This site is archived.

distributed computing

How to do parallel processing and map/reduce with Gearman.

Giuseppe Maxia 14 June 2010
Type:  Not planned session

Parallel processing, in any programming language, is hard to achieve and easy to get wrong.
However, you can leverage a distributed client/server paradigm to create parallel processed tasks quite easily, with a scalable and fail-resistant system. With Gearman you can run tasks in parallel, no matter where they should be performed.

A common scenario in web development is a delay in serving a page, due to
the need of performing several unrelated operations in sequence.
Sequential development is the norm in every major programming language,
and, although parallel processing is possible, implementing it in your
application is often painful and results in complex and debug unfriendly code.
Enters Gearman, a distributed client/server framework, which can be easily
integrated in any programming language, including SQL (through a MySQL User
Defined Function), and makes parallel processing a trivial task. This