I'm looking for a way to manage asynchronous, potentially long-running PHP jobs (ex. feed import and generation). One could start with a simple cron-based system, but this has a few drawbacks :
* Too much 'downtime' if you set the sample interval too long
* Overlapping jobs if jobs don't finish
* No ability to handle events or dependency issues
I've previously used AppWorx (appworx.com) in a large ETL operation. AppWorx is quite sophisticated with a distributed slave architecture, chains, forks and joins. But AppWorx is probably too complex for my needs.
In my simplest case I would need a queue manager so I could keep the queue full without overlapping jobs. I also have modest needs for distributed processing.
For my purposes, I thought about using Hudson. Although Hudson is a job manager, it's really meant to manage build jobs. Not the best for the types of jobs I'm looking to process.
Next I looked at Zend Server. It comes from PHP-based company Zend, so the PHP integration should be good. Here's an article regarding queues in Zend Server :
Unfortunately, there's no queue facility in Zend's free Community Edition :
Then I found Gearman. Gearman is a free job manager that seems to be well suited for PHP-based jobs :
Gearman is used at large sites (Yahoo and Digg). Here are articles on Gearman for feed processing, and Gearman for PHP applications :
A more general overview of job schedulers can be found here :