MongoQueue is a PHP queue that allows for moving tasks and jobs into an asynchronous process for completion in the background. The queue is managed by Mongo
MongoQueue is an extraction from online classifieds site Oodle. Oodle uses MongoQueue to background common tasks in order to keep page response times low.
- mongod 1.5.4+
- PHP 5.3+
- Mongo PECL (PHP mongo driver)
Extract the source files into a directory in your PHP library path. Because the queue is backed by Mongo, there is no need to migrate or set up a table for MongoQueue. It will use the mongo_queue collection in a database of your choice.
Before any MongoQueue actions are taken you’ll need to initialize the MongoQueue class with the proper settings:
$mongo = new Mongo('mongodb://host:port', array('connect' => false));
MongoQueue::$connection = $mongo;
MongoQueue::$database = 'my_database';
Jobs are PHP objects that extend from MongoJob. Object states are not serialized so jobs must be defined as public static function.
require_once('MongoQueue/lib/MongoJob.php');
class QueueTracer
extends MongoJob
{
public static $context;
public static function trace()
{
echo "Tracer hit\n";
}
}
MongoQueue::push('QueueTracer', 'trace', array(), time() + 500); # run QueueTracer 500 seconds later
You can also take advantage of @MongoJob@’s built in later() method which automatically delays a method for asynchronous running:
QueueTracer::later(500)->trace();
You can also batch calls which have the same parameter so that they’ll only be run once by workers. The run time will be decided by the first job inserted and the subsequent jobs will not have their run times obeyed.
QueueTracer::batchLater(500)->trace();
MongoQueue also supports prioritization. Priorities are ascending and null (the default) is considered to have the lowerst priority.
QueueTracer::later(500, false, /*priority:*/ 1)->trace();
Jobs are run via MongoQueue::run(). Before running job, you’ll need to set two extra MongoQueue initializers: environment and, optionally, context. MongoQueue currently does not include a daemon runner, but here is an example runner:
$mongo = new Mongo('mongodb://host:port', array('connect' => false));
MongoQueue::$connection = $mongo;
MongoQueue::$database = 'my_database';
MongoQueue::$environment = 'classes/jobs';
MongoQueue::$context = array('context' => array('foo', 'bar'));
if (MongoQueue::run())
echo("Found a job to run");
else
echo("Nothing to run");
$environment is the path under which jobs are automatically loaded by the runner. The convention is that if your job class is called FooBar then the file that is loaded is $environment/FooBar.php
Any keys set in $context are available as static variables inside the Job class. (e.g. ‘context’ in the QueueTracer example)
By default, MongoQueue will use priorites (which are all null by default). If you want to run the next job regardless of priority, you can pass false as the third param to run. e.g:
MongoQueue::run(null, null, false)
You’ll want to add indices to MongoQueue to ensure high performance. You’ll need at least the following index for basic operation:
db.mongo_queue.ensureIndex({locked: 1, when: 1, priority: 1});
If you plan to use the batching feature you want an index on the type of jobs already in the queue:
db.mongo_queue.ensureIndex({object_class: 1, object_method: 1, parameters: 1, locked: 1});
MongoQueue comes with built-in support for manual sharding. Note that due to the key restriction on updates for auto-sharding, the ability to scale with only auto-sharding is limited.
If you initialize MongoQueue::$connection to an array instead of a single Mongo connection object, MongoQueue will insert or pull jobs at random from the set of connections. This way, you can have dedicated workers per physical machine and insert in a distributed way (or vice versa). For example:
MongoQueue::$connection = array($mongo1, $mongo2, $mongo3, $mongo4);
MongoQueue will send batchable jobs to the same connection so that batched jobs with identical parameters will not be distribued across multiple mongo instances.
All jobs are stored as Mongo documents in the mongo_queue collection. MongoQueue does not come with any built in administrative scripts, but you can view the queue via your mongo console. locked_at will tell you whether or not a job is currently being processed.