Queue-Automator is a super simple multi-node queue module/server for nodejs. to distribute work to clients easily. Its easy to setup, and even easier to add more clients ad-hoc when you need it.

##Described by example Here is a short example of a situation where this can be used:

  • Lets say you have a web site that is taking competition entries.
  • For each entry, you need to process something, for example compose an image.
  • The composed image takes a little while to do
  • Since your campaign will potentially have many users, you need to be able to process lots of images quickly.

You can:

  1. Let them wait until each image is processed by the server
  2. Let them wait, but use a queuing system on one computer
  3. Let them wait, but distribute the queue on many computers

What queue-automator does is answer with 3

How to use it

First install it

npm install queue-automator

Second, create one server, which holds the queue and distributes it. It needs to have a callback for when a queue is added (which happens trough a request to http://localhost:20301/add?......).

// queue-server-test.js

var QA = require('queue-automator');
// starts the server
var server = QA.server({ port: 20301 });

// Add a callback for each time a queue item is supposed to be added
server.setAddCallback(function(data, next) {
    // data.key is just something we use to identify the process. The "data" comes when something is added to the queue, more on this further down.
    next(null, data.key, {foo: data.foo}); 

// Add a callback whenever an item has been processed by a client
server.setProcessCallback(function(key, data, next) {
    // Do whatever you need to do with the processed data. 
    console.log(key, data);

Then you need at least one client to process the queue:

// queue-client-test.js

var QA = require('queue-automator');

// connect to the server
var client = QA.client( {url: "http://localhost", port: 20301 });

// set a callback for processing a queue item
client.setProcessCallback(function(key, data, next) {
    // Do whatever you need to do when a queue item needs to be processed
    next({ foo2: data.foo + " done"});

And now add an item to the queue by calling the :

curl http://localhost:20301/add?key=1&foo=baq

Now, when the server gets an item with the request, it takes the data and passes it to the queue. As soon as the server has something in the queue, it will distribute it to the next free client.

The client will process it, and tell the server its done.


Queue-Automator has a simpel UI to watch how the queues are going. Just visit http://localhost:20301/info.

More info

You can see a working example at https://github.com/camme/node-queue-automator