Movable Type features a new capability that enables publishing your blog in the background. Depending on your specific circumstances, requirements and system configuration, the Publish Queue may help you reduce the time your users (both authors/administrators and site commenters) wait while Movable Type publishes files to your file system, and improve overall system performance.

System Requirements

• There are no special system requirements for running Movable Type's Publishing Queue.
• To execute run-periodic-tasks on a predefined schedule, users will need access to a scheduling framework like cron in Unix or Window's scheduled task framework.

Using Publish Queue

Using the Publish Queue requires you to:

• Configure your blog to use the queue
• Run the periodic tasks script as a cron, or as a daemon

Configure your blog to use the Publish Queue

Each blog in your system can be configured to use the Publish Queue. To configure your blog to use the Publish Queue:

Movable Type 4.2 and Movable Type 5

• Edit the template you want to be published in the background via Publish Queue.

• Select "Via Publish Queue" under the Publishing pull down menu under "Template Options."

Once you have configured your blog to use the Publish Queue, the system will no longer automatically publish pages when a user creates a new entry, leaves a comment or does any other action that requires Movable Type to publish files. Instead, jobs for those tasks will be placed in the job queue.

Running the run-periodic-tasks script

Jobs are processed from the queue with the script run-periodic-tasks. This is shipped with Movable Type in the tools directory.

» See Setting up run-periodic-tasks.pl

Running multiple instances of the Publish Queue

Movable Type allows you to setup multiple instances of the run-periodic-tasks daemon. This allows administrators to more easily scale the capacity and speed of the Publish Queue. There are a few things to note when running multiple instances of the Publish Queue:

• Users are free to run multiple instances of run-periodic-tasks on a single machine, as well as running multiple instances across multiple servers.
• When running multiple instances concurrently, it is recommended that at least one daemon be instructed to process jobs in a more random manner by using the --randomly flag on run-periodic-tasks (supported in Movable Type 4.2 and later).

How the Publish Queue works

The Publish Queue consists of two key components:

• The publishing job queue
• The publishing worker(s)

The publishing job queue

If you have enabled and configured Publish Queue, when a user saves an entry, leaves a comment, or performs any action that would publish new content to your blog, the system puts that publish action into a queue. Jobs in this queue are then processed by one or more publishing worker(s).

The publishing job queue is managed in your Movable Type database in tables prepended with mt_ts.

The publishing worker(s)

The publishing worker is launched via the script run-periodic-tasks and is responsible for processing jobs in the queue. Parameters can be passed to run-periodic-tasks when it is invoked to modify and fine tune its behavior according to your specific needs.

Behind the scenes: The Schwartz

The publish queue and its two primary components -- the job queue and the worker -- is actually an open source set of tools built and maintained by Six Apart called The Schwartz.

The Schwartz is taglined "a reliable job queue system" (and named after an engineering manager at Six Apart) and was originally developed as a generic job processing system for Six Apart's hosted services. It is used in production today on TypePad, LiveJournal and Vox for managing tasks that can be performed by the system without user interaction.

The Schwartz follows a very simple request cycle:

• A user (or the application itself) makes a request which the application decides to put into the Schwartz job queue. (In the case of Movable Type's publish queue, this would be a file publish request.)

• The application registers a Schwartz job in the Schwartz database. In Movable Type, Schwartz jobs are stored in the Movable Type database with the table prefix mt_ts.

• A worker, typically running as a daemon, grabs one or more Schwartz jobs off of the queue in a relatively random manner. In Movable Type, a worker is managed through the script run-periodic-tasks. Therefore multiple workers can be deployed by spawning multiple instances of the run-periodic-tasks script. Starting in Movable Type 4.2, jobs are pulled off the queue according to their assigned priority. Users wishing to process jobs random can use the --randomly flag on `run-periodic-tasks.pl script.**

• The worker completes its assigned work, and when it is complete removed the job from the queue completely. If the worker fails in completing the task then the job is requeued to be processed again at some point in the future (typically 5 minutes later).