I’m happy to share my new Clojure project – Byplay. It’s a background job queue on top of PostgreSQL 9.5. Check it out at github.com/metametadata/byplay.
Byplay allows creating background jobs, placing those jobs on multiple queues, and processing them later. Background jobs can be any named Clojure function.
The project is mostly inspired by Que, Resque and Celery.
It may be inefficient to store a queue in a relational database. But it allows using transactions and doesn’t bloat your project with a dedicated queue storage in case you already have PostgreSQL (YAGNI). I believe it should be sufficient for a lot of projects. For instance, this approach worked for GitHub for several months while they were using delayed_job.
Features
- Durability: queue can survive app restarts because it is stored inside a PostgreSQL table. All done and failed jobs are left in a table so that at any time user is able inspect, retry or purge them manually.
- Embedment: queue consumption worker can be easily started in a background thread.
- Parallelism: queue can be consumed by several threads on different machines to better utilize multiple CPU cores. The parallel queue consumption is based on a new
FOR UPDATE/SKIP LOCKED feature from PostgreSQL 9.5. - Transactional guarantees:
- Every job is executed inside its own database transaction.
- If a job is marked done than all its database statements have been committed.
- In case of exception inside a job all job’s database statements are rolled back and a job is marked failed. Thus if a job is marked new or failed then none of its database statements have been committed yet.
- Scheduling can be executed in a transaction where a data needed for a job is committed. So that a worker will not pick up the job before the database commits.
- Multiple queues: jobs can be scheduled to different queues/tags. E.g. you can schedule heavy jobs into a separate “slow” queue/worker in order to not block an execution of more important jobs from a “light” queue.
- Fewer dependencies: if you already use PostgreSQL, a separate queue (Redis, RabbitMQ, etc.) is another moving part that can break.
- Small: the implementation with docstrings is less than 300 LOC.
Anti-Features
It hasn’t been proven yet, but Byplay can experience the problem described in Que docs:
Que’s job table undergoes a lot of churn when it is under high load, and like any heavily-written table, is susceptible to bloat and slowness if Postgres isn’t able to clean it up. The most common cause of this is long-running transactions, so it’s recommended to try to keep all transactions against the database housing Que’s job table as short as possible. This is good advice to remember for any high-activity database, but bears emphasizing when using tables that undergo a lot of writes.
This PostgreSQL issue is explained in more detail in the article “Postgres Job Queues & Failure By MVCC”.
metametadata 