Byplay – Clojure background job queue on top of PostgreSQL

I’m happy to share my new Clojure project – Byplay. It’s a background job queue on top of PostgreSQL 9.5. Check it out at

Byplay allows creating background jobs, placing those jobs on multiple queues, and processing them later. Background jobs can be any named Clojure function.

The project is mostly inspired by Que, Resque and Celery.

It may be inefficient to store a queue in a relational database. But it allows using transactions and doesn’t bloat your project with a dedicated queue storage in case you already have PostgreSQL (YAGNI). I believe it should be sufficient for a lot of projects. For instance, this approach worked for GitHub for several months while they were using delayed_job.


  • Durability: queue can survive app restarts because it is stored inside a PostgreSQL table. All done and failed jobs are left in a table so that at any time user is able inspect, retry or purge them manually.
  • Embedment: queue consumption worker can be easily started in a background thread.
  • Parallelism: queue can be consumed by several threads on different machines to better utilize multiple CPU cores. The parallel queue consumption is based on a new
    FOR UPDATE/SKIP LOCKED feature from PostgreSQL 9.5.
  • Transactional guarantees:
    • Every job is executed inside its own database transaction.
    • If a job is marked done than all its database statements have been committed.
    • In case of exception inside a job all job’s database statements are rolled back and a job is marked failed. Thus if a job is marked new or failed then none of its database statements have been committed yet.
    • Scheduling can be executed in a transaction where a data needed for a job is committed. So that a worker will not pick up the job before the database commits.
  • Multiple queues: jobs can be scheduled to different queues/tags. E.g. you can schedule heavy jobs into a separate “slow” queue/worker in order to not block an execution of more important jobs from a “light” queue.
  • Fewer dependencies: if you already use PostgreSQL, a separate queue (Redis, RabbitMQ, etc.) is another moving part that can break.
  • Small: the implementation with docstrings is less than 300 LOC.


It hasn’t been proven yet, but Byplay can experience the problem described in Que docs:

Que’s job table undergoes a lot of churn when it is under high load, and like any heavily-written table, is susceptible to bloat and slowness if Postgres isn’t able to clean it up. The most common cause of this is long-running transactions, so it’s recommended to try to keep all transactions against the database housing Que’s job table as short as possible. This is good advice to remember for any high-activity database, but bears emphasizing when using tables that undergo a lot of writes.

This PostgreSQL issue is explained in more detail in the article “Postgres Job Queues & Failure By MVCC”.

How to package a Clojure desktop app for Mac OS X?

This is something I unexpectedly couldn’t google quickly. The short answer is:

  1. build uberjar;
  2. package it using javapackager.

Javapackager can be used to create .app, .pkg, .dmg files, and even mac.appStore ones.

Here’s an example of creating an executable .app bundle for my application:

$ lein uberjar
Compiling pickings.core
Compiling pickings.keylistener
Compiling pickings.logic
Compiling pickings.main
Compiling pickings.mvsa
Compiling pickings.ui
Created /Users/yuri/dev/pickings/target/pickings-0.1.0-SNAPSHOT.jar
Created /Users/yuri/dev/pickings/target/pickings-0.1.0-SNAPSHOT-standalone.jar

$ /Library/Java/JavaVirtualMachines/jdk1.8.0_31.jdk/Contents/Home/bin/javapackager -deploy -native image -outdir out -outfile -srcfiles target/pickings-0.1.0-SNAPSHOT-standalone.jar -appclass pickings.main -name "Pickings" -title "Pickings" -Bruntime= -Bicon=resources/icon.icns
Package is configured to ship without a JRE.
Creating app bundle: /Users/yuri/dev/pickings/out/bundles/


Elm-ish architecture in ClojureScript

Implemented Elm-ish architecture examples in ClojureScript using Reagent, core.match and Specter.

GitHub Repo

Here’s how a counter example looks like:

(ns frontend.counter
(:require [frontend.ui :as ui]
[cljs.core.match :refer-macros [match]]
[reagent.core :as r]))
(defn init
"Creates a model intance."
(defn control
"Non-pure signal handler.
Based on current model snapshot and received signal it can dispatch an action further to reconcile."
[_model_ signal dispatch]
(match signal
:on-connect nil
:on-increment (dispatch :increment)
:on-decrement (dispatch :decrement)))
(defn reconcile
"Pure function. It returns a new model based on current model snapshot and received action."
[model action]
(match action
:increment (inc model)
:decrement (dec model)))
(defn view-model
"Pure function. Given a model snapshot returns an immutable value for view to display."
(str "#" model))
(defn view
"Pure function. View is given an immutable view-model and a signal dispatching function."
[view-model dispatch]
[:button {:on-click #(dispatch :on-increment)} "+"]
[:span view-model]
[:button {:on-click #(dispatch :on-decrement)} ""]])
(defn view-with-remove-button
"dispatch-on-remove is passed because it's up to container to decide how to handle removing."
[view-model {:keys [dispatch dispatch-on-remove] :as _context_}]
[:button {:on-click #(dispatch :on-increment)} "+"]
[:span view-model]
[:button {:on-click #(dispatch :on-decrement)} ""]
[:button {:on-click #(dispatch-on-remove)} "X"]])
(defonce model (r/atom (init 1)))
(defn example
(ui/connect model view-model view (ui/wrap-log-signals control) (ui/wrap-log-actions reconcile)))

view raw
hosted with ❤ by GitHub

Portable Clojure/ClojureScript exception assertions

I found it not convenient to use (is (thrown-with-msg? …)) form in .cljc unit tests. Because codebase usually depends on different exception classes depending on host’s language.

So I used these macros instead:

(defn- cljs-env?
"Take the &env from a macro, and tell whether we are expanding into cljs.
(boolean (:ns env))))
(defmacro is-exception-thrown
"(is (thrown-with-msg? …)) for specified exceptions in Clojure/ClojureScript."
[clj-exc-class cljs-exc-class re expr]
(let [is (if (cljs-env? &env) 'cljs.test/is
exc-class (if (cljs-env? &env) cljs-exc-class
`(~is (~'thrown-with-msg? ~exc-class ~re ~expr)))))
(defmacro is-error-thrown
"(is (thrown-with-msg? …)) for general exceptions in Clojure/ClojureScript."
[re expr]
`(is-exception-thrown java.lang.Exception js/Error ~re ~expr)))
(defmacro is-assertion-error-thrown
"(is (thrown-with-msg? …)) for assert exceptions in Clojure/ClojureScript."
[re expr]
`(is-exception-thrown java.lang.AssertionError js/Error ~re ~expr)))
;; example
#"expected exception"
(throw (ex-info "expected exception" {})))

view raw
hosted with ❤ by GitHub

Error handling in core.async

Played around using core.async in ClojureScript for coordinating async AJAX requests and app state updates. See this gist for more code and explanation.

(defrecord Controller [state router]
  (on-navigation [_ file-id]
          (<?initialize-state! state)
          (<?set-current-file! state file-id router))
        (catch js/Object e
          (println "!!! ERROR - on-navigation - caught"
                   (if (channel-error? e)
                     (str "error from some channel, error data: " (channel-error-data e))
                     (str "js exception: " e)))))))
  (on-change-current-file [_ file-id]
        ; <? is needed to wait for result or raise an exception
        (<? (<?set-current-file! state file-id router))
        (catch js/Object e
          (.error js/console "on-change-current-file exception:" e))))))