PERMALINK

Using test assertions in a multithreaded test often seems to work by magic or silently break. I found that even Clojure’s own test suite has trouble getting it right. Let’s dig into how to do it correctly.

clojure.test uses thread-local state to record the progress of a unit test, and is used by test assertions (is forms) to report test results. Without this state, it can lead to test failures that don’t fail the overall test suite:

FAIL in foo
...
ERROR in bar
...
Ran 15 tests, 0 failures, 0 errors

Always ensure test assertions are run in a thread that has thread-local bindings conveyed from the main deftest thread. The simplest example is to run in the deftest thread itself.

(deftest my-test
  (is nil))

Binding conveyance means future will work seamlessly in this respect:

(deftest my-test
  @(future (is nil)))

For other threading constructs, use bound-fn instead of fn to get the same effect:

(deftest my-test
  (doto (Thread. (bound-fn [] (is nil)))
    .start .join))

testing contexts will also be preserved with these strategies.

(deftest my-test
  (testing "foo"
    @(future (testing "bar"
               (is nil)))))
; FAIL in foo bar
; actual: nil

Another robust alternative is to use an atom to accumulate results, and then test it in the main thread at the end.

(deftest my-test
  (let [reqs (atom [])
        resp (http/stub {:get (fn [req]
                                ;; don't assert here...
                                (swap! reqs conj req)
                                {:status 200, ...})}
               (http/get "/my-url" ...))]
    ;; ...assert here instead
    (is (= [{:request :get, :url "/my-url", ...}] @reqs))
    (is (= {:status 200, ...} resp))))

Personally, I prefer this atom approach. Binding conveyance can sometimes seem to be supported by accident without being part of specification of function: it’s barely mentioned in the docs for future, and almost never in functions that wrap it.

Notice that in these examples, all test assertions complete before the deftest finishes. The following assertion may be executed after a testing report has been generated.

(deftest my-test
  (future (is nil)))
; Ran 1 test, 0 failures, 0 errors
; ...
; FAIL, actual: nil

I made sure to preserve all these invariants in my recent clojure.test enhancement proposal. It has now been proposed upstream. With this patch, the following no longer reports a generic message, but actually mentions the testing context (even with nested futures).

(deftest my-test
  (testing "foo"
    @(future (testing "bar"
               (throw (ex-info "an error" {}))))))
; ERROR in my-test
; foo bar
; #error {...}

To summarize:

1. convey thread bindings from the deftest thread to any threads that perform test assertions.
2. await asserting threads before ending your deftest.