Future Memory Leaks
Java 19 introduces “Virtual Threads”, which is a new kind of thread on the JVM. One big difference with regular Threads is that it’s unidiomatic to create thread pools:
[Using synchronization constructs] is more effective and convenient than thread pools [of Virtual Threads], and is also more secure since there is no risk of thread-local data accidentally leaking from one task to another.
The designers of Clojure were well aware of this security risk when designing Futures and Agents, which both manage pools of regular Threads for running arbitrary tasks. Before running a new task on a pooled Thread, all thread-local data on that Thread is cleared.
This prevents data leakage, however Futures can leak memory, as reported by António Monteiro. The problem is that Futures don’t clear thread-local data after a task is completed. That means any thread-local data that you convey to a Future is referenced by the Thread until the next task is run on the same Thread. If this never happens and you were banking on your thread-local data being garbage collected, you might be in for a nasty shock.
I believe the same problem happens with Agents, but I don’t think there is a reproduction or ticket yet. A closed ticket from 2012 even seems to predict this, but the assessment that it’s not Clojure’s problem leads me to think this is referring to a different issue:
There is still the opportunity for memory leaks if agents or futures are used, and the executors used for them are not shutdown when the app is undeployed. That’s a solvable problem, but should probably be solved by the containers themselves (and/or the war generation tools) instead of in clojure itself.
I attempted to fix both problems at once but I quickly abandoned the Agents fix. I speculated on what might need to be done to fix this memory leak in Agents:
The fix for agents is different since thread bindings must be propagated to error-handlers (and watchers?). After an attempt, the fix for agents can probably be safely done in another issue, as the entirety of the fix probably resides in the Agent implementation (not any code touched in this issue).
I have proposed a patch resolving this memory leak in Futures, which is awaiting review.
If you want to use it now, you’re in luck! It is available today via clearing-future in fully-satisfies. There are also drop-in replacements for future and future-call.