---
title: "Managed and Functional Languages on the Desk"
book: "Low-Latency Software"
subject: quant
language: en
chapter: 10
exercises: 8
source: https://one-course.com/books/quant/13/en/chapter/10-managed-and-functional-languages-on-the-desk
---

# Chapter 10 — Managed and Functional Languages on the Desk

In 2011 a retail exchange described in public how its matching logic processed six million orders a second on a single thread, written in Java (Fowler, 2011). Its engineers had not escaped the [garbage collector](#def-ll-managed-and-functional-languages-on-the-desk-gc); they had designed around it: everything pre-allocated, ring buffers of millions of slots created at start-up, no locks in the business logic. A few years earlier a proprietary trading firm had described why it wrote its critical trading systems in OCaml, a functional language with a [garbage collector](#def-ll-managed-and-functional-languages-on-the-desk-gc) of its own (Minsky and Weeks, 2008). Managed languages are on trading desks, sometimes on the hot path. This chapter explains what their runtimes do that C++ and Rust do not, how a latency-sensitive program written in them is structured, and what the firms that use them have said in public about why.

## 10.1 Garbage collection and its pauses

**Definition 10.1 (Garbage collector, stop-the-world pause).**

A *garbage collector* reclaims memory that the program can no longer reach, by tracing references from the program’s roots (stacks, globals). Most divide the heap into generations: new objects in a small young generation, collected often and cheaply; survivors promoted to an old generation, collected rarely and at a cost that grows with its size. A *stop-the-world pause* is an interval during which the collector has stopped every application thread.

The trade is explicit: the program never frees memory and never frees it twice; in exchange, it runs on a schedule set by its allocation. Modern collectors shorten pauses by doing most of their work concurrently with the program; a widely used Java collector aimed first at pauses under ten milliseconds and later at under one millisecond spent at the points where all threads stop (dated box below). Either figure is enormous against a microsecond budget, and the concurrent work itself takes cores and memory bandwidth from the program.

![A generational heap. Allocation fills the young generation, whose collections cost in proportion to what survives; survivors move to the old generation, whose full collections scan everything alive, and stop or slow the program for longer as the heap grows.](https://one-course.com/images/onecourse/chapters/quant-13/ll-managed-and-functional-languages-on-the-desk/fig-c97a8ab1068a.svg)

***Figure 10.1.** A generational heap. Allocation fills the young generation, whose collections cost in proportion to what survives; survivors move to the old generation, whose full collections scan everything alive, and stop or slow the program for longer as the heap grows.*

**As of September 2026 — Pause goals of a modern Java collector.**

OpenJDK’s ZGC was proposed (JEP 333) with the goal that garbage-collection pause times should not exceed 10 milliseconds, its stop-the-world phases limited to scanning roots. A later proposal (JEP 376) moved the scanning of thread stacks to a concurrent phase, with the expectation that less than one millisecond be spent inside the collector’s safepoints on typical machines.

## 10.2 The low-latency managed style: pre-allocation, off-heap memory, warm-up

**Definition 10.2 (Off-heap memory).**

*Off-heap memory* is memory a managed program uses outside its collected heap (direct byte buffers, memory-mapped files, native arrays), laid out by the program and invisible to the collector: it is never scanned, moved or freed by it.

The style that the exchange above described follows from the collector’s economics. Allocate nothing per message after start-up: pre-allocate every object the hot path needs and reuse it (the pools of chapter 6); keep large, long-lived data (order books, rings, journals) off the heap or in flat arrays of primitives, so that full collections have little to scan; size the young generation so that the little garbage that remains fills it rarely; and run a quiet collector. What is left is a program whose collector has nothing to do during the trading day. The limit of the style is its brittleness: one library call that allocates, one boxed integer, puts the collector back on the path, and nothing in the language says so; the allocation must be measured, as in chapter 6.

**Proposition 10.3 (The garbage budget).**

A program that allocates $b$ bytes per message at $r$ messages a second, with a young generation of $Y$ bytes, collects its young generation every $Y/(rb)$ seconds. It collects at most $k$ times in a session of length $T$ if and only if $b \le kY/(rT)$.

**Proof.** Allocation fills the young generation at $rb$ bytes a second and each collection empties it, so collections occur every $Y/(rb)$ seconds and number $rbT/Y$ in the session; $rbT/Y \le k$ is the stated condition. ∎

With a young generation of 2 GiB, 200 000 messages a second and a session of six and a half hours, the budget for one collection a session is under half a byte per message: in practice, zero.

## 10.3 Just-in-time compilation and deoptimisation

**Definition 10.4 (Just-in-time compilation, warm-up period).**

In *just-in-time compilation* a runtime first interprets or lightly compiles a program, measures which code is hot and how it behaves, and then compiles the hot code to optimised machine code using that profile. The *warm-up period* is the time, or number of executions, before the hot path runs as optimised code.

**Definition 10.5 (Deoptimisation).**

*Deoptimisation* is the runtime’s fall-back from optimised code to a slower tier when an assumption the optimiser made from the profile is violated (a call site that had seen one type sees another, a branch never taken is taken); the code may later be recompiled.

A just-in-time compiler is [profile-guided optimisation](https://one-course.com/books/quant/13/en/chapter/8-c-for-latency-iii-the-compiler#def-ll-cpp-for-latency-iii-the-compiler-pgo) (chapter 8) done while the program runs, with the same risk: the profile is whatever the program has seen so far. A trading process that starts in the morning is slowest on its first messages, exactly when the open is busiest, and a rare path taken for the first time at a volatile moment (an unusual message type, a halt) may trigger [deoptimisation](#def-ll-managed-and-functional-languages-on-the-desk-deopt) just then. The standard remedies are warm-up by replaying synthetic or recorded traffic through the real code before trading (with orders sent to a simulator, never the market), and keeping the hot path’s types and branches stable. Ahead-of-time compilation removes the warm-up at the cost of the profile.

## 10.4 The functional-language shops

The proprietary firm of the opening described its choice in a journal paper: OCaml as its primary development language, over twenty OCaml programmers and hundreds of thousands of lines of code, used for critical trading systems, research, systems software and administration, valued because it let the firm “rapidly produce readable, correct, efficient code” (Minsky and Weeks, 2008). The argument was not speed per message but correctness per engineer: an expressive type system that encodes invariants (an order’s state as a variant whose cases the compiler checks are all handled), immutable data by default, and code short enough to be reviewed by people who trade with it. The same firm’s later essay argued that the next language a programmer learns should be functional (Minsky, 2011). OCaml’s collector is generational too, with a small minor heap collected often and a major heap collected incrementally; latency-critical OCaml, like latency-critical Java, is written to allocate little on the hot path.

## 10.5 Choosing a language for a desk

The choice is less binary than debates suggest. Several languages usually coexist in one firm: C++ or Rust where nanoseconds are traded, a managed or functional language for the large body of logic whose [latency budget](https://one-course.com/books/quant/13/en/chapter/1-where-latency-comes-from#def-ll-where-latency-comes-from-budget) is microseconds to milliseconds, and Python for research and operations (One Quant Book 15, chapter 8). The questions that decide are measurable: the [latency budget](https://one-course.com/books/quant/13/en/chapter/1-where-latency-comes-from#def-ll-where-latency-comes-from-budget) at the percentile that matters; the allocation discipline the team can sustain and test; the cost of the bugs each language prevents or permits; and the size and experience of the team. In Python, this book’s research language, the lessons of this chapter apply directly, as the tutorial shows.

## 10.6 Tutorial: the collector in a Python message loop

**Goal.** Watch CPython’s collector interrupt a message loop, and remove it by pre-allocation. **End state:** [Figure 10.2](#fig-ll-managed-and-functional-languages-on-the-desk-gc).

1. **Two books.** One keeps each live order as a Python dictionary, adding two orders for each one removed, as a book fills during the morning; the other writes orders into a pre-allocated NumPy record array. `def dict_book (): book = {} def step (k): book[k] = {" id " : k, " px " : 1_000_000 + k % 7 , " qty " : 100 , " fills " : [k]} if k % 2 == 0 and k - 500 in book: del book[k - 500 ] return step def pooled_book (capacity=1 << 18 ): pool = gw.RecordPool(capacity, gw.MESSAGE) buf = pool.buf def step (k): i = k % capacity buf[" ts " ][i] = k buf[" price " ][i] = 1_000_000 + k % 7 buf[" qty " ][i] = 100 return step` **Listing 10.1.** A book of dictionaries and a pre-allocated book. code/low-latency/10-managed-and-functional-languages-on-the-desk/python/ll_gc.py
2. **Time the collector.** `GcWatch` registers a callback that CPython calls before and after every collection, and records each pause, by generation, in `firm.lathist`. `class GcWatch : def __init__(self ): self .pauses = {g: LatHist() for g in range (3 )} self .collections = {0 : 0 , 1 : 0 , 2 : 0 } self .total_ns = 0 self ._t0 = 0 def _cb (self , phase, info): if phase == " start " : self ._t0 = time.perf_counter_ns() else : dt = time.perf_counter_ns() - self ._t0 g = info[" generation " ] self .pauses[g].record(dt) self .collections[g] += 1 self .total_ns += dt def __enter__(self ): gc.callbacks.append(self ._cb) return self def __exit__(self , *exc): gc.callbacks.remove(self ._cb) return False` **Listing 10.2.** Timing every collection through the gc module’s callbacks. code/firm/gcwatch/firm_gcwatch.py
3. **Measure the loop** over 200 000 messages with `python bench_gc.py` ; the loop stores timestamps only, so that nothing but the book’s own work happens between two of them.

**What to change next.** Keep the dictionary book but call `gc.freeze()` after the warm-up and again every 100 000 messages, and see what happens to the full collections; bound the dictionary book at 1 000 live orders and explain why the collector then never runs.

![A Python message loop over 200 000 orders: a book of dictionaries, which triggered 259 young, 23 intermediate and 2 full collections, and a pre-allocated NumPy book, which triggered none. The dictionary book’s maximum is its full collection. CPython 3.10. Measured on a laptop (Intel Core Ultra 7 155H) under WSL2, no isolated cores. Data: bench_gc.py.](https://one-course.com/images/onecourse/chapters/quant-13/ll-managed-and-functional-languages-on-the-desk/fig-2e0c5e61d1cc.svg)

***Figure 10.2.** A Python message loop over 200 000 orders: a book of dictionaries, which triggered 259 young, 23 intermediate and 2 full collections, and a pre-allocated NumPy book, which triggered none. The dictionary book’s maximum is its full collection. CPython 3.10. Measured on a laptop (Intel Core Ultra 7 155H) under WSL2, no isolated cores. Data: `bench_gc.py`.*

The result has two halves, and both matter. The pre-allocated book removed the collector: no collection in 200 000 messages, and a tail more than twenty times shorter at its maximum. It is not faster at the median: writing three fields of a NumPy array from Python costs more than creating a small dictionary. And the dictionary book’s worst message took as long as its last full collection, some twenty milliseconds, which scanned every order still alive. CPython adds a twist of its own: its collector counts allocations minus deallocations, so a loop whose live set stays bounded never triggers a collection at all; a book that grows does.

## 10.7 Build: collector watch and message pool

**Purpose.** The firm’s Python services (research servers, reporting, the simulator drivers of One Quant Book 10) measure their collector and keep it off their message loops.

**Interface.** Python `firm_gcwatch`: `GcWatch()` as a context manager with `pauses[gen]` (`LatHist`, nanoseconds), `collections[gen]`, `total_ns`; `RecordPool(n, dtype)` with `next()` and `buf`; `MESSAGE` (timestamp, price, quantity, side); `steady_state(step, n, warm)`.

**Rules.** The callback is removed when the context ends, even on error; `steady_state` freezes the heap after warm-up and unfreezes it afterwards; the pool never grows.

**Acceptance tests.** `code/firm/gcwatch/tests/`: a pooled loop of 50 000 messages causes no collection; a bounded churn causes none either; a growing book causes young collections, each timed.

**Stretch.** Report the allocation rate (bytes a message) with `tracemalloc` in a test build; alarm when a full collection happens during market hours.

Sources and further reading

- M. Fowler, “The LMAX architecture”, 12 July 2011.
- Y. Minsky and S. Weeks, “Caml trading: experiences with functional programming on Wall Street”, *Journal of Functional Programming* 18(4), 2008; Y. Minsky, “OCaml for the masses”, *ACM Queue* , 2011.
- OpenJDK, JEP 333 (ZGC) and JEP 376 (concurrent thread-stack processing).
- Python documentation, module `gc` .

## 10.8 Exercises

**Exercise 10.1 ★.**

A Java engine allocates 64 bytes per message at 100 000 messages a second with a 1 GiB young generation. How often does it collect the young generation, and how many times in a 6.5-hour session?

**Solution of Exercise 10.1.**

$64 \times 100\,000 = 6.4$ MB a second fills 1 GiB in 167.8 seconds: about 139 young collections in a 6.5-hour session ([Proposition 10.3](#prop-ll-managed-and-functional-languages-on-the-desk-budget)).

**Exercise 10.2 ★.**

A collector’s pause is $1\,\mathrm{m}\mathrm{s}$. How many messages arriving at 200 000 a second queue behind it, and how long does the last of them wait?

**Solution of Exercise 10.2.**

$0.001 \times 200\,000 = 200$ messages; the last arrives just before the pause ends and the first waits about a millisecond, and all of them are then processed late in turn, so the queue takes a further 200 processing times to drain.

**Exercise 10.3 ★.**

From [Figure 10.2](#fig-ll-managed-and-functional-languages-on-the-desk-gc), by how much does the pooled book shorten the maximum and the p99.9, and how does its median compare?

**Solution of Exercise 10.3.**

On the committed measurement the maximum falls from about $19.6\,\mathrm{m}\mathrm{s}$ to $0.58\,\mathrm{m}\mathrm{s}$, some thirty times, and the p99.9 from about $26\,\text{µ}\mathrm{s}$ to $1.7\,\text{µ}\mathrm{s}$; the median is slightly worse, $335\,\mathrm{n}\mathrm{s}$ against $313\,\mathrm{n}\mathrm{s}$.

**Exercise 10.4 ★★.**

Why does a Python loop that keeps the last 1 000 orders in a bounded deque never trigger the collector, while a growing book does?

**Solution of Exercise 10.4.**

CPython triggers a young collection when allocations of tracked objects exceed deallocations by the threshold (700 by default on Python 3.10). A bounded deque frees an old order for every new one, so the difference never grows; a growing book adds more than it frees and crosses the threshold every few hundred messages.

**Exercise 10.5 ★★.**

A Java strategy is fastest after ten minutes of trading and slowest at the open. Explain, and propose two remedies.

**Solution of Exercise 10.5.**

The hot path is interpreted or lightly compiled until the just-in-time compiler has seen enough executions; the open is also when rare paths are first taken and may deoptimise. Remedies: warm up by replaying recorded traffic through the real code against a simulator before the open; keep types and branches on the hot path stable; or compile ahead of time.

**Exercise 10.6 ★★.**

The exchange of the opening kept ring buffers of 20 million slots on the input side. At 6 million orders a second, how many seconds of input does such a ring hold?

**Solution of Exercise 10.6.**

$20 \times 10^6 / 6 \times 10^6 \approx 3.33$ seconds of input at full rate.

**Exercise 10.7 ★★★.**

*Coding.* Call `gc.freeze()` in the dictionary book after the warm-up and then every 100 000 messages. Count the full collections and record the worst message. What does freezing trade away?

**Solution of Exercise 10.7.**

Freezing moves every object alive at that moment to a permanent generation that collections ignore, so full collections scan only the orders added since the last freeze and are much shorter. The price is memory: frozen objects that die are never reclaimed until the heap is unfrozen, so a book whose orders are cancelled keeps their memory.

**Exercise 10.8 ★★★.**

*Find the flaw.* “Our Java gateway uses the concurrent collector with pauses under a millisecond, so garbage collection does not affect our latency.”

**Solution of Exercise 10.8.**

Under a millisecond is a thousand microseconds, far beyond a microsecond budget; and a concurrent collector still takes cores, memory bandwidth and cache from the program while it runs, and pauses the threads at safepoints. It affects latency unless the gateway allocates nothing during the session.

## 10.9 Problem: The Garbage Budget

**Problem 10.1.**

Weekend problem — a JVM engine on a busy day

A firm runs an order-routing engine on the JVM with a 2 GiB young generation. On a normal day it handles 200 000 messages a second for a 6.5-hour session and allocates 48 bytes per message on average. A code review finds that most of it comes from boxed integers and a logging call that formats a string.

**Part I — The budget.**

1. How many bytes does the engine allocate per second, and per session?
2. How often does it collect the young generation, and how many times per session?
3. What is the allocation per message that allows at most one young collection per session?
4. What does that budget mean in practice?

**Part II — The pauses.**

5. If each young collection pauses the engine for $0.5\,\mathrm{m}\mathrm{s}$ , how much time a session does it spend paused?
6. How many messages queue behind one such pause, and what does the one at the back of the queue wait?
7. On a day with three times the traffic, how do the answers change?
8. Why is the busy day the one that matters?

**Part III — The fixes.**

9. What replaces boxed integers on the hot path?
10. What replaces formatted logging (chapter 23)?
11. Where should the order book and the rings live, and why?
12. How would you verify, in the test suite, that the allocation per message is zero?

**Part IV — The verdict.**

13. State the *named result* : the allocation per message that keeps the engine to one young collection a session, and the pause time a session at the current 48 bytes.
14. What remains of the collector’s cost once allocation is zero?
15. Why must the engine still be warmed up before the open?
16. What did the Python experiment of the tutorial show that carries over to the JVM?
17. Why did the functional-language firm value its language, according to its own paper?
18. When would you choose C++ or Rust over a managed language for this engine?
19. What must the firm measure before deciding?
20. In one sentence: what does a [garbage collector](#def-ll-managed-and-functional-languages-on-the-desk-gc) cost a hot path?

**Solution of Problem 10.1.**

1. $48 \times 200\,000 = 9.6$ MB a second; 224.64 GB in a session.
2. Every $2~\text{GiB}/9.6~\text{MB/s} = 223.7$ seconds: about 105 times a session.
3. $2~\text{GiB}/(200\,000 \times 23\,400) = 0.459$ bytes.
4. Zero: no allocation per message on the hot path.
5. $105 \times 0.5\,\mathrm{m}\mathrm{s} \approx 52\,\mathrm{m}\mathrm{s}$ .
6. $0.0005 \times 200\,000 = 100$ messages; the first waits about $0.5\,\mathrm{m}\mathrm{s}$ , and the queue then takes a further 100 processing times to drain.
7. The allocation rate triples: about 314 collections and $157\,\mathrm{m}\mathrm{s}$ paused, with 300 messages behind each pause.
8. Queues and pauses scale with traffic, and the busiest days are the ones where latency is worth most.
9. Primitive fields and primitive collections (arrays of `long` ), with no boxing.
10. A binary log: the hot path writes a format identifier and raw arguments to a pre-allocated ring; formatting happens elsewhere, later.
11. Off the heap or in flat arrays of primitives allocated at start-up, so that no collection scans or moves them.
12. Measure allocated bytes per thread before and after a replay of messages (the JVM’s per-thread allocation counters, or an allocation profiler) and assert zero after warm-up.
13. **Named result.** The engine may allocate at most 0.459 bytes per message, in practice nothing, to collect once a session; at 48 bytes it collects about 105 times and spends about $52\,\mathrm{m}\mathrm{s}$ paused.
14. Safepoints and the collector’s background threads, which still take cores and cache; and the start-up and warm-up cost.
15. Because the just-in-time compiler still needs to see the hot path executed before it is optimised.
16. That removing per-message allocation removes the collector from the loop, and that the worst message is the full collection over the live heap.
17. Readable, correct, efficient code, written and changed quickly: correctness per engineer more than speed per message.
18. When the budget is in the low microseconds at a high percentile, and the team cannot guarantee zero allocation in every release.
19. The [latency budget](https://one-course.com/books/quant/13/en/chapter/1-where-latency-comes-from#def-ll-where-latency-comes-from-budget) at the percentile that matters, the allocation per message, and the measured pauses under a replayed busy day.
20. A pause proportional to what the hot path allocates and to what the heap keeps alive.

## 10.10 Interview questions

**Interview question 10.1 ★ developer.**

How does a generational [garbage collector](#def-ll-managed-and-functional-languages-on-the-desk-gc) work, and why is it generational?

**Solution of Interview question 10.1.**

It traces reachable objects from the roots and reclaims the rest. Most objects die young, so new objects go to a small young generation collected often, cheaply, in proportion to survivors; survivors are promoted to an old generation collected rarely.

*What the interviewer is looking for: the weak generational hypothesis and cost in proportion to survivors.*

**Interview question 10.2 ★★ developer.**

How would you write a low-latency service in Java so that the collector does not interfere?

**Solution of Interview question 10.2.**

Allocate nothing per message after warm-up: pre-allocated pools and ring buffers, primitive arrays instead of boxed collections, [off-heap memory](#def-ll-managed-and-functional-languages-on-the-desk-offheap) for large state, binary logging; a quiet collector and a young generation sized for the residual; warm-up before the open; measure allocation in tests.

*What the interviewer is looking for: no allocation, off-heap state, warm-up, measurement.*

**Interview question 10.3 ★★ developer.**

What is warm-up in a JIT-compiled runtime, and how do you deal with it before the open?

**Solution of Interview question 10.3.**

The time before the JIT has compiled the hot path with its profile; before the open, replay recorded traffic through the real code (orders to a simulator) until the hot methods are compiled, and keep the profile representative.

*What the interviewer is looking for: warm-up with realistic traffic, never with real orders.*

**Interview question 10.4 ★★ developer.**

What is [deoptimisation](#def-ll-managed-and-functional-languages-on-the-desk-deopt), and why can it happen at the worst moment?

**Solution of Interview question 10.4.**

The runtime discards optimised code whose assumptions (a single type at a call site, a branch never taken) have been broken, and falls back to slower code; the first rare event of the day, often at a volatile moment, is what breaks them.

*What the interviewer is looking for: speculative optimisation and its fall-back.*

**Interview question 10.5 ★★ developer, researcher.**

Why might a trading firm choose a functional language for its trading systems?

**Solution of Interview question 10.5.**

Correctness and speed of development: expressive types that encode invariants (every case of an order state handled), immutable data by default, concise code that traders can review, as the firm that published its experience in 2008 argued; with performance good enough for its budget.

*What the interviewer is looking for: correctness and review, backed by the published account.*

**Interview question 10.6 ★★★ developer.**

A Python service’s p99.9 is fine but its maximum is 20 milliseconds once every few minutes. How do you find out whether it is the [garbage collector](#def-ll-managed-and-functional-languages-on-the-desk-gc), and what do you do about it?

**Solution of Interview question 10.6.**

Record every collection’s duration with the `gc` module’s callbacks and correlate them with the slow messages; count collections by generation. Then stop allocating per message (pre-allocated arrays), freeze long-lived objects after start-up, or schedule full collections explicitly in quiet moments.

*What the interviewer is looking for: measure the collector, then remove its cause.*