18The Feed Handler

At the open the message rate jumps: on the simulator’s busy instrument of this chapter, from about 570 messages a second at midday to more than 9 000 in the first seconds; on the consolidated feed of U.S. options, OPRA published a peak of 63.9 million messages in one second of July 2026 and 350 000 in its busiest millisecond. A handler that falls behind lets its socket buffer fill; the kernel drops what does not fit; line A loses a packet that line B has also lost; and the firm’s book is stale until the handler has recovered, by retransmission if the gap is small and recent, from the next snapshot if not. This chapter builds the first component of Part IV’s trading system, the feed handler: it decodes and arbitrates the two lines of chapter 16, detects gaps and recovers from them, normalises every message into the firm’s event record and publishes it on the rings of chapter 12, and says at every moment whether the book it feeds can be trusted.

18.1 Decode and arbitrate

The handler of the build is written first in Python, as the reference, then in C++20 and Rust, and all three publish exactly the same events (compared by a 64-bit hash of their 48-byte records) on the same inputs. Its time is the arrival time recorded with each packet, so that a recorded day replays exactly: the same inputs give the same events, the same staleness intervals and the same counters. Arbitration is chapter 16’s; what the handler adds is what to do while a sequence number is missing.

18.2 Detect gaps and recover

A missing sequence number is usually not lost, only late: it is on its way on the other line, or reordered by a switch. The handler therefore holds the messages that arrive after it and waits a short time, here $0.5\,\mathrm{m}\mathrm{s}$, for the missing ones. If they come, the held messages are released in order. If not, the handler asks the retransmission server for the range (chapter 16; the simulator answers up to 1 000 messages per request from its last 100 000), and applies the answer when it arrives, a round trip later. If the server cannot help, because the gap is older than its window or the server is unreachable, the handler waits for the next snapshot of the instrument, loads it, discards the held messages it already reflects and continues from the sequence number it gives, the procedure CME’s snapshot messages support with their last incremental sequence number processed.

    void incremental(std::uint64_t t, const std::uint8_t* p) {
        ++c.packets;
        const std::uint64_t seq = be64(p + 10), cnt = be16(p + 18);
        if (cnt != 0 && cnt != 0xFFFF && seq + cnt <= next_) {
            ++c.duplicates;
            return;
        }
        blocks(p, [&](std::uint64_t s, const std::uint8_t* m) {
            if (s < next_) return;
            if (s == next_ && mode == Mode::live) {
                publish(normalise(m, s));
                ++c.messages;
                ++next_;
            } else {
                pending_.emplace(s, m);
            }
        });
        if (!pending_.empty() && mode == Mode::live) {
            mode = Mode::await_line;
            since_ = t;
            deadline_ = t + timeout_;
            ++c.gaps;
        }
        drain(t);
    }

The snapshot path is the delicate one: the snapshot must be at least as recent as the book, the held messages it already reflects must go, and a hole after it must be detected again.

    def _apply_snapshot(self, t):
        upto = be(self._snap[0], 11, 8)
        if upto + 1 < self.next:               # older than what the book already holds: wait for the next cycle
            self._snap = []
            return
        for m in self._snap:
            self._publish(normalise(m, upto))
        self._snap = []
        self.next = upto + 1
        for s in [s for s in self.pending if s <= upto]:
            del self.pending[s]
        self._drain(t)
        if self.pending and self.mode != "live":             # a hole after the snapshot: detect it again
            self.mode, self._deadline = "await_line", t + self.gap_timeout_ns

Figure 18.2 shows the handler at work on a simulated minute from the open, both lines impaired: 0.2% of packets lost independently on each line, bursts of loss, jitter that reorders packets, and an outage on each line overlapping for 15 milliseconds at 40 seconds. Out of 4 232 gaps, 4 224 are filled by the other line within the half millisecond; the retransmission server closes the other eight within $2.7\,\mathrm{m}\mathrm{s}$. Without the server, seven gaps need a snapshot, and with a snapshot every second the book stays stale for 0.3 to 1 second each time, five seconds of the minute in all. Staleness is then not a property of the network but of the recovery path.

18.3 Normalise and publish

Every message leaves the handler as one 48-byte event: its kind, side, instrument, sequence number, timestamp, order reference, second reference (the new reference of a replace, the match number of an execution, the sequence number of a snapshot), price and quantity. A normalised record has three virtues downstream. Its size is fixed, so it fits a ring slot (chapter 12) and a consumer reads it without decoding. It is the same for every venue the firm reads, so the book builder (chapter 19) and the strategies (chapter 20) are written once. And it carries the venue’s sequence number, so a consumer can tell exactly which events it has seen, and a snapshot’s events can say which sequence number they reflect. The build’s test publishes a whole run on an SPSC ring of 64-byte slots and reads it back, hash for hash.

On the laptop the C++ handler takes about $39\,\mathrm{n}\mathrm{s}$ per packet at the median, $160\,\mathrm{n}\mathrm{s}$ at the p99 and $290\,\mathrm{n}\mathrm{s}$ at the p99.9, measured over the minute of Figure 18.2 with the clock read around each input; the rare slow ones are gaps, whose held messages go through an ordered map, and retransmission answers. At 9 000 messages a second the handler is busy about 0.04% of the time: the burst of this simulated instrument is nothing for it. The real bursts are the ones of Box 18.1.

18.4 Bursts, buffers and slow consumers

During a burst the difference between the arrival rate and the handler’s service rate accumulates in the socket’s receive buffer, the M/M/1 queue of One Quant Book 4, chapter 8, with a hard wall: whatever does not fit is dropped. The buffer’s size is set with SO_RCVBUF, and two details of Linux decide what it holds. The kernel “doubles this value (to allow space for bookkeeping overhead)” and returns the doubled value (socket(7)), capped by net.core.rmem_max, 4 MiB on the laptop, so that asking for 4 MiB grants 8. And the buffer is charged with each datagram’s whole memory footprint, not its payload. Figure 18.3 measures what a stalled reader keeps: 50 000 datagrams sent on loopback while nobody reads, then the socket drained. Each 64-byte datagram costs 960 bytes of buffer and each 1 000-byte datagram 2 305: the largest buffer the laptop grants holds 8 738 small datagrams, a fortieth of the messages of OPRA’s busiest millisecond.

The handler is in turn a producer. Its consumers read the ring at their own pace, and a consumer slower than the feed (a slow consumer, chapter 12) must not slow the handler down: the broadcast ring never waits, and a lapped consumer resynchronises. The two defences against bursts are therefore the same at both ends of the handler: enough buffer for the burst the data says to expect, and a way to recover when even that is not enough.

18.5 Monitoring a feed

A handler that is wrong silently is worse than one that stops. The build counts packets, messages, duplicates, gaps, gaps filled by the other line, retransmission requests and snapshot recoveries, and keeps every staleness interval with its start and end. Four of these numbers deserve alarms: the rate of gaps on each line (a line going bad, while the other hides it), the gaps filled by neither line (a common cause, chapter 16), any snapshot recovery (a book that was wrong for up to a snapshot interval), and the receive queue’s high-water mark (a burst closer to the buffer than expected). The stale-book flag itself goes to every consumer with each event, so that a strategy never quotes on a book that its handler knows to be wrong.

18.6 Tutorial: the handler on impaired lines

18.7 Build: the feed handler

18.8 Exercises

18.9 Problem: The Opening Burst

18.10 Interview questions