1Networking for Trading

The operator of the consolidated U.S. options feed tells its subscribers how much network to buy, and it does not quote an average. For July 2026 it projected 4.403 gigabits in the busiest 100 milliseconds and 0.501 gigabits in the busiest 10 milliseconds, per copy of the feed: $44\,\mathrm{G}\mathrm{bit}/\mathrm{s}$ over the longer window, $50\,\mathrm{G}\mathrm{bit}/\mathrm{s}$ over the shorter, twice that for a subscriber that takes both redundant copies. Its own statistics say the same thing from the other side: the busiest millisecond of July 2026 carried messages at five times the rate of the busiest second. A network sized for the average of a trading day loses packets in its busiest milliseconds, and those are the milliseconds a trading firm is paid to see.

This chapter opens the book’s first part, the network between a venue and the firm’s servers. It prices a byte on the wire, shows how one packet reaches many subscribers, explains why market data travels in datagrams and orders in streams, and ends where most losses begin: at a switch port that receives more than it can send.

1.1 The Ethernet frame and what a byte costs on the wire

Table 1.1 is the first latency table of this book and the one most often forgotten. A full-size frame at $10\,\mathrm{G}\mathrm{bit}/\mathrm{s}$ takes $1.23\,\text{µ}\mathrm{s}$ to serialise, half of the $2.5\,\text{µ}\mathrm{s}$ that an engineer of a large market maker gave in a public talk as a very good wire-to-wire time for a whole software trading system (One Quant Book 13, chapter 1). Moving from 10 to $25\,\mathrm{G}\mathrm{bit}/\mathrm{s}$ saves 60% of every serialisation, a reason to buy the faster port even when the slower one would carry the traffic. Serialisation is also why small frames matter: an order is rarely larger than a hundred bytes, and a switch that forwards it only once it has all of it pays for all of it.

MIN_FRAME, MAX_FRAME = 64, 1518
MOLD_HDR = 20          # session 10 + sequence number 8 + message count 2


def frame_bytes(payload, l4="udp", vlan=False):
    l4h = UDP if l4 == "udp" else TCP
    return max(MIN_FRAME, ETH + (VLAN if vlan else 0) + IPV4 + l4h + int(payload) + FCS)


def mold_payload(msg_lengths):
    return MOLD_HDR + sum(2 + int(n) for n in msg_lengths)


def wire_bytes(frame):
    return int(frame) + OVERHEAD

1.2 IP multicast: groups, membership and snooping

A venue sends the same market data to hundreds of subscribers. Sending one copy to each over its own connection would make the last subscriber wait for all the others (chapter 12 studies a venue that did exactly that); sending one copy that the network replicates makes them all equal to within the replication.

A venue’s feed is usually spread over many groups, one per channel or partition of instruments, so that a subscriber receives only the groups it needs (chapter 22 sizes the options feeds this way). A group must also be delivered on the Ethernet segment, and Ethernet addresses are 48 bits: a group address is mapped to the Ethernet address 01:00:5e followed by the low 23 bits of the group, so the upper 5 of the 28 bits that distinguish groups are lost and 32 groups share each Ethernet address. A card filtering on Ethernet addresses may therefore let through groups the host never joined, and the kernel discards them after having paid for them.

1.3 Datagrams against streams

Two transport protocols carry almost everything in a trading network. UDP sends datagrams: each is delivered whole or not at all, with no ordering and no retransmission, and one datagram sent to a group reaches every member. TCP carries a byte stream between two endpoints: it numbers the bytes, acknowledges them, retransmits what is lost and delivers everything in order.

Market data travels in UDP multicast because only multicast sends one copy to all, and because a feed must not slow down for its slowest subscriber: TCP’s flow control would do exactly that. The price is that loss is the application’s problem. The venue sends every packet on two independent lines (One Quant Book 10, chapter 26), numbers them, and runs retransmission and snapshot services; the firm’s handler arbitrates and recovers (One Quant Book 13, chapters 16 and 18). Orders travel over TCP because an order must arrive exactly once, in order with the firm’s other orders, and because the venue must know which connection sent it: the session layer of the order-entry protocol builds on those guarantees.

Head-of-line blocking is the cost of TCP’s ordering. A cancel sent a microsecond after a lost new-order message waits for the new order’s retransmission, which comes after at least a round trip and usually after a timer: on a clean colocation link loss is rare, and when it happens the whole session stalls. Ordering is also a feature: the venue must never see the cancel before the order it cancels.

1.4 Congestion: egress queues, incast and microbursts

A switch forwards each frame to an output port. If frames destined to one port arrive faster than the port’s line rate, they wait.

A feed that uses half of a port on average can overflow it for a millisecond, and a trading network is built to make this happen: every venue’s feed, and every partition of every feed, reacts to the same market events at the same time.

Figure 1.3 is the published version of the microburst: the same feed, in the same month, is 63.9 million messages a second or 350 million a second depending on the window one reads it through. The operator’s capacity notice, which subscribers use to size their links, is written in 10-millisecond peaks for that reason (Box 1.1); a switch port does not average over 10 milliseconds, it queues frame by frame.

To see the mechanism rather than its footprint, the chapter’s tutorial simulates eight feeds into one $10\,\mathrm{G}\mathrm{bit}/\mathrm{s}$ port with firm.netsim. Each feed averages 400 000 packets a second in clusters (a market event produces a run of messages), one to thirty add-order messages a packet; the eight together offer about $5.2\,\mathrm{G}\mathrm{bit}/\mathrm{s}$, half the port. In one version every feed’s clusters are independent. In the other, 30% of each feed’s packets come in clusters triggered by common events, 400 a second, that hit all eight feeds at once, as a macro release or a large trade in an index future does.

The two versions have the same average load, and the difference is all in the correlation (Figure 1.4). With independent bursts the busiest 100 microseconds carry $9.1\,\mathrm{G}\mathrm{bit}/\mathrm{s}$, and the buffer that would have lost nothing, the largest backlog of the run, is 21 to 31 KB over twenty seeds. With common events the busiest 100 microseconds carry $68\,\mathrm{G}\mathrm{bit}/\mathrm{s}$, and that buffer is between 0.44 and 2.4 MB over the same twenty seeds, 1.1 MB on average. Removing the common events is the ablation: the megabyte is caused by the correlation, not by the load. A port’s traffic statistics averaged over a second show half a link in both cases.

Read against the simulation, the box says that a port which can borrow the whole shared block survives the common-event bursts of Figure 1.4, and a port that gets only its sixteenth does not: at 500 KB of buffer the eight feeds with common events still lose 2.6% of their frames. The events that fill every port at once are the events that leave each port with only its share. The remedies are a faster egress port (the eight feeds’ common bursts reach $16\,\mathrm{G}\mathrm{bit}/\mathrm{s}$ over a millisecond, within a $25\,\mathrm{G}\mathrm{bit}/\mathrm{s}$ port), fewer feeds per port, or a switch whose buffer is dedicated rather than shared; chapter 2 adds the devices that do not queue at all.

1.5 Tutorial: microbursts from frames to drops

def feed(seed, common, share, seconds=SECONDS, pps=FEED_PPS):
    """One simulated feed: clusters of packets; 1 to 30 add-order messages per packet."""
    t = ns.bursty_arrivals(pps, seconds, 8, 100, seed, common, share)
    k = np.minimum(np.random.default_rng(seed + 1000).geometric(1 / 3, len(t)), 30)
    frames = np.array([ns.frame_bytes(ns.mold_payload([ADD_ORDER] * int(x))) for x in k])
    return t, frames


def fanin_traffic(n_feeds=8, share=0.3, seed=1, seconds=SECONDS, event_hz=400):
    ev = ns.event_times(event_hz, seconds, 99 + seed)
    return ns.merge(*[feed(10 * seed + i, ev, share, seconds) for i in range(n_feeds)])


def fanin(buffers, n_feeds=8, share=0.3, seed=1, seconds=SECONDS, gbps=10.0):
    t, f, _ = fanin_traffic(n_feeds, share, seed, seconds)
    return [100.0 * ns.egress(t, f, gbps, b).n_dropped / len(t) for b in buffers]

    t = np.asarray(t_ns, dtype=np.int64)
    w = np.asarray(frames, dtype=np.int64) + OVERHEAD
    n = len(t)
    dep = np.full(n, -1.0)
    drop = np.zeros(n, dtype=bool)
    back = np.zeros(n)
    rate = gbps / 8.0                                         # bytes per ns
    q, prev, mx, nd, db = 0.0, 0, 0.0, 0, 0
    tl, wl = t.tolist(), w.tolist()
    for i in range(n):
        ti, wi = tl[i], wl[i]
        q = max(0.0, q - (ti - prev) * rate)
        prev = ti
        back[i] = q
        if q + wi > buffer_bytes:
            drop[i] = True
            nd += 1
            db += wi
            continue
        q += wi
        mx = max(mx, q)
        dep[i] = ti + q / rate

1.6 Build: the network model

1.7 Exercises

1.8 Problem: The Open That Did Not Fit

1.9 Interview questions