2Switches and Layer-1 Devices

A store-and-forward switch cannot send a 1 518-byte frame on before it has received all of it: at $10\,\mathrm{G}\mathrm{bit}/\mathrm{s}$ that is $1.2\,\text{µ}\mathrm{s}$ of waiting, whatever the switch does afterwards. A layer-1 switch sold to trading firms forwards between its ports in $4\,\mathrm{n}\mathrm{s}$, with less than 100 picoseconds of jitter, and in the words of its data sheet it does “not buffer or queue data”: it never looks at the frame at all. Between the two sit the cut-through switches, which look only at the first bytes. Choosing among them, port by port, is most of what designing the network inside a trading cage means.

This chapter follows a frame through each kind of device, then through the passive devices that let a firm watch its own traffic, and ends with three designs of the same small cage: the latency of each path, the devices each path crosses, and the single devices whose failure would cut a server off.

2.1 Store-and-forward against cut-through

A store-and-forward switch drops a frame whose check sequence is wrong before it goes further; a cut-through switch has already sent most of it when it finds out, and can only mark the end of the frame as bad. That is the whole price of cut-through, and in a colocation cage, where links are short and errors rare, it is a price firms pay.

2.2 Layer-1 devices: fan-out, multiplexing and patching

The fan-out is the natural device for market data: the venue’s feed arrives once on the cross-connect and must reach every server that trades on it, all at the same time, with nothing to decide. The multiplexer is the natural device for orders: many servers, one order-entry handoff, and no routing decision beyond “all to that port”. A multiplexer cannot be as fast as a fan-out, because two servers can send at once.

The chapter’s tutorial runs this race for two, four and eight servers (Figure 2.4). With four servers answering within $50\,\mathrm{n}\mathrm{s}$ of one another, an order waits $93\,\mathrm{n}\mathrm{s}$ on average and the last one $186\,\mathrm{n}\mathrm{s}$, more than the multiplexer’s own $39\,\mathrm{n}\mathrm{s}$; spread the answers over a microsecond and the average falls to $6\,\mathrm{n}\mathrm{s}$. The tighter a firm’s servers are, the more they queue behind one another.

A layer-1 switch is also a patch panel under software control: any port can be connected to any other, or to several, by a configuration change instead of a technician moving a cable. Firms use that to move a server from one venue’s line to another, to swap a failed device for a spare, or to copy a link to a capture device for an hour.

2.3 Taps and mirror ports

A firm that wants to know what it sent and received, and when, must copy its traffic without disturbing it.

A tap is on the link itself: it cannot drop, reorder or delay the copy relative to the original, and it keeps working when the switch does not. Its costs are the light it takes (a 50/50 split halves the power reaching the far end, which a long cross-connect may not afford) and a monitoring port for each direction. A mirror port is free, since the switch is already there, but it is a switch port like any other: mirroring both directions of a busy $10\,\mathrm{G}\mathrm{bit}/\mathrm{s}$ link sends up to $20\,\mathrm{G}\mathrm{bit}/\mathrm{s}$ to a $10\,\mathrm{G}\mathrm{bit}/\mathrm{s}$ port, which drops half of it when both directions are full, exactly in the bursts one wanted to see. Capture devices (chapter 5) therefore sit on taps, and on layer-1 mirror outputs, which copy at line rate.

2.4 Designing the network inside one cage

Table 2.1 compares three designs of the same cage, two servers and the same cables (30 metres of cross-connect to each handoff, 3 metres inside the cage), computed with firm.cagenet. The commodity design puts two store-and-forward switches in series; the cut-through design gives each line its own switch; the layer-1 design is Figure 2.3.

The layer-1 design’s feed path is 263 nanoseconds, of which 176 are the 36 metres of fibre and 83 the frame’s own serialisation: the devices add 4. Its order path is slower than its feed path by the multiplexer’s 39 nanoseconds less the fan-out’s 4, and slower still when two servers answer the same event. The next nanoseconds are no longer in the devices but in the cables and in the servers.

2.5 Tutorial: three cages

    def breakdown(self, path, frame, gbps):
        hops = [self._cable(a, b) for a, b in zip(path, path[1:], strict=False)]
        prop = sum(ns.prop_ns(c.length_m, c.n_g) for c in hops)
        dev = 0.0
        for n in path[1:-1]:
            d = self.dev[n]
            if d.kind in ("l1", "mux"):
                dev += ns.forward_ns(frame, gbps, "layer-1", d.fabric_ns)
            elif d.kind in ("cut-through", "store-and-forward"):
                dev += ns.forward_ns(frame, gbps, d.kind, d.fabric_ns)
        return {"serialisation": 8.0 * frame / gbps, "propagation": prop, "devices": dev}

    rng = np.random.default_rng(seed)
    t = np.abs(rng.normal(1000.0, jitter_ns, (n_events, k)))
    t.sort(axis=1)
    ser = ns.ser_ns(frame, gbps)
    free = np.zeros(n_events)
    waits = np.zeros((n_events, k))
    for j in range(k):
        start = np.maximum(t[:, j], free)
        waits[:, j] = start - t[:, j]
        free = start + ser
    return {"mean": float(waits.mean()), "p_wait": float((waits > 0).mean()),
            "last": float(waits[:, -1].mean()), "ser": ser}

2.6 Build: the cage network model

2.7 Exercises

2.8 Problem: Three Designs for One Cage

2.9 Interview questions