23Futures Connectivity

In January 2018 an order on Eurex entered the exchange through one of sixteen high-frequency gateways, and the path on which competing orders raced was 22 microseconds long “with multiple µs variance”. Each gateway had its own queue and its own jitter, so a firm that opened sessions on many gateways and sent a copy down each could beat a faster firm with one. The exchange’s own account is blunt: “Latency jitter on parallel inbound paths had incentivised multiplicity to reduce latency.” Its answer was to change the gateways, and by September 2018 the competitive path was under two microseconds with nanosecond variance. The history of those changes is a history of what “fair” means at the microsecond.

Book 1 introduced futures commission merchants and clearing members, Book 10 the order-entry session, Book 11 the kill switch and the pre-trade check, Book 13 the drop copy and the binary market data. This chapter is about how a firm connects to a futures exchange: the sessions and the gateways behind them, the fairness designs the exchanges adopted, the clearing firm’s controls on the path, and the market data channels. The race between gateway designs is a labelled simulation built on the exchanges’ published numbers.

23.1 Order-entry sessions and market-segment gateways

CME Group offers both models. A Convenience Gateway accepts orders for every market segment and routes them; a Market Segment Gateway connects to one segment, and the firm connects to each segment it trades, with one session identity shared across all of them and separate primary and backup addresses for each (Box 23.2). Deutsche Börse’s T7 has the same split: partition-specific gateways, for high-frequency sessions only, reach one partition; low-frequency gateways reach all partitions through them, 43 microseconds slower at the median; FIX gateways go through the low-frequency ones.

A session is therefore a set of choices: which gateway type, which segments or partitions, what throttle, and how many. On T7 a high-frequency session is throttled to 150 transactions a second (“Full”) or 250 (“Ultra”), a participant may order up to 80 sessions, and the price list charges more for the seventh Full session than for the sixth (Box 23.1).

A session planner turns a transaction rate into the cheapest mix: firm_gwmodel.plan_sessions carries 3 000 transactions a second on one partition with five Full sessions and nine Ultra ones, EUR 8 320 a month, cheaper than twenty Full sessions (EUR 8 840) because the Full price doubles from the seventh.

23.2 Gateway fairness and its history

The model reproduces the incentive. Ten firms react to the same event at the same instant. Under the parallel design, each copy of an order crosses the network to a gateway ($20\,\text{µ}\mathrm{s}$, jitter of $1\,\text{µ}\mathrm{s}$), waits behind that gateway’s queue (two messages already there on average, and the race’s own copies that arrived first, $1.5\,\text{µ}\mathrm{s}$ each), and the firm’s first copy to finish counts. Under the segment design every copy enters one gateway whose order is fixed at the network card, over connections equal to ±2.5 ns. All parameters are assumptions in the range the exchanges describe.

def race(design, sessions, rng):
    """sessions: copies per competitor. Returns the winner, the arrival of its first copy (us) and the completion of a
    bystander message reaching a random gateway just after the race's copies (us)."""
    owner = np.repeat(np.arange(len(sessions)), sessions)
    if design.kind == "segment":
        arrive = design.net_us + rng.uniform(-design.cable_ns, design.cable_ns, len(sessions)) / 1000
        queue = rng.poisson(design.background) + int(np.sum(sessions))
        return (int(np.argmin(arrive)), float(arrive.min() + design.service_us),
                float(design.net_us + (queue + 1) * design.service_us))
    gw = np.concatenate([rng.choice(design.gateways, size=s, replace=False) for s in sessions])
    at_gw = design.net_us + design.net_sd_us * rng.standard_normal(len(gw))
    queue = rng.poisson(design.background, design.gateways).astype(float)
    done = np.empty(len(gw))
    for i in np.argsort(at_gw):
        queue[gw[i]] += 1
        done[i] = at_gw[i] + queue[gw[i]] * design.service_us
    best = np.full(len(sessions), np.inf)
    np.minimum.at(best, owner, done)
    by = design.net_us + (queue[rng.integers(design.gateways)] + 1) * design.service_us
    return int(np.argmin(best)), float(best.min()), float(by)

Alone, a firm that sprays copies over twelve gateways wins 59.7% of the races instead of 10.0%. When everyone does it, the win rate returns to one in ten, and what remains is the load: a bystander message reaching a random gateway just after the race takes $25.4\,\text{µ}\mathrm{s}$ when every firm sends one copy and $35.8\,\text{µ}\mathrm{s}$ when every firm sends twelve, the “higher, less predictable latencies” of the exchange’s account. Under the segment design the firm wins 9.8% of races with one session or twelve: the order is decided by nanoseconds of cable, and copies only load the gateway.

The history reads as a sequence of such repairs. Parallel gateways with independent queues rewarded multiplicity; FIFO within a gateway and deterministic networks reduced the variance; a single entry point per partition removed the parallel paths; equal cables moved the tie-break to nanoseconds. Each repair closes one strategy and exposes the next margin. CME’s safeguards, which its documentation calls a market integrity control, target messages split across packets: sending the start of a message before its end is ready is a way to arrive first, and a session that does it now waits at least three microseconds.

23.3 Drop copy and clearing-firm risk layers

A futures firm trades under a clearing member’s guarantee, and the clearing member controls it at the exchange. CME’s Globex Credit Controls let clearing-firm risk administrators set exposure limits and maximum order sizes per executing firm and choose what happens on a breach: an e-mail, blocked orders, cancelled orders; for mass quotes the check applies only after execution. Unlike the broker’s layer of chapter 22, this control sits inside the exchange, so it adds nothing to the firm’s path; its cost is the limit itself, which can stop a strategy in the middle of a busy day.

The drop copy is the firm’s independent record of every execution and cancellation, from the exchange. With several sessions it covers them all, and reconciliation must be per session: an order identifier is unique within its session, not across them. firm.gwmodel splits the drop copy by session and runs Book 13’s reconciler on each.

def reconcile_sessions(gateways, drop_copy):
    """Reconcile each session's gateway with its part of the firm's drop copy (firm_ordergw.reconcile)."""
    breaks = []
    for s, gw in gateways.items():
        mine = [(k, cl, q) for (sess, k, cl, q) in drop_copy if sess == s]
        breaks += [f"session {s}: {b}" for b in og.reconcile(gw, mine)]
    unknown = sorted({sess for (sess, *_rest) in drop_copy} - set(gateways))
    breaks += [f"session {s}: in the drop copy, not in the firm" for s in unknown]
    return breaks

Multiplicity has a cost here too. Twelve copies of an immediate-or-cancel order produce up to twelve executions and cancellations to reconcile, and more than one copy may fill: a firm that sprays orders must size each copy as if all might execute, which is the worst-case accounting of Book 13’s gateway.

23.4 Market data: channels, recovery and bandwidth

CME disseminates market data by channel, a channel per product group (the E-mini S&P 500 futures on channel 310, their options on 311, interest rate futures on 312, NYMEX crude and refined futures on 382, among others), each on two UDP feeds, A and B, which the exchange strongly recommends processing both. A gap in the packet sequence numbers means that every book on the channel may be wrong; recovery is from snapshot loops, a TCP replay of missed packets or an instrument loop. The configuration file, not the channel guide, is authoritative for the addresses, and both change.

Two feeds protect only if their losses are independent. With a loss probability of $10^{-5}$ per packet on each, independent losses lose a packet on both once in $10^{10}$; with a correlation of 0.5 between the two feeds’ losses (a shared switch, a shared buffer), once in 200 000 (firm_gwmodel.both_lost). The two feeds must reach the firm by separate paths, into separate switch ports, or the second one mostly duplicates the first’s failures.

23.5 The session budget

A firm can compute what an extra session is worth. At 20 000 races a month and EUR 0.50 for each race won (assumptions), the second session is worth EUR 840 a month under the parallel design and the twelfth EUR 180.5; against Eurex’s fees, the last session that pays for itself is the sixth, where the price per session doubles.

def marginal(curve):
    p = [c[0] for c in curve]
    return [(k, (p[i] - p[i - 1]) * RACES * VALUE_EUR) for i, k in enumerate(K) if i > 0]


def fee(k):
    """The monthly fee of the k-th Full session under the dated schedule."""
    return FULL.fee_first if k <= FULL.first_n else FULL.fee_after


def last_paying(curve):
    last = 1
    for k, v in marginal(curve):
        if v >= fee(k):
            last = k
        else:
            break
    return last

The fee schedule is part of the fairness design: a price that doubles from the seventh session makes the marginal copy cost more than it wins exactly where the curve flattens. Under a segment gateway the calculation is simpler: the second session wins nothing, and sessions are bought for throughput, segments and redundancy only.

23.6 Tutorial: racing through gateways

23.7 Build: the gateway and session model

23.8 Exercises

23.9 Problem: Twelve Sessions for One Order

23.10 Interview questions