22Pre-Trade Risk and Kill Switches

On 1 August 2012, one of the eight servers of a broker-dealer’s order router, Knight Capital Americas, still carried an old piece of code that a new deployment should have replaced. For 212 incoming orders it sent child orders to the market “without regard to the number of share executions” already received, in the words of the regulator’s order: 4 million executions in 154 stocks, more than 397 million shares, in about 45 minutes, and a loss of $460 million (One Quant Book 7, chapter 21, tells the deployment story, and One Quant Book 11, chapter 27, the risk policy it produced). This chapter reads the same order as an engineer would: as a list of requirements. Every paragraph that says what the firm lacked (a control that compared orders leaving the router with those that entered it, a position limit that counted outstanding orders, a firm-wide threshold that stopped orders automatically, a way to halt the router on its own misbehaviour) is a check to implement, on the path of every order, in a few tens of nanoseconds. The chapter builds that gate, then replays a runaway of the 2012 shape through it with one check at a time, and finds that most of the checks a firm would think of first would not have stopped it.

22.1 The requirements document

The regulator’s order (Release 34-70694) is precise about what was missing. The firm had controls before orders reached the router, but none after it: nothing compared “orders leaving SMARS with those that entered it”, and there were no procedures “to halt SMARS’s operations in response to its own aberrant activity”. It had a price cap, 9.5% from the best price when the parent order arrived, which did not help while prices moved less than that and did not apply to orders meant for the opening auction. It had position limits for some groups, but “these limits did not account for the firm’s exposure from outstanding orders”. It had no firm-wide capital thresholds linked to automated controls, and its position monitor “relied entirely on human monitoring”. The rule it was charged under, the market access rule, asks for controls that prevent the entry of orders that exceed “pre-set credit or capital thresholds” and of erroneous orders that “exceed appropriate price or size parameters, on an order-by-order basis or over a short period of time, or that indicate duplicative orders”. The engineering translation is a list: a price collar, size and notional limits per order, a position limit that counts orders in flight, a capital threshold across the firm, a rate limit, a duplicate check, and a kill switch that the gate itself can pull. The pre-trade risk check and the price collar are defined in One Quant Book 11 (chapter 27), which sets the policy; this chapter is about doing them fast and doing them right.

The gate sits in the trading process, on the strategy engine’s thread or right after it, before the order gateway (Figure 22.1): the engine’s order becomes the gateway’s message only if the gate says so. Everything else in the figure feeds it. The limits come from a risk service, as whole snapshots, with a heartbeat; the reference prices come from the book builder; the fills and cancellations come back from the gateway, so that the gate’s view of the position and of the orders in flight is the gateway’s; and the kill switch can be pulled by the gate itself, by a person on the risk desk, or by the venue.

22.2 Checks in nanoseconds

A risk gate is checked on every order, so it must cost little, and cost the same whether it accepts or refuses: a gate that is fast on the common path and slow on the rare one is slowest exactly when a runaway is sending orders. The build’s gate is organised for that. The limits are held in dense tables indexed by instrument (the venue’s locate code) and by strategy, not in maps keyed by name. Whatever can be computed before the order arrives is: the collar is kept as two prices, recomputed when the reference price or the limits change, so that the collar check is one comparison; the exposures are running sums, updated by the reports as in chapter 21, so that the position check is an addition and a comparison. And every check is computed, without a branch, into one bit of a mask (Listing 22.1); the order passes if the mask is zero, and a refusal reports the lowest bit set, while the audit record keeps the whole mask, because the second reason matters when the first is fixed.

        const std::int64_t long_ = pos_[instr] + open_buy_[instr] + qty;
        const std::int64_t short_ = -pos_[instr] + open_sell_[instr] + qty;
        std::uint32_t m = 0;
        m |= std::uint32_t(firm_killed_ | desk_killed_[desk] | st.killed) << Kill;
        m |= std::uint32_t(t - lim_t_ > lim_.max_age) << Stale;
        m |= std::uint32_t((ref_[instr] <= 0) | (t - ref_t_[instr] > lim_.ref_max_age)) << Ref;
        m |= std::uint32_t(buy ? price > hi_[instr] : price < lo_[instr]) << Collar;
        m |= std::uint32_t(qty > il.max_qty) << Qty;
        m |= std::uint32_t(notional > il.max_notional) << Notional;
        m |= std::uint32_t(buy & (long_ > il.max_long)) << Long;
        m |= std::uint32_t(!buy & (short_ > il.max_short)) << Short;
        m |= std::uint32_t(st.open >= sl.max_open) << Open;
        m |= std::uint32_t((desk_gross_[desk] + notional > lim_.desk_gross[desk]) |
                           (firm_gross_ + notional > lim_.firm_gross)) << Gross;
        m |= std::uint32_t(tokens < kOne) << Throttle;
        m |= std::uint32_t(dup) << Dup;
        if (m) return {kCodes[__builtin_ctz(m)], m};

Figure 22.2 measures it. With every order accepted and the duplicate check off, a check costs about $11\,\mathrm{n}\mathrm{s}$ at the median; the duplicate check, which compares the order with the strategy’s sixteen most recent ones, adds about $10\,\mathrm{n}\mathrm{s}$. The first version of that check compared five fields per remembered order and cost more than all the other checks together; packing quantity and price into one word and instrument and side into another (the wire’s quantity and price are 32-bit fields, so the packing is exact) brought it to this. On the fixture’s stream, where every kind of refusal occurs and each check follows the parsing of its input line, a check costs about $26\,\mathrm{n}\mathrm{s}$ at the median, the same for accepted and refused orders: the mask makes the refusals no cheaper and no dearer.

Tens of nanoseconds are small next to the path of an order through the kernel and the network (chapter 1), and there is no reason to make them smaller by checking less. The temptation runs the other way: to move the checks out of the trading process, into a separate risk process that sees the orders on a ring and answers. That costs a round trip between cores, about $200\,\mathrm{n}\mathrm{s}$ through the rings of chapter 12, eight times the check itself, and, worse, it creates a second failure to handle: if the risk process stops answering, the trading process must either wait or send unchecked. A gate in the trading process, with limits it can check alone, fails in one way only.

22.3 Limits in a hierarchy, updated live

Limits are set at several levels: the firm has a capital threshold, each desk a share of it, each strategy a rate, a number of open orders and its own share, each instrument a collar, a size and a position limit (the risk hierarchy of One Quant Book 6, chapter 29, and the policy of Book 11, chapter 27). An order is checked at every level it belongs to, and the gate keeps one running exposure per level. The limits change during the day, when a trader asks for more or the risk desk takes some away, and the gate applies changes the way the strategy engine applies parameters (chapter 20): as whole, versioned snapshots, between two orders, never in the middle of a check, so that no order is checked against half an old snapshot and half a new one.

The gate fails closed. A snapshot carries a maximum age: the risk service sends heartbeats, and if the last snapshot or heartbeat is older than its age, every order is refused as stale. The build’s fixture includes a risk service that goes quiet for three seconds with a maximum age of two: from the moment its last heartbeat is two seconds old until the next one arrives, the gate refuses every order, 50 of them. The same holds for the reference prices behind the collar: an instrument whose reference is older than its maximum age cannot be traded, because a collar around a stale price is no collar. Failing closed has a cost, since a strategy stops when a monitoring service fails, and the firm must decide which services the trading depends on. The rule of the build is that anything a check needs is such a service.

22.4 Kill switches: block, cancel, flatten

A kill switch stops a node of the hierarchy (a strategy, a desk, the firm) and can do three things, in increasing order of consequence. Block refuses every new order under the node: nothing more goes out, and nothing already out is touched. Cancel also sends a mass cancel for the node’s open orders; until each cancellation is confirmed, the orders count in the exposure, and those that fill in the meantime are races of the kind of chapter 21. Flatten also sends orders that close the positions, at the reference price, which the collar admits; it is the most dangerous of the three, since it trades, into a market the runaway may have moved, and the build returns those orders for a person to confirm rather than sending them. Who pulls the switch is a policy question (Book 11): the gate pulls it on a breach of a hard limit, the risk desk can pull it by hand, and many venues offer their own. Unkilling is always a person’s decision.

    def kill(self, t, level, name="", action="block"):
        """block: refuse every new order under the node; cancel: also return the open orders to cancel (a mass
        cancel); flatten: also return the orders that close the node's positions (at the reference price, which the
        collar admits). Positions are kept per instrument for the firm, so flatten closes the firm's positions."""
        self.killed.add((level, name if level != "firm" else ""))
        self.audit.append((t, 0, "K", BIT["K"]))
        if action == "block":
            return []
        cancels = sorted(cl for cl, o in self.orders.items() if self._under(level, name, o))
        if action == "cancel":
            return cancels
        return cancels + [(instr, "S" if p > 0 else "B", abs(p), self.ref.get(instr, (0, 0))[0])
                          for instr, p in sorted(self.pos.items()) if p]

22.5 Where the checks sit

Checks can sit in four places, and a firm uses several. In the trading process, as here: the cheapest and the only place that sees every order before it leaves, with the firm’s own view of the orders in flight. In a separate risk process or appliance on the path: independent of the trading code, at the cost of a hop, and the only protection against a bug in the trading process itself. At the broker, when the firm reaches the venue through a broker’s sponsored access (One Quant Book 1, chapter 4): the broker is responsible under the market access rule for the orders it lets through, whatever the firm checks. And at the venue, whose collars, limits and kill switches protect the market rather than the firm. The 2012 router had checks upstream and a monitor downstream, and nothing on the path of its own output. A useful test of any architecture is to ask, for each component that generates orders, which check sees its output before the venue does.

22.6 Tutorial: a runaway, one check at a time

firm_riskgate_runaway.py rebuilds the 2012 shape against Book 10’s matching engine. The runaway sends buy child orders of 100 shares in bursts of fifteen, a microsecond apart, every $10\,\mathrm{m}\mathrm{s}$: 1 500 a second on average, the rate of 4 million executions in 45 minutes, of about 99 shares each. It prices them five ticks above the last trade and never looks at its fills. A liquidity provider keeps a ladder of sell orders and moves it up a tick every ten fills, so the runaway pushes the price up as it buys. Table 22.1 and Figure 22.3 give five seconds of it, with each check switched on in turn.

The table is the chapter’s argument. The checks that look at each order alone (collar, size, notional) never trip: every child order is small, and priced near a market that the runaway itself is moving, so a collar around the last trade moves with it (the price rose 7.5% in the five seconds, more than the collar, and the collar never noticed). A throttle slows the runaway to its rate and does not stop it: 135 million shares in 45 minutes is still a disaster. The duplicate check nearly stops this runaway, because its child orders repeat, and would miss one that varied its quantity. What stops it is a limit on what the firm holds, counted in the worst case: the position limit at 20 000 shares after 200 orders, the capital threshold at 24 900. Counting only fills overshoots by the orders in flight, here one burst of ten orders, 1 000 shares; with more orders in flight, or a slower report path, it overshoots by more. The kill switch changes what happens next: without it, the runaway keeps sending and the gate keeps refusing, 7 300 refusals in five seconds; with it, the strategy is blocked at the first breach and its open orders are cancelled.

22.7 Build: the risk gate

22.8 Exercises

22.9 Problem: Forty-Five Minutes

22.10 Interview questions