21Live Experiments

A new execution algorithm handles ten per cent of the firm’s orders for two weeks, 2 485 of 24 000, and saves 0.55 basis point of implementation shortfall on average, with a standard error of 0.48. The 95% interval runs from $-1.49$ to 0.39: the experiment cannot tell a saving from a cost. At half the flow it would need 77 trading days to detect a saving of 0.3 basis point, and 38 with the variance reduction of this chapter. Even a perfect answer would be to a narrower question than the desk’s: in the chapter’s simulated flow the new algorithm saves 0.30 basis point on its own orders but adds 0.25 to the firm’s other orders in the same stock, side and day, so that switching everything over saves 0.08. Research reaches production through paper trading, canaries and staged rollouts, and is then measured on live flow; this chapter covers both, with firm.abtest.

21.1 Paper trading and shadow mode

Paper trading tests everything that does not depend on the market’s reaction. The live feeds arrive with their real timing and gaps; the signals computed live can be compared with the same signals recomputed by the research code on the same data, value by value; the orders pass or fail the pre-trade risk checks; the strategy lives through the operational calendar (start-up and shutdown, corporate actions, holidays, the index rebalance) that a backtest compresses into rows. What paper trading cannot test is the fills. It is a simulator running on live data, with the blind spots of every simulator: chapter 19 showed that a replay without the strategy’s own orders misses the trades that stopped at them and the levels where they were alone. Shadow trading (chapter 19) compares the simulator with the live trading of an existing strategy; paper trading is the stage before a new strategy has any.

The exit condition of paper trading is therefore not its P&L. It is parity: live signals equal to the research signals within a stated tolerance, every order it would have sent accepted by the risk checks, and a full operational cycle crossed without manual repair. A strategy whose paper P&L is good and whose live signals differ from research has not been tested; it has been replaced by another one.

21.2 Canaries and staged rollouts

The most expensive deployment failure in the public record was a staged deployment. From 27 July 2012 Knight Capital placed new order-routing code on a limited number of servers on successive days, ahead of a new NYSE programme starting on 1 August. A technician did not copy the new code to one of the eight servers; no second person reviewed the deployment, and no written procedure required it. On 1 August the eighth server ran an old, defective function that the new code’s repurposed flag reactivated: for 212 incoming parent orders it sent millions of child orders, 4 million executions in 154 stocks for more than 397 million shares in about 45 minutes. Knight was left with a net long position of about $3.5 billion in 80 stocks and a net short of about $3.15 billion in 74, and realised a loss of $460 million. Before the open, its systems had sent 97 automated e-mails citing the old function by name; they were not designed as alerts and nobody read them. The SEC’s order of October 2013 imposed a $12 million penalty.

Three lessons follow, none of them statistical. A stage is only as good as its check of the deployed state: every server, process and configuration runs the version the stage says it runs. An alert is a message designed and routed to be acted on, not a log line that happens to be e-mailed. And a stage must be capped in what it can do: a canary with 1% of the flow must hold 1% of the risk limits, and invariants such as “child quantity never exceeds the parent’s” must be checked on every order, where they fire on the first violation. The statistical comparison of canary and control is for a different class of defect.

With the chapter’s flow ($\sigma = 23.1$ basis points, 2 400 orders a day), a defect costing 5 basis points an order and an alarm at $z = 3$, a 1% canary raises the alarm after 194 damaged orders and 8.1 days, a 10% canary after 214 orders and 0.89 day, a 50% canary after 385 orders and 0.32 day. A smaller canary does not reduce the damage done before a statistical alarm; it spreads it over more time. What a small canary buys is a small exposure for defects caught by invariants and limits, and time for people to look.

21.3 Randomised A/B tests on live flow

The chapter’s simulated flow has 2 400 parent orders a day in 500 stocks, drawn with popularity inversely proportional to rank. An order’s implementation shortfall, in basis points against its arrival price, is the firm’s pre-trade cost estimate (standard deviation 6.9 across orders), the estimate’s error, the beta-adjusted market move over the order’s scheduled window signed by its side, idiosyncratic noise, and a day effect common to all orders; the mean is 11.7 and the standard deviation 23.1. The new algorithm saves 0.30 basis point on each order it handles, and adds 0.25 to the other orders of the same stock, side and day when all of them are on the new algorithm (it trades earlier in the window and leaves its siblings more impact).

Assignment. firm.abtest.assign hashes the unit’s identifier with the experiment’s name and compares the first eight bytes of the SHA-256 digest, read as a fraction, with the planned share. The assignment is deterministic (the same order always goes to the same arm, on every server, without a shared table), independent across experiments with different names, and reproducible afterwards by anyone holding the identifiers. In the two-week experiment of the opening, 2 485 of 24 000 orders went to the new algorithm against a planned 2 400: $z = 1.83$, a p-value of 0.07, no mismatch at 5%. A mismatch would not be a statistical curiosity: it is the symptom of orders lost, duplicated or re-routed in one arm, and it invalidates the comparison until explained.

Interference. In the flow, 88% of orders share their stock, side and day with another of the firm’s orders. When orders are randomised one by one, a treated order and a control order have, on average, the same share of treated siblings, so the difference between the arms measures the direct saving, $-0.30$. The desk’s question is what happens when every order moves: $-0.30 + 0.25 \times 0.883 = -0.08$. A randomisation unit must contain the interference: here the stock-day, all of a stock’s orders on a day in the same arm. The day itself contains every interference within a day, but offers only one unit a day, and the day effect is then the noise; alternating the treatment through time in this way is a switchback design (Bojinov, Simchi-Levi and Zhao), common where units interfere, at the price of few units and of carryover from one period to the next.

def cluster_diff(y, t, cluster):
    """Order-weighted difference in means when whole clusters are assigned, with a cluster-robust standard error:
    each arm's mean is a ratio of cluster totals, linearised per cluster."""
    y, t, cluster = np.asarray(y, float), np.asarray(t, bool), np.asarray(cluster)
    _, inv = np.unique(cluster, return_inverse=True)
    n = np.bincount(inv).astype(float)
    tot = np.bincount(inv, weights=y)
    arm = np.bincount(inv, weights=t) / n
    if np.any((arm > 0) & (arm < 1)):
        raise ValueError("an arm must be constant within each cluster")
    mean, var = [], 0.0
    for m in (arm > 0.5, arm < 0.5):
        mu = tot[m].sum() / n[m].sum()
        e = tot[m] - mu * n[m]
        k = int(m.sum())
        mean.append(mu)
        var += k / (k - 1) * float((e * e).sum()) / n[m].sum() ** 2
    return float(mean[0] - mean[1]), math.sqrt(var)

21.4 Variance reduction

The covariates must be fixed before assignment or lie outside the firm’s reach. The pre-trade cost estimate qualifies, and explains 9% of the variance of the shortfall. So does the market’s beta-adjusted move over the order’s scheduled window: the portfolio manager fixes the window before the order is assigned, and the market does not move because one of the firm’s algorithms rather than the other is working it. With both covariates $R^2$ is 0.51, close to the halving Deng and co-authors reported for Bing’s experiments. The order’s realised duration, its fill rate or its participation do not qualify: the algorithm changes them, and adjusting for them removes part of the effect. Post-stratification (Miratrix, Sekhon and Yu) is CUPED with stratum indicators as covariates, nearly as efficient as assigning within strata in advance; for a design randomised by stock-day, analysing within each day (subtracting each day’s mean from outcome and covariates) is the stratification that removes the day effect.

Each design below was run as 100 simulated experiments of 60 days on half the flow (the mean estimate, the spread of the estimates, the mean reported standard error, and the days that standard error implies for 80% power on 0.3 basis point):

Every order-level design estimates the direct saving, $-0.30$; every design that assigns whole stock-days or days estimates the rollout’s, $-0.08$ (the day design’s $-0.16$ is within its spread of 0.66). The reported standard errors match the spread of the estimates across experiments. CUPED with the market covariate halves the days needed; the pre-trade estimate alone and stratification on it buy little, because the estimate explains little. Randomising by day multiplies the days by 42. Randomising by stock-day and analysing within the day, with CUPED, is as precise as the best order-level design and answers the right question; the answer is small, and detecting a saving of 0.08 basis point at 80% power would take 503 trading days.

21.5 Sizing and stopping an experiment

The standard deviation of 23.1 basis points and 2 400 orders a day give 77.7 days at half the flow for 0.3 basis point, and 38.2 with CUPED’s residual standard deviation of 16.2; the simulated experiments, rejecting in 78% of runs at 80 days without CUPED and 76% at 40 days with it, confirm the formula (Figure 21.2). The equal split minimises the sample size; a 10% experiment, as in the opening, needs $(1/0.1 + 1/0.9)/4 = 2.8$ times as many orders.

Peeking. A desk watching an experiment’s dashboard every morning and stopping the first day its p-value falls below 5% runs a different test from the one the p-value describes. On 4 000 simulated experiments with no effect at all, a fixed-horizon test looked at once rejects in 5% of them; looked at every day, it rejects at least once in 29% within 38 days, 33% within 60 and 38% within 120 (Figure 21.3).

The mixture sequential probability ratio test, a descendant of Wald’s sequential test (1945), turns continuous monitoring into a valid procedure (Johari, Koomen, Pekelis and Walsh, who deployed it on a commercial testing platform). For an estimate $\hat\Delta$ with variance $V$ at the current look, and effects drawn from $N(0, \tau^2)$ under the alternative, the mixture likelihood ratio against no effect is

Under no effect $\Lambda$ is a nonnegative martingale starting at 1, so the probability that it ever exceeds $1/\alpha$ is at most $\alpha$ (Ville’s inequality). The always-valid p-value is the running minimum of $1/\Lambda$; stopping whenever it falls below $\alpha$ keeps the error at $\alpha$ however often one looks. On the same null experiments it crossed 5% in 46, 62 and 82 of the 4 000 by days 38, 60 and 120. With the real saving of 0.30 and CUPED, and $\tau = 0.3$, half the experiments stop by day 38, the fixed horizon’s length, 4% by day 10 and 98% by day 120. A harmful change of $+1$ basis point stops at a median of 6 days. The sequential test gives up a fixed end date; it keeps the size under monitoring and stops early when the effect is large, which is when stopping matters.

The plan. Everything that decides the outcome is written down before the first order is assigned: the metric and its guardrails (fill rate, rejects, mark-outs), the randomisation unit, the share, the covariates, the horizon or the sequential rule with its $\tau$, the minimum detectable effect, and what will be done at each outcome. Chapter 1’s research log records it; chapter 20’s arithmetic about trials applies to experiments as much as to backtests, and a desk that runs twenty experiments a year will see one false saving a year at 5%.

21.6 Tutorial: is the new algorithm better?

21.7 Build: the experiment toolkit

21.8 Exercises

21.9 Problem: Is the New Algorithm Better?

21.10 Interview questions