27Testing and Continuous Delivery

In the last days of July 2012 a broker-dealer deployed new order-routing code to its eight routing servers, in stages. The new code reused a flag that had once switched on a function retired years before and never deleted. A technician did not copy the new code to one of the eight servers; no second person reviewed the deployment, and no written procedure required one. On 1 August orders carrying the reused flag reached the eighth server, woke the retired function, and in forty-five minutes the firm sent millions of orders and lost more than $460 million. Book 11 reads the episode for its missing risk limits; this chapter reads it for its deployment. Every piece of it — the untested old code, the reused flag, the partial deployment, the missing check — is something a test pipeline and a deployment procedure exist to catch.

27.1 The test pyramid for trading code

Trading code adds levels of its own. Book 13, chapter 25, describes property-based, differential and fuzz tests, performance regression gates and exchange certification for the low-latency path; chapter 12’s parity harness is an integration test between the simulator and the production adapter; Book 6’s model validation is a review, not a test, but it produces the reference values tests use. This chapter adds the test a pricing library needs most, and the controls that decide what reaches production.

27.2 Golden files for pricers

A pricing library’s outputs are numbers without an independent oracle for most products: a closed form covers the simplest, and Book 5’s reference pricers cover some more. What a golden file adds is memory: the value the library gave yesterday, reviewed then, against which today’s build is compared. The JPMorgan task force report on the 2012 losses in the Chief Investment Office found that the new VaR model ran through spreadsheets filled by copying and pasting, and that one of them divided by the sum of two rates instead of their average, which likely muted volatility by a factor of two. A golden value for the VaR of a reference portfolio, reviewed once, would have moved when the model changed.

The chapter’s portfolio has sixty instruments priced by Book 5’s library: twenty European options (analytic engine), ten American puts (finite differences), twenty Asian calls (Monte Carlo, 20 000 paths) and ten down-and-out calls (closed form). The tolerance is the engine’s own error times a factor $k$: a relative $10^{-10}$ for the closed forms, $k$ times the gap between the 200-point and the 400-point grid for the PDE, $k$ times the Monte Carlo standard error — estimated from eight independent seeds — for Monte Carlo.

def tolerance(key: str, inst, eng, k: float) -> float:
    """The engine's own error, times k (closed forms: a relative 1e-10)."""
    if isinstance(eng, FP.MonteCarloEngine):
        xs = [pv(inst, dataclasses.replace(eng, seed_offset=100 + j))
              for j in range(N_SE_SEEDS)]
        return k * float(np.std(xs, ddof=1))
    if isinstance(eng, FP.PDEEngine):
        return k * abs(pv(inst, eng) - pv(inst, FP.PDEEngine(m=400, n_t=400)))
    return 1e-10 * max(1.0, abs(pv(inst, eng)))

def compare(store: GoldenStore, results: dict, product_of=None) -> Report:
    """Each result against its golden value; a key missing from either side is a failure."""
    product_of = product_of or (lambda k: k.split(":")[0])
    fails, by = [], {}
    for k in sorted(set(store.items) | set(results)):
        if k not in store.items or k not in results:
            fails.append((k, math.inf, 0.0))
        else:
            g = store.items[k]
            diff = results[k] - g.value
            if not abs(diff) <= g.tol:
                fails.append((k, diff, g.tol))
                by[product_of(k)] = by.get(product_of(k), 0) + 1
    return Report(fails, by, len(results))

Five builds are then compared with the golden file. Each plants one change in the library’s behaviour.

def build(change: str, key: str, inst, eng, seed: int = 1) -> float:
    """The new build's price of one instrument, for each planted change."""
    if change == "seed" and isinstance(eng, FP.MonteCarloEngine):
        return pv(inst, dataclasses.replace(eng, seed_offset=seed))
    if change == "day count":
        kw = {"expiry": _stretch(inst.expiry)}
        if isinstance(inst, FP.AsianOption):
            kw["fixing_dates"] = tuple(_stretch(x) for x in inst.fixing_dates)
        return pv(dataclasses.replace(inst, **kw), eng)
    if change == "put sign" and getattr(inst, "right", "") == "P":
        return pv(inst, eng, market(rate=-0.03))
    if change == "Asian fixing" and isinstance(inst, FP.AsianOption):
        return pv(dataclasses.replace(inst, fixing_dates=inst.fixing_dates[:-1]), eng)
    if change == "finer grid" and isinstance(eng, FP.PDEEngine):
        return pv(inst, FP.PDEEngine(m=400, n_t=400))
    return pv(inst, eng)

The two large regressions are caught at every tolerance: the sign error changes all twenty puts by hundreds of error units and more, and the day count changes every European, American and barrier price by far more than its tolerance. The trouble is Monte Carlo (Figure 27.2). With the seed free to change, the golden test faces two estimates of the same price and must tolerate their difference. At $k=2$ noise alone flags 18% of the Asian calls and fails every one of forty runs; at $k=5$ it fails 2.5% of runs, but the Asian-fixing bug — a small change, two to ten standard errors — is caught on only 9 of the 20 Asian calls, and the day count’s effect on them, never more than two standard errors, on none. With a false alarm costing $1 000 of investigation and a missed single-product regression $500 000 — the model’s assumptions — the cheapest tolerance is $k\le2$, a test that fails on every run. No tolerance works.

The fix is not a tolerance but a seed. Book 5’s Monte Carlo engine draws its random numbers from a seed fixed per instrument; if the build pins it, today’s Monte Carlo price is a deterministic function of the code, the same random numbers give the same price to the last digit, and the Monte Carlo instruments can be compared at a closed form’s tolerance. Pinned, the golden test catches the Asian-fixing bug on all 20 instruments and the day count on all 60. A new seed then fails the test like any other change — which it is — and is released as an approved update of the golden values. The same holds for the finer PDE grid: it moves each American put by exactly one error unit, passes every $k\ge1$, and should reach production as a reviewed golden update with the new grid in its provenance, not as a silent pass.

27.3 Continuous integration

The pipeline’s order is the pyramid’s: cheap tests first, so that most failures are found in seconds. On the chapter’s portfolio, the unit tier (put–call parity on a thousand options) takes 0.02 seconds, the golden tier (the sixty instruments against their pinned values) 0.33 and the end-to-end tier (the portfolio revalued in five scenarios) 1.62, measured on an otherwise idle machine. The golden file is small enough to run on every merge; a firm’s real one, with thousands of instruments, runs in the nightly integration build and on every change to the library.

27.4 Release trains and change control

The train’s cadence sets a trade-off (Table 27.1). Over a model year of 533 changes, deploying each change as it passes the pipeline gives no wait and one change per deployment; a daily train makes changes wait half a day on average and ships 1.8 at a time; a weekly train, 3.5 days and 10.3 changes; a fortnightly one, 7 days and 20.5. The batch matters when a deployment goes wrong: finding the bad change among ten takes about four bisection steps, among twenty about five, and each step is a rollback, a redeployment and a wait for the symptom. Trading firms usually settle between the two: small changes often for research and analytics, trains with a change window for the order path, where RTS 6 requires a clearly delineated methodology to develop and test a system before its deployment or substantial update, in an environment separated from production.

27.5 Deploying and rolling back

The hook’s deployment failed in the step between installation and switch: seven hosts ran the new release, one ran the old, and nothing compared them. A deployment check is simple to write. Each host reports the hash of the build it runs and of its configuration, and the release is refused — the traffic switch does not happen — unless every host reports the release’s hashes. Run on the hook’s eight servers, it names the eighth before the market opens.

def host_state(build: dict, config: dict) -> tuple[str, str]:
    """What a host reports: the hash of the build it runs and of its configuration."""
    return _h(build), _h(config)


def consistency(hosts: dict, release: tuple) -> list[str]:
    """Hosts whose build or configuration differs from the release's: the release is refused
    unless the list is empty."""
    return sorted(h for h, state in hosts.items() if state != release)

The order also lists what else was missing: the retired code was still present and callable, and the flag it had once used was given a new meaning. Both are testing failures in this chapter’s sense. Dead code should be deleted, not left behind a flag; a flag should never be reused, since an old host or an old message will still give it the old meaning; and a feature flag (chapter 16) switched on by a release is a deployment of its own, with its own staged rollout and its own check that every host agrees on what it means.

27.6 Tutorial: seven of eight servers

27.7 Build: golden tests and deployment checks

27.8 Exercises

27.9 Problem: Seven of Eight Servers

27.10 Interview questions