28Disaster Recovery and Regulatory Requirements

When Superstorm Sandy reached New York in October 2012, the U.S. national securities exchanges “jointly decided not to open for trading on October 29 and October 30”. The securities industry’s annual test of trading from backup sites had run on October 27, two days before, and found nothing that would have stopped the markets from opening on their backup systems; many firms, the SEC later wrote, “remained uncomfortable about switching over”. Two years later Regulation SCI required the exchanges’ recovery plans to be reasonably designed to resume their critical systems within two hours of a wide-scale disruption, and required the members each venue designates to test its backup site with it at least once a year.

Book 13 described failover, primary–backup replication and the split brain; Book 6 operational risk; Book 11 the kill switch; Book 15 incident response. This chapter is about sites and rules: where a firm puts its recovery site, how exchanges test theirs with their members, which resilience rules a trading firm is examined on, and how a recovery design turns into a recovery time and a recovery point. The recovery simulation is labelled as such; the rules are quoted.

28.1 Secondary sites and how they are placed

Two distances pull against each other. The recovery site must be outside the hazards that could take the primary, a storm, a flood, a regional power failure; and it must be close enough for its replication to be cheap and its connections to the venues usable. The exchanges’ own choices show the scale: CME’s U.S. BrokerTec markets run in Secaucus and recover in Aurora, its European ones run in Slough and recover in Secaucus; LSEG’s spot matching runs in Slough and recovers in Secaucus (chapters 23 to 25). Recovery across a continent or an ocean, not across the river.

The New Jersey sites are all inside any storm that closes Secaucus. Aurora is not, and costs what distance costs: synchronous replication to it would add $17.4\,\mathrm{m}\mathrm{s}$ of round trip to every write at a route factor of 1.5, against $0.5\,\mathrm{m}\mathrm{s}$ to Mahwah, so the firm replicates asynchronously and accepts a recovery point.

28.2 Exchange failover tests

Tests exist because Sandy showed that a tested backup is not a trusted one. Regulation SCI makes them mandatory for the members an exchange designates, at least once every twelve months, coordinated across the industry. The futures industry runs its own: FIA’s 2025 test, on Saturday 25 October with a pre-test in September, had more than a hundred organisations verify round-trip connectivity and process recovery between primary and backup sites (Box 28.1). Exchanges also fail over routinely: CME’s market segment gateways switch to their backup components and back in weekly windows.

28.3 The resilience rules firms are examined on

The rules differ by who they bind but share a shape. The exchange must resume its critical systems within two hours; the trading firm must have arrangements to resume or shut down in order, tested every year; and in the United Kingdom the firm itself names the services that matter and the disruption it will tolerate, then shows it can stay within it. A trading firm’s recovery design is therefore judged on two numbers it sets itself, a recovery time and a recovery point, against the tolerance it declared.

28.4 Recovery objectives and runbooks

A recovery is a runbook: detect the loss and decide to fail over, switch the sessions to the recovery addresses, restore or start what is not running, load positions and reference data, reconnect and verify sessions, reconcile orders and positions against the exchanges’ drop copies, resume. Each step has a duration that varies; the recovery time is their sum.

def rto_samples(plan, n=100000, seed=0):
    rng = np.random.default_rng(seed)
    total = np.zeros(n)
    for s in plan.steps:
        total += s.median_min * np.exp(s.sigma * rng.standard_normal(n))
    return total


def p_within(plan, target_min, n=100000, seed=0):
    return float(np.mean(rto_samples(plan, n, seed) <= target_min))


def rpo(plan, table):
    if plan.replication == "sync":
        return 0.0
    if plan.replication == "async":
        d = separation_km(plan.primary, plan.recovery, table) * 1000.0
        return (plan.lag_ms + gm.floor_us(d, "fibre") / 1000.0) / 1000.0
    return plan.snapshot_min * 60.0


def sync_penalty_ms(plan, table, factor=1.5):
    d = separation_km(plan.primary, plan.recovery, table) * 1000.0
    return 2 * gm.floor_us(d, "fibre") * factor / 1000.0

The chapter’s firm, trading from Secaucus with a recovery site in Aurora, compares two designs (Figure 28.1). The hot design keeps servers running in Aurora with asynchronous replication (a lag of $50\,\mathrm{m}\mathrm{s}$) and sessions ready: its runbook is five steps, with medians of 15, 10, 20, 10 and 5 minutes. The warm design keeps the site and a few servers, restores from snapshots taken every fifteen minutes, and has six steps, including 45 minutes to start and restore and 30 to load data. All durations are assumptions, lognormal around their medians.

HOT = dr.Plan("hot", PRIMARY, "aurora",
              (S("detect and decide", 15, 0.5), S("switch sessions to recovery addresses", 10, 0.4),
               S("reconcile orders and positions", 20, 0.5), S("check venue connectivity", 10, 0.4),
               S("resume trading", 5, 0.3)),
              "async", lag_ms=50.0, annual_cost=0.70 * PRIMARY_COST)
WARM = dr.Plan("warm", PRIMARY, "aurora",
               (S("detect and decide", 15, 0.5), S("start servers and restore", 45, 0.6),
                S("load reference data and positions", 30, 0.5),
                S("reconnect and verify sessions", 20, 0.5),
                S("reconcile orders and positions", 20, 0.5), S("resume trading", 5, 0.3)),
               "snapshot", snapshot_min=15.0, annual_cost=0.25 * PRIMARY_COST)

The hot design resumes within two hours in 99.2% of simulated recoveries, with a median of 64 minutes and a 95th percentile of 96; its recovery point is $56\,\mathrm{m}\mathrm{s}$, the lag plus the one-way delay to Aurora. The warm design meets two hours in 20.3%, with a median of 148 minutes and a 95th percentile of 237, and can lose fifteen minutes of data. If the hot site costs 70% of the primary’s yearly infrastructure and the warm one 25% (chapter 27’s example stack, USD 3.19 million a year; assumptions), the difference is USD 1.44 million a year: the price of the two-hour line for this firm.

28.5 Tutorial: the two-hour rule

28.6 Build: the disaster-recovery plan

28.7 Exercises

28.8 Problem: The Two-Hour Rule

28.9 Interview questions