11Backtest-Engine Architecture

A quoting strategy looked rich in the bar backtest, flat in the order-book replay, and lost money in the simulator. The firm had four backtesters, written by four teams in four styles, each with its own idea of a strategy, a clock, an order and a result; the strategy had been ported to each by hand. It took a week to be sure that the three answers differed because the markets differed — bars against queues, queues against a market that reacts — and not because the four ports of the strategy did. This chapter builds the engine that makes that question a query: one strategy object, one event model, one data-access layer and one results store over the four fidelity levels of One Quant Book 7 (chapter 16), the fourth being a reactive simulation on the exchange simulator of One Quant Book 10.

11.1 One engine, four levels

The firm’s backtesters already exist: Book 7’s vectorised backtester (firm.vecbt, level 1), its event-driven backtester on bars (firm.evbt, level 2) and its order-book replay with queue positions (firm.lobreplay, level 3), and Book 10’s exchange simulator (firm.exchsim). Book 7 defined level 4 as live trading at small size or paper trading, and in its chapter 19 let an agent trade inside the synthetic market as the stand-in for live; here the exchange simulator plays that part, with the same session replayed as real orders against which the strategy’s own orders rest and trade. The engine does not rewrite any of them. Each level has an adapter that turns the one strategy object into what that backtester expects, and turns what the backtester reports into the engine’s events (Figure 11.1).

The strategy interface has two forms, because the levels differ in what a strategy is. At level 1 there are no orders: a strategy is a rule that maps closes to a target position, run over all dates at once. At levels 2 to 4 a strategy reacts to the market and to its own fills by sending and cancelling orders. A TargetStrategy states its rule once, target(close, i), and the engine runs it vectorised at level 1 and bar by bar at level 2; a QuoteStrategy has two callbacks, on_market and on_fill, and runs at levels 2 to 4. Asking for a quoting strategy at level 1 is an error, not a silently empty backtest.

class Quoter(B.QuoteStrategy):
    """A lot at the best bid and ask; requote when the touch moves; stop at the limit."""

    name = "quoter"

    def __init__(self, lots: int = 1, limit: int = 3):
        self.qty, self.limit, self.mine = 100 * lots, 100 * lots * limit, {}

    def on_market(self, ctx, t, snap):
        b, a = snap["bid"], snap["ask"]
        if b is None or a is None:
            return
        live = set(ctx.working())
        self.mine = {v: sp for v, sp in self.mine.items() if v in live}
        have = set()
        for v, (side, px) in list(self.mine.items()):
            if px != (b if side == 1 else a):               # the touch moved: cancel
                ctx.cancel(v)
                del self.mine[v]
            else:
                have.add(side)
        for side, px in ((1, b), (-1, a)):
            if side not in have and side * ctx.position < self.limit:
                self.mine[ctx.send(side, px, self.qty)] = (side, px)

class Engine:
    def __init__(self, level: int, strategy, data: DataAccess, config: dict):
        if level not in LEVELS:
            raise ValueError(f"level {level} not in {LEVELS}")
        if level == 1 and not isinstance(strategy, TargetStrategy):
            raise TypeError("level 1 runs a TargetStrategy: no orders, no queues")
        self.level, self.strategy, self.data = level, strategy, data
        self.cfg = dict(config)

    def manifest(self, pnl: float, n_fills: int) -> dict:
        name = self.cfg["dataset"]
        return {"level": self.level, "strategy": type(self.strategy).__name__,
                "strategy_code": code_hash(self.strategy), "engine_code": ENGINE_CODE,
                "data": {name: self.data.hash(name)},
                "params": {k: v for k, v in sorted(self.cfg.items()) if k != "dataset"},
                "seed": self.cfg.get("seed", 0), "pnl": round(float(pnl), 6),
                "fills": n_fills}

    def run(self) -> Run:
        level = {1: self._level1, 2: self._level2, 3: self._level3, 4: self._level4}
        pnl, pos, fills, log, mids = level[self.level]()
        fills = np.array(fills, dtype=float).reshape(-1, 4)
        mids = np.array(mids).reshape(-1, 2)
        return Run(self.level, pnl, pos, fills, log.ordered(), mids,
                   self.manifest(pnl, len(fills)))

11.2 The event model and the clock

Four classes suffice for the chapter’s strategies, in this rank order: market (the top of the book the strategy saw), fill (what the market did to its orders), order and cancel (what the strategy did in response). The rank settles the question every event-driven engine must answer and most answer by accident: when a quote update and a fill carry the same time, which does the strategy see first? A fixed answer makes two runs of the same inputs identical and two levels comparable; the sequence number breaks the remaining ties in arrival order.

LEVELS = (1, 2, 3, 4)
CLASSES = ("market", "fill", "order", "cancel")
RANK = {c: i for i, c in enumerate(CLASSES)}


# ------------------------------------------------------------ event model
@dataclass(order=True, frozen=True)
class Event:
    t: float                        # seconds from the session start (level 1-2: bar end)
    rank: int                       # RANK[cls]: simultaneous events in a fixed class order
    seq: int                        # arrival order within the run: ties broken the same way
    cls: str = field(compare=False)
    data: tuple = field(compare=False, default=())


class EventLog:
    def __init__(self):
        self.events: list[Event] = []

    def add(self, t: float, cls: str, *data) -> None:
        self.events.append(Event(float(t), RANK[cls], len(self.events), cls, tuple(data)))

    def ordered(self) -> list[Event]:
        return sorted(self.events)

The clock is the simulated clock of Book 7 (chapter 17): time moves from event to event, never from the machine. Each backtester keeps its own clock internally — bar ends at level 2, message times at level 3, nanoseconds of the simulated day at level 4 — and the adapters convert all of them to seconds from the start of the session, so that events of different levels can be laid side by side.

11.3 Data access

The chapter’s layer stores named datasets as Parquet (chapters 3 and 4) and keeps a content hash in each file’s metadata: a firm.tape session of Book 7 (the order-by-order messages of a synthetic hour), and daily bars. The hash is what makes “the same data” checkable: two runs that read datasets with equal hashes read the same bytes, whatever the files were called, and a run’s manifest names the hash, not the path. In the firm the same interface fronts the tick store of chapter 4 and the reference data of chapter 6; a strategy never opens a file.

11.4 The results store

A run’s identifier is the hash of its manifest (Listing 11.4): the same strategy code, engine code, data, parameters and seed give the same identifier, so a rerun is recognised rather than stored twice, and a changed line of the strategy gives a new one. The store answers the questions the week of the chapter’s hook was spent on: which runs of this strategy exist, at which levels, on which data, with which code; and, for two runs, where their events first differ.

    def save(self, run: Run) -> str:
        d = self.root / run.run_id
        d.mkdir(exist_ok=True)
        (d / "manifest.json").write_text(json.dumps(run.manifest, sort_keys=True, indent=1))
        f, ev = run.fills, run.events
        fills = {"t": f[:, 0], "side": f[:, 1], "qty": f[:, 2], "price": f[:, 3]}
        pq.write_table(pa.table(fills), d / "fills.parquet")
        events = {"t": [e.t for e in ev], "cls": [e.cls for e in ev],
                  "data": [json.dumps(e.data) for e in ev]}
        pq.write_table(pa.table(events), d / "events.parquet")
        return run.run_id

11.5 Adding a level: the reactive simulation

At level 3 the strategy’s orders are shadows (Book 7, chapter 18): they take a place in the replayed queue and fill when the recorded executions reach them, but the replay does not change — a market order that filled a shadow also fills, in the recording, the orders it filled that day. At level 4 the same firm.tape session is replayed through Book 10’s simulator as real orders: each recorded addition becomes a limit order, each cancellation a cancel, each trade a market order of its size. The strategy’s orders are among them. A market order that meets the strategy’s order trades with it and not with the order behind it, which then rests longer and trades later with something else; when the orders at the touch leave, the strategy’s order is left alone at the front, visible to everyone, where the replay’s shadow would have vanished with the level it was hiding in.

    def _level4(self):
        meta = self.data.meta(self.cfg["dataset"])
        secs, lat = float(meta["seconds"]), int(round(self.cfg.get("latency", 0.0) * 1e9))
        sim = X.Simulator(_venue(secs), seed=self.cfg.get("seed", 1))
        tape = TapeConfig(seconds=secs, seed=int(meta["seed"]), news_at=None)
        sim.add_background(X.TapeBackground(tape))     # the same session, as real orders
        log, fills, side, rctx = EventLog(), [], {}, [None]

        class Inner(firm_lobreplay.Strategy):          # what ReplayStrategyAdapter drives
            def on_market(self, ctx, t, snap):
                rctx[0] = ctx
                wrap.market(t, snap)

            def on_fill(self, ctx, vid, t, qty, price):
                rctx[0] = ctx
                fills.append((t, side[vid], qty, price))
                wrap.fill(vid, t, qty, price, side[vid])

        def send(s, price, qty):
            vid = rctx[0].send(s, price, qty)
            side[vid] = s
            return vid

        back = _Back(send, lambda v: rctx[0].cancel(v), lambda: rctx[0].working(),
                     lambda: rctx[0].position)
        wrap = _Wrap(self.strategy, _Ctx(back, log))
        lm = X.LatencyModel(entry_ns=lat, ack_ns=lat, data_ns=lat)
        sim.add_agent(X.ReplayStrategyAdapter(Inner()), X.SessionSpec("BT", latency=lm))
        sim.run()
        pos = sum(s * q for _, s, q, _ in fills)
        cash = -sum(s * q * p for _, s, q, p in fills)
        mid = wrap.mids[-1][1] if wrap.mids else 0.0
        return cash + pos * mid, pos, fills, log, wrap.mids

Level 4 is the counterfactual impact of Book 7 (chapter 18) made concrete, with its limits: the background flow is a recording, so its limit orders do not requote around the strategy and its informed traders do not change their minds. It measures the mechanical part of the strategy’s presence — queues and matching — and not the strategic part, which needs agents (Book 10’s firm.agentmkt).

11.6 The P&L ladder

The chapter’s experiment runs the quoting strategy of Listing 11.1 on twenty one-hour sessions of firm.tape (seeds 1 to 20) at levels 2, 3 and 4, with 20 $\mu$s of data and order-entry latency and no fees at any level. At level 2 the strategy sees the touch at the end of each one-second bar that has trades and a quote fills in full when the bar trades at its price (the touch fill of Book 7, chapter 17); at level 3 it takes its place at the back of the queue; at level 4 it is in the queue.

Figure 11.2 is the ladder. At level 2 the strategy makes $228 an hour ($\pm 29$, one standard error over the sessions) from 841 fills: a quote that fills whenever the bar trades at its price never waits in a queue. At level 3 it loses $12 an hour ($\pm 13$, indistinguishable from zero) from 333 fills: behind the queue, it fills less and mostly when the price is about to move through it. At level 4 it loses $102 an hour ($\pm 18$) from 434 fills. The last step is not noise: level 4 is below level 3 in every one of the twenty sessions (Figure 11.3), by $90 an hour on average ($\pm 7.4$).

Because the strategy object is one and the same at every level, the difference between two runs is the market’s; the engine’s event logs say where it begins. first_divergence compares the fills of the level-3 and level-4 runs of each session: they part within the first five fills in all twenty sessions, at a median of 11 seconds into the hour. From then on the two runs are different histories of the same session.

11.7 Where the gap comes from

Which fills make the difference? Each level-4 fill either has a level-3 counterpart — a fill of the same side within a millisecond — or it exists only because the strategy’s order was in the book. Symmetrically, some replay fills have no reactive counterpart. The ten-second markout of each fill (the mid ten seconds later against the fill price, signed) measures what the fill earned before the inventory it created was marked; Figure 11.4 splits the markouts of both runs into the three groups, in the manner of the divergence decomposition of Book 7 (chapter 19).

11.8 A rule at levels 1 and 2

The same engine runs a daily rule — long when the close is above its level fifty days earlier, short when below — on a thousand days of synthetic bars at level 1 (firm.vecbt, the rule’s weight earned from the next close) and at level 2 (firm.evbt, market orders filled at the next open). Level 1 returns $-26.5\%$ over the period and level 2 $-31.5\%$; the five points are the overnight moves on the days the position changes, earned by the old position at level 2 and by the new one at level 1, which trades at a close it has only just seen. The rule is written once; the two runs sit next to each other in the results store under different identifiers.

11.9 Tutorial: one strategy through four backtesters

11.10 Build: the backtest engine

11.11 Exercises

11.12 Problem: Four Backtesters, One Strategy

11.13 Interview questions