Research Craft: Predictors, Backtests, Measurement, Portfolios · Research

17Event-Driven Backtests

A strategy that rests limit orders one tick either side of the last price fills, in a bar backtest, every time a bar’s range reaches its price. Asked instead to see the price trade one tick through its order before counting a fill, the same backtest fills 55% as often and turns a small daily profit into a loss of $584 a day, with fills followed by a two-tick move against it. Add an order-entry latency of five seconds, and the backtest’s answer depends on a question the bars cannot answer: did the price touch the order while it was live? Level 1 of chapter 16 cannot even express such a strategy. The event-driven backtest can: it simulates orders on a clock, and it makes every assumption about fills an explicit, replaceable model. This chapter builds it (firm.evbt) and shows how much of a passive strategy’s result is the fill model’s.

17.1 Architecture: clock, events, handlers

Definition 17.1 (Event-driven backtest, simulated clock)

An event-driven backtest processes a time-ordered stream of events (market data, order acknowledgements, fills, cancellations, corporate actions) one at a time, each changing the state of the strategy, its orders and its portfolio. Its simulated clock is the time of the event being processed; nothing may use information stamped later.

The architecture of firm.evbt: one queue of events ordered by time, kind and sequence number, a dispatcher, and handlers that may only schedule events in the future. — **Figure 17.1.** The architecture of `firm.evbt`: one queue of events ordered by time, kind and sequence number, a dispatcher, and handlers that may only schedule events in the future.

firm.evbt keeps one priority queue. At each bar’s end the dispatcher first matches the working orders against the bar through the fill model, then marks the portfolio, then calls the strategy, which may submit or cancel orders; a submission becomes a working order only when its acknowledgement event arrives, after the order-entry latency, and a cancellation takes effect only when its own event does. Events at the same time are ordered by kind (bars before order events) and then by a sequence number, so a run is deterministic: the same data, strategy and models give the same fills to the last digit. The result is the BacktestResult of chapter 16, with the orders and fills attached.

17.2 Orders and fills at bar level

Definition 17.2 (Order lifecycle, partial fill)

The order lifecycle is the sequence of states an order passes through: pending (sent, not yet acknowledged), working, partially filled, and one of filled or cancelled. A partial fill executes part of an order’s quantity, leaving the rest working.

Definition 17.3 (Fill model, touch fill, penetration fill, volume participation cap)

A fill model decides from market data whether a working order executes, at what price and for what quantity. The touch fill fills a limit order in full when a bar trades at its price; the penetration fill only when the bar trades a given number of ticks beyond it; a volume participation cap limits the quantity filled in a bar to a share of the bar’s volume.

A bar records four prices and a volume; it does not record where the strategy’s order stood in the queue at its price, how much traded there, or whether the trades at that price came before or after the order arrived. Each fill model fills that gap with an assumption. The touch fill assumes the order was first in the queue. The penetration fill assumes it was last: if the price traded through, everything ahead of it must have traded. The volume cap assumes the order can take a fixed share of the bar’s volume; the open-source backtester Zipline’s VolumeShareSlippage model, for instance, caps fills at 2.5% of each bar’s volume by default and prices them off the bar’s close with a quadratic impact. The truth lies between the touch and penetration fills and depends on the queue (chapter 18).

The chapter’s strategy (rs_evbt.Quoter) runs on one-minute bars built from four simulated 6.5-hour days of firm.tape: at every bar’s close it cancels its orders and rests one lot (100 shares) one tick below the close and one lot one tick above, within an inventory of five lots, marking to the close. Per day, on average:

fill model (no latency)	lots filled	fill rate	P&L ($)	mark-out (ticks)
touch	523	70%	$+17$	$-0.16$
penetration by one tick	290	41%	$-584$	$-2.00$
touch, capped at 2.5% of volume	388	55%	$+20$	$-0.18$
level 1: buy after a down bar, sell after an up bar	—	—	$-215$	—

The mark-out (the close five bars after a fill, in the fill’s direction) measures adverse selection: the penetration fills are the ones that happen when the price is moving through the order, and they lose two ticks on average in five minutes (Figure 17.2). The touch fill’s daily P&L (from $-\$333$ to $+\$206$ over the four days) is within its noise of zero; the penetration fill’s is negative every day. A fill rate of 70% for an order one tick away from the price is itself a warning: a real order joins the back of a queue that may hold thousands of shares.

17.3 Decision time and latency

Definition 17.4 (Latency and the live-in-bar policy)

With an order-entry latency, an order sent at a bar’s close is live from some moment within the next bar. A conservative policy lets an order fill in a bar only if it was live for the whole bar; an optimistic policy, if it was live at any moment of it. A bar cannot say which is right, because it does not say when within it the price reached the order.

With a latency of five seconds, the quoter’s orders, cancelled and replaced at every bar’s close, are never live for a whole bar: under the conservative policy they never fill. Under the optimistic policy they fill more often than with no latency (585 lots a day for the touch fill), and lose more: $-\$41$ a day for the touch fill, $-\$776$ for the penetration fill, $-\$190$ for the capped fill, with the capped fills’ mark-out falling from $-0.18$ to $-0.87$ ticks. The two policies bracket the answer, and a strategy whose result swings from nothing to a loss between them cannot be judged at level 2. That is the signal to move to level 3, where the order’s arrival, its queue position and every trade at its price are events of their own.

The passive quoter on four simulated days under each fill model, with no latency and with a five-second latency under the optimistic policy (under the conservative policy it never fills), and the level-1 version of the same idea. Data: rs_evbt on firm.tape bars. — **Figure 17.2.** The passive quoter on four simulated days under each fill model, with no latency and with a five-second latency under the optimistic policy (under the conservative policy it never fills), and the level-1 version of the same idea. Data: `rs_evbt` on `firm.tape` bars.

17.4 Calendars and corporate actions in the loop

Events that a vectorised backtest folds into adjusted prices are, in an event-driven one, events in the queue. A split multiplies the position and the working orders’ quantities by its ratio and divides their limit prices by it, at the moment it takes effect; the bars keep the prices as traded, so a limit order that survived a split unadjusted would sit at twice the new price and fill instantly. A dividend pays cash on its payment date to positions held at the record date. Half-days end the session early, auctions replace continuous trading at the open and the close (Book 1, chapter 13), and a halt freezes the book; each belongs in the calendar the dispatcher reads, with its own handler, and each has caused a backtest to trade at a time the market was closed. firm.evbt implements splits and leaves the others as the build’s stretch; its acceptance test holds a lot through a two-for-one split and checks that the position doubles and the capital does not move.

17.5 Determinism

A backtest that gives different answers on two runs cannot be debugged, compared or reviewed. Determinism needs three things: a total order on events (time, then kind, then sequence number; never the order in which a dictionary happens to iterate), no randomness except from seeded generators passed in explicitly, and no dependence on the wall clock. firm.evbt’s tests run the same backtest twice and compare the net returns bit for bit. The same discipline lets a level-2 and a level-3 backtest be compared on the same events, and a live run be replayed against its own logs (chapter 19).

17.6 Tutorial: the limit order that always filled

Goal. Run the passive quoter in firm.evbt under three fill models and two latencies with both live-in-bar policies, and compare with the level-1 version of the same idea. End state: Figure 17.2; the table.

The fill models: touch, penetration, and a cap on the bar’s volume.

class TouchFill:
    """Limit: fills in full at its price if the bar's range reaches it. Market: fills at the bar's open."""

    def fill(self, order: Order, bar: dict, used: float = 0.0):
        if order.kind == "market":
            return order.remaining, bar["open"]
        if order.side > 0 and bar["low"] <= order.price or order.side < 0 and bar["high"] >= order.price:
            return order.remaining, order.price
        return None


class PenetrationFill(TouchFill):
    def __init__(self, ticks: float = 1.0):
        self.ticks = ticks

    def fill(self, order: Order, bar: dict, used: float = 0.0):
        if order.kind == "market":
            return order.remaining, bar["open"]
        k = self.ticks
        if order.side > 0 and bar["low"] <= order.price - k or order.side < 0 and bar["high"] >= order.price + k:
            return order.remaining, order.price
        return None


class VolumeCapFill:
    def __init__(self, inner, participation: float = 0.1):
        self.inner, self.participation = inner, participation

    def fill(self, order: Order, bar: dict, used: float = 0.0):
        f = self.inner.fill(order, bar)
        if f is None:
            return None
        room = self.participation * bar["volume"] - used
        q = min(f[0], max(room, 0.0))
        return (q, f[1]) if q > 0 else None

Listing 17.1. Three fill models for bars. code/firm/evbt/firm_evbt.py

Matching: which orders were live in the bar, and what the model fills.

    def _live(self, o: Order, bar: dict) -> bool:
        if self.policy == "conservative":
            return o.active <= bar["start"] and o.cancelled_at >= bar["end"] and o.status != "cancelled"
        return o.active < bar["end"] and o.cancelled_at > bar["start"]

    def _match(self, i: int, bar: dict) -> None:
        used = 0.0
        cand = [o for o in self.orders.values() if o.remaining > 1e-12 and o.status not in ("filled", "pending")]
        for o in sorted(cand, key=lambda o: o.oid):
            if not self._live(o, bar):
                continue
            f = self.fill_model.fill(o, bar, used)
            if f is None:
                continue
            q, px = f
            used += q
            o.avg_price = (o.avg_price * o.filled + px * q) / (o.filled + q)
            o.filled += q
            self._status(o, "filled" if o.remaining <= 1e-12 else "partial", bar["end"])
            fee = self.fee * q
            self.position += o.side * q
            self.cash -= o.side * q * px + fee
            fl = Fill(o.oid, bar["end"], o.side * q, px, fee)
            self.fills.append(fl)
            self.strategy.on_fill(self, fl)

Listing 17.2. Matching working orders against a bar. code/firm/evbt/firm_evbt.py

Run summary(), level1() and fig_evbt.py.

What to change next. Rest the orders at the close instead of one tick away and watch the touch model’s fill rate approach 100%; add a cancel-only-if-the-price-moved rule so orders survive whole bars under latency.

17.7 Build: the level-2 backtester

Purpose. The firm’s backtester for strategies whose orders, fills and timing matter at bar resolution: limit orders, stop orders, intraday schedules, corporate actions in the loop.

Interface. Order, Fill, TouchFill(), PenetrationFill(ticks), VolumeCapFill(inner, participation), Strategy.on_bar(ctx, i, bar), Strategy.on_fill(ctx, fill), Engine(bars, strategy, fill_model, latency, fee, capital, splits, policy).run() returning the BacktestResult, the orders and the fills.

Rules. Handlers schedule only future events; orders fill only after their acknowledgement; every fill assumption lives in a fill model; the live-in-bar policy is stated; the run is deterministic.

Acceptance tests. code/firm/evbt/tests/: touch, penetration and capped fills on hand bars with their lifecycles; a market order at the next open; a latency that misses a bar; cancellations; the optimistic policy filling an order live for an instant; fees, P&L and a split; two runs identical.

Stretch. Dividends, half-days, auctions and halts as calendar events; stop orders; several instruments on one clock.

Sources and further reading

Zipline (open-source Python backtester), zipline/finance/slippage.py: VolumeShareSlippage, default volume limit of 2.5% of each bar.

17.8 Exercises

Exercise 17.1 ★

A buy limit at 99.00 is working. The next bar has open 99.20, high 99.30, low 99.00 and close 99.10. Does it fill under the touch fill? Under a one-tick penetration fill with a tick of 0.01?

Solution

Solution of Exercise 17.1.

Touch fill: the low reached 99.00, so yes, in full at 99.00. Penetration by one tick: the low would have to reach 98.99; it did not, so no.

Exercise 17.2 ★

An order for 5 000 shares faces bars of 40 000, 10 000 and 60 000 shares with a 2.5% volume cap. How much fills in each bar?

Solution

Solution of Exercise 17.2.

2.5% of each bar: 1 000, 250 and 1 500 shares, 2 750 in all, leaving 2 250 working after the third bar.

Exercise 17.3 ★

Draw the lifecycle of an order for 300 shares that fills 100, then is cancelled with 200 remaining.

Solution

Solution of Exercise 17.3.

pending (sent) $\to$ working (acknowledged) $\to$ partial (100 filled, 200 remaining) $\to$ cancelled (the 200 withdrawn when the cancellation takes effect); the 100 filled shares stay filled.

Exercise 17.4 ★★

Why are penetration fills followed by adverse price moves more than touch fills? Relate it to the queue at the order’s price.

Solution

Solution of Exercise 17.4.

A trade through the order’s price means the whole queue at that price was consumed, which happens when the flow on the other side is heavy and persistent, often informed (chapter 9); the price then keeps moving. A touch can happen with a queue barely dented and the price turning back. The penetration fill therefore selects the fills a real order at the back of the queue would get, and those are the adverse ones: on the synthetic days, $-2.0$ ticks of mark-out against $-0.16$ .

Exercise 17.5 ★★

With a latency of 5 seconds and one-minute bars, why does the quoter never fill under the conservative policy? How would a strategy that cancels only when the price moves fare?

Solution

Solution of Exercise 17.5.

Every order is sent at a bar’s close, becomes live 5 seconds into the next bar and is cancelled 5 seconds into the one after; it is never live for a whole bar. A strategy that leaves its orders alone while the price is unchanged keeps them live for many whole bars, and the conservative policy can fill them.

Exercise 17.6 ★★

A two-for-one split takes effect while a sell limit at $120 for 100 shares is working and the position is 100 shares. What must the engine do, and what goes wrong if it does not?

Solution

Solution of Exercise 17.6.

At the split’s effective time: position to 200 shares, the order to 200 shares at $60. Without it, the order still asks $120 for 100 shares in a market now around $60: it never fills (the book is unhedged against its intended exit), or, for a buy limit left at the old price, it fills at once at twice the market.

Exercise 17.7 ★★★

Coding. Run the quoter with the orders at the close price (not one tick away) under the touch and penetration fills. What happens to the fill rates and the mark-outs?

Solution

Solution of Exercise 17.7.

rs_evbt.at_close: under the touch fill 95% of orders fill, with a mark-out of $-0.17$ ticks and a loss of $188 a day; under the penetration fill 70% fill, with $-1.16$ ticks and a loss of $506 a day. Orders at the price that just traded are touched almost always, which says nothing about whether a real order at the back of its queue would fill.

Exercise 17.8 ★★★

Find the flaw. “Our backtest fills limit orders when the bar’s low reaches them, and the strategy made money every week for two years.”

Solution

Solution of Exercise 17.8.

The fill model assumes the order was first in the queue at its price, and fills it on every touch, including all the touches where the price turned back, the fills a real order would not get; it ignores the adverse fills’ selection. Rerun with penetration fills and a volume cap, check the mark-outs, and confirm at level 3 or with live fills before believing it.

17.9 Problem: The Limit Order That Always Filled

Problem 17.1

Weekend problem — a passive strategy, three fill models

The chapter’s quoter on four simulated days of firm.tape one-minute bars.

Part I — The engine.

In what order does firm.evbt process a bar’s close: fills, marks, strategy? Why that order?
When does a submitted order first become able to fill?
What makes two runs identical?
What does the engine return?

Part II — Fill models.

Give the lots filled, fill rate, P&L and mark-out a day under each fill model with no latency.
Which assumption about the queue does each model make?
Why is the penetration fill’s P&L negative every day while the touch fill’s is not?
What is the level-1 version of the idea, and what does it earn?

Part III — Latency.

What happens at a latency of 5 seconds under the conservative policy?
And under the optimistic policy, for each fill model?
What does the difference between the two policies tell you about the strategy?
How would you change the strategy so that latency matters less?

Part IV — The verdict.

State the named result: the fill rate and P&L of the passive bar strategy under touch, penetration and capped fills, with and without latency.
Which result would you report to a portfolio manager, and with which caveat?
What would a level-3 replay add (chapter 18)?
What does Zipline’s default cap mean for a strategy that trades 5% of volume?
Which corporate actions belong in the event queue for this strategy?
Why is a 70% fill rate for orders one tick away suspicious?
What would you log from a live run to calibrate the fill model?
In one sentence: what is the fill model?

Solution

Solution of Problem 17.1.

Fills against the bar first (the orders were working during it), then the mark at its close, then the strategy’s reaction; so the strategy sees the fills and the portfolio it actually had, and its new orders act only on later bars.
When its acknowledgement event arrives, after the order-entry latency; and under the conservative policy only in bars that begin after that.
A total order on events (time, kind, sequence number), seeded randomness passed in explicitly, no wall-clock dependence.
The BacktestResult of chapter 16, the orders with their lifecycles, and the fills.
Touch: 523 lots, 70%, $+\$17$ , $-0.16$ ticks. Penetration: 290 lots, 41%, $-\$584$ , $-2.00$ ticks. Capped at 2.5%: 388 lots, 55%, $+\$20$ , $-0.18$ ticks.
Touch: first in the queue. Penetration: last in the queue. Capped: a fixed share of the volume, wherever the order stands.
Its fills are only those where the price traded through, which are followed by moves against the position; the touch fill adds the benign fills where the price turned back.
Hold one lot long after a down bar and one short after an up bar, traded at the close with half a tick of cost: $-\$215$ a day.
No fills at all: no order is live for a whole bar.
Touch: 585 lots, $-\$41$ ; penetration: 372 lots, $-\$776$ ; capped: 440 lots, $-\$190$ , with its mark-out falling to $-0.87$ ticks.
That the answer depends on when within the bar the price reached the orders, which bars do not record: the strategy must be judged at level 3.
Keep orders working while the price is unchanged, cancel only when it moves, and quote further from the price.
Named result. With no latency: touch 70% of orders filled and $+\$17$ a day, penetration 41% and $-\$584$ , capped 55% and $+\$20$ ; with a 5-second latency: nothing under the conservative policy, and $-\$41$ , $-\$776$ and $-\$190$ under the optimistic one. The fill model moves the answer by $800 a day; the strategy’s own edge is within its noise.
The penetration result, as the pessimistic bound, with the statement that the touch result assumes first place in every queue and that the level-2 results bracket the answer without locating it.
Our own orders in the queue at their prices, every trade at those prices, and latency on both the order and the data side: fills that happen or not for the reasons they would live.
At most 2.5% of each bar fills; a strategy that needs 5% leaves half its orders working and trades later, at other prices.
Splits, dividends (for the cash and the price gap), half-days, auctions and halts.
Because a real order joins the back of a queue that may hold thousands of shares at its price; to fill 70% of the time, the price would have to trade through that queue in most minutes.
Each order’s send, acknowledgement, queue estimate, fills and cancellations with timestamps, and the market’s trades at its price; the ratio of live fills to touch fills calibrates the model.
The backtest’s assumption about where its orders stood in the queue, made explicit and replaceable.

17.10 Interview questions

Interview question 17.1 ★ developer, researcher

Describe the architecture of an event-driven backtester.

Solution

Solution of Interview question 17.1.

A priority queue of time-stamped events (market data, order acknowledgements, fills, cancellations, calendar and corporate events), a dispatcher that pops them in order and advances the simulated clock, handlers for the strategy, the order manager (lifecycle), the fill model and the portfolio, all of which may only schedule events in the future; and a recorder that produces the result and the audit trail.

Interview question 17.2 ★★ researcher, trader

How would you simulate limit-order fills on bar data, and what biases does each approach have?

Solution

Solution of Interview question 17.2.

Touch fills (optimistic: first in the queue), penetration fills (pessimistic: last in the queue), volume-capped fills (a share of the bar’s volume), probabilistic fills calibrated on live data; each needs a policy for orders live for part of a bar. Biases: touch overstates fills and understates adverse selection; penetration understates fills; caps ignore where the volume traded.

Interview question 17.3 ★★ developer

How do you make a backtest deterministic?

Solution

Solution of Interview question 17.3.

Order events totally (time, then kind, then a sequence number), pass seeded generators explicitly, avoid iteration over unordered containers and any use of the wall clock, and test that two runs produce identical results.

Interview question 17.4 ★★ researcher

What is adverse selection for a passive order, and how would you measure it in a backtest?

Solution

Solution of Interview question 17.4.

The tendency of passive orders to be filled when the price is about to move against them, because the flow that fills them is informed or persistent. Measure the mark-out of each fill: the mid a few seconds or bars later minus the fill price, in the fill’s direction, averaged by fill model, time and size.

Interview question 17.5 ★★ developer, trader

How should a backtester handle a stock split while orders are working?

Solution

Solution of Interview question 17.5.

As an event at its effective time: multiply positions and working quantities by the ratio, divide limit and stop prices by it, keep the bars as traded, and log the adjustment; the same for reverse splits, and a check that capital is unchanged by the event.

Interview question 17.6 ★★★ researcher

When is a level-2 backtest not enough? Give a test that tells you.

Solution

Solution of Interview question 17.6.

When the result depends on assumptions the bars cannot settle: run it under touch and penetration fills, and under conservative and optimistic live-in-bar policies; if the answers bracket zero or differ by more than the edge, it needs level 3 (queue and latency) or live evidence.

Terms defined in this chapter

See all 2333 terms in the glossary