Microstructure and Execution · Execution
12Transient Impact and Propagator Models
The signs of trades are predictable for hours: a buy is followed by buys long after any one trader’s order has finished. Yet prices do not trend. Something must undo the impact of each trade, and how fast it does decides what an execution costs. This chapter measures the memory of order flow in a simulated market, fits the propagator that maps trades into prices, and asks which shapes of decay a model of impact can have without letting a trader make money by trading in circles.
12.1 The long memory of order flow
Lillo and Farmer (2004) found that on the London Stock Exchange the autocorrelation of order signs decays roughly as a power law in the lag, with an exponent of about 0.6: order flow has long memory (One Quant Book 4, chapter 17). Lillo, Mike and Farmer (2005) explained it by splitting: if metaorders have sizes whose cumulative distribution falls like and are executed in equal pieces, the signs’ autocorrelation falls like , a long memory whenever .
The chapter’s market is chapter 11’s firm.agentmkt population, with the noise takers’ signs now drawn in runs of Pareto length with tail 1.5, as if each run were a metaorder. Over two simulated hours, 10 764 market orders (executions merged), the sign autocorrelation is 0.27 at lag one, 0.106 at lag ten and 0.023 at lag fifty; a power-law fit over lags 2 to 50 gives an exponent of 0.62 (Figure 12.1, left), against 0.5 for the runs alone: the fundamentalists’ orders, whose signs follow the price, dilute the memory. The response keeps growing: 0.52 ticks after one order, 1.05 after ten, 2.3 after a hundred.
12.2 The propagator model
Definition 12.1 (Propagator model, decay kernel)
The propagator model writes the price before trade as the sum of the impacts of all past trades,
where the decay kernel (the propagator) is the impact of one trade trades later, independent of the others.
Bouchaud, Gefen, Potters and Wyart (2004) proposed it and showed that the market sits at a critical point: with correlated signs, a constant would make prices trend and a fast-decaying would make them mean-revert; prices are diffusive only when decays at the rate that compensates the signs’ memory. The kernel is estimated from the data by writing the price change as with and taking expectations against past signs: , a Toeplitz system solved by Levinson recursion.
def fit_kernel(eps, p, maxlag: int):
e, p = np.asarray(eps, float), np.asarray(p, float)
dp = np.diff(p) # dp_t = p_{t+1} - p_t
e = e[: len(dp)]
c = sign_acf(e, maxlag)
s = np.array([float(np.mean(e[: len(e) - k] * dp[k:])) if k else float(np.mean(e * dp))
for k in range(maxlag + 1)])
k = solve_toeplitz(c, s) # Levinson recursion
return k, np.concatenate([[0.0], np.cumsum(k)])
On the simulated flow the fitted kernel is 0.541 ticks one order after a trade, 0.452 ten orders later and 0.419 a hundred later: it decays, slowly. Does it make prices diffusive? The variance of price changes over orders divided by is flat for a random walk (Figure 12.1, right). For the market it falls from 1.98 at lag one to about 1.35 from lag ten on. The propagator driven by the same signs gives 0.29 at lag one, rising to 1.34 at a hundred: it explains the long-lag variance but not the short-lag noise (quotes that move without trades), and its decay, cut at a hundred lags, is not quite fast enough. Driven by the same signs shuffled, with no memory, the same kernel makes prices sub-diffusive, 0.32 falling to 0.18: its decay is what compensates the signs’ memory.
mx_prop.kernel_study.12.3 Transient impact in continuous time
Definition 12.2 (Transient impact model)
A transient impact model prices an execution as the continuous-time propagator: trading at rate moves the price at time by , so an execution at times costs with .
Definition 12.3 (Obizhaeva–Wang model)
The Obizhaeva–Wang model is the transient impact model of a limit order book with a block shape and exponential resilience: the book refills at rate after being consumed, so up to a constant.
Obizhaeva and Wang (2013) found that the optimal execution in their model mixes discrete and continuous trades: selling over , it trades a block of at the start, sells at a constant rate , and trades another block of at the end. The first block uses the book as it is; the continuous trading matches the book’s refilling; the last block uses the refilled book when no later trade will pay for its impact. Minimising under gives , and on a fine grid with an exponential kernel it reproduces the two blocks.
12.4 No-manipulation conditions
Definition 12.4 (No-dynamic-arbitrage condition)
An impact model satisfies the no-dynamic-arbitrage condition when no round trip, a sequence of buys and sells with zero net position, has a negative expected cost: whenever (the kernel is positive definite on round trips).
Definition 12.5 (Transaction-triggered price manipulation)
Transaction-triggered price manipulation is present when the cheapest way to sell (or buy) a position includes trades in the opposite direction, although no round trip is profitable.
Huberman and Stanzl (2004) showed that with time-stationary impact only linear permanent impact rules out quasi-arbitrage; Gatheral (2010) studied which combinations of impact function and decay kernel allow dynamic arbitrage; Alfonsi, Schied and Slynko (2012) proved that impact must decay as a convex non-increasing function of time to rule out both standard and transaction-triggered manipulation. The family shows the three regimes (Figure 12.2): convex for ; for it is flat, then falls, and is still positive definite (it is the characteristic function of a stable law); beyond 2 it decays too fast after a flat start and is not.
def round_trip(gamma) -> float:
n = len(gamma)
q, _ = np.linalg.qr(np.column_stack([np.ones(n), np.eye(n)[:, : n - 1]]))
b = q[:, 1:] # an orthonormal basis of sum x = 0
lam = np.linalg.eigvalsh(b.T @ gamma @ b)
return float(-lam[0] / 2)
def optimal_liquidation(gamma, x_total: float) -> np.ndarray:
one = np.ones(len(gamma))
w = np.linalg.solve(gamma, one)
return w * x_total / (one @ w)
On 21 trades over one unit of time with , the best round trip loses 0.136 (in units of the instantaneous impact times the squared size) for , 0.042 for , 0.008 for 1.5 and 0.002 for 1.8, and earns 0.028 for , 0.091 for 2.5 and 0.213 for 3. At the Gaussian kernel’s matrix is singular on a fine grid, the boundary itself. For and 1.8 no round trip pays but the optimal sale includes two buys, of and of the order: transaction-triggered manipulation. With it sells only.
mx_prop.kernel_family.12.5 Estimating the kernel
The Toeplitz estimator needs only signs and prices, but its answers depend on choices. The maximum lag truncates the kernel, and beyond it the fit treats impact as permanent. Merging executions into orders, or not, changes what a trade is. Other orders move prices too: quote changes without trades add noise at short lags, as the simulated market showed. And the model’s linearity in signs ignores sizes; extensions let the kernel depend on trade size or on whether the trade changed the price. The diffusion check is the model’s own test: a kernel that leaves prices super- or sub-diffusive is missing some of the decay.
12.6 Tutorial: the kernel that paid you to trade
Goal. Measure the memory of order flow, fit a propagator, check diffusion, and find the kernels that pay for round trips. End state: Figures 12.1 and 12.2.
- Flow.
mx_prop.flow(seed, seconds): signs and mids of merged market orders fromfirm.agentmktwith long-memory signs. - Kernel.
firm_propagator.sign_acf,response,fit_kernel;propagatefor the model’s prices. - Diffusion. Variance ratios for the market, the model and the model on shuffled signs (
kernel_study()). - Manipulation.
impact_matrix,round_trip,optimal_liquidationover the kernel family (kernel_family());obizhaeva_wang(X, rho, T); draw withfig_prop.py.
What to change next. Fit the kernel separately for orders that moved the price and those that did not; raise the run tail to 1.8 and see the kernel decay faster.
12.7 Build: the propagator
Purpose. Transient impact for the execution models of chapters 14 and 15, the market-impact simulations of chapter 13, and Book 11’s inventory costs.
Interface. sign_acf, response, fit_kernel(eps, p, maxlag), propagate(eps, G), impact_matrix(times, kernel), round_trip(Gamma), optimal_liquidation(Gamma, X), obizhaeva_wang(X, rho, T).
Rules. Time in trades for the discrete model; the Toeplitz system solved by scipy.linalg.solve_toeplitz (Levinson); round trips on the subspace with unit norm.
Acceptance tests. code/firm/propagator/tests/: a planted power-law kernel recovered from persistent signs; round trips and liquidations for an exponential, a compressed and a stretched kernel; the discrete optimum close to Obizhaeva–Wang’s blocks.
Stretch. Size-dependent and price-changing kernels; the continuous-time kernel from timestamps; the non-linear transient impact of Gatheral.
Sources and further reading
- J.-P. Bouchaud, Y. Gefen, M. Potters and M. Wyart, “Fluctuations and response in financial markets: the subtle nature of ‘random’ price changes”, Quantitative Finance 4(2), 2004.
- F. Lillo and J. D. Farmer, “The long memory of the efficient market”, Studies in Nonlinear Dynamics and Econometrics 8(3), 2004.
- G. Huberman and W. Stanzl, “Price manipulation and quasi-arbitrage”, Econometrica 72(4), 2004.
- F. Lillo, S. Mike and J. D. Farmer, “Theory for long memory in supply and demand”, Physical Review E 71, 2005.
- J. Gatheral, “No-dynamic-arbitrage and market impact”, Quantitative Finance 10(7), 2010.
- A. Alfonsi, A. Schied and A. Slynko, “Order book resilience, price manipulation, and the positive portfolio problem”, SIAM Journal on Financial Mathematics 3(1), 2012.
- A. A. Obizhaeva and J. Wang, “Optimal trading strategy and supply/demand dynamics”, Journal of Financial Markets 16(1), 2013.
12.8 Exercises
Exercise 12.1 ★
The sign autocorrelation is 0.27 at lag one. If the next sign were predicted to equal the last, how often would the prediction be right?
Solution
Solution of Exercise 12.1.
With and signs , , so .
Exercise 12.2 ★
Why would a constant kernel make prices trend when signs have long memory?
Solution
Solution of Exercise 12.2.
With a constant kernel each trade’s impact is permanent, and persistent signs keep adding impacts of the same sign: the price would drift in the direction of the flow, predictably, for as long as the signs’ memory lasts.
Exercise 12.3 ★
In the Obizhaeva–Wang model, sell 10 000 shares over two hours with resilience per hour. Give the two blocks and the continuous rate.
Solution
Solution of Exercise 12.3.
: a block of 1 000 shares at the start, 4 000 shares an hour for two hours, a block of 1 000 at the end.
Exercise 12.4 ★★
Two trades at times 0 and 1 with . What does the round trip cost, and how would you sell one unit with these two trades?
Solution
Solution of Exercise 12.4.
; the round trip costs , positive. To sell one unit, , which is symmetric: half at each time.
Exercise 12.5 ★★
Explain why the propagator on shuffled signs makes prices sub-diffusive.
Solution
Solution of Exercise 12.5.
Without memory, each trade’s impact decays and nothing replaces it on average: price changes over long lags are smaller than the sum of short ones, so the variance per lag falls (0.32 to 0.18).
Exercise 12.6 ★★
Metaorder sizes have a cumulative distribution falling like and are executed in equal pieces. What exponent should the sign autocorrelation have, and why did the simulation find 0.62?
Solution
Solution of Exercise 12.6.
. The measured 0.62 is steeper because the fundamentalists’ orders, whose signs follow the gap to the value rather than long runs, dilute the noise takers’ memory, and the fit uses lags 2 to 50 of a sample of 10 764 orders.
Exercise 12.7 ★★★
Coding. For , find the most profitable round trip’s trades on the 21-point grid and describe its shape. Why does it earn money?
Solution
Solution of Exercise 12.7.
The eigenvector of the most negative eigenvalue ( on the round-trip subspace, a profit of 0.091) alternates buys and sells in seven legs: at the start, at the fourth trade, at the seventh, at the eleventh, and so on symmetrically. The kernel stays near its full value for about and then falls abruptly, so its spectrum is negative at the frequency of this oscillation: each leg trades against the price its predecessor left before that impact has decayed, and after it has, in a pattern that earns more than it pays.
Exercise 12.8 ★★★
Find the flaw. “Our fitted kernel is positive definite, so our execution model is free of manipulation.”
Solution
Solution of Exercise 12.8.
Positive definiteness rules out profitable round trips but not transaction-triggered manipulation: a kernel that is positive definite but not convex ( in the chapter) makes the cheapest sale include buys. The safe condition is a convex non-increasing decay.
12.9 Problem: The Kernel That Paid You to Trade
Problem 12.1
Weekend problem — the kernel that paid you to trade
A quant proposes an impact model with a kernel that stays flat for a while and then drops. Check what it implies.
Part I — Memory.
- What did Lillo and Farmer find on the London Stock Exchange?
- State the splitting explanation and its exponent.
- Give the simulated sign autocorrelation at lags 1, 10 and 50, and its fitted exponent.
- Give the response after 1, 10 and 100 orders.
Part II — The propagator.
- Write the model and the Toeplitz system for its kernel.
- Give the fitted kernel at lags 1, 10 and 100.
- Compare the variance ratios of the market, the model and the model on shuffled signs.
- Why is the market critical in Bouchaud and co-authors’ sense?
Part III — Continuous time.
- Write the cost of an execution in the transient model.
- Derive the optimal liquidation .
- State the Obizhaeva–Wang solution and explain its two blocks.
- Solve exercise 3.
Part IV — Manipulation and the verdict.
- Define dynamic arbitrage and transaction-triggered manipulation.
- State the condition of Alfonsi, Schied and Slynko.
- Give the best round trip for , 1, 1.5, 1.8, 2.2, 2.5 and 3.
- Which kernels make the optimal sale include buys?
- What happens at exactly?
- State the named result: the round-trip profit under a kernel that decays too fast after a flat start, and the decay parameter at which it vanishes.
- Which regime is the proposed kernel in?
- In one sentence: what must a kernel look like to be safe?
Solution
Solution of Problem 12.1.
1. Order signs with a power-law autocorrelation, exponent about 0.6 (Hurst 0.7). 2. Splitting power-law metaorders into equal pieces: . 3. 0.27, 0.106 and 0.023; an exponent of 0.62. 4. 0.52, 1.05 and 2.3 ticks. 5. ; . 6. 0.541, 0.452 and 0.419 ticks. 7. Market 1.98 falling to 1.35; model 0.29 rising to 1.34; shuffled 0.32 falling to 0.18. 8. The kernel’s decay exactly compensates the signs’ memory, leaving prices diffusive. 9. , . 10. Minimise : . 11. Blocks of at both ends and a rate between: the first block uses the existing book, the last uses the refilled book with no later trade to pay for it. 12. 1 000 shares, 4 000 an hour, 1 000 shares. 13. A round trip with negative expected cost; a cheapest execution with opposite trades. 14. Impact must decay as a convex non-increasing function of time. 15. , , , , , , . 16. and 1.8 (two buys each), and every . 17. The Gaussian kernel’s matrix is singular on a fine grid: the boundary of positive definiteness. 18. Named result: with on 21 trades, the best round trip earns 0.028, 0.091 and 0.213 for , 2.5 and 3 and loses for every ; manipulation of the transaction-triggered kind starts at , where convexity is lost. 19. Flat then dropping is : at least transaction-triggered manipulation, and profitable round trips if the drop is sharp (). 20. Convex and non-increasing.
12.10 Interview questions
Interview question 12.1 ★ researcher
Order signs are predictable but prices are not. How is that possible?
Solution
Solution of Interview question 12.1.
Each trade’s impact decays (transient impact), and the decay offsets the predictable continuation of order flow; liquidity providers who see the flow’s persistence adjust quotes so that prices stay close to a martingale.
What the interviewer is looking for: Transient impact; compensation; liquidity providers.
Interview question 12.2 ★★ researcher
How would you estimate the impact kernel from trades and quotes?
Solution
Solution of Interview question 12.2.
Signs and pre-trade mids; the sign autocorrelation and the cross-correlation of signs with later mid changes; solve the Toeplitz system for the kernel’s increments (Levinson); check that the fitted model makes prices diffusive.
What the interviewer is looking for: The linear system; a diagnostic.
Interview question 12.3 ★★ researcher
What condition must an impact model satisfy to rule out profitable round trips?
Solution
Solution of Interview question 12.3.
The quadratic form of the kernel must be non-negative on round trips (positive definiteness); to exclude transaction-triggered manipulation too, the decay must be convex and non-increasing.
What the interviewer is looking for: Positive definiteness; convexity.
Interview question 12.4 ★★ trader
Why does the optimal execution in the Obizhaeva–Wang model start and end with blocks?
Solution
Solution of Interview question 12.4.
The first block takes the liquidity already in the book; the continuous part trades at the book’s refill rate; the last block takes liquidity that will never have to be paid for by later trades, since the execution ends.
What the interviewer is looking for: Resilience; end effects.
Interview question 12.5 ★★ developer
Compute the cost of a schedule of trades under a transient kernel. What is the complexity, and how would you make it faster?
Solution
Solution of Interview question 12.5.
is ; for an exponential kernel a running sum makes it , for a sum of exponentials , and for general kernels an FFT convolution on a regular grid is .
What the interviewer is looking for: Complexity; recursive or FFT tricks.
Interview question 12.6 ★★★ researcher
Your optimiser, fed a fitted kernel, tells a liquidation to buy for a while. Is it a bug?
Solution
Solution of Interview question 12.6.
Probably not a bug: the fitted kernel is not convex (or not positive definite), and the optimiser is exploiting it. Fix the model, by fitting within convex kernels or projecting the fit, rather than the optimiser.
What the interviewer is looking for: Transaction-triggered manipulation; model constraints.