9Managing Researchers, Traders and Engineers

Three researchers’ signals went into one combined forecast that made $30 million in a year. Asked what each had contributed, the desk measured what the forecast would have lost without each signal in turn: $2.5 million, $2.7 million and $4.6 million, less than a third of the profit between them. Two of the signals were two routes to the same effect, so each looked nearly worthless once the other was there. The rest of the $30 million had to be split some other way, and the way it was split decided which of the three stayed.

9.1 Choosing projects: a research portfolio

A quantitative desk spends much of its budget on work whose value is uncertain: new signals, new markets, new models, better execution. The head of desk chooses among projects as an investor chooses among assets, with the same two questions: what is each expected to return, and how do the returns combine.

A project’s expected net value is $pV(1-\text{overlap})-c$. The chapter’s desk has twelve candidates (illustrative, $ million): a new futures signal (cost 1.5, probability 0.30, value 12), options-flow features (2.0, 0.20, 15), an execution model (1.0, 0.60, 4), an alternative-data trial (2.5, 0.10, 20), a second momentum variant that overlaps the book by 70% (1.0, 0.40, 6), and seven others. With a budget of $10 million, choosing greedily by expected net value per dollar of cost funds six projects with an expected net value of $8.15 million. Over 20 000 simulated years the chosen portfolio returns $8.08 million on average and loses money in 28.5% of years; a portfolio chosen at random within the same budget returns $3.65 million and loses in 44.8% (Figure 9.1).

9.2 Review without theatre

A research review (One Quant Book 7, chapter 1) is where a project meets its evidence. Reviews become theatre when their outcome does not depend on what is presented: when the project’s sponsor is senior, when the backtest is shown without the number of variants tried, when a kill criterion was never written. The procedures that prevent it are already in the series: pre-registration of the hypothesis and of the test, a trial count kept in the research log, stage gates with kill criteria agreed before the evidence arrives.

The trial count matters more than any other number in the review. A backtest with a Sharpe ratio of 1.2 over five years of daily data has a probabilistic Sharpe ratio of 0.996: taken alone, it is almost certainly real. If it is the best of 40 variants tried, the expected best of 40 worthless variants (with a spread of Sharpe ratios of 0.5 a year) is 1.09, and the deflated Sharpe ratio falls to 0.59 (Figure 9.2). The same result, after 200 variants, is more likely luck than skill.

9.3 Who made the money: attributing credit

A combined forecast is a game in the sense of the Shapley value (One Quant Book 12, chapter 6): each set $S$ of contributors has a worth $v(S)$, the P&L of the best combination of their signals. Leave-one-out credit, $v(N)-v(N\setminus\{i\})$, measures what each adds last; when two contributors bring the same information, each adds little last and both look worthless. Order-dependent credit gives each contributor what they added when they arrived, so the first of two overlapping signals gets everything and the second nothing. The Shapley value averages the order over all orders.

def leave_one_out(v, n: int) -> np.ndarray:
    full = frozenset(range(n))
    return np.array([v(full) - v(full - {i}) for i in range(n)])


def ordered(v, order) -> np.ndarray:
    out = np.zeros(len(order))
    seen = set()
    for i in order:
        before = v(seen)
        seen.add(i)
        out[i] = v(seen) - before
    return out


def shapley(v, n: int, samples: int | None = None, rng=None) -> np.ndarray:
    if samples is None:
        tot = np.zeros(n)
        for order in itertools.permutations(range(n)):
            tot += ordered(v, order)
        return tot / math.factorial(n)
    tot = np.zeros(n)
    for _ in range(samples):
        tot += ordered(v, list(rng.permutation(n)))
    return tot / samples

In the chapter’s desk (illustrative: signals A and B correlated at 0.8, C independent, ten years of daily data, the combined P&L scaled to $30 million), the rules disagree completely (Figure 9.3):

A bonus pool of $6 million split in proportion to each rule’s credits gives A, B and C $1.51, $1.64 and $2.85 million by leave-one-out, $2.53, $2.55 and $0.92 million by Shapley, and $4.55, $0.52 and $0.93 million by order of arrival. Leave-one-out pays C most, because C is the only one whose signal cannot be replaced; order of arrival pays whoever came first. Shapley pays A and B as a pair for the effect they both found, and C for what only C found. The credits are themselves estimates: rerun on another ten years of data (exercise 7), the split moves by several million.

9.4 Keeping research honest

Credit rules shape research as much as they measure it. Paid by leave-one-out, researchers avoid overlapping work, including the replication that would expose a spurious signal; paid by order of arrival, they race to be first and hoard; paid by stand-alone P&L, they rediscover what the book already holds. Paid by Shapley, a researcher who finds a second route to an effect the desk already exploits is paid for part of it, which rewards robustness and replication. Three further rules keep research honest whatever the credit rule: every trial goes in the research log; results are presented against a pre-registered test; and the people who review a signal are not those who are paid for it.

9.5 Traders and engineers: different work, different feedback

Researchers, traders and engineers get feedback at different speeds, and managing them the same way fails all three (Figure 9.4). A trader sees the market’s verdict in minutes and must be judged on decisions, not only on the day’s P&L, whose noise dominates it: attribution of P&L to risk taken and to execution (One Quant Book 1, chapter 7) is the tool. A researcher’s work pays off, if ever, after months of testing and years of live trading: the measure is the quality of the process, trials and evidence, until the P&L can speak. An engineer’s work shows in systems that run or do not: delivery, reliability and the time it takes to restore service are measured daily (One Quant Book 15, chapter 28), and the value of the work is the trading it enables, which is why engineers belong in the credit attribution of the strategies they make possible.

9.6 Tutorial: a year of research, and the credit for it

What to change next. Make A and B identical and check that Shapley pays them equally and leave-one-out pays them nothing; rerun on another seed and measure how uncertain each credit is.

9.7 Build: research portfolio and credit

9.8 Exercises

9.9 Problem: Who Made the Money

9.10 Interview questions