30Surveillance and Compliance Technology

On 27 September 2022 the SEC announced that sixteen firms had agreed to pay more than $1.1 billion in penalties, and the CFTC the same day $710 million against eleven: from 2018 to 2021 their employees had discussed business in text messages and messaging applications on personal phones, and the firms had not kept the substantial majority of those messages. The rules were old; the phones were new; the archive had a hole the size of a pocket. Book 9, chapter 29, built the detectors that find spoofing, marking the close and wash trades in account-days. This chapter builds what surrounds them in a firm: the pipeline from orders to alerts, the queue in which people review alerts, the archive of what people said, the reports a firm sends its regulator about every trade, and the measurements that say whether any of it works.

30.1 The surveillance pipeline

firm.survpipe extracts, from a stream of order events, the account-day features Book 9’s detectors use: orders placed and filled by size, large orders cancelled, and large cancellations that follow a fill on the account’s other side within a second. The chapter’s quarter then runs on Book 9’s own populations of account-days — market makers, quote refreshers, deep liquidity providers, directional traders, index funds, multi-algorithm firms — 2 200 legitimate account-days a day, with planted spoofing, marking-the-close and wash episodes arriving at random: 45 in the quarter’s 63 days. Each detector’s threshold is set on a separate calibration quarter at a false-positive rate of 0.2%, 0.5% or 1% of legitimate account-days.

30.2 From alert to case

The team is two analysts who can each review eight alerts a day. Each day’s alerts join the queue, and the team takes sixteen: the oldest first, or the highest score first.

def work_queue(alerts_by_day: list, per_day: int, policy: str = "oldest") -> QueueRun:
    """Each day the day's alerts join the queue and the team reviews `per_day` of them:
    the oldest first, or the highest score first."""
    run, queue = QueueRun(), []
    for day, new in enumerate(alerts_by_day):
        queue.extend(new)
        if policy == "score":
            queue.sort(key=lambda a: (-a.score, a.day, a.aid))
        else:
            queue.sort(key=lambda a: (a.day, a.aid))
        take, queue = queue[:per_day], queue[per_day:]
        run.reviewed += [(a, day) for a in take]
        run.backlog.append(len(queue))
    return run

The threshold is a staffing decision (Figure 30.2). At 0.2% the detectors raise 32 of the 45 planted episodes and the team reviews every one on the day it appears; at 0.5%, 36. At 1% they raise 38, but the alerts outnumber the team’s capacity, and the order of the queue decides what that is worth (Figure 30.3). Oldest first, the team reviews 28 of the 38 by the quarter’s end and only 11 within five days, with a median wait of seven days; highest score first, 37 of 38 within five days, while the queue’s backlog is made of low scores. A queue that is always behind is not a surveillance system: an alert reviewed a month late finds a trader who has done it thirty more times.

30.3 Communications archiving and surveillance

The chapter’s archive holds a quarter of messages from 200 people, 252 025 of them. Forty conversations move off channel; 25 leave a pointer in the archive ("let’s take this to whatsapp", "call my cell") and 15 leave nothing. Two lexicons search it (Table 30.1).

The table is the hook in numbers. A lexicon finds only phrasings it knows, so a quarter’s catch is ten of forty conversations, and precision comes from writing phrases rather than words. And the fifteen conversations with no pointer are outside any search: an archive cannot surveil what it does not hold. The control that matters is upstream — the firm’s devices and approved channels capturing everything, and attestations and phone reviews for the rest — which is what the 2022 orders required the firms to fix.

30.4 Transaction reporting

A week of 2 500 equity executions for five clients is reported twice. The legacy generator copies each trade when it is booked and has four faults: a client LEI left empty on 1% of reports, a mistyped ISIN on 0.5%, amendments never reported (41 trades, 1.6%, are amended later in the week) and a batch occasionally sent twice. The fixed generator reads chapter 21’s event-sourced trade store, so an amendment becomes a cancellation of the first report and a new one, and takes identifiers from validated reference data.

def event_reports(trades: dict) -> list[dict]:
    """The fixed generator reads the event-sourced trade store: an amendment cancels the
    first report and sends a new one (CANC, then NEWT); identifiers from reference data."""
    out = []
    for trn, t in trades.items():
        if t["amended"]:
            out.append(_report(trn, t, qty=t["quantity"] + 100))       # as first booked
            out.append(_report(trn, t, status="CANC", qty=t["quantity"] + 100))
        out.append(_report(trn, t))
    return out

def reconcile(reports, trades) -> dict:
    """Net each reference's reports (a CANC removes the report it cancels) and compare with
    the firm's trades by reference: missing, extra, and mismatched quantity or price."""
    live: dict = {}
    for r in reports:
        if r["status"] == "CANC":
            live.pop(r["trn"], None)
        elif r["trn"] in live:
            live[r["trn"] + "#dup"] = r
        else:
            live[r["trn"]] = r
    out = {"missing": [], "extra": [], "mismatched": []}
    for trn, t in trades.items():
        r = live.get(trn)
        if r is None:
            out["missing"].append(trn)
        elif r["quantity"] != t["quantity"] or abs(r["price"] - t["price"]) > 1e-9:
            out["mismatched"].append(trn)
    out["extra"] = sorted(k for k in live if k not in trades)
    return out

Of the legacy generator’s 2 506 reports, 87 are wrong in one of three ways (Figure 30.4): 81 of the week’s 2 500 trades (3.2%) are either unreported or misreported, and 6 are reported twice. Validation catches the malformed ones before sending; only reconciliation with the trades catches the amendments, because each unreported amendment leaves a perfectly valid report of the trade as first booked. The event-sourced generator sends 2 582 reports — more, because each amendment costs two — and reconciles exactly.

30.5 Measuring a surveillance system

Every component of this chapter has a measure, and a surveillance function that cannot state them cannot say whether it works. For detectors, the true-positive rate at a false-positive rate, on labelled or planted episodes (Book 9), and the precision–recall curve (Book 12, chapter 20) as the base rate falls. For the queue, alerts a day against capacity, backlog, and the age at review of what matters. For the archive, the share of channels captured — which no search statistic can show — and a lexicon’s hits against known cases. For reporting, the shares that fail validation and reconciliation, every day. Planted episodes, run through the whole pipeline on a schedule like the replayed incidents of chapter 28, are the end-to-end test.

30.6 Tutorial: a hole the size of a pocket

30.7 Build: the surveillance pipeline

30.8 Exercises

30.9 Problem: A Hole the Size of a Pocket

30.10 Interview questions