23Market-Data Distribution and Entitlements

An exchange’s auditor asks a firm for the list of every person and every program that used the exchange’s depth data in the past three years. The firm can name forty screens: it pays for forty. It cannot name, at first, the eleven applications that read the same feed from the firm’s distribution platform — strategies, a risk service, a tick recorder, a surveillance system, a research dashboard — and each of them is a use the exchange charges for. The firm in this chapter is a model, but the documents it is audited against are real: one exchange’s price list, data policies and subscriber agreement. The chapter builds the platform that distributes market data inside a firm and the part of it that is usually missing: the record of who used what, for which purpose, counted the way the venue counts.

23.1 Distributing data inside the firm

A direct feed (Book 1, chapter 9) arrives once at the firm’s feed handlers. From there it must reach dozens of consumers with very different needs: a strategy that wants every message within microseconds, a risk service that wants current prices, a trader’s screen that cannot show more than a few updates a second, a research job that records everything. Connecting each of them to the feed would multiply the network load, the licences and the points of failure; the firm puts one platform in between.

The chapter’s bus (firm.entitle) is simulated in virtual time. Ten minutes of depth updates for twenty instruments — 240 267 messages, from Book 7’s tape generator, the order flow that Book 10’s simulator replays — go to four subscribers: a strategy that takes 20 microseconds per message, a risk service that takes 2 milliseconds, a research dashboard that takes 10, and a trader’s screen following five instruments at 50 milliseconds a refresh. Each subscriber has its own queue, so that a slow one cannot delay the others: the publish–subscribe design of chapter 24, and the slow-consumer problem of Book 13, chapter 12, one level up.

    def _enqueue(self, s: _Sub, ts: float, topic: str, value) -> None:
        if s.policy == "conflate":
            s.queue.pop(topic, None)             # keep only the latest per topic
            s.queue[topic] = (ts, value)
        else:
            s.queue.append((ts, topic, value))
        s.stats.max_depth = max(s.stats.max_depth, len(s.queue))

    def _pop(self, s: _Sub):
        if s.policy == "conflate":
            _topic, (ts, value) = s.queue.popitem(last=False)
            return ts, value
        ts, _topic, value = s.queue.popleft()
        return ts, value

The research dashboard is the lesson of Table 23.1. It can process 100 messages a second and the feed brings about 400: with every message queued, its queue grows through the whole session to 180 267 messages, it shows prices fifteen minutes old on average, and it is still working through the session half an hour after it ended (Figure 23.2). Conflated, it processes a quarter of the messages and never shows a price older than 134 ms. The risk service can keep up on average but not in bursts; conflation takes its worst age from 2.4 seconds to 32 ms. A screen or a risk service wants the current state, not the history, and should be conflated; a strategy or a recorder needs every message and must be fast enough not to need it.

23.2 Licences, uses and units of count

What the firm pays for is not the feed: it is each use of it. Venues distinguish display use, by a person looking at a screen, from non-display use by a machine, which Book 1, chapter 29, introduced with the non-display fee; and they count each in a stated unit.

The definition is the venue’s, and it is where firms lose money. The chapter’s venue — Nasdaq, whose documents are in the ledger — counts a display subscriber as "a device, computer terminal, automated service, or unique user identification and password combination", and non-display use by the servers that receive and benefit from the data, including those that compute on it or create derived data from it. Servers that only transport, normalise or store the data are not necessarily fee-liable, but the firm must be able to show which they are. A firm that counts people instead of identifiers, or applications instead of servers, undercounts by construction.

23.3 Entitlements

The question is when to check. Checking at subscription is cheap and enough while nothing changes; but a trader who leaves the desk, or an application whose licence lapses, keeps receiving until it next subscribes. In the chapter’s run the screen user’s entitlement is revoked five minutes into the session. With the check made only at subscription, the screen receives another 6 005 updates it was not entitled to; checked at every delivery, none.

    def _serve(self, s: _Sub, t: float, ready: list, i: int) -> None:
        if not s.queue:
            return
        ts, _value = self._pop(s)
        if self.check_on_delivery and not self.ents.check(s.subject, s.source, s.use, t):
            s.stats.denied += 1
        else:
            s.stats.delivered += 1
            s.ages.append(t + s.service - ts)
            s.stats.max_age = max(s.stats.max_age, t + s.service - ts)
            if self.trace_every and s.stats.delivered % self.trace_every == 0:
                s.stats.trace.append((t + s.service, t + s.service - ts))
        s.busy_until = t + s.service
        if s.queue:
            heapq.heappush(ready, (s.busy_until, i))

A real platform does not call the entitlement system for each of millions of messages: it caches the answer per subscription and invalidates the cache when an entitlement changes, which has the same effect at almost no cost. What matters is that revocation is pushed to the platform rather than discovered at the next login.

23.4 Usage reporting and exchange audits

The chapter’s firm reported 40 display subscribers every month for three years and no non-display use. Its usage log — which the platform writes on every subscription — tells a different story (Figure 23.3). Traders grew from 38 to 47, and six of them have a second login, so display identifiers went from 43 to 53; the eleven applications came into use one after another and by the end ran on 22 servers.

def usage_report(log: UsageLog, month: int, source: str) -> dict:
    """The monthly count a venue asks for: distinct units of count by use."""
    units = {u: set() for u in USES}
    for r in log.rows:
        if r.month == month and r.source == source:
            units[r.use].add(r.unit_id)
    return {u: len(v) for u, v in units.items()}


@dataclass
class AuditRow:
    month: int
    used: dict
    licensed: dict
    shortfall: dict
    owed: float


def audit(log: UsageLog, source: str, months, licensed: dict, fees: dict) -> list[AuditRow]:
    """Reconcile each month's usage with the licences held; the amount owed is the fee for
    what was used minus the fee for what was licensed, by use."""
    out = []
    for m in months:
        used = usage_report(log, m, source)
        short = {u: max(0, used[u] - licensed.get(u, 0)) for u in USES}
        owed = sum(max(0.0, fees[u](used[u]) - fees[u](licensed.get(u, 0))) for u in USES)
        out.append(AuditRow(m, used, dict(licensed), short, owed))
    return out

Priced with the venue’s 2026 fees for its depth product — $84 a month per professional display subscriber, and for non-display use $412 a month per subscriber up to 39, then a flat $16 490 a firm from 40 to 99 (Figure 23.4) — the shortfall is $1 900 in the first month and $10 156 in the last, $258 596 over the three years: $22 932 for display and $235 664 for non-display. The unreported units reach 87.5% of those reported, far above the 10% at which the venue’s subscriber agreement adds the audit’s costs; interest comes on top, at a rate the model does not price. The display shortfall is the smaller part. The firm’s real mistake was believing that data it had paid to receive could be used by any program.

The remedy is technical as much as contractual. The platform’s usage log is the evidence an audit asks for, and it must record the unit of count the venue uses — identifiers, not people; servers, not applications — for as long as the look-back. The chapter’s venue limits liability for a good-faith error to three years before the underreporting was found, so three years of logs is the minimum. A monthly reconciliation of the log against the licences, run by the firm before any auditor does, turns a back-bill into a licence purchase.

23.5 Derived data

A conflated quote is not derived data: it is the venue’s data, delivered less often, and needs the same licence. A mid-price published every second may be a substitute for the quote; a volatility estimate or a basket’s fair value usually is not. The distinction matters in two directions: derived data may be redistributed under terms that the raw data does not allow, and the servers that create it are, for the chapter’s venue, among the non-display units that count. The entitlement model should therefore carry the use — display, non-display, derived — and not only the source, and the tick store of chapter 4 should record which licence each dataset was captured under, since the licence follows the data into research.

23.6 Tutorial: distribute, entitle, report, audit

23.7 Build: entitlements

23.8 Exercises

23.9 Problem: Eleven Applications

23.10 Interview questions