2Capturing and Storing Tick Data

The consolidated US options feed is planned for hundreds of billions of messages on a busy day (the dated box of Section 2.5 gives the figure). A firm that keeps every one of those messages for five years is running a storage business on the side, and the first decision it makes is not which database to buy but what exactly to keep: the packets as they arrived, with their duplicates, their gaps and their arrival times, or the events they meant, cleaned, arbitrated and laid out for reading. This chapter captures ten busy minutes of Book 10’s simulated exchange on both of its feed lines, turns them into the firm’s event records, measures what each layout costs in bytes, and scales the result, with cited message counts and prices, to the question of the title of its weekend problem: a petabyte a year?

2.1 What tick data is, and where it is captured

Tick data is the finest record of a market a firm can keep, and everything coarser is derived from it: bars (One Quant Book 7, chapter 2), order-book features (chapter 8 of that book), the replays of its chapter 18. What is less obvious is that “the” tick data of a day depends on where it was captured. Figure 2.1 shows the three usual capture points on the path of Book 13’s market-data handling.

Each point answers different questions. Only the wire capture can say which line lost which packet and when each packet arrived, which is what a latency or network investigation needs. Only the capture after arbitration is the market as the firm’s book saw it, one message at a time. Only the capture after conflation is what a slow consumer (One Quant Book 13, chapter 12) actually acted on. Research wants the second; operations and compliance sometimes need the first or the third.

The chapter’s capture is at the first point, in the recorded-file format of Book 10’s simulator: for every packet received on a line, its receive time (8 bytes), its length (4 bytes) and the MoldUDP64 packet itself. Ten minutes of the simulator’s busy open, two instruments, with its seeded impairments — independent losses on each line, bursts of loss, duplicated packets, jitter and a five-second outage on line B — give 15.8 MB on line A and 15.6 MB on line B.

2.2 Raw capture against normalised storage

To turn the capture into the market the book saw, the two lines are merged by sequence number: the first copy of each message wins, later copies are duplicates, and a gap is held open until the other line has had time to fill it. What neither line fills is a gap in the record; a live feed handler would ask the venue’s retransmission service or rebuild from a snapshot (One Quant Book 13, chapter 18), and a capture that is written for research records it and moves on. Listing 2.1 is the arbitration of firm.tickcap.

def arbitrate(lines, timeout_ns: int = 500_000):
    """Merge redundant lines by sequence number. A message is delivered once, at the receive
    time of its first copy; a gap is held open until a later sequence number has waited
    `timeout_ns`, then recorded and skipped."""
    ev = sorted((t, k, p) for k, line in enumerate(lines) for t, p in line)
    nxt, pending, out, gaps = 1, {}, [], []
    cnt = {"packets": 0, "messages": 0, "duplicates": 0, "gap_messages": 0}
    held_since = None
    for t, _k, p in ev:
        cnt["packets"] += 1
        seq, msgs = blocks(p)
        for j, m in enumerate(msgs):
            s = seq + j
            if s < nxt or s in pending:
                cnt["duplicates"] += 1
            else:
                pending[s] = (t, m)
        while nxt in pending:
            tt, m = pending.pop(nxt)
            out.append((tt, nxt, m))
            nxt += 1
        if pending and held_since is None:
            held_since = t
        if pending and t - held_since >= timeout_ns:
            first = min(pending)
            gaps.append((nxt, first - 1))
            cnt["gap_messages"] += first - nxt
            nxt = first
            while nxt in pending:
                tt, m = pending.pop(nxt)
                out.append((tt, nxt, m))
                nxt += 1
            held_since = t if pending else None
        elif not pending:
            held_since = None

On the ten-minute session the two lines carry 503 024 packets. Arbitration delivers 258 535 messages, discards 255 899 duplicates (nearly every packet arrives twice, once per line) and records four gaps, eight messages in all, lost on both lines at once. Each delivered message is then normalised into the firm’s event record: the receive time followed by the 48-byte event of Book 13’s feed handler (kind, side, instrument, sequence number, exchange time, order references, price, quantity), 56 bytes in all. The store and the trading path therefore share one layout, and a test checks that the capture’s events are, message for message, the feed handler’s.

The proposition is the whole argument for keeping raw captures. A normaliser has bugs, and a field that nobody needed when it was written (a flag, a new message type after a venue’s upgrade) is needed a year later; with the raw capture both are fixed by running the corrected normaliser again, without it they are lost. The price is the bytes, which the rest of this chapter counts.

2.3 Record layouts and encodings

Figure 2.2 counts bytes per message. The raw capture costs 61.0 bytes a message on one line: the simulator sends one message per packet, so each message carries a 12-byte file record header, a 20-byte MoldUDP64 header and a 2-byte length besides its own 23 to 44 bytes. Real feeds pack many messages into a packet and pay less framing; the share is a property of the feed, and it is one reason to measure on the feed one actually captures. The normalised record is 56 bytes, fixed: it is barely smaller, but every record is at a known offset, which chapter 4 exploits.

A generic compressor on the raw capture gets a ratio of 3.32; on the normalised records, 3.67. The columnar layouts do better, and the best of the chapter is the simplest columnar trick: write each field as its own column, and write the columns whose values increase slowly (receive time, sequence number, exchange time) or move by small steps (price) as differences from the previous value, before a generic compressor. The deltas of the sequence number are all 1, those of the receive time are small integers, and the ratio rises to 4.70, 11.9 bytes a message. Chapter 3 explains why columns compress better than rows; this chapter only counts.

def delta_encode(rec: np.ndarray) -> bytes:
    """Columns one after another; time, sequence, exchange time and price as differences
    from the previous row."""
    cols = []
    for name in RECORD.names:
        delta = name in ("recv", "seq", "ts", "price")
        c = rec[name].astype(np.int64) if delta else rec[name]
        if name in ("recv", "seq", "ts", "price"):
            c = np.diff(c, prepend=c[:1] * 0)
        cols.append(np.ascontiguousarray(c).tobytes())
    return b"".join(cols)

2.4 Partitioning and compression

The chapter’s session is written under date=2026-09-28/locate=N/. Its two instruments are very unequal: instrument 1 takes 811 160 bytes of the 14 477 960, 5.6% of the day, so a query about it reads 5.6% of what the unpartitioned file would cost. The overhead term is the limit. Partition a US options day by option series and a query for one underlying opens thousands of small files, each costing a directory lookup and an open; partition by underlying and each file is large and the query opens few. The rule of thumb that follows — choose the partition key from the filters queries actually use, and keep partitions large enough that their overhead is negligible — is the subject of chapter 4’s tick store.

Compression has its own trade-off, between ratio and speed (Figure 2.3). On this laptop zlib at level 9 gains about 2% of ratio over level 6 at a twelfth of its speed, and zstd at level 9 inside a columnar file is both faster and better than zlib at level 6 on the same records. Capture writes once and in real time, so it wants speed; the historical store writes once at night and is read thousands of times, so it wants ratio.

2.5 Retention, tiers and the economics of petabytes

The storage model of firm.tickcap multiplies it out: bytes a day are messages times bytes per message times the number of copies; a year is 252 trading days; the steady-state cost of keeping five years is the last month (21 trading days) in hot storage, the rest of the first year in infrequent access and the four older years in the archive. Table 2.1 applies it to the options feed’s planned capacity with the session’s bytes per message.

Two cautions keep the table honest. The message count is a capacity the feed’s operator plans for, a ceiling rather than an average day, so the table is an upper bound. And the bytes per message are the simulator’s: real options messages have other sizes and other redundancy, and a real ratio can be higher or lower. The method transfers; the digits must be re-measured on the feed itself. What does transfer is the shape: the cheapest petabyte is the one not written twice (both lines), not written in its heaviest layout, and not kept in the hot tier after the month in which anyone reads it.

2.6 Tutorial: ten minutes on two lines

2.7 Build: the tick capture

2.8 Exercises

2.9 Problem: A Petabyte a Year?

2.10 Interview questions