26Build: Tick-to-Trade, Measured

A packet leaves the simulated exchange on two lines at once. A few microseconds later the feed handler has it, a microsecond after that the strategy has decided, and an order leaves for the venue through the risk gate and the order gateway. Every one of those steps has been built and measured in isolation in this book; this chapter puts them on one path, on four threads of one machine, and times the whole thing with a stamp at every boundary. The result is the latency budget of chapter 1 with the guesses replaced by measurements: which stage costs what, at the median and in the tail, and where a real server would have to be different. The answer on this laptop is sobering in one place and reassuring in another. The code the book has written takes a few hundred nanoseconds; the kernel, the network stack and a shared machine take the rest, and the rest is most of it.

26.1 Assembling the path

The path is the book’s components in order (Figure 26.1). An exchange thread replays the recorded line: the small-tick day of chapter 19, 6 000 events in 1 200 MoldUDP64 packets, sent over UDP on the loopback interface on line A and, five microseconds later, on line B, at twenty times its recorded spacing (a packet every $100\,\text{µ}\mathrm{s}$). A feed thread receives both lines and runs the feed handler of chapter 18, which arbitrates them and publishes each event on a ring (chapter 12). An engine thread takes each event off the ring and runs firm.ticktotrade’s Path: the strategy engine of chapter 20, with its book builder (chapter 19), then, for each order it decides, the risk gate of chapter 22 and the order gateway of chapter 21, which writes the venue’s binary message (chapter 16) and sends it on a TCP session. A venue thread receives the session’s frames. Each thread is pinned to its own CPU (chapter 4), and nothing on the path allocates after warm-up (chapter 6): the harness counts, and finds zero in every run.

The path is checked before it is timed. The C++ test replays the same events offline through the same Path and obtains 1 050 orders (new orders, replaces and cancels) whose hash is committed; the Rust twin, assembled from the Rust feed handler, engine, risk gate and gateway, produces the same hash; and the live harness, with its threads, sockets and arbitration between two lines, produces it again, in every run. What is timed is therefore the path that computes the right orders.

    template <class Send>
    void on_event(const strat::Event& e, std::uint64_t (*now)(), Stamp* st, Send&& send) {
        const std::size_t first = engine_.actions.size();
        engine_.stamp = now;
        one_[0] = e;
        engine_.run(one_);
        if (now && st) st->t[Book] = engine_.book_stamp, st->t[Decide] = now();
        for (std::size_t i = first; i < engine_.actions.size(); ++i)
            act(engine_.actions[i], now, st, send);
    }

26.2 Instrumentation points and stage timestamps

Every point reads the time-stamp counter (chapter 5), which all cores of this machine share, so that a reading on the feed thread and one on the engine thread are comparable. The ten points of Figure 26.1 give eight stages: receive, from the exchange’s sendto to the feed thread’s recv, the kernel’s UDP path on the loopback interface; decode, the handler’s arbitration and normalisation until the event is on the ring; ring, the wait on the ring; book, strategy, risk and gateway on the engine thread; and transmit, from the written message to the venue thread’s receipt, the kernel’s TCP path. The stamps travel with the event (in the ring’s message) and with the order (in a preallocated record), and the exchange’s and venue’s stamps are joined afterwards by packet sequence number and client order identifier (Listing 26.2). Two traps shaped the design. Reading the clock is not free: each point costs some $11\,\mathrm{n}\mathrm{s}$ with rdtscp (chapter 5), so the points are few and none is inside a loop. And when an event causes two orders, the second waits for the first to be sent, a few microseconds that the naive attribution charges to its risk check; the stages are therefore attributed on the first order of each event, and the end-to-end figures on every order.

        for (;;) {
            if (ring_in.try_read(&m, sizeof m) < 0) {
                if (feed_done.load(std::memory_order_acquire) && ring_in.try_read(&m, sizeof m) < 0)
                    break;
                continue;
            }
            if (n_ev++ == cfg.warmup_events)
                alloc_mark.store(static_cast<long long>(firm::arena::allocations()));
            Stamp st;
            st.t[Recv] = m.recv, st.t[Pub] = m.pub, st.t[Pop] = tsc();
            st.pkt_seq = m.pkt_seq, st.line = m.line;
            int nth = 0;
            auto send = [&](const std::uint8_t* msg, std::size_t n, char, std::uint64_t) {
                frame[0] = static_cast<std::uint8_t>((n + 1) >> 8);  // SoupBinTCP: length, 'U', message
                frame[1] = static_cast<std::uint8_t>(n + 1);
                frame[2] = 'U';
                std::memcpy(frame + 3, msg, n);
                ::send(cli, frame, n + 3, 0);
                OrderStamp os;
                os.s = st;
                os.s.t[Sent] = tsc();
                os.nth = nth++;  // a second order of the same event has waited for the first one's send
                res.orders.push_back(os);
            };
            path.on_event(m.e, &tsc, &st, send);
        }

26.3 The budget against the measurement

Figure 26.2 gives the stages over ten runs, 5 250 first orders (10 500 in all), on this laptop with no isolated cores, the machine otherwise idle. The book’s own code is fast: the book update takes about $160\,\mathrm{n}\mathrm{s}$ at the median, the strategy about 100, the risk gate about 50 and the gateway about 30, all well under a microsecond at the 99th percentile. The decode stage, about $440\,\mathrm{n}\mathrm{s}$, includes waiting for the earlier messages of the same packet: five events arrive together and leave one after another. The ring’s median of about $0.8\,\text{µ}\mathrm{s}$ is the same queueing, on the engine side, behind the orders of the previous events. The kernel’s stages dwarf the rest: receiving a UDP packet on the loopback interface takes about $1.4\,\text{µ}\mathrm{s}$, sending the order over TCP about $2.9\,\text{µ}\mathrm{s}$, and in the tail the ring waits milliseconds whenever the engine thread is preempted on its CPU (by the kernel, the hypervisor or any other task), which no user-space code can prevent on a machine that has not isolated it (chapter 13).

End to end, from the exchange’s send of a packet to the venue’s receipt of the order it caused, the median is about $9\,\text{µ}\mathrm{s}$; in the process, from the packet’s receipt to the order’s message, about $3.7\,\text{µ}\mathrm{s}$ (Figure 26.3). The distributions have the shape the book has predicted since chapter 1: a tight body and a tail three orders of magnitude further out, set by the machine rather than the code. Table 26.1 checks chapter 1’s three-microsecond budget with firm.latbudget: the strategy and the risk gate meet their targets; the book and the decode miss theirs (the book by a third, the decode because of the packet’s queue), and the kernel’s receive and transmit miss by factors of 1.6 and 3.6, as they were bound to: the budget assumed a kernel-bypass network card.

26.4 What a tuned server would change

Three changes separate this laptop from a trading server, and each targets a part of the table. A kernel-bypass network card removes the system calls from the data path: the ef_vi interface of one vendor’s adapters, for example, gives the application “direct access” to the adapter’s datapath, with data path operations that “do not require system calls”; chapter 1’s budget assumed $0.9\,\text{µ}\mathrm{s}$ to receive and $0.8\,\text{µ}\mathrm{s}$ to transmit on such a path, card and wire included. Isolated cores (chapter 13) keep other work off the path’s CPUs, which is what the tail of the ring stage is made of. And the path itself can be shortened: the packet’s queue behind the decode and the ring disappears if events are handed over as a batch or if the feed thread runs the strategy for simple cases, and the book’s $160\,\mathrm{n}\mathrm{s}$ is the ladder of chapter 19 on a small-tick instrument with a per-event engine call that the batch would amortise.

Replacing the measured receive and transmit by the budget’s kernel-bypass figures and keeping the measured in-process stages gives a projected median of about $3.2\,\text{µ}\mathrm{s}$: $0.9 + 0.8$ for the network, plus about $1.5\,\text{µ}\mathrm{s}$ for decode, ring, book, strategy, risk and gateway, of which the queueing in decode and ring is some two-thirds. That misses the three-microsecond budget by the queue, which is the next thing to remove, and not by any of the components the book has built. The projection is arithmetic on a stated assumption, not a measurement; the measurement on the target server is the only proof, and the harness of this chapter is how it is taken.

26.5 Tutorial: run the path, read the budget

26.6 Build: tick-to-trade

26.7 Exercises

26.8 Problem: The Budget, Proven

26.9 Interview questions