17WebSocket, REST, TLS and JSON at Speed

Crypto venues and many brokers do not speak FIX or binary protocols to most of their clients: market data arrives as JSON text inside WebSocket frames inside TLS records, and orders leave as signed HTTP requests or as JSON over the same kind of connection. A trading server placed in the venue’s cloud region (One Quant Book 3, chapter 26) sits a fraction of a millisecond from its matching engine. On this book’s laptop, the Python reference of this chapter’s build spends almost $50\,\text{µ}\mathrm{s}$ of processor time to receive one depth update and send one order (opening and sealing TLS records, unmasking and masking frames, decoding the JSON, signing); the C++ components spend about one. The first is a tenth of a half-millisecond round trip; the second disappears in it. This chapter takes the web stack apart layer by layer: HTTP and the cost of opening connections, TLS and what its handshake and records cost, WebSocket framing, JSON decoding with a structural index, and request signatures computed without rehashing the key.

17.1 HTTP, REST and connection reuse

A crypto venue’s order entry is typically a REST interface: an order is a POST with the order’s parameters, the client’s API key and a signature (One Quant Book 3, chapter 15), and the answer is JSON. The expensive part is not the request but the connection under it. Opening one costs a TCP handshake, one round trip, then a TLS handshake, another round trip plus the cryptography of key exchange and certificate verification; a WebSocket adds an HTTP upgrade, a third. HTTP/1.1 “defaults to the use of persistent connections” (RFC 9112), and a client that uses them pays these costs once per connection instead of once per order. The build’s pool keeps a fixed number of connections open and passes every request through Book 3’s rate-limit governor first, since a venue’s limits count requests and weights (Book 3, chapter 15), not connections.

17.2 TLS: the handshake and the record layer

TLS 1.3 (RFC 8446) cut the full handshake to one round trip and replaced the older session identifiers and tickets by resumption with a pre-shared key: “the key derived from the initial handshake is used to bootstrap the cryptographic state instead of a full handshake”. A zero-round-trip mode lets a resuming client send data with its first message, “at the cost of certain security properties” (the data can be replayed), which is why an order must never travel in it. After the handshake, the record layer cuts the stream into records of at most $2^{14}$ bytes and protects each with an authenticated cipher.

Table 17.1 gives the costs on the laptop’s loopback, where the network costs almost nothing and what remains is processing. A full TLS 1.3 handshake with an elliptic-curve certificate takes about $2.1\,\mathrm{m}\mathrm{s}$ with Python’s ssl module (both ends on the same machine), a resumed one about $1.6\,\mathrm{m}\mathrm{s}$. A round trip of 300 bytes on an open connection takes about $45\,\text{µ}\mathrm{s}$ over TLS and $33\,\text{µ}\mathrm{s}$ over plain TCP, both in Python; the protection itself, one AES-128-GCM seal or open of a 300-byte record with OpenSSL called from C++, is under $0.2\,\text{µ}\mathrm{s}$. The handshake is ten thousand times dearer than a record: the reason to keep connections open.

17.3 WebSocket framing

Three rules of RFC 6455 shape a client. “A client MUST mask all frames that it sends to the server”, with a new random key per frame, even over TLS: the payload is XORed with the key repeated, a defence of intercepting proxies, not of the data. A message may be split into fragments (a first frame with the opcode, continuation frames, the last with its final bit), and control frames (ping, pong, close) may arrive between fragments, carry at most 125 bytes and are never fragmented. And the venue checks the connection with pings that the client must answer: one large venue sends a ping every 20 seconds and drops a connection that has not answered within a minute. The build’s decoder unmasks in place eight bytes at a time.

// XOR with the 4-byte key, eight bytes at a time (the key repeated twice in a 64-bit word), then the tail.
inline void unmask(std::uint8_t* p, std::size_t n, const std::uint8_t key[4]) {
    std::uint64_t k8;
    std::uint8_t kk[8] = {key[0], key[1], key[2], key[3], key[0], key[1], key[2], key[3]};
    std::memcpy(&k8, kk, 8);
    std::size_t i = 0;
    for (; i + 8 <= n; i += 8) {
        std::uint64_t w;
        std::memcpy(&w, p + i, 8);
        w ^= k8;
        std::memcpy(p + i, &w, 8);
    }
    for (; i < n; ++i) p[i] ^= key[i % 4];

Framing costs little in C++: about $24\,\mathrm{n}\mathrm{s}$ to decode and unmask a 300-byte frame, $16\,\mathrm{n}\mathrm{s}$ to encode and mask one. The build’s Python reference, which XORs the payload one byte at a time, takes about $16\,\text{µ}\mathrm{s}$ for the same frame: several hundred times more, and the largest item of its budget.

17.4 JSON at speed

The parser of Langdale and Lemire (chapter 14) works in two stages: the first writes “the location of all structural characters … as integer indexes in a separate array”, the second walks that index. The build applies the idea to one message shape, the depth update a large venue documents for its diff-depth stream: an event time, a symbol, the first and last update identifiers (U, u) that Book 3’s book builder checks for gaps, and arrays of bids and asks as [price, quantity] strings. The first stage is chapter 14’s positions_any over the seven characters {}[]:, and the double quote; the second walks the positions with the shape known in advance, reads each number where it stands, and parses prices into integers of $10^{-8}$ without floating point. Anything unexpected (an escape, a missing key) makes it return false, and a general parser takes over.

    while (k < m) {
        std::string_view key;
        if (!str(key) || k >= m || s[ix[k++]] != ':') return false;
        if (key == "b" || key == "a") {
            if (!(key == "b" ? levels(d.bids, d.nb) : levels(d.asks, d.na))) return false;
        } else if (s[ix[k]] == '"') {
            std::string_view v;
            if (!str(v)) return false;
            if (key == "s") {
                const std::size_t n = v.size() < sizeof d.symbol - 1 ? v.size() : sizeof d.symbol - 1;
                std::memcpy(d.symbol, v.data(), n);
                d.symbol[n] = 0;
            }
        } else {   // a number: from after ':' to the next ',' or '}'
            const std::size_t a = ix[k - 1] + 1, b = ix[k];
            std::uint64_t v = 0;
            std::size_t used = 0;
            if (!simdscan::parse_uint(s + a, b - a, v, used) || used != b - a) return false;
            if (key == "E") d.event_time = v;
            else if (key == "U") d.first = v;
            else if (key == "u") d.last = v;
        }
        const char c = s[ix[k++]];
        if (c == '}') return k == m;
        if (c != ',') return false;
    }
    return false;
}

On the fixture’s 1 000 depth updates (about 286 bytes each, up to ten levels a side), the indexed decoder takes about $0.39\,\text{µ}\mathrm{s}$ per update; a general parser that builds a tree of maps, vectors and strings and converts prices with strtod about $2.6\,\text{µ}\mathrm{s}$, seven times more; Python’s json module and the conversion to integers about $10\,\text{µ}\mathrm{s}$. The decoded levels, divided by the instrument’s tick and lot, are what Book 3’s firm.wsbook applies.

17.5 Signing requests off the hot path

HMAC (RFC 2104) computes $H\big((K \oplus \mathit{opad}) \,\|\, H((K \oplus \mathit{ipad}) \,\|\, m)\big)$. Both padded keys are one 64-byte block, so the hash’s state after absorbing each of them depends only on the key; the RFC notes that these intermediate results “can be precomputed only once at the time of generation of the key”. A signer that does so absorbs the key once, copies the two states for each request and hashes only the message. The build’s signer does this in C++, Rust and Python, and is checked against RFC 4231’s vectors and against the worked example of a large venue’s documentation, whose published key and order payload give a published signature.

class HmacSha256 {
public:
    explicit HmacSha256(std::string_view key) {
        std::uint8_t k[64] = {}, pad[64];
        if (key.size() > 64) {
            Sha256 h;
            h.update(reinterpret_cast<const std::uint8_t*>(key.data()), key.size());
            h.final(k);
        } else {
            std::memcpy(k, key.data(), key.size());
        }
        for (int i = 0; i < 64; ++i) pad[i] = k[i] ^ 0x36;
        inner_.update(pad, 64);
        for (int i = 0; i < 64; ++i) pad[i] = k[i] ^ 0x5c;
        outer_.update(pad, 64);
    }
    void sign(std::string_view msg, std::uint8_t out[32]) const {
        Sha256 in = inner_, o = outer_;
        in.update(reinterpret_cast<const std::uint8_t*>(msg.data()), msg.size());
        std::uint8_t ih[32];
        in.final(ih);
        o.update(ih, 32);
        o.final(out);

The costs teach two lessons. Precomputing the pads saves about two-fifths with the build’s portable SHA-256 (about $0.64\,\text{µ}\mathrm{s}$ against $1.1\,\text{µ}\mathrm{s}$ for a 119-byte order), and four-fifths with OpenSSL (about $0.19\,\text{µ}\mathrm{s}$ against $0.92\,\text{µ}\mathrm{s}$ for its one-shot HMAC()). And the library is several times faster than a portable implementation because it uses the processor’s SHA instructions: cryptography is the one place where the fast path is someone else’s code, called well.

Figure 17.2 puts the pieces together: the processor time of receiving one depth update and sending one order, step by step, for the build’s C++ components and for its Python reference.

17.6 Tutorial: the web stack, measured

17.7 Build: the web client

17.8 Exercises

17.9 Problem: Cloud to Venue

17.10 Interview questions