18Private Connectivity to Crypto Venues

A venue that runs in a public cloud can offer its market makers a private endpoint inside the provider’s network: their orders never touch the public internet, and, if the endpoint has an interface in the same zone as their machines, they never cross a zone boundary either. The zone identifiers of chapter 16 let a firm check which is the case. The difference is not small: in this chapter’s model, the same private endpoint served from another zone adds about $285\,\text{µ}\mathrm{s}$ to every median round trip, and costs less than a cent more per million messages. Private connectivity is cheap; getting it in the wrong place is expensive only in time.

Chapter 17 found the venue’s region and zone; this chapter connects to it. Virtual networks and the two ways of joining them, peering and private endpoints; what the venues offer (from shared cluster placement to colocation cross-connects for the few that run in data centres); and the tiers of access that unlock the faster paths. The venues’ documentation and the provider’s are cited and dated; the latencies are a labelled simulation built on chapter 16’s published measurement.

18.1 Virtual networks, peering and private endpoints

Peering and endpoints solve different problems. Peering joins two networks and lets them route to each other: fast, simple, but both sides must agree on addresses and trust each other’s routes. An endpoint publishes one service to many customers without joining their networks: each customer gets an interface in its own network, and the provider carries its packets through a load balancer to the service. AWS documents both: peering makes instances “communicate with each other as if they are within the same network”; PrivateLink connects a network to a service “as if they were in your VPC”.

The endpoint’s zone is the catch. An endpoint has interfaces in the zones where it is created; the load balancer behind it sends each connection to targets in its zone unless cross-zone balancing is enabled, which AWS documents as off by default for network load balancers. If the venue’s gateways are in one physical zone and the firm’s instance in another, every packet crosses the boundary somewhere; the only question is where. Chapter 16’s identifiers let the firm compare its instance’s zone with the endpoint’s interfaces before it sends a single order.

def rtt_percentiles(path, qs=(50, 99), n=200_000, seed=0):
    x = cp.sample_rtt(BODIES[path.rtt_body], n, seed) + path.extra_hop_us + 1000.0 * path.edge_ms
    return {q: float(np.percentile(x, q)) for q in qs}


def aligned(instance_zone_id, endpoint_zone_ids):
    """True when the endpoint has a network interface in the instance's physical zone."""
    return instance_zone_id in set(endpoint_zone_ids)


def cost_per_million(path, bytes_per_message=400):
    """Data processed by the endpoint plus cross-zone transfer (out of one zone, into the other), per million
    messages of `bytes_per_message` (request and response together)."""
    gb = 1e6 * bytes_per_message / 1e9
    return gb * path.usd_per_gb + gb * 2 * path.cross_zone_gb_each_way


def monthly_fixed(path, zones=1, hours=730.0):
    return path.usd_hour * zones * hours

18.2 Venue colocation offers

A shared cluster placement group is the cloud’s version of colocation: the venue creates the group in its account and shares it, so that the market maker’s instances land in the same high-bisection segment of the network as the venue’s gateways and matching engine. It is the only offer on the list that brings the customer’s machine physically closer; the others choose a route to wherever the machine happens to be.

18.3 Cross-connects at data-centre-hosted venues

Some crypto venues do not run in a cloud at all. Coinbase’s derivatives exchange runs in a Chicago data centre with its disaster recovery in New Jersey, the same buildings as chapter 10’s venues, and sells the connections of chapter 9: cross-connects to the exchange’s network in the building. Its documentation is explicit about the hierarchy: cross-connects for the lowest latency, PrivateLink for private connectivity without a cross-connect, the internet for the rest. For such a venue the cloud is an access network, not the venue’s home, and the firm’s choice is chapter 9’s.

18.4 Dedicated and market-maker gateways, and the tiers that unlock them

The faster paths are not open to everyone. Venues reserve shared placement groups, private endpoints and dedicated gateways for customers above a volume or a fee, as One Quant Book 3’s dated box on dedicated endpoints and tier matching describes, and may apply separate rate limits to them. For the firm that qualifies, the question is the one of this chapter: which path, from which zone. For the one that does not, it is chapter 17’s: the best instance it can find on the public path.

def wrong_zone():
    s, o = PATHS["endpoint_same"], PATHS["endpoint_other"]
    qs, qo = pl.rtt_percentiles(s), pl.rtt_percentiles(o)
    cost = pl.cost_per_million(o, BYTES) - pl.cost_per_million(s, BYTES)
    return {"p50_added": qo[50] - qs[50], "p99_added": qo[99] - qs[99], "cost_added_per_million": cost}

The costs are an afterthought: one endpoint in one zone for a month is 10.22 USD of endpoint-hours, and even ten billion messages add 40 USD of data processing, 120 across zones. What matters is the zone. The endpoint in the wrong zone adds $285\,\text{µ}\mathrm{s}$ to the median round trip and $360\,\text{µ}\mathrm{s}$ to the 99th percentile, more than the whole difference between peering and a same-zone endpoint.

18.5 Tutorial: which path, which zone

18.6 Build: the private-path planner

18.7 Exercises

18.8 Problem: The Endpoint in the Wrong Zone

18.9 Interview questions