29Build a Connectivity Plan

A new firm will trade three venues: equities on an exchange in New Jersey, futures on an exchange outside Chicago, and a crypto venue whose matching engine runs in a cloud region in Tokyo. Before it signs anything it has to say what latency it will get at each, what it must buy, from whom, on what terms, and what it will cost a year. The book’s last chapter answers with one configuration and the book’s own components: USD 1.02 million a year all in, three of four latency targets met, and a missed one that a private endpoint fixes for USD 123 a year.

Every earlier chapter supplied a piece. Chapter 10 gave the sites and the latency floors; chapter 13 fibre routes and chapter 14 radio; chapters 2 to 8 the cage, cards, software path and servers; chapter 9 colocation prices; chapter 15 circuits and service levels; chapters 16 to 18 the cloud; chapter 27 the vendor map; chapter 28 recovery. This chapter joins them in firm.connplan, the book’s capstone, and runs it on a plan whose prices are dated rows from the venues’ own schedules.

29.1 Requirements: strategies, venues and latency targets

The firm’s strategies set the targets. Its equity strategy reacts to the equity venue’s own book, so it colocates in Mahwah and needs a wire-to-wire time under $40\,\text{µ}\mathrm{s}$ at the 99th percentile. Its futures strategy hedges on CME: it needs Aurora’s prices in Mahwah within $4.2\,\mathrm{m}\mathrm{s}$ and its orders in Aurora within $8\,\mathrm{m}\mathrm{s}$. Its crypto strategy trades a venue in AWS’s Tokyo region from instances there, and needs round trips to the venue under one millisecond at the 99th percentile (all targets are the firm’s, and assumptions here).

VENUES = (
    V("NYSE equities (Mahwah)", "colocated", 40.0, site="mahwah", metric="wire-to-wire p99"),
    V("CME data (Aurora to Mahwah)", "radio", 4200.0, site="aurora", published_us=3986.0,
      metric="one way"),
    V("CME orders (Mahwah to Aurora)", "fibre", 8000.0, site="aurora", factor=1.3,
      metric="one way"),
    V("Binance (AWS Tokyo)", "cloud", 1000.0, path="internet",
      offers=("internet", "endpoint_same", "endpoint_other"), metric="round trip p99"),
)
LINES = (
    L("nyse_cabinet", 1, "shared", "colocation"), L("nyse_kw", 8, "shared", "colocation"),
    L("nyse_xc", 4, "shared", "colocation"),
    L("nyse_lcn_10g", 1, "NYSE equities (Mahwah)", "connectivity"),
    L("nyse_ip_1g", 1, "NYSE equities (Mahwah)", "connectivity"),
    L("wave_mah_aur", 1, "CME orders (Mahwah to Aurora)", "connectivity"),
    L("nyse_wl_cme", 1, "CME data (Aurora to Mahwah)", "market data"),
    L("nyse_cme_feed", 1, "CME data (Aurora to Mahwah)", "market data"),
    L("aws_tokyo_c7i_4xl", 2, "Binance (AWS Tokyo)", "cloud"),
)
PLAN = cp.Plan(VENUES, LINES)

29.2 The latency table

Each access kind takes its latency from a component: a colocated path from chapter 2’s cage and chapter 5’s software path (firm.cagenet, firm.wirepath, Book 13’s measured stages); a radio route from its published figure; a fibre route from the geodesic and a route factor (firm.fibreroute); a cloud path from chapter 18’s private paths (firm.privlink).

def latency_us(v, plan):
    if v.access == "colocated":
        return _colocated_p99()
    if v.access == "radio":
        return v.published_us
    if v.access == "fibre":
        a, b = SITES[plan.home], SITES[v.site]
        km = gm.geodesic_m(a.lat, a.lon, b.lat, b.lon) / 1000.0
        return fr.components(fr.build(v.name, km, v.factor))["total"]
    if v.access == "cloud":
        return pl.rtt_percentiles(pl.load_paths()[v.path])[99]
    raise ValueError(v.access)

The missed target is the one the plan can fix cheaply. The configuration gives the crypto venue chapter 18’s three paths (an assumption about what it offers): its public endpoint through an edge network, and a private endpoint in the firm’s zone or in another zone. The plan tries each and ranks those that meet the target by cost.

def cheapest_fixes(plan, prices, bytes_per_month=0.0):
    """Each cloud venue missing its target: the offered paths that meet it, cheapest first."""
    fixes = []
    paths = pl.load_paths()
    for v in plan.venues:
        if v.access != "cloud" or latency_us(v, plan) <= v.target_us:
            continue
        for option in v.offers:
            alt = dataclasses.replace(v, path=option)
            x = latency_us(alt, plan)
            if x <= v.target_us:
                p = paths[option]
                extra = 12 * (p.usd_hour * 730 + p.usd_per_gb * bytes_per_month / 1e9)
                fixes.append({"venue": v.name, "option": option, "achieved_us": x,
                              "extra_annual": extra})
    return sorted(fixes, key=lambda f: f["extra_annual"])

A private endpoint in the instance’s zone brings the 99th-percentile round trip from $2.40\,\mathrm{m}\mathrm{s}$ to $0.44\,\mathrm{m}\mathrm{s}$, for USD 122.64 a year of endpoint hours (at chapter 18’s price; its traffic charge adds to that with volume); one in another zone meets the target too, at $0.80\,\mathrm{m}\mathrm{s}$, for the same fixed price plus cross-zone transfer. The fix is to align the endpoint with the instance’s zone identifier, chapter 16’s lesson.

29.3 The bill of materials

Two checks run before the plan is trusted: every price row must name its ledger source, and none may be older than 60 days. On the day of writing, one row fails the first check, the Mahwah–Aurora wavelength, whose USD 20 000 a month is an assumption the firm must replace with a carrier’s quote; none fails the second. The table is also the plan’s export: export_cost_table writes it with its sources and dates for the firm’s budget owners (Book 16 reads it).

29.4 Contracts and service levels

The plan’s access to the equity venue is one 10 Gb connection, with a 1 Gb IP-network circuit as a backup. With each circuit failing once in 4 000 hours and repaired in four (assumptions), one connection alone is available 99.900% of the time; with the backup and a common-mode event (a shared duct or room) twice in ten years for eight hours, 99.982% (firm.slamodel, chapter 15). A second 10 Gb connection beside the backup would add USD 366 250 a year to the budget with the recovery share: the price of keeping full bandwidth through a failure. The fee schedule read for the plan mentions no service-level credits for these circuits; they belong in the contract.

29.5 The annual budget and its sensitivities

The plan costs USD 68 812 a month plus 39 500 in one-time fees: USD 1 017 289 a year with the fees amortised over three years and the recovery site at a quarter of the on-site spend. Connectivity is 56% of it, and one connection, the 10 Gb LCN and NMS link, is 29% alone. The sensitivities show where the budget moves: a second LCN connection adds USD 366 250 a year; four more kW of power 72 000; a wavelength priced 50% above the assumption 152 083; doubling the Tokyo instances only 15 747; dropping the backup circuit saves 38 542 and costs the availability above. The cloud leg, the one that missed its target, is the cheapest to fix and to grow.

29.6 Tutorial: three venues, one budget

29.7 Build: the connectivity plan

29.8 Exercises

29.9 Problem: Three Venues, One Budget

29.10 Interview questions