13Linux Tuning

Run a thread that does nothing but read the time-stamp counter in a loop, and record every gap between two consecutive readings longer than $200\,\mathrm{n}\mathrm{s}$, when one pass of the loop takes about fifteen. Each such gap is a moment when the thread was not running: something else had the CPU. On the laptop of this book, pinned to a CPU that no other program of ours uses, the thread loses about 1.3% of its time: some 750 times a second for five microseconds or more, a few times a second for more than a hundred, and every few seconds for a millisecond or more. A market-data message that arrives during a gap waits for the rest of it. None of this is the program’s fault, and none of it can be fixed in the program: it is the operating system doing its own work on the CPU the program thought it had. This chapter is about taking the CPU back. It moves the kernel’s work, the interrupts and the timer tick off the cores that run the hot path, replaces sleeping with polling, keeps the program’s memory in RAM, and ends with a build that audits a host against the placement plan of chapter 4 before a trading process is allowed to start on it.

13.1 Core isolation and CPU affinity

Affinity alone fixes where a thread runs, not what else runs there: a pinned thread still shares its CPU with any other runnable task the scheduler places on it, with kernel threads, and with interrupts. The kernel documentation calls all of these “noise”, asynchronous (interrupts, timers, work queues, kernel threads) or synchronous (system calls and page faults), and notes that it usually goes unnoticed: the timer interrupt can run 1 024 times a second “without a significant and measurable impact most of the time”. The exceptions it names are workloads such as very low latency network processing. Isolation is how the noise is moved: the boot parameter isolcpus (with its default domain flag) takes CPUs out of “the general SMP balancing and scheduling algorithms”, irreversibly until the next boot, and control-group cpusets do the same at run time. A thread reaches an isolated CPU only through its affinity, which is exactly what chapter 9’s firm.affinity sets from chapter 4’s plan.

Figure 13.1 shows what the hook’s thread sees in three situations on this untuned laptop. Pinned alone or left free to move, it loses about one percent of its time in gaps of microseconds, and a few gaps a second reach a hundred microseconds or more. Sharing its CPU with a second spinning thread, it loses about half of it, about 120 times a second for about four milliseconds each: the scheduler hands the CPU to the rival for a slice, and a slice of that length matches the period of this kernel’s tick, which runs 250 times a second (its configuration, CONFIG_HZ). The shared case is the one that isolation removes outright; the small gaps of the other two need the rest of this chapter, and some of them, on a virtual machine, belong to the hypervisor, which the guest cannot see or tune.

On a tuned host the plan of chapter 4 becomes a boot configuration. Figure 13.2 draws it for the two-socket fixture of that chapter: the three hot threads on three cores of the network card’s socket, their hyper-threading siblings taken offline (the kernel’s own checklist says to avoid SMT, “to prevent your hardware thread from being ‘preempted’ by another one”), the card’s interrupts on housekeeping CPUs of the same socket, and everything else on the rest. The documentation asks for at least one housekeeping CPU, preferably one per NUMA node; a trading host keeps several, since the logger, the monitoring agents and the kernel all live there, and a housekeeping CPU that is itself overloaded delays the work (log draining, timers) that the hot path hands to it.

13.2 Interrupts and their affinity

An interrupt runs its handler on whichever CPU receives it, in the middle of whatever that CPU was doing: the network card announcing a packet, a disk finishing a write, another CPU asking this one to flush its translation buffer. On an untuned machine every line may go to every CPU: the audit of this laptop in the build below finds the default mask and all of its active lines allowed on the CPUs that chapter 4 would give the hot path. The kernel documentation’s instruction is short: “isolate the IRQs whenever possible, so that they don’t fire on the target CPUs”. Two details make it harder than writing a mask. Many distributions run irqbalance, a daemon that periodically spreads interrupts across CPUs and would undo a hand-made placement; the Red Hat real-time tuning guide says it is not needed where applications are bound to CPUs, and disables it. And some interrupts belong on a hot core’s socket but not on the core: the network card’s receive queues should interrupt a housekeeping CPU of the card’s node (Figure 13.2), so that the kernel’s packet processing leaves the data in the socket’s shared cache, where the hot threads read it, rather than across the interconnect (chapter 4).

13.3 Tickless kernels and housekeeping

The scheduling-clock interrupt, the tick, lets the kernel account time, run timers and decide whether to switch tasks, 100 to 1 000 times a second depending on the build. A CPU with a single runnable task has nothing to switch to, and the kernel’s documentation on adaptive ticks puts the gain precisely: real-time applications “improve their worst-case response times by the maximum duration of a scheduling-clock interrupt”. The same document lists the conditions: the kernel must be built with CONFIG_NO_HZ_FULL (this laptop’s is not), the CPU must run one task only, the thread must stay out of the kernel (system calls and page faults bring the noise back), and at least one CPU must keep ticking for timekeeping. Read-copy-update callbacks, deferred work the kernel queues on each CPU, go with the tick: nohz_full offloads them from the listed CPUs, and rcu_nocbs does so explicitly. The price is paid elsewhere: entering and leaving the kernel costs more on a tickless CPU, and the housekeeping CPUs take the offloaded work.

What does a gap cost a message? The hook’s thread spends its time either running or stopped, and a message arriving at a random moment waits only if it arrives during a stop.

The squares matter: the wait is dominated by the long gaps, not the frequent ones. A tick of $2\,\text{µ}\mathrm{s}$ at 250 Hz steals 0.05% of the time and adds half a nanosecond on average; a single gap of $1\,\mathrm{m}\mathrm{s}$ a second steals 0.1% but adds $500\,\mathrm{n}\mathrm{s}$ on average and makes one message in two thousand wait more than half a millisecond. On the laptop, pinned alone, the meter’s gaps give an average added wait of about three microseconds (it moves by a factor of two or more from run to run, with the run’s longest gap: the squares at work), a wait beyond $10\,\text{µ}\mathrm{s}$ for about 0.45% of messages and beyond $100\,\text{µ}\mathrm{s}$ for 0.1 to 0.2%: the host adds a 99.9th percentile of over a hundred microseconds to a path whose own work takes a few microseconds or less (chapter 1). The proposition assumes messages arrive independently of the gaps; bursts that coincide with the kernel’s work (an interrupt from the same card that brings the data) make it worse.

13.4 Busy polling and kernel bypass

A blocking receive is the default and the right choice for almost every program: the thread sleeps, the CPU does other work or saves power, and the kernel wakes the thread when the packet has gone through the interrupt handler, the protocol stack and the socket’s buffer (Figure 13.3, left). Each step costs microseconds, and the wake-up is the worst of them: the sleeping thread’s CPU may be idle in a low-power state, and the thread must be scheduled back in. Busy polling removes the sleep: the thread asks for data in a loop, with a non-blocking call, and finds it as soon as the kernel has put it in the socket (middle). The socket option SO_BUSY_POLL makes the kernel itself spin for up to a set number of microseconds inside a blocking receive (socket(7)); the tutorial polls from user space instead, which works on any socket. Kernel bypass removes the kernel: in the open-source DPDK, a poll-mode driver “accesses the RX and TX descriptors directly without any interrupts”, in a run-to-completion loop on a dedicated core (right). One Quant Book 14 covers the stacks and the cards; what matters here is that each step to the right trades a CPU spinning at 100% for microseconds.

Figure 13.4 measures the first two on the laptop’s loopback interface, where no card is involved and the difference is the sleep alone. A round trip of 64 bytes between two pinned threads takes about $30\,\text{µ}\mathrm{s}$ at the median with blocking receives and about $2.3\,\text{µ}\mathrm{s}$ busy-polled, over ten times less. Two threads bouncing a byte through a pair of pipes on the same CPU complete a round trip in about $1.5\,\text{µ}\mathrm{s}$: two context switches with their system calls, each well under a microsecond when the CPU is awake. The same ping-pong between two CPUs takes about $30\,\text{µ}\mathrm{s}$: the cost is not the switch but waking a CPU that went idle while waiting, which is also why the blocking receive is slow. Kernel bypass, which removes the system calls as well, needs a card that supports it, which this laptop does not have.

The tails tell the rest: busy polling cuts the median, not the p99.9, which on this laptop is still tens of microseconds, because the spinning thread meets the gaps of Figure 13.1 like any other. Polling and isolation go together: a thread that polls on a CPU where the kernel still interrupts it is fast on average and slow in the tail, and an isolated CPU whose thread sleeps in a blocking call wakes up slowly. The idle states are the third part: a CPU that is never idle never enters a deep one, but the housekeeping CPUs that wake the hot path’s helpers can, and the kernel documentation lists intel_idle.max_cstate= as the way to limit their depth, and idle=poll as a stronger one that “can cause your CPU to overheat” and disables turbo frequencies.

13.5 Real-time priorities and memory locking

A real-time policy answers the shared-CPU curve of Figure 13.1: the rival no longer gets a slice. On an isolated core with a single thread there is no rival to preempt, and the policy brings a risk instead. A real-time thread that spins forever would starve the CPU’s kernel threads, so Linux throttles real-time threads by default: they may run $950\,\mathrm{m}\mathrm{s}$ of every second, and the remaining $50\,\mathrm{m}\mathrm{s}$ go to ordinary tasks (the kernel’s documentation of real-time group scheduling). A spinning SCHED_FIFO thread on a CPU that also has an ordinary task runnable is therefore stopped for up to $50\,\mathrm{m}\mathrm{s}$ once a second, a hiccup a thousand times longer than any on the untuned laptop. Throttling can be disabled (sched_rt_runtime_us set to $-1$), which the Red Hat guide calls adequate only when the real-time tasks are well engineered. An ordinary process may not ask for the policy at all: on this laptop the request fails with EPERM. The firm’s design therefore spins ordinary threads on isolated cores, where there is nothing to preempt, and keeps real-time priorities for threads that must preempt others on shared CPUs.

Page faults are the synchronous noise of the kernel documentation, and chapter 6 measured them: the first write to a page costs a microsecond or more. The mlockall manual page names the reason to lock: “paging is one major cause of unexpected program execution delays” for real-time algorithms. On the laptop, writing one byte to each page of a fresh 32 MiB mapping takes 8 192 page faults and about one to two microseconds a page. After mlockall(MCL_CURRENT | MCL_FUTURE), the same writes take no fault and about $35\,\mathrm{n}\mathrm{s}$ a page: the cost has moved to the mmap call, which now populates every page up front, at start-up, where it belongs. Locking is limited by RLIMIT_MEMLOCK, 64 MiB for an ordinary user on this machine: a process whose book and rings need a gigabyte must be given the limit (the build checks it), or mlockall fails with ENOMEM.

13.6 Tutorial: measure an untuned machine

13.7 Build: the host audit

firm.tuneaudit reads the same files an engineer would, from /proc and /sys or from a fixture tree with their layout, and checks each rule of this chapter against the plan of firm.coreplan: the hot CPUs isolated, their siblings offline or isolated, the hot CPUs tickless and their read-copy-update callbacks offloaded, no interrupt line (nor the default mask) allowed on a hot CPU, irqbalance not running, the performance frequency governor on the hot CPUs, the idle states capped on the boot line, transparent huge pages only where asked for (madvise, with no stalling always defragmentation, which the kernel documentation says makes an allocation “stall on allocation failure and directly reclaim pages and compact memory”), a locked-memory limit large enough, and real-time throttling off if the hot threads are real-time.

    isolated = set(parse_cpulist(_text(root, "sys/devices/system/cpu/isolated") or ""))
    miss = hot_cpus - isolated
    add("isolated", not miss, f"hot CPUs not isolated: {_fmt(miss)}" if miss else f"isolated {_fmt(isolated)}")

    online = {c for c in idle if _text(root, f"sys/devices/system/cpu/cpu{c}/online") != "0"}
    busy = online - isolated
    add("siblings", not busy, f"idle siblings online and not isolated: {_fmt(busy)}" if busy else "siblings idle")

    nohz = _text(root, "sys/devices/system/cpu/nohz_full")
    if nohz is None:
        add("nohz_full", False, "kernel built without adaptive ticks (no nohz_full file)")
    else:
        miss = hot_cpus - set(parse_cpulist(nohz))
        add("nohz_full", not miss, f"ticking hot CPUs: {_fmt(miss)}" if miss else f"nohz_full {nohz}")

    offloaded = set(parse_cpulist(cmd.get("rcu_nocbs", ""))) | set(parse_cpulist(nohz or ""))
    miss = hot_cpus - offloaded
    add("rcu_nocbs", not miss, f"RCU callbacks run on {_fmt(miss)}" if miss else "RCU callbacks offloaded")

    bad = []
    dflt = _text(root, "proc/irq/default_smp_affinity")
    if dflt is not None and mask_cpus(dflt) & hot_cpus:
        bad.append("default")
    for d in sorted((root / "proc/irq").glob("[0-9]*"), key=lambda p: int(p.name)):
        cpus = _text(root, f"proc/irq/{d.name}/smp_affinity_list")
        if cpus is not None and set(parse_cpulist(cpus)) & hot_cpus:
            bad.append(d.name)
    shown = ", ".join(bad[:5]) + (f" and {len(bad) - 5} more" if len(bad) > 5 else "")
    add("irq_affinity", not bad, f"interrupts allowed on hot CPUs: {shown}" if bad else "no interrupt on hot CPUs")

A setting the host does not expose is reported unknown, not passed: this laptop has no frequency governor to read, as virtual machines often do not. Its report fails seven of the eleven rules (the hot CPUs neither isolated nor tickless, their siblings busy, every interrupt allowed on them, no idle-state cap, a 64 MiB locked-memory limit), passes three and cannot judge one. The tuned fixture, which describes Figure 13.2, passes all eleven; the mistuned fixture fails nine, one per usual mistake.

13.8 Exercises

13.9 Problem: The Hiccup

13.10 Interview questions