Quantitative Finance · Book 18 · Careers

The Interview Book

The Interview Book · Careers

24Python

A helper that appends each fill to a default list argument works in the unit test and doubles every position by the end of the first live day. The candidate asked about it in an interview has twenty seconds to say why the list was shared, and another minute to say how he would have found it without being told. Python interviews for researchers and developers test the data model (names, objects, mutability), iteration, the interpreter’s concurrency model, and the numerical stack’s views, copies and vectorisation; the platform side is One Quant Book 15, chapter 8. Every snippet here runs in the chapter’s tests with the pinned interpreter (CPython 3.10), and where an answer depends on CPython rather than on the language, the solution says so.

24.1 The data model: names, objects and mutability

A Python variable is a name bound to an object; assignment rebinds a name and never copies. Mutable objects (lists, dicts, sets, most user classes) can change in place, and every name bound to them sees the change. Default argument values are evaluated once, when the def statement runs, so a mutable default is shared by every call. Equality (==) compares values through __eq__; identity (is) compares objects, and whether two equal immutable values are the same object is an implementation detail. A type used as a dictionary key needs __hash__ consistent with __eq__ and must not change while it is a key, which is what a frozen dataclass provides.

def add_fill(fill, fills=[]):  # noqa: B006 -- the bug the question is about
    fills.append(fill)
    return fills


print(add_fill(1))
print(add_fill(2))
print(add_fill(3, []))
Listing 24.1. A mutable default argument. code/interviews/24-python/python/snippets/iv_snip_mutable_default.py

24.2 Iterators, generators and context managers

An iterator produces values on demand and is exhausted after one pass; a generator is an iterator written as a function with yield, or as a generator expression. Pipelines of generators process a file of any size in constant memory, because each stage pulls one record at a time from the one before. A context manager pairs set-up with guaranteed tear-down (with), which is how files, locks and temporary changes of state are made safe against exceptions.

def read_fills(path):
    """Yield (symbol, qty, price) from a CSV of fills one line at a time: constant memory."""
    with open(path, encoding="ascii") as f:
        next(f)  # header
        for line in f:
            sym, qty, px = line.rstrip("\n").split(",")
            yield sym, int(qty), float(px)


def large_fills(rows, min_notional):
    return (r for r in rows if abs(r[1]) * r[2] >= min_notional)


def notional_by_symbol(rows):
    out = {}
    for sym, qty, px in rows:
        out[sym] = out.get(sym, 0.0) + qty * px
    return out
Listing 24.2. A generator pipeline over a file of fills: read, filter, aggregate, one line in memory at a time. code/interviews/24-python/python/iv_py.py

Closures capture variables, not values: a function created in a loop sees the loop variable’s final value when called later (Interview question 24.2). Mutating a dictionary while iterating over it raises an error; build a new one instead.

24.3 The interpreter lock, threads, processes and async

CPython’s global interpreter lock (One Quant Book 15, chapter 8) lets one thread execute Python bytecode at a time. Threads therefore speed up work that waits (network, disk) or that runs in C code releasing the lock (most of numpy’s heavy operations, compression, many I/O libraries), and do not speed up pure-Python computation, for which processes are the answer, at the cost of serialising data between them. asyncio runs many waiting tasks on one thread with explicit yield points, suited to many network connections (a crypto market-data client with hundreds of websocket streams). The lock does not make compound operations atomic: x += 1 on a shared variable is several bytecodes, and threads can interleave between them.

24.4 The numerical stack

Vectorisation replaces a Python loop by one call into compiled code over an array, typically tens to hundreds of times faster for arithmetic (One Quant Book 15, chapter 8, on interpreter overhead and array programming). Three rules avoid the classic bugs. Basic slicing of a numpy array returns a view that shares memory with the original; fancy indexing (a list or boolean mask) returns a copy. In pandas, assign through .loc in one step: chained indexing such as df[mask][’col’] = v may assign to a temporary copy and leave the frame unchanged. And floating-point sums depend on order: math.fsum or pairwise summation (numpy’s) keeps error small where a naive running sum does not (One Quant Book 4, chapter 25).

def rolling_mean_vec(x, k):
    """Rolling mean by cumulative sums: O(n), no Python loop."""
    c = np.cumsum(np.concatenate([[0.0], np.asarray(x, dtype=float)]))
    return (c[k:] - c[:-k]) / k
Listing 24.3. A rolling mean by cumulative sums: linear time and no Python loop. code/interviews/24-python/python/iv_py.py

Method 24.1 (Answering a Python output question)

  1. Track names and the objects they are bound to; ask which objects are mutable and shared.
  2. Note when each expression is evaluated: defaults at definition, closures at call, generators on demand.
  3. Separate language guarantees from CPython details (small-integer caching, dictionary internals).
  4. For numpy and pandas, ask for every indexing operation whether it returns a view or a copy.

24.5 Question bank

Interview question 24.1 ★ developer, researcher • systematic fund

What does Listing 24.1 print, and how do you fix the function?

Solution

Solution of Interview question 24.1.

[1], [1, 2], [3]. The default list is created once, when def runs, and every call without the argument appends to the same list; the third call passes its own. Fix: def add_fill(fill, fills=None) and if fills is None: fills = []. A linter flags the pattern (ruff’s B006), which is how to find it without being told.

What the interviewer is looking for: evaluation time of defaults, the standard fix, and tooling that catches it.

pricers = [lambda x: x * k for k in range(3)]  # noqa: B023 -- the bug the question is about
print([p(10) for p in pricers])
fixed = [lambda x, k=k: x * k for k in range(3)]
print([p(10) for p in fixed])
Listing 24.4. Pricers made in a loop. code/interviews/24-python/python/snippets/iv_snip_late_binding.py

Interview question 24.2 ★ developer, researcher • any

What do the two lines of Listing 24.4 print, and why does the second fix the first?

Solution

Solution of Interview question 24.2.

[20, 20, 20] then [0, 10, 20]. The lambdas capture the variable k, not its value, and are called after the loop ends with k = 2. A default argument k=k is evaluated when each lambda is created, capturing the current value; functools.partial does the same.

What the interviewer is looking for: late binding of closures and the default-argument fix.

a = int("256")
b = int("256")
x = int("1000")
y = int("1000")
print(a == b, a is b)
print(x == y, x is y)
Listing 24.5. Equality and identity. code/interviews/24-python/python/snippets/iv_snip_identity.py

Interview question 24.3 ★ developer • any

What does Listing 24.5 print under CPython? Which parts of that answer does the language guarantee?

Solution

Solution of Interview question 24.3.

True True then True False. The language guarantees only the equalities. CPython keeps one object for each integer from −5-5 to 256, so the two 256s are the same object, while int(’1000’) creates a new object each time. Never use is to compare numbers or strings; use it for None and sentinels.

What the interviewer is looking for: equality against identity, and naming the CPython detail as a detail.

squares = (q * q for q in range(3))
print(sum(squares))
print(sum(squares))
Listing 24.6. A generator used twice. code/interviews/24-python/python/snippets/iv_snip_generator_twice.py

Interview question 24.4 ★ researcher, mle • systematic fund

What does Listing 24.6 print?

Solution

Solution of Interview question 24.4.

5 then 0: the first sum consumes the generator (0+1+40 + 1 + 4) and the second finds it exhausted. Use a list if the values are needed twice, or recreate the generator.

What the interviewer is looking for: single-pass iterators.

positions = {"A": 0, "B": 5, "C": 0}
try:
    for sym, qty in positions.items():
        if qty == 0:
            del positions[sym]
except RuntimeError as e:
    print("RuntimeError:", e)
positions = {s: q for s, q in {"A": 0, "B": 5, "C": 0}.items() if q != 0}
print(positions)
Listing 24.7. Removing flat positions. code/interviews/24-python/python/snippets/iv_snip_dict_mutation.py

Interview question 24.5 ★★ developer, researcher • multi-manager fund

What happens in the loop of Listing 24.7, and why is the last line the right way to do it?

Solution

Solution of Interview question 24.5.

Deleting a key while iterating over the dictionary changes its size, and CPython raises RuntimeError: dictionary changed size during iteration at the next step. Build a new dictionary with a comprehension (the last line), or iterate over a copy of the keys (for sym in list(positions)).

What the interviewer is looking for: the mutation-during-iteration rule and two safe patterns.

Interview question 24.6 ★★ developer, researcher • systematic fund

Compute notional by symbol over the large fills in a 50-gigabyte CSV file on a machine with 16 gigabytes of memory, in plain Python.

Solution

Solution of Interview question 24.6.

Stream it: a generator that reads one line at a time, a generator that filters, and a dictionary accumulating notional per symbol (Listing 24.2); memory is the dictionary plus one line, whatever the file’s size. The chapter’s test runs it on 5 000 generated fills and checks it against a direct computation. For speed rather than memory, read in chunks with pandas or Polars and aggregate per chunk.

What the interviewer is looking for: constant-memory streaming with generators.

Interview question 24.7 ★★ developer • bank

Write a context manager that changes a risk limit for the duration of a block and restores it even if the block raises.

Solution

Solution of Interview question 24.7.

@contextmanager
def temporary_limit(limits, key, value):
    """Change a risk limit for a block and restore it even if the block raises."""
    old = limits[key]
    limits[key] = value
    try:
        yield limits
    finally:
        limits[key] = old
Restore the limit in a finally clause. code/interviews/24-python/python/iv_py.py

The finally runs whether the block ends normally or by an exception, and the exception still propagates. The chapter’s test checks both paths.

What the interviewer is looking for: contextlib.contextmanager with try/finally.

import numpy as np

prices = np.array([100.0, 101.0, 102.0, 103.0])
window = prices[1:3]
window[0] = 0.0
picked = prices[[2, 3]]
picked[0] = 0.0
print(prices)
print(np.shares_memory(prices, window), np.shares_memory(prices, picked))
Listing 24.8. Views and copies. code/interviews/24-python/python/snippets/iv_snip_views.py

Interview question 24.8 ★★ researcher, mle • systematic fund

What does Listing 24.8 print?

Solution

Solution of Interview question 24.8.

[100. 0. 102. 103.] and True False. prices[1:3] is a view, so writing to it changes prices[1]; prices[[2, 3]] is a copy, so writing to it changes nothing in prices. The function np.shares_memory confirms which is which.

What the interviewer is looking for: basic slicing gives views, fancy indexing gives copies.

import math

ticks = [0.1] * 10
print(sum(ticks) == 1.0, sum(ticks))
print(math.fsum(ticks) == 1.0)
Listing 24.9. Summing ten ticks of 0.1. code/interviews/24-python/python/snippets/iv_snip_float_sum.py

Interview question 24.9 ★★ researcher, developer • market maker

What does Listing 24.9 print? When does the difference matter on a desk?

Solution

Solution of Interview question 24.9.

False 0.9999999999999999 and True: 0.1 is not exactly representable in binary, and the running sum accumulates the rounding; math.fsum tracks the lost low-order bits and returns the correctly rounded sum. On a desk it matters when a position or cash balance is checked for exact zero, when many small fills are summed, and when two systems reconcile totals computed in different orders: keep money in integer units (cents, ticks) and compare floats with a tolerance.

What the interviewer is looking for: binary representation, compensated summation, and integer units for money.

Interview question 24.10 ★★★ developer, mle • crypto firm

A service subscribes to 300 websocket market-data streams and computes a statistic per message. When do threads help, when do processes, and when asyncio? What does the interpreter lock not protect you from?

Solution

Solution of Interview question 24.10.

The work is mostly waiting on sockets, so asyncio (one thread, 300 streams, cheap task switches) or a thread per group of streams both fit; threads help because they release the lock while blocked. If the per-message statistic is heavy pure Python, the lock serialises it: move it to numpy (which releases the lock), to a compiled extension, or to worker processes fed through queues. The lock does not make compound operations atomic: a shared counter or dictionary updated from several threads still needs a lock (or a queue to a single owner).

What the interviewer is looking for: matching the concurrency model to I/O-bound and CPU-bound work, and the limits of the GIL.

Interview question 24.11 ★★★ researcher, mle • systematic fund

Vectorise a rolling mean of window kk over a price array without a Python loop, and say how you check it.

Solution

Solution of Interview question 24.11.

With cc the cumulative sum of the prices preceded by a zero, the mean of the window ending at ii is (ci+1−ci+1−k)/k(c_{i+1} - c_{i+1-k})/k (Listing 24.3). Check it against the obvious loop on random inputs of random lengths and windows, as the chapter’s test does for 50 seeds; for long series of large numbers, beware the cancellation in differences of large cumulative sums, and use np.convolve or pandas’ rolling for stability.

What the interviewer is looking for: the cumulative-sum trick, an oracle check, and its numerical caveat.

import warnings

import pandas as pd

with warnings.catch_warnings(record=True) as caught:
    warnings.simplefilter("always")
    df = pd.DataFrame({"qty": [5, -3, 2], "flag": [0, 0, 0]})
    df[df["qty"] < 0]["flag"] = 1
    print(df["flag"].tolist())
    print(any("SettingWithCopy" in type(w.message).__name__ for w in caught))
df.loc[df["qty"] < 0, "flag"] = 1
print(df["flag"].tolist())
Listing 24.10. Flagging short positions in pandas. code/interviews/24-python/python/snippets/iv_snip_chained.py

Interview question 24.12 ★★★ researcher, developer • multi-manager fund

What does Listing 24.10 print, and why? How does copy-on-write, announced as the default for pandas 3.0, change the answer?

Solution

Solution of Interview question 24.12.

[0, 0, 0], True, [0, 1, 0]. df[df[’qty’] < 0] (a boolean mask) returns a copy, and [’flag’] = 1 assigns into that copy, which is discarded: the frame is unchanged and pandas 2.3 emits a SettingWithCopyWarning. The .loc[mask, ’flag’] = 1 form assigns in one step on the frame itself. Under copy-on-write every indexing result behaves as a copy, so chained assignment never modifies the original and pandas warns about it; .loc remains the way to write.

What the interviewer is looking for: copies from masks, one-step assignment with .loc, and awareness of copy-on-write.

Interview question 24.13 ★★★ developer • proprietary firm

Write a value type for an order key (venue, client order identifier) that can be used in a dictionary. What goes wrong if a mutable object is used as a key and then changed?

Solution

Solution of Interview question 24.13.

@dataclass(frozen=True)
class OrderKey:
    """A value type usable as a dict key: equality and hash from the fields, immutable."""

    venue: str
    client_order_id: int
A frozen dataclass: equality and hash from the fields. code/interviews/24-python/python/iv_py.py

A mutable key changed after insertion keeps its old hash bucket, so lookups by the new value and by the old one can both fail, and the entry is lost in the dictionary. A frozen dataclass cannot be changed, and its generated __hash__ is consistent with __eq__. A plain tuple works too.

What the interviewer is looking for: hash and equality consistency, immutability, and what breaks otherwise.

Sources and further reading

  • The Python Language Reference and Library Reference, version 3.10 (the series’ pinned interpreter); the pandas 2.3 and numpy 2.2 user guides on indexing, views and copies, and copy-on-write.
  • One Quant Book 15, chapter 8 (Python at scale); One Quant Book 4, chapter 25 (floating point).