15The Research Environment

In 2013 three economists replicating an influential paper on public debt and growth obtained the authors’ working spreadsheet. A coding error in it excluded five countries — Australia, Austria, Belgium, Canada and Denmark — from the analysis; with that and two other problems corrected, the average growth of countries with public debt above 90% of output was 2.2%, not the $-0.1\%$ published (Herndon, Ash and Pollin). The economics was not what failed. What failed was an environment in which a calculation could not be rerun from the top by anyone but its author, and not even by its author once the state of the spreadsheet had drifted from what it showed. A research platform’s notebooks have the same property, more so. This chapter audits a notebook that lies in the same way, reruns it from the top, promotes its calculation to a library, and reproduces the corrected number from a clean directory.

15.1 Notebooks and their hidden state

The notebook is the fastest way to explore data and the easiest way to publish a number nobody can recompute. Pimentel, Murta, Braganholo and Freire collected Jupyter notebooks from GitHub and tried to run them: out of 863 878 attempted executions of valid notebooks, only 24.11% executed without errors and only 4.03% produced the same results. Execution counters give hidden state away — a counter skipped means a cell re-run or run and deleted, a counter lower than the cell above it means execution out of document order — but only if something reads them.

The chapter’s planted notebook computes the headline of the hook’s debate on a synthetic panel of twenty countries from 1946 to 2009: the average real growth of country-years with debt above 90% of output, averaged by country. Its history is written down in the code: the cells, in document order, and what was run, in the order it was run.

CELLS = {                       # the notebook's code cells, in document order
    "load": 'import pandas as pd\ndf = pd.read_csv("panel.csv")',
    "clean": "df = df[df.year >= 1950]",
    "high": "high = df[df.debt > 90]",
    "select": "sel = high[~high.country.isin(exclude)]",
    "result": 'result = sel.groupby("country").growth.mean().mean()\n'
              'print(f"{result:.2f}")',
}
DELETED = f"exclude = {EXCLUDED!r}"
HISTORY = ["load", "high", "DELETED", "select", "result", "clean"]    # what ran, in that order

Saved, the notebook shows 1.96 under its result cell. Nothing in the document says that five countries were excluded, or that the years before 1950 were still in the data when the number was computed.

15.2 Auditing the document

firm.envaudit reads the notebook’s JSON and nothing else (Listing 15.2). It walks the code cells in document order with their counters, and parses each cell to see which names it reads and which it binds.

def audit(nb: dict) -> list[Finding]:
    out: list[Finding] = []
    cells = code_cells(nb)
    ran = [(i, n) for i, _s, n, _o in cells if n is not None]
    for (i, n), (j, m) in zip(ran, ran[1:], strict=False):
        if m < n:
            why = f"counter {m} below the cell above it ({n}, cell {i})"
            out.append(Finding("out-of-order", j, why))
    counters = sorted(n for _i, n in ran)
    for a, b in zip(counters, counters[1:], strict=False):
        if b - a > 1:
            gap = f"{a + 1}" if b - a == 2 else f"{a + 1} to {b - 1}"
            why = f"counter {gap} missing: a re-run or a deleted cell"
            out.append(Finding("skipped-counter", -1, why))
    everywhere = set()
    for _i, s, _n, _o in cells:
        everywhere |= _names(s)[0]
    defined = set()
    for i, s, n, outputs in cells:
        if n is None and outputs:
            why = "outputs saved but no execution counter"
            out.append(Finding("output-without-run", i, why))
        if n is None and not outputs and s.strip():
            out.append(Finding("never-run", i, "code never executed"))
        bound, read = _names(s)
        for name in sorted(read - defined - BUILTINS):
            kind = "used-before-defined" if name in everywhere else "never-defined"
            out.append(Finding(kind, i, name))
        defined |= bound
    return out

On the planted notebook it finds three things (Table 15.1). The cell that selects country-years above 90% ran second, but a cell above it ran sixth: the cleaning cell was run after the result, so the displayed data are not the data the result used. Counter 3 is missing: a cell was run and is gone. And the selection cell reads exclude, which no cell in the notebook defines.

Rerun from the top in a fresh interpreter (Listing 15.3), the notebook fails: NameError: name ’exclude’ is not defined. The number on the page cannot be produced by the page.

def rerun(nb: dict, workdir, timeout: int = 120, env: dict | None = None) -> Rerun:
    """All code cells, top to bottom, as one script in a fresh interpreter."""
    workdir = pathlib.Path(workdir)
    workdir.mkdir(parents=True, exist_ok=True)
    script = workdir / "rerun.py"
    script.write_text("\n\n".join(s for _i, s, _n, _o in code_cells(nb)) + "\n")
    r = subprocess.run([sys.executable, str(script)], cwd=workdir, capture_output=True,
                       text=True, timeout=timeout, env=env)
    return Rerun(r.returncode == 0, r.stdout, r.stderr, r.returncode)

15.3 From notebook to library

The fix is to decide, in the open, what the calculation is. With no exclusion and the cells run as they stand, the notebook prints 2.44. The calculation belongs in the library (Listing 15.4): the threshold, the first year, the excluded countries and the weighting become arguments, and each has a test. The library reproduces all three numbers the debate turns on: 1.96 with the five countries excluded and all years (the saved number, explained), 2.44 with every country from 1950 (the corrected number), and 2.90 if every country-year counts once instead of every country — the weighting choice the hook’s critique also questioned. The five excluded countries hold 204 of the 455 country-years above 90% since 1950.

def average_growth(df: pd.DataFrame, threshold: float = 90.0, since: int | None = None,
                   exclude=(), weight: str = "country") -> float:
    """Average growth of country-years with debt above `threshold` (percent of output).

    weight='country': the mean of each country's mean (each country counts once);
    weight='country-year': the pooled mean of the country-years. `since` drops earlier
    years and `exclude` drops countries: the caller states both, no notebook remembers them."""
    d = df[df["debt"] > threshold]
    if since is not None:
        d = d[d["year"] >= since]
    if len(exclude):
        d = d[~d["country"].isin(list(exclude))]
    if weight == "country":
        return float(d.groupby("country")["growth"].mean().mean())
    if weight == "country-year":
        return float(d["growth"].mean())
    raise ValueError(weight)

15.4 Review

Review is where the research review of Book 7 (chapter 1) meets the codebase. A reviewer cannot review a notebook’s hidden state, only a diff: the promotion of a calculation to the library is what makes it reviewable. The chapter’s promotion checklist, run by promotion_checklist, is the minimum a reviewer asks for: the audit finds no hidden state, the notebook reruns from the top, the rerun reproduces the saved result, the library functions are tested, the environment matches the lock, the data match the snapshot.

Google’s account of its own (Potvin and Levenberg, 2016) describes a single repository serving as a common source of truth for tens of thousands of developers, made workable by systems and workflows built for its scale. This series’ own code (code/firm/) is a monorepo in miniature: a component of one book is used by the next, and a change to it is tested against every chapter that imports it.

15.5 Environments and containers

The Open Container Initiative specifies the format: an image manifest, an optional image index, a set of file-system layers and a configuration, so that tools for building, transporting and running images interoperate. A container freezes an environment; an environment lock (Book 7, chapter 29) describes one. The chapter uses the lighter of the two: capture_env lists the interpreter and every installed distribution and compare_env checks them against the repository’s requirements.txt; the data are checked against a manifest of hashes (snapshot). Where the lock is enough, a container adds nothing but weight; where system libraries or compilers matter (chapter 9’s extensions), it is the only honest record.

15.6 Reproducibility as a routine

The last step reproduces the corrected number from a clean directory: the promoted library, the data file, and a three-line script that imports the library and prints the headline. The data file is checked against the snapshot’s hash (no difference), the environment against the lock (no difference), and the script prints 2.44 in a fresh interpreter. It is the reproducible result of Book 7 (chapter 29), obtained not by care but by routine: a result is published only with the command that recomputes it from a clean checkout.

15.7 Tutorial: audit, rerun, promote, reproduce

15.8 Build: the environment audit

15.9 Exercises

15.10 Problem: Rerun From the Top

15.11 Interview questions