---
title: "Software Engineer"
book: "The Industry: Firms, Roles and Careers"
subject: quant
language: en
chapter: 20
exercises: 8
source: https://one-course.com/books/quant/17/en/chapter/20-software-engineer
---

# Chapter 20 — Software Engineer

In February 2026 Virtu Financial, one of the largest listed electronic market makers, had about 1 027 employees in thirteen countries. Five months later Microsoft reported 77 000 people in product research and development alone, seventy-five times Virtu’s whole staff. A software engineer at a trading firm is one of a few hundred engineers at most, a few steps from the profit and loss, and on call while markets are open; at a technology company the same engineer is one of tens of thousands, many steps from revenue, with a product that runs around the clock. This chapter describes the engineering jobs at trading firms, shows how they differ from the same jobs in technology, and compares the pay the filings and the survey record for them.

| **Role cards: three engineering specialisms** |
| --- |
|  | [low-latency engineer](#def-in-software-engineer-lowlat) | [FPGA engineer](#def-in-software-engineer-fpga) | [site reliability engineer](#def-in-software-engineer-sre) |
| works on | the software path from market data to order | logic in programmable hardware on that path | the production systems and their operation |
| units | microseconds and their tails | nanoseconds and clock cycles | availability, incidents, time to recover |
| reports to | head of trading technology | head of hardware | head of infrastructure |
| codes in filings | 15-1252 | 15-1252, 17-2061 | 15-1252, 15-1299.08 |
| taught in | Book 13, ch. 1; Book 14, ch. 1 | Book 14, ch. 6–7 | Book 15, ch. 28; Book 16, ch. 21 |

## 20.1 Core, low-latency, infrastructure, hardware and network engineering

A trading firm’s engineers divide by what they build. *Core* engineers build the trading system’s logic and the tools around it: the strategy engine, order management, risk checks, the configuration that traders change, and the research engineering of chapter 19. The rest specialise by the layer they own.

**Definition 20.1 (Low-latency engineer).**

A *low-latency engineer* is a software engineer whose work is to shorten and stabilise the time between an event in the market and the firm’s response to it: the code, memory layout, operating-system settings and network stack on that path, measured in microseconds and in the tail of their distribution.

The [low-latency engineer](#def-in-software-engineer-lowlat)’s craft is the subject of Book 13: where latency comes from (chapter 1), how processors, caches and the operating system add to it, and the kernel bypass that removes the kernel from the network path (chapter 13). The work is measured, not argued: a change ships when its latency histogram, replayed on the same data, is better where it matters, usually in the tail.

**Definition 20.2 (FPGA engineer).**

An *FPGA engineer* is an engineer who designs, verifies and maintains logic for field-programmable gate arrays: feed handlers, order builders and risk checks that run in hardware, where the time from a packet’s arrival to a response is counted in clock cycles.

Book 14 covers the hardware (chapters 6 and 7) and the network around it: switches, network cards, time synchronisation, colocation. The [FPGA engineer](#def-in-software-engineer-fpga) writes a hardware description language, verifies it in simulation far more thoroughly than software is tested (a hardware bug cannot be patched in the middle of a trading day), and works with the network engineers who own the cables, switches and the colocated racks. It is the smallest of the engineering specialisms and the one most specific to trading.

Two more groups complete the map. *Network engineers* own the links from the firm to its venues: the colocated racks and cross-connects (Book 14, chapter 9), the switches and layer-1 devices (chapter 2), the clocks that timestamp every packet (chapter 4), the capture that records them (chapter 5), and, for the firms that race between cities, the long-haul fibre and wireless links (chapters 13 and 14). *Infrastructure engineers* own what research and trading run on away from the exchange: the compute clusters and their schedulers, storage for tick data, the build and release systems, and the cloud accounts that some firms now use for research (Book 15). The two groups are measured by capacity, cost and failure rather than by microseconds, but a failure in either stops trading as surely as a bug in the strategy.

**Definition 20.3 (Site reliability engineer).**

A *site reliability engineer* is a software engineer who owns the reliability of production systems: monitoring and alerting, the on-call rotation, incident response and review, capacity, and the automation that removes manual operational work.

The name comes from Google, whose book on the practice describes it as “what happens when you ask a software engineer to design an operations team” and caps operational work at half of the team’s time so that the other half goes to engineering it away. Trading firms often call the job production engineering or trading support; the tools are those of Book 15, chapter 28: service-level objectives, the on-call rotation and blameless reviews.

## 20.2 How a trading firm differs from a technology company

Four differences shape an engineer’s day.

- **Distance to the money.** A trading firm’s system earns or loses money directly, within seconds of a change. An engineer can see the result of a release in the next day’s profit and loss, and a failure can cost more in minutes than the engineer’s year’s pay. A technology company’s engineer is usually several layers from revenue.
- **The market’s clock.** A trading system must be up while its markets are open and can change while they are closed. Releases go out between sessions, on-call matters most while markets are open, and weekends are for maintenance. Crypto markets, which never close, remove that rhythm (chapter 26).
- **Size.** A trading firm of a thousand people has a few hundred engineers; the product teams and platform teams of Book 16, chapter 19, have a few people each, and one engineer may own a system end to end.
- **What cannot be bought.** A trading firm builds some of what other companies buy as a service, when latency, control or secrecy requires it (Book 16, chapter 20); and much of what it builds, it does not publish.

**Example 20.4 (An error budget on the market’s clock).**

A service-level objective of 99.9% availability measured over the whole year allows 525.6 minutes of failure. Measured over US equity trading hours alone (6.5 hours on 252 days, 1 638 hours), the same 99.9% allows 98.3 minutes, and 99.99% allows 9.8. The trading firm’s budget is smaller, and every minute of it falls when the market is moving.

## 20.3 Pay and progression compared with technology firms

The pay evidence comes from two sources that answer different questions. The labour condition applications of chapter 14 compare trading firms with banks and exchanges for the same occupation code at the same [wage level](https://one-course.com/books/quant/17/en/chapter/14-pay-levels-by-role-firm-type-and-seniority#def-in-pay-levels-by-role-firm-type-and-seniority-soc), with intervals; they do not cover technology employers here, because the one pass over the files kept only the sourced finance employers. The occupational survey compares industries, including technology, but only as published percentiles without intervals.

**As of September 2025 — What software engineers are paid: the public evidence.**

Fiscal 2025 labour condition applications coded as software developers (15-1252), median offered base: systematic funds $207 000 (67 applications, 6 employers), platforms $200 000 (35, 3), market makers $175 000 (153, 9), banks $155 688 (4 790, 8), exchanges $147 900 (296, 4). Trading firms over banks, ratio of medians 1.19 (95% interval 1.16–1.23), 1.17 at level IV. Occupational survey, May 2025, software developers’ median: web search portals and other information $213 520; software publishers $164 550; securities industry $163 290; computing infrastructure and data processing $156 950; banks $136 620; computer systems design $132 050. [Base salary](https://one-course.com/books/quant/17/en/chapter/13-how-pay-works#def-in-how-pay-works-base) only.

![Software developers’ offered base in fiscal 2025 labour condition applications by kind of employer (code 15-1252, all titles): 10th to 90th percentile (thin), interquartile range (thick), median (mark). Data: data/industry/lca_soc.csv, through in_engineer.cells.](https://one-course.com/images/onecourse/chapters/quant-17/in-software-engineer/fig-bc6d52bc51b0.svg)

***Figure 20.1.** Software developers’ offered base in fiscal 2025 labour condition applications by kind of employer (code 15-1252, all titles): 10th to 90th percentile (thin), interquartile range (thick), median (mark). Data: `data/industry/lca_soc.csv`, through `in_engineer.cells`.*

The market makers’ range is the widest: their 90th percentile, $295 000, is the highest of any kind, and their 10th is close to the banks’. Pooled and compared level by level, the trading firms’ premium can be put in a single number with an interval.

![Median offered base of software-developer filings at trading firms (market makers, systematic funds, platforms) over banks’ and exchanges’, overall and by wage level, with 95% bootstrap intervals. Missing points are comparisons with fewer than ten filings or three employers on a side. Data: data/industry/lca_kind_gap.csv, through in_engineer.kind_gaps.](https://one-course.com/images/onecourse/chapters/quant-17/in-software-engineer/fig-f10a39d1cd5b.svg)

***Figure 20.2.** Median offered base of software-developer filings at trading firms (market makers, systematic funds, platforms) over banks’ and exchanges’, overall and by [wage level](https://one-course.com/books/quant/17/en/chapter/14-pay-levels-by-role-firm-type-and-seniority#def-in-pay-levels-by-role-firm-type-and-seniority-soc), with 95% bootstrap intervals. Missing points are comparisons with fewer than ten filings or three employers on a side. Data: `data/industry/lca_kind_gap.csv`, through `in_engineer.kind_gaps`.*

The trading firms’ premium over banks is about a fifth at every level where it can be measured in fiscal 2025, and it narrowed from about a third in fiscal 2021 as the banks’ offers rose faster ([Figure 20.2](#fig-in-software-engineer-ratio)). Over exchanges it is larger. [Base salary](https://one-course.com/books/quant/17/en/chapter/13-how-pay-works#def-in-how-pay-works-base) understates the difference: chapter 14’s implied ratios of total pay to base were several times higher at market makers than at banks.

![Software developers’ annual wages by industry at five percentiles, May 2025 occupational survey. The securities industry pays like the software publishers at the median and above them at both ends; web search and information pays more at every percentile. Data: data/industry/oews_roles.csv, through in_engineer.survey.](https://one-course.com/images/onecourse/chapters/quant-17/in-software-engineer/fig-8d398f9ae4d2.svg)

***Figure 20.3.** Software developers’ annual wages by industry at five percentiles, May 2025 occupational survey. The securities industry pays like the software publishers at the median and above them at both ends; web search and information pays more at every percentile. Data: `data/industry/oews_roles.csv`, through `in_engineer.survey`.*

The survey places the securities industry beside the software publishers, not above them: the ratio of their percentiles is 1.27 at the 10th, 0.99 at the median and 1.12 at the 90th ([Figure 20.3](#fig-in-software-engineer-survey)). The industry that contains the largest internet companies pays more at every percentile: the securities industry’s median is 0.77 of theirs. Two cautions apply. The securities industry in the survey includes brokers and asset managers as well as the trading firms, whose filings sit above the survey’s median; and the survey measures wages, not the equity awards that make up much of a technology engineer’s pay (chapter 13). The honest conclusion is narrow: for base pay, a trading firm competes with the technology industry’s best-paying part, not above it.

Progression differs with size. An employer with tens of thousands of engineers needs a ladder of levels to compare them with one another; a firm with a few hundred can judge each engineer by the systems they own, and its pay follows the firm’s year (chapter 13). Chapter 28 follows the paths.

## 20.4 Public engineering: blogs, talks and open source

Most of a trading firm’s code is private, but some firms publish a great deal, and what they publish is the best public evidence of what their engineers do.

**As of September 2026 — Trading firms’ public engineering.**

Jane Street writes “everything that we can in OCaml”, which it calls its “primary development platform”, publishes open-source OCaml libraries and tools (Base, the Dune build system, the OxCaml language extensions, and Hardcaml, “A language for designing and simulating hardware in OCaml”), has funded work on the OCaml compiler and its package manager, and runs a public engineering blog; its GitHub organisation listed 411 public repositories on 29 September 2026. Jump Crypto’s Firedancer, a validator client for the Solana blockchain, is open source; its README says that “The concurrency model draws from experience in the low latency trading space”.

For a candidate, public engineering answers questions that interviews do not: which language the firm writes in, how it tests, what it considers a hard problem. For the firm it is recruiting, and a way to have outside engineers improve the tools it depends on. What is never published is the trading logic, and the engineer who joins accepts that most of their work will stay private (chapter 28 discusses what a candidate can show from such a job).

## 20.5 Tutorial: trading firms, banks and technology

**Goal.** Compare software engineers’ pay at trading firms with banks, exchanges and technology industries, with the uncertainty each source allows. **End state:** Figures [20.1](#fig-in-software-engineer-kinds), [20.2](#fig-in-software-engineer-ratio) and [20.3](#fig-in-software-engineer-survey).

1. **Cells by kind.** Chapter 19’s `lca_soc.csv` for code 15-1252 by employer kind, at all levels.
2. **Ratio of medians.** `in_lca_kind_gap_derive.py` pools the trading firms and applies `firm.roles.median_gap` against banks and against exchanges, overall and by level, from chapter 14’s cache, keeping a comparison only when each side has ten filings from three employers.
3. **Percentile ratios.** `percentile_ratio(a, b)` divides two published distributions percentile by percentile and returns nothing where a value is marked ([Listing 20.1](#lst-in-software-engineer-ratio)). `def percentile_ratio (a, b, keys=(" p10 " , " p25 " , " p50 " , " p75 " , " p90 " )): """Ratio of two published wage distributions at each percentile; a value that is not a number (a survey's top-code mark or a missing cell) gives None at that percentile.""" out = {} for k in keys: try : out[k] = float (a[k]) / float (b[k]) except (TypeError, ValueError, ZeroDivisionError): out[k] = None return out` **Listing 20.1.** Two published wage distributions compared percentile by percentile. code/firm/roles/firm_roles.py
4. **Scale.** The headcounts from the filings give the hook’s ratios.

For fiscal 2025, trading firms over banks: 1.19 (1.16–1.23) overall, 1.25 (1.14–1.38) at level II, 1.18 (1.12–1.27) at level III, 1.17 (1.11–1.29) at level IV; level I’s interval (0.78–1.43) includes one. Over exchanges: 1.25 overall and 1.30 at level IV. Securities over software publishers in the survey: 1.27, 1.07, 0.99, 1.00, 1.12 from the 10th to the 90th percentile.

**What to change next.** Add large technology employers to the filings with a sourced list, which needs a second pass over the files; weight the comparison by level so that the overall ratio is not a mix effect (chapter 19); add bonus and equity from the proxy statements’ [median-employee pay](https://one-course.com/books/quant/17/en/chapter/14-pay-levels-by-role-firm-type-and-seniority#def-in-pay-levels-by-role-firm-type-and-seniority-median) (chapter 14).

## 20.6 Build: engineering cards and percentile comparison

**Purpose.** Add the three engineering specialisms to the role registry and a comparison of published distributions.

**Interface.** `firm.roles`: the cards `low-latency engineer`, `FPGA engineer`, `site reliability engineer`; `percentile_ratio(a, b, keys) -> {key: ratio or None}`; with `median_gap` (chapter 19) for data with values.

**Rules.** A marked or missing percentile gives no ratio, never a guess; ratios of published percentiles carry no interval and the chapter says so.

**Acceptance tests.** `code/firm/roles/tests/`: known ratios; a top-code mark gives `None`.

**Stretch.** Bounds for a ratio when one side is top-coded; interpolate a distribution from its five percentiles and compare shares above a threshold.

Sources and further reading

- Virtu Financial, Form 10-K for 2025; Flow Traders, Annual Report 2025; Microsoft, Form 10-K for fiscal 2026.
- Beyer, B., C. Jones, J. Petoff and N. R. Murphy (eds., 2016), *Site Reliability Engineering* , O’Reilly.
- Jane Street technology page and GitHub organisation; Firedancer repository.
- Chapter 14’s tables from the Department of Labor’s LCA files; BLS occupational survey, May 2025.

## 20.7 Exercises

**Exercise 20.1 ★.**

How many times Virtu’s staff is Microsoft’s product research and development staff, and its whole staff?

**Solution of Exercise 20.1.**

$77\,000/1\,027=75$ times for research and development; $223\,000/1\,027=217$ times for the whole staff.

**Exercise 20.2 ★.**

How many minutes of failure a year does a 99.95% objective allow over US equity trading hours?

**Solution of Exercise 20.2.**

$0.0005\times1\,638\times60=49.1$ minutes a year.

**Exercise 20.3 ★.**

Flow Traders’ 2025 annual report gives its 2024 year-end staff as 609 FTEs in one section and 625 in another. What should a reader do with such a difference?

**Solution of Exercise 20.3.**

Note both, find the definitions (year-end or average, permanent only or all workers, the date of the count) and use the figure whose definition matches the question, citing the page; never average them or pick the convenient one.

**Exercise 20.4 ★★.**

Why is the trading firms’ premium over exchanges larger at level II than at level IV?

**Solution of Exercise 20.4.**

The trading firms’ medians hardly change with the stated level ($175 000 at level II, $195 000 at level IV), while the exchanges’ rise from $105 300 to $149 800: offers far above the [prevailing wage](https://one-course.com/books/quant/17/en/chapter/14-pay-levels-by-role-firm-type-and-seniority#def-in-pay-levels-by-role-firm-type-and-seniority-lca) are not ordered by level (chapter 14), so the premium is largest where the comparison group is paid least.

**Exercise 20.5 ★★.**

The ratio of trading firms’ to banks’ median fell from 1.32 in fiscal 2021 to 1.19 in fiscal 2025. Using the medians of the derived table ($165 000 and $125 000 in 2021; $185 000 and $155 688 in 2025), which side moved more?

**Solution of Exercise 20.5.**

The trading firms’ median rose 12.1% ($185\,000/165\,000$), the banks’ 24.6% ($155\,688/125\,000$): the banks moved more.

**Exercise 20.6 ★★.**

An engineer choosing between a trading firm and a large technology company compares base salaries only. What does that leave out on each side?

**Solution of Exercise 20.6.**

At the trading firm, the bonus, which chapter 14’s implied ratios show can be several times base, and its variation with the firm’s year. At the technology company, the equity awards and their vesting, and the value of the stock. On both, deferral and [forfeiture](https://one-course.com/books/quant/17/en/chapter/13-how-pay-works#def-in-how-pay-works-rsu) (chapter 13), and hours and on-call (chapter 26).

**Exercise 20.7 ★★★.**

*Coding.* With `percentile_ratio`, compare the securities industry’s software developers with each of the other five industries of the survey table. At how many of the 25 percentile comparisons does the securities industry pay more?

**Solution of Exercise 20.7.**

18 of 25: at every percentile against banks, computing infrastructure and computer systems design; at the 10th, 25th and 90th against software publishers; at none against web search and information.

**Exercise 20.8 ★★★.**

*Find the flaw.* “The survey shows the securities industry pays software developers no more than software publishers do, so trading firms pay engineers like the technology industry.”

**Solution of Exercise 20.8.**

The survey’s securities industry includes brokers, asset managers and banks’ broker-dealers as well as trading firms, and the trading firms’ filings sit above its median; the survey measures wages without bonus or equity; and the technology side is not one industry: web search and information pays more than the securities industry at every percentile.

## 20.8 Problem: Trading Firm or Technology Company?

**Problem 20.1.**

Weekend problem — trading firm or technology company?

A software engineer with five years at a large technology company has an offer from a market maker. She wants to know how the work and the pay differ.

**Part I — The jobs.**

1. Define the [low-latency engineer](#def-in-software-engineer-lowlat) , the [FPGA engineer](#def-in-software-engineer-fpga) and the [site reliability engineer](#def-in-software-engineer-sre) .
2. What do core engineers build?
3. Where does the [site reliability engineer](#def-in-software-engineer-sre) ’s name come from, and what rule on operational work goes with it?
4. What makes the [FPGA engineer](#def-in-software-engineer-fpga) ’s testing stricter than a software engineer’s?
5. Which books of the series teach each specialism?

**Part II — The firm.**

6. Compare the scale of the two employers with the hook’s numbers.
7. State the four differences between a trading firm and a technology company.
8. Compute the error budget of a 99.9% objective over a year and over trading hours.
9. What does the market’s clock do to releases and on-call?
10. What do trading firms publish about their engineering, and what not?

**Part III — The pay.**

11. What can each source (filings, survey) compare, and what not?
12. Give the trading firms’ medians by kind and the banks’.
13. Give the ratios over banks by level, with intervals.
14. Compare the securities industry with software publishers and with web search in the survey.
15. What does [base salary](https://one-course.com/books/quant/17/en/chapter/13-how-pay-works#def-in-how-pay-works-base) leave out on each side?

**Part IV — The verdict.**

16. State the *named result* : the ratio of median offered base at trading firms to banks for software developers at each [wage level](https://one-course.com/books/quant/17/en/chapter/14-pay-levels-by-role-firm-type-and-seniority#def-in-pay-levels-by-role-firm-type-and-seniority-soc) , with bootstrap intervals, and the survey’s comparison with technology industries.
17. Why is the comparison with banks sharper than the comparison with technology?
18. How does progression differ?
19. What should she ask the market maker about on-call?
20. In two sentences, answer her.

**Solution of Problem 20.1.**

1. As in the chapter’s three definitions: the software path from market to order in microseconds; logic in programmable hardware in clock cycles; reliability and operation of production systems.
2. The trading system’s logic and tools: strategy engine, order management, risk checks, configuration, research engineering.
3. Google: an operations team designed by software engineers, with operational work capped at half the team’s time.
4. A hardware bug cannot be patched during the trading day, so verification in simulation must be exhaustive.
5. Book 13 (low latency), Book 14 (hardware and network), Book 15, chapter 28, and Book 16 (operations and organisation).
6. Virtu had about 1 027 employees; Microsoft 223 000, of whom 77 000 in research and development.
7. Distance to the money; the market’s clock; size; what cannot be bought.
8. 525.6 minutes over a year; 98.3 minutes over trading hours.
9. Releases between sessions, on-call while markets are open, maintenance at weekends.
10. Languages, libraries, tools, blogs and talks; never trading logic.
11. Filings: kinds of finance employer, same code and level, with intervals, but no technology employers. Survey: industries including technology, but only published percentiles without intervals.
12. Systematic funds $207 000, platforms $200 000, market makers $175 000; banks $155 688.
13. 1.14 (0.78–1.43) at level I, 1.25 (1.14–1.38) at II, 1.18 (1.12–1.27) at III, 1.17 (1.11–1.29) at IV.
14. Against software publishers: 1.27, 1.07, 0.99, 1.00, 1.12 from the 10th to the 90th percentile; against web search: below at every percentile (0.77 at the median).
15. Bonus at the trading firm; equity at the technology company.
16. Trading firms over banks about 1.17–1.25 by level (overall 1.19, 1.16–1.23); the securities industry equal to software publishers at the median and below web search and information at every percentile.
17. The filings compare the same code and level with resampling intervals; the survey mixes firm types within an industry and has no intervals.
18. A ladder of levels at scale against scope by owned systems and pay by the firm’s year.
19. The rotation, the hours it covers, how incidents are reviewed, and how often she would be woken.
20. The market maker’s base pay will be above a bank’s and about level with the best-paying technology employers, with a bonus instead of equity. The work is closer to the money and runs on the market’s clock, with a smaller team and more ownership.

## 20.9 Interview questions

**Interview question 20.1 ★ developer.**

What is tail latency, and why does a trading firm care more about the 99.9th percentile than the mean?

**Solution of Interview question 20.1.**

The latency that a small share of events exceed. A trading firm’s worst responses come in the busiest moments, when being late costs most, and a mean hides them.

*What the interviewer is looking for: percentiles, and why the busy moments are the tail.*

**Interview question 20.2 ★ developer.**

A service must be up 99.9% of trading hours. How much downtime is that a year, and how would you spend it?

**Solution of Interview question 20.2.**

$0.001\times1\,638\times60=98.3$ minutes. Spend it on planned risk (releases, failover tests) rather than let incidents consume it, and stop releasing when it is spent.

*What the interviewer is looking for: the arithmetic and the error-budget policy.*

**Interview question 20.3 ★★ developer.**

How does kernel bypass reduce latency, and what do you give up?

**Solution of Interview question 20.3.**

The application reads packets from the network card directly, avoiding system calls, copies and interrupts. The cost is the kernel’s services: the network stack, isolation and standard tooling, so monitoring and security must be rebuilt.

*What the interviewer is looking for: what the kernel did that now must be done elsewhere.*

**Interview question 20.4 ★★ developer, trader.**

Your system sent duplicate orders for two seconds at the open. Walk through the first hour after.

**Solution of Interview question 20.4.**

Stop the cause (kill switch or disable the strategy), establish the position and exposure and hand them to the traders, inform compliance and the venues as required, preserve logs, find the root cause, and hold a blameless review with fixes that prevent the class of error.

*What the interviewer is looking for: contain first, position second, root cause and review after.*

**Interview question 20.5 ★★ developer.**

When would you put a risk check in an FPGA rather than in software?

**Solution of Interview question 20.5.**

When the check must not add latency to an order path already in hardware, or must act on every order at line rate; the price is harder change and verification, so the check must be simple and stable.

*What the interviewer is looking for: latency on the path against flexibility.*

**Interview question 20.6 ★★★ developer.**

Design the release process for a trading system that must not change while the market is open but must ship every day.

**Solution of Interview question 20.6.**

Build and test every day; deploy between sessions with an automatic check against replayed market data; feature flags so that code ships dark and is enabled by configuration; a fast rollback; and a freeze while markets are open except for fixes approved under an incident.

*What the interviewer is looking for: separating deployment from activation, and rollback.*