All books

Professional

Apps About Coach Log in Start reading

Quantitative Finance · Glossary

What is Minimum detectable effect, sequential test?

Also known as: minimum detectable effect · sequential test

Definition 21.7 Research Craft: Predictors, Backtests, Measurement, Portfolios · Chapter 21 — Live Experiments

The minimum detectable effect of a design is the smallest true effect it detects with a stated power at a stated size. A sequential test examines the data as they arrive and may stop at any look, with its size controlled over all the looks it might take.

Power to detect a saving of 0.3 basis point against the length of the experiment: the formula at each design’s standard error (lines) and the share of 100 simulated experiments that rejected (dots). The dotted line is 80%. Data: rs_abtest.
Figure 21.2. Power to detect a saving of 0.3 basis point against the length of the experiment: the formula at each design’s standard error (lines) and the share of 100 simulated experiments that rejected (dots). The dotted line is 80%. Data: rs_abtest.
Left: the share of 4 000 null experiments declared significant at some daily look, with a fixed-horizon test (grey) and the mixture sequential test (blue); the dotted line is 5%. Right: the day the sequential test stops under the real saving, with CUPED; the dashed line is the 38-day fixed horizon that has 80% power. Data: rs_abtest.peeking, stopping.
Figure 21.3. Left: the share of 4 000 null experiments declared significant at some daily look, with a fixed-horizon test (grey) and the mixture sequential test (blue); the dotted line is 5%. Right: the day the sequential test stops under the real saving, with CUPED; the dashed line is the 38-day fixed horizon that has 80% power. Data: rs_abtest.peeking, stopping.
Read in context →