On this page

Statistics for A/B Testing

A/B testing is the scientific method applied to product development. You have a hypothesis ("changing the button color increases clicks"), you run a controlled experiment, and you use statistics to determine whether the observed difference is real or just noise. Getting this wrong means either shipping changes that do not work (false positive) or killing changes that actually help (false negative). This page covers the statistics you need to run experiments correctly.

Each concept follows the same pattern: intuition, then the formula, then a worked numeric example, then how it shows up in a real experimentation platform. All Python uses numpy and scipy only, uses fixed random seeds, and the outputs shown in comments come from running it. The probability background (Bayes, distributions, tail bounds) is on Probability for System Design.

flowchart LR
    D["Describe the data: percentiles, spread"] --> S["Sampling distributions and the CLT"]
    S --> CI["Confidence intervals and the bootstrap"]
    CI --> HT["Hypothesis tests, power, sample size"]
    HT --> P["Pitfalls: peeking, multiple tests, SRM, interference"]
    P --> VR["Variance reduction: CUPED"]
    VR --> RM["Ratio metrics: delta method"]
    HT --> B["Bayesian A/B and bandits"]

Descriptive Statistics: Summarising What You Measured

Intuition

Before testing anything, you need numbers that describe a pile of measurements. The centre (mean, median) says what is typical; the spread (standard deviation, interquartile range) says how much individual values vary; percentiles say what fraction of values fall below a threshold.

The -th percentile is the value below which of observations fall. The median is p50. The in the sample variance (Bessel's correction) compensates for measuring spread around the sample's own mean, which is always a little closer to the data than the true mean is.

Why Percentiles Beat Means for Latency

Latency distributions are bounded below by zero and have a long right tail (often roughly log-normal). In such distributions the mean is dragged upward by a few slow requests, so it describes neither the typical user nor the unlucky one.

Worked example. Ten requests take (in ms): 12, 13, 13, 14, 15, 15, 16, 18, 20, 900. The mean is 103.6 ms, a value no request actually took. The median is 15 ms. The one 900 ms request (a GC pause, a cold cache, a retry) is invisible in the median and dominates the mean. Neither number alone tells you that 10% of users waited almost a second: only a high percentile does.

Rules that follow: report p50, p95 and p99 (the mean is still right for capacity, since total work is mean times count); never average percentiles across hosts, merge histograms (HdrHistogram, t-digest) instead; and remember a p99 from 200 requests rests on 2 data points.

import numpy as np

rng = np.random.default_rng(42)
# 10 hosts; one is unhealthy (3x slower). Log-normal latency, median 20 ms.
hosts = [rng.lognormal(np.log(20 * (3 if h == 0 else 1)), 0.6, 10_000) for h in range(10)]
fleet = np.concatenate(hosts)

print(f"mean  = {fleet.mean():6.1f} ms")
print(f"p50   = {np.percentile(fleet, 50):6.1f} ms")
print(f"p99   = {np.percentile(fleet, 99):6.1f} ms   (true fleet p99)")
avg_of_p99 = np.mean([np.percentile(h, 99) for h in hosts])
print(f"mean of per-host p99s = {avg_of_p99:6.1f} ms   (wrong: not a percentile of anything)")
# mean  =   28.7 ms
# p50   =   21.5 ms
# p99   =  132.7 ms   (true fleet p99)
# mean of per-host p99s =   98.0 ms   (wrong: not a percentile of anything)

The averaged p99 (98 ms) understates the true fleet p99 (133 ms) by a quarter because the unhealthy host's slow requests are most of the fleet's tail but only one tenth of the average. Averaging percentiles hides exactly the outlier hosts you want to find.


Sampling Distributions and the Central Limit Theorem

Intuition

Run the same experiment twice and you get two different conversion rates, purely from which users happened to land in it. The sampling distribution is the distribution of a statistic (the mean, a conversion rate, a difference) across all the samples you could have drawn. Every test and interval in this page is a statement about that distribution.

The central limit theorem says that for large , the sample mean of independent observations is approximately normal, whatever the shape of the original data, with

For a conversion rate (a mean of 0/1 values), , so .

Worked Example

A 5% conversion rate measured on 10,000 users has

So a rerun would typically land within about percentage points ( SE) of the truth. A difference of 0.5 points between two such groups is right at the edge of noise, which is why the first example below is not significant.

How Large Is "Large"?

The CLT kicks in after a few dozen observations for well-behaved data but can need tens of thousands for revenue per user, where a handful of users spend thousands. Winsorize (cap at, say, the 99.9th percentile) to cut variance at the price of a little bias, and check with the bootstrap when unsure.

Engineering Application

The law is the budget equation of experimentation: to halve the noise you need four times the users. Every variance reduction technique (CUPED, winsorizing, better metrics) is really a way of buying the equivalent of more users for free.


Confidence Intervals

Intuition

A point estimate ("lift is +10%") is almost certainly wrong in the decimals. A confidence interval gives a range that would contain the true value in 95% of repeated experiments, if the method's assumptions hold.

For a difference of two proportions:

The precise meaning is about the procedure: if you repeated the experiment many times, 95% of the intervals constructed this way would contain the true difference. It is not "there is a 95% probability the true value is in this particular interval" (that is a Bayesian credible interval, covered later), although with flat priors the two often coincide numerically.

Why Confidence Intervals Matter More Than p-values

A p-value tells you IF there is a difference. A confidence interval tells you HOW BIG the difference is. Product decisions need both.

graph TD
    subgraph "Three Experiment Results"
        A["Test A<br/>CI: [+0.5%, +4.5%]<br/>p = 0.014<br/>Significant, meaningful"]
        B["Test B<br/>CI: [+0.01%, +0.09%]<br/>p = 0.02<br/>Significant, but tiny effect"]
        C["Test C<br/>CI: [-1.0%, +5.0%]<br/>p = 0.19<br/>Not significant, wide CI"]
    end

    A --> SHIP["Ship it"]
    B --> SKIP["Skip — not worth the complexity"]
    C --> MORE["Need more data or bigger change"]

    style A fill:#059669,color:#fff
    style B fill:#d97706,color:#fff
    style C fill:#dc2626,color:#fff

Interpreting Confidence Intervals

CI RangeInterpretationDecision
[+2%, +8%]Entire range is positiveShip — effect is reliably positive
[+0.1%, +6%]Positive but lower bound near zeroShip with caution — might be small
[-1%, +5%]Range includes zeroNot significant — need more data
[-3%, -0.5%]Entire range is negativeTreatment is worse — do not ship
def interpret_ci(ci_lower: float, ci_upper: float) -> str:
    """Interpret a confidence interval for product decisions."""
    if ci_lower > 0 and ci_upper > 0:
        if ci_lower > 0.005:  # Lower bound > 0.5%
            return "SHIP: Significant positive effect"
        return "SHIP WITH CAUTION: Positive but potentially small"
    elif ci_lower < 0 and ci_upper > 0:
        return "INCONCLUSIVE: CI includes zero, need more data"
    elif ci_lower < 0 and ci_upper < 0:
        return "DO NOT SHIP: Treatment is worse"
    return "EDGE CASE: Review manually"

print(interpret_ci(0.02, 0.08))
print(interpret_ci(-0.01, 0.05))
# SHIP: Significant positive effect
# INCONCLUSIVE: CI includes zero, need more data

"Inconclusive" is not "no effect": is consistent with a 4% win. Only when the whole interval sits inside a negligible range can you claim "no meaningful effect" (equivalence testing).

The Bootstrap: Confidence Intervals for Anything

Intuition. The formulas above work for means and proportions. For a p99, a median, a trimmed mean or a ratio, there is often no tidy formula. The bootstrap fakes "rerunning the experiment" by resampling your own data with replacement: each resample is a plausible alternative dataset, and the spread of the statistic across thousands of resamples estimates its sampling distribution.

Algorithm (percentile bootstrap).

  1. Resample values from group A and from group B, with replacement.
  2. Compute the statistic (for example, ).
  3. Repeat times (2,000–10,000).
  4. The 2.5th and 97.5th percentiles of the results are the 95% CI.

Worked example. A new serialization library is deployed to half the traffic. Did p99 latency change?

import numpy as np

rng = np.random.default_rng(7)
control = rng.lognormal(np.log(40), 0.5, 20_000)     # median ~40 ms
treatment = rng.lognormal(np.log(40), 0.45, 20_000)  # same median, thinner tail

def p99(x):
    return np.percentile(x, 99)

observed = p99(treatment) - p99(control)
boot = np.empty(2_000)
for i in range(boot.size):
    c = rng.choice(control, control.size, replace=True)
    t = rng.choice(treatment, treatment.size, replace=True)
    boot[i] = p99(t) - p99(c)

lo, hi = np.percentile(boot, [2.5, 97.5])
print(f"p99 control={p99(control):.1f} ms  treatment={p99(treatment):.1f} ms")
print(f"difference={observed:+.1f} ms  95% bootstrap CI=[{lo:+.1f}, {hi:+.1f}] ms")
# p99 control=126.4 ms  treatment=111.0 ms
# difference=-15.3 ms  95% bootstrap CI=[-18.9, -10.6] ms

The medians are identical, so a test on medians or a look at the dashboards' p50 would see nothing. The bootstrap CI on the p99 difference excludes zero: the change genuinely trimmed the tail.

Caveats. Resample the unit that was randomized: if users have many requests, resample users, not requests, or the CI will be far too narrow. Extreme quantiles from small samples are unreliable (a p99.9 from 5,000 points rests on 5 values).


Hypothesis Testing

The Framework

Every A/B test is a hypothesis test with four components:

graph TD
    H0["Null Hypothesis (H₀)<br/>'No difference between<br/>control and treatment'"]
    H1["Alternative Hypothesis (H₁)<br/>'There IS a difference<br/>(treatment is better/worse)'"]
    DATA["Collect Data<br/>Run experiment for<br/>predetermined duration"]
    STAT["Compute Test Statistic<br/>(Z-score, t-statistic)"]
    PV["Calculate p-value<br/>'How likely is this data<br/>if H₀ is true?'"]
    DEC{"p < α?"}
    REJ["Reject H₀<br/>'Statistically significant'"]
    FAIL["Fail to Reject H₀<br/>'Not enough evidence'"]

    H0 & H1 --> DATA --> STAT --> PV --> DEC
    DEC -->|"Yes"| REJ
    DEC -->|"No"| FAIL

    style REJ fill:#059669,color:#fff
    style FAIL fill:#d97706,color:#fff
TermDefinitionTypical Value
Significance level (alpha)P(reject H0 when H0 is true) — false positive rate0.05 (5%)
Power (1 - beta)P(reject H0 when H0 is false) — true positive rate0.80 (80%)
p-valueProbability of observing data this extreme if H0 is trueComputed from data
Effect sizeMagnitude of the difference you want to detectDomain-specific
MDEMinimum Detectable Effect — smallest meaningful differencee.g., 2% lift

The logic is proof by contradiction with noise. Assume there is no effect. Compute how surprising the observed data would be under that assumption. If it would be very surprising (p-value below ), reject the assumption.

Computing a p-value: Two-Proportion Z-Test

The most common A/B test compares conversion rates between two groups.

Under both groups share one rate, so the standard error uses the pooled rate . The confidence interval, which is not computed under , uses the unpooled standard error from the previous section. That is why the code below has two different SE lines.

import math
from scipy import stats

def two_proportion_z_test(
    conversions_a: int, visitors_a: int,
    conversions_b: int, visitors_b: int
) -> dict:
    """
    Two-proportion Z-test for A/B test significance.
    Returns z-statistic, p-value, and confidence interval for the difference.
    """
    p_a = conversions_a / visitors_a
    p_b = conversions_b / visitors_b
    p_pooled = (conversions_a + conversions_b) / (visitors_a + visitors_b)

    se = math.sqrt(p_pooled * (1 - p_pooled) * (1/visitors_a + 1/visitors_b))
    z = (p_b - p_a) / se
    p_value = 2 * (1 - stats.norm.cdf(abs(z)))  # Two-tailed

    # 95% confidence interval for the difference
    se_diff = math.sqrt(p_a * (1 - p_a) / visitors_a + p_b * (1 - p_b) / visitors_b)
    ci_lower = (p_b - p_a) - 1.96 * se_diff
    ci_upper = (p_b - p_a) + 1.96 * se_diff

    return {
        "control_rate": p_a,
        "treatment_rate": p_b,
        "relative_lift": (p_b - p_a) / p_a * 100,
        "z_statistic": z,
        "p_value": p_value,
        "significant": p_value < 0.05,
        "ci_95": (ci_lower, ci_upper),
    }

# Example: checkout page A/B test
result = two_proportion_z_test(
    conversions_a=500, visitors_a=10000,  # 5.0% conversion
    conversions_b=550, visitors_b=10000,  # 5.5% conversion
)
print(f"Lift: {result['relative_lift']:.1f}%")
print(f"p-value: {result['p_value']:.4f}")
print(f"Significant: {result['significant']}")
print(f"95% CI: [{result['ci_95'][0]:.4f}, {result['ci_95'][1]:.4f}]")
# Lift: 10.0%
# p-value: 0.1129
# Significant: False
# 95% CI: [-0.0012, 0.0112]

A 10% relative lift sounds big, but with 10,000 users per group it is not significant: the CI runs from to percentage points. The next sections explain why this test was doomed from the start. It was underpowered: about 31,000 users per group were needed to reliably detect a 10% lift at this baseline.

What a p-value Is NOT

  • It is NOT the probability that H0 is true
  • It is NOT the probability the result is due to chance
  • It IS the probability of seeing data this extreme (or more extreme) IF H0 were true
  • A p-value of 0.03 means: "If there were truly no difference, we would see data this extreme only 3% of the time"

Nor is it the effect size ( on 50 million users can be a 0.01% lift), the replication probability (a result at exactly replicates at only about half the time), or evidence for when large ( means "no evidence of a difference"). And it is invalid if you stopped the first time it dipped below 0.05.

Why does "probability is true" differ from the p-value? Bayes, exactly as with alert base rates on the probability page. If only 10% of the ideas you test actually work, and your tests have 80% power at , then among significant results the fraction that are false is . More than a third of your "wins" are noise, even with perfect methodology.

Type I and Type II Errors

true (no real effect) false (real effect)
Reject ("ship it")Type I error (false positive), probability Correct, probability (power)
Fail to reject ("no ship")Correct, probability Type II error (false negative), probability

Intuition. is the rate at which you ship things that do nothing. is the rate at which you throw away things that work. You can shrink both only by collecting more data (or reducing variance). For a fixed sample, lowering raises .

The costs are asymmetric: a false positive on a ranking change that actually hurts revenue by 0.5% can cost millions, so mature platforms use stricter thresholds for risky launches and one-sided harm tests on guardrails.

Power and the Minimum Detectable Effect

Power is the probability of detecting an effect of a given size, if it exists. It depends on four things: effect size, noise (), sample size, and .

Intuition. Picture two bell curves for the test statistic: one centred at 0 (no effect) and one centred at the true effect divided by its SE. The rejection threshold sits at on the first curve. Power is the area of the second curve beyond that threshold. To get 80% power, the second curve's centre must sit standard errors beyond the threshold, so the effect must be at least standard errors.

That gives the MDE for a fixed sample:

Worked example. With and per group: , so the MDE is percentage points, a 17% relative lift. The checkout test above was hunting for a 10% lift with a test that could only reliably see 17%. Its power for a true 10% lift was only about 35%.

import math
from scipy.stats import norm

def power_two_proportions(p1: float, p2: float, n: int, alpha: float = 0.05) -> float:
    """Approximate power of a two-sided two-proportion z-test with n users per group."""
    se = math.sqrt(p1 * (1 - p1) / n + p2 * (1 - p2) / n)
    z_crit = norm.ppf(1 - alpha / 2)
    return norm.cdf(abs(p2 - p1) / se - z_crit)

def mde(p: float, n: int, alpha: float = 0.05, power: float = 0.8) -> float:
    """Absolute minimum detectable effect for a conversion rate p with n users per group."""
    return (norm.ppf(1 - alpha / 2) + norm.ppf(power)) * math.sqrt(2 * p * (1 - p) / n)

print(f"power for 5.0% -> 5.5% at n=10,000: {power_two_proportions(0.05, 0.055, 10_000):.0%}")
print(f"power for 5.0% -> 5.5% at n=31,231: {power_two_proportions(0.05, 0.055, 31_231):.0%}")
print(f"MDE at n=10,000: {mde(0.05, 10_000):.4f} absolute = {mde(0.05, 10_000) / 0.05:.0%} relative")
# power for 5.0% -> 5.5% at n=10,000: 35%
# power for 5.0% -> 5.5% at n=31,231: 80%
# MDE at n=10,000: 0.0086 absolute = 17% relative

Underpowered tests lie twice

An underpowered test usually misses real effects. Worse, when it does reach significance, the estimated effect is inflated, because only the lucky overestimates cross the threshold (the "winner's curse", or Type M error). A 35%-power test that comes out significant typically overstates the true lift substantially. Plan for 80% power or more.


Sample Size Calculation

Why You Calculate Sample Size Before Running the Test

Running a test without a predetermined sample size leads to two problems:

  1. Underpowered tests: Not enough data to detect real effects, so you miss real improvements.
  2. Peeking: Checking results early inflates false positive rates.

The Formula

For a two-proportion test with significance level alpha and power (1 - beta):

n = (Z_{α/2} + Z_β)² × (p₁(1-p₁) + p₂(1-p₂)) / (p₂ - p₁)²

Where:
  Z_{α/2} = 1.96 for α = 0.05
  Z_β     = 0.84 for power = 0.80
  p₁      = baseline conversion rate
  p₂      = expected new conversion rate (p₁ + MDE)

In math notation, per group:

Worked by Hand

Baseline 5%, target a 10% relative lift (so ), , power 80%:

  1. Variance term:
  2. Squared difference:
  3. per group

So about 62,500 users in total. At 10,000 eligible visitors per day that is 6.2 days, which you round up to a whole week (7 days) so every weekday is represented equally, and often to two weeks to let novelty effects decay.

import math
from scipy.stats import norm

def sample_size(
    baseline_rate: float,
    mde_relative: float,
    alpha: float = 0.05,
    power: float = 0.80,
) -> int:
    """
    Calculate required sample size per group for a two-proportion Z-test.

    Args:
        baseline_rate: Current conversion rate (e.g., 0.05 for 5%)
        mde_relative: Minimum detectable relative lift (e.g., 0.10 for 10% lift)
        alpha: Significance level
        power: Statistical power
    """
    p1 = baseline_rate
    p2 = baseline_rate * (1 + mde_relative)

    z_alpha = norm.ppf(1 - alpha / 2)
    z_beta = norm.ppf(power)

    n = ((z_alpha + z_beta) ** 2 * (p1 * (1 - p1) + p2 * (1 - p2))) / (p2 - p1) ** 2

    return math.ceil(n)

# How many users per group to detect a 10% relative lift on 5% conversion?
n = sample_size(baseline_rate=0.05, mde_relative=0.10)
print(f"Sample size per group: {n:,}")
# Sample size per group: 31,231
# ~31,234 per group → ~62,468 total
# At 10,000 visitors/day → ~6.2 days

(The function prints 31,231; the older "~31,234" comment is the same figure rounded slightly differently.)

Sample Size Quick Reference

Correction (2026-11-09)

Two rows were off by more than rounding: 5% baseline with 5% MDE needs about 122,000 per group (previously "~125,000"), and 10% baseline with 20% MDE needs about 3,800 (previously "~3,700"). Values below are from sample_size() at , power 80%.

Baseline RateMDE (Relative)Sample Per GroupTotal Sample
5%5%~122,000~244,000
5%10%~31,000~62,000
5%20%~8,000~16,000
10%5%~57,000~114,000
10%10%~14,500~29,000
10%20%~3,800~7,700
20%5%~25,000~50,000
20%10%~6,500~13,000

The Smaller the Effect, the More Data You Need

Detecting a 1% relative lift requires roughly 100x more data than detecting a 10% lift. If your traffic is limited, focus on changes that produce large effects — not micro-optimizations.

(Exactly: about 3.0 million per group for a 1% lift on a 5% baseline, 96 times the 10% figure. Sample size scales with .)

Continuous Metrics

For a mean (revenue per user, session length) with standard deviation and absolute MDE :

Worked example. Revenue per user has mean USD 10 and standard deviation USD 50 (most users spend nothing, a few spend a lot). To detect a USD 1 change (10%): per group. Cut in half with winsorizing or CUPED and drops by a factor of four, to under 10,000. Variance is the lever.


Choosing a Test

Intuition

Different tests ask different questions and assume different things about the data. Choosing is mostly about what kind of metric you have and what claim you want to make.

flowchart TD
    Q["What is the metric?"] --> BIN["Binary: converted or not"]
    Q --> CONT["Continuous: revenue, time, count"]
    Q --> RATIO["Ratio of sums: CTR per view, revenue per session"]
    BIN --> Z["Two-proportion z-test, or chi-square"]
    BIN --> SMALL["Very small counts: Fisher exact test"]
    CONT --> MEANS["Care about the mean?"]
    MEANS -->|"yes"| WELCH["Welch t-test, CLT makes it robust at large n"]
    MEANS -->|"no, any shift"| MW["Mann-Whitney U, rank based"]
    CONT --> QUANT["Care about p95 or p99?"]
    QUANT --> BOOT["Bootstrap the quantile difference"]
    RATIO --> DELTA["Delta method or bootstrap over users"]
TestUse whenAssumesTests
Two-proportion z-testBinary metric, large in each cellDifference in rates
Chi-squareBinary or categorical, large ; also SRM checksExpected counts Independence of variant and outcome
Fisher exactBinary with tiny countsNothing asymptoticSame as chi-square, exact
Student t-testMeans, equal variancesNormal-ish means, equal Difference in means
Welch t-testMeans, the safe defaultNormal-ish means (CLT)Difference in means
Mann-Whitney UOrdinal or very skewed data, small Independent observationsWhether one group tends to have larger values
BootstrapQuantiles, medians, anything customRepresentative, independent unitsWhatever statistic you compute

z versus t. The t-distribution accounts for estimating from the sample; with thousands of users per group the two are identical, so it matters only below about 30 per group. For proportions the variance follows from itself, hence the z-test. Use Welch (equal_var=False) by default: it does not assume equal variances and loses almost nothing when they are equal.

Mann-Whitney is not a test of means or medians. It tests whether a random user from B tends to beat a random user from A (). Its robustness to outliers is exactly why it ignores the whales that drive revenue.

Worked Example: Skewed Revenue

Revenue per user is zero for most users and log-normal for buyers. Two different treatments both raise mean revenue by 10%: one converts 10% more users into buyers, the other makes existing buyers spend 10% more. How often does each test detect each change? The simulation runs each scenario 300 times with 50,000 users per arm and counts rejections at , which is the test's power.

import numpy as np
from scipy import stats

rng = np.random.default_rng(3)
n, sims = 50_000, 300

def revenue(buy_rate, mean_order):
    """Most users spend 0; buyers spend a log-normal amount with the given mean."""
    buys = rng.random(n) < buy_rate
    amount = rng.lognormal(np.log(mean_order) - 0.5, 1.0, n)
    return np.where(buys, amount, 0.0)

scenarios = {
    "10% more buyers": (0.044, 50.0),
    "10% bigger orders": (0.040, 55.0),
}
for name, (rate, order) in scenarios.items():
    t_hits = u_hits = 0
    for _ in range(sims):
        a, b = revenue(0.040, 50.0), revenue(rate, order)
        t_hits += stats.ttest_ind(b, a, equal_var=False).pvalue < 0.05
        u_hits += stats.mannwhitneyu(b, a, alternative="two-sided").pvalue < 0.05
    print(f"{name:18s} power: Welch t-test {t_hits / sims:.0%}   Mann-Whitney {u_hits / sims:.0%}")
# 10% more buyers    power: Welch t-test 48%   Mann-Whitney 92%
# 10% bigger orders  power: Welch t-test 47%   Mann-Whitney 6%

The Welch t-test detects both changes equally (and weakly, because order amounts are noisy). Mann-Whitney detects the "more buyers" change almost every time, because extra non-zero values shift the ranks, but it is blind to bigger orders: 6% is barely above the 5% false-positive rate. Neither test is better in general. They answer different questions, so decide which question you are asking before you look. If the business decision is about mean revenue, use the t-test (after winsorizing and CUPED to cut variance) or a bootstrap of the mean.


Common Mistakes

The pitfalls below are where most real-world experimentation errors live. Each one has a fix that is standard practice on mature platforms.

1. Peeking (Early Stopping)

Checking your test results daily and stopping when significant inflates the false positive rate dramatically.

Days CheckedNominal AlphaActual False Positive Rate
1 (at end)5%5%
5 (daily)5%~14%
10 (daily)5%~19%
20 (daily)5%~25%

Fix: Pre-determine sample size. If you must peek, use sequential testing (e.g., O'Brien-Fleming boundaries or always-valid p-values).

Why it happens. Under the z-statistic performs a random walk as data accumulates. It wanders above 1.96 and back again many times; checking often and stopping at the first crossing gives the walk many chances to "win". With continuous monitoring, the probability of eventually crossing approaches 100%. The table is easy to verify by simulation (100 looks gives about 37%):

import numpy as np

rng = np.random.default_rng(0)

def peeking_false_positive_rate(looks: int, sims: int = 200_000) -> float:
    """A/A test: the z-statistic after each of `looks` equal batches; stop at first |z| > 1.96."""
    increments = rng.standard_normal((sims, looks))
    z = increments.cumsum(axis=1) / np.sqrt(np.arange(1, looks + 1))
    return float((np.abs(z) > 1.96).any(axis=1).mean())

for looks in (1, 5, 10, 20, 100):
    print(f"{looks:>3} looks: false positive rate = {peeking_false_positive_rate(looks):.1%}")
#   1 looks: false positive rate = 5.0%
#   5 looks: false positive rate = 14.1%
#  10 looks: false positive rate = 19.3%
#  20 looks: false positive rate = 24.8%
# 100 looks: false positive rate = 37.4%

Sequential Testing Done Properly

There are two mainstream ways to look early without inflating .

Group sequential designs with alpha spending. Decide in advance how many interim looks you will take, and "spend" the 5% across them. The O'Brien-Fleming boundary is very strict early and close to 1.96 at the end, so you can stop for a huge effect but lose almost no power at the final analysis. For 5 equally spaced looks at overall two-sided :

LookFraction of dataO'Brien-Fleming boundaryPocock boundary
120%4.562.41
240%3.232.41
360%2.632.41
480%2.282.41
5100%2.042.41

O'Brien-Fleming boundaries are at look ; Pocock uses one constant. Both give an overall false-positive rate of 5% (verified by simulation in the snippet below). The Lan–DeMets alpha spending function generalises this to unplanned look times: you spend of the budget by information fraction , so the timing of looks does not have to be fixed in advance.

Always-valid inference: mSPRT. The mixture sequential probability ratio test (used by Optimizely's Stats Engine, following Johari, Pekelis and Walsh) lets you look after every observation. It computes a likelihood ratio of "some effect, drawn from a prior " against "no effect":

where is the difference in means after users per arm and . Reject when ; the always-valid p-value is . By a martingale argument (Ville's inequality), the chance that ever reaches under is at most , no matter how often you look.

import numpy as np

rng = np.random.default_rng(11)

def msprt_rejects(diffs: np.ndarray, var: float, tau2: float, alpha: float = 0.05) -> bool:
    """diffs: per-pair differences (B_i - A_i). Look after every pair; reject if Lambda >= 1/alpha."""
    n = np.arange(1, diffs.size + 1)
    dbar = diffs.cumsum() / n
    log_lam = 0.5 * np.log(var / (var + n * tau2)) + (n**2 * tau2 * dbar**2) / (2 * var * (var + n * tau2))
    return bool((log_lam >= np.log(1 / alpha)).any())

def naive_rejects(diffs: np.ndarray, var: float, every: int = 100) -> bool:
    """Peek every `every` pairs with a fixed-horizon z-test at 1.96."""
    n = np.arange(1, diffs.size + 1)
    z = diffs.cumsum() / np.sqrt(n * var)
    return bool((np.abs(z[every - 1::every]) > 1.96).any())

sigma2, n_pairs, sims = 1.0, 10_000, 2_000
var = 2 * sigma2                      # Var(B_i - A_i)
for effect in (0.0, 0.05):
    msprt = naive = 0
    for _ in range(sims):
        d = rng.normal(effect, np.sqrt(var), n_pairs)
        msprt += msprt_rejects(d, var, tau2=0.01)
        naive += naive_rejects(d, var)
    label = "A/A (no effect)" if effect == 0 else "effect = 0.05 sd"
    print(f"{label:17s} naive peeking rejects {naive / sims:.1%}   mSPRT rejects {msprt / sims:.1%}")

# Check the O'Brien-Fleming and Pocock boundaries above
z = rng.standard_normal((400_000, 5)).cumsum(axis=1) / np.sqrt(np.arange(1, 6))
obf = 2.04 * np.sqrt(5 / np.arange(1, 6))
print(f"O'Brien-Fleming overall alpha: {(np.abs(z) > obf).any(axis=1).mean():.3f}")
print(f"Pocock overall alpha:          {(np.abs(z) > 2.413).any(axis=1).mean():.3f}")
# A/A (no effect)   naive peeking rejects 37.3%   mSPRT rejects 3.2%
# effect = 0.05 sd  naive peeking rejects 97.0%   mSPRT rejects 74.5%
# O'Brien-Fleming overall alpha: 0.050
# Pocock overall alpha:          0.050

Under the null, naive peeking every 100 users (100 looks) rejects 37% of the time while mSPRT stays at 3%, under its 5% guarantee. With a real effect, mSPRT still detects it 75% of the time, at the cost of somewhat lower power than a fixed-horizon test of the same size. That is the trade: you pay a little power for the right to look whenever you want.

ApproachLooks allowedPower costUsed by
Fixed horizonOnce, at the endNoneClassic textbook tests
Group sequential (O'Brien-Fleming, alpha spending)A few, plannedSmallClinical trials, many in-house platforms
mSPRT / always-valid p-valuesContinuouslyModerateOptimizely and others

2. Multiple Comparisons

Testing 20 variants simultaneously means one will be "significant" by chance (5% × 20 = 1 expected false positive).

Fix: Apply Bonferroni correction (alpha/n) or use False Discovery Rate (FDR) control.

The probability of at least one false positive across independent null tests is : 40% for 10 tests and 64% for 20. The same applies to testing one variant on 20 metrics, or slicing one test into 20 segments ("it works for Android users in Brazil!").

Bonferroni controls the family-wise error rate (the chance of any false positive) by testing each hypothesis at . It is the union bound from the probability page: . Simple and valid under any dependence, but conservative: with many metrics it destroys power.

Benjamini–Hochberg controls the false discovery rate (the expected fraction of your "discoveries" that are false), which is usually what a product team cares about. Procedure: sort the p-values ascending, ; find the largest with ; reject hypotheses .

Worked example with the ten p-values below and :

Rank p-valueBH threshold Below?Bonferroni ()?
10.0010.005yesyes
20.0080.010yesno
30.0390.015nono
40.0410.020nono
50.150.025nono

The largest rank under its threshold is , so BH rejects two hypotheses; Bonferroni rejects one. The p-values 0.039 and 0.041, "significant" without correction, are not. scipy's adjusted p-values say the same: the first two are at or below 0.05.

# Bonferroni correction
n_comparisons = 10
adjusted_alpha = 0.05 / n_comparisons  # 0.005
# Much stricter threshold — reduces false positives

# Benjamini-Hochberg FDR (less conservative)
from scipy.stats import false_discovery_control
p_values = [0.001, 0.008, 0.039, 0.041, 0.15, 0.28, 0.55, 0.67, 0.78, 0.92]
adjusted = false_discovery_control(p_values, method='bh')
print([round(float(x), 4) for x in adjusted])
print("BH discoveries:", [p for p, adj in zip(p_values, adjusted) if adj <= 0.05])
print("Bonferroni discoveries:", [p for p in p_values if p < adjusted_alpha])
# [0.01, 0.04, 0.1025, 0.1025, 0.3, 0.4667, 0.7857, 0.8375, 0.8667, 0.92]
# BH discoveries: [0.001, 0.008]
# Bonferroni discoveries: [0.001]

In practice: declare one primary metric (the decision metric) before the test and do not correct it; treat secondary metrics and segment cuts as exploratory, corrected with BH, and confirm surprising segment results in a follow-up test.

3. Novelty and Primacy Effects

  • Novelty effect: Users click new things because they are new. Effect fades after 1-2 weeks.
  • Primacy effect: Returning users prefer what they are used to. Engagement dips initially then recovers.

Fix: Run tests for at least 2 full weeks. Segment analysis by new vs returning users.

Detecting it. Plot the daily treatment effect, not the cumulative one: novelty looks like a decaying early lift, primacy like an early dip that recovers. New users have no habits, so their effect estimates the long run. For long-term consequences, keep a small holdout on the old experience for months.

4. Simpson's Paradox

A treatment can appear better overall but be worse in every segment — because the segments have different sizes and different baseline rates.

Correction (2026-11-09)

The original example had control at 55% on mobile and 65% on desktop but 52% overall, which is impossible: a weighted average of 55% and 65% must lie between them. Its explanation also had the direction backwards. For treatment to win overall while losing in each segment, treatment must be over-weighted toward the high-baseline segment (desktop). The example below uses a consistent traffic mix.

Overall: Treatment wins (58% vs 57%)

But segment by device:
  Mobile:  Treatment 50% vs Control 55% — Control wins
  Desktop: Treatment 60% vs Control 65% — Control wins

How? Treatment's traffic was 80% desktop (high baseline): 0.2 × 50% + 0.8 × 60% = 58%.
Control's traffic was 80% mobile (low baseline):          0.8 × 55% + 0.2 × 65% = 57%.
Treatment's overall average is propped up by its desktop-heavy mix, not by being better.

Fix: Always segment your analysis. Check key segments (device, country, new vs returning) for consistency.

In a properly randomized experiment the mix should be the same in both arms, so Simpson's paradox appearing between arms is itself a red flag for broken randomization (see sample ratio mismatch below). It shows up legitimately when you pool across time periods with different traffic splits, for example ramping treatment from 5% to 50% mid-test while the weekday/weekend mix shifts. Analyse each ramp phase separately, or only analyse data from the final split.

5. Not Accounting for Network Effects

Standard A/B tests assume independence between users. Social features (sharing, messaging, marketplace) violate this assumption.

Fix: Use cluster randomization (randomize by city, company, or social graph cluster).

The assumption has a name, SUTVA (the stable unit treatment value assumption): one user's outcome depends only on their own assignment. It fails in two broad ways.

  • Network effects. A treated user's new "share" button creates notifications for control users, who then engage more. The control group is contaminated, and the measured difference underestimates the true effect.
  • Marketplace interference. In a two-sided market, a treatment that helps buyers win auctions (or riders get cars) takes supply away from the control group. The measured difference overestimates the effect, because treatment wins partly at control's expense; at 100% rollout there is nobody to take from.

Designs that reduce interference:

DesignHowCost
Cluster randomizationRandomize by city, school, company or graph clusterFar fewer independent units, so much wider CIs
Switchback testsAlternate the whole market between A and B by time slot (e.g. every 30 minutes)Carryover between slots; needs time-series aware analysis
Geo experimentsTreat whole regions, compare with matched control regionsFew units; use synthetic control methods
Budget-split marketplacesSplit supply (e.g. ad budget) as well as demandEngineering complexity
flowchart LR
    subgraph USER["User-level randomization"]
        U1["Treated user"] -->|"shares, messages"| U2["Control user"]
    end
    subgraph CLUSTER["Cluster randomization"]
        C1["City A: all treated"]
        C2["City B: all control"]
    end
    USER -->|"contamination biases the estimate"| BAD["Effect underestimated"]
    CLUSTER -->|"interference stays inside clusters"| OK["Unbiased, but fewer units"]

With cluster randomization, the effective sample size is the number of clusters (adjusted by the intra-cluster correlation), not the number of users. Analyse at the cluster level, or use cluster-robust standard errors, or the confidence interval will be far too narrow.

6. Sample Ratio Mismatch (SRM)

Intuition. If you configured a 50/50 split, the observed counts should be close to 50/50, within binomial noise. When they are not, something in the pipeline is removing or adding users differently in one arm: a redirect that loses slow clients, a bot filter that triggers on the new page, a crash before the logging event, a cache serving the old variant. Any result from an experiment with SRM is untrustworthy, because the arms are no longer comparable.

The test. A chi-square goodness-of-fit test of observed counts against the configured split:

Use a strict threshold such as , because you run this check on every experiment and do not want false alarms.

Worked example. Configured 50/50, observed 100,000 control and 101,500 treatment users. Expected 100,750 each. , . A 0.75% imbalance looks trivial on a dashboard and is a clear SRM.

from scipy.stats import chisquare

def srm_check(counts: list[int], ratios: list[float], threshold: float = 0.001) -> None:
    total = sum(counts)
    expected = [total * r for r in ratios]
    stat, p = chisquare(counts, f_exp=expected)
    status = "SRM DETECTED - do not trust results" if p < threshold else "ok"
    print(f"counts={counts} chi2={stat:.2f} p={p:.4f} -> {status}")

srm_check([100_000, 101_500], [0.5, 0.5])
srm_check([100_000, 100_400], [0.5, 0.5])
srm_check([10_000, 10_150], [0.5, 0.5])
# counts=[100000, 101500] chi2=11.17 p=0.0008 -> SRM DETECTED - do not trust results
# counts=[100000, 100400] chi2=0.80 p=0.3716 -> ok
# counts=[10000, 10150] chi2=1.12 p=0.2906 -> ok

The last line shows why SRM is checked on counts, not percentages: the same 0.75% imbalance on 20,000 users is well within noise. Microsoft's experimentation team reported SRM in roughly 6% of experiments (Fabijan et al., "Diagnosing Sample Ratio Mismatch in Online Controlled Experiments", KDD 2019), so automate the check.


Variance Reduction with CUPED

Intuition

Much of the noise in a metric has nothing to do with your experiment: heavy users are heavy users whether they get A or B. If you know how active each user was before the experiment started, you can subtract the part of their behaviour you could have predicted, and compare only the unpredictable remainder. Same effect estimate, much less noise. This is CUPED (Controlled-experiment Using Pre-Experiment Data, Deng, Xu, Kohavi and Walker, WSDM 2013).

The Formula

For each user, let be the metric during the experiment and a pre-experiment covariate (usually the same metric over the prior weeks). Define

Because was measured before randomization, it is independent of the assignment, so differs between arms by exactly the same treatment effect. The variance shrinks:

where is the correlation between and . With , variance drops by 49%, which is equivalent to running the experiment with almost twice as many users.

Worked Example

import numpy as np
from scipy import stats

rng = np.random.default_rng(5)
n = 20_000
# Pre-period activity drives the in-experiment metric; correlation about 0.7
pre = rng.gamma(shape=2.0, scale=5.0, size=2 * n)
post = 0.8 * pre + rng.normal(0, 6.0, 2 * n)
arm = np.repeat([0, 1], n)
post = post + 0.25 * arm          # true treatment effect = +0.25

theta = np.cov(pre, post)[0, 1] / np.var(pre, ddof=1)   # pooled across arms
cuped = post - theta * (pre - pre.mean())

for name, y in (("raw", post), ("CUPED", cuped)):
    a, b = y[arm == 0], y[arm == 1]
    diff = b.mean() - a.mean()
    se = np.sqrt(a.var(ddof=1) / n + b.var(ddof=1) / n)
    p = stats.ttest_ind(b, a, equal_var=False).pvalue
    print(f"{name:5s}: effect={diff:+.3f}  95% CI=[{diff - 1.96 * se:+.3f}, {diff + 1.96 * se:+.3f}]  p={p:.4f}")

rho = np.corrcoef(pre, post)[0, 1]
print(f"correlation={rho:.2f}  predicted variance reduction={rho**2:.0%}")
# raw  : effect=+0.238  95% CI=[+0.076, +0.399]  p=0.0039
# CUPED: effect=+0.303  95% CI=[+0.185, +0.421]  p=0.0000
# correlation=0.68  predicted variance reduction=47%

Both estimates are consistent with the true +0.25 (CUPED's point estimate differs because it removes a different slice of noise, not because it is biased). The CI width shrinks from 0.32 to 0.24, a 27% cut, matching the predicted , and the p-value drops from 0.004 to below 0.0001.

Engineering Application

  • Covariate choice: the same metric over the 1–4 weeks before the test is usually best. New users have no history; set their to the mean (or add a "has history" indicator) so they get no adjustment.
  • Compute on pooled data (both arms together), never per arm, so it cannot absorb the treatment effect.
  • CUPED is linear regression adjustment. Regressing on the treatment indicator and gives the same answer; generalisations use several covariates or machine-learned predictions (sometimes called CUPAC).

Guardrail Metrics and Ratio Metrics

Guardrail Metrics

Intuition. Your primary metric says whether the change achieved its goal. Guardrails say whether it broke something else on the way: latency, error rate, crash rate, unsubscribes, revenue, page load time. A checkout redesign that raises conversion 2% while adding 400 ms of latency to every page may lose more than it gains elsewhere.

Guardrails are usually one-sided non-inferiority tests: ship only if the CI's upper bound for, say, latency increase is below +2%. "Not significant" is not enough, because an underpowered guardrail is always non-significant. Trust guardrails (SRM, logging loss) block analysis; organisational guardrails (revenue, latency) block shipping.

Ratio Metrics and the Delta Method

Intuition. Many metrics are ratios of sums: click-through rate = total clicks / total page views, revenue per session, errors per request. If you randomize by user but compute the ratio over page views, page views from the same user are correlated (a heavy clicker clicks on all their views). Treating each page view as an independent Bernoulli trial makes the standard error too small, sometimes by a factor of two or more, and you will declare noise significant.

The fix is to treat the ratio as a function of two per-user means. For per-user clicks and views :

The delta method linearises around the means (a first-order Taylor expansion) to get its variance:

with and so on. The same tool gives the variance of a relative lift , which is how platforms report "+3.2% ± 1.1%".

import numpy as np

rng = np.random.default_rng(9)
n_users = 5_000
# Users differ in how many pages they view and how click-happy they are
views = rng.poisson(rng.gamma(2.0, 5.0, n_users)) + 1
ctr_user = rng.beta(2, 30, n_users)                 # per-user click propensity
clicks = rng.binomial(views, ctr_user)

ratio = clicks.sum() / views.sum()

# Naive: pretend every page view is an independent Bernoulli trial
se_naive = np.sqrt(ratio * (1 - ratio) / views.sum())

# Delta method on per-user sums
mc, mv = clicks.mean(), views.mean()
var_c, var_v = clicks.var(ddof=1), views.var(ddof=1)
cov_cv = np.cov(clicks, views)[0, 1]
var_r = (var_c / mv**2 + mc**2 * var_v / mv**4 - 2 * mc * cov_cv / mv**3) / n_users
se_delta = np.sqrt(var_r)

# Ground truth: bootstrap over users
boot = []
for _ in range(2_000):
    idx = rng.integers(0, n_users, n_users)
    boot.append(clicks[idx].sum() / views[idx].sum())
se_boot = np.std(boot, ddof=1)

print(f"CTR = {ratio:.4f}")
print(f"naive SE     = {se_naive:.5f}")
print(f"delta SE     = {se_delta:.5f}")
print(f"bootstrap SE = {se_boot:.5f}")
# CTR = 0.0630
# naive SE     = 0.00104
# delta SE     = 0.00126
# bootstrap SE = 0.00125

The delta-method SE matches the user-level bootstrap and is about 21% larger than the naive one; with more heterogeneous users the gap is often 2x or more. Any experimentation platform that computes CTR's SE from page-view counts alone has an inflated false-positive rate on every ratio metric. The general rule: the unit of analysis must match the unit of randomization, or you must correct for the mismatch.


Bayesian vs Frequentist

Frequentist Approach (What We Covered Above)

  • Ask: "How likely is this data if the null hypothesis is true?"
  • Output: p-value, confidence interval
  • Decision: Reject or fail to reject H0 at a fixed alpha
  • Limitation: Cannot say "probability treatment is better is 95%"

Bayesian Approach

  • Ask: "Given this data, what is the probability that treatment is better?"
  • Output: Posterior probability distribution over the effect size
  • Decision: "P(treatment > control) = 96.3%"
  • Advantage: More intuitive interpretation, works with smaller samples

Correction (2026-11-09)

"Works with smaller samples" needs a qualifier. A Bayesian analysis does not extract more information from the same data; with little data the answer is driven mostly by the prior. It expresses small-sample uncertainty in a more intuitive form, and an informative prior based on past experiments can genuinely help, but a flat-prior Bayesian test on 500 users is no more reliable than a frequentist one.

The Beta-Binomial Model

Intuition. Treat the unknown conversion rate as a random variable. Start with a prior belief, observe conversions, and update. For conversion data the Beta distribution is the natural choice because it is conjugate to the binomial: a prior plus conversions out of gives a posterior. No integration needed; you just add counts.

Worked example. With a flat prior, control's 500/10,000 becomes and treatment's 550/10,000 becomes . Comparing samples gives , the counterpart of the frequentist on the same data.

import numpy as np
from scipy import stats

def bayesian_ab_test(
    conversions_a: int, visitors_a: int,
    conversions_b: int, visitors_b: int,
    n_simulations: int = 100_000,
) -> dict:
    """
    Bayesian A/B test using Beta-Binomial model.
    Prior: Beta(1, 1) — uniform prior (non-informative).
    """
    # Posterior distributions (conjugate prior: Beta + Binomial → Beta)
    alpha_a = 1 + conversions_a       # Prior alpha + successes
    beta_a  = 1 + visitors_a - conversions_a  # Prior beta + failures
    alpha_b = 1 + conversions_b
    beta_b  = 1 + visitors_b - conversions_b

    # Sample from posteriors
    samples_a = np.random.beta(alpha_a, beta_a, n_simulations)
    samples_b = np.random.beta(alpha_b, beta_b, n_simulations)

    # P(B > A)
    prob_b_better = (samples_b > samples_a).mean()

    # Expected lift distribution
    lift = (samples_b - samples_a) / samples_a
    lift_mean = lift.mean()
    lift_ci = np.percentile(lift, [2.5, 97.5])

    return {
        "prob_b_better": prob_b_better,
        "expected_lift": lift_mean,
        "lift_95_ci": (lift_ci[0], lift_ci[1]),
    }

np.random.seed(0)  # added so the printed numbers are reproducible
result = bayesian_ab_test(
    conversions_a=500, visitors_a=10000,
    conversions_b=550, visitors_b=10000,
)
print(f"P(B > A): {result['prob_b_better']:.1%}")
print(f"Expected lift: {result['expected_lift']:.1%}")
print(f"95% credible interval: [{result['lift_95_ci'][0]:.1%}, {result['lift_95_ci'][1]:.1%}]")
# P(B > A): 94.4%
# Expected lift: 10.2%
# 95% credible interval: [-2.2%, 23.6%]

Expected Loss: A Better Decision Rule

"" ignores how much you lose when you are wrong. The expected loss of shipping B is the average shortfall in the scenarios where A is actually better:

Ship B when its expected loss falls below a "threshold of caring", for example 0.01 percentage points of conversion. This rule naturally stops early when B is clearly better or when the two are so close that choosing wrongly costs nothing, which is often the right business decision.

import numpy as np

rng = np.random.default_rng(0)
a = rng.beta(1 + 500, 1 + 9_500, 200_000)
b = rng.beta(1 + 550, 1 + 9_450, 200_000)
print(f"P(B > A)            = {np.mean(b > a):.1%}")
print(f"expected loss of B  = {np.mean(np.maximum(a - b, 0)):.5f}  (absolute conversion)")
print(f"expected loss of A  = {np.mean(np.maximum(b - a, 0)):.5f}")
# P(B > A)            = 94.3%
# expected loss of B  = 0.00008  (absolute conversion)
# expected loss of A  = 0.00506

Shipping B risks an expected loss of 0.008 percentage points of conversion; staying on A risks about 0.5 points, some sixty times as much. That asymmetry, not a 95% threshold, is the decision.

Comparison

Correction (2026-11-09)

The "Peeking problem" row previously said the Bayesian approach "naturally handles sequential analysis". The posterior is valid at any moment, but a decision rule such as "stop as soon as " checked daily still produces many more false winners than 5% when there is no real difference, just as frequentist peeking does. Expected-loss stopping rules and informative priors reduce the problem; they do not remove it. The row now says so.

AspectFrequentistBayesian
Interpretation"How extreme is this data?""How likely is the treatment better?"
Sample sizeMust pre-determineCan update as data arrives
Peeking problemInflates false positivesLess sensitive, but "stop when P(B > A) > 95%" still inflates false positives; use expected-loss thresholds
Decision thresholdp < 0.05P(B > A) > 0.95 (or loss threshold)
ComputationSimple formulasSimulation or MCMC
Prior influenceNo priorRequires choosing a prior
Industry adoptionGoogle, LinkedInVWO, Dynamic Yield

When to Use Which

Frequentist: When you have high traffic, can pre-determine sample size, and need results that are easy to explain to stakeholders ("p < 0.05").

Bayesian: When you have low traffic, need to peek at results, or want to answer "what is the probability B is better?" directly.


Multi-Armed Bandits

Multi-armed bandits (MABs) balance exploration (trying different variants) and exploitation (showing the best-known variant). Unlike A/B tests that split traffic 50/50, bandits dynamically allocate traffic to better-performing variants.

The quantity a bandit minimises is regret: the total reward lost compared with always playing the best arm. Good bandit algorithms have regret growing only logarithmically with time, meaning they spend an ever-smaller fraction of traffic on bad arms.

Epsilon-Greedy

The simplest bandit: with probability epsilon, pick a random arm (explore); otherwise pick the best-known arm (exploit).

import random

class EpsilonGreedy:
    def __init__(self, n_arms: int, epsilon: float = 0.1):
        self.epsilon = epsilon
        self.counts = [0] * n_arms        # Times each arm was pulled
        self.rewards = [0.0] * n_arms     # Total reward per arm

    def select_arm(self) -> int:
        if random.random() < self.epsilon:
            return random.randint(0, len(self.counts) - 1)  # Explore
        # Exploit — pick arm with highest average reward
        averages = [
            self.rewards[i] / self.counts[i] if self.counts[i] > 0 else float('inf')
            for i in range(len(self.counts))
        ]
        return averages.index(max(averages))

    def update(self, arm: int, reward: float):
        self.counts[arm] += 1
        self.rewards[arm] += reward

Thompson Sampling

More sophisticated: maintain a probability distribution for each arm's reward rate and sample from it. Arms with more uncertainty get explored more.

import numpy as np

class ThompsonSampling:
    def __init__(self, n_arms: int):
        self.alpha = np.ones(n_arms)  # Successes + 1 (Beta prior)
        self.beta = np.ones(n_arms)   # Failures + 1

    def select_arm(self) -> int:
        # Sample from each arm's posterior Beta distribution
        samples = [
            np.random.beta(self.alpha[i], self.beta[i])
            for i in range(len(self.alpha))
        ]
        return int(np.argmax(samples))

    def update(self, arm: int, reward: float):
        if reward > 0:
            self.alpha[arm] += 1
        else:
            self.beta[arm] += 1

# Quick check: three banners with true click rates 4%, 5% and 6%
np.random.seed(1)
true_rates = [0.04, 0.05, 0.06]
bandit = ThompsonSampling(3)
pulls = [0, 0, 0]
for _ in range(20_000):
    arm = bandit.select_arm()
    pulls[arm] += 1
    bandit.update(arm, float(np.random.random() < true_rates[arm]))
print("share of traffic per arm:", [round(p / sum(pulls), 2) for p in pulls])
# share of traffic per arm: [0.01, 0.15, 0.84]

After 20,000 impressions Thompson sampling has sent 84% of traffic to the 6% arm and only 1% to the 4% arm, while still occasionally sampling the others in case it is wrong.

Upper Confidence Bound (UCB1)

Pick the arm with the highest upper confidence bound — balances estimated reward with exploration bonus.

The bonus term is large for arms pulled rarely ( small) and grows slowly with total pulls , so neglected arms are eventually retried. It comes from a Chernoff–Hoeffding tail bound: the true mean is very unlikely to exceed the sample mean by more than the bonus.

import math

class UCB1:
    def __init__(self, n_arms: int):
        self.counts = [0] * n_arms
        self.rewards = [0.0] * n_arms
        self.total = 0

    def select_arm(self) -> int:
        # Play each arm once first
        for i in range(len(self.counts)):
            if self.counts[i] == 0:
                return i

        ucb_values = [
            (self.rewards[i] / self.counts[i]) +
            math.sqrt(2 * math.log(self.total) / self.counts[i])
            for i in range(len(self.counts))
        ]
        return ucb_values.index(max(ucb_values))

    def update(self, arm: int, reward: float):
        self.counts[arm] += 1
        self.rewards[arm] += reward
        self.total += 1
graph LR
    subgraph "A/B Test"
        AB1["50% Traffic → A"]
        AB2["50% Traffic → B"]
        AB3["Wait for sample size"]
        AB4["Analyze and decide"]
        AB1 --> AB3 --> AB4
        AB2 --> AB3
    end

    subgraph "Multi-Armed Bandit"
        MAB1["Start equal"]
        MAB2["Shift traffic to winner"]
        MAB3["Converge to best arm"]
        MAB1 --> MAB2 --> MAB3
    end

    style AB3 fill:#d97706,color:#fff
    style MAB3 fill:#059669,color:#fff

Bandits vs A/B Tests

Bandits minimize regret (lost conversions during the experiment) but make it harder to measure the true effect size. Use bandits when you care more about optimization than measurement — for example, choosing which promotional banner to show. Use A/B tests when you need rigorous causal measurement.

Bandits also assume quick rewards (a click, not 30-day retention) and stable arm quality; a bandit that converged on Monday's winner can be slow to notice a weekend shift.


Testing Checklist

graph TD
    A["1. Define metric and MDE"] --> B["2. Calculate sample size"]
    B --> C["3. Set up experiment<br/>(random assignment, logging)"]
    C --> D["4. Run for predetermined duration"]
    D --> E["5. Check for SRM<br/>(Sample Ratio Mismatch)"]
    E --> F["6. Analyze: CI + p-value"]
    F --> G["7. Segment analysis"]
    G --> H["8. Document and decide"]

    style E fill:#d97706,color:#fff
    style F fill:#059669,color:#fff
StepWhat to Check
Sample Ratio MismatchAre the groups actually 50/50? If not, something is wrong with randomization.
Guardrail metricsDid we hurt revenue, latency, error rate, or engagement?
Novelty effectIs the lift consistent across the experiment duration?
Segment consistencyDoes the treatment help across platforms, geos, and user types?
Practical significanceEven if statistically significant, is the effect worth shipping?
Unit of analysisDoes it match the unit of randomization? If not, use the delta method or bootstrap over users.
Multiple testingOne pre-registered primary metric; BH on the rest.
A/A testsRun A/A tests regularly: about 5% should be "significant". Many more means the pipeline is broken.

Interview Questions

1. Your test shows a 10% lift with p = 0.11 on 10,000 users per arm. The PM wants to run it another week "to get significance". What do you say?

Answer. The test was underpowered: at a 5% baseline, 10,000 per arm only reliably detects about a 17% relative lift (MDE points), and the power for a true 10% lift is about 35%. Extending is fine if it is planned: compute the required size (about 31,000 per arm for 10% at 80% power) and run to that size. What is not fine is extending repeatedly until , which is peeking. If repeated looks are needed, use a sequential design.

2. How do you compute the sample size for detecting a 10% relative lift on a 5% conversion rate?

Answer. per arm, about 62,500 total. At 10,000 visitors per day that is 6.2 days, rounded up to whole weeks (7 or 14 days) for weekly seasonality and novelty.

3. What is sample ratio mismatch and how do you test for it?

Answer. The observed split differs from the configured split by more than chance, indicating that something (redirects, bot filtering, crashes, logging) drops users unevenly. Test with a chi-square goodness-of-fit at a strict threshold such as . Example: 100,000 vs 101,500 on a 50/50 split gives , : SRM. Do not analyse the results; find the cause.

4. Your metric is click-through rate per page view but you randomize by user. What goes wrong and how do you fix it?

Answer. Page views from one user are correlated, so treating them as independent Bernoulli trials underestimates the variance and inflates false positives. Compute the ratio of per-user means and use the delta method for its variance, or bootstrap over users. The unit of analysis must match the unit of randomization.

5. What is CUPED and when does it help most?

Answer. It subtracts the part of each user's metric predicted by pre-experiment data: with . The effect estimate is unchanged and variance falls by . It helps most for metrics that are stable per user (engagement, sessions, revenue for repeat customers); a correlation of 0.7 halves the variance, equivalent to doubling traffic. It does nothing for brand-new users with no history.

6. You test 20 metrics and 3 are significant at 0.05. How do you report this?

Answer. With 20 null metrics you would expect one false positive and have a 64% chance of at least one. Report the pre-registered primary metric as the decision; correct secondary metrics with Benjamini–Hochberg (or Bonferroni, , if any false positive is costly); and treat surprising secondary or segment wins as hypotheses for a follow-up test.

7. Why can't you just stop the test the first day it is significant? What can you do instead?

Answer. Under the null the z-statistic random-walks and crosses 1.96 repeatedly; daily checks over 20 days push the false-positive rate to about 25%. Instead use a group sequential design (O'Brien-Fleming boundaries such as 4.56, 3.23, 2.63, 2.28, 2.04 for five looks) or always-valid inference such as mSPRT, which permits continuous monitoring at the cost of some power.

8. A marketplace test shows treatment buyers convert 8% more. Why might the real effect at 100% launch be smaller?

Answer. Interference: treatment buyers compete with control buyers for the same supply, so part of the lift is taken from control rather than created. At full launch there is no control to take from. Use switchback or geo-randomized designs, or split supply as well as demand, to estimate the global effect.