Why does a small holdout take so long?

The comparison is only as precise as its smaller group. A 10% holdout gives a cheap test because you decline few leads, but the held-out group fills slowly, so you need more weeks of traffic than a 50/50 split would.

What incremental share should I try to detect?

The smallest share that would change your decision. If the source pays off only when at least 40% of its funded loans are incremental, size the test to detect 40%, not 10%: smaller effects need far more leads.

Why add the outcome lag to the test length?

Funded loans arrive weeks after the lead. You can stop declining leads once you have enough, but you cannot read the result until the last declined borrowers have had time to fund elsewhere.

What if I stop the test early?

The smallest effect you can detect gets bigger. A test stopped before it reaches the required size can still find a large effect, but a "no difference" result then says little about smaller ones.

All tools

Holdout Test Size Calculator

How many leads must I hold out, and for how long, to detect a given incremental effect on funded loans?

Enter the source's volume and funded rate and the smallest incremental share you care about. The calculator works out how many leads to decline, how long that takes and what it costs. The defaults are the illustrative Source A example from the article.

Leads per month from the source10,000Funded-loan rate among bought leads6.0%Smallest incremental share to detect25%Share of leads held out10%

Confidence (two-sided)Power

Outcome lag (days until loans fund)60 daysIf you stop the test after10 weeksValue you give up per declined lead$18

Held-out funded rate at that share: 4.50% versus 6.0% for bought leads.

Leads to hold out

2,005

Bought leads in the test

18,045

Weeks of traffic needed

8.7 weeks

Until a readable result

17.3 weeks

Detectable share after 10 wk

23%

Leads declined in 10 wk

2,308

Cost of 10 wk of holdout

$41,538

The line shows the smallest incremental share the test could detect if you stopped after each week; the test is big enough once it drops below your target. It uses the standard two-proportion z-test with unequal groups, so it assumes leads are randomized one by one and borrowers fund independently. Weeks count from the first declined lead; the lag is added on top because the last leads still need time to fund. The default cost per declined lead ($18) is what the article's last-touch report implies: 6% of leads funding at $800 margin, less the $30 price. If the source is mostly not incremental, the true cost of declining is lower.

How it works

Before you decline a random share of a lead source to measure its incrementality, check that the test can actually find the effect you care about. Enter the source's volume, its funded rate and the smallest incremental share worth detecting, and the calculator returns the leads to hold out, the weeks it takes and what declining them costs. The default inputs are an illustrative example, not benchmarks.

If a share X of the source's funded loans is truly incremental, held-out leads fund at the bought rate × (1 − X). The calculator uses the standard sample size for a two-sided two-proportion z-test (normal approximation, no continuity correction) with unequal groups: k bought leads for every held-out lead, where k = (1 − holdout share) ÷ holdout share. Weeks needed = held-out leads ÷ (holdout share × monthly leads × 12 ÷ 52); the outcome lag is added on top for a readable result. The detectable share after a chosen number of weeks is the X at which the required held-out leads equal the leads declined by then.

p_H = p_B\,(1 - X), \qquad k = \frac{1 - h}{h}

\bar p = \frac{p_H + k\,p_B}{1 + k}

A = z_{\alpha/2}\sqrt{\bar p\,(1 - \bar p)\,(1 + 1/k)}

C = z_{\beta}\sqrt{p_H(1 - p_H) + p_B(1 - p_B)/k}

n_H = \frac{(A + C)^2}{(p_B - p_H)^2}, \qquad n_B = k\,n_H

where

$p_B$: funded rate of bought leads
$p_H$: funded rate of held-out leads
$X$: incremental share to detect
$h$: holdout share of traffic
$z_{\alpha/2},\ z_{\beta}$: normal quantiles for confidence and power
$n_H,\ n_B$: leads needed in the holdout and bought groups

Frequently asked questions

Why does a small holdout take so long?: The comparison is only as precise as its smaller group. A 10% holdout gives a cheap test because you decline few leads, but the held-out group fills slowly, so you need more weeks of traffic than a 50/50 split would.
What incremental share should I try to detect?: The smallest share that would change your decision. If the source pays off only when at least 40% of its funded loans are incremental, size the test to detect 40%, not 10%: smaller effects need far more leads.
Why add the outcome lag to the test length?: Funded loans arrive weeks after the lead. You can stop declining leads once you have enough, but you cannot read the result until the last declined borrowers have had time to fund elsewhere.
What if I stop the test early?: The smallest effect you can detect gets bigger. A test stopped before it reaches the required size can still find a large effect, but a "no difference" result then says little about smaller ones.

Read the articleAttribution Models for Multi-Touch Lead JourneysLast-click attribution is broken for complex lead journeys. Learn how AI-powered multi-touch attribution assigns credit based on actual influence.

Strategy