All frameworks

Holdout Test Size Calculator

How many leads must I hold out, and for how long, to detect a given incremental effect on funded loans?

Enter the source's volume and funded rate and the smallest incremental share you care about. The calculator works out how many leads to decline, how long that takes and what it costs. The defaults are the illustrative Source A example from the article.

Held-out funded rate at that share: 4.50% versus 6.0% for bought leads.

Leads to hold out
2,005
Bought leads in the test
18,045
Weeks of traffic needed
8.7 weeks
Until a readable result
17.3 weeks
Detectable share after 10 wk
23%
Leads declined in 10 wk
2,308
Cost of 10 wk of holdout
$41,538
Target 25%1 week26 weeks

The line shows the smallest incremental share the test could detect if you stopped after each week; the test is big enough once it drops below your target. It uses the standard two-proportion z-test with unequal groups, so it assumes leads are randomized one by one and borrowers fund independently. Weeks count from the first declined lead; the lag is added on top because the last leads still need time to fund. The default cost per declined lead ($18) is what the article's last-touch report implies: 6% of leads funding at $800 margin, less the $30 price. If the source is mostly not incremental, the true cost of declining is lower.

How it works

Before you decline a random share of a lead source to measure its incrementality, check that the test can actually find the effect you care about. Enter the source's volume, its funded rate and the smallest incremental share worth detecting, and the calculator returns the leads to hold out, the weeks it takes and what declining them costs. The default inputs are an illustrative example, not benchmarks.

If a share X of the source's funded loans is truly incremental, held-out leads fund at the bought rate × (1 − X). The calculator uses the standard sample size for a two-sided two-proportion z-test (normal approximation, no continuity correction) with unequal groups: k bought leads for every held-out lead, where k = (1 − holdout share) ÷ holdout share. Weeks needed = held-out leads ÷ (holdout share × monthly leads × 12 ÷ 52); the outcome lag is added on top for a readable result. The detectable share after a chosen number of weeks is the X at which the required held-out leads equal the leads declined by then.

pH=pB (1−X),k=1−hhp_H = p_B\,(1 - X), \qquad k = \frac{1 - h}{h}
pˉ=pH+k pB1+k\bar p = \frac{p_H + k\,p_B}{1 + k}
A=zα/2pˉ (1−pˉ) (1+1/k)A = z_{\alpha/2}\sqrt{\bar p\,(1 - \bar p)\,(1 + 1/k)}
C=zβpH(1−pH)+pB(1−pB)/kC = z_{\beta}\sqrt{p_H(1 - p_H) + p_B(1 - p_B)/k}
nH=(A+C)2(pB−pH)2,nB=k nHn_H = \frac{(A + C)^2}{(p_B - p_H)^2}, \qquad n_B = k\,n_H
where
pBp_B
funded rate of bought leads
pHp_H
funded rate of held-out leads
XX
incremental share to detect
hh
holdout share of traffic
zα/2, zβz_{\alpha/2},\ z_{\beta}
normal quantiles for confidence and power
nH, nBn_H,\ n_B
leads needed in the holdout and bought groups

What a real decision adds

  • Clean randomization is harder than it looks. The same borrower can arrive through several sources or sessions and land in both groups, and some channels cannot be withheld lead by lead at all, which pushes the test to regions.
  • Declined borrowers still fund through other routes, sometimes much later. Counting them depends on matching people across channels and waiting out the full outcome lag.
  • Funded-loan outcomes are noisy relative to the effects that matter, and source mix, lender appetite and seasonality shift while the test runs, adding uncertainty the formula does not include.
  • Declining leads costs more than lost margin: volume commitments, caps and seller relationships can limit how large a holdout can be and how long it can run.

Working through this decision for your business?

Talk to us

Frequently asked questions

Why does a small holdout take so long?
The comparison is only as precise as its smaller group. A 10% holdout gives a cheap test because you decline few leads, but the held-out group fills slowly, so you need more weeks of traffic than a 50/50 split would.
What incremental share should I try to detect?
The smallest share that would change your decision. If the source pays off only when at least 40% of its funded loans are incremental, size the test to detect 40%, not 10%: smaller effects need far more leads.
Why add the outcome lag to the test length?
Funded loans arrive weeks after the lead. You can stop declining leads once you have enough, but you cannot read the result until the last declined borrowers have had time to fund elsewhere.
What if I stop the test early?
The smallest effect you can detect gets bigger. A test stopped before it reaches the required size can still find a large effect, but a "no difference" result then says little about smaller ones.
Read the articleAttribution Models for Multi-Touch Lead JourneysLast-click attribution is broken for complex lead journeys. Learn how AI-powered multi-touch attribution assigns credit based on actual influence.