---
title: "Holdout Test Size Calculator"
category: Strategy
canonical: https://plato.ai/tools/holdout-test-size/
---

# Holdout Test Size Calculator

How many leads must I hold out, and for how long, to detect a given incremental effect on funded loans?

Free interactive calculator at https://plato.ai/tools/holdout-test-size/ (runs in the browser).

## How it works

Before you decline a random share of a lead source to measure its incrementality, check that the test can actually find the effect you care about. Enter the source's volume, its funded rate and the smallest incremental share worth detecting, and the calculator returns the leads to hold out, the weeks it takes and what declining them costs. The default inputs are an illustrative example, not benchmarks.

If a share X of the source's funded loans is truly incremental, held-out leads fund at the bought rate × (1 − X). The calculator uses the standard sample size for a two-sided two-proportion z-test (normal approximation, no continuity correction) with unequal groups: k bought leads for every held-out lead, where k = (1 − holdout share) ÷ holdout share. Weeks needed = held-out leads ÷ (holdout share × monthly leads × 12 ÷ 52); the outcome lag is added on top for a readable result. The detectable share after a chosen number of weeks is the X at which the required held-out leads equal the leads declined by then.

`pH = pB × (1 − X); k = (1 − h) ÷ h; p̄ = (pH + k·pB) ÷ (1 + k); nHoldout = [zα/2·√(p̄(1 − p̄)(1 + 1/k)) + zβ·√(pH(1 − pH) + pB(1 − pB)/k)]² ÷ (pB − pH)²; nBought = k × nHoldout`

## Frequently Asked Questions

### Why does a small holdout take so long?
The comparison is only as precise as its smaller group. A 10% holdout gives a cheap test because you decline few leads, but the held-out group fills slowly, so you need more weeks of traffic than a 50/50 split would.

### What incremental share should I try to detect?
The smallest share that would change your decision. If the source pays off only when at least 40% of its funded loans are incremental, size the test to detect 40%, not 10%: smaller effects need far more leads.

### Why add the outcome lag to the test length?
Funded loans arrive weeks after the lead. You can stop declining leads once you have enough, but you cannot read the result until the last declined borrowers have had time to fund elsewhere.

### What if I stop the test early?
The smallest effect you can detect gets bigger. A test stopped before it reaches the required size can still find a large effect, but a "no difference" result then says little about smaller ones.

## Read the article

[Attribution Models for Multi-Touch Lead Journeys](https://plato.ai/md/ai-attribution-multi-touch-leads.md)