How do I calculate the payback period of a machine learning model?

Treat it as cash flow. Spend the build cost up front, then each month add the extra margin the model earns over your rules and subtract what it costs to run. The payback month is when the running total first gets back to zero. In the default example, a $150,000 build that adds $27,000 a month at full uplift and costs $15,000 a month to run pays back in month 15.

Why does the model never pay back in some cases?

Because the running cost is a monthly bill, not a one-off. If the margin uplift on your volume is no bigger than the people and infrastructure needed to keep the model running, every month adds to the loss and no build budget can fix that. More decisions, more margin per decision or a cheaper way to run the model can.

Why does the holdout reduce the gain?

Holdout traffic stays on the old rules, so it earns no uplift. That is the price of proving the lift rather than asserting it: comparing the model with the holdout is how you find out whether the uplift you entered is real.

What uplift should I enter?

Your own estimate, and then a lower one. The uplift drives everything, and before a model is live it is a guess. Try the number at which payback stops being worth it, then measure the real uplift against a holdout once the model is running. The default inputs are an illustrative example, not a benchmark.

All tools

Model Payback Calculator

If a model makes better decisions than your current rules, how long until it pays back what it costs to build and run?

Enter how many decisions your rules make each month, what each is worth and how much better a model would do. The calculator works out when the model pays back what it costs to build and run. The defaults are an illustrative example, not a benchmark.

Decisions per month (leads bought, routed or priced)20,000Average margin per decision today$25Expected uplift in margin from the model6.0%Build cost (one-off)$150,000Running cost per month (people, infrastructure)$15,000Ramp-up months before full uplift3 monthsShare of traffic kept on the rules as a holdout10%

Monthly gain at full uplift

$27,000

Net monthly gain after running cost

$12,000

Payback

Month 15

Net return after 12 months

−$33,000

Net return after 24 months

$111,000

The line is cumulative cash flow: the build cost is spent in month 0, the model goes live in month 1, and the uplift ramps up in a straight line to its full value in the last ramp month. Running costs are paid in full from month 1. Holdout traffic stays on the rules, so it earns no uplift; that is the price of being able to prove the lift rather than assume it. The uplift is the number to be most sceptical of: measure it against the holdout before you count on it. Cash is not discounted.

How it works

Replacing hand-written rules with a model is an investment: a one-off build, a monthly running cost, and a margin uplift that takes a few months to arrive. Enter how many leads your rules buy, route or price each month, the margin on each, the uplift you expect and what the model costs, and the calculator returns the payback month, the net return after 12 and 24 months and the cumulative cash flow. The default inputs are an illustrative example, not benchmarks.

At full uplift the model adds decisions per month × margin per decision × uplift, but only on the share of traffic it decides; the holdout stays on the rules. The uplift ramps up in a straight line, reaching its full value in the last ramp month (no ramp means full uplift from month 1). Each month's net gain is the ramped gain minus the running cost. Cumulative cash flow starts at minus the build cost in month 0 and adds each month's net gain; payback is the first month it reaches zero. If the full gain is no more than the running cost, it never pays back. Cash flows are not discounted.

G = N\,m\,u\,(1 - h)

r_t = \min(t / T,\ 1)

\mathrm{CF}_0 = -B

\mathrm{CF}_t = \mathrm{CF}_{t-1} + r_t\,G - R

t^{*} = \min\{\,t \ge 1 : \mathrm{CF}_t \ge 0\,\}

where

$N$: decisions per month
$m$: average margin per decision under the rules
$u$: uplift in margin from the model
$h$: share of traffic in the holdout
$G$: monthly gain at full uplift
$T$: ramp-up months (full uplift from month 1 if T = 0)
$r_t$: share of the full uplift reached in month t
$B,\ R$: build cost, running cost per month
$\mathrm{CF}_t$: cumulative cash flow after month t
$t^{*}$: payback month (never if G ≤ R)

Frequently asked questions

How do I calculate the payback period of a machine learning model?: Treat it as cash flow. Spend the build cost up front, then each month add the extra margin the model earns over your rules and subtract what it costs to run. The payback month is when the running total first gets back to zero. In the default example, a $150,000 build that adds $27,000 a month at full uplift and costs $15,000 a month to run pays back in month 15.
Why does the model never pay back in some cases?: Because the running cost is a monthly bill, not a one-off. If the margin uplift on your volume is no bigger than the people and infrastructure needed to keep the model running, every month adds to the loss and no build budget can fix that. More decisions, more margin per decision or a cheaper way to run the model can.
Why does the holdout reduce the gain?: Holdout traffic stays on the old rules, so it earns no uplift. That is the price of proving the lift rather than asserting it: comparing the model with the holdout is how you find out whether the uplift you entered is real.
What uplift should I enter?: Your own estimate, and then a lower one. The uplift drives everything, and before a model is live it is a guess. Try the number at which payback stops being worth it, then measure the real uplift against a holdout once the model is running. The default inputs are an illustrative example, not a benchmark.

Read the articleFrom Heuristics to Algorithms: Moving Beyond 'If-Then' RulesIntuition gets you started, but algorithms get you to the finish line. Transition your marketing from rigid rules to fluid models.

Strategy