PBO, Explained in Plain English: The Probability Your Best Backtest Is a Fluke

You ran 8 versions of a strategy. Seven were mediocre. One was brilliant. You ship the brilliant one. Question: did you find an edge, or did you run a lottery and crown the winning ticket?

That is exactly what PBO — the Probability of Backtest Overfitting — measures. And it is one of the least explained ideas in trading, because the people who understand it write papers, and the people who write blog posts mostly skip it. Here it is in plain English, with a worked example.

The core idea

PBO does not ask “is this strategy good?” It asks something sneakier: if you pick the best performer from a set of candidates, how likely is it to disappoint out of sample? It answers by splitting history into many pieces, ranking the candidates in each piece, and checking whether the in-sample winner keeps winning out of sample — or whether the winner keeps changing, which is what luck looks like.

The formal method is called Combinatorially Symmetric Cross-Validation (CSCV). You do not need the maths to use the intuition: a real edge is stable. A fluke is a rotating cast of winners.

A worked example

Imagine 8 configurations of the same strategy, tested over 4 years.

In-sample, configuration #5 wins: Sharpe 2.1, best by a clear margin.

Split history into halves, quarters, and many combinations. In most out-of-sample slices, #5 is mid-pack. The winner changes from slice to slice: #2 here, #7 there, #5 rarely.

PBO comes out at 0.62. Interpretation: a 62% chance that the in-sample winner underperforms out of sample — the selection process itself is likely what produced the “winner.”

Now the same setup, different outcome: configuration #3 wins in-sample, and it also ranks near the top in most out-of-sample slices. The winner is stable. PBO comes out at 0.14. That does not prove the strategy will make money — nothing proves that — but it does mean the selection was probably not a lottery.

As rough bands used by validation services: below about 0.20 looks clean, 0.20–0.50 is caution, above 0.50 means the winner is more likely than not a fluke. Treat the bands as guidance, not law.

The two honesty notes nobody states

First, PBO needs a candidates file — the full set of versions you tried. With a single model and no candidates, PBO is null. It is not zero, it is not “passed,” it is undefined, and any tool that reports a PBO for one lonely model is inventing a number. This is why BuildIT Index states it plainly: PBO is null for single models, and never a gate.

Second, PBO is only as honest as the candidate list. If you tried 200 versions and hand over 8, the computation is being fed a curated history. The trial count problem never goes away; it just moves. Demand the full list, or discount the score.

How to use PBO

Before you trust a “best” strategy, ask for its PBO and the candidate count behind it.

If PBO is above 0.50, the correct response is not to tune harder — it is to suspect the whole selection process and start over with fewer degrees of freedom.

If a vendor cannot produce a PBO at all, that is not a fail — but ask why, and ask what else they ran instead: walk-forward folds, deflated Sharpe, a leakage gate. One missing check is fine. All of them missing is the answer.

A backtest that survives recomputation, beats its baselines, holds up fold by fold, and carries a low PBO has earned a forward test. A “winner” with a high PBO has earned nothing but suspicion — and suspicion, in this business, is free.

A backtest that survives every check has earned a forward test. A backtest that fails one has saved you the money you would have spent learning it live.