CSCV and PBO: is your best strategy just the luckiest loser?

You tested twenty strategies. One has a Sharpe of 1.8 and the other nineteen look ordinary. The usual next question is whether 1.8 is good. The better question is whether the winner was selected or discovered.

The Probability of Backtest Overfitting (PBO) answers the second question. It is computed with Combinatorially Symmetric Cross-Validation (CSCV), and it needs the full matrix of returns for every strategy you tried — including the ones you deleted.

How CSCV works

Split the return history into S contiguous blocks. For S = 10, take every way of choosing 5 blocks as in-sample. That is 252 splits, each leaving the other 5 blocks as out-of-sample.

For each split: rank the strategies in-sample, take the winner, then find that same strategy’s rank out-of-sample. Convert the rank to a relative position and take the logit. If the logit is at or below zero, the in-sample winner landed in the bottom half out-of-sample.

PBO is the share of splits where that happens. A PBO of 0.60 means the winner underperforms the median out-of-sample in 60% of splits.

The numbers

Twenty strategies, 1,000 observations, 1% daily volatility, 252 splits. The only thing that changes between rows is whether any strategy has a genuine edge.

• All 20 zero-edge — PBO 0.607, mean out-of-sample rank of the in-sample winner 9.3 of 20.
• One strategy with a 0.05% daily edge — PBO 0.313, mean rank 13.5 of 20.
• One strategy with a 0.20% daily edge — PBO 0.000, mean rank 20.0 of 20.

The first row is the point. With twenty strategies that have no edge at all, the in-sample winner lands at the median out-of-sample. Rank 9.3 out of 20 is what a coin flip looks like. The strategy that looked best was best because it was measured on the data that made it look best.

What a small edge buys you

A 0.05% daily edge — about 13% annualised against 16% volatility — lifts the mean rank to 13.5 and drops PBO to 0.313. That is a real improvement and still not a pass. In roughly a third of splits, the winner is below the median out-of-sample.

At 0.20% daily, the true strategy wins every split and lands last out-of-sample every time. PBO collapses to zero. That is what a dominant edge looks like in this framework: not a high Sharpe, but a strategy that keeps winning when the data is rearranged.

What PBO does not tell you

• It does not validate the winner. A low PBO means selection was probably not a lottery. The winner can still lose money after costs.
• It does not correct for costs. Feed it net returns. A matrix of gross returns produces a confident answer about a strategy you cannot trade.
• It does not fix a biased candidate set. If the twenty strategies are all variants of one idea, PBO measures selection within that idea, not across the space you actually searched — which is the trial-count question in another form.
• It needs the full matrix. Deleting the losers is the failure mode PBO exists to detect, which makes it the one check you cannot run on a vendor’s summary page.

Implementation notes

Use an even number of blocks, at least 8, and contiguous blocks rather than shuffled observations, so each block preserves local market conditions. Ten blocks give 252 splits, which is enough to estimate a proportion. Twenty blocks give 184,756 splits and a better estimate, at the cost of fewer observations per block.

Report the PBO next to the mean out-of-sample rank. The rank is easier to read and harder to game: “the winner was, on average, 13th of 20 when the data changed” is a sentence a buyer understands.

PBO and the deflated Sharpe ratio are complements. The DSR asks whether one strategy’s Sharpe survives the number of trials; PBO asks whether the selection process picks a strategy that persists. The plain-English PBO explainer covers the concept; this page covers the calculation and its calibration.

The rule

A backtest tells you how good the winner looked. CSCV tells you how much of that was the looking.

If a vendor cannot supply the full candidate matrix, PBO is not computable. That is not a technicality. It means the one number that would separate skill from selection is the one number being withheld.

How these numbers were produced

Seeded simulation: Mulberry32(4242), 20 strategies, 1,000 daily observations, 1% volatility, 10 contiguous blocks and all 252 in-sample splits. The edge rows add 0.05% or 0.20% to the mean of one strategy only. Re-running with that generator reproduces these figures exactly.

Questions

What is PBO?

The Probability of Backtest Overfitting. It estimates how often the best strategy from a set of candidates underperforms the median out-of-sample, which measures the selection process rather than any single strategy.

How is PBO computed?

With Combinatorially Symmetric Cross-Validation. Split history into blocks, take every way of choosing half as in-sample, rank the candidates, and check where the in-sample winner lands out-of-sample. PBO is the share of splits where it lands in the bottom half.

What does a PBO of 0.61 mean?

In 61% of splits the in-sample winner underperformed the median out-of-sample. In a simulation of 20 zero-edge strategies, PBO was 0.607 and the winner’s mean out-of-sample rank was 9.3 of 20.

What PBO value is acceptable?

There is no universal threshold. Below 0.20 is clean, 0.20 to 0.50 is caution, and above 0.50 means the winner is more likely than not a selection artefact. Report the mean out-of-sample rank alongside it.

Does a low PBO prove the strategy works?

No. It means selection probably did not produce the winner. Costs, capacity and regime change sit outside the calculation, and a low PBO can still belong to a losing strategy.