The permutation test: telling an edge from a coincidence

You have 250 trades and a mean return of +0.18% per trade. Is that an edge, or is it what 250 coin flips look like when you only look at the ones that landed well?

The permutation test answers that question by building the distribution of the result under the assumption that there is no edge, then asking how often chance produces something at least as good.

The null, made concrete

Under the null, the sign of each trade’s return is arbitrary. If a trade made +1.2%, the same market move could just as easily have produced -1.2%. So take the 250 returns, flip each sign with probability one half, and compute the mean. Repeat a few thousand times. That collection of means is the distribution of “no edge”.

The p-value is the share of shuffled means at least as large as the one you observed. If 2% of shuffles match or beat it, p is about 0.02.

Why sign flipping and not bootstrapping

A bootstrap resamples your observed returns, so the edge you are testing for is inside the resampled distribution. It estimates the sampling variability of the mean, not the probability of the mean under no edge. Sign flipping destroys the direction while preserving the magnitude, which is exactly the hypothesis you want to reject.

That distinction is the whole test. Use the wrong null and you get a confident answer to a question nobody asked.

The false-positive rate

Simulate 500 strategies with zero edge. Each has 250 trades drawn from a normal distribution with mean zero and 1% volatility. Run 2,000 sign-flip shuffles and call anything with p < 0.05 significant.

• False positives at p < 0.05: 2.0% of 500 zero-edge strategies.

That is slightly conservative, which is expected. The p-value from a finite number of shuffles is discrete, and the estimator (k+1)/(n+1) cannot return zero. A test that fires below its nominal rate on pure noise is working correctly.

The power

Now give the same setup a real edge: 0.25% per trade against 1% volatility, about 0.25 in per-trade Sharpe terms.

• Detection rate at p < 0.05: 98.6%.

So the test is not the bottleneck. At 250 trades and a 1% daily standard deviation, a 0.25% per-trade edge is almost always found. When traders fail to detect a real edge, the cause is a short sample, a small edge, or costs that eat the difference before the test sees it.

What the test does not do

• It does not correct for multiple testing. Run 20 strategies at p < 0.05 and roughly one will pass on noise alone. Correct the threshold, or use a deflated Sharpe ratio.
• It does not model costs. Test net returns. A gross edge the test can see may be invisible after spread and slippage.
• It does not handle non-stationarity. If the edge existed for six months and vanished, the full-sample mean understates the early period and overstates the late one. Test in blocks, and read the folds rather than the average.
• It does not tell you the size of the edge. Significance and magnitude are different questions, and only one of them pays.

How to run it

Use at least 1,000 shuffles, preferably 10,000 when the sample is small. Report the p-value with the shuffle count. Never report p = 0; the smallest honest value from n shuffles is 1/(n+1). And report the observed mean next to the p-value, because a statistically significant 0.01% edge is still smaller than the spread.

If the sample is short, the more useful question is how many trades you would need. The sample-size calculator turns a win rate into the interval it actually supports; the Wilson interval explains why a 70% win rate over 20 trades cannot be distinguished from a coin flip.

The rule

A p-value answers one question: how often would chance produce this? It never answers whether the edge is big enough to trade.

Run the test before you run the strategy. A record that fails a permutation test is not a small edge with a wide interval; it is a result that chance explains without help.

How these numbers were produced

Seeded simulation: Mulberry32(31337), 500 independent runs of 250 trades, 2,000 sign-flip shuffles per run, 1% daily volatility. The zero-edge runs use mean zero; the edge runs add 0.25% per trade. Re-running with that generator reproduces these figures exactly.

Questions

What is a permutation test in trading?

A test that builds the distribution of a result under the assumption of no edge by shuffling the data, then measures how often chance produces a result at least as good as the observed one.

Why flip signs instead of resampling returns?

Because the null is no directional edge. Flipping signs keeps each trade’s magnitude and destroys its direction, which is the hypothesis being tested. A bootstrap keeps the edge inside the resampled distribution.

How accurate is the permutation test?

In 500 simulated zero-edge strategies with 250 trades and 2,000 shuffles, 2.0% were flagged at p < 0.05. A real 0.25% per-trade edge was detected 98.6% of the time.

How many shuffles do I need?

At least 1,000, and 10,000 when the sample is small. Report the shuffle count with the p-value, and never report p = 0, because the smallest honest value is 1/(n+1).

Does a significant p-value mean the strategy is profitable?

No. It means the mean return is unlikely under no edge. Costs, multiple testing and non-stationarity sit outside the test, and a significant edge can still be too small to trade.