I keep coming back to permutation tests because they answer the one question every backtest quietly dodges. Your strategy made money in the test window. Fine. But would a coin flip with the same trade sizes and the same market have made money too? A Sharpe ratio does not tell you that. A pretty equity curve definitely does not. What you want is a way to generate thousands of fake versions of your strategy that keep everything real except the thing you think is your edge, then check where your real result lands in that pile of fakes. That is the whole idea, and it is more honest than most of the significance math people staple onto trading results.
The reason I trust this approach more than a textbook t-test is that market returns break almost every assumption a t-test needs. They are fat-tailed, they cluster in volatility, they are serially correlated, and your trades are not independent draws from anything. A permutation test does not assume any of that. It builds the null distribution out of your own data by scrambling it, so whatever weird structure lives in your returns comes along for the ride. You are comparing your strategy against a scrambled version of the exact same reality, not against a bell curve that never existed.
The three questions and the three shuffles
The mistake I made early on was treating permutation testing as one procedure. It is not. It is a family, and each member answers a different question. Pick the wrong shuffle and you get a p-value that looks rigorous and means nothing. Here is how I map them now.
- Shuffle the order of your trades. This asks whether the sequence of your wins and losses mattered, which is really a question about position sizing, compounding, and path dependence. If your equity curve only looks good because a few big winners happened to land early while you were sized up, shuffling the trade order will wash that out. If your average edge per trade is genuine, the total return survives reordering just fine because the same trades are still there, just in a different order.
- Resample your returns with replacement. This is the bootstrap, and it asks how fragile your result is to the specific set of trades you happened to get. You draw a new sequence of trades from your real trade returns, allowing repeats, and rebuild the curve. Do it thousands of times and you get a spread of outcomes. A strategy whose edge disappears when a handful of lucky trades get dropped is telling you something you need to hear before you fund it.
- Randomize the signal timing. This is the one that actually tests your signal. Keep the market returns exactly as they were, but fire your entries at random times instead of when your rule said. If a random-entry version of your strategy, holding for the same durations and sizes, does about as well as your real signal, then your signal is decoration. The market did the work and your rule just happened to be along for the ride.
Most people reach for the first two because they are easy and they feel scientific. The third is the uncomfortable one, and it is usually the one worth running.
Building the null distribution and reading the p-value
The mechanics are boring in a good way. You compute one number from your real strategy, call it the test statistic. It can be total return, Sharpe, profit factor, whatever you actually care about. Then you generate a few thousand shuffled versions using whichever scheme matches your question, compute the same statistic for each, and collect those numbers into a distribution. That distribution is your null, the world where your edge is not real.
The p-value is just the fraction of shuffled runs that did as well as or better than your real run. If you ran roughly ten thousand permutations and forty of them beat your real strategy, your p-value is around 0.004. Read it plainly. It means that if your signal had no edge, you would see a result this good by luck less than one time in two hundred. It does not mean your strategy has a 99.6 percent chance of working. That reversal is the single most common way people overclaim, and I have done it myself in a hurry.
A rule of thumb I use for the run count: at least a couple thousand permutations to get a stable read, and closer to ten thousand if the p-value is landing near your decision threshold. You cannot resolve a p-value of 0.01 with only five hundred shuffles, because your smallest measurable p-value is one over the number of runs.
Where it quietly lies to you
Permutation tests are honest about the question you ask and completely silent about the questions you forgot. A few failure modes have burned me or people I have worked with.
The big one is testing a strategy you already tuned on the same data. If you tried thirty parameter sets and kept the best, then permutation-tested that winner, your p-value is optimistic and you have no idea by how much. The shuffle knows nothing about the twenty-nine you threw away. The fix is to run the whole selection procedure inside each permutation, or to test on data you never touched during tuning. Otherwise you are measuring luck against a version of luck you already cherry-picked.
The second is destroying structure you meant to keep. If your returns have serial correlation and you shuffle individual bars, you break the autocorrelation and your null becomes easier to beat than reality, so everything looks significant. When the timing matters, block bootstrap methods that resample chunks of consecutive returns preserve more of that structure. Match the shuffle to what you are willing to assume is exchangeable.
The third is small samples wearing a lab coat. Forty trades will give you a p-value, and it will be almost meaningless because the null distribution built from forty trades is lumpy and unstable. A clean statistical procedure on thin data is still thin data.
A procedure you can copy
Here is the loop I actually run when someone hands me a backtest and asks whether it beats luck.
- Write down the exact question first. Is it the sizing, the trade selection, or the signal timing? That choice picks your shuffle before you touch any code.
- Compute your real test statistic once and set it aside.
- Generate at least a couple thousand permutations under the matching scheme, recomputing the statistic each time, and if you tuned parameters, run the tuning inside the loop.
- Count how many permutations matched or beat your real number, divide by the run count, and that is your p-value.
- Plot your real statistic as a vertical line against the histogram of the null. If it sits deep in the right tail, good. If it sits in the fat middle, your edge is probably the market's edge wearing your logo.
- State the claim in the weak, correct form. Not this works, but this would be unlikely under no edge, given this data and this test.
None of this proves a strategy will make money going forward, and I want to be clear about that. It only rules out the boring explanation that you got lucky in a fixed window. That is a smaller claim than most backtests pretend to make, and it is exactly the claim worth defending. When we build the strategy backtesting tools inside Blockcircle, this is the check we lean on hardest, because it is the one that survives contact with real market data instead of assuming the data is nicer than it is.
If you only take one habit from this, make it the signal-timing shuffle. Run your entries at random and see how much of your edge survives. Most of the time the answer is humbling, and it is a lot cheaper to be humbled by a histogram than by a funded account.