The number that scares me most in strategy research is not the Sharpe ratio on the winner. It is the count of everything I tried before I found it. If I run one backtest and it clears a significance bar, that means something. If I run two hundred and pick the best one, the bar means almost nothing, and I have quietly turned a statistics test into a lottery. This is data snooping, and it is the single most common way a smart, careful person ends up trading a strategy that was never real.
The math is not subtle once you look at it. Suppose you use the usual five percent threshold, so a truly worthless strategy still looks significant roughly one time in twenty by chance. Test one strategy and you have a small, honest risk of fooling yourself. Test a hundred independent worthless strategies and you should expect around five of them to clear the bar anyway. Not because they work. Because you asked the question a hundred times and randomness answered yes a handful of them. The winner you are so pleased with is often just the luckiest coin in a big pile of coins.
Why your true trial count is always bigger than you think
The honest trial count is rarely the number of backtests you clicked run on. It includes every variation you considered, even the ones you tried in your head or tweaked and reran. Every lookback window you swept. Every asset you added and dropped. Every entry threshold you nudged from 70 to 65 to 60 because 70 looked weak. Every time you changed the stop and re-ran. All of that is testing, and all of it spends your statistical budget.
There is also the snooping you inherit. If you read a paper or a thread that already searched ten thousand parameter combinations and reported the best, and you go verify that exact combination on the same data, you are standing on top of their search without paying for it. Your one clean-looking backtest is really their ten thousand. This is why replicating a published edge on the original sample almost never tells you anything. The only real test is fresh data the search never touched.
A concrete failure mode I have watched more than once: someone builds a mean-reversion signal, it looks flat, so they add a trend filter. Still flat, so they add a volatility filter. Now it looks great. They report the three-filter version as if it were one idea they had up front. In reality they ran a small search and kept the survivor, and the equity curve is fitted to the noise in that specific window. Out of sample it reverts to flat, or worse.
The corrections, from blunt to proper
Once you accept that the trial count matters, there are a few standard ways to adjust for it. They differ in how much they punish you and how much they know about your strategies.
- Bonferroni. The blunt instrument. Divide your significance threshold by the number of trials. If you want five percent overall and you ran twenty strategies, each one now has to clear roughly a quarter of a percent to count. It is easy, it is conservative, and it assumes your trials are independent, which they usually are not. Twenty variations of the same momentum idea are highly correlated, so Bonferroni over-punishes and you throw away strategies that might be fine. Still, as a back-of-envelope sanity check it is hard to beat. If your winner cannot survive a rough Bonferroni cut, be suspicious.
- Holm and step-down variants. Slightly smarter versions that rank your p-values and apply a graded threshold instead of the same harsh cut to everyone. You get a bit more power without giving up much rigor. Reach for these when Bonferroni feels too brutal but you still want something simple.
- White's Reality Check and the Hansen SPA test. These are the ones built specifically for this problem. Instead of assuming independence, they use a bootstrap. You resample your returns many times to build the distribution of the best strategy's performance under the null that none of them have real edge, accounting for how correlated your strategies are with each other. Then you ask where your actual best performer falls in that distribution. If it is not out past the tail, the best result you found is consistent with pure luck given how many bets you placed. Hansen's version corrects a known weakness where padding your set with obviously bad strategies could make a mediocre winner look better than it should.
The bootstrap approach is more work and it needs your full set of trial returns, not just the winner, which is exactly why logging matters. You cannot run a reality check on trials you never wrote down.
The research ledger, which does most of the work
All of the corrections above depend on one honest number: how many things you actually tried. The corrections are the easy part. Keeping that number honest is the discipline nobody wants to do, and it is where the real edge in research hygiene lives. So I keep a ledger. Every trial gets a row, logged at the moment I run it, before I know the result. Logging after you see the outcome is how the losers quietly disappear from the record and the count shrinks to flatter you.
A workable ledger row does not need to be fancy. Mine has:
- A timestamp and a short hypothesis in plain words, written before the run. If I cannot state what I expect and why, I am fishing, not testing.
- The exact parameters and the data window, including which assets and which date range.
- The result, whatever it is, including the ugly ones. Especially the ugly ones.
- A running count of trials against this same dataset, so I always know my current denominator.
- A note on whether this run reused data I have already searched, because if it did, the trial count that matters is cumulative, not one.
The point of the denominator is that it feeds the significance test directly. When I finally find something promising, I do not ask whether it beats the bar. I ask whether it beats the bar given that I have now run, say, eighty trials on this data. Often the answer is no, and the ledger is the only reason I know it. The ledger turns a vague sense of I have been at this a while into a number I can actually plug into a correction.
So how many backtests is too many
There is no magic ceiling, but there is a working rule. You are allowed as many trials as your out-of-sample data and your corrections can honestly support. If you are going to run a hundred variations, you need either a real multiple-testing correction on the front end or a genuinely untouched holdout on the back end, and ideally both. What you cannot do is run a hundred, pick the best, and report it as if it were the only one.
The cheapest habit that protects you is holdout hygiene. Split your history early, before you start searching. Do all your fishing on the training portion, log every trial, and keep the final slice sealed. When you have a candidate, it gets exactly one look at the sealed data. One. If it survives that single honest test, you have something. If you find yourself peeking at the holdout, tweaking, and peeking again, you have just made the holdout part of your training set and spent it. At that point you need fresh data, not another peek.
None of this makes strategy research safe. It just makes it honest, which is the most you can ask of a process that is mostly you trying to tell signal from noise on a finite pile of numbers. When we built the backtesting tools inside Blockcircle, the feature I care about most is not the optimizer, it is the trial log sitting quietly underneath it, counting. The optimizer finds the winner. The log is what tells you whether to believe it.