The thing that makes ML trading models dangerous is not that they are hard to build. It is that they are easy to build wrong in a way that looks spectacular. You train a classifier, it hits something like 90 percent accuracy on your held-out data, the equity curve goes up and to the right with a Sharpe you would be embarrassed to claim out loud, and then you put a dollar on it and it behaves like a coin flip with fees. Almost every time I have chased that gap, the answer was not the model. It was that some piece of information from the future had quietly leaked into training, and the backtest was grading the model on an exam it had already seen.
Leakage in a normal ML problem is annoying. In a trading problem it is close to inevitable, because time is the whole point. Every row in your dataset has a neighbor an hour later that it is not supposed to know about, and there are more ways for that neighbor to sneak in than most people expect. Here are the four paths I keep tripping over, each with a way to catch it and the pipeline shape that closes it.
Scaling and fitting on the full dataset
This is the most common one and the most forgivable, because every tutorial teaches it wrong. You load your features, you normalize them so the model trains nicely, you split into train and test, and you evaluate. The problem is the order. If you computed the mean and standard deviation, or the min and max, over the whole dataset before splitting, every one of those statistics contains information from your test period. Your scaler knows what the future volatility looked like. That knowledge bleeds into the training features through the normalization, and the model gets a faint whiff of what is coming.
The same trap hides in anything that fits parameters across all your data. A PCA rotation, a feature-selection step that ranks columns by their correlation with the label, an imputation that fills gaps with the column mean, a target encoding that averages the label by category. If it learns anything from the rows, it has to be fit on training rows only and then applied to test rows.
The detection test is simple and a little brutal. Take your pipeline, shuffle the target labels randomly so there is no real relationship left to learn, and run the whole thing. If your model still scores meaningfully above chance on the test set, you have leakage somewhere, because there is nothing legitimate left to predict. A clean pipeline on shuffled labels should score like a coin flip. The fix is to wrap every fitted transform inside cross-validation so it only ever sees training data, and for time series that means a walk-forward split where the test fold is always later in time than the train fold, never interleaved.
Features that quietly embed future bars
This one is sneakier because the feature looks completely reasonable. Say you build a feature that centers a moving average, or you compute a rolling z-score using a window that extends symmetrically around each point. A centered window at time t uses bars from t plus a few. That is the future. Or you resample a higher-timeframe indicator and forward-fill it in a way that stamps the closed daily value onto intraday bars before the day has actually closed. Or you use the high and low of the current bar to build a feature, then try to trade on the open of that same bar, which you could not have known yet.
The label side has the same disease. If your target is next-period return, you must make absolutely sure the feature row is timestamped strictly before the return window opens. One off-by-one shift, one .shift() in the wrong direction, and the model is reading the answer key.
The way I catch these now is to build the whole feature matrix, then delete every row after some cutoff date, recompute the features on the truncated data, and compare the surviving rows to what they were in the full run. A feature that only uses past information will produce identical values for those rows whether or not the future existed in the dataframe. A feature that peeks will change. Any column whose historical values shift when you add future data is leaking, full stop. It is a five-minute check and it has saved me more than once.
Overlapping labels that inflate everything
This is the one people almost never think about, and it is specific to how we label financial data. Say you label each bar by whether price is up 2 percent before it is down 2 percent within the next 20 bars, a triple-barrier style label. Two rows that are only three bars apart share seventeen bars of the future that determines their outcome. Their labels are not independent. They are heavily correlated by construction.
Now you do a random train-test split. Nearly every test sample has a near-twin sitting in the training set that shares most of its outcome window. The model is not generalizing, it is recognizing samples it has effectively already seen. Cross-validation scores look fantastic and mean almost nothing. You can spot it by checking how far your label windows reach and comparing that to the gap between your train and test folds. If a test row's outcome window overlaps any training row's outcome window, they are entangled.
The fix has two parts. Purge any training samples whose label window overlaps the test period, and add an embargo, a small gap of dropped samples right after the test fold, so serial correlation does not bridge the seam. If you want to weight it properly, down-weight samples by how much their label windows overlap with their neighbors, so a cluster of near-identical outcomes does not count as a cluster of independent evidence.
Tuning on the fold you are grading
The last one is subtle because it feels like good practice. You run a grid search or a Bayesian sweep over hyperparameters, you pick the settings with the best cross-validated score, and you report that score as your expected performance. The problem is that you used the test folds to choose the model, so the number you are reporting is the maximum over many tries, not an honest estimate. Search hard enough over enough parameters and thresholds and you will find a configuration that fit the noise in that particular split. Threshold tuning is the worst offender here, because picking the probability cutoff that maximized backtest profit is fitting one more parameter to the test set.
The honest structure is nested. An inner loop tunes hyperparameters on training data only, an outer loop touches each test fold exactly once to score the already-tuned model, and you never let the outer folds influence a single choice. It is more expensive and the reported numbers come out lower, which is exactly the point. If your Sharpe drops by half when you go nested, the other half was the search overfitting.
The audit list
When a model looks too good, I walk this before I believe any of it:
- Shuffle the labels and rerun. Above-chance on nonsense means leakage.
- Confirm every scaler, encoder, imputer, and selector is fit inside the CV loop on training rows only.
- Truncate the data, recompute features, and check that past-only feature values did not change.
- Verify the feature timestamp sits strictly before the label window opens, and check the shift direction by hand.
- Measure label-window length against the train-test gap, then purge and embargo the overlap.
- Make hyperparameter and threshold tuning nested, so the scoring fold never influences a choice.
- Sanity check that the strategy still works after realistic fees and slippage, because a lot of paper edge lives inside the spread.
None of this makes the model good. It just makes the number trustworthy, which is the harder problem. A mediocre strategy you can measure honestly is worth more than a brilliant one you cannot, because you can only size and improve what you can measure. When we run backtests inside Blockcircle, the walk-forward split with purge and embargo is baked into the harness for exactly this reason, so the equity curve you see is the one you would have actually traded. The rest of the time, the discipline is on you, and the tell is always the same. If it looks too good live is where you find out it was.