Overfitting in backtesting, explained with a real example

Quantooo Lab · 2026-10-22

The more you tune a strategy on past data, the better its past results get. That is exactly the problem. Past data contains real patterns and a great deal of random noise, and a flexible enough rule will happily fit both.

What is overfitting?

Overfitting means a model or trading rule has learned the noise in its training data rather than a pattern that will repeat. It performs well on the history it was tuned on and poorly on new data. In trading, it is the main reason backtests look better than live results.

How does overfitting happen in practice?

Rarely through one big mistake. Usually through many small, reasonable-looking choices:

  • trying a 50-day average, then 100, then 200, and keeping the best;
  • adding a filter that removes the worst year;
  • testing ten indicators and combining the three that worked;
  • re-running after "fixing a bug" only when results disappointed.

Each choice uses the same data to make a decision. After enough of them, the strategy is shaped around that particular history.

Can you show a real example?

Our stock score combines five parts: trend, stability, fundamentals, market backdrop and news tone. Before freezing it, we tried 15 different weightings on 2020 to 2025 data. The best was "trend only", which beat equal weights by +0.0063 in rank correlation.

That sounds like a win. But the difference had a p-value of 0.26, which means a gap that size would appear by chance about a quarter of the time. Our rule, written down before testing, said a candidate must beat equal weights with p below 0.05, otherwise simple wins ties. So we kept equal weights.

Then we checked what would have happened if we had chased the best performer each year, a process called walk-forward selection. Picking the best weighting on earlier years and using it the next year gave an average rank correlation of −0.0093, worse than the +0.0024 from simply keeping equal weights on the same days. Chasing the past winner made things worse. That is overfitting in one number.

How do you detect overfitting?

  • Hold data back. Keep a final period untouched until every decision is made, then test once. Ours is a sealed holdout.
  • Count your trials. Report how many versions you tried, and adjust for it. The deflated Sharpe ratio estimates how likely the best of N attempts is to be real. Ours was 0.14 for the score, which is weak.
  • Compare with random. Run the same test with random rankings. If your best idea sits inside the range of random results, it is not an idea.
  • Look year by year. A strategy that works in one year and fails in the others is fitting that year.

How do you avoid overfitting?

Fewer knobs, rules written before testing, a strict bar for adding complexity, and a final test you only run once. None of this is exciting, and all of it is what separates a backtest from a story. Every trial we ran is listed on the research page, including the ones that looked good.

Is a simpler model always better?

Not always, but when a complex model and a simple one perform about the same, the simple one is more likely to keep working. It has fewer ways to have fitted noise. In our model comparison, neural networks with thousands of parameters did no better than a one-line trend rule.

Open questions for students and researchers

Questions without a single right answer. Pick one and share your reasoning in the discussion below.

  1. We tried 15 weightings. How would the deflated Sharpe change if we had tried 150? Derive the intuition before checking the formula.
  2. Walk-forward selection of weights did worse than fixed equal weights. Under what market conditions would adaptive selection beat fixed weights?
  3. Is 'simple wins ties' a statistical principle or a prior belief? Can you justify it formally, for example through model complexity penalties?

Debate: three perspectives

An illustrative debate written by the author to lay out the main viewpoints. The three voices are roles, not real people. Your view is welcome in the discussion below.

The Sceptic: Every backtest you see is the survivor of many attempts you never see. Discount all of them.

The Practitioner: You cannot avoid trying things. The discipline is to log every attempt, fix the bar before testing, and keep a final holdout. That is cheap and almost nobody does it.

The Academic: Overfitting is the finance version of the replication crisis in science. The remedies are the same: pre-registration, multiple-testing corrections and out-of-sample replication.

Which view is closest to yours, and what evidence would change your mind?