Quant backtesting checklist - 12 checks before you trust a result

Quantooo Lab ยท 2026-10-22

A backtest is easy to run and easy to get wrong. Most impressive-looking results fail one of a handful of checks. Here is the list we run every test through, with the mistake each check catches.

What should a quant backtesting checklist include?

At minimum: point-in-time data, a tradable timeline, realistic costs, walk-forward evaluation, a count of every variation tried, comparison with random controls, results by year, concentration checks, and a final sealed holdout. The twelve checks below cover each one.

The data checks

  1. Is every input point-in-time? Each value must be used only from the moment it was public: filing dates for company results, release dates for economic data, publication times for news. Prevents look-ahead bias.
  2. Does the universe include companies that later disappeared? Using today's index members for past years removes the failures. Prevents survivorship bias.
  3. Are prices adjusted correctly? Splits and dividends must be handled consistently, or returns jump on corporate actions.

The timeline checks

  1. Can you actually trade when the backtest says you trade? Decide with data up to one day and trade at the next open, not at the close you used to decide.
  2. Is the evaluation walk-forward? Train on the past, test on the next period, roll forward, with a gap at least as long as your return horizon. Prevents leakage from random splits (walk-forward testing).

The cost checks

  1. Are trading costs included? Charge a realistic cost per trade and report turnover. Prevents strategies that only win on paper (trading costs).
  2. Does it survive double costs? If doubling costs flips the result, the edge is too thin to trade.

The statistics checks

  1. How many variations were tried? Count them all and adjust, for example with the deflated Sharpe ratio. Prevents overfitting by selection.
  2. Does it beat random controls? Run the same test with random rankings and shuffled labels. A real signal sits clearly outside their range.
  3. Is it consistent across years? One great year and five flat ones is luck, not a strategy. Use a t-statistic that accounts for overlapping returns.
  4. Does it depend on a few stocks? Report the share of profit from the top three names, and the result without them.

The final check

  1. Was a final period sealed and opened once? Freeze everything, then test on data no decision was allowed to see. One look only. This is the closest a backtest gets to live trading (sealed holdout).

What happens when a result fails the checklist?

Usually it disappears. Our own stock score passed the data, timeline and cost checks, was indistinguishable from random in development, and was significantly negative on the sealed holdout. Twelve machine learning models went through the same checklist and none survived. We publish those results on the research page and model leaderboard because a checklist only means something if failures are reported too.

After the checklist, what next?

Run it live with every decision logged before the outcome is known. Our track record fingerprints each day's scores before the market opens and timestamps them on the Bitcoin blockchain, so the live record cannot be edited after the fact.

Open questions for students and researchers

Questions without a single right answer. Pick one and share your reasoning in the discussion below.

  1. Which three checks on this list would you drop if you had only one day? Defend your choice.
  2. Suggest a thirteenth check that catches a failure none of the twelve would. Give an example of the failure.
  3. Could a checklist itself become a source of overfitting, by passing results that are tuned to pass checklists? How?

Debate: three perspectives

An illustrative debate written by the author to lay out the main viewpoints. The three voices are roles, not real people. Your view is welcome in the discussion below.

The Sceptic: Checklists give false comfort. A determined researcher can pass every box and still fool themselves.

The Practitioner: Checklists catch the common mistakes, which are most mistakes. The rare clever ones are why you also need a live record.

The Academic: Checklists work in medicine and aviation because they reduce routine errors, not because they guarantee outcomes. Backtesting is no different.

Which view is closest to yours, and what evidence would change your mind?