AI and Backtesting: The Risk of Overfitting
What is a backtest?
A simulation of how a strategy would have performed on historical price data. In AI and algorithmic systems, new ideas are usually tried on past data first.
What is overfitting?
Bailey, Borwein, López de Prado and Zhu (2014) show that trying even a relatively small number of strategy configurations on historical data can produce impressive but misleading simulated results; the more configurations are tried, the more likely a backtest is overfit[1]. The authors also note that the number of configurations tried is rarely disclosed, so investors often cannot judge how overfit a proposal is[1].
Out-of-sample expectation
The same paper argues that, under memory effects, overfitting can lead to negative rather than zero expected out-of-sample returns[1]. A rule that looked excellent in the past can therefore perform worse than average in real time.
A checklist for AI-assisted analysis
Record how many rules and parameters were tried; test results on a separate out-of-sample period; account for commission, slippage and spread; watch on paper before using real money; do not draw firm conclusions from small samples. These are general principles and do not guarantee success.
How to read AI output
AI summaries and model outputs are decision aids; if the source and data date are unknown they should not be used without verification. NX24 content is not investment advice and the limits of model outputs are stated openly; see the Editorial Principles page.
Disclaimer
Past performance is not indicative of future results. This page is educational, not investment advice.
Frequently asked questions
- What is overfitting?
- Fitting a strategy too closely to past data, so that a rule that looked good in the past does not repeat its success in real time[1].
- Why can a backtest mislead?
- When many configurations are tried, the chance of finding a combination that looks good by luck rises, and the number tried is rarely disclosed[1].
- Can AI predict markets?
- No method guarantees returns. The source for this page examines when backtest results can be unreliable; AI outputs are subject to the same limits[1].
Related guides
Sources
- D. H. Bailey, J. Borwein, M. López de Prado, Q. J. Zhu. Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on Out-of-Sample Performance. Notices of the American Mathematical Society 61(5), 2014. Accessed: 2026-10-22.