Why your AI-built backtest looks too good
The short answer
An AI can write a trading strategy in a few minutes. The backtest often looks excellent. Most of these strategies still lose money live, for four reasons that are easy to check.
“I have a strategy built by Claude and it works great on backtest but I have zero clue how it works.” (r/algotrading)
1. It was tested on the data it was built on
If you ask an AI to improve a strategy, it adjusts the rules until the results on your data get better. Each tweak makes the past look better. None of them prove the future will.
Check: keep the most recent part of your history aside. Build on the older part, then test once on the part you held back. If the result collapses, the strategy learned the past, not the market.
2. You tried many versions and kept the best
Try enough variations of anything and one will look great by luck. With two years of daily data, roughly eight independent attempts are enough to produce a result that looks like a good strategy from pure noise. An AI can try thousands in a minute.
Check: count how many versions you tried. The more you tried, the higher the bar the winner has to clear.
3. The test peeked at the future
This is the most common bug in AI-written code. A rule uses the day’s high to decide a trade taken in the morning, or a signal is calculated with data from a candle that had not closed yet. The backtest looks brilliant because it can see the future.
Check: for every input the strategy reads, ask whether you would have known that number at the moment of the trade.
4. Costs were left out
Spreads, commissions and slippage are small per trade and large in total. A strategy that makes a tenth of its risk per trade before costs often loses after them.
Check: charge every trade the real cost of the market and broker you use, then look again.
What a fair test looks like
A fair test does all four: it tests on history the strategy never saw, it accounts for how many versions were tried, it only uses information available at the time, and it charges real costs. Then the strategy runs live without trading, then on paper, before it touches real money.
That is what Fortuna’s Edge Engine does for every idea, yours or its own. Most ideas do not survive it. The ones that do are worth your attention.
Questions
What is overfitting in trading?
Overfitting is when a strategy has been adjusted so closely to past data that it describes the noise in that data rather than a real pattern. It looks excellent on the past and fails on new data.
How many trades do I need to trust a backtest?
As a rough rule, fewer than 30 trades tells you very little, and 100 or more is where results start to mean something. The more variations you tried, the more trades you need.
What is a walk-forward test?
A walk-forward test builds the strategy on one stretch of history and then scores it on the next stretch it has never seen, repeating this through time. It is the closest thing to trading the strategy live without risking money.
Talk to the chart.
Fortuna reads your chart, answers in charts, and keeps watching after you close the app.
Early access opens in waves. First in get a live alert on their own setup.