ChatGPT Can’t Backtest: Why “AI” Trading Bots With 10-Year Backtests Are Fiction

Here is a fact that eliminates most “AI trading bots” on the market in one sentence: a bot whose decisions require live calls to a language model cannot be backtested — because the model did not exist for most of the backtest window.

ChatGPT launched in 2022. A product showing you a flawless 2019 backtest for its “ChatGPT-powered” bot is showing you fiction — or, more charitably, a conventional strategy with AI branding. Real AI decisions require live API connectivity, cost money per call, and take seconds per decision (roughly 3–7 seconds end to end in published breakdowns). You cannot retroactively generate historical decisions from a model that did not exist, and you cannot backtest a system whose response times and model versions change underneath you. GPT-5 thinks differently from GPT-4. A backtest run on one is not evidence about the other.

The five red flags that expose fake AI EAs

Perfect long backtests. Five to ten years of flawless “AI-optimised” results. If the AI did not exist for those years, the backtest is not AI. Period.

No live record. No public wallet, no vault, no exchange-verified account. Only screenshots — which can be cropped from a cherry-picked window.

Hidden recovery systems. “Our AI intelligently scales positions for optimal recovery” is usually a martingale system doubling down on losses, described in softer words.

Annualised short windows. A good fortnight dressed as a triple-digit APR. Always ask: over what window, in what market, net of what fees?

No losing months shown. Every real trading record has losing months. A record without them is a curated exhibit, not a record.

How to validate a real AI trading system, then?

Forward testing. Since backtesting is impossible for genuine LLM-driven bots, the only honest validation is: demo for at least 30 days, then small live capital, with every trade logged and every cost counted. If a vendor refuses forward testing and offers only a backtest, that refusal is the answer.

The broader lesson applies to every trading claim, AI or not: recompute from raw data, beat the baselines, demand the trial count, check the folds. A claim that cannot survive those checks is marketing, whatever technology word is printed on it.

BuildIT Index applies that standard to any prediction file: recompute accuracy, beat baselines, check folds, count trials, return a verdict with an exit code.

If a vendor refuses forward testing and offers only a backtest, that refusal is the answer.