How to Audit a Vendor’s Backtest Before You Pay a Single Rupee
You are not the one who built the strategy — you are the one being asked to pay for it. That makes you the auditor. Here is the buyer-side checklist, in order.
Almost everything written about backtest validation is written for the person building the strategy. But if you are considering paying for one — a bot, a signal service, a course strategy, a “proven” model — you are not the builder. You are the buyer. And the buyer needs a different checklist, because the builder’s incentives run the other way.
Here it is, in order. Any vendor who fails the first three is done.
1. Show me the raw predictions
Not a screenshot. Not an equity curve. The raw file: predictions with timestamps, and the actual outcomes. If you cannot recompute the accuracy yourself — or have it recomputed — from raw predictions, you have nothing verifiable. A claimed-accuracy column in a vendor’s file is the easiest number in finance to inflate, which is why any serious check ignores it and recomputes from the predictions themselves.
2. How many did you try — and where are the losers?
The trial count is the single most predictive question in this whole field. A vendor who tried 500 versions and shows you the best is showing you a lottery winner. Ask for the number, and ask to see the discarded versions. A vendor who cannot or will not answer has answered.
3. What do the dumb baselines score?
Always-bullish and no-change, on the same data. If the vendor’s model cannot beat both, it adds nothing — it is riding the base rate. On a bare CSV with no price history, those two baselines are the honest ones; anything claiming five baselines without price history is faking three of them.
4. Show me the folds, not the average
A single overall accuracy hides decay. Ask for walk-forward results fold by fold, earliest to latest. A model decaying from 80% to 56% across folds is a dying edge with a flattering average. Also ask which version of “walk-forward” they mean: period stability (accuracy across time periods) or purged, embargoed training (leak-proof fold construction). Vendors who say “walk-forward validated” without saying which have told you nothing precise.
5. What did it cost — in the backtest?
Commissions, spread, slippage, taxes. A backtest with zero costs is a fantasy. Ask for net-of-cost results, and ask at what slippage the edge dies (the break-even cost per trade). If the edge survives only at zero friction, it does not survive.
6. What market regime was this?
A strategy that only worked in one regime — one trend, one volatility environment — is a bet on that regime repeating. Ask for results sliced by year and by regime. Concentration in one slice is a warning, not a track record.
7. Can I forward-test before paying?
The strongest answer a vendor can give is: “Here is a forward-test window — paper-trade it live for 30 days and watch.” The weakest is pressure to pay now for a backtest you cannot reproduce. Time is the cheapest audit there is.
The sentence that ends most pitches
“I’d like the raw prediction file, the trial count, the baseline scores, and the fold-by-fold results. If those check out, I’ll forward-test before paying.”
Most vendors will not survive that sentence. The ones who do are the only ones worth a second conversation — and they are rare, which is exactly why the sentence works.
BuildIT Index exists so you can run that sentence mechanically: hand it a prediction file, get a recomputed accuracy, baseline scores, walk-forward folds, leakage gate, multiple-testing correction, calibration, and a verdict with an exit code. The tool does the audit; you make the decision.
The buyer who asks the right questions rarely needs to pay for the wrong answer.
This article is educational, not financial advice. Past backtest results do not predict future returns.