Six ways a backtest lies, and the habit that catches them

Look-ahead is the one that hides best

Look-ahead is using information the strategy could not have had at the moment it traded, and it almost never looks like cheating while you are doing it. Index membership taken as of today rather than as of the trade date. Fundamentals stamped with the period they describe rather than the date they were filed. Macro series used at their current revised value rather than the value published that morning, which for payrolls and GDP can differ substantially.

This is exactly why the forecast machinery on this site stores an issue timestamp, the inputs as they stood at that moment, and a rule that states whether the initial release or the revised history controls. Point-in-time discipline is not bureaucracy, it is the only thing standing between a backtest and a very convincing hallucination.

Survivorship, and the strategies you did not report

A universe assembled from instruments that still exist has already deleted the failures. Funds close, tickers delist, tokens die, and a dataset that quietly drops them will show a strategy avoiding disasters it never actually avoided. The same applies to the country level: a long United States equity backtest is a study of the century’s most successful market.

The subtler version is multiple testing. Try twenty variants at a 5% significance threshold and one is expected to clear it on noise alone. Most published backtests are the survivor of an unreported search, which means the reported significance is not the significance. If you cannot state how many variants were tried, the result carries no information at all.

Costs are not a rounding error

Spread, commission, slippage, borrow cost, and tax. A strategy that rebalances daily across 252 trading days at a 5 basis point round trip is paying about 12.6% a year before it earns anything, and that is a modest cost assumption. High-turnover results that look extraordinary gross and merely good net are the normal case; results that stay extraordinary net usually have not modelled the costs at all.

Slippage in particular is not a constant. It grows with position size and it grows precisely when you most want to trade, because everyone else wants to trade then too. A backtest that fills at the close, in unlimited size, at zero impact, is describing a market that does not exist.

• Count the round trips per year and multiply by a realistic cost.
• Assume fills are worse in the periods the strategy makes its money.
• Report gross and net, and treat the gap as the strategy’s real fragility.
• Recheck the edge at ten times the position size.

The habit that catches most of it

Write the rule, the universe, the costs, and the evaluation window down before running anything, and do not touch them afterwards. Then run it forward in time you have not lived through. A holdout period you have already looked at is not out of sample; it is in sample with extra steps, because your knowledge of it shaped the rule.

This site holds itself to the same standard, which is why the public ledger starts at zero and refuses to backfill. Replaying history would have produced a record instantly and it would have been worth nothing. Our position: the only legitimate output of a backtest is a hypothesis plus the sample size needed to test it. Anything presented as a track record was not a backtest, it was an advertisement.