Backtest curves can look great—first check whether they’re “peeking at the answers”
When reviewing a strategy backtest, before looking at the return figures, ask: Could you really have accessed this data at the time?
The most straightforward example: A strategy decides to buy at 9 a.m. but uses data that won’t be released until that evening. The code may run just fine, and the curve may look great, but at 9 a.m. in the real world, that information wasn’t available. That’s bringing future data into the past.
The second problem is less obvious: trying lots of parameter combinations on the same stretch of historical data, then showing only the best-looking one. The process of picking the winner has already made use of the results from that period. Using the same data to prove that the strategy “works really well” is therefore less convincing.
So, when checking a backtest, keep at least three gates in place:
① Data timing gate. The date on an indicator isn’t enough; you also need to know when it was actually released and became available. Trading decisions can only use information that was known at the time.
② Sample separation gate. Keep the data used for designing and tuning the strategy separate from the data used for the final test. Once you repeatedly use the latter to change the rules, it’s no longer a clean, unseen exam.
③ Implementation conditions gate. Are trading costs, execution conditions, and sample coverage reasonable? List the constraints omitted from the model; don’t mistake omissions for things that don’t exist in reality.
A simple way to keep track: For every test, record the rule version, training period, test period, and what you changed. When you see a result improve, you can then tell whether the mechanism has become more reasonable or whether you’ve simply switched to a more favorable slice of history.
Good out-of-sample performance doesn’t mean you’re guaranteed to make money in the future. What it can do is reduce some forms of self-deception and provide clearer evidence that “this idea is worth researching further.”
A backtest is first and foremost a testing tool. If you keep making changes until you’re happy with the result, you may just end up memorizing the past.
#TradingReview #Backtesting #QuantitativeFinance
When reviewing a strategy backtest, before looking at the return figures, ask: Could you really have accessed this data at the time?
The most straightforward example: A strategy decides to buy at 9 a.m. but uses data that won’t be released until that evening. The code may run just fine, and the curve may look great, but at 9 a.m. in the real world, that information wasn’t available. That’s bringing future data into the past.
The second problem is less obvious: trying lots of parameter combinations on the same stretch of historical data, then showing only the best-looking one. The process of picking the winner has already made use of the results from that period. Using the same data to prove that the strategy “works really well” is therefore less convincing.
So, when checking a backtest, keep at least three gates in place:
① Data timing gate. The date on an indicator isn’t enough; you also need to know when it was actually released and became available. Trading decisions can only use information that was known at the time.
② Sample separation gate. Keep the data used for designing and tuning the strategy separate from the data used for the final test. Once you repeatedly use the latter to change the rules, it’s no longer a clean, unseen exam.
③ Implementation conditions gate. Are trading costs, execution conditions, and sample coverage reasonable? List the constraints omitted from the model; don’t mistake omissions for things that don’t exist in reality.
A simple way to keep track: For every test, record the rule version, training period, test period, and what you changed. When you see a result improve, you can then tell whether the mechanism has become more reasonable or whether you’ve simply switched to a more favorable slice of history.
Good out-of-sample performance doesn’t mean you’re guaranteed to make money in the future. What it can do is reduce some forms of self-deception and provide clearer evidence that “this idea is worth researching further.”
A backtest is first and foremost a testing tool. If you keep making changes until you’re happy with the result, you may just end up memorizing the past.
#TradingReview #Backtesting #QuantitativeFinance