Originally posted by Hristo H, here: https://www.linkedin.com/feed/update/urn:li:activity:7450440055514570752/
The 9 Deadly Sins of Backtesting (and How to Fix Them) - Part 2
Robust modeling requires moving beyond simple performance metrics to identify false positives. Even clean data can be fit to a mirage.
5. Regime Analysis Markets are non-stationary; states characterized by specific volatility shift abruptly. A "bull hero" strategy that thrived in 2020 often bleeds out during inflationary, rising-rate regimes like 2022. Fix: Stress test across multiple 10+ year cycles and extreme events (1987, 2008, 2020) to ensure survival through non-stationary shifts.
6. Overfitting (Curve-Fitting) Tuning 50 parameters fits the noise of the past, not a repeatable signal. As John von Neumann said, with enough parameters, you can even make an elephant wiggle its trunk. Fix: Keep parameters in single digits—I recommend up to 5. Limit exploration to a strict 8–32 hour window. If out-of-sample performance drops below 70% of in-sample, the strategy likely lacks predictive power.
7. Multiple Testing Fallacy Testing 200 strategies and only reporting the winner is an "error of selection". It is like a blindfolded archer firing 1,000 arrows and only showing the one that hit the bullseye. An impressive Sharpe ratio is often just a "random artifact" of luck. Fix: The "Statistical Tax." Raise your standard of proof based on total trials. For 200 tests, a 2.0 t-stat is insufficient; you need 3.66+ to prove skill. Use the Superior Predictive Ability (SPA) test or BHY adjustments to "haircut" results and ensure the winner isn't just a mathematical fluke.
8. Short-Selling Reality Most backtests assume you can short for free. In reality, you face high fees and "hard-to-borrow" lists. I faced this on Binance Spot—the backtest was profitable, but I couldn't actually borrow the assets, and no historical record of borrow availability existed. Fix: Add a shortable provider model to your backtest to verify actual share availability and margin interest costs.
9. Implementation and AI Risk AI models often "memorize" past surges—like NVIDIA’s 2023 rally—and recite them as "predictions," creating a profit mirage. Fix: Use Point-in-Time (PiT) AI models trained strictly on data available before a specific cutoff date to ensure no future knowledge leaks into the signal.
Real quantitative work is about ruthlessly killing bad ideas early. The real question is: "What had to be true for this backtest to look good?". That is where genuine edges live.
hashtag#QuantitativeFinance hashtag#TradingStrategies hashtag#Backtesting hashtag#DataScience hashtag#FinTech hashtag#AlgorithmicTrading
