Your backtest looks amazing. Sharpe ratio of 2.5, maximum drawdown of 8%, consistent returns across market regimes. You deploy to paper trading, then live. Six months later, the strategy is flat—or worse.
This story is so common it's almost a rite of passage in quantitative trading. We've lived it ourselves, and we've seen it happen to smart, experienced researchers. The culprit is usually the same: backtesting errors that silently inflate expected performance.
At Referential Labs, we've reviewed hundreds of backtests. The same mistakes appear repeatedly—and they're not always obvious. We've seen lookahead bias hiding in a feature that looked perfectly innocent. We've seen survivorship bias enter through a vendor's "convenience" function. This guide covers the most common pitfalls and how to detect and prevent them.
Why Backtests Fail
A backtest is a simulation. It makes assumptions about data availability, execution, and market behavior. When those assumptions don't match reality, backtested returns evaporate.
The tricky part: these errors don't throw exceptions. Your code runs fine. The numbers look good. The problem only becomes apparent when real money is on the line.
The Top 10 Pitfalls
1. Lookahead Bias
What it is: Using information that wouldn't have been available at the time of the trade decision.
Examples:
- Using today's close price to make a decision before the close
- Using a feature calculated from future data (e.g., monthly statistics computed from the full month when you're only partway through)
- Using adjusted prices without respecting the adjustment date
- Using fundamental data before its actual release date
The insidious forms: The obvious cases are easy to catch. The subtle ones are dangerous:
- Corporate action adjustments: Split-adjusted prices use adjustment factors that weren't known at the time. TSLA's August 2020 split shows retroactively in adjusted data—but you didn't know that adjustment would happen in June 2020 when you "traded."
- Restated financials: Earnings numbers get revised. Enron's "historical" earnings looked great in databases until the restatements.
- Index membership: If you're trading S&P 500 stocks, the current membership list includes survivors and excludes failures. The September 2008 list included Lehman Brothers (removed September 15).
How to detect:
- Trace data flow through your pipeline. For each feature, verify the timestamp of every input—not just the data timestamp, but when the data became available.
- Run your backtest with data cutoffs at different points. If truncating history changes historical signals, you have lookahead.
- Check if performance is suspiciously good around data release times (earnings, economic releases). This often indicates using the release value before it was released.
- Compare your backtest to paper trading on the same period. Signal differences often indicate lookahead in the backtest.
How to prevent: Build your data infrastructure with point-in-time semantics from the start. Every query should specify "as of when"—and the system should return only data that was available at that time.
2. Survivorship Bias
What it is: Only testing on assets that still exist today, missing delisted stocks, failed funds, or bankrupt companies.
Examples:
- Backtesting a stock strategy using the current S&P 500 constituents
- Testing a crypto strategy only on coins that still trade
- Evaluating a momentum strategy without including the losers that got delisted
How to detect:
- Check your universe construction. Does it include historical constituents?
- Look for unusually high returns. Survivorship bias typically adds 1-3% annually.
- Verify your data source explicitly includes delisted securities.
How to prevent:
- Use point-in-time universe data that reflects what was actually available
- Include delisted securities with their final prices
- Track the reason for delisting (merger, bankruptcy, etc.)
3. Look-Ahead Bias in Data Preprocessing
What it is: Using statistics from the entire dataset (including future data) to preprocess training data.
Examples:
- Normalizing features using the full-sample mean and standard deviation
- Filling missing values with forward-looking interpolation
- Winsorizing outliers based on full-sample percentiles
- PCA or factor analysis using the full dataset
Why it matters: This form of lookahead is particularly insidious because it happens before model training. You might have perfect train/test splits for your model, but if the preprocessing used future information, you've already leaked.
The magnitude: Preprocessing lookahead can inflate Sharpe ratios by 0.3-0.5 or more. We've seen strategies go from Sharpe 1.8 to Sharpe 1.2 after fixing preprocessing lookahead—the difference between "exciting" and "marginal."
How to detect:
- Review all preprocessing steps. Does any step use aggregate statistics? If so, what data are those statistics computed from?
- Truncate your dataset to an earlier end date and rerun preprocessing. If the preprocessed values for historical data change, you have lookahead.
- Check your feature pipeline's dependencies. If feature X depends on feature Y, and Y uses full-sample statistics, X inherits the lookahead.
How to prevent: Every preprocessing step must use only data available at that point in time. Normalization should use expanding or rolling windows with a lag. Missing value imputation should use only past values. Outlier detection should use historical percentiles, not full-sample percentiles.
4. Overfitting
What it is: Tuning parameters until they fit historical noise rather than signal.
Signs of overfitting:
- Too many parameters relative to data points
- Performance degrades significantly out-of-sample
- Strategy only works on specific date ranges
- Results are extremely sensitive to small parameter changes
How to detect:
- Calculate degrees of freedom: parameters / independent data points
- Compare in-sample vs out-of-sample performance
- Test parameter stability: do optimal parameters change with more data?
How to prevent:
- Use walk-forward optimization instead of single-period optimization
- Apply regularization or parameter penalties
- Prefer simpler models with fewer parameters
- Reserve a true holdout set you never touch during development
5. Ignoring Transaction Costs
What it is: Backtesting without accounting for commissions, slippage, and market impact.
The math: A high-turnover strategy might turn over its portfolio 10-20x per year. At 10 bps round-trip cost per trade, that's 1-2% annual drag just from transaction costs.
Why high-Sharpe strategies are most vulnerable: The strategies with the highest gross Sharpe ratios are often the highest-turnover strategies. A stat-arb strategy might have gross Sharpe of 3.0 but net Sharpe of 1.0 after transaction costs. The gap between gross and net is where many strategies die.
What to include:
- Commissions: Explicit fees paid to brokers.
- Spread cost: You buy at the ask and sell at the bid.
- Market impact: Your order moves the price against you.
- Borrowing costs: For short positions, you pay to borrow shares.
- Financing costs: If you're leveraged, you pay interest on borrowed capital.
How to detect: Calculate your strategy's turnover. Multiply by realistic transaction cost estimates. Compare gross returns to net returns. If transaction costs consume more than 20-30% of gross returns, your strategy may not survive contact with reality.
6. Insufficient Sample Size
What it is: Drawing conclusions from too few trades or too short a time period.
How to detect:
- Count independent observations.
- Calculate confidence intervals on performance metrics.
- Check if results are consistent across subperiods.
How to prevent:
- Require minimum sample sizes before trusting results.
- Report confidence intervals, not point estimates.
- Test on multiple independent periods.
7. Not Accounting for Market Regimes
What it is: Testing on a single market regime and assuming the strategy will work in all conditions.
Examples:
- Backtesting 2010-2020 (mostly bull market) and expecting similar results in a bear market.
- Testing a volatility strategy only during high-vol periods.
How to detect:
- Segment your backtest by market regime.
- Check if performance is concentrated in specific periods.
How to prevent:
- Explicitly label regimes and report performance separately.
- Include regime detection in your strategy.
8. Survivorship Bias in Features
What it is: Using features that wouldn't exist in production.
Examples:
- A feature that requires data from a vendor you've since dropped.
- A feature calculated from a field the exchange stopped providing.
How to detect:
- Document the data source and availability for every feature.
- Track when each data source was available historically.
How to prevent:
- Maintain a feature registry with metadata on data sources.
- Version your feature pipeline.
9. Data Quality Issues
What it is: Backtesting on corrupted, missing, or erroneous data.
Examples:
- Stock splits not properly adjusted.
- Missing data filled with incorrect values.
How to detect:
- Validate data before backtesting.
- Look for anomalous returns around specific dates.
How to prevent:
- Implement data validation in your pipeline.
- Maintain data quality metrics.
10. Unrealistic Execution Models
What it is: Assuming you can execute at prices that wouldn't be achievable in practice.
Common unrealistic assumptions:
- "Execute at close" when you signal before close.
- "Execute at the mid-price".
How to detect:
- Compare backtest execution timestamps to signal generation timestamps.
- Check if your backtest execution prices match actual traded prices in the historical data.
How to prevent: Be conservative. Assume you execute at the worse side of the spread, with some market impact, after some delay.
A Practical Validation Checklist
Before trusting a backtest, verify:
Data:
- Data source is documented and validated
- Point-in-time accuracy verified
- Corporate actions properly handled
- No gaps or quality issues in test period
Methodology:
- No lookahead bias in features or signals
- Survivorship-free universe
- Expanding window for any preprocessing
- Degrees of freedom reasonable
Execution:
- Transaction costs modeled realistically
- Slippage and market impact included
- Execution timing matches reality
Validation:
- Out-of-sample testing performed
- Multiple market regimes tested
- Confidence intervals calculated
- Parameter sensitivity analyzed
Conclusion
Backtesting errors are subtle and systematic. They don't announce themselves—they just quietly inflate your expected returns until live trading reveals the truth.
The best defense is systematic validation. Check every assumption. Trace every data flow. Question every result that looks too good. And critically, compare backtest performance to paper trading and live performance. Divergence is diagnostic—it tells you something is wrong, even if it doesn't tell you what.
If you need help validating your backtesting methodology or building more robust validation infrastructure, contact us. We've seen these pitfalls hundreds of times and can help you avoid them.
Frequently Asked Questions
What is lookahead bias in backtesting?
Lookahead bias occurs when your backtest uses information that wouldn't have been available at the time of the trade decision.
How much does survivorship bias affect backtests?
Survivorship bias typically inflates annual returns by 1-3% for broad indices.
How do I know if my backtest is overfitted?
Signs of overfitting include too many parameters relative to data points.
What transaction costs should I include in backtests?
Include commissions, spread costs, market impact, borrowing costs for shorts, and financing costs for leverage.