Overfitting and how we test for it

Overfitting is when a strategy has been tuned so closely to past prices that it learned their noise instead of a real pattern. It's the main danger in any backtest. This page explains why, what we do to catch it, and what you can check yourself.

Every strategy has a few settings: how many bars to average, how many votes must agree before buying, how far the price must move. Try enough combinations on the same price history and one of them will always look spectacular, purely by chance. It has fitted itself to the particular ups and downs of that history. Shown new prices, it falls apart.

That's overfitting, and it's the main reason backtests disappoint. The awkward part is that an overfitted strategy looks exactly as good on a chart as an honest one. Same smooth account line, same impressive CAGR, nothing on the card to warn you. The difference only shows once it meets prices it was never tuned on.

A made-up example

"Buy BTC when it has fallen exactly 4.2% over five days, but only if the bar before was red." Test 4.1%, 4.2%, 4.3% and every other threshold on 2015 to 2025, and one of them will shine. Not because 4.2% means anything, but because with enough tries something always fits. If 4.1% and 4.3% do badly while 4.2% does brilliantly, you've found luck, not a pattern.

What we check before publishing

During research, a strategy's settings are searched systematically, and the whole landscape of results is examined, not just the best point.

  1. Plateau or spike? If a setting of 40 works well, do 38 and 42 work too? A broad plateau of good neighbours suggests the strategy is picking up something real. A lone spike that collapses one step to either side suggests luck, and isn't published.
  2. Does it hold up in different periods? The history is cut into consecutive stretches and each is checked on its own. A setting that does reasonably in every stretch is preferred over one that's brilliant in one year and poor in the rest.
  3. Could the search itself explain it? The more combinations are tried, the more likely the best one is just the luckiest. Statistical tests that account for how many combinations were tried (such as the probability of backtest overfitting and a deflated Sharpe ratio) estimate how much of a result could come from searching alone.
  4. Does it survive data it wasn't tuned on? In newer research rounds, the history is split: settings are chosen on the earlier part only, then judged on a held-out later stretch.

When a setting is chosen, it's taken from the middle of a stable area, not from the single highest peak. The peak is usually the luckiest point; the middle is a little worse on paper but far more likely to behave the same way next year.

Other markets, other timeframes

A strategy is tuned on one market and one bar length. Taking it somewhere else, without touching a single setting, is one of the most telling tests there is, because the noise it might have memorised doesn't come along.

Does it work on other tickers?

A strategy fitted on BTC that also does well on ETH, SOL and other coins is more likely to have found something real than one that only shines where it was tuned. The same goes the other way round: rules built on ETH that hold up on BTC earn extra trust. It counts for even more when they carry over to a different sector altogether, such as stocks or commodities like gold, which move for very different reasons than crypto does.

Can BTC's chart steer other markets?

The (BTC) strategies take their signals from BTC's chart (BTCUSD) and use them to trade something else: ETH, SOL, or whole-market indexes like TOTAL and TOTAL2. That makes them a test in their own right. If BTC's signal works as a proxy for those markets, the rules are picking up the broad crypto cycle rather than the quirks of one price history. In practice it works quite well, which is one more sign that the signals hold across assets. Markets and asset classes explains how these strategies are named.

Does it survive a different bar length?

Most strategies here are tuned on 5-day bars. Run the same rules with the same settings on weekly, 3-day or daily bars and most of them do worse, which is only to be expected when the settings were picked for one rhythm. What matters is how much worse. A strategy that still does fairly well on 1W, 3D and 1D bars is probably riding a real trend; one that falls apart the moment the bar length changes was most likely fitted to the exact timing of its own history.

A human look at every backtest

Statistics only go so far. A backtest can look great on paper and still show its problems once you go through it trade by trade on the chart. The things we look for by eye include:

  • Drawdowns that turn out deeper or longer than the headline figures suggest.
  • Signals at odd moments, such as buying into the top of a spike or selling right after a bottom, in a way that doesn't fit the idea behind the rules.
  • Too many or too few signals. Lots of trades usually means the rules react to noise; very few leaves the whole result resting on a handful of lucky calls.
  • A result carried by the early years. Some strategies made most of their money in BTC's wild early days and have done much less since. The full-history numbers hide that; the chart doesn't.

That kind of review rests on years of experience with these markets and this kind of system. It's partly a judgement call and can't be written down as a formula, but it still catches things the automated tests let through.

How the design itself resists it

  • Few, simple rules. Fewer settings leave fewer ways to fit noise.
  • 5-day bars. Strategies trade a few times a year, so being a day early or late hardly matters and a result doesn't hinge on one lucky timing.
  • Voting. Index strategies (the names with C or T and a number, like God Emperor T9) combine several single strategies that look at different things, such as trend, momentum and volume. One indicator fitting the noise is outvoted by the others.
  • The same rules on several markets. We avoid hard-coded levels that worked in one period, and prefer rules that do reasonably on many assets over ones that look perfect on one. For altcoins, the signal often comes from BTC's chart rather than being fitted to each coin's short, erratic history.

Why we stay on long timeframes

This is the core idea behind the whole approach, so it's worth spelling out, together with the honest caveat that it's a thesis and not a promise.

The first reason is the one this page is about. Slow bars and few trades act as a filter against overfitting: there's less noise to latch onto, and a result doesn't depend on catching the exact day of a turn.

The second reason looks ahead. Short-term trading is turning into a race of speed and computing power, and high-frequency, AI-driven bots will most likely dominate it, reacting faster and reading more data than any person or slower system can. We'd rather not compete there. What no algorithm can do is push a market worth many billions around for months or years at a time. The short-term noise may well get louder, but the longer trends should keep trending, because they come from large amounts of money moving in and out of an asset over long periods.

The strategies here aim to catch a sizeable part of those long moves. How much varies over time: in one cycle a strategy might be in for 90% of a rise, in the next for only 40%, depending on how the trend starts and ends. Catching a good share of the big moves while sitting out much of the big falls is, in theory, what adds up to a profit over a long horizon. That's what we believe and what the backtests support so far. It isn't proof. Markets can change in ways that break the idea, so treat it as our reasoning, not as a guarantee.

Where it can still go wrong

These checks lower the risk. They don't remove it.

  • Few trades. A strategy with 10 trades since 2015 is a small sample, however carefully it was built. See Trade statistics.
  • One kind of market. Much of the history here is one long crypto bull market. A strategy can be honest and still only have been tested in mostly good times.
  • Short histories. HYPE and VVV have about two years of prices. Nothing about their strategies has been tested through a full cycle.
  • Known traps only. Passing every check means none of the usual warning signs showed up, not that a strategy is sound.

No test can prove a strategy will keep working.

What you can check yourself

You don't need the research tools to spot warning signs:

  1. Try several windows. Click 3Y, 5Y and All on a strategy page. A robust strategy usually looks reasonable in every window, not great in one and poor in the others.
  2. Look at the trade count. Under about 10, be careful with every number.
  3. Compare with Buy & Hold. If a strategy barely beats simply holding the market, its extra complexity may not be worth it.
  4. Look at the same rules on other markets. Search a strategy's name in the All folder. EEMA Bands v3, for instance, runs on BTC, ETH, SOL, HYPE and VVV; all five show a CAGR between about 77% and 138% and a Sharpe between 1.1 and 1.4. Consistency like that is a good sign. (BTC) God Emperor T9, which trades altcoins on BTC's signal, did very well on ETH and SOL but lost money on VVV, a reminder that a rule working on one market says little about the next.
  5. Watch it after publication. Every night adds a bar the rules have never seen. A strategy that keeps behaving like its backtest over the coming months is more trustworthy than it was the day it was published.
  6. Spread your bets. A portfolio of several different strategies depends less on any single one being right.

In Strategies, open All Strategies, type EEMA Bands v3 in the search box and compare up to four of the versions side by side with the scales icon scale icon.

Related