Hi guys, how long should the history for backtesting be?
A question I received from a student and the answer is: it depends!
My wife gets mad at me when I say “it depends”, but it depends.
It depends on the market, it depends on the conditions.
How much data for backtesting on stocks and indexes?
Generally speaking, it is good to have as much as possible to cover different market conditions as a bull market, bear market, choppy market, wild crashes.
All these things, the more we have, the brighter we have the idea about the reactions to our models.
The problem is that not always the market phases show the same reaction in our approaches, so you might argue that it’s the approach being bad or wrong or whatever, but sometimes, really, the market moves reflect a different way to move and therefore different reaction to our strategies.
Considering, for example, the indexes as miniS&P500.
I say: “You have to have a bull market in your data, and we have a lot of bull market but also bad crashes or bear market”.
It’s true, but I can tell you that, in many cases, the results of some approaches on the markets, are different in the bear market of the 2001/2002 compared to the bear market of 2008 and again on the crashes we saw for example in August 2015, February 2018 or at the end of 2018 again.
So it’s difficult to find a common way of behaving for that market in those different conditions.
Now the more we have the better because we see how stable the approach is, but again it’s difficult to understand why.
Backtest on market crashes
For sure a violent crash is different from a slow down phase or prolonged down phase better said, because down phases are never too slow, they are always a bit active, you know, but sitting at the monitor I saw why, for example, after a market crash, for a while the breakout strategies on miniS&P look like being more efficient, more satisfactory in terms of results, compared to the classical counter-trend approach.
The point is that normally when you look at the order book on miniS&P you see that there are, depending on the time of the day, 300, 400, 900 contracts at each level of the order book, each level of price bid/ask, while after those markets crash as in February 2018 for example, you could see there were 30, 40 sometimes.
It’s clear that sending a big order, a large order, if I said 500 contracts and I find 900, I don’t move that much, but if I find only 30 I’m eating out different levels of bid or ask and I’m moving the prices, so I’m working in favor of the breakout.
So, actually, once the big players go out of the market waiting for a situation to calm down, the reaction is completely different.
We have to be aware of this, we have to understand that this might happen.
Eventually, we can choose not to use any strategy or to choose some strategies specific for those periods but probably is that we don’t know how long they last.
One week, one month, six months?
We don’t know!
So actually we have to be very careful about this, but again we need enough data to understand this.
So on the indexes, it’s better to have a large variety, on other markets we have sometimes to understand that there are conditions which will never come back.
Backtest on commodities
On commodities, for example, crude oil, gold, grains, we have intraday data starting in 2008 more or less.
Normally I develop starting from 1st of January 2010, so why am I throwing away 2 years of data I paid for?
I’m throwing this date away because I think that the first 2 years were absolutely not significant for anything on these markets.
The truth is that these markets became electronic in 2007/2008, during that period and in those first years they were getting accustomed to the new situation, they were learning how to trade electronically together with the pit and so on.
This led to performances which were not real, we have all these systems showing wonderful performances in the first years as 2008/2009 and then slowly going down.
So that kind of reaction present, if the data are correct, if they are not again we don’t need them, but if they are correct in the first years 2008/2009, first years in terms of electronic market, they are not significant and those conditions will never come back again, because those conditions were linked to a specific condition present, new stuff, which is no longer, it and will never be new again in that way.
So those results are useless for me to understand the behavior of a market, to understand what they could do.
On the other hand, the result after those days show also a different reaction to trend-following, to breakout on many markets, showing how there is a tendency to lose efficiency in breakout strategies because the markets gained efficiency, that means that they show a stronger opposition to breakout, a strong resistance to breakout.
Soybeans, for example, showed a very strong change in its behavior in 2011 or 2012 I think, also gold future in 2018 showed that change.
Sometimes they go back to the past behavior because they get again some trend-following periods and so on, but the situation is going in a direction where the trend-following is getting weaker and weaker and we have to be aware.
By reading the backtest on a wider range of data can help us in understanding what’s going on, so we need that.
As said with commodities I would say 2010 not anything before, on indexes futures is better to go back as far as you can trying to understand as much as possible on every single specific phenomenon we see, and in case ask if you were not sitting there, if you don’t understand why or if you wonder why in 2008 your strategy was doing that, why in 2001 it was doing that other thing like that.
Backtest on ForEX
The currencies or forex had more or less the same behavior throughout the years, but it’s also true that what changes the bid/ask is the level of volatility.
So the volatility which was present in the older years is now much smaller and many approaches developed based on that kind of volatility and today they find harder times to produce any profit.
So again it’s better to look back, but to understand that overall conditions change and to read carefully the reports try to understand what works today, still looking back if it was working in the past.
Last 12 months backtest, is it helpful?
Another thing is, if I had a strategy that lost money for years and in the last 12 months it makes a lot of money, I would not feel confident in putting that strategy live, because 12 months is, in my opinion, a too short period to make any decision.
If this period strongly differs from what was shown before.
If it’s just an improvement it’s okay, but if it’s a total difference to what happened before, unless I have a clear reason why this happens, I would wait for a longer time to decide to put this live.
Maybe these 12 months are the great light to become the new behavior for the next 10 years, in that case, I will lose some time waiting for further confirmation, but 12 months, in my opinion, are not enough.
So if you develop in 12 months, if you develop on 12 months only, I would not be happy to use those strategies if I did so, because I don’t consider 12 months of history enough, unless you trade high frequency on tick data or so, but it’s not our case, it’s not my case, I would not use that.
So again, enough history 2010 on commodities, currency go back but be careful about the changing volatility, indexes go back as much as possible, try to find a proper interpretation of all results, looking throughout a bearish or bullish period, some violent crashes, choppy market, everything you find.
I hope it helped, write your comments on what you prefer and how long you look back when you develop your strategies.
Ciao from Andrea Unger