Hi guys, hi from Andrea Unger! Today, I’d like to discuss a bit about historical data and the choice of the right timeframe.
People often ask me questions about historical data. Most of them are about how far back in history the historical data we use to test our strategies should go, in order for the backtests to be effective, and what timeframe we should use to get a clear picture of the market we are testing.
The answers to these questions depend on the instruments you intend to trade.
Index Futures and Historical Data
Let’s start with index futures, such as the DAX or the e-Mini S&P 500. The point with these markets is that you need go back enough to be able to cover different market scenarios.
If your data start from, say, 2012, you won’t get a clear picture. Indeed, since that date, these markets have been almost always characterized by the same uptrend behaviour.
This means you would develop your strategies resting upon data that show a strong directional bias, so those strategies and systems might not be effective under different market conditions. We all know what happened to index futures in 2008.
So, I recommend that you use data that include at least that strong 2008 downtrend, so that you can get a clearer picture of trends and volatility movements in these markets.
These markets experienced strong downtrends in 2001 and 2002, too. However, even though it could be a good idea to take a look at what happened during that period, you should keep in mind that the kind of downtrend that occurred in those years had a smaller algorithmic presence of the market.
The dynamics of the moves were quite different compared to the following years, so what works on those data may not work today. Therefore, those data are not as useful as those from the 2008 on, when it comes to get an idea of what happens when violent and sudden falls occur in these markets.
So, 2006 could be a good compromise on index futures.
As for the timeframe, five minutes work very well. I don’t think that smaller timeframes are necessary. Not only would one-minute or tick data be very expensive, but they are also not necessary for normal retail traders as we are. In fact, developing resting upon tick data requires a highly technical infrastructure in terms of hardware, software, connectivity, etc.
So, actually, a 5-minute timeframe is enough to clearly define how the market moves and what kind of actions to take.
When it comes to the historical data of US commodities (Crude Oil, Gold, Soybeans or whatever) starting from 2010 is certainly enough. Data from 2008 and 2009 are available, but since the electronic markets of these futures were then newly born, their initial moves are not clear enough to develop strategies that would effectively work today.
When you backtest on the Grains, always keep in mind that session times have changed over the years.
As for currency futures and Forex, you can go back to 2003. Starting from 2008 is good, but taking data that start from 2003 into account would be better. In fact, I noticed that, from time to time, certain dynamics that occurred in some past periods tend to happen again on these markets.
If you refer to Forex instead of currency futures, be careful about the source of the data you use. As you know, Forex is an OTC market, so everybody has its own data, let’s say. This means that it is very important that you develop on data that are most probably the ones you use when you go live.
I don’t think there are many people who are interested in Bonds, however if you trade and develop on Bonds, you can apply what we saw about index futures.
Historical Data and Software
Of course, you should always bear in mind that the higher the quality of the data you gather, the higher the quality of the software you use to backtest the data.
You can’t get tick data for thousands of dollars and then use a small hand calculator to develop! So, the higher the quality of the data you focus on, the higher the quality of the backtesting environment you use to devise your strategy.
That’s it, I hope this was useful.
If you want, ask questions in the comments and I will be glad to answer.
See you next time!
Ciao from Andrea Unger!