Hi guys, hi from Andrea Unger! Today, I’d like to talk about the robustness of trading systems.
The robustness of trading systems is a recurring topic in the trading industry. However, it’s quite in line with those myths and stereotypes that should be busted, as sometimes people portray it as an excessively algorithmic process.
If, on the one hand, there’s a part of truth in this, on the other it sometimes happens that some things are not properly explained, so let’s try to clarify the concept.
As a first thing, there are many tools out there that allow you to evaluate the robustness and stability of a system’s performances. These tools take into account the choices we make when we select the parameters of a system. They focus on the inputs we choose throughout the optimization process, exploring their impact on performances.
These tools allow you to move through different input levels and see how performances vary as they change.
They are certainly useful but, if you think about that, you simply need to avoid cherry-picking the best values.
In general, there are two categories of developers. The first gathers “tricky” developers, i.e. those who intentionally build systems that look very nice but actually are absolutely unstable. By choosing certain parameters and inputs, they build systems whose performances look very nice, even though, in reality, they know that they wouldn’t endure real-time challenges.
The second category gathers what we may call “dumb” developers (I mean no offense), i.e. those who cheat themselves.
When I develop a system and I see that it performs wonderfully only with a specific input value, producing a disaster with all the other ones, and I choose that value, I’m clearly cheating myself, because I’m trying to use something I know is going to fail in the next five minutes.
These tools can be useful to evaluate the robustness of trading systems in such cases. However, if you choose parameters properly, you’ll be able to get an overall scenario of the stability of your system. This is something that is useful of course, if by doing so you understand that there is some stability.
However, there’s also another important point. What happens if the edge your system is based on disappears?
If you build a system based on a specific feature or edge of the market and this thing disappears, what happens to your system?
Suppose you need to be at work by 9 a.m. and always catch the bus that stops in front of your house at 8:20. If, for some reason, the bus starts leaving at 8:35 instead of 8:20, you’ll be late. So, a change in the environment causes a change in your system, because you’re no longer able to arrive at work on time.
In this case, you simply need to keep up with the timetable updates your bus company provides. By doing this, you become able to find a solution in advance.
Unfortunately, the markets don’t tell us when they change their schedules, and this is a great issue. So, we need to analyse our systems and try to fully understand what we put in them.
What we have to do is to identify the motor of the system, the very thing it revolves around. Why did we put those rules together? Because we are looking for a specific edge. So, if we identify the edge we are working on, we might be able to find out in advance or, at least, in time, when that edge stops being there.
Normally, when you work on a specific edge, then you add more filters and combine more patterns. This leads to a system that may survive to a period when the edge disappears. This happens thanks to the further conditions you add.
By the way, the higher the number of conditions, the higher the chances of an unstable system. In fact, the higher the degree of freedom of your system, the better.
When you add many conditions, you force your system into something that might be very unstable. Like a beautiful house of cards, it might collapse as soon as your cat passes by.
So, actually, you have to be very careful. Let’s imagine that these additional conditions help your system survive the disappearance of the edge. If the motor of your system was based on that edge only, and the edge disappeared, you could have seen it in the results, if there were no additional conditions.
Seeing this is very difficult and takes experience. So, if you aren’t able to do it, the only solution is to create some performance indicators that will tell you when to stop a system. This will open up to a completely new world.
However, the bad news is that there is no specific performance indicator. You have to choose the ones you prefer, so you should build something that makes sense.
You might be wrong – we can’t be right all the time – but, if you have something that tells you when to switch a system off, this will obviously be ahead. So, this is the only way to avoid losing a lot of money with “ill” systems.
This might not appear very scientific, because instead of evaluating the robustness of trading systems, you are simply evaluating their performances. Unfortunately, when you have to fight against a disappearing or disappeared edge, I think this is the only way to go.
I hope this was useful! If you have any questions or observations, let me know in the comments and I’ll try to answer as usual.
Ciao from Andrea Unger!