“How do I know if I have over-optimized my trading system?”
Despite the fact that traders have struggled to answer this question since the dawn of algo trading, over-optimization is probably the number one reason why many traders fail to ever produce a profitable trading system.
The question isn't easy to answer, that's for sure. But we are convinced it all comes down to statistical significance. If you don't have the required level of statistical significance in your back test results, then your results are meaningless and you will be trading a system that will not replicate your back test results in a live account - fact.
So that's the answer right? Well yes... but how do you know if results are statistically significant I hear you ask? Well, read on...
So what does this best practice entail? First and foremost is the concept of statistical significance, and it is this that we will focus on. The need for any type of back testing to have statistical significance is absolutely imperative.
Without it, test results are completely meaningless, and you will have wasted several hours of your life undertaking them. In fact, a lack of statistical significance is the number one reason (by far) why traders fail to optimize and back test effectively.
For WFA in particular, understanding how statistical significance impacts the methodology, is a slightly more complex concept, but in this article (and the next in the series) I aim to fully cover the subject so that you can incorporate the techniques into your own testing.
A quick recap of Walk Forward Analysis before we start
The out-of-sample back test phases are commonly termed ‘walk forward’ phases (since they were not used in the optimization itself) and they are used to validate the optimal parameters that were identified in the previous optimization phase.
The fact that the entire WFA uses multiple stages, covering different time epochs (and therefore different market conditions / price action personalities) is what makes walk forward analysis so effective. It shows the ability of your system to adapt and perform well across many changing market conditions.
Firstly, the way we need to look at the statistical significance of the in-sample optimization phases is fundamentally different to how we look at the statistical significance of the out-of-sample back tests. Also the consequences of a lack of statistical significance, results in very different issues depending which we are considering.
Statistical Significance of the in-sample optimizations
When we undertake optimizations without sufficient statistical significance, this has the effect of reducing (or in extreme cases, completely eliminating) the predictive power of selecting the best parameter values from the optimization. When this is the case, the selection of parameters tends to be based more on randomness and chance, than by the effectiveness of the parameters of your trading system. This then leads to poor performing out-of-sample back tests, and poor performance if traded in a real money account. This is what the industry often terms ‘over-optimization’ or ‘over-fitting’.
I’ll let you into a secret. It is much easier to over-optimize than you would ever imagine. If I had to guess, I would estimate that 90% - 95% of algorithmic traders are over-optimizing. This is a real shame because it means that trading systems that could work well are probably thrown away by the trader. They don’t have a chance to work, because they are based predominantly on ‘random’ parameter values, that performed well by chance in the optimization, but will, in all likelihood, never perform well again in the future.
Statistical Significance of the out-of-sample back tests
There are two extreme scenarios here, but most people only consider the former. That is that the results could be inflated or over-optimistic, and in fact the system would never achieve results to a similar level in a live account.
This is of course true but also, if the statistical significance is really bad, this can also result in poor back-test results, that if given a chance to perform longer term in a live trading account would actually achieve great results. However, in the meantime the trader has probably thrown this (perfectly good) system away.
So to summarise
In the next articles...
Then in part 3, we will consider how to architect your walk forward optimization settings to achieve the optimal balance between in-sample optimization significance and out-of-sample back-test significance. This will allow you to maximise the effectiveness of your walk forward analysis process and produce more robust systems – another important element of best practice. Coming soon.
If you use MetaTrader for your back testing...
Find out exactly how your out-of-sample and in-sample statistical significance holds up. Are you currently trading a system that is based more on random and insignificant parameter values, than on the ones that will achieve the best results in your live account? Interested? Find out more here