How Much Can We Trust Backtesting?

I believe a lot of readers would be wondering how much can we trust the result of statistical backtesting based on historical data.   I guess a lot of you have definitely came across articles or gurus that equate doing live trading based on backtested result as "driving with rear view mirror" or something similar.

I would like to point out that the analogy is not quite correct.  In live trading, if we have sufficient capital to withstand a few drawdowns, we will survive losing trades.  In fact. there is no way to avoid a few losing trades once in a while.  However, when we drive, we can't afford to make a wrong turn or go straight when the road in front is winding, in fact, we can't really make a single mistake, as that would likely be disastrous!

Having said that, we must also realise the limitations of statistical backtesting as historical data is the only sample that we have and to make an educated guess on the likelihood of repeating patterns of the mass human psychology (market) and we can't be 100% sure that the same will happen again in the future, although a variation of the sample is likely to happen again as long as human has flaws!  I know this is confusing, as we have no choice, but to make a deterministic (same input, same output) conclusion (trading system) by just studying a sample (historical data) from hundreds of thousands of possible variations of the future!

To make the matter worst, the process of doing backtesting is susceptible to a lot of human biases, assumptions and errors such as worng historical record, selecting a bull or bear only cycle (period too short or sample size too small), curve-fitting the strategy to match data, wrong programmming or calculations etc.  One of the most dangerous example is hindsight bias (giving advice such as buy high sell low, or quickly take profit when market is ranging) and seeing one occurance and start assuming that the secret has been found! 

Remember, we can't prove that a hypothesis is correct, all we can do is repeating the process of trying to prove it wrong so that after thousands or millions of tests, we can have a degree of confidence in our model's successfulness in describing the pattern that it tries to capture.  The hypothesis just remains as the best solution so far.  We only need one observation that our model does not correctly describing the pattern to fix it or throw it away (such as failing an out-of-sample test)!  Just think of this as innocent until proven guilty.  If no one can prove you guilty, you remain innocent so far, it doesn't mean that you are really innocent!

On top of that, we need to do a stress test of the trading system using out-sample data and simulations on the variations of the result.  At the moment, one way of testing the trading system's result is to generate random sample data of the future via Monte Carlo Simulation which by itself is not perfect as certain assumptions about the pattern of distribution of the future data (possible data points of the results) has to be made, for example, a normal distribution, which may or may not be representing the real future data as the future is always unknown.  Besides, how many retail speculators or even the so call gurus know how to design and build their own Monte Carlo Simulator or even do and interpret the result from a commercial progam not to mention it usually is quite expensive.  Furthermore, can you really trust the simulation results?  Is there any correlation between the random inputs generated by the random number generator?

If I have scared you already, please don't be.  There are at least hundreds of ways we could have been knocked down by a car when we cross the roads but we still do it, why?  Simple, once we really understand what we are doing and are in control, we would have the courage to take the risk already, in fact, it has become second nature for us to take calculated risks in our everyday life without we second guessing them! 

Just always respect the market and have faith that humans will always have flaws!  I will try to explain how to design and evaluate robust trading systems for normal people like you and me in my future postings.  For now, good luck and happy trading!