Historical Data

                     
The 1st step before you start designing your simple trading system is to collect the historical prices of the products that you want to test your strategies on.  Many in statistics would refer this as sample collection process.

In my opinion, it is one of the most important processes that will determine the chances of success or failure of your trading systems.  Why?  Imagine if you have a strategy that is trend following and you have collected and test your strategy on crude oil futures data from Aug 2007 - Mar 2009!  Your strategy will be able to ride the trend on the way up as well as on the way down giving you a rate of return that would be very high.  Even though you may have done every calculation correctly but the robustness (the ability to withstand different cycles) of the system was jeopardize because your sample size is not reflecting the actual population's profile.  If you are a computer programmer, that’s what you call garbage in garbage out.  The analogy would be to choose only 10 males to do your survey before designing a car or mobile phone.

This is also how the unscrupulous systems sellers or trading coachers or some “professionals” fool most beginners.  They would usually choose a strategy that will match perfectly with a certain period (usually very small) of the historical data and show you the performance summary that is too good to be true.  But if you were to verify the strategy on your own with longer term data and more trades, you’ll see that it is not as rosy as it seems.  If it is really so good they would have sold everything they have and trade on their own already no matter what they choose to say to the contrary such as “my interest is coaching and help people”, “I’m giving back to the society” nonsense!  But if you are ok accepting advice from someone who can't do what they teach, then by all means, don't let me stop you.

Therefore a few general guidelines are very important when you choose your period of historical data.  First, we must at least make sure that the period is long enough (generally more than 5 years and would generate at least more than 30 trades) and the data has all the different cycles of the market such as bull, bear, range (bigger cycles of ups and downs) and congestion (tight ups and downs).  The next thing is to go through all the data and look for possible errors such as prices that deviates too far away and cross check with other sources if possible.

As futures contracts have expiration, we’ll need a way to make it into a continuous data.  Sometimes you’ll have huge gaps between the most active month (most trading volume and highest open interest) and the month after it and it’ll affect your entry, stop or exit signal unnecessarily and affect your back-test result when it rolls over.  Either you do some adjustment when the expiration nears or no adjustment at all or you just continue with new data when the old one expires (roll on expiration).  There is always a trade-off in choosing different ways of adjusting the data.  The professionals would mostly use back-adjusted data (google yourself) but unless you can download free or at a reasonable fee from the internet, it will be too tedious for us mere mortals to do the adjustments ourselves.

The best is to use back-adjusted continuous data, but if you can’t get the data, then you’ll just have to use the roll on expiration method but you’ll have to check all the trades and make sure that they are not affected by a huge gap when rollover.  In addition, you’ll have to factor in the rolls into your performance summary as sometimes they are not favorable to your positions when you roll them.

You can start by downloading the FKLI and FCPO historical data which is available free on Bursa Malaysia’s website http://www.bursamalaysia.com/website/bm/market_information/market_statistics/derivatives/downloading.html

Usually you’ll get 3 complete years of daily data before the current year in Excel format.  Just download the FKLI and FCPO from 2006 – 2009.  For FKLI, after downloading, just construct a continuous data from the spot month contract section (date, open high, low, close and volume).  As for FCPO, just use the next two month section.  You’ll notice strange change in open position column on the 16th of every month as the expiration is on the 15th of every month. The data is in continuous roll on expiration or unadjusted format.

But for my example, I’ll use the S&P 500 index futures continuous back-adjusted data from 1997 – 2008 (12 years) which you can download free from the internet.

I’ll use the data from 1997 – 2002 (6 years) as our in-sample to design, test and optimize on, using a simple strategy as an example to show the development, testing and optimization process in my future postings.