Computing The Rank List

April, 12 2016

Some of you asked us how we compute the rank list for our algorithmic trading competitions. Thanks for your questions! Here is the answer - from upload to final results.

Eligible and Secure?

Once you upload your trading algorithm to our webserver we check it for eligibility. What does that mean? The competition rules state that your trading algorithm needs to be:
  • built on futures
  • unique (and not just one of our own sample systems)
  • deterministic
The last point is important because your code remains a black box to us. We need to make sure that it returns the exact same results if we evaluate it twice. The algorithm you submit is the one we might use in production trading significant investments. Re-running it and verifying matching results ensures that the back- and live-tests are reproducible and represent the actual performance of your algorithm. 
Finally, there is also a security check in which we test if your code performs any file operations or network requests. If both checks are negative, we flag your system as eligible for the competition and it shows up on the competition page.
 

Same Data Means Same Chances For All Systems

During the submission phase you can upload as many algorithms as you want. We start to simulate them with daily live data immediately after it shows up as an eligible system in the competition. This is why you see some computed live-results for your system during the submission phase. They can be helpful to give you preliminary clues about how your algo is doing on live-data. However, with the start of the official live-simulation (= end of the submission phase) we reset the competition page. This reset is done to give all systems equal chances in the competition. From this initial start on all systems are simulated on the same live data. Throughout the competition we update the data on a daily basis. We receive the data from our data vendor on market close at which point we start the evaluation of the systems – overnight (Pacific Time).
 


 

Incremental Versus Final Evaluation?

So, how does live-simulation exactly work? With the start of the live-simulation we run all submitted and eligible algorithms on the latest day’s data. Day after day we append the latest trading results to the existing trading days. We do this for all trading days throughout the competition. At the end of the competition we reevaluate all systems from backtest to the last live day to verify the computed results. This way we eliminate potential errors arising from ex-post changes of the input data (e.g. corrected settlements or missing data points). This is also a crosscheck of the incrementally built live-simulation.    
The result of this final re-evaluation is the confirmed live-simulation result. The lower of this live Sharpe Ratio and the Sharpe Ratio of your backtest is your final score in the competition.

Let us know if you have any questions. Write in the forum or shoot me an email: dirk@quantiacs.com