Advanced search
Quantiacs > update on quantiacs, data tools, evaluating historical performance View modes: 
wheel_of_law - 4/10/2019 7:20:19 AM
update on quantiacs, data tools, evaluating historical performance
 I had some discussions with Martin when he was at Quantiacs, about how the firm was structured at the time, and what the offerings were available for instutional investors looking to place allocations with Quantiacs. 

I felt kindly disposed towards Quantiacs and their mission, but wanted to wait and see what happened in the future because I had some mild misgivings about competition scoring as well as the way trading system results are reported. 

I am happy to see that the backtesting SDK has been updated. 

It is still difficult to get a picture of how all the various strategies that have been uploaded have performed out of sample, either live or on-paper out of sample. I have picked through quite a few strategies that have been live for over 700 days, and it is cumbersome to say the least to zoom in on the live portion of the equity curve to get a glimpse of the true performance numbers. 

I fear that the current contest system is vulnerable to selection bias. I am not sure that this fear is borne out, because I cannot effectively analyse the data. 

My fear then, is the same fear now - strategies that happen to have high performance during the short contest period are favored over stratgies that might have less variance in performance, and which are the true winners in the long term. 

A possible fix to this would be to roll each competition forward with a year or two years of out of sample testing. Then time is moved forward another year, everyone gets updated data, and the oracle scores everyone on the next segment. Once you get to the present day, you have many years of "out of sample" data. This method does not address way to game the system of course. 

Another approach is to just not trust any results for a few years. Thankfully this is possible now that Quantiacs has existed for a while. But it seems slightly sad for the authors of reasonably constructed strategies to potentially "lose" these competitions to strategies that have high variance, or are the beneficiaries of random selection.
Are my concerns misplaced?  Any thoughts from long time participants?