Q20 contest results
-
@azure-machamp We are checking all the systems. It is clear that in many cases a hand-picking of stocks has been done, which is not a quantitative criterion.
-
@support: Wasn't this an issue that should have been reported at the start of the competition? I understand the business principle. But given the large differences in market value and growth of the Nasdaq 100 shares, was this somewhat to be expected? I tried, perhaps not very clearly, to indicate this problem in my post of Nov 10, 2022 about the zipline pipeline functionality. Better risk and diversification requirements may be needed for strategies in future competitions.
-
@theflyingdutchman The rules of all stock contests clearly say that "Each trading system must use quantitative rules only". That was stated before any contest.
Hand-picking of stocks is not a quantitative rule. Hand-picking of stocks allows to easily game the contest, for example by submitting 15 long only trading systems on single stocks chosen in a discretionary way. The purpose of these contests is to develop systems which are working on stocks and are free from survivorship bias.
Unfortunately, in contrast with rules which can be checked automatically, like "Sharpe ratio in sample must be larger than 1", it is more difficult to look at the generated positions and say: Hey, this user is hand-picking stocks.
It is well possible that the selected stocks are the results of a dynamic selection criteria which, for whatever reason, leads to select a total of 20 stocks. We can only say that, for several systems, it is very plausible that stocks have been hand-picked.
That is why we are investigating the results.
-
@theflyingdutchman said in Q20 contest results:
I tried, perhaps not very clearly, to indicate this problem in my post of Nov 10, 2022 about the zipline pipeline functionality. Better risk and diversification requirements may be needed for strategies in future competitions
You are absolutely correct, we agree and are working on criteria which will allow better risk and diversification as you say.
-
@support said in Q20 contest results:
Hand-picking of stocks allows to easily game the contest, for example by submitting 15 long only trading systems on single stocks chosen in a discretionary way. The purpose of these contests is to develop systems which are working on stocks and are free from survivo
Hello,
if there's a hint of manual stock selection,
how do you rule on disqualification?
thanks in advance -
If a user selects stocks dynamically, we expect to see some kind of output, namely a large data structure with several empty columns, as some stocks will never be traded, and others partially traded.
We are already contacting the users and getting from them the details on the way stocks were selected.
-
@support said in Q20 contest results:
W
Hi,
thank you for these answers. So the disqualification will depend on how the user selected the actions? -
@dark-pidgeot The correct way to load stocks, free from survivorship bias, is:
data = qndata.stocks.load_ndx_data(min_date="2005-06-01")
Later you can select index members dynamically as:
is_liquid = data.sel(field="is_liquid")
weights = weights * is_liquidassuming that you computed your weights.
Survivorship bias follows from loading a specific hand-picked set of stocks in the load function. Or selecting specific stocks later by indicating explicitely their name.
-
@support Am I disqualified if I do this?
data = qndata.stocks.load_ndx_data(min_date="2005-06-01", assets=["NAS:AAPL", "NAS:AMZN", "NAS:MRNA", "NAS:MSFT"])
I select a few stocks, it's not a dynamic selection
-
@support I don't really understand, if I have a strategy that works on 10 stocks for example, to be eligible for the price, I absolutely have to trade all the stocks?
-
@dark-pidgeot I would think it would be fair if the criteria were that the strategy added value somewhere. In the period from January 2005 to now, additional value should have been created compared to a long-only investment in those four shares.
-
@dark-pidgeot No, you can trade 10 stocks, you do not have to trade all of them. For example you can load all the stocks, then select dynamically the liquid ones, then sort them according to some criterion (example: trade at each point in time only the top 10 in terms of inverse volatility) and that is fine.
But if you choose by hand 10 stocks, then you incur into survivorship bias, as you are selecting 10 stocks which are alive (and most likely successful if you go long only) now, and are using future information (now) to make your selection (hand-picking these 10 stocks at the beginning of the simulation).
-
@theflyingdutchman That is also a good observation. In general, we would like a long-only system with low turnover to beat the Nasdaq index for example. But still, we see a lot of problems with hand-picking stocks and survivorship-bias affected systems.
-
@support I always ask myself the question, when we look at the winners of other competitions, there are some who trade just 5 stocks. can we consider this to be hand-picking?
-
@support said in Q20 contest results:
That is also a good observation. In general, we would like a long-only system with low turnover to beat the Nasdaq index for example. But still, we see a lot of problems with hand-picking stocks and survivorship-bias affected systems.
if I understand correctly
if I initially select 10 stocks, and even if I have a good sharpe ratio, my strategy can make long and short positions. I could be disqualified
-
@theflyingdutchman it seemed to me that strategies should be able to open long and short positions
-
@dark-pidgeot said in Q20 contest results:
@support said in Q20 contest results:
That is also a good observation. In general, we would like a long-only system with low turnover to beat the Nasdaq index for example. But still, we see a lot of problems with hand-picking stocks and survivorship-bias affected systems.
if I understand correctly
if I initially select 10 stocks, and even if I have a good sharpe ratio, my strategy can make long and short positions. I could be disqualified
That's my understanding too. If you initially select 10 stocks, that already implies survivorship-bias, and is in fact "forward looking" because in for instance year 2010 you could not foresee which 10 stocks would still exist in the index in 2024.
-
@buyers_are_back Yes, that is the forward looking associated to survivorship bias. In order to avoid that, the dataset includes also stocks which were part of the index but have been delisted at some point. These stocks should be taken potentially in consideration by any algorithm, which does not happen by hand-picking stocks.
One can reverse the logic and conclude the same for stocks which "should" have been shorted in the pasted because of observing an a posteriori delisting.
-
@dark-pidgeot that is fine
-
@dark-pidgeot yes, ybut ou can start from the full index, and selec dynamically 10 according to some criteria, and then take long and short positions.
