However, the participants had the opportunity to submit the same code (provided it is correct) to the Q21 (in other words no correlations check are performed).

It is very cumbersome to remove strategies from the web page, so they are currently displayed.

If your strategy will continue to perform good and we believe it can be traded, we will contact you.

One can reverse the logic and conclude the same for stocks which "should" have been shorted in the pasted because of observing an a posteriori delisting.

@support said in Q20 contest results:

That is also a good observation. In general, we would like a long-only system with low turnover to beat the Nasdaq index for example. But still, we see a lot of problems with hand-picking stocks and survivorship-bias affected systems.

if I understand correctly

if I initially select 10 stocks, and even if I have a good sharpe ratio, my strategy can make long and short positions. I could be disqualified

That's my understanding too. If you initially select 10 stocks, that already implies survivorship-bias, and is in fact "forward looking" because in for instance year 2010 you could not foresee which 10 stocks would still exist in the index in 2024.

That is also a good observation. In general, we would like a long-only system with low turnover to beat the Nasdaq index for example. But still, we see a lot of problems with hand-picking stocks and survivorship-bias affected systems.

if I understand correctly

if I initially select 10 stocks, and even if I have a good sharpe ratio, my strategy can make long and short positions. I could be disqualified

But if you choose by hand 10 stocks, then you incur into survivorship bias, as you are selecting 10 stocks which are alive (and most likely successful if you go long only) now, and are using future information (now) to make your selection (hand-picking these 10 stocks at the beginning of the simulation).

data = qndata.stocks.load_ndx_data(min_date="2005-06-01", assets=["NAS:AAPL", "NAS:AMZN", "NAS:MRNA", "NAS:MSFT"])

I select a few stocks, it's not a dynamic selection

data = qndata.stocks.load_ndx_data(min_date="2005-06-01")

Later you can select index members dynamically as:

is_liquid = data.sel(field="is_liquid")
weights = weights * is_liquid

assuming that you computed your weights.

Survivorship bias follows from loading a specific hand-picked set of stocks in the load function. Or selecting specific stocks later by indicating explicitely their name.

W
Hi,
thank you for these answers. So the disqualification will depend on how the user selected the actions?

If a user selects stocks dynamically, we expect to see some kind of output, namely a large data structure with several empty columns, as some stocks will never be traded, and others partially traded.

We are already contacting the users and getting from them the details on the way stocks were selected.

Hand-picking of stocks allows to easily game the contest, for example by submitting 15 long only trading systems on single stocks chosen in a discretionary way. The purpose of these contests is to develop systems which are working on stocks and are free from survivo

Hello,

if there's a hint of manual stock selection,
how do you rule on disqualification?
thanks in advance

I tried, perhaps not very clearly, to indicate this problem in my post of Nov 10, 2022 about the zipline pipeline functionality. Better risk and diversification requirements may be needed for strategies in future competitions

You are absolutely correct, we agree and are working on criteria which will allow better risk and diversification as you say.

Hand-picking of stocks is not a quantitative rule. Hand-picking of stocks allows to easily game the contest, for example by submitting 15 long only trading systems on single stocks chosen in a discretionary way. The purpose of these contests is to develop systems which are working on stocks and are free from survivorship bias.

Unfortunately, in contrast with rules which can be checked automatically, like "Sharpe ratio in sample must be larger than 1", it is more difficult to look at the generated positions and say: Hey, this user is hand-picking stocks.

It is well possible that the selected stocks are the results of a dynamic selection criteria which, for whatever reason, leads to select a total of 20 stocks. We can only say that, for several systems, it is very plausible that stocks have been hand-picked.

That is why we are investigating the results.