Submission Logic Questions
-
Hi guys, after exploring quantics for over a week, I came up with some questions I could not answer after reading the documentation.
-
How often are the OOS results computed after submission? Weekly basis? It looks like it will take some days for your submissions to display OOS metrics.
-
How does the submission logic work? To be more precise, let's suppose that you submitted your strategy. For today, 16th Jan, the weights used are the ones the model sent for 15th Jan? What I am trying to say is that usually, the notebook will download all available data from 2006 up to 15th Jan and will follow whatever logic to generate predictions for all that time period (including 15th Jan), so does Quantiacs take the last weights of the model (15th Jan which was the latest available data) to use them for 16th Jan? or how does it work?
-
Regarding the 10-minute limit, it seems that you must develop a model (ML or not) within that time. This means that if you try to train a model that takes more than 10 minutes, it might not work. Can we train our model locally, save it in a folder in the notebook, and use it during the live contest? Otherwise, most of the more complex ML might not be doable.
-
-
Dear @auxiliary-snail,
- Out of sample score is currently calculated every 4-5 days, depending on the internal configuration and currently available data, but the plan is to do the calculation immediately after getting new available data.
- Yes, when you generate weights for some day, those weights are used with the data on the following day.
- Yes, it is possible, in your strategy environment you can upload additional files that you would like to use, and then you could load that model directly into your code and use it for generating weights only.
Best regards
-
@support I forgot one more question. This is about the strategies allowed to participate in the contest, which have required a shape ratio of 0.7 since 2006. My take is that requirement limits the universe of strategies to those that generalize over long-term periods, meaning that if the patterns persist over time, it will profit from them in the long run. However, some strategies that might work well now (2024-2025) might not have worked well in the past, implying that there's a limit to using dynamic strategy as the market evolves. Some strategies might have a good Sharpe ratio in old periods (let's say 2006-2015), whereas other strategies might perform well after 2015 due to changes in market dynamics. I guess it might not be clear, but summarizing, how can we deal with strategies that might work in recent times but not in the past, which might lead to a low Sharpe ratio (past = bad, present = good, overall= bad) and no consideration for the contest. Can I submit a notebook with different models depending on the moment in time we are in? Let's say I want to use X model for 2006-2015 and Y model for 2015 onwards. If the performance is good, can the notebook (which contains multiple models) be accepted?
-
@auxiliary-snail Hi,
unfortunately, this is not allowed and in accordance with the rules. Using hard-coded time periods in which trading algorithm will work differently, is not a quantitative method (just like manual asset selection, e.g. "trade only Apple or Microsoft"). We still haven't implemented a mechanism for automatic recognition of such behaviors in trading strategies, and even though a strategy could be successfully submitted, it will not be eligible for prize winning.
What we are searching for, is well performing strategy over entire in_sample period (SR>0.7), robust to all market movements 2006-2025, so we can expect it will perform well in future, too.
