Recently it takes an awful long time for algos to get checked after submission. I guess there are a lot of uploads these days...
My problem is, that I submitted one that took 5 days to evaluate just to get rejected.
Not enough bid information. The strategy must trade from January 1, 2014.
But it does start trading on this date, so I don't know what the issue is.
When I look at the logs, the date that failed is 2021-05-07. Perhaps the problem is that I submitted it on 2021-05-06?
In the short description for that day it says
FAIL: 2021-05-11T22:27:50Z: filter failed: in sample period is too short:0 < 1656
Any ideas what I could do to avoid this kind of rejections? Trial and error would definitely take to long in this case
Thanks in advance!
@antinomy Hello, thanks for reporting, 2 questions:
are you using the stateless or stateful version of the backtester? First one uses parallelization, second one obviously not as it needs to process data sequentially;
can you try to print the weights? One common problem is generation of vanishing weights. The system should start generating non-zero weights from the beginning, otherwise we should compare systems with effectively different track records.
Thanks for getting back to me.
- Yes I'm using the stateful version, allthough this algo doesn't really need it (now that you mention it, I think you said already elsewhere that stateful tests take longer to evaluate).
- I checked the weights:
weights, state = qnbt.backtest( strategy=trade, competition_type="cryptofutures", load_data=load_data, lookback_period=1000, start_date='2014-01-01', window=build_data_for_one_step, ) df = weights.to_pandas() zero = df == 0 zero
At least the first 5 weights aren't 0, in fact there are only a few days with zero weights:
I also checked for non-finite values (allthough I think the backtester already sets them to 0)
not_finite = ~np.isfinite(df) not_finite.sum()
@antinomy Thank you.
How do you persist the state? Using some persistable data type (dict, list, xarray.DataArray, pandas.DataFrame,...) or maybe a class?
Could you print the state, with:
maybe we have a problem with that.
It's a class, the code looks like this
class State: pass def trade(data, state): if state is None: state = State """ calculating weights and in some algos store sth. in the state class... """ return weights, state
The output of
state <class 'main.State'>
Maybe something else of interest:
There's also some code that will return zero-weights in the first 2 iterations. I put it in there because the backtester always runs the first and last pass before actually backtesting, so the variables at the beginning of the actual backtest were in fact the ones from the last pass.
I didn't think this could be a problem because the resulting weights were as expected, but now I'm wondering...
The stateless version got accepted now, thanks!
@antinomy Ok, good to know. About the delay of 5 days you experienced, we are sorry, there was a network issue which has been slowing down processing of algorithms for some days.
Also, stateful strategies can't be processed in a parallel manner, so the processing will be slower a stateless one. It may be 10 times slower.
You can see in the log how long it takes to process one day and simply multiply it by the number of days in the in-sample period.
def trade(data, state):
if state is None:
state = State
calculating weights and in some algos store sth. in the state class...
return weights, state
Class as the state also looks suspicious. I guess that pickle can't correctly serialize and deserialize it. Can you use a dict instead of the class?
@support Sure, I just liked the dot-notation for classes
Just out of curiousity I did some testing and it looks like the class actually was the culprit.
I submitted a simple strategy in 2 versions, one with a class and the other with a dictionary as state. The class version was rejected (exaclty like the one from my 1st post) and the dictionary version got accepted.