Submission Issue

antinomy

Hello.

Recently it takes an awful long time for algos to get checked after submission. I guess there are a lot of uploads these days...
My problem is, that I submitted one that took 5 days to evaluate just to get rejected.
It says

Not enough bid information. The strategy must trade from January 1, 2014.

But it does start trading on this date, so I don't know what the issue is.
When I look at the logs, the date that failed is 2021-05-07. Perhaps the problem is that I submitted it on 2021-05-06?
In the short description for that day it says

FAIL: 2021-05-11T22:27:50Z: filter failed: in sample period is too short:0 < 1656

Any ideas what I could do to avoid this kind of rejections? Trial and error would definitely take to long in this case
Thanks in advance!

support

@antinomy Hello, thanks for reporting, 2 questions:

are you using the stateless or stateful version of the backtester? First one uses parallelization, second one obviously not as it needs to process data sequentially;
can you try to print the weights? One common problem is generation of vanishing weights. The system should start generating non-zero weights from the beginning, otherwise we should compare systems with effectively different track records.

antinomy

@support
Thanks for getting back to me.

Yes I'm using the stateful version, allthough this algo doesn't really need it (now that you mention it, I think you said already elsewhere that stateful tests take longer to evaluate).
I checked the weights:

weights, state = qnbt.backtest(
    strategy=trade,
    competition_type="cryptofutures",
    load_data=load_data,
    lookback_period=1000,
    start_date='2014-01-01',
    window=build_data_for_one_step,
)
df = weights.to_pandas()
zero = df == 0
zero

asset BTC
time
2014-01-01 False
2014-01-02 False
2014-01-03 False
2014-01-04 False
2014-01-05 False
... ...
2021-05-09 False
2021-05-10 False
2021-05-11 False
2021-05-12 False
2021-05-13 False

At least the first 5 weights aren't 0, in fact there are only a few days with zero weights:

zero[zero.values]

asset BTC
time
2014-04-27 True
2014-04-28 True
2014-05-09 True
2014-05-29 True
2014-06-05 True
2019-04-19 True
2019-05-30 True

I also checked for non-finite values (allthough I think the backtester already sets them to 0)

not_finite = ~np.isfinite(df)
not_finite.sum()

asset
BTC 0
dtype: int64

support

@antinomy Thank you.

How do you persist the state? Using some persistable data type (dict, list, xarray.DataArray, pandas.DataFrame,...) or maybe a class?

Could you print the state, with:

print('state:', state)

maybe we have a problem with that.

antinomy

@support
It's a class, the code looks like this

class State:
    pass

def trade(data, state):
    if state is None:
        state = State
    """
    calculating weights and in some algos store sth. in the state class...
    """
    return weights, state

The output of

print('state', state)

is:
state <class 'main.State'>

antinomy

Maybe something else of interest:
There's also some code that will return zero-weights in the first 2 iterations. I put it in there because the backtester always runs the first and last pass before actually backtesting, so the variables at the beginning of the actual backtest were in fact the ones from the last pass.
I didn't think this could be a problem because the resulting weights were as expected, but now I'm wondering...

antinomy

Update:
The stateless version got accepted now, thanks!

support

@antinomy Ok, good to know. About the delay of 5 days you experienced, we are sorry, there was a network issue which has been slowing down processing of algorithms for some days.

support

Also, stateful strategies can't be processed in a parallel manner, so the processing will be slower a stateless one. It may be 10 times slower.

You can see in the log how long it takes to process one day and simply multiply it by the number of days in the in-sample period.

support

@antinomy said in Submission Issue:

class State:
pass

def trade(data, state):
if state is None:
state = State
"""
calculating weights and in some algos store sth. in the state class...
"""
return weights, state

Class as the state also looks suspicious. I guess that pickle can't correctly serialize and deserialize it. Can you use a dict instead of the class?

antinomy

@support Sure, I just liked the dot-notation for classes

antinomy

Just out of curiousity I did some testing and it looks like the class actually was the culprit.
I submitted a simple strategy in 2 versions, one with a class and the other with a dictionary as state. The class version was rejected (exaclty like the one from my 1st post) and the dictionary version got accepted.