Submission Issue

support

@antinomy Hello, thanks for reporting, 2 questions:

are you using the stateless or stateful version of the backtester? First one uses parallelization, second one obviously not as it needs to process data sequentially;
can you try to print the weights? One common problem is generation of vanishing weights. The system should start generating non-zero weights from the beginning, otherwise we should compare systems with effectively different track records.

antinomy

@support
Thanks for getting back to me.

Yes I'm using the stateful version, allthough this algo doesn't really need it (now that you mention it, I think you said already elsewhere that stateful tests take longer to evaluate).
I checked the weights:

weights, state = qnbt.backtest(
    strategy=trade,
    competition_type="cryptofutures",
    load_data=load_data,
    lookback_period=1000,
    start_date='2014-01-01',
    window=build_data_for_one_step,
)
df = weights.to_pandas()
zero = df == 0
zero

asset BTC
time
2014-01-01 False
2014-01-02 False
2014-01-03 False
2014-01-04 False
2014-01-05 False
... ...
2021-05-09 False
2021-05-10 False
2021-05-11 False
2021-05-12 False
2021-05-13 False

At least the first 5 weights aren't 0, in fact there are only a few days with zero weights:

zero[zero.values]

asset BTC
time
2014-04-27 True
2014-04-28 True
2014-05-09 True
2014-05-29 True
2014-06-05 True
2019-04-19 True
2019-05-30 True

I also checked for non-finite values (allthough I think the backtester already sets them to 0)

not_finite = ~np.isfinite(df)
not_finite.sum()

asset
BTC 0
dtype: int64

support

@antinomy Thank you.

How do you persist the state? Using some persistable data type (dict, list, xarray.DataArray, pandas.DataFrame,...) or maybe a class?

Could you print the state, with:

print('state:', state)

maybe we have a problem with that.

antinomy

@support
It's a class, the code looks like this

class State:
    pass

def trade(data, state):
    if state is None:
        state = State
    """
    calculating weights and in some algos store sth. in the state class...
    """
    return weights, state

The output of

print('state', state)

is:
state <class 'main.State'>

antinomy

Maybe something else of interest:
There's also some code that will return zero-weights in the first 2 iterations. I put it in there because the backtester always runs the first and last pass before actually backtesting, so the variables at the beginning of the actual backtest were in fact the ones from the last pass.
I didn't think this could be a problem because the resulting weights were as expected, but now I'm wondering...

antinomy

Update:
The stateless version got accepted now, thanks!

support

@antinomy Ok, good to know. About the delay of 5 days you experienced, we are sorry, there was a network issue which has been slowing down processing of algorithms for some days.

support

Also, stateful strategies can't be processed in a parallel manner, so the processing will be slower a stateless one. It may be 10 times slower.

You can see in the log how long it takes to process one day and simply multiply it by the number of days in the in-sample period.

support

@antinomy said in Submission Issue:

class State:
pass

def trade(data, state):
if state is None:
state = State
"""
calculating weights and in some algos store sth. in the state class...
"""
return weights, state

Class as the state also looks suspicious. I guess that pickle can't correctly serialize and deserialize it. Can you use a dict instead of the class?

antinomy

@support Sure, I just liked the dot-notation for classes

antinomy

Just out of curiousity I did some testing and it looks like the class actually was the culprit.
I submitted a simple strategy in 2 versions, one with a class and the other with a dictionary as state. The class version was rejected (exaclty like the one from my 1st post) and the dictionary version got accepted.