Please advise on p settings. Thanks.

Hello.
Ok, if it is just an example, I reveal it.
You sent me this strategy:
import numpy as np import pandas as pd def mySettings(): settings = {} settings['markets'] = ['F_CL', 'F_BC'] settings['beginInSample'] = '20171115' settings['endInSample'] = '20210201' settings['lookback'] = 84 settings['budget'] = 10**6 settings['slippage'] = 0.05 settings['zthreshold'] = 1.75 settings['p'] = np.array([0., 0.]) settings['count'] = 0 return settings def myTradingSystem(DATE, CLOSE, exposure, equity, settings): s1 = pd.Series(CLOSE[settings['lookback']:,0]) s2 = pd.Series(CLOSE[settings['lookback']:,1]) # Compute mean of the spread up to now mvavg = np.mean(np.log(s1/s2)) # Compute stdev of the spread up to now stdev = np.std(np.log(s1/s2)) # Compute spread current_spread = np.log(CLOSE[1,0] / CLOSE[1,1]) # Compute zscore zscore = (current_spread  mvavg) / stdev if stdev > 0 else 0 p = settings['p'] if zscore >= settings['zthreshold']: p = np.array([0.5,0.5]) settings['p'] = p elif zscore <= settings['zthreshold']: p = np.array([0.5,0.5]) settings['p'] = p else: p = settings['p'] return p, settings

At first, I rewrote it using the multipass approach:
import numpy as np import pandas as pd import xarray as xr import qnt.data as qndata import qnt.ta as qnta import qnt.backtester as qnbt import qnt.stats as qns zthreshold = 1.75 lookback_trading_days = 84 markets = ['F_CL', 'F_BC'] def load_data(period): return qndata.futures_load_data(assets=markets, tail=period) def strategy(data): close = data.sel(field='close', asset=markets).transpose('time', 'asset') s1 = close[lookback_trading_days:, 0] s2 = close[lookback_trading_days:, 1] # Compute mean of the spread up to now mvavg = np.mean(np.log(s1/s2)) # Compute stdev of the spread up to now stdev = np.std(np.log(s1/s2)) # Compute spread current_spread = np.log(close[1, 0] / close[1, 1]) # Compute zscore zscore = (current_spread  mvavg) / stdev if stdev > 0 else 0 if zscore >= zthreshold: p = [0.5,0.5] elif zscore <= zthreshold: p = [0.5,0.5] else: p = [0,0] # there is no way to use variable 'p' from previous pass # So, probably, the strategy is broken (*) return xr.DataArray(p, dims=['asset'], coords=dict(asset=markets)) weights = qnbt.backtest( competition_type= "futures", load_data= load_data, lookback_period= 365, start_date= "20060101", strategy= multi_pass_strategy )
Unfortunately, the backtester does not support passing variables between iterations (*). So this conversion is not fully correct.

Next, I modified this strategy using the singlepass approach.
import numpy as np import pandas as pd import xarray as xr import qnt.data as qndata import qnt.ta as qnta import qnt.backtester as qnbt import qnt.output as qnout import qnt.stats as qns zthreshold = 1.75 lookback_trading_days = 84 markets = ['F_CL', 'F_BC'] def single_pass_strategy(data): close = data.sel(field='close', asset=markets).transpose('time', 'asset') s1 = close.sel(asset=markets[0]) s2 = close.sel(asset=markets[1]) # Compute mean of the spread up to now mvavg = np.log(s1 / s2).rolling(time=lookback_trading_days, min_periods=2).mean() # Compute stdev of the spread up to now stdev = np.log(s1 / s2).rolling(time=lookback_trading_days, min_periods=2).std() # Compute spread current_spread = np.log(s1/s2) # Compute zscore zscore = xr.where(stdev > 0, (current_spread  mvavg) / stdev, 0) p1 = xr.where(zscore >= zthreshold, 0.5, xr.where(zscore <= zthreshold, 0.5, np.nan)) p2 = xr.where(zscore >= zthreshold, 0.5, xr.where(zscore <= zthreshold, 0.5, np.nan)) p = xr.concat([p1, p2], pd.Index(markets, name='asset')) p = p.ffill('time') # (*) return p data = qndata.futures_load_data(assets=markets, min_date='20050101') output = single_pass_strategy(data) output = qnout.clean(output, data) qnout.write(output) stat = qns.calc_stat(data, output) display(stat.to_pandas().tail()) # Or you can use the backtester to check looking forward. # # def load_data(period): # return qndata.futures_load_data(assets=markets, tail=period) # weights = qnbt.backtest( # competition_type= "futures", # load_data= load_data, # lookback_period= 2*365, # start_date= "20060101", # strategy= single_pass_strategy # )
As you see, this strategy calculates the whole output series in one pass using vector operations. It calculates p, when it is possible and uses ffill to fill missed values (*).

How do I align the two series (dropna for both) to make sure I'm using good data for both series on the same dates?

When you use the backtester or data load functions, you receive one xarray.DataArray as an argument. All data is already aligned. When the data is not available for some assets on some days, you will see the NaN value there.
When I generate a buy signal, that is when the zscore < zthreshold, how do I make the allocation to buy Contract1 and sell Contract2.
In the same way as it was in the previous backtester. But use xarray instead of np.array.
How do I keep that allocation when the backtest runs for the next day and so on until the signal changes to sell or the algo sends an exit signal?
There is no way to do that yet. We already created a feature request for this. Share the state between iterations
I wrote an example of how to bypass this in your strategy. But I understand that it is not the easiest way... Well, we will inform you when we add this feature to the new backtester.
Regards.


Hi @support,
Fantastico! Thank you very much, you are very helpful!!
I'll try these out.
Much appreciated! 
Hi @support,
I seem to be getting this error:Wrong output dimensions. ('time', 'asset') is not ('asset',) 100% (1289 of 1289) #################### Elapsed Time: 0:00:00 ETA: 00:00:00
Please advise.
Also, for a different strategy can you help me with the line of code that will tell me how many assets (the count) are up today:
today = data.sel(field="close") yesterday = data.sel(field="close").shift(time=1) uptoday = today.isel(1) > yesterday.isel[1] How do I count how many are uptoday?
Thanks for your help, greatly appreciated as always.

Wrong output dimensions.
I guess you see this error when you try to run the second example using qnbt.backtest
There was a requirement to return 1d array from the strategy function, so you can modify the code this way:
weights = qnbt.backtest( competition_type= "futures", lookback_period= 2*365, start_date= "20060101", strategy= lambda d: single_pass_strategy(d).isel(time=1) )
Recently, we updated the library: added the optimizer, and slightly improved the backtest function. Now it extracts the last day automatically. So you can just update the library, run this in your env:
conda install 'quantiacssource::qnt>=0.0.228'

@spancham said in Please advise on p settings. Thanks.:
Also, for a different strategy can you help me with the line of code that will tell me how many assets (the count) are up today:
print(len(uptoday.isel(time=1).dropna('asset').asset))
Does it suit?
Regards.

Hi @support
Thank you, I'll try that. 
This post is deleted!