Please advise on p settings. Thanks.

spancham

Hello, first of all, thanks for your great platform. Beats having to build my own backtester so I am trying to learn it.
I am attempting a simple pairs trading strategy for the WTI vs Brent Crude spread, code below.
For some reason, it does not stop running & also does not plot the Equity chart. I don't believe I am understanding correctly how to set the position exposure.

And the following warning is given:
C:\Users\sheikh\anaconda3\lib\site-packages\quantiacsToolbox\quantiacsToolbox.py:881: RuntimeWarning: invalid value encountered in true_divide
position = position / np.sum(abs(position))

import numpy as np
import pandas as pd
  
def  mySettings():
    settings = {}
    settings['markets'] = ['F_CL', 'F_BC']
    settings['beginInSample'] = '20171115'
    settings['endInSample'] = '20210201'
    settings['lookback'] = 84
    settings['budget'] = 10**6
    settings['slippage'] = 0.05
    settings['zthreshold'] = 1.75
    settings['p'] = np.array([0., 0.])
    settings['count'] = 0
    return settings

def myTradingSystem(DATE, CLOSE, exposure, equity, settings):
    s1 = pd.Series(CLOSE[-settings['lookback']:,0])
    s2 = pd.Series(CLOSE[-settings['lookback']:,1])

    # Compute mean of the spread up to now
    mvavg = np.mean(np.log(s1/s2))

    # Compute stdev of the spread up to now
    stdev = np.std(np.log(s1/s2))

    # Compute spread
    current_spread = np.log(CLOSE[-1,0] / CLOSE[-1,1])

    # Compute z-score
    zscore = (current_spread - mvavg) / stdev if stdev > 0 else 0
    
    p = settings['p']

    if zscore >= settings['zthreshold']:
        p = np.array([-0.5,0.5])
        settings['p'] = p
    elif zscore <= -settings['zthreshold']:
        p = np.array([0.5,-0.5])
        settings['p'] = p
    else:
        p = settings['p']
        
    settings['count'] += 1
    if settings['count'] % 50 == 0:
        print(settings['count'], equity, exposure)
    
    return p, settings

support

@spancham Hello, the backtester you are using is the "old" Quantiacs backtester we used for the contests until Q14 (in other words, until end of 2020). The new code is available at https://github.com/Quantiacs, and can be installed locally using conda and reading the instructions at:

https://quantiacs.com/documentation/en/user_guide/local_development.html

It is based on exposure, but much faster and more flexible.

Anyhow, I copied and pasted your example and it works fine, but at the end of your file I added:

if __name__ == '__main__':
    from quantiacsToolbox.quantiacsToolbox import runts

    results = runts(__file__)

This is a call to the "runts" function from the "old" Quantiacs toolbox.

The warning you see is probably related to an attempted division by a null value, when the denominator is vanishing.

Regards

spancham

@support Hi again, thanks very much for your earlier response.
I am now using the latest Quantiacs set up based on the links & direction you provided. So that is greatly appreciated!
I tried some of the Quantiacs sample strategies to double check that things are running properly. Things look okay so far. I'll now turn to converting the strategy I'm working on to the new format.
I do have a question however:
I can run strategy.py from the command prompt but when I try to run it from my local Jupyter notebook it's not finding the qnt module:

# run from Jupyter
%run C:\Users\sheikh\anaconda3\envs\qntdev\Scripts\strategy.py

ModuleNotFoundError                       
Traceback (most recent call last)
~\anaconda3\envs\qntdev\Scripts\strategy.py in <module>
      1 # Multi-Pass implementation
----> 2 import qnt.ta as qnta
      3 import qnt.data as qndata
      4 import qnt.backtester as qnbk

ModuleNotFoundError: No module named 'qnt'

I even placed the strategy.py file in the qntdev\Scripts folder.
Would you happen to know what I can try differently to get it to run from my local Jupyter notebook?
Thanks again.

support

@spancham

Hello.

I guess your jupyter uses the wrong environment.

I am starting my jupyter this way:

C:\Users\User> conda activate qntdev

(qntdev) C:\Users\User> where jupyter
C:\Users\User\anaconda3\Scripts\jupyter.exe
# it means that there is no jupyter in qntdev env
# so I have to install it

(qntdev) C:\Users\User>conda install -y notebook
...

(qntdev) C:\Users\User> where jupyter
C:\Users\User\anaconda3\envs\qntdev\Scripts\jupyter.exe
C:\Users\User\anaconda3\Scripts\jupyter.exe

(qntdev) C:\Users\User> C:\Users\User\anaconda3\envs\qntdev\Scripts\jupyter.exe notebook

When the jupyter starts, the %run is working. Ensure that you activated the 'qntdev' env and started the jupyter from this environment.

I hope this will help.

Regards.

spancham

Hi again @support
I read through everything and all the examples but to be honest the new format is not straightforward to me. I was wondering if there are Quantiacs developers available that you can recommend or put me in contact with to help convert my pairs strategy from the legacy toolbox format to the new format. I do not mind paying for their services.

Once I see how it's done using my strategy, I think I can understand how to set up my other strategies from there on.
Thanks for your help.
Sheikh

support

@spancham Hi, sure, you can simply send a mail with details to support@quantiacs.com and we will come back to you.

support

@spancham No payment needed.....

support

@spancham sure, pls write the support@quantiacs.com for details

spancham

This post is deleted!

support

This post is deleted!

spancham

@support
Thanks very much for your help.
Don't worry about me posting my strategy to a public forum.
I would never post my "secret sauce"
I'm just trying to understand the new format for the platform by setting up a pairs strategy.
I read through your documentation which is good, but none of your examples show how to set up a pairs trading strategy. I'm sure others would benefit from a pairs trading example.

I have a number of questions that I hope you'll illustrate in your example to come:

How do I align the two series (dropna for both) to make sure I'm using good data for both series on the same dates?
When I generate a buy signal, that is when the zscore < -zthreshold, how do I make the allocation to buy Contract1 and sell Contract2.
How do I keep that allocation when the backtest runs for the next day and so on until the signal changes to sell or the algo sends an exit signal?
Thanks again.
Sheikh

support

Hello.

Ok, if it is just an example, I reveal it.

You sent me this strategy:

import numpy as np
import pandas as pd

def  mySettings():
    settings = {}
    settings['markets'] = ['F_CL', 'F_BC']
    settings['beginInSample'] = '20171115'
    settings['endInSample'] = '20210201'
    settings['lookback'] = 84
    settings['budget'] = 10**6
    settings['slippage'] = 0.05
    settings['zthreshold'] = 1.75
    settings['p'] = np.array([0., 0.])
    settings['count'] = 0
    return settings

def myTradingSystem(DATE, CLOSE, exposure, equity, settings):
    s1 = pd.Series(CLOSE[-settings['lookback']:,0])
    s2 = pd.Series(CLOSE[-settings['lookback']:,1])

    # Compute mean of the spread up to now
    mvavg = np.mean(np.log(s1/s2))

    # Compute stdev of the spread up to now
    stdev = np.std(np.log(s1/s2))

    # Compute spread
    current_spread = np.log(CLOSE[-1,0] / CLOSE[-1,1])

    # Compute z-score
    zscore = (current_spread - mvavg) / stdev if stdev > 0 else 0

    p = settings['p']

    if zscore >= settings['zthreshold']:
        p = np.array([-0.5,0.5])
        settings['p'] = p
    elif zscore <= -settings['zthreshold']:
        p = np.array([0.5,-0.5])
        settings['p'] = p
    else:
        p = settings['p']

    return p, settings

support

At first, I rewrote it using the multi-pass approach:

import numpy as np
import pandas as pd
import xarray as xr

import qnt.data as qndata
import qnt.ta as qnta
import qnt.backtester as qnbt
import qnt.stats as qns


zthreshold =  1.75
lookback_trading_days = 84
markets = ['F_CL', 'F_BC']


def load_data(period):
    return qndata.futures_load_data(assets=markets, tail=period)


def strategy(data):
    close = data.sel(field='close', asset=markets).transpose('time', 'asset')
    s1 = close[-lookback_trading_days:, 0]
    s2 = close[-lookback_trading_days:, 1]
    
    # Compute mean of the spread up to now
    mvavg = np.mean(np.log(s1/s2))
    
    # Compute stdev of the spread up to now
    stdev = np.std(np.log(s1/s2))
    
    # Compute spread
    current_spread = np.log(close[-1, 0] / close[-1, 1])

    # Compute z-score
    zscore = (current_spread - mvavg) / stdev if stdev > 0 else 0
    
    if zscore >= zthreshold:
        p = [-0.5,0.5]
    elif zscore <= -zthreshold:
        p = [0.5,-0.5]
    else:
        p = [0,0] # there is no way to use variable 'p' from previous pass
        # So, probably, the strategy is broken (*)
    
    return xr.DataArray(p, dims=['asset'], coords=dict(asset=markets))


weights = qnbt.backtest(
    competition_type= "futures",
    load_data= load_data,
    lookback_period= 365,
    start_date= "2006-01-01",
    strategy= multi_pass_strategy
)

Unfortunately, the backtester does not support passing variables between iterations (*). So this conversion is not fully correct.

support

Next, I modified this strategy using the single-pass approach.

import numpy as np
import pandas as pd
import xarray as xr

import qnt.data as qndata
import qnt.ta as qnta
import qnt.backtester as qnbt
import qnt.output as qnout
import qnt.stats as qns


zthreshold =  1.75
lookback_trading_days = 84
markets = ['F_CL', 'F_BC']


def single_pass_strategy(data):
    close = data.sel(field='close', asset=markets).transpose('time', 'asset')
    s1 = close.sel(asset=markets[0])
    s2 = close.sel(asset=markets[1])
       
    # Compute mean of the spread up to now
    mvavg = np.log(s1 / s2).rolling(time=lookback_trading_days, min_periods=2).mean()
    
    # Compute stdev of the spread up to now
    stdev = np.log(s1 / s2).rolling(time=lookback_trading_days, min_periods=2).std()
       
    
    # Compute spread
    current_spread = np.log(s1/s2)
    
    # Compute z-score
    zscore = xr.where(stdev > 0, (current_spread - mvavg) / stdev, 0)
    
    p1 = xr.where(zscore >= zthreshold, -0.5, xr.where(zscore <= -zthreshold, 0.5, np.nan))
    p2 = xr.where(zscore >= zthreshold, 0.5, xr.where(zscore <= -zthreshold, -0.5, np.nan))
    
    p = xr.concat([p1, p2], pd.Index(markets, name='asset'))
    
    p = p.ffill('time') # (*)
    
    return p


data = qndata.futures_load_data(assets=markets, min_date='2005-01-01')
output = single_pass_strategy(data)
output = qnout.clean(output, data)
qnout.write(output)
stat = qns.calc_stat(data, output)
display(stat.to_pandas().tail())


# Or you can use the backtester to check looking forward.
#
# def load_data(period):
#     return qndata.futures_load_data(assets=markets, tail=period)

# weights = qnbt.backtest(
#     competition_type= "futures",
#     load_data= load_data,
#     lookback_period= 2*365,
#     start_date= "2006-01-01",
#     strategy= single_pass_strategy
# )

As you see, this strategy calculates the whole output series in one pass using vector operations. It calculates p, when it is possible and uses ffill to fill missed values (*).

support

How do I align the two series (dropna for both) to make sure I'm using good data for both series on the same dates?

When you use the backtester or data load functions, you receive one xarray.DataArray as an argument. All data is already aligned. When the data is not available for some assets on some days, you will see the NaN value there.
You can use dropna or ffill

When I generate a buy signal, that is when the zscore < -zthreshold, how do I make the allocation to buy Contract1 and sell Contract2.

In the same way as it was in the previous backtester. But use xarray instead of np.array.

How do I keep that allocation when the backtest runs for the next day and so on until the signal changes to sell or the algo sends an exit signal?

There is no way to do that yet. We already created a feature request for this. Share the state between iterations

I wrote an example of how to bypass this in your strategy. But I understand that it is not the easiest way... Well, we will inform you when we add this feature to the new backtester.

Regards.

spancham

Hi @support,
Fantastico! Thank you very much, you are very helpful!!
I'll try these out.
Much appreciated!

spancham

Hi @support,
I seem to be getting this error:

Wrong output dimensions.  ('time', 'asset') is not ('asset',)
100% (1289 of 1289) |####################| Elapsed Time: 0:00:00 ETA:  00:00:00

Please advise.

Also, for a different strategy can you help me with the line of code that will tell me how many assets (the count) are up today:

today = data.sel(field="close")
yesterday = data.sel(field="close").shift(time=1)
uptoday = today.isel(-1) > yesterday.isel[-1]

How do I count how many are uptoday?

Thanks for your help, greatly appreciated as always.

support

@spancham

Wrong output dimensions.

I guess you see this error when you try to run the second example using qnbt.backtest

There was a requirement to return 1d array from the strategy function, so you can modify the code this way:

weights = qnbt.backtest(
    competition_type= "futures",
    lookback_period= 2*365,
    start_date= "2006-01-01",
    strategy= lambda d: single_pass_strategy(d).isel(time=-1)
)

Recently, we updated the library: added the optimizer, and slightly improved the backtest function. Now it extracts the last day automatically. So you can just update the library, run this in your env:

 conda install 'quantiacs-source::qnt>=0.0.228'

support

@spancham said in Please advise on p settings. Thanks.:

Also, for a different strategy can you help me with the line of code that will tell me how many assets (the count) are up today:

print(len(uptoday.isel(time=-1).dropna('asset').asset))

Does it suit?

Regards.

spancham

Hi @support
Thank you, I'll try that.