# Evaluation¶

## Single-Pass Backtesting¶

This approach allows to evaluate your strategy very fast. It is very useful when you use any kind of optimization.

Often, single-pass strategies are looking forward.
So it is necessary to test your strategy using multi-pass approach to avoid looking forward.
It is possible to adapt multi-pass strategy for multi-pass backtester.
See the files in the folder `examples`

in the jupyter environment on the platform (or in the package `qnt.examples`

).

### Write Weights¶

The necessary input for backtesting is a set of allocation **weights** (fractions of capital to be invested) for all
assets over the backtesting period. Let us suppose that we wrote the code:

```
import qnt.data as qndata
import qnt.ta as qnta
futures = qndata.futures.load_data(min_date="2006-01-01")
price_open = futures.sel(field="open")
price_open_one_day_ago = qnta.shift(price_open, periods=1)
strategy = price_open - price_open_one_day_ago
weights = strategy / abs(strategy).sum("asset")
```

Then we can submit code by importing in addition the **qnt.output** library:

```
import qnt.output as output
```

and a call to the **write** function:

```
output.write(weights)
```

**Function**

```
qnt.output.write(weights)
```

**Parameters**

Parameter | Explanation |
---|---|

weights | xarray.DataArray with allocation weights for all assets in the backtesting period. |

**Output**

None, the call is mandatory as it will write weights to file used for evaluating performance.

### Clean Weights¶

We provide you with a **clean** function which can be used for performing sanity checks on the defined weights. The
function can be called before writing:

```
weights = output.clean(weights, futures, "futures")
```

It will perform 2 operations:

if there are trading days where the user did not specify any exposure, an exposure of “0” (no allocation) will be used;

if the total sum of the absolute exposure is larger than 1, normalization to 1 will be applied (i.e. max. allowed leverage is 1).

**Function**

```
qnt.output.clean(weights, data, kind)
```

**Parameters**

Parameter | Explanation |
---|---|

weights | xarray.DataArray with allocation weights for all assets in the backtesting period. |

data | xarray.DataArray with input data. |

kind | "futures" or "crypto". |

If kind=”crypto”, the code will check in addition if other futures in addition to Bitcoin are defined. In positive case, it will remove them and leave Bitcoin futures only.

**Output**

An xarray.DataArray with the cleaned weights ready for submission.

### Check Weights¶

In addition to the clean function we provide a **check** function which will return you warnings if issues with weights
are present. The function can be called before writing:

```
output.check(weights, futures, "futures")
```

The first check is connected to the possible presence of missing values in your algorithm. With the previous call to the clean function, this problem is automatically solved.

The second check computes the In-Sample Sharpe ratio of your system. The In-Sample Sharpe ratio must be larger than 0.7 for a successful submission.

The third check controls correlation with existing templates and with all systems submitted to previous contests.

**Function**

```
qnt.output.check(weights, data, kind)
```

**Parameters**

Parameter | Explanation |
---|---|

weights | xarray.DataArray with allocation weights for all assets in the backtesting period. |

data | xarray.DataArray with input data. |

kind | "futures" or "crypto". |

If kind=”crypto”, the code will check in addition if other futures in addition to Bitcoin are defined.

**Output**

None, only warning messages will be displayed.

## Multi-Pass Backtesting¶

We provide you with a function for performing an optional backtesting which explicitely forbids looking-forward issues with a multi-pass implementation where at timestamp “t” only data until timestamp “t” are available by construction. It can be used with:

```
import xarray as xr
import qnt.ta as qnta
import qnt.backtester as qnbt
import qnt.data as qndata
def load_data(period):
return qndata.futures_load_data(
assets=["F_AD", "F_BO", "F_BP", "F_C"],
tail=period)
def strategy(data):
close = data.sel(field="close")
sma_long = qnta.sma(close, 200).isel(time=-1)
sma_short = qnta.sma(close, 40).isel(time=-1)
pos = xr.where(sma_short > sma_long, 1, -1)
return pos / abs(pos).sum("asset")
weights = qnbt.backtest(
competition_type="futures",
load_data=load_data,
lookback_period=365,
test_period=365 * 15,
strategy=strategy)
```

Note that multi-pass backtesting is considerably slower than single-pass one. We suggest you to start with a simple single-pass implementation and to cross-check your results with the multi-pass implementation before submission, to get a realistic estimate of the performance of your algorithm.

**Function**

```
qnt.backtester.backtest(competition_type, load_data, lookback_period, test_period, strategy)
```

**Parameters**

Parameter | Explanation |
---|---|

competition_type | "futures" or "crypto". |

load_data | data load function which returns data time series, accepts tail argument. |

lookback_period | calendar days, max. lookback period for building indicators. |

start_date | start date for backtesting, overrides test period |

end_date | end date for backtesting, by default - now |

test_period | calendar days, period used for the simulation. |

strategy | strategy function, accepts data and returns weights for all assets for the last day. |

**Output**

It returns the xarray.DataArray of **weights** and performs automatically calls to **clean**, **check** and **write**.

## Stateful Multi-pass Backtesting¶

If you want to pass the state between iterations using the multi-pass backtesting, you can do it this way:

```
import xarray as xr
import numpy as np
import qnt.backtester as qnbt
def strategy(data, state):
close = data.sel(field="close")
last_close = close.ffill('time').isel(time=-1)
# state may be null, so define a default value
if state is None:
state = {
"ma": last_close,
"ma_prev": last_close,
"output": xr.zeros_like(last_close)
}
ma_prev = state['ma']
ma_prev_prev = state['ma_prev']
output_prev = state['output']
# align the arrays to prevent problems in case the asset list changes
ma_prev, ma_prev_prev, last_close = xr.align(ma_prev, ma_prev_prev, last_close, join='right')
ma = ma_prev.where(np.isfinite(ma_prev), last_close) * 0.97 + last_close * 0.03
buy_signal = np.logical_and(ma > ma_prev, ma_prev > ma_prev_prev)
stop_signal = ma < ma_prev_prev
output = xr.where(buy_signal, 1, output_prev)
output = xr.where(stop_signal, 0, output)
next_state = {
"ma": ma,
"ma_prev": ma_prev,
"output": output
}
return output, next_state
weights, state = qnbt.backtest(
competition_type="futures", # Futures contest
lookback_period=365, # lookback in calendar days
start_date="2006-01-01",
strategy=strategy,
analyze=True,
build_plots=True,
collect_all_states=False # if it is False, then the function returns the last state, otherwise - all states
)
```

As you see, the strategy function accepts 2 arguments and return 2 values. The backtester takes the 2nd return value and pass it to the next iteration as 2nd argument.

For the first iteration, the `state`

will be None, so you have to define a default value.

The evaluator will treat the state as persistent between iterations on our server.
You can observe this state in the **log table**.

The evaluation of a stateful strategy may be slower than the evaluation of a stateless strategy (which does not use states), because the evaluator cannot parallelize the running of iterations.

### Stateful Multi-pass Backtest ML¶

LSTM Neural Network for Predicting Stock Price Movements using State

This repository provides an example of using the **backtest_ml** function with **state**, calculating complex indicators, dynamically selecting stocks for
trading, and implementing basic risk management measures, such as normalizing and reducing large positions. It also
includes recommendations for submitting strategies to the competition.

## Statistics¶

### Calculating Statistics¶

For estimating the profitability of our algorithm we measure the Sharpe ratio, the most important and popular metric. We use the annualized Sharpe ratio and assume that there are ≈252 trading days on average per year. The annualized Sharpe ratio must be larger than 0.7 at submission time for the In-Sample test. The In-Sample period depends on the competition kind:

Futures: since January 1, 2006 to submission time;

Bitcoin Futures: since January 1, 2014 to submission time.

The **calc_stat** function allows to calculate the complete statistics of an algorithm including the Sharpe ratio.

**Function**

```
qnt.stats.calc_stat(data, portfolio_history, slippage_factor=None, roll_slippage_factor=None,
min_periods=1, max_periods=None)
```

**Parameters**

Parameter | Explanation |
---|---|

data | xarray.DataArray with historical asset data. |

portfolio_history | xarray.DataArray filled with allocation weights. |

slippage_factor | fraction of ATR14 used for punishing trades. Default for futures/BTC futures is 0.04. |

roll_slippage_factor | fraction of ATR14 used for futures rollovers. Default for futures/BTC futures is 0.02. |

min_periods | minimal number of days used for statistics. |

max_periods | maximal number of days used for statistics. Default None: complete time series. |

**Output**

The output is an xarray.DataArray with statistical indicators computed on a cumulative (max_periods=None) or rolling ( for example, max_periods=252) basis.

Output Field | Explanation |
---|---|

equity | Cumulative profit and losses. We start with 1 M USD. |

relative_return | Relative daily variation of equity. |

volatility | Annualized standard deviation of relative returns. |

underwater | Time evolution of drawdowns. |

max_drawdown | Absolute minimum of underwater. |

sharpe_ratio | Annualized Sharpe ratio: ratio of mean return / volatility. |

mean_return | Annualized mean return. |

bias | Daily asymmetry between long and short exposure: 1 for a long-only system, -1 for a short-only one. |

instruments | Number of assets which get allocations on a given day. |

avg_turnover | Average turnover. |

avg_holding_time | Average holding time. |

Note that some indicators are defined on a daily basis:

equity;

relative_return;

underwater;

max_drawdown;

bias;

instruments.

Other indicators imply an average over time:

volatility;

sharpe_ratio;

mean_return;

avg_turnover;

avg_holding_time.

With default arguments:

```
qnt.stats.calc_stat(data, portfolio_history)
```

all indicators will be displayed since inception (cumulative basis).

To get results on a rolling basis, one has to specify max_periods, for example 252 trading days:

```
qnt.stats.calc_stat(data, portfolio_history, max_periods=252)
```

To get In-Sample results one can use slicing, for example:

```
qnt.stats.calc_stat(data, portfolio_history.sel(time=slice("2006-01-01", None)))
```

**Example**

Let us assume that you are writing a simple strategy:

```
import xarray as xr
import qnt.ta as qnta
import qnt.data as qndata
import qnt.output as qnout
import qnt.stats as qnstats
data = qndata.futures_load_data(
assets=["F_AD", "F_BO", "F_BP", "F_C"],
min_date="2000-01-01")
close = data.sel(field="close")
sma_long = qnta.sma(close, 200)
sma_short = qnta.sma(close, 40)
weights = xr.where(sma_short > sma_long, 1, -1)
weights = weights / abs(weights).sum("asset")
weights = qnout.clean(weights, data, "futures")
qnout.check(weights, data, "futures")
qnout.write(weights)
```

After the weights have been computed, you can calculate the statistics in order to evaluate the algorithm in the In-Sample perios using:

```
stat = qnstats.calc_stat(data, weights.sel(time=slice("2006-01-01", None)))
display(stat.to_pandas().tail())
```

field time |
equity | relative_return | volatility | underwater | max_drawdown | sharpe_ratio | mean_return | bias | instruments | avg_turnover | avg_holding_time |
---|---|---|---|---|---|---|---|---|---|---|---|

2021-01-13 | 0.689775 | -0.000568 | 0.100944 | -0.640707 | -0.704302 | -0.172620 | -0.017425 | 1.0 | 4.0 | 0.018621 | 111.681818 |

2021-01-14 | 0.698379 | 0.012474 | 0.100971 | -0.636225 | -0.704302 | -0.166830 | -0.016845 | 1.0 | 4.0 | 0.018621 | 111.681818 |

2021-01-15 | 0.689371 | -0.012898 | 0.101000 | -0.640917 | -0.704302 | -0.172728 | -0.017446 | 1.0 | 4.0 | 0.018622 | 111.681818 |

2021-01-19 | 0.687289 | -0.003021 | 0.100993 | -0.642002 | -0.704302 | -0.174101 | -0.017583 | 1.0 | 4.0 | 0.018620 | 111.681818 |

2021-01-20 | 0.690715 | 0.004985 | 0.100989 | -0.640217 | -0.704302 | -0.171786 | -0.017349 | 1.0 | 4.0 | 0.018618 | 111.588235 |

We can also display plots as follows:

```
import qnt.graph as qngraph
performance = stat.to_pandas()["equity"]
qngraph.make_plot_filled(performance.index, performance, name="PnL (Equity)", type="log")
```

```
UWchart = stat.to_pandas()["underwater"]
qngraph.make_plot_filled(UWchart.index, UWchart, color="darkred", name="Underwater Chart", range_max=0)
```

```
SRchart = stat.to_pandas()["sharpe_ratio"]
qngraph.make_plot_filled(SRchart.index, SRchart, color="#F442C5", name="SR")
```