Notebook

Q18 Quick Start

This template shows how to make a submission to the Q18 Nasdaq-100 contest and contains some useful code snippets.

You can clone and edit this example there (tab Examples).


This template shows you the basic steps for taking part to the Q18 NASDAQ-100 Stock Long-Short contest.

In [ ]:
import xarray as xr

import qnt.ta as qnta
import qnt.data as qndata
import qnt.output as qnout
import qnt.stats as qns

data = qndata.stocks.load_ndx_data(min_date="2005-06-01")

close     = data.sel(field="close")
is_liquid = data.sel(field="is_liquid")
sma_slow  = qnta.sma(close, 200)
sma_fast  = qnta.sma(close, 20)
weights   = xr.where(sma_slow < sma_fast, 1, -1)
weights   = weights * is_liquid

weights = qnout.clean(weights, data, "stocks_nasdaq100")

# calc stats
stats = qns.calc_stat(data, weights.sel(time=slice("2006-01-01", None)))
display(stats.to_pandas().tail())

# graph
performance = stats.to_pandas()["equity"]
import qnt.graph as qngraph

qngraph.make_plot_filled(performance.index, performance, name="PnL (Equity)", type="log")
In [ ]:
weights = weights.sel(time=slice("2006-01-01",None))

qnout.check(weights, data, "stocks_nasdaq100")
qnout.write(weights) # to participate in the competition

Description

1. Data

data_example

The variable qndata.stocks.load_ndx_data(tail=period) is an xarray.DataArray structure which contains historical market data for the last (tail=period) days and whose coordinates are:

  • time: a date in format yyyy-mm-dd;
  • field: an attribute, for example the opening daily price;
  • asset: the identifying symbol for the asset, for example NAS:APPL for Apple.

More details on xarray can be found at https://xarray.pydata.org/en/stable/.

2. Strategy. Weights allocation

weights_example

For each date, the algorithm calculates the portfolio weights which should be used at the opening of the next day's trading.

Quantiacs uses an exposure-based backtester. The trading algorithm should define the fractions of capital which will be distributed to the assets (allocation weights). A positive weight means a long position (buy), a negative value means a short position (sell).

Note that algorithm decisions can use all data available at the close of the session, and will be applied at the opening of the next day's session. The chosen allocation weights are translated to positions (number of contracts to be bought/sold) immediately after the close of the session and transactions are executed at the open of the next day.

3. Performance estimation

statistic_example

After we have built the algorithm, we can evaluate its performance calculating statistics.

We can display the values of statistical indicators on a cumulative basis, assuming that we have 1M USD at the starting point.

The call will produce:

  • equity: the cumulative value of profits and losses since inception (1M USD);
  • relative_return: the relative daily variation of equity;
  • volatility: the volatility of the investment since inception (i.e. the annualized standard deviation of the daily returns);
  • underwater: the time evolution of drawdowns;
  • max_drawdown: the absolute minimum of the underwater chart;
  • sharpe_ratio: the annualized Sharpe ratio since inception; the value must be larger than 1 for taking part to contests;
  • mean_return: the annualized mean return of the investment since inception;
  • bias: the daily asymmetry between long and short exposure: 1 for a long-only system, -1 for a short-only one;
  • instruments: the number of instruments which get allocations on a given day;
  • avg_turnover: the average turnover;
  • avg_holding_time: the average holding time in days.

A detailed explanation can be found inspecting the source code for the library in your directory at /qnt/stats.py

Moreover we can produce a chart which shows the cumulative profits and losses.

Alternative way of writing strategies

A quick prototpye can be written using a single-pass implementation where all the time series are processed at once. Beware that unintentional forward looking can take place!

If strategy sharpe coefficient in the competition differs from the sharpe coefficient in jupyter, rewrite the strategy for Multi-Pass


Single-pass (quick check of the result) Multi-Pass equivalent
import xarray as xr

import qnt.ta as qnta
import qnt.data as qndata
import qnt.output as qnout
import qnt.stats as qns

data = qndata.stocks.load_ndx_data(min_date="2006-01-01")

close     = data.sel(field="close")
is_liquid = data.sel(field="is_liquid")
sma_slow  = qnta.sma(close, 200)
sma_fast  = qnta.sma(close, 20)
weights   = xr.where(sma_slow < sma_fast, 1, -1)
weights   = weights * is_liquid

weights = qnout.clean(weights, data, "stocks_nasdaq100")

qnout.check(weights, data, "stocks_nasdaq100")

qnout.write(weights)

# calc stats
stats = qns.calc_stat(data, weights.sel(time=slice("2006-01-01",None)))
stats.to_pandas().tail()
import xarray as xr

import qnt.ta as qnta
import qnt.backtester as qnbt
import qnt.data as qndata



def load_data(period):
    return qndata.stocks.load_ndx_data(tail=period)



def strategy(data):
    close     = data.sel(field="close")
    is_liquid = data.sel(field="is_liquid")
    sma_slow  = qnta.sma(close, 200).isel(time=-1)
    sma_fast  = qnta.sma(close, 20).isel(time=-1)
    weights   = xr.where(sma_slow < sma_fast, 1, -1)
    weights   = weights * is_liquid
    return weights



weights = qnbt.backtest(
    competition_type = "stocks_nasdaq100",
    load_data        = load_data,
    lookback_period  = 365*4,
    start_date       = "2006-01-01",
    strategy         = strategy,
    analyze          = True,
    build_plots      = True
)

Example of a strategy using technical analysis indicators

The example of a strategy with a sharpe ratio of 0.9615 trading 215 financial instruments

The strategy use sma, ema, adl (Advance–Decline line)

from IPython.display import display
import xarray as xr
import qnt.data as qndata
import qnt.output as qnout
import qnt.ta as qnta
import qnt.stats as qns

data = qndata.stocks.load_ndx_data(min_date="2005-01-01")


def get_strategy_1(data, params):
    buy = 1
    not_trade = 0
    close = data.sel(field="close")

    strategy_1 = xr.where(qnta.sma(close, params[1]) > qnta.sma(close, params[0]), buy, not_trade)
    strategy_2 = xr.where(qnta.ema(close, params[2]) > qnta.ema(close, params[3]), buy, not_trade)

    weights = strategy_1 * strategy_2 * data.sel(field="is_liquid")
    weights = weights / 100.0
    return weights.fillna(0)


def get_strategy_2(data, params):
    buy = 1
    not_trade = 0
    close = data.sel(field="close") * data.sel(field="is_liquid")

    adl = qnta.ad_line(close) * 1.0
    adl_dif = adl.shift(time=params[0]) - adl.shift(time=params[1])
    positive_trend = adl_dif > 0
    strategy_1 = xr.where(positive_trend, buy, not_trade)

    weights = strategy_1 * data.sel(field="is_liquid")
    return weights.fillna(0)


weights_1 = get_strategy_1(data, [25, 40, 12, 132])  # 0.6108887689714039 Sharpe Ratio
weights_2 = get_strategy_2(data, [34, 183])  # 0.6012686822757577

weights_all = 2 * weights_1 * weights_2 - weights_1
weights = qnout.clean(output=weights_all, data=data, kind="stocks_nasdaq100")  # 0.9615

qnout.check(weights, data, "stocks_nasdaq100")
qnout.write(weights)

What libraries are available?

# Import basic libraries.
import xarray as xr
import pandas as pd
import numpy as np

# Import quantnet libraries.
import qnt.data    as qndata  # load and manipulate data
import qnt.output as output   # manage output
import qnt.backtester as qnbt # backtester
import qnt.stats   as qnstats # statistical functions for analysis
import qnt.graph   as qngraph # graphical tools
import qnt.ta      as qnta    # indicators library

May I import libraries?

Yes, please refer to the file init.ipynb in your home directory. You can for example use:

! conda install -y scikit-learn

How to load data?

Daily stock data for the Q18 Nasdaq-100 contest can be loaded using:

data = qndata.stocks.load_ndx_data(tail = 17*365, dims = ("time", "field", "asset"))

Cryptocurrency daily data used for the Q16/Q17 contests can be loaded using:

data = qndata.cryptodaily.load_data(tail = 17*365, dims = ("time", "field", "asset"))

Futures data for the Q15 contest can be loaded using:

data= qndata.futures.load_data(tail = 17*365, dims = ("time", "field", "asset"))

BTC Futures data for the Q15 contest can be loaded using:

data= qndata.cryptofutures.load_data(tail = 17*365, dims = ("time", "field", "asset"))

How to view a list of all tickers?

data.asset.to_pandas().to_list()

How to see which fields are available?

data.field.to_pandas().to_list()

How to load specific tickers?

data = qndata.stocks.load_ndx_data(tail=17 * 365, assets=["NAS:AAPL", "NAS:AMZN"])

How to select specific tickers after loading all data?

def get_data_filter(data, assets):
    filler= data.sel(asset=assets)
    return filler

get_data_filter(data, ["NAS:AAPL", "NAS:AMZN"])

How to get the prices for the previous day?

qnta.shift(data.sel(field="open"), periods=1)

or:

data.sel(field="open").shift(time=1)

How to get the Sharpe ratio?

import qnt.stats as qnstats

def get_sharpe(market_data, weights):
    rr = qnstats.calc_relative_return(market_data, weights)
    sharpe = qnstats.calc_sharpe_ratio_annualized(rr).values[-1]
    return sharpe

sharpe = get_sharpe(data, weights) # weights.sel(time=slice("2006-01-01",None))

How do I get a list of the top 3 assets ranked by Sharpe ratio?

import qnt.stats as qnstats

data = qndata.stocks.load_ndx_data(tail = 17*365, dims = ("time", "field", "asset"))

def get_best_instruments(data, weights, top_size):
    # compute statistics:
    stats_per_asset = qnstats.calc_stat(data, weights, per_asset=True)
    # calculate ranks of assets by "sharpe_ratio":
    ranks = (-stats_per_asset.sel(field="sharpe_ratio")).rank("asset")
    # select top assets by rank "top_period" days ago:
    top_period = 1
    rank = ranks.isel(time=-top_period)
    top = rank.where(rank <= top_size).dropna("asset").asset

    # select top stats:
    top_stats = stats_per_asset.sel(asset=top.values)

    # print results:
    print("SR tail of the top assets:")
    display(top_stats.sel(field="sharpe_ratio").to_pandas().tail())

    print("avg SR = ", top_stats[-top_period:].sel(field="sharpe_ratio").mean("asset")[-1].item())
    display(top_stats)
    return top_stats.coords["asset"].values

get_best_instruments(data, weights, 3)

How can I check the results for only the top 3 assets ranked by Sharpe ratio?

Select the top assets and then load their data:

best_assets= get_best_instruments(data, weights, 3)

data= qndata.stocks.load_ndx_data(tail = 17*365, assets=best_assets)

How can prices be processed?

Simply import standard libraries, for example numpy:

import numpy as np

high= np.log(data.sel(field="high"))

How can you reduce slippage impace when trading?

Just apply some technique to reduce turnover:

def get_lower_slippage(weights, rolling_time=6):
    return weights.rolling({"time": rolling_time}).max()

improved_weights = get_lower_slippage(weights, rolling_time=6)

How to use technical analysis indicators?

For available indicators see the source code of the library: /qnt/ta

ATR

def get_atr(data, days=14):
    high = data.sel(field="high") * 1.0 
    low  = data.sel(field="low") * 1.0 
    close= data.sel(field="close") * 1.0

    return qnta.atr(high, low, close, days)

atr= get_atr(data, days=14)

EMA

prices= data.sel(field="high")
prices_ema= qnta.ema(prices, 15)

TRIX

prices= data.sel(field="high")
prices_trix= qnta.trix(prices, 15)

ADL and EMA

close= data.sel(field="close") * data.sel(field="is_liquid")
adl= qnta.ad_line(close) * 1.0
adl_ema= qnta.ema(adl, 18)

How can you check the quality of your strategy?

import qnt.output as qnout
qnout.check(weights, data, "stocks_nasdaq100")

or

stat= qnstats.calc_stat(data, weights)
display(stat.to_pandas().tail())

or

import qnt.graph   as qngraph
statistics= qnstats.calc_stat(data, weights)
display(statistics.to_pandas().tail())

performance= statistics.to_pandas()["equity"]
qngraph.make_plot_filled(performance.index, performance, name="PnL (Equity)", type="log")

display(statistics[-1:].sel(field = ["sharpe_ratio"]).transpose().to_pandas())
qnstats.print_correlation(weights, data)

An example using pandas

One can work with pandas DataFrames at intermediate steps and at the end convert them to xarray data structures:

def get_price_pct_change(prices):
    prices_pandas = prices.to_pandas()
    assets = data.coords["asset"].values
    for asset in assets:
        prices_pandas[asset] = prices_pandas[asset].pct_change()
    return prices_pandas

prices = data.sel(field="close") * 1.0
prices_pct_change = get_price_pct_change(prices).unstack().to_xarray()

Disable widget scrolling

%%javascript
window.IPython && (IPython.OutputArea.prototype._should_scroll = function(lines) { return false; })
// disable widget scrolling

How can I combine datasets?

Let us suppose that we want to use some Futures data as external indicators for taking positions on Nasdaq-100 stocks. This can easily be achieved as follows:

import xarray as xr
import numpy as np

import qnt.backtester as qnbt
import qnt.data as qndata
import qnt.ta as qnta



def load_data(period):
    futures = qndata.futures.load_data(tail=period, assets=["F_DX"]).isel(asset=0)
    stocks  = qndata.stocks.load_ndx_data(tail=period)
    return {"futures": futures, "stocks": stocks}, futures.time.values



def window(data, max_date: np.datetime64, lookback_period: int):
    min_date = max_date - np.timedelta64(lookback_period, "D")
    return {
        "futures": data["futures"].sel(time=slice(min_date, max_date)),
        "stocks":  data["stocks"].sel(time=slice(min_date, max_date)),
    }



def strategy(data):
    close_futures = data["futures"].sel(field="close")
    close_stocks  = data["stocks"].sel(field="close")
    sma20 = qnta.sma(close_futures, 20).isel(time=-1)
    sma20_stocks = qnta.sma(close_stocks, 20).isel(time=-1)
    is_liquid = data["stocks"].sel(field="is_liquid").isel(time=-1)
    weights = xr.where(sma20 < sma20_stocks, 1, -1)
    weights = weights * is_liquid 
    weights = weights / 100.0
    return weights



qnbt.backtest(
    competition_type= "stocks_nasdaq100",
    load_data= load_data,
    lookback_period= 365,
    start_date= "2006-01-01",
    strategy= strategy,
    window= window
)

How to submit a strategy to the competition?

Check that weights are fine:

import qnt.output as qnout
qnout.check(weights, data, "stocks_nasdaq100")

If everything is ok, write the weights to file:

qnout.write(weights)

In your personal account:

  • choose a strategy;
  • click on the Submit button;
  • select the type of competition.

At the beginning you will find the strategy under the Checking area:

  • Sent strategies > Checking.

If technical checks are successful, the strategy will go under the Candidates area:

  • Sent strategies > Candidates.

Otherwise it will be Filtered:

  • Sent strategies > Filtered

and you should inspect error and warning messages.

Note that a strategy under the Candidates area should have a Sharpe ratio larger than 1 for being eligible for a prize. Please check warning messages in your Candidates area!

Please note that:

  • Your trading algorithm can open short and long positions.

  • At each point in time your algorithm can trade all or a subset of the stocks which at that point of time are or were part of the NASDAQ-100 stock index. Note that the composition of this set changes in time, and Quantiacs provides you with an appropriate filter function for selecting them.

  • The Sharpe ratio of your system since January 1st, 2006, has to be larger than 1.

  • Your system cannot be a copy of the current examples. We run a correlation filter on the submissions and detect duplicates.

More details on the rules can be found here.

How to find good parameters for my algorithm?

See examples

  • Trading System Optimization
  • Trading System Optimization by Asset

Read more on our article.

The main reasons for submission rejection

Detailed explanation with examples.

Missed call to write_output

Save algorithm weights, run code

qnt.output.write(weights)

Not eligible send to contest. In-Sample Sharpe must be larger than 1

Improve your algorithm. Аor example, you can use sections and get an algorithm that will pass the filter

  • Example of a strategy using technical analysis indicators
  • How do I get a list of the top 3 assets ranked by Sharpe ratio?

Need help? Check the Documentation and find solutions/report problems in the Forum section.

Not enough bid information.

Run code

min_time = weights.time[abs(weights).fillna(0).sum('asset')> 0].min()
min_time

min_time must be less than or equal to January 1, 2006.

If min_time is larger than the starting date, we recommend to fill the starting values of the time series with non-vanishing values, for example a simple buy-and-hold strategy.

def get_enough_bid_for(data_, weights_):
    time_traded = weights_.time[abs(weights_).fillna(0).sum('asset') > 0]
    is_strategy_traded = len(time_traded)
    if is_strategy_traded:
        return xr.where(weights_.time < time_traded.min(), data_.sel(field="is_liquid"), weights_)
    return weights_


weights_new = get_enough_bid_for(data, weights)
weights_new = weights_new.sel(time=slice("2006-01-01",None))