Navigation

    Quantiacs Community

    • Register
    • Login
    • Search
    • Categories
    • News
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    1. Home
    2. antinomy
    3. Posts
    A
    • Profile
    • Following 0
    • Followers 1
    • Topics 9
    • Posts 65
    • Best 30
    • Groups 0
    • Blog

    Posts made by antinomy

    • RE: Some top S&P 500 companies are not available?

      The symbols are all in there, but if they are listed on NYSE you have to prepend NYS: not NAS: to the symbol. Also, I believe by 'BKR.B' you mean 'BRK.B'

      [sym for sym in data.asset.values if any(map(lambda x: x in sym, ['JPM', 'LLY', 'BRK.B']))]
      

      ['NYS:BRK.B', 'NYS:JPM', 'NYS:LLY']

      You can also search for symbols in qndata.stocks_load_spx_list() and get a little more infos like this:

      syms = qndata.stocks_load_spx_list()
      [sym for sym in syms if sym['symbol'] in ['JPM', 'LLY', 'BRK.B']]
      
      [{'name': 'Berkshire Hathaway Inc',
        'sector': 'Finance',
        'symbol': 'BRK.B',
        'exchange': 'NYS',
        'id': 'NYS:BRK.B',
        'cik': '1067983',
        'FIGI': 'tts-824192'},
       {'name': 'JP Morgan Chase and Co',
        'sector': 'Finance',
        'symbol': 'JPM',
        'exchange': 'NYS',
        'id': 'NYS:JPM',
        'cik': '19617',
        'FIGI': 'tts-825840'},
       {'name': 'Eli Lilly and Co',
        'sector': 'Healthcare',
        'symbol': 'LLY',
        'exchange': 'NYS',
        'id': 'NYS:LLY',
        'cik': '59478',
        'FIGI': 'tts-820450'}]
      

      The value for the key 'id' is what you will find in data.asset

      posted in Support
      A
      antinomy
    • RE: toolbox not working in colab

      I got the same error after installing qnt locally with pip.
      There is indeed a circular import in the current Github repo for the toolbox, introduced by this commit:
      https://github.com/quantiacs/toolbox/commit/78beafa93775f33606156169b3e6b8f995804151#diff-89350fe373763b439e4697f9b11cceb811b4a3f0adc7a655707a936ce5646c01R6-R10
      when some of the imports in output.py which were inside of fuctions before were moved to the top level.
      Now output imports from stats and stats imports from output.

      @support Can you please have a look?

      @alexeigor @omohyoid
      The conda version of qnt doesn't seem to be affected, so if that's an option for you install that one instead.
      Otherwise we can use the git version previous to the commit above:

      pip uninstall qnt
      pip install git+https://github.com/quantiacs/toolbox.git@a1e6351446cd936532af185fb519ef92f5b1ac6d
      
      posted in Support
      A
      antinomy
    • RE: Error for importing quantiacs module

      @steel-camel

      !pip install --force-reinstall python_utils
      

      should fix the issue.
      But I have no idea what would have caused it, the line in converters.py is totally messed up. The only thing that comes to my mind is a cat on the keyboard 😉

      posted in Support
      A
      antinomy
    • RE: Why .interpolate_na dosen't work well ?

      @cyan-gloom
      interpolate_na() only eliminates NaNs between 2 valid data points. Take a look at this example:

      import qnt.data as qndata
      import numpy as np
      
      stocks = qndata.stocks_load_ndx_data()
      sample = stocks[:, -5:, -6:] # The latest 5 dates for the last 6 assets
      
      print(sample.sel(field='close').to_pandas())
      """
      asset       NYS:NCLH  NYS:ORCL  NYS:PRGO  NYS:QGEN  NYS:RHT  NYS:TEVA
      time                                                                 
      2023-05-12     13.24     97.85     35.21     45.09      NaN      8.03
      2023-05-15     13.71     97.26     34.23     45.36      NaN      8.07
      2023-05-16     13.48     98.25     32.84     45.25      NaN      8.13
      2023-05-17     14.35     99.77     32.86     44.95      NaN      8.13
      2023-05-18     14.53    102.34     33.43     44.92      NaN      8.26
      """
      
      # Let's add some more NaN values:
      sample.values[3, (1,3), 0] = np.nan
      sample.values[3, 1:4, 1] = np.nan
      sample.values[3, :2, 2] = np.nan
      sample.values[3, 2:, 3] = np.nan
      sample.values[3, :-1, 5] = np.nan
      print(sample.sel(field='close').to_pandas())
      """
      asset       NYS:NCLH  NYS:ORCL  NYS:PRGO  NYS:QGEN  NYS:RHT  NYS:TEVA
      time                                                                 
      2023-05-12     13.24     97.85       NaN     45.09      NaN       NaN
      2023-05-15       NaN       NaN       NaN     45.36      NaN       NaN
      2023-05-16     13.48       NaN     32.84       NaN      NaN       NaN
      2023-05-17       NaN       NaN     32.86       NaN      NaN       NaN
      2023-05-18     14.53    102.34     33.43       NaN      NaN      8.26
      """
      
      # Interpolate the NaN values:
      print(sample.interpolate_na('time').sel(field='close').to_pandas())
      """
      asset       NYS:NCLH    NYS:ORCL  NYS:PRGO  NYS:QGEN  NYS:RHT  NYS:TEVA
      time                                                                   
      2023-05-12    13.240   97.850000       NaN     45.09      NaN       NaN
      2023-05-15    13.420  100.095000       NaN     45.36      NaN       NaN
      2023-05-16    13.480  100.843333     32.84       NaN      NaN       NaN
      2023-05-17    14.005  101.591667     32.86       NaN      NaN       NaN
      2023-05-18    14.530  102.340000     33.43       NaN      NaN      8.26
      """
      

      As you can see, only the NaNs in the first 2 columns are being replaced. The others remain untouched and might be dropped when you use dropna().

      Another thing you should keep in mind is that you might introduce lookahead bias with interpoloation, e. g. in a single run backtest. In my example for instance (pretend the NaNs I added were already in the data) you would know on 2023-05-15 that ORCL will rise when in reality you would first know that on 2023-05-18.

      posted in Support
      A
      antinomy
    • RE: How to fix this error

      Asuming whatever train is has a similar structure as the usual stock data, I get the same error as you with:

      import itertools
      import qnt.data as qndata
      
      stocks = qndata.stocks_load_ndx_data(tail=100)
      
      for comb in itertools.combinations(stocks.asset, 2):
          print(stocks.sel(asset=[comb]))
      

      There are 2 things to consider:

      1. comb is a tuple and you can't use tuples as value for the asset argument. You are putting brackets around it, but that gives you a list with one element wich is a tuple, hence the error about setting an array element as a sequence. Using stocks.sel(asset=list(comb)) instead resolves this issue but then you'll get an index error which leads to the second point
      2. each element in comb is a DataArray and cannot be used as an index element to select from the data. You want the string values instead, for this you can iterate over asset.values for instance.

      My example works when the loop looks like this:

      for comb in itertools.combinations(stocks.asset.values, 2):
          print(stocks.sel(asset=list(comb)))
      
      posted in Support
      A
      antinomy
    • RE: Python

      I'm a huge fan of Sentdex, he really tought me a lot about Python in his tutorials.
      Have a look at his website and his Youtube channel, for instance there's a tutorial for Python beginners.

      posted in General Discussion
      A
      antinomy
    • RE: Local Development with Notifications

      It's safe to ignore these notices but if they bother you, you can set the variables together with your API key using the defaults and the messages go away:

      import os
      
      os.environ['API_KEY'] = 'YOUR-API-KEY'
      os.environ['DATA_BASE_URL'] = 'https://data-api.quantiacs.io/'
      os.environ['CACHE_RETENTION'] = '7'
      os.environ['CACHE_DIR'] = 'data-cache'
      
      posted in Support
      A
      antinomy
    • Fundamental Data

      Hello @support
      Could you please add CIKs to the NASDAQ100 stock list?
      In order to load fundamental data from secgov we need the CIKs for the stocks but they're currently not in the list we get from qnt.data.stocks_load_ndx_list().
      Allthough it is still possible to get fundamentals using qnt.data.stocks_load_list(), it takes a little bit acrobatics like this for instance:

      import pandas as pd
      import qnt.data as qndata
      
      
      stocks = qndata.stocks_load_ndx_data()
      df_ndx = pd.DataFrame(qndata.stocks_load_ndx_list()).set_index('symbol')
      df_all = pd.DataFrame(qndata.stocks_load_list()).set_index('symbol')
      idx = sorted(set(df_ndx.index) & set(df_all.index))
      df = df_ndx.loc[idx]
      df['cik'] = df_all.cik[idx]
      symbols = list(df.reset_index().T.to_dict().values())
      fundamentals = qndata.secgov_load_indicators(symbols, stocks.time)
      
      

      It would be nice if we could get them with just 2 lines like so:

      stocks = qndata.stocks_load_ndx_data()
      fundamentals = qndata.secgov_load_indicators(qndata.stocks_load_ndx_list(), stocks.time)
      
      

      Also, the workaround doesn't work locally because qndata.stocks_load_list() seems to return the same list as qndata.stocks_load_ndx_list().

      Thanks in advance!

      posted in Support
      A
      antinomy
    • RE: Local Development Error "No module named 'qnt'"

      @eddiee Try step 4 without quotes, this should start jupyter notebook. And if that's your real API-key we see in the image, delete your last post. It's a bad idea to post it in a public forum 😉

      posted in Support
      A
      antinomy
    • RE: Q17 Neural Networks Algo Template; is there an error in train_model()?

      Yes, I noticed that too. And after fixing it the backtest takes forever...
      Another thing to consider is that it redefines the model with each training but I belive you can retrain already trainded NNs with new Data so they learn based on what they previously learned.

      posted in Strategy help
      A
      antinomy
    • RE: Weights different in testing and submission

      About the slicing error, I had that too a while ago. It took me some time to figure out that it wasn't enough to have the right pandas version in the environment. Because I had another python install with the same version in my PATH, the qntdev-python also looked there and always used the newer pandas. So I placed the -s flag everywhere the qntdev python is supposed to run (PyCharm, Jupyter, terminal) like this

      /path/to/quantiacs/python -s strategy.py
      

      Of course one could simply remove the other python install from PATH but I needed it there.

      posted in Support
      A
      antinomy
    • RE: Saving and recalling a dictionary of trained models

      @alfredaita
      In case you don't want to run init.py every time in order to install external libraries, I came up with a solution for this. You basically install the library in a folder in your home directory and let the strategy create symlinks to the module path at runtime. More details in this post.

      posted in Support
      A
      antinomy
    • Erroneous Data?

      Hello @support
      Since the live period for the Q16 contest is coming to an end I'm watching my participating algorithms more closely and noticed something odd:
      The closing prices are the same on 2022-02-24 and 2022-02-25 to the last decimal for allmost all cryptos (49 out of 54).

      import qnt.data as qndata
      
      crypto = qndata.cryptodaily.load_data(tail=10)
      c = crypto.sel(field='close').to_pandas().iloc[-3:]
      liquid = crypto.sel(field='is_liquid').fillna(0).values.astype(bool)[-3:]
      # only showing the cryptos which were liquid for the last 3 days:
      c.iloc[:, liquid.all(axis=0)]
      
      asset 	ADA 	AVAX 	BNB 	BTC 	DOGE 	DOT 	ETH 	LINK 	SOL 	XRP
      time 										
      2022-02-23 	0.8664 	73.47 	365.6 	37264.053 	0.1274 	15.97 	2580.9977 	13.34 	84.64 	0.696515
      2022-02-24 	0.8533 	76.39 	361.2 	38348.744 	0.1242 	16.16 	2598.0195 	13.27 	89.41 	0.696359
      2022-02-25 	0.8533 	76.39 	361.2 	38348.744 	0.1242 	16.16 	2598.0195 	13.27 	89.41 	0.696359
      
      (c.values[-1] == c.values[-2]).sum(), c.shape[1]
      
      (49, 54)
      

      Could you please have a look?
      Thanks!

      posted in Support
      A
      antinomy
    • RE: External Libraries

      @support
      Yes, pip is way faster. Thanks!
      I might have found an even faster solution but I guess I have to wait a few hours to find out if it really works.

      Here's what I did:

      1. I created a folder in /root/books called "modules" to install cvxpy there to make it persistent:
      !mkdir modules && pip install --target=modules cvxpy
      
      1. Then if the import fails in the strategy, it creates symbolic links in /usr/local/lib/python3.7/site-packages/ that point to the content of /root/books/modules/
      try:
          import cvxpy as cp
      except ImportError:
          import os
          source = '/root/book/modules/'
          target = '/usr/local/lib/python3.7/site-packages/'
          for dirpath, dirnames, filenames in os.walk(source):
              source_path = dirpath.replace(source, '')
              target_path = os.path.join(target, source_path)
              if not os.path.exists(target_path) and not os.path.islink(target_path):
                  os.symlink(dirpath, target_path)
                  continue
              for file in filenames:
                  source_file = os.path.join(dirpath, file)
                  target_file = os.path.join(target, source_path, file)
                  if not os.path.exists(target_file) and not os.path.islink(target_file):
                      os.symlink(source_file, target_file)
          import cvxpy as cp
      

      Creating the symlinks only takes 0.07 seconds, so fingers crossed 🙂

      UPDATE (a few hours later):
      It actually worked. When I just reopened the strategy, the environment was newly initialized. First I tried just importing cvxpy and got the ModuleNotFoundError. Then I ran the strategy including the code above: cvxpy was imported correctly and the strategy ran.

      I'm not sure if that solution works for every module because I don't know if pip might also write something to other directories than site-packages.

      Anyway, I'm happy with this solution.
      Regards

      posted in Support
      A
      antinomy
    • RE: External Libraries

      @support
      It's actually the same strategy / environment, not a new one.
      If I haven't used it for a while (say, a few hours or a day) and open it again by clicking on the Jupyter button, it says:
      Initialization of the virtual environment. The notebook will be ready in 15 seconds.
      And when I try to run the strategy that worked fine a few hours or a day ago, I get the ModuleNotFoundError and have to install the module again.
      Everything else is still there as it was before - the strategy, custom files - just not cvxpy.

      posted in Support
      A
      antinomy
    • External Libraries

      Hello @support ,

      I've been using cvxpy in the server environment which I installed by running

      !conda install -y -c conda-forge cvxpy
      

      in init.ipynb. But whenever this environment is newly initialized, the module is gone and I have to run this cell again (which takes awfully long).

      Is this normal or is there something wrong with my environment?
      My current workaround is placing these lines before the import

      try:
          import cvxpy as cp
      except ImportError:
          import subprocess
      
          cmd = 'conda install -y -c conda-forge cvxpy'.split()
          rn = subprocess.run(cmd)
      
          import cvxpy as cp
      

      Is there a better way?

      Best regards.

      posted in Support
      A
      antinomy
    • RE: Issues with the Legacy Website

      @jeppe_and Ok, thanks for the quick reply!

      posted in Support
      A
      antinomy
    • Issues with the Legacy Website

      Hello,

      I'm having 2 issues with legacy.quantiacs.com:

      1. Trouble accessing the site
        Since a few days ago Firefox won't open the legacy website, shwoing this message:

      legacy_issue.png

      1. Accessing the website with Chromium by adding an exception I found that the strategy charts haven't been updated since 2021-10-26.

      Could you please take a look?
      Thanks!

      posted in Support
      A
      antinomy
    • RE: Output the results in an excel or other format file

      @anshul96go
      To get the actual statistics you currently have to calculate them like so:

      import qnt.stats as qns
      
      data = qndata.cryptodaily_load_data(min_date="2014-01-01") # or whenever your backtest started
      stats = qns.calc_stat(data, weights)
      

      And if you really need them as xls file you can do:

      stats.to_pandas().to_excel('stats.xls') # I got a ModuleNotFoundError the first time - pip install did the trick.
      

      Allthough I can't recommend xls because at least LibreOffice becomes very slow / unresponsive when handling such a file.

      Getting the statistics after a backtest could be a little simpler, which brings me to a feature request:
      @support
      Do you think you could add a parameter to the backtester which makes it return the statistics? They get calculated anyway by default, but we only see a truncated printout or the plots and can't use them for further analysis.
      .
      In my local environment I did it like this in qnt.backtester.py:

      1. Add the parameter return_stats: bool = False to the parameters of the backtest function
      2. From line 353 onward my backtester now looks like this:
      	qnout.write(result)
              qnstate.write(state)
      
              if return_stats:
                  analyze = True
      
              out = [result]
      
              if analyze:
                  log_info("---")
                  stats = analyze_results(result, data, competition_type, build_plots, start_date)
                  if return_stats:
                      out.append(stats)
      
              if args_count > 1:
                  out.append(state)
      
              if len(out) == 1:
                  out = out[0]
      
              return out
          finally:
              qndc.set_max_datetime(None)
      
      1. And of course I made analyze_results return the statistics like so (line 458 in the original):
          if not build_plots:
              log_info(stat_global.to_pandas().tail())
              return stat_global # here
          log_info("---")
          log_info("Calc stats per asset...")
          stat_per_asset = qnstat.calc_stat(data, output, per_asset=True)
          stat_per_asset = stat_per_asset.loc[output.time.values[0]:]
      
          if is_notebook():
              build_plots_jupyter(output, stat_global, stat_per_asset)
          else:
              build_plots_dash(output, stat_global, stat_per_asset)
      
          return stat_global # and there
      

      This might not be the most elegant solution but you get the idea.
      Now I can get the statistics immediately after the backtest with

      weights, stats = backtest(...return_stats=True)
      

      and can do further analysis.
      For instance, I started to calculate the correlations between my strategies to avoid uploading more of the same to the contest.

      It would be nice to have this feature in a future version, so I don't have to mess with the backtester after each update 😉

      Best regards

      posted in Support
      A
      antinomy
    • Correlation Check always fails

      Hello,
      whenever I run a backtest on the server I get the message

      WARNING! Can't calculate correlation.

      This has been happening since I started developing for the Q16 contest.
      I don't know if this has any influence on the actual submission check and we'll soon have x times the quickstart template in the contest 😉
      Anyway, it would be good if the correlation check would work before we submitt algos that will eventually fail the correlation filter.

      posted in Support
      A
      antinomy
    • Documentation
    • About
    • Career
    • My account
    • Privacy policy
    • Terms and Conditions
    • Cookies policy
    Home
    Copyright © 2014 - 2021 Quantiacs LLC.
    Powered by NodeBB | Contributors