External Libraries

antinomy

I've been using cvxpy in the server environment which I installed by running

!conda install -y -c conda-forge cvxpy

in init.ipynb. But whenever this environment is newly initialized, the module is gone and I have to run this cell again (which takes awfully long).

Is this normal or is there something wrong with my environment?
My current workaround is placing these lines before the import

try:
    import cvxpy as cp
except ImportError:
    import subprocess

    cmd = 'conda install -y -c conda-forge cvxpy'.split()
    rn = subprocess.run(cmd)

    import cvxpy as cp

Is there a better way?

Best regards.

support

@antinomy Sorry for delay, maybe we do not completely understand. When is exactly gone the module? Each strategy has its own environment. So the behavior looks ok, the module should be intalled each time a new strategy is defined. But maybe you mean a different problem, correct?

antinomy

@support
It's actually the same strategy / environment, not a new one.
If I haven't used it for a while (say, a few hours or a day) and open it again by clicking on the Jupyter button, it says:
Initialization of the virtual environment. The notebook will be ready in 15 seconds.
And when I try to run the strategy that worked fine a few hours or a day ago, I get the ModuleNotFoundError and have to install the module again.
Everything else is still there as it was before - the strategy, custom files - just not cvxpy.

support

@antinomy Hi, we checked, this is the default behaviour. The solution you used is also the recommended one...

The reason is that after a timeout, the strategy container is removed and it has to be recreated.

support

@antinomy Conda may take a lot of time to install just one module. Try to use pip install cvxpy instead.

antinomy

@support
Yes, pip is way faster. Thanks!
I might have found an even faster solution but I guess I have to wait a few hours to find out if it really works.

Here's what I did:

I created a folder in /root/books called "modules" to install cvxpy there to make it persistent:

!mkdir modules && pip install --target=modules cvxpy

Then if the import fails in the strategy, it creates symbolic links in /usr/local/lib/python3.7/site-packages/ that point to the content of /root/books/modules/

try:
    import cvxpy as cp
except ImportError:
    import os
    source = '/root/book/modules/'
    target = '/usr/local/lib/python3.7/site-packages/'
    for dirpath, dirnames, filenames in os.walk(source):
        source_path = dirpath.replace(source, '')
        target_path = os.path.join(target, source_path)
        if not os.path.exists(target_path) and not os.path.islink(target_path):
            os.symlink(dirpath, target_path)
            continue
        for file in filenames:
            source_file = os.path.join(dirpath, file)
            target_file = os.path.join(target, source_path, file)
            if not os.path.exists(target_file) and not os.path.islink(target_file):
                os.symlink(source_file, target_file)
    import cvxpy as cp

Creating the symlinks only takes 0.07 seconds, so fingers crossed

UPDATE (a few hours later):
It actually worked. When I just reopened the strategy, the environment was newly initialized. First I tried just importing cvxpy and got the ModuleNotFoundError. Then I ran the strategy including the code above: cvxpy was imported correctly and the strategy ran.

I'm not sure if that solution works for every module because I don't know if pip might also write something to other directories than site-packages.

Anyway, I'm happy with this solution.
Regards

Sjackson3289

This post is deleted!

giant.wallaby

This post is deleted!