I am relatively new in iPOP-up (and in python in general) and I have some basic questions concerning jupyterhub in relation to the server.
I have noticed that sometimes I (pip) install some python libraries in jupyterhub and set up a script that works fine within the jupyter notebook. Then, when I run the same commands in a script on the terminal I need to install these libraries again on the server (using conda and pip) for the script to run, because otherwise it errors because it can not find the libraries.
The same is true when I do it the other way around (I install the libraries on the server and then move to jupyterhub and they have to be installed again).
I would like to know the relationship between the terminal and the jupyterhub.
Is there a way to avoid installing the same libraries twice? can I just install everything via conda and then “let jupyterhub know” where everything is so that I don’t have to install everything again? or do I have to do it twice every time?
I have a followup question.
One of the tools I am using requires a gpu.
I have chosen 1 gpu when logging in jupyterhub, but when I run the command in the new kernel (the one that is connected with the new conda env) it does not “see” the gpu.
However, it does “see” the gpu when I am on the default kernel (Python3 ipykernel).
What should I do?
It seems in those examples that your are trying to use pytorch.
Pythorch libraries exist in gpu enable or cpu enable versions. They are called CPU or CUDA version.
You will need to select the right conda package or pip repos when installing (The release that you need and the GPU enabled one).
Depending on the torch release the installation could differ.
Usually you will also need to install (In the conda env) a gxx and gcc compatible version with the cuda version use by your software in order to build it.
If you can give us more information on what software, you are trying to use; we could probably help you find a recipe for youre jupyter kernel.
I used conda install jax[cuda11_pip]==0.4.23 -c conda-forge but now the scvi library can not be imported (see the error bellow). From what I understand that is because of the mismatch between the version of jax and the version of jaxlib
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
Cell In[4], line 1
----> 1 import scvi
File /shared/projects/compdevneuro/envs/conda_jupyter/lib/python3.12/site-packages/scvi/__init__.py:11
8 from ._settings import settings
10 # this import needs to come after prior imports to prevent circular import
---> 11 from . import data, model, external, utils
13 from importlib.metadata import version
15 package_name = "scvi-tools"
File /shared/projects/compdevneuro/envs/conda_jupyter/lib/python3.12/site-packages/scvi/data/__init__.py:3
1 from anndata import read_csv, read_h5ad, read_loom, read_text
----> 3 from ._anntorchdataset import AnnTorchDataset
4 from ._datasets import (
5 annotation_simulation,
6 brainlarge_dataset,
(...)
22 synthetic_iid,
23 )
24 from ._manager import AnnDataManager, AnnDataManagerValidationCheck
File /shared/projects/compdevneuro/envs/conda_jupyter/lib/python3.12/site-packages/scvi/data/_anntorchdataset.py:27
25 if TYPE_CHECKING:
26 from ._manager import AnnDataManager
---> 27 from ._utils import registry_key_to_default_dtype, scipy_to_torch_sparse
29 logger = logging.getLogger(__name__)
32 class AnnTorchDataset(Dataset):
File /shared/projects/compdevneuro/envs/conda_jupyter/lib/python3.12/site-packages/scvi/data/_utils.py:7
4 from uuid import uuid4
6 import h5py
----> 7 import jax
8 import jax.numpy as jnp
9 import numpy as np
File /shared/projects/compdevneuro/envs/conda_jupyter/lib/python3.12/site-packages/jax/__init__.py:39
34 del _cloud_tpu_init
36 # Confusingly there are two things named "config": the module and the class.
37 # We want the exported object to be the class, so we first import the module
38 # to make sure a later import doesn't overwrite the class.
---> 39 from jax import config as _config_module
40 del _config_module
42 # Force early import, allowing use of `jax.core` after importing `jax`.
File /shared/projects/compdevneuro/envs/conda_jupyter/lib/python3.12/site-packages/jax/config.py:15
1 # Copyright 2018 The JAX Authors.
2 #
3 # Licensed under the Apache License, Version 2.0 (the "License");
(...)
12 # See the License for the specific language governing permissions and
13 # limitations under the License.
---> 15 from jax._src.config import config as _deprecated_config # noqa: F401
17 # Deprecations
19 _deprecations = {
20 # Added October 27, 2023
21 "config": (
22 "Accessing jax.config via the jax.config submodule is deprecated.",
23 _deprecated_config),
24 }
File /shared/projects/compdevneuro/envs/conda_jupyter/lib/python3.12/site-packages/jax/_src/config.py:28
25 from typing import Any, Callable, Generic, NamedTuple, NoReturn, TypeVar
26 import warnings
---> 28 from jax._src import lib
29 from jax._src.lib import jax_jit
30 from jax._src.lib import transfer_guard_lib
File /shared/projects/compdevneuro/envs/conda_jupyter/lib/python3.12/site-packages/jax/_src/lib/__init__.py:75
70 return _jaxlib_version
73 version_str = jaxlib.version.__version__
74 version = check_jaxlib_version(
---> 75 jax_version=jax.version.__version__,
76 jaxlib_version=jaxlib.version.__version__,
77 minimum_jaxlib_version=jax.version._minimum_jaxlib_version)
79 # Before importing any C compiled modules from jaxlib, first import the CPU
80 # feature guard module to verify that jaxlib was compiled in a way that only
81 # uses instructions that are present on this machine.
82 import jaxlib.cpu_feature_guard as cpu_feature_guard
AttributeError: partially initialized module 'jax' has no attribute 'version' (most likely due to a circular import)
I am afraid that it does not work conda install jax[cuda12_pip]==0.4.28 -c conda-forge. conda install jaxlib==*cuda* -c conda-forge does not work either
Here is the error:
Collecting package metadata (current_repodata.json): \ WARNING conda.models.version:get_matcher(546): Using .* with relational operator is superfluous and deprecated and will be removed in a future version of conda. Your spec was 1.7.1.*, but conda is ignoring the .* and treating it as 1.7.1
done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Collecting package metadata (repodata.json): | WARNING conda.models.version:get_matcher(546): Using .* with relational operator is superfluous and deprecated and will be removed in a future version of conda. Your spec was 1.8.0.*, but conda is ignoring the .* and treating it as 1.8.0
WARNING conda.models.version:get_matcher(546): Using .* with relational operator is superfluous and deprecated and will be removed in a future version of conda. Your spec was 1.9.0.*, but conda is ignoring the .* and treating it as 1.9.0
WARNING conda.models.version:get_matcher(546): Using .* with relational operator is superfluous and deprecated and will be removed in a future version of conda. Your spec was 1.6.0.*, but conda is ignoring the .* and treating it as 1.6.0
done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
PackagesNotFoundError: The following packages are not available from current channels:
- jax==0.4.28
Current channels:
- https://conda.anaconda.org/conda-forge/linux-64
- https://conda.anaconda.org/conda-forge/noarch
- https://repo.anaconda.com/pkgs/main/linux-64
- https://repo.anaconda.com/pkgs/main/noarch
- https://repo.anaconda.com/pkgs/r/linux-64
- https://repo.anaconda.com/pkgs/r/noarch
To search for alternate channels that may provide the conda package you're
looking for, navigate to
https://anaconda.org
and use the search bar at the top of the page.
With this I was able to import torch and scvi from the custom kernel my_env in jupyter and detect my GPU with the python command " torch.cuda.get_device_name(0)".
I think you should try this out and see if you get any bugs. (it’s a bit long to install)
Thank you very much for your answer.
That solved my issue.
conda install pytorch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 pytorch-cuda=11.7 -c pytorch -c nvidia did not work, so I just installed those packages one package at a time.
The model is now running with a gpu in the new kernel.
