Persistent Python Packages¶

Installing additional Python packages into a persistent folder is useful in UCloud workflows when an application already provides a working Python environment with preinstalled packages (for example, the latest PyTorch app).

Instead of reinstalling all dependencies from scratch or creating a new environment, only the extra packages needed for a project are installed into a separate folder. This folder can then be reused across multiple job instances, while still relying on the preinstalled packages provided by the app.

Create a Persistent Folder¶

A persistent folder can be created by running the following command in a terminal:

$ pip install <packages name> --target=/path/to/target/dir

The --target flag allows to specify the desired installation location. For example:

$ pip install <packages name> --target=/work/myfolder

This installs the specified package and its dependencies under /work/myfolder.

To install packages according to the requirements in a requirements.txt file:

pip install -r requirements.txt --target=/work/myfolder

The same flag can be used with uv to achieve faster installation:

$ uv pip install <packages name> --target=/work/myfolder

Reuse a Persistent Folder¶

A persistent folder created previously can be reused across job instances. The following steps show how to make it available in both terminal sessions and Jupyter notebooks.

Mount and activate the folder¶

Mount the folder during job submission by selecting the option Select folders to use.
Add the folder to PYTHONPATH:
```
$ export PYTHONPATH=/work/myfolder:$PYTHONPATH
```
Here, myfolder is the name of the folder mounted in the job submission page.

To make this change automatic for all terminal sessions:

$ echo 'export PYTHONPATH=/work/myfolder:$PYTHONPATH' >> ~/.bashrc
$ source ~/.bashrc

Install additional packages¶

After updating PYTHONPATH, additional packages can be added to the same persistent folder at any time:

$ pip install <packages name> --target=/work/myfolder

Note

The --target <dir> flag installs packages in the specified directory. Existing packages are not replaced unless --upgrade is specified.

Use the folder in Jupyter¶

Within a notebook cell, the persistent folder path can be added at runtime as follows:

import os
os.environ['PYTHONPATH'] = '/work/myfolder:' + os.environ.get('PYTHONPATH','')

Verification can be done with:

import os
print(os.environ['PYTHONPATH'])

To ensure Python imports prioritize the persistent folder:

import sys
sys.path.insert(0,'/work/myfolder')

The index 0 ensures the path has the highest priority.

Make it permanent in Jupyter¶

To automatically include the persistent folder for all future Jupyter sessions:

Create an IPython profile (if not already present):
```
$ ipython profile create
```

Edit the file:

/home/ucloud/.ipython/profile_default/ipython_config.py

Add the following lines at the end of the file:

c.InteractiveShellApp.exec_lines = [
    'import sys; sys.path.append("/work/myfolder")'
]

Automate Setup¶

The setup process can be automated by using an initialization script that runs automatically when the app starts.

General case¶

To install packages and configure the environment automatically, use the following Bash script:

#!/bin/bash

pip install <packages name> --target=/work/myfolder
echo 'export PYTHONPATH=/work/myfolder:$PYTHONPATH' >> ~/.bashrc

If the persistent folder has already been created and mounted:

#!/bin/bash

echo 'export PYTHONPATH=/work/myfolder:$PYTHONPATH' >> ~/.bashrc

JupyterLab case¶

For Jupyter notebooks:

#!/bin/bash

pip install <packages name> --target=/work/myfolder
ipython profile create
echo "c.InteractiveShellApp.exec_lines = [" >> ~/.ipython/profile_default/ipython_config.py
echo "    'import sys'," >> ~/.ipython/profile_default/ipython_config.py
echo "    'sys.path.append(\"/work/myfolder\")'," >> ~/.ipython/profile_default/ipython_config.py
echo "]" >> ~/.ipython/profile_default/ipython_config.py

The pip install command can be omitted if the persistent folder is already present and mounted.