Snakemake¶

The Snakemake workflow management system is a tool to create reproducible and scalable data analyses. Workflows are described via a human readable, Python based language.

Input Parameters¶

The app requires one mandatory parameter, which specifies the directory where the Snakemake command is executed. By default, Snakemake searches for a Snakefile in the following order within this directory: Snakefile, snakefile, workflow/Snakefile, and workflow/snakefile.

Additionally, users can provide an optional parameter to specify a different path for the Snakefile.

Initialization¶

For information on how to use the Initialization parameter, please refer to the Initialization: Bash Script section of the documentation.

Configure SSH Access¶

The app provides optional support for SSH access from an external client. An SSH public key must be uploaded using the corresponding panel in Resources section of the UCloud navigation menu.

Using Slurm¶

In multi-node Snakemake jobs on UCloud, Slurm is the default executor. It is configured within a dedicated profile that is automatically updated to fully utilize the resources allocated to the UCloud job.

By default, a Slurm cluster is launched when a Snakemake job starts. If the cluster is not required, users can prevent it from starting by setting the optional Start Slurm cluster parameter to false during job submission.

If a Slurm cluster is started but remains unused, it will stay idle for the duration of the job.

Note

For single-node jobs, the Slurm configuration profile is not applied. Instead, Snakemake uses the default executor unless the user specifies otherwise. Therefore, in single-node jobs, users can always set the Start Slurm cluster parameter to false.

Slurm configuration profile¶

The Slurm configuration profile is specified with the following information:

cores: Set to the total number of logical CPUs per node for the selected machine type.
jobs: Set to the number of allocated nodes.
mem_mb: Set to the total memory per node for the chosen machine type.
runtime: Effectively unlimited, set to 100000.

Additionally, the Slurm configuration profile includes two slurm_extra options:

--nodes=1-<num-nodes>: Specifies the range of nodes allocated to the UCloud job, where <num-nodes> is the total number of nodes.
--gres=gpu:<gpu-type>:<num-gpu>: Defines the GPU resource allocation. Here, <gpu-type> refers to the GPU type (e.g., h100), and <num-gpu> is the number of GPUs per node. This option is only included when the UCloud job runs on a GPU-enabled machine type.

Adding `sbatch` options¶

Users can customize the Slurm configuration profile by specifying additional native Slurm options using the Slurm extra arguments optional parameter.

Options provided via this parameter are appended to the predefined slurm_extra values in the Slurm configuration profile (see above).

To add multiple Slurm options, users must provide them as a single string, with each option separated by a single whitespace:

--option1=value1 --option2=value2 --option3=value3

If duplicate slurm_extra options are specified, Slurm uses the value from the last occurrence of the option. This allows users to override default slurm_extra values (such as --nodes and --gres) by specifying their own values using the Slurm extra arguments parameter.

Multiple Environments¶

Users can install the required software dependencies in separate environments using conda or mamba by specifying the path(s) to the corresponding configuration YAML file(s) directly in the Snakefile. To enable this, users must use the --sdm conda option.

For more details on managing environments in Snakemake, refer to the official Snakemake documentation.

Interactive Mode¶

The Interactive mode parameter allows users to start an interactive job session. This enables them to open a terminal window from the job progress page and execute shell commands.