Snakemake¶

The Snakemake workflow management system provides a framework for constructing reproducible and scalable data analyses. Workflows are defined using a human-readable language based on Python.

Input Parameters¶

To execute the workflow, the Project directory containing the Snakefile must be provided. Within this directory, Snakemake searches for a Snakefile in the following order: Snakefile, snakefile, workflow/Snakefile, and workflow/snakefile.

An optional parameter may be provided to specify an alternative path to the Snakefile.

Initialization¶

Details on the use of the Initialization parameter are provided in the Initialization: Bash Script section of the documentation.

Configure SSH Access¶

Optional SSH access from an external client is supported. To enable this feature, an SSH public key must be uploaded through the corresponding panel in the Resources section of the UCloud navigation menu.

Using Slurm¶

In multi-node Snakemake jobs on UCloud, Slurm serves as an executor. The number of jobs is initially set equal to the number of nodes allocated in the cluster, but this can be adjusted using the --jobs option.

A Slurm cluster is automatically launched when a Snakemake workflow runs in multi-node mode.

Note

For single-node jobs, the Slurm configuration profile is not applied. Instead, Snakemake uses the local executor. In this case, the degree of parallelism corresponds to the number of cores allocated on the node and can be modified with the --cores option.

Additional native Slurm options may be added through the --default-resources setting. These options are passed to the sbatch command in the Slurm configuration profile (see below).

For example:

mem_mb=16000 runtime=2h threads=1 slurm_extra="--nodes=1-2 --gres=gpu:h100:2"

Load a custom profile¶

Snakemake allows the use of custom profiles to simplify executor configuration. A profile is defined in a directory containing a config.yaml file. The directory path can be supplied via the optional --profile parameter when invoking Snakemake.

For a Slurm executor, a minimal profile could be defined as follows:

# ---- Snakemake SLURM profile (executor-based) ----
# Customize the values marked <...> for your cluster.

executor: slurm          # use the built-in SLURM executor (not legacy --cluster)

# Submission / scheduling
jobs: 3                  # max number of concurrent jobs to submit
keep-going: true         # continue independent jobs after a failure
printshellcmds: true     # echo shell commands
rerun-incomplete: true

# Software environments (optional but common)
use-conda: true
# conda-prefix: /work/test/.snakemake/conda  # uncomment to pin env location

# Default resources (apply to rules that don’t specify their own)
# Adjust these to your site’s limits.
default-resources:
  - runtime=2h                 # walltime -> maps to sbatch --time
  - mem_mb=16000               # total memory MiB -> sbatch --mem
  # OR use per-CPU memory instead (uncomment and remove mem_mb if preferred):
  # - mem_mb_per_cpu=4000      # -> sbatch --mem-per-cpu
  - threads=2
  # Append any extra sbatch flags that aren’t mapped by resources:
  # e.g., QoS and mail notifications.
  - slurm_extra="--gres=gpu:h100:2"

# Per-rule resource tweaks (optional examples)
set-resources:
  # Example: give map_reads more memory and time, or move it to a big-mem queue
  map_reads:
    mem_mb: 16000
    runtime: 4h

In this example:

executor specifies Slurm as the backend.
jobs sets the maximum number of parallel jobs (here, three).
mem_mb and runtime define default resource values used by Snakemake rules.
slurm_extra adds native Slurm arguments, such as the number of nodes and GPU allocation.

Additional or overriding Slurm arguments can also be provided at runtime using Snakemake’s --default-resources option.

Multiple Environments¶

Software dependencies can be isolated in separate environments using conda or mamba. This is achieved by specifying the path to one or more configuration YAML files directly in the Snakefile.

Further information on environment management in Snakemake is available in the official documentation.

Interactive Mode¶

The Interactive mode parameter allows users to start an interactive job session. This enables them to open a terminal window from the job progress page and execute shell commands.