cardinal_pythonlib.slurm
Original code copyright (C) 2009-2022 Rudolf Cardinal (rudolf@pobox.com).
This file is part of cardinal_pythonlib.
Licensed under the Apache License, Version 2.0 (the “License”); you may not use this file except in compliance with the License. You may obtain a copy of the License at
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an “AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
Library functions to launch jobs under the Slurm workload manager.
See https://slurm.schedmd.com/.
RNC: SLURM job launcher.
You can’t use environment variables in
#SBATCH
syntax.sbatch
stops looking at the first non-comment line that is not an#SBATCH
line (and the rest goes to the job’s shell).We want to be able to use variable substitution in places, and no substitution in others (for later substitution).
The HPHI has Python 3.4.2 (as of 2018-02-17) so no Python 3.5 stuff without installing a newer Python and
venv
(but let’s do that).
To find out what you have available in terms of partitions, clusters, etc.:
$ sinfo # summarizes partitions, nodes
# NB: default partition has "*" appended
$ scontrol show node <NODENAME> # details of one node
$ sacctmgr show qos # show Quality of Service options
$ squeue -u <USERNAME> --sort=+i # show my running jobs
- cardinal_pythonlib.slurm.launch_cambridge_hphi(jobname: str, cmd: str, memory_mb: int, qos: str, email: str, duration: timedelta, cpus_per_task: int, project: str = 'hphi', tasks_per_node: int = 1, partition: str = 'wbic-cs', modules: List[str] | None = None, directory: str = '/home/docs/checkouts/readthedocs.org/user_builds/cardinalpythonlib/checkouts/latest/docs/source', encoding: str = 'ascii') None [source]
Specialization of
launch_slurm()
(q.v.) with defaults for the University of Cambridge WBIC HPHI.
- cardinal_pythonlib.slurm.launch_slurm(jobname: str, cmd: str, memory_mb: int, project: str, qos: str, email: str, duration: timedelta, tasks_per_node: int, cpus_per_task: int, partition: str = '', modules: List[str] | None = None, directory: str = '/home/docs/checkouts/readthedocs.org/user_builds/cardinalpythonlib/checkouts/latest/docs/source', encoding: str = 'ascii') None [source]
Launch a job into the SLURM environment.
- Parameters:
jobname¶ – name of the job
cmd¶ – command to be executed
memory_mb¶ – maximum memory requirement per process (Mb)
project¶ – project name
qos¶ – quality-of-service name
email¶ – user’s e-mail address
duration¶ – maximum duration per job
tasks_per_node¶ – tasks per (cluster) node
cpus_per_task¶ – CPUs per task
partition¶ – cluster partition name
modules¶ – SLURM modules to load
directory¶ – directory to change to
encoding¶ – encoding to apply to launch script as sent to
sbatch