Managing an HPC cluster efficiently requires more than simply installing Slurm. Daily administration involves monitoring jobs, inspecting nodes, maintaining partitions, troubleshooting scheduling issues, and responding quickly to unexpected events. While Slurm provides hundreds of commands and options, a relatively small set is used regularly by administrators to keep clusters operating smoothly.
This article covers the core Slurm administration commands that every HPC administrator should know. The examples assume a standard Slurm installation and focus on practical usage rather than exhaustive command references.
Verifying the Slurm Controller
Before troubleshooting users or jobs, verify that the Slurm controller is operational.
The most common command is:
scontrol ping
A healthy controller returns something similar to:
Slurmctld(primary) at controller is UP
If the controller is unreachable, users will typically experience job submission failures, and many other Slurm commands will become unavailable. This should always be one of the first checks during incident response.
Checking Cluster Status
The sinfo command provides an overview of the cluster.
Display all partitions:
sinfo
Example output:
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
compute* up infinite 32 idle node[01-32]
gpu up infinite 4 alloc gpu[01-04]
For additional details:
sinfo -N
This displays node-level information instead of partition summaries.
To view node states in a compact format:
sinfo -R
This command highlights nodes that are down, drained, or unavailable, along with the reason recorded by the administrator.
Inspecting Nodes
Detailed information about a specific compute node can be obtained with:
scontrol show node node05
Useful fields include:
- CPU allocation;
- Memory usage;
- Node state;
- Boot time;
- Active features;
- Running jobs.
Administrators frequently use this command while diagnosing resource allocation problems or hardware failures.
To display every node:
scontrol show nodes
Although verbose, this output contains nearly every configuration parameter known by Slurm.
Monitoring Running Jobs
The primary command for job monitoring is:
squeue
Typical output:
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
1024 compute simulation alice R 02:35 2 node[05-06]
1025 compute analysis bob PD 00:00 1 (Priority)
To display jobs for a specific user:
squeue -u alice
To inspect jobs on a specific node:
squeue -w node05
Pending jobs often include a scheduling reason, helping administrators determine whether a job is waiting because of insufficient resources, priority, reservations, or partition limits.
Viewing Detailed Job Information
While squeue provides a summary, administrators often need complete job details.
Use:
scontrol show job 1024
The output includes:
- Requested resources;
- Allocated CPUs;
- Memory limits;
- Execution node;
- Time limits;
- Job state;
- Exit information.
This command is especially useful when investigating scheduling behavior or unexpected resource consumption.
Examining Completed Jobs
Once a job finishes, squeue no longer displays it.
Instead, use accounting data:
sacct -j 1024
Example:
JobID State ExitCode Elapsed
1024 COMPLETED 0:0 01:12:45
To display resource utilization:
sacct -j 1024 --format=JobID,User,State,Elapsed,MaxRSS,AllocCPUS
Historical accounting is essential for capacity planning, user support, and performance analysis.
Canceling Jobs
Administrators frequently terminate jobs that are malfunctioning or violating cluster policies.
Cancel a single job:
scancel 1024
Cancel every job owned by a user:
scancel -u alice
Cancel every job on a partition:
scancel -p debug
⚠️ Because scancel immediately affects running workloads, administrators should verify the target jobs before issuing large-scale cancellations.
Managing Node States
Hardware maintenance and troubleshooting often require changing node states.
Drain a node – the node will stop accepting new jobs while allowing existing workloads to finish:
scontrol update NodeName=node05 State=DRAIN Reason="Memory errors"
After maintenance, return the node to service:
scontrol update NodeName=node05 State=RESUME
If a node unexpectedly becomes unavailable:
scontrol update NodeName=node05 State=DOWN Reason="Hardware failure"
Providing a meaningful reason helps both administrators and users understand why resources are unavailable.
Viewing Partition Configuration
Inspect partition settings with:
scontrol show partition
To examine a specific partition:
scontrol show partition compute
Administrators can verify:
- Maximum runtime;
- Default runtime;
- Allowed nodes;
- Scheduling policy;
- Default partition;
- Access restrictions.
This information is especially valuable when users report unexpected scheduling behavior.
Reconfiguring Slurm
After modifying the configuration file slurm.conf, administrators typically reload the configuration without restarting the controller.
Execute:
scontrol reconfigure
This instructs the controller to reread its configuration files.
💡 Whenever possible, validate configuration changes before applying them in production, particularly in large clusters where configuration errors can affect every node.
Reviewing Reservations
Reservations are commonly used for maintenance windows or dedicated projects.
Display active reservations:
scontrol show reservations
Example output may include:
- Reservation name;
- Start and end time;
- Reserved nodes;
- Authorized users.
Reservations can explain why jobs remain pending even when compute nodes appear idle.
Checking Licenses
Clusters that schedule licensed software can display license information with:
scontrol show licenses
This allows administrators to verify available license counts and determine whether jobs are waiting for software licenses rather than compute resources.
Reviewing Daemon Logs
Many operational problems require examining Slurm log files.
Typical locations include (for the Slurm controller):
/var/log/slurm/slurmctld.log
and (for the Slurm compute nodes):
/var/log/slurm/slurmd.log
Useful examples:
tail -f /var/log/slurm/slurmctld.log
or
grep ERROR /var/log/slurm/slurmctld.log
Log analysis often provides immediate insight into node registration failures, authentication problems, configuration errors, or scheduler issues.
Managing Users and Accounts with sacctmgr
In clusters that use Slurm accounting (slurmdbd), the sacctmgr command is the primary administrative tool for managing users, accounts, Quality of Service (QoS), and resource associations. Most production HPC environments rely on it to enforce accounting policies and usage limits.
To list all configured accounts:
sacctmgr show accounts
Display registered users:
sacctmgr show users
To display user associations, including accounts and partitions:
sacctmgr show associations
Add a new account:
sacctmgr add account research
Add a user and associate them with an account:
sacctmgr add user alice account=research
Verify the new association:
sacctmgr show user alice withassoc
Administrators can also modify existing associations. For example, changing a user’s default account:
sacctmgr modify user where name=alice set defaultaccount=research
If a user leaves the organization, the account association can be removed:
sacctmgr delete user alice
Because sacctmgr modifies the accounting database directly, changes take effect immediately and generally do not require restarting Slurm services. Before removing users or accounts, it is recommended to verify existing associations to avoid unintentionally affecting active projects or historical accounting records.
Inspecting Configuration
To display the controller’s active configuration:
scontrol show config
Administrators frequently use this command to verify:
- Cluster name;
- Scheduler type;
- Authentication method;
- Accounting configuration;
- Default plugins;
- Resource limits.
Since the output reflects the running configuration, it is useful for confirming whether recent configuration changes have been successfully applied.
Useful Administrative Workflow
A common troubleshooting sequence might look like this:
scontrol ping
sinfo
squeue
scontrol show node node05
scontrol show job 1024
sacct -j 1024
tail -f /var/log/slurm/slurmctld.log
This progression moves from validating controller availability to inspecting cluster resources, examining affected jobs, and finally reviewing controller logs. Following a consistent workflow helps reduce troubleshooting time and ensures that common failure points are not overlooked.
To Wrap This Up
Slurm administration becomes significantly easier once administrators are comfortable with its core command-line tools. Commands such as sinfo, squeue, scontrol, sacct, and scancel provide immediate visibility into cluster health, job execution, and resource allocation, while node management and log inspection commands enable rapid diagnosis of operational issues.
Although larger HPC environments may incorporate accounting databases, monitoring platforms, and automation frameworks, these foundational commands remain the primary interface for day-to-day cluster management.
Developing familiarity with them not only speeds up routine administration but also provides the confidence needed to diagnose scheduling issues, manage maintenance activities, and keep production HPC systems running reliably.
