Essential Slurm Administration Commands Every HPC Administrator Should Know

Slurm administration commands are part of my daily routine when managing HPC clusters. Whether I need to investigate scheduling issues, monitor running jobs, inspect compute nodes, or respond to unexpected incidents, having the right commands readily available makes cluster administration faster and more reliable.

Although Slurm provides hundreds of commands and options, only a core set is required for most administrative tasks. In my experience, mastering these commands significantly reduces troubleshooting time and helps maintain cluster stability while supporting users more efficiently.

This guide presents the essential Slurm administration commands I regularly use to manage HPC environments. The examples assume a standard Slurm installation and demonstrate practical commands for monitoring jobs, inspecting nodes, maintaining partitions, and resolving common operational issues.

Slurm Administration Commands: Verifying the Slurm Controller

Before troubleshooting users or jobs, verify that the Slurm controller is operational.

The most common command is:

scontrol ping

A healthy controller returns something similar to:

Slurmctld(primary) at controller is UP

If the controller is unreachable, users will typically experience job submission failures, and many other Slurm commands will become unavailable. This should always be one of the first checks during incident response.

Checking Cluster Status

The sinfo command provides an overview of the cluster.

Display all partitions:

sinfo

Example output:

PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
compute*    up    infinite   32 idle node[01-32]
gpu         up    infinite    4 alloc gpu[01-04]

For additional details:

sinfo -N

This displays node-level information instead of partition summaries.

To quickly identify unavailable nodes and the reasons recorded for them:

sinfo -R

This is particularly useful during incident response because it shows why Slurm considers a node unavailable. For the complete node information, follow it with:

scontrol show node <node-name>

This command highlights nodes that are down, drained, or unavailable, along with the reason recorded by the administrator.

Inspecting Nodes

Detailed information about a specific compute node can be obtained with:

scontrol show node node05

Useful fields include:

• CPU allocation;
• Memory usage;
• Node state;
• Boot time;
• Active features;
• Running jobs.

Administrators frequently use this command while diagnosing resource allocation problems or hardware failures.

To display every node:

scontrol show nodes

Although verbose, this output contains nearly every configuration parameter known by Slurm.

Monitoring Running Jobs

The primary command for job monitoring is:

squeue

Typical output:

JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
1024 compute simulation alice R 02:35 2 node[05-06]
1025 compute analysis bob PD 00:00 1 (Priority)

To display jobs for a specific user:

squeue -u alice

To inspect jobs on a specific node:

squeue -w node05

Pending jobs often include a scheduling reason, helping administrators determine whether a job is waiting because of insufficient resources, priority, reservations, or partition limits.

Understand Job Priority with sprio

When the cluster uses Slurm’s multifactor priority plugin, inspect a pending job with:

sprio -j 1025

For normalized factors:

sprio -n -j 1025

sprio breaks the job priority into components such as age, fair-share, job size, partition, QOS, and other configured factors. It is therefore much more useful than looking only at the final numeric priority.

If the main question is why a job remains queued, see “Why Is My Slurm Job Pending? How to Decode Every Common Reason“.

Viewing Detailed Job Information

While squeue provides a summary, administrators often need complete job details.

Use:

scontrol show job 1024

The output includes:

• Requested resources;
• Allocated CPUs;
• Memory limits;
• Execution node;
• Time limits;
• Job state;
• Exit information.

This command is especially useful when investigating scheduling behavior or unexpected resource consumption.

Examining Completed Jobs

Once a job finishes, squeue no longer displays it.

Instead, use accounting data:

sacct -j 1024

Example:

JobID State ExitCode Elapsed
1024 COMPLETED 0:0 01:12:45

To display resource utilization:

sacct -j 1024 --format=JobID,User,State,Elapsed,MaxRSS,AllocCPUS

Note: Resource-utilization fields such as MaxRSS depend on job accounting collection being configured. They are calculated from job-step accounting data, so some fields may be blank when the required JobAcctGather data is not available.

Historical accounting is essential for capacity planning, user support, and performance analysis.

Monitor Resource Usage of a Running Job with sstat

While sacct is commonly used for accounting and completed jobs, sstat provides resource-usage information for currently running job steps:

sstat -j 1024 \
  --format=JobID,AveCPU,MaxRSS,AveRSS

This can help investigate memory consumption and CPU usage while a workload is still executing.

Note: sstat requires job accounting collection to be enabled, and the available metrics depend on the configured jobacct_gather plugin.

For a practical introduction to creating and monitoring Slurm jobs, see “Slurm Job Submission: Practical Guide to srun, sbatch, and salloc“.

Canceling Jobs

Administrators frequently terminate jobs that are malfunctioning or violating cluster policies.

Cancel a single job:

scancel 1024

Cancel every job owned by a user:

scancel -u alice

Cancel every job on a partition:

scancel -p debug

⚠️ Because scancel immediately affects running workloads, administrators should verify the target jobs before issuing large-scale cancellations.

Managing Node States

Hardware maintenance and troubleshooting often require changing node states.

Drain a node – the node will stop accepting new jobs while allowing existing workloads to finish:

scontrol update NodeName=node05 State=DRAIN Reason="Memory errors"

After the root cause has been corrected, use RESUME to request that Slurm return the node toward normal operation:

scontrol update NodeName=node05 State=RESUME

Note: RESUME is an administrative action. Slurm requires the compute node to register successfully with the controller before it can return to normal scheduling. Always verify the result:

sinfo -N -n node05

scontrol show node node05

If a node unexpectedly becomes unavailable:

scontrol update NodeName=node05 State=DOWN Reason="Hardware failure"

Providing a meaningful reason helps both administrators and users understand why resources are unavailable.

Important: State=DOWN is more disruptive than DRAIN and should not be used as an interchangeable maintenance state. Setting a node to DOWN can affect running workloads associated with that node. Use it when the node genuinely needs to be taken out of service.

If a node enters DRAINED state unexpectedly, do not repeatedly run RESUME without identifying the underlying reason. See “Slurm Node Is DRAINED: How to Find the Exact Reason“ for the complete troubleshooting workflow.

Viewing Partition Configuration

Inspect partition settings with:

scontrol show partition

To examine a specific partition:

scontrol show partition compute

Administrators can verify:

• Maximum runtime;
• Default runtime;
• Allowed nodes;
• Scheduling policy;
• Default partition;
• Access restrictions.

This information is especially valuable when users report unexpected scheduling behavior.

Reconfiguring Slurm

Many Slurm configuration changes can be applied with:

Execute:

scontrol reconfigure

This causes the Slurm daemons to reread their configuration. However, not every configuration file or parameter can necessarily be applied non-disruptively through a reconfigure operation. Always check the documentation for the specific setting being changed.

💡 Whenever possible, validate configuration changes before applying them in production, particularly in large clusters where configuration errors can affect every node.

Reviewing Reservations

Reservations are commonly used for maintenance windows or dedicated projects.

Display active reservations:

scontrol show reservations

Example output may include:

• Reservation name;
• Start and end time;
• Reserved nodes;
• Authorized users.

Reservations can explain why jobs remain pending even when compute nodes appear idle.

Checking Licenses

Clusters that schedule licensed software can display license information with:

scontrol show licenses

This allows administrators to verify available license counts and determine whether jobs are waiting for software licenses rather than compute resources.

Reviewing Daemon Logs

The Slurm log location depends on the cluster configuration.

Check the active values first:

scontrol show config | grep -E 'SlurmctldLogFile|SlurmdLogFile'

A cluster may use paths such as:

/var/log/slurm/slurmctld.log
/var/log/slurm/slurmd.log

When the daemons are managed by systemd, journalctl is also useful:

Controller:

journalctl -u slurmctld

Compute node:

journalctl -u slurmd

Recent errors:

journalctl -u slurmd --since "30 minutes ago"

Log analysis often provides immediate insight into node registration failures, authentication problems, configuration errors, or scheduler issues.

Managing Users and Accounts with sacctmgr

In clusters that use Slurm accounting (slurmdbd), the sacctmgr command is the primary administrative tool for managing users, accounts, Quality of Service (QoS), and resource associations. Most production HPC environments rely on it to enforce accounting policies and usage limits.

To list all configured accounts:

sacctmgr show accounts

Display registered users:

sacctmgr show users

To display user associations, including accounts and partitions:

sacctmgr show associations

Add a new account:

sacctmgr add account research

Add a user and associate them with an account:

sacctmgr add user alice \
  cluster=hpc-lab \
  account=research

Verify the new association:

sacctmgr show user alice withassoc

Administrators can also modify existing associations. For example, changing a user’s default account:

sacctmgr modify user \
  where name=alice \
  cluster=hpc-lab \
  set defaultaccount=research

If a user leaves the organization, the account association can be removed:

sacctmgr delete user \
  where name=alice \
  cluster=hpc-lab \
  account=research

This removes the specified Slurm accounting association. It does not delete the Linux user account from the operating system.

Because sacctmgr modifies the accounting database directly, changes take effect immediately and generally do not require restarting Slurm services. Before removing users or accounts, it is recommended to verify existing associations to avoid unintentionally affecting active projects or historical accounting records.

Inspecting Configuration

To display the controller’s active configuration:

scontrol show config

Administrators frequently use this command to verify:

• Cluster name;
• Scheduler type;
• Authentication method;
• Accounting configuration;
• Default plugins;
• Resource limits.

Since the output reflects the running configuration, it is useful for confirming whether recent configuration changes have been successfully applied.

Useful Administrative Workflow

A common troubleshooting sequence might look like this:

scontrol ping
sinfo
squeue
scontrol show node node05
scontrol show job 1024
sacct -j 1024
tail -f /var/log/slurm/slurmctld.log

This progression moves from validating controller availability to inspecting cluster resources, examining affected jobs, and finally reviewing controller logs. Following a consistent workflow helps reduce troubleshooting time and ensures that common failure points are not overlooked.

Building a Practical Slurm Administration Toolkit

Effective Slurm administration does not require memorizing every available command. A relatively small set of tools provides visibility into most day-to-day cluster operations.

sinfo shows the state of cluster resources, squeue exposes active and pending jobs, scontrol provides detailed scheduler information and administrative actions, while sacct and sstat help analyze resource usage. Commands such as sprio, scancel, and sacctmgr then provide focused control over priority, workloads, and accounting associations.

The important part is to use these commands as an investigation workflow rather than making changes immediately. Check the controller, identify the affected resource or job, inspect its detailed state and reason, and only then perform the appropriate administrative action.

External References