Essential Slurm Administration Commands Every HPC Administrator Should Know

Reading Time: 5 minutes

Managing an HPC cluster efficiently requires more than simply installing Slurm. Daily administration involves monitoring jobs, inspecting nodes, maintaining partitions, troubleshooting scheduling issues, and responding quickly to unexpected events. While Slurm provides hundreds of commands and options, a relatively small set is used regularly by administrators to keep clusters operating smoothly.

This article covers the core Slurm administration commands that every HPC administrator should know. The examples assume a standard Slurm installation and focus on practical usage rather than exhaustive command references.

Verifying the Slurm Controller

Before troubleshooting users or jobs, verify that the Slurm controller is operational.

The most common command is:

scontrol ping

A healthy controller returns something similar to:

Slurmctld(primary) at controller is UP

If the controller is unreachable, users will typically experience job submission failures, and many other Slurm commands will become unavailable. This should always be one of the first checks during incident response.

Checking Cluster Status

The sinfo command provides an overview of the cluster.

Display all partitions:

sinfo

Example output:

PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
compute*    up    infinite   32 idle node[01-32]
gpu         up    infinite    4 alloc gpu[01-04]

For additional details:

sinfo -N

This displays node-level information instead of partition summaries.

To view node states in a compact format:

sinfo -R

This command highlights nodes that are down, drained, or unavailable, along with the reason recorded by the administrator.

Inspecting Nodes

Detailed information about a specific compute node can be obtained with:

scontrol show node node05

Useful fields include:

• CPU allocation;
• Memory usage;
• Node state;
• Boot time;
• Active features;
• Running jobs.

Administrators frequently use this command while diagnosing resource allocation problems or hardware failures.

To display every node:

scontrol show nodes

Although verbose, this output contains nearly every configuration parameter known by Slurm.

Monitoring Running Jobs

The primary command for job monitoring is:

squeue

Typical output:

JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
1024 compute simulation alice R 02:35 2 node[05-06]
1025 compute analysis bob PD 00:00 1 (Priority)

To display jobs for a specific user:

squeue -u alice

To inspect jobs on a specific node:

squeue -w node05

Pending jobs often include a scheduling reason, helping administrators determine whether a job is waiting because of insufficient resources, priority, reservations, or partition limits.

Viewing Detailed Job Information

While squeue provides a summary, administrators often need complete job details.

Use:

scontrol show job 1024

The output includes:

• Requested resources;
• Allocated CPUs;
• Memory limits;
• Execution node;
• Time limits;
• Job state;
• Exit information.

This command is especially useful when investigating scheduling behavior or unexpected resource consumption.

Examining Completed Jobs

Once a job finishes, squeue no longer displays it.

Instead, use accounting data:

sacct -j 1024

Example:

JobID State ExitCode Elapsed
1024 COMPLETED 0:0 01:12:45

To display resource utilization:

sacct -j 1024 --format=JobID,User,State,Elapsed,MaxRSS,AllocCPUS

Historical accounting is essential for capacity planning, user support, and performance analysis.

Canceling Jobs

Administrators frequently terminate jobs that are malfunctioning or violating cluster policies.

Cancel a single job:

scancel 1024

Cancel every job owned by a user:

scancel -u alice

Cancel every job on a partition:

scancel -p debug

⚠️ Because scancel immediately affects running workloads, administrators should verify the target jobs before issuing large-scale cancellations.

Managing Node States

Hardware maintenance and troubleshooting often require changing node states.

Drain a node – the node will stop accepting new jobs while allowing existing workloads to finish:

scontrol update NodeName=node05 State=DRAIN Reason="Memory errors"

After maintenance, return the node to service:

scontrol update NodeName=node05 State=RESUME

If a node unexpectedly becomes unavailable:

scontrol update NodeName=node05 State=DOWN Reason="Hardware failure"

Providing a meaningful reason helps both administrators and users understand why resources are unavailable.

Viewing Partition Configuration

Inspect partition settings with:

scontrol show partition

To examine a specific partition:

scontrol show partition compute

Administrators can verify:

• Maximum runtime;
• Default runtime;
• Allowed nodes;
• Scheduling policy;
• Default partition;
• Access restrictions.

This information is especially valuable when users report unexpected scheduling behavior.

Reconfiguring Slurm

After modifying the configuration file slurm.conf, administrators typically reload the configuration without restarting the controller.

Execute:

scontrol reconfigure

This instructs the controller to reread its configuration files.

💡 Whenever possible, validate configuration changes before applying them in production, particularly in large clusters where configuration errors can affect every node.

Reviewing Reservations

Reservations are commonly used for maintenance windows or dedicated projects.

Display active reservations:

scontrol show reservations

Example output may include:

• Reservation name;
• Start and end time;
• Reserved nodes;
• Authorized users.

Reservations can explain why jobs remain pending even when compute nodes appear idle.

Checking Licenses

Clusters that schedule licensed software can display license information with:

scontrol show licenses

This allows administrators to verify available license counts and determine whether jobs are waiting for software licenses rather than compute resources.

Reviewing Daemon Logs

Many operational problems require examining Slurm log files.

Typical locations include (for the Slurm controller):

/var/log/slurm/slurmctld.log

and (for the Slurm compute nodes):

/var/log/slurm/slurmd.log

Useful examples:

tail -f /var/log/slurm/slurmctld.log

or

grep ERROR /var/log/slurm/slurmctld.log

Log analysis often provides immediate insight into node registration failures, authentication problems, configuration errors, or scheduler issues.

Managing Users and Accounts with sacctmgr

In clusters that use Slurm accounting (slurmdbd), the sacctmgr command is the primary administrative tool for managing users, accounts, Quality of Service (QoS), and resource associations. Most production HPC environments rely on it to enforce accounting policies and usage limits.

To list all configured accounts:

sacctmgr show accounts

Display registered users:

sacctmgr show users

To display user associations, including accounts and partitions:

sacctmgr show associations

Add a new account:

sacctmgr add account research

Add a user and associate them with an account:

sacctmgr add user alice account=research

Verify the new association:

sacctmgr show user alice withassoc

Administrators can also modify existing associations. For example, changing a user’s default account:

sacctmgr modify user where name=alice set defaultaccount=research

If a user leaves the organization, the account association can be removed:

sacctmgr delete user alice

Because sacctmgr modifies the accounting database directly, changes take effect immediately and generally do not require restarting Slurm services. Before removing users or accounts, it is recommended to verify existing associations to avoid unintentionally affecting active projects or historical accounting records.

Inspecting Configuration

To display the controller’s active configuration:

scontrol show config

Administrators frequently use this command to verify:

• Cluster name;
• Scheduler type;
• Authentication method;
• Accounting configuration;
• Default plugins;
• Resource limits.

Since the output reflects the running configuration, it is useful for confirming whether recent configuration changes have been successfully applied.

Useful Administrative Workflow

A common troubleshooting sequence might look like this:

scontrol ping
sinfo
squeue
scontrol show node node05
scontrol show job 1024
sacct -j 1024
tail -f /var/log/slurm/slurmctld.log

This progression moves from validating controller availability to inspecting cluster resources, examining affected jobs, and finally reviewing controller logs. Following a consistent workflow helps reduce troubleshooting time and ensures that common failure points are not overlooked.

To Wrap This Up

Slurm administration becomes significantly easier once administrators are comfortable with its core command-line tools. Commands such as sinfo, squeue, scontrol, sacct, and scancel provide immediate visibility into cluster health, job execution, and resource allocation, while node management and log inspection commands enable rapid diagnosis of operational issues.

Although larger HPC environments may incorporate accounting databases, monitoring platforms, and automation frameworks, these foundational commands remain the primary interface for day-to-day cluster management.

Developing familiarity with them not only speeds up routine administration but also provides the confidence needed to diagnose scheduling issues, manage maintenance activities, and keep production HPC systems running reliably.