Why Is My Slurm Job Pending? How to Decode Every Common Reason

A Slurm job pending in the queue does not necessarily mean something is wrong with the cluster or with your batch script. Pending simply means that Slurm accepted the job, but the scheduler has not yet found the conditions required to start it. The important question is why the job is waiting.

Fortunately, Slurm normally provides that answer directly through its job reason codes. A job may be waiting for CPUs, memory, GPUs, another job, a reservation, a Quality of Service limit, or simply because other jobs currently have higher scheduling priority.

Understanding these reason codes turns a vague “my job is stuck” situation into a much more structured troubleshooting process. Instead of resubmitting jobs or immediately contacting the cluster administrator, you can usually identify the restriction with a few Slurm commands.

Start with squeue

The first command to run is:

squeue -u $USER

A typical result might look like this:

JOBID PARTITION     NAME     USER ST TIME NODES NODELIST(REASON)
18432 compute   simulation  user PD 0:00     4 (Resources)
18435 compute   analysis    user PD 0:00     1 (Priority)

The ST column shows the current job state. PD means PENDING, while the final column displays the reason associated with that state.

Slurm defines a pending job as a queued job waiting for execution, and pending jobs will typically have a reason code explaining why they have not started. Only one reason is normally displayed even when several conditions could prevent the job from running.

For a cleaner troubleshooting view, try:

squeue -u $USER -o "%.18i %.9P %.20j %.8T %.10M %.30R"

This makes the state and reason easier to identify when the cluster contains many jobs.

Each format specifier has a specific meaning:

• %.18i — displays the Job ID using up to 18 characters.
• %.9P — displays the partition where the job was submitted.
• %.20j — displays the job name, limited to 20 characters.
• %.8T — displays the full job state, such as PENDING, RUNNING, or COMPLETED.
• %.10M — displays the amount of time the job has been running.
• %.30R — displays the reason why a pending job is waiting, or the allocated node list when the job is already running.

The numbers define the width of each column, while the leading dot tells squeue to right-justify the value within that space. Increasing these values can be useful when job names, partition names, or pending reasons are being truncated.

For example, the output may look like this:

             JOBID PARTITION             NAME    STATE       TIME                         NODELIST(REASON)
             18432   compute       simulation  PENDING       0:00                         Resources
             18435   compute          analysis  PENDING       0:00                          Priority
             18440       gpu        training    RUNNING      12:43                           gpu-node02

This customized view is especially useful during troubleshooting because it exposes the most important information in a single command: where the job was submitted, its current state, how long it has been running, and most importantly, why Slurm is keeping it pending.

Resources: The Requested Hardware Is Not Available

One of the most common messages is:

(Resources)

This means Slurm cannot currently allocate the resources requested by the job. The official Slurm quick-start documentation identifies Resources as one of the typical reasons jobs remain pending.

Suppose the job requests:

#SBATCH --nodes=4
#SBATCH --ntasks-per-node=32
#SBATCH --mem=120G
#SBATCH --time=08:00:00

Slurm needs four nodes satisfying all those requirements at the same time. There may be idle CPUs somewhere in the cluster, but that does not mean four suitable nodes with 120 GB of available memory each are immediately available.

Check the cluster with:

sinfo

For more detail:

sinfo -N -l

You can also inspect the job itself:

scontrol show job <jobid>

Look especially at fields describing requested CPUs, nodes, memory, features, GRES, partition, and constraints.

Important: A common mistake is assuming that Resources means the cluster is completely full. It may instead mean the particular combination requested by your job is unavailable.

Priority: Other Jobs Are Ahead of Yours

Another very common reason is:

(Priority)

This does not indicate an error in the job.

It means one or more higher-priority jobs currently take precedence in the relevant partition or reservation. Slurm explicitly defines the Priority reason as the presence of higher-priority jobs ahead of the pending job.

You can inspect job priorities with:

sprio

Or examine a specific job:

scontrol show job <jobid>

On clusters using multifactor priority scheduling, several components may influence priority, including age, fair-share usage, partition settings, job size, and QOS configuration.

Important: Do not automatically cancel and resubmit a job showing Priority. Resubmission may reset factors associated with waiting time and can make the situation worse rather than better.

Dependency: Your Job Is Waiting for Another Job

Workflows frequently use job dependencies.

For example:

jid=$(sbatch preprocess.sh | awk '{print $4}')
sbatch --dependency=afterok:$jid analysis.sh

The second job cannot start until the first one completes successfully.

While it waits, you may see:

(Dependency)

Check the complete job definition:

scontrol show job <jobid>

Look for:

Dependency=afterok:18421

Then inspect the parent job – in this case, for instance, the parent job ID is 18421:

sacct -j 18421

A more problematic condition is:

DependencyNeverSatisfied

This indicates that the dependency can no longer be satisfied under the current circumstances. Slurm can leave such a job pending depending on the cluster configuration.

This often occurs when a required parent job fails, is cancelled, or does not reach the state required by the dependency expression.

QOS Limits

Quality of Service policies are another frequent source of pending jobs.

Depending on the environment, you may encounter reasons such as:

QOSGrpCpuLimit
QOSGrpJobsLimit
QOSMaxCpuPerJobLimit
QOSMaxMemoryPerJob
QOSMaxJobsPerUserLimit
QOSGrpMemLimit

QOS rules can affect scheduling priority, preemption, and resource limits.

For example, your cluster might allow a user to consume a maximum of 256 CPUs simultaneously. If your existing jobs already use all 256 CPUs, a newly submitted job can remain pending even when unused nodes exist elsewhere in the partition.

Users with permission can inspect associations and QOS information with commands such as:

sacctmgr show qos

and:

sacctmgr show assoc user=$USER

On managed HPC systems, access to some accounting information may be restricted. In that case, the reason code itself provides useful information to give the administrator.

Association Limits

Slurm accounting associations can impose limits independently of QOS.

You may encounter messages such as:

AssocGrpCpuLimit
AssocGrpJobsLimit
AssocMaxJobsLimit
AssocGrpMemLimit

An association normally connects a user, account, and possibly cluster or partition with accounting policies.

Imagine a research project account is limited to 500 CPUs. Several members of the same project could collectively reach that limit even though your personal CPU usage is low.

This distinction matters because reducing resources in your own running jobs may not solve the issue if another user under the same account is consuming the shared allocation.

Check your association when permitted:

sacctmgr show assoc where user=$USER

Partition Restrictions

Sometimes the problem is the partition selected in the job script.

For example:

#SBATCH --partition=gpu

Inspect available partitions with:

sinfo

Then inspect the partition configuration:

scontrol show partition gpu

A partition may restrict maximum execution time, allowed nodes, accounts, QOS values, job size, or other resources.

A job requesting 72 hours, for example, cannot simply run in a partition whose configured maximum wall time is 48 hours.

Tip: Always compare the job request with the partition where it was submitted.

Reservations

Clusters commonly use reservations for maintenance, training sessions, special projects, or dedicated computing periods.

A pending job may show a reservation-related reason when nodes that would otherwise satisfy its requirements are reserved.

Check reservations with:

scontrol show reservation

Slurm reservations can cover resources including nodes, cores, licenses, and other resources for selected users, accounts, partitions, or QOS configurations.

Note: This explains a confusing scenario where sinfo appears to show idle nodes but your job still cannot use them. The nodes may be idle, but they are not necessarily available to your job.

ReqNodeNotAvail: Requested Nodes Are Unavailable

Another important message is:

ReqNodeNotAvail

This commonly appears when nodes required by the job are unavailable.

Start with:

sinfo -R

Then inspect individual nodes:

scontrol show node <nodename>

Nodes can be drained, down, reserved, undergoing maintenance, or unavailable for another administrative reason. sinfocan report the reason associated with unavailable nodes.

Pay particular attention to jobs using explicit node requirements:

#SBATCH --nodelist=node05,node06

or feature constraints such as:

#SBATCH --constraint=avx512

💡 The more restrictive the job specification becomes, the smaller the set of nodes Slurm can select.

Waiting for GPUs or Other GRES Resources

GPU clusters introduce another scheduling dimension through Generic Resources, commonly called GRES.

In Slurm, GRES (Generic Resources) is the mechanism used to manage resources that are not represented only by standard CPU or memory allocations, such as GPUs and other specialized devices. It allows administrators to define these resources on compute nodes and lets users request them explicitly in their jobs, for example with --gres=gpu:2. This helps Slurm track which devices are available, allocate them correctly, and prevent multiple jobs from using the same restricted hardware resource at the same time.

A job might request:

#SBATCH --gres=gpu:4

Even if a node has free CPUs and memory, the job cannot start unless Slurm can also satisfy the GPU request.

Inspect available nodes with:

sinfo -N -o "%N %G %t"

The command below provides a quick node-level view of the Slurm cluster:

sinfo -N -o "%N %G %t"

The -N option displays information for each individual node, while -o defines a custom output format. In this custom format, %Nshows the node name, %G displays the configured GRES resources, such as GPUs, and %t shows the node’s current state, such as idle, mix, alloc, or down.

For example:

NODELIST   GRES          STATE
gpu01      gpu:a100:4    idle
gpu02      gpu:a100:4    mix
gpu03      gpu:v100:2    alloc

This is useful when troubleshooting GPU jobs because it quickly shows which nodes have GPUs configured and whether those nodes are currently available, partially allocated, fully allocated, or unavailable.

Then examine candidate nodes:

scontrol show node <nodename>

Problems become particularly easy to encounter when requesting specific GPU types:

#SBATCH --gres=gpu:a100:4

Note: A four-GPU job may wait much longer than four separate one-GPU jobs because all required GPUs must become allocatable under the scheduling requirements.

JobHeldUser and JobHeldAdmin

A pending job can also be intentionally held.

Check it with:

scontrol show job <jobid>

If the job was held by the user, you may be able to release it with:

scontrol release <jobid>

Administrative holds are different. If the reason indicates an administrator placed the job on hold, investigate the message and contact the cluster administrator when necessary.

Holds are useful because the job remains in Slurm instead of being deleted, allowing its configuration to be examined before execution resumes.

BeginTime: The Job Was Scheduled for Later

Not every pending job is supposed to start immediately.

A job can be submitted with:

sbatch --begin=now+2hours job.sh

Until that start condition is reached, Slurm can report a time-related pending reason.

Check:

scontrol show job <jobid>

and inspect the job’s timing fields.

This is particularly worth checking when jobs are generated by workflow systems or automation rather than submitted manually.

A Practical Troubleshooting Sequence

When a Slurm job remains pending, avoid guessing. Use the scheduler information systematically.

Start with:

squeue -j <jobid>

Then collect the complete job definition:

scontrol show job <jobid>

Check cluster capacity and node states:

sinfo
sinfo -R

If priority is involved:

sprio -j <jobid>

If previous jobs or dependencies matter:

sacct -j <jobid>

For administrator-level investigations, scheduler diagnostics can also be useful:

sdiag

To Wrap This Up

Slurm itself recommends examining scontrol show job <jobid> and particularly the job state and reason when investigating jobs that are not being scheduled.

The key is to read the pending reason as the beginning of the investigation rather than the final diagnosis. Resources may lead you toward memory, GPUs, node count, or constraints; Priority points toward queue ordering; QOS and association codes lead toward policy limits.

Once you become familiar with those patterns, a pending job becomes much less mysterious. In most cases, squeue, scontrol, sinfo, sprio, and sacct provide enough information to understand what Slurm is waiting for—and whether the correct action is changing the job request, waiting for capacity, or involving the cluster administrator.

Are you new on the HPC world? No worries about that. We have written an article to introduce HPC. Click here to access it.

Maybe you would like a hands-on experience. So, there is an article explaining how to deploy a simple HPC cluster in a lab environment. Click here to access it.

Enjoy 🙂