Why Is My Slurm Job Pending? How to Decode Every Common Reason

A Slurm job pending in the queue does not necessarily mean something is wrong with the cluster or with your batch script. Pending simply means that Slurm accepted the job, but the scheduler has not yet found the conditions required to start it. The important question is why the job is waiting.

Fortunately, Slurm normally provides that answer directly through its job reason codes. A job may be waiting for CPUs, memory, GPUs, another job, a reservation, a Quality of Service limit, or simply because other jobs currently have higher scheduling priority.

Understanding these reason codes turns a vague “my job is stuck” situation into a much more structured troubleshooting process. Instead of resubmitting jobs or immediately contacting the cluster administrator, you can usually identify the restriction with a few Slurm commands.

If you are still getting familiar with Slurm job submission and the differences between sbatch, srun, and salloc, see “Slurm Job Submission: Practical Guide to srun, sbatch, and salloc” before continuing.

Start with squeue

The first command to run is:

squeue -u $USER

A typical result might look like this:

JOBID PARTITION     NAME     USER ST TIME NODES NODELIST(REASON)
18432 compute   simulation  user PD 0:00     4 (Resources)
18435 compute   analysis    user PD 0:00     1 (Priority)

The ST column shows the current job state. PD means PENDING, while the final column displays the reason associated with that state.

Slurm defines a pending job as a queued job waiting for execution, and pending jobs will typically have a reason code explaining why they have not started. Only one reason is normally displayed even when several conditions could prevent the job from running.

For a cleaner troubleshooting view, try:

squeue -u $USER -o "%.18i %.9P %.20j %.8T %.10M %.30R"

This makes the state and reason easier to identify when the cluster contains many jobs.

Each format specifier has a specific meaning:

• %.18i — Job ID, right-justified in a field with a minimum width of 18 characters.
• %.9P — Partition name, right-justified with a minimum width of 9 characters.
• %.20j — Job name, right-justified with a minimum width of 20 characters.
• %.8T — Extended job state, such as PENDING, RUNNING, or COMPLETED.
• %.10M — displays the amount of time the job has been running (Elapsed job time).
• %.30R — Pending reason for a pending job, or the allocated node list for a running job.

The numbers define the width of each column, while the leading dot tells squeue to right-justify the value within that space. Increasing these values can be useful when job names, partition names, or pending reasons are being truncated.

For example, the output may look like this:

             JOBID PARTITION             NAME    STATE       TIME                         NODELIST(REASON)
             18432   compute       simulation  PENDING       0:00                         Resources
             18435   compute          analysis  PENDING       0:00                          Priority
             18440       gpu        training    RUNNING      12:43                           gpu-node02

This customized view is especially useful during troubleshooting because it exposes the most important information in a single command: where the job was submitted, its current state, how long it has been running, and most importantly, why Slurm is keeping it pending.

Resources: The Requested Hardware Is Not Available

One of the most common messages is:

(Resources)

This means Slurm cannot currently allocate the resources requested by the job. The official Slurm quick-start documentation identifies Resources as one of the typical reasons jobs remain pending.

Suppose the job requests:

#SBATCH --nodes=4
#SBATCH --ntasks-per-node=32
#SBATCH --mem=120G
#SBATCH --time=08:00:00

Slurm needs four nodes satisfying all those requirements at the same time. There may be idle CPUs somewhere in the cluster, but that does not mean four suitable nodes with 120 GB of available memory each are immediately available.

Check the cluster with:

sinfo

For more detail:

sinfo -N -l

You can also inspect the job itself:

scontrol show job <jobid>

Look especially at fields describing requested CPUs, nodes, memory, features, GRES, partition, and constraints.

Important: A common mistake is assuming that Resources means the cluster is completely full. It may instead mean the particular combination requested by your job is unavailable.

Priority: Other Jobs Are Ahead of Yours

Another very common reason is:

(Priority)

This does not indicate an error in the job.

It means one or more higher-priority jobs currently take precedence in the relevant partition or reservation. Slurm explicitly defines the Priority reason as the presence of higher-priority jobs ahead of the pending job.

You can inspect job priorities with:

sprio -j <jobid>

sprio -n -j <jobid>

Or examine a specific job:

scontrol show job <jobid>

On clusters using the multifactor priority plugin, sprio can break the job priority into components such as age, fair-share, job size, partition, QOS, association, and other configured factors.

Important: Do not automatically cancel and resubmit a job showing Priority. Resubmission may reset factors associated with waiting time and can make the situation worse rather than better.

Dependency: Your Job Is Waiting for Another Job

Workflows frequently use job dependencies.

For example:

jid=$(sbatch preprocess.sh | awk '{print $4}')
sbatch --dependency=afterok:$jid analysis.sh

The second job cannot start until the first one completes successfully.

While it waits, you may see:

(Dependency)

Check the complete job definition:

scontrol show job <jobid>

Look for:

Dependency=afterok:18421

Then inspect the parent job – in this case, for instance, the parent job ID is 18421:

sacct -j 18421

A more problematic condition is:

DependencyNeverSatisfied

When a dependency can no longer be satisfied, Slurm can leave the job pending with DependencyNeverSatisfied. Depending on the system-wide kill_invalid_depend configuration or the job’s --kill-on-invalid-dep option, Slurm may instead cancel the job automatically.

This often occurs when a required parent job fails, is cancelled, or does not reach the state required by the dependency expression.

Can Slurm Estimate When the Job Will Start?

squeue --start -j <jobid>

When the cluster uses Slurm’s backfill scheduler, squeue --start can display an estimated start time for eligible pending jobs.

This estimate is based on the scheduler’s current view of running jobs, requested time limits, resource availability, reservations, and scheduling priorities.

Important: The value is an estimate, not a guarantee. It can change as jobs finish earlier or later than expected, new jobs enter the queue, priorities change, or resources become unavailable.

QOS Limits

Quality of Service policies are another frequent source of pending jobs.

Depending on the environment, you may encounter reasons such as:

QOSGrpCpuLimit
QOSGrpJobsLimit
QOSMaxCpuPerJobLimit
QOSMaxMemoryPerJob
QOSMaxJobsPerUserLimit
QOSGrpMemLimit

QOS rules can affect scheduling priority, preemption, and resource limits.

For example, your cluster might allow a user to consume a maximum of 256 CPUs simultaneously. If your existing jobs already use all 256 CPUs, a newly submitted job can remain pending even when unused nodes exist elsewhere in the partition.

Users with permission can inspect associations and QOS information with commands such as:

sacctmgr show qos

and:

sacctmgr show assoc user=$USER

On managed HPC systems, access to some accounting information may be restricted. In that case, the reason code itself provides useful information to give the administrator.

Association Limits

Slurm accounting associations can impose limits independently of QOS.

You may encounter messages such as:

AssocGrpCpuLimit
AssocGrpJobsLimit
AssocMaxJobsLimit
AssocGrpMemLimit

An association normally connects a user, account, and possibly cluster or partition with accounting policies.

Imagine a research project account is limited to 500 CPUs. Several members of the same project could collectively reach that limit even though your personal CPU usage is low.

This distinction matters because reducing resources in your own running jobs may not solve the issue if another user under the same account is consuming the shared allocation.

Check your association when permitted:

sacctmgr show assoc where user=$USER

Partition Restrictions

Sometimes the problem is the partition selected in the job script.

For example:

#SBATCH --partition=gpu

Inspect available partitions with:

sinfo

Then inspect the partition configuration:

scontrol show partition gpu

A partition may restrict maximum execution time, allowed nodes, accounts, QOS values, job size, or other resources.

A job requesting 72 hours from a partition configured with a 48-hour maximum can remain pending with a reason such as:

PartitionTimeLimit

Other partition-related reasons worth recognizing include PartitionDown, PartitionInactive, and PartitionNodeLimit.

Tip: Always compare the job request with the partition where it was submitted.

Reservations

Clusters commonly use reservations for maintenance, training sessions, special projects, or dedicated computing periods.

A pending job may show a reservation-related reason when nodes that would otherwise satisfy its requirements are reserved.

Check reservations with:

scontrol show reservation

Slurm reservations can cover resources including nodes, cores, licenses, and other resources for selected users, accounts, partitions, or QOS configurations.

Note: This explains a confusing scenario where sinfo appears to show idle nodes but your job still cannot use them. The nodes may be idle, but they are not necessarily available to your job.

ReqNodeNotAvail: Requested Nodes Are Unavailable

Another important message is:

ReqNodeNotAvail

This commonly appears when nodes required by the job are unavailable.

Start with:

sinfo -R

Then inspect individual nodes:

scontrol show node <nodename>

Nodes can be drained, down, reserved, undergoing maintenance, or unavailable for another administrative reason. sinfo can report the reason associated with unavailable nodes.

If the required node is in a DRAINED state, see “Slurm Node Is DRAINED: How to Find the Exact Reason“ for a complete node-level troubleshooting workflow.

Pay particular attention to jobs using explicit node requirements:

#SBATCH --nodelist=node05,node06

or feature constraints such as:

#SBATCH --constraint=avx512

💡 The more restrictive the job specification becomes, the smaller the set of nodes Slurm can select.

Waiting for GPUs or Other GRES Resources

GPU clusters introduce another scheduling dimension through Generic Resources, commonly called GRES.

In Slurm, GRES (Generic Resources) is the mechanism used to manage resources that are not represented only by standard CPU or memory allocations, such as GPUs and other specialized devices. It allows administrators to define these resources on compute nodes and lets users request them explicitly in their jobs, for example with --gres=gpu:2. This helps Slurm track which devices are available, allocate them correctly, and prevent multiple jobs from using the same restricted hardware resource at the same time.

A job might request:

#SBATCH --gres=gpu:4

Even if a node has free CPUs and memory, the job cannot start unless Slurm can also satisfy the GPU request.

Inspect the node-level GRES configuration and state:

sinfo -N -o "%N %G %t"

The -N option displays information for each individual node, while -o defines a custom output format. In this custom format, %Nshows the node name, %G displays the configured GRES resources, such as GPUs, and %t shows the node’s current state, such as idle, mix, alloc, or down.

For example:

NODELIST   GRES          STATE
gpu01      gpu:a100:4    idle
gpu02      gpu:a100:4    mix
gpu03      gpu:v100:2    alloc

This is useful when troubleshooting GPU jobs because it quickly shows which nodes have GPUs configured and whether those nodes are currently available, partially allocated, fully allocated, or unavailable.

Then examine candidate nodes:

scontrol show node <nodename>

Problems become particularly easy to encounter when requesting specific GPU types:

#SBATCH --gres=gpu:a100:4

Note: A four-GPU job may wait much longer than four separate one-GPU jobs because all required GPUs must become allocatable under the scheduling requirements.

JobHeldUser and JobHeldAdmin

A pending job can also be intentionally held.

Check it with:

scontrol show job <jobid>

JobHeldUser normally indicates that the job was held by the user or an account coordinator:

scontrol release <jobid>

Administrative holds are different. If the reason indicates an administrator placed the job on hold, investigate the message and contact the cluster administrator when necessary.

Holds are useful because the job remains in Slurm instead of being deleted, allowing its configuration to be examined before execution resumes.

BeginTime: The Job Was Scheduled for Later

Not every pending job is supposed to start immediately.

A job can be submitted with:

sbatch --begin=now+2hours job.sh

Until that start condition is reached, Slurm can report a time-related pending reason.

Check:

scontrol show job <jobid>

and inspect the job’s timing fields.

This is particularly worth checking when jobs are generated by workflow systems or automation rather than submitted manually.

Other Pending Reasons Worth Recognizing

Reason	What It Usually Means
InvalidAccount	The account requested by the job is not valid for the user or job context.
InvalidQOS	The requested QOS is invalid or not available to the job.
Licenses	The job is waiting for a configured license resource.
PartitionDown	The requested partition is administratively down.
PartitionInactive	The requested partition is not currently accepting jobs for execution.
PartitionTimeLimit	The job requests more wall time than the partition allows.
JobArrayTaskLimit	The job array has reached its configured number of simultaneously running tasks.
WaitingForScheduling	The scheduler has not yet assigned a more specific reason.

If the pending reason is InvalidAccount or Slurm rejects a submission with an account/partition error, see “Slurm srun Invalid Account or Account/Partition Combination Specified“ for a focused troubleshooting workflow.

A Practical Troubleshooting Sequence

When a Slurm job remains pending, avoid guessing. Use the scheduler information systematically.

Start with:

squeue -j <jobid>

Then collect the complete job definition:

scontrol show job <jobid>

Check cluster capacity and node states:

sinfo
sinfo -R

If priority is involved:

sprio -j <jobid>

After that:

squeue --start -j <jobid>

If previous jobs or dependencies matter:

sacct -j <jobid>

For administrator-level investigations, scheduler diagnostics can also be useful:

sdiag

Turning a Pending Reason into a Troubleshooting Direction

A Slurm pending reason should be treated as the starting point of the investigation rather than as a complete diagnosis.

Resources tells you to compare the job request with currently allocatable hardware. Priority points toward scheduling order and priority factors. QOS and association reasons point toward accounting policy limits, while ReqNodeNotAvail directs the investigation toward specific compute nodes.

Also remember that Slurm normally displays only one pending reason at a time. As the scheduler reevaluates the job and conditions change, the reason may change as well.

In most cases, combining squeue, scontrol show job, sinfo, sprio, and sacct is enough to determine whether the correct action is to wait for capacity, adjust the job request, fix an unavailable node, or involve the cluster administrator.

External References