SlurmDBD Is Down: What Continues Working and What Does Not

When SlurmDBD is down in a Slurm HPC cluster, the impact can look more serious than it actually is. SlurmDBD is central to persistent accounting and account-management data, but it is not the daemon that directly schedules jobs or controls compute nodes during normal cluster operation.

That distinction matters during an outage. If slurmctld was already running and had previously synchronized with SlurmDBD, the controller can usually continue operating from cached association, limit, and fair-share information while accounting messages accumulate locally for later delivery.

The situation becomes more critical when SlurmDBD remains unavailable for an extended period, administrators need database-backed tools, or slurmctld must start without usable cached information. Understanding these boundaries helps avoid unnecessary cluster-wide disruption while the accounting service is being restored.

Where SlurmDBD Fits in the Architecture

A typical Slurm deployment has several independent components. slurmctld is the central controller responsible for scheduling jobs and managing cluster state, while slurmd runs on compute nodes and launches or supervises workloads. SlurmDBD sits beside this execution path and provides the interface between Slurm and its accounting database.

When AccountingStorageType=accounting_storage/slurmdbd is configured, accounting records are sent through SlurmDBD to the underlying MySQL or MariaDB database. The same database commonly stores users, accounts, associations, QOS definitions, usage history, and information needed for fair-share and resource-limit policies.

This separation is intentional. SchedMD notes that SlurmDBD offloads database processing from the controller, helping prevent a slow or overloaded database from directly slowing normal controller operations. Therefore, a SlurmDBD outage is not automatically equivalent to a Slurm controller outage.

What Continues Working

The most important operational point is that an already-running slurmctld can normally continue scheduling during a temporary SlurmDBD outage.

Existing jobs continue running on their compute nodes. Slurm does not terminate healthy workloads simply because the accounting daemon becomes unreachable. The slurmd daemons continue managing job execution, and the controller (slurmctld) continues maintaining the active cluster state.

New jobs can also generally continue to be submitted and scheduled because the controller retains the accounting-related information it already received. SchedMD specifically documents that slurmctld caches user limits and fair-share information, allowing a short SlurmDBD outage to be tolerated.

Commands that obtain live scheduling information from slurmctld also remain useful. For example:

squeue

still queries jobs managed by the controller, while:

sinfo

continues showing partition and node information.

Administrative commands such as:

scontrol show job <jobid>
scontrol show node <node>

also depend primarily on the controller rather than the accounting database.

This is why users may continue submitting jobs, checking queues, and running workloads even while administrators are receiving SlurmDBD connection errors in the logs.

What Happens to Accounting Records

The next concern can be: if SlurmDBD cannot accept accounting updates, are job records immediately lost?

For a short outage, normally no.

When SlurmDBD becomes unavailable, slurmctld queues accounting messages instead of requiring every database operation to complete synchronously. Once communication is restored, the queued records are transferred to SlurmDBD. Slurm also documents that cached controller information is written to local storage during shutdown and recovered when possible at startup.

This behavior provides an important buffer between scheduling operations and the accounting database.

An administrator can inspect the queue with:

sdiag

Look for:

DBD Agent queue size

According to the sdiag documentation, this value grows when messages intended for SlurmDBD cannot be processed because SlurmDBD or the database is unavailable. A small temporary increase is expected during an outage. A continuously growing queue requires attention.

The MaxDBDMsgs Limit Matters

An important point must be clear here:

• The Slurm controller (slurmctld) cannot store an unlimited number of database messages.

The relevant setting is:

MaxDBDMsgs

Check it with:

scontrol show config | grep -i MaxDBDMsgs

When communication with SlurmDBD is unavailable, slurmctld queues messages, but MaxDBDMsgs limits how large that queue can become so the controller does not consume memory indefinitely. Current Slurm documentation defines a minimum of 10,000 messages and calculates the default using cluster size when that results in a higher value.

This changes the severity of a long outage.

If the queue reaches its limit, behavior depends on max_dbd_msg_action. With the default discard behavior, Slurm starts purging selected accounting messages and can eventually stop tracking new ones, creating accounting data loss. With exit, slurmctld terminates instead of discarding the messages.

For that reason, do not assume accounting information can be buffered forever.

What Stops Working or Becomes Unreliable

Database-oriented tools are where the outage becomes immediately visible.

sacct normally retrieves job and job-step accounting information from Slurm accounting storage. Historical queries may therefore fail or become unavailable while SlurmDBD cannot be reached. Even after service returns, very recent records may temporarily appear incomplete until queued updates have been processed.

The impact is even clearer with:

sacctmgr

sacctmgr uses the database interface provided by SlurmDBD to view and modify accounts, users, associations, and related accounting objects. If SlurmDBD is unavailable, normal database-backed account-management operations should not be expected to work.

The same applies to reporting tools such as:

sreport

because these reports are generated from accounting data and database rollups.

So while:

squeue

may look completely normal, commands such as the following will not work as expected:

sacct
sacctmgr
sreport

Associations, QOS, and Fair Share During the Outage

Resource policies require a little more attention. If the controller already has associations, QOS definitions, user limits, and fair-share information in its internal cache, it can continue using that information while SlurmDBD is unavailable. SchedMD explicitly documents caching of user limits and fair-share information by slurmctld.

We can inspect portions of the controller’s cached accounting information with:

scontrol show assoc_mgr

The assoc_mgr output displays the controller’s internal cache for users, associations, and QOS records.

This means existing policies do not simply disappear when SlurmDBD goes offline. However, the cache represents information already known to the controller. Database changes are a different matter. Adding a new account, creating a new association, or modifying a QOS through sacctmgr requires access to SlurmDBD.

Operationally, the distinction can be summarized as:

Existing cached policy --> Continues to be enforced

New accounting database change --> Unavailable until SlurmDBD returns

That is especially important in clusters using AccountingStorageEnforce, associations, QOS limits, or fair-share scheduling.

Example:

Be Careful with slurmctld Restarts

A running controller surviving a SlurmDBD outage is not the same situation as a controller starting for the first time.

Slurm can persist cached account, association, and limit information locally so that an existing controller installation can recover it across restarts. However, SchedMD also warns that SlurmDBD must be available when slurmctld is initially started without such cached information, because the controller has no authoritative local copy yet.

⚠️ This is an important troubleshooting detail!

If SlurmDBD is down but slurmctld is healthy and jobs are scheduling normally, restarting the controller without a specific reason can introduce additional risk.

First and foremost, stay calm, take a breath, and determine whether the actual problem is:

• the slurmdbd service;
• network connectivity between slurmctld and SlurmDBD;
• MUNGE authentication;
• MySQL or MariaDB availability;
• database storage;
• or a configuration mismatch.

⚠️ Do not turn an accounting outage into a controller outage unnecessarily!

A Practical SlurmDBD Outage Check

We can start on the SlurmDBD server. Access it from the command line and check the slurmdbd service status:

systemctl status slurmdbd

Then inspect recent log messages:

journalctl -u slurmdbd --since "-30 min"

On the controller (slurmctld), look for database communication errors:

journalctl -u slurmctld --since "-30 min"

Then check the database-message backlog:

sdiag

Pay particular attention to:

DBD Agent queue size

Next, confirm that the core scheduling path is healthy:

scontrol ping
squeue
sinfo

These commands help establish whether you have only an accounting-service failure or a broader Slurm control-plane problem.

We can then test database-backed functionality separately, using the following commands:

sacct -S today
sacctmgr show cluster

If squeue and sinfo work while sacctmgr fails, the evidence strongly points toward the accounting path rather than the scheduler itself.

If SlurmDBD is running but still cannot operate correctly, inspect the database service and connectivity as well:

systemctl status mariadb

or:

systemctl status mysqld

depending on the platform.

Also check authentication, available disk space, database logs, firewall rules, and DNS or hostname resolution where applicable.

Consider a Backup SlurmDBD

As we told you before, database outages can create a high impact for the HPC environment. Based on that, sites that require additional resilience can configure a backup SlurmDBD.

Relevant settings include:

AccountingStorageBackupHost

in slurm.conf, and:

DbdBackupHost

in slurmdbd.conf.

SchedMD states that the backup SlurmDBD must have access to the same underlying database as the primary daemon. The primary and backup instances also need compatible authentication and database access.

This can protect against the failure of the SlurmDBD host itself, although it does not remove the database backend from the overall failure domain.

What to Expect After SlurmDBD Returns

Once SlurmDBD becomes reachable again, slurmctld begins forwarding the accounting records accumulated during the outage.

Important: Do not assume that every accounting query will be complete immediately after the service starts. Depending on the amount of data in the controller cache, it may take some time to commit to SlurmDBD.

We can use the following command to check the value on DBD Agent queue size:

sdiag | grep -i "DBD Agent queue size"

The value should move back toward its normal level as the backlog is processed.

Then compare current scheduler state with accounting information:

squeue

and:

sacct -S today

Recent completed jobs should gradually appear correctly in the accounting database.

Final Words

A temporary SlurmDBD outage is therefore usually an accounting-service incident rather than an immediate scheduling emergency. An already-initialized controller can keep scheduling jobs, enforcing cached policies, and buffering accounting messages while the database path is repaired.

The important boundary is time. The longer SlurmDBD remains unavailable, the larger the queued accounting backlog becomes and the closer the controller moves toward MaxDBDMsgs. Watching that queue, preserving a healthy slurmctld, and restoring SlurmDBD before the buffer becomes exhausted is usually the safest way to handle the incident.

If you are new to the Slurm HPC world, don’t be afraid of that. We have written an article introducing this subject. Click here to access it.

Additionally, if you want a hands-on adventure, we have written an article explaining how to deploy a Slurm cluster in a lab environment. Click here to access the article.

Enjoy 🙂