Author: Danilo Chiacchio

When SlurmDBD is down in a Slurm HPC cluster, the impact can look more serious than it actually is. SlurmDBD is central to persistent accounting and account-management data, but it is not the daemon that directly schedules jobs or controls compute nodes during normal cluster operation. That distinction matters during an outage. If slurmctld was already running and had previously synchronized with SlurmDBD, the controller can usually continue operating from cached association, limit, and fair-share information while accounting messages accumulate locally for later delivery. The situation becomes more critical when SlurmDBD remains unavailable for an extended period, administrators need database-backed…

Deploy VMware vSphere VMs with Ansible is a practical way to create consistent lab environments without manually cloning and configuring each VM through the vSphere Client. In this lab, I use a Red Hat Enterprise Linux 8 virtual machine as the source for deploying multiple servers that will later become part of an HPC environment. Ansible connects to vCenter Server, clones the prepared RHEL template, assigns each VM its hostname and static IP configuration, and powers the systems on. The most important part of this workflow is preparing the source VM correctly before cloning it. Machine identity, SSH host keys,…

Configuring the ESXi management network from the command line can be useful when the host has lost network connectivity and the vSphere Host Client or SSH is no longer available. In this lab, the ESXi host does not have a working VMkernel interface for management traffic. I therefore accessed the host through the DCUI and ESXi Shell and rebuilt the management network using esxcli. The workflow creates a vSphere Standard Switch, adds the required port groups and physical uplink, creates a Management VMkernel adapter, assigns its IP configuration and default gateway, and validates connectivity with vmkping. Important: Changing the Management…

A Slurm job pending in the queue does not necessarily mean something is wrong with the cluster or with your batch script. Pending simply means that Slurm accepted the job, but the scheduler has not yet found the conditions required to start it. The important question is why the job is waiting. Fortunately, Slurm normally provides that answer directly through its job reason codes. A job may be waiting for CPUs, memory, GPUs, another job, a reservation, a Quality of Service limit, or simply because other jobs currently have higher scheduling priority. Understanding these reason codes turns a vague “my job is…

The Slurm Invalid account or account/partition combination specified error usually means that the user submitting the job does not have a valid Slurm accounting association for the requested cluster, account, and, when applicable, partition. I encountered this problem while testing job submission in my Slurm lab. The Linux user existed normally on the login and compute nodes, and the shared home directory was available, but srun still rejected the job. The problem was not the Linux account itself. The user was missing from the Slurm accounting associations managed through slurmdbd. In this guide, I will show how to identify the…

Linux ss lsof fuser commands are essential tools for investigating network connections, open files, busy ports, and processes using system resources. When a Linux server refuses connections, reports that a port is already in use, or prevents a filesystem from being unmounted, these commands can quickly reveal what is happening. Although their functions sometimes overlap, each command approaches troubleshooting differently. The ss command examines network sockets, lsof connects open files and network endpoints to running processes, and fuser identifies processes using a specific file, directory, mount point, or network port. Knowing how to combine these Linux troubleshooting commands can save…