Author: Danilo

A Slurm node DRAINED state means the scheduler has intentionally prevented a compute node from receiving new jobs. This usually happens because Slurm, a health-check mechanism, or a cluster administrator identified a condition that makes the node unsuitable for additional workloads. Seeing drain or drained in sinfo does not immediately tell you what failed. The node may have a hardware problem, a full filesystem, an invalid Slurm configuration, a failed Prolog or Epilog, or simply have been drained manually for maintenance. Fortunately, Slurm normally stores a reason with the node state. By combining sinfo, scontrol, system logs, and a few Linux checks, an administrator can usually determine exactly why the…

A Slurm job pending in the queue does not necessarily mean something is wrong with the cluster or with your batch script. Pending simply means that Slurm accepted the job, but the scheduler has not yet found the conditions required to start it. The important question is why the job is waiting. Fortunately, Slurm normally provides that answer directly through its job reason codes. A job may be waiting for CPUs, memory, GPUs, another job, a reservation, a Quality of Service limit, or simply because other jobs currently have higher scheduling priority. Understanding these reason codes turns a vague “my job is…

Linux process resource usage analysis is one of the first steps to take when a server becomes slow, unstable, or unresponsive. Instead of immediately restarting services or adding hardware, administrators should identify which resource is under pressure and which process is responsible for it. A high load average does not always mean that the CPUs are overloaded, just as low free memory does not automatically indicate a memory problem. Linux uses available memory for cache and may report high system load when processes are waiting for disk operations rather than CPU time. In this small guide, we will investigate CPU,…

Lustre filesystem commands can seem intimidating when an administrator first starts managing an HPC cluster. A single filesystem may involve metadata servers, object storage servers, multiple targets, specialized networking, and hundreds of clients accessing data at the same time. Fortunately, daily Lustre administration does not always require advanced knowledge of its internal architecture. A relatively small set of commands can reveal whether the filesystem is mounted, how much storage remains, where files are located, and which components may be experiencing problems. This practical guide covers the basic Lustre filesystem commands administrators can use for routine checks and first-level troubleshooting. The…

Linux ss lsof fuser commands are essential tools for investigating network connections, open files, busy ports, and processes using system resources. When a Linux server refuses connections, reports that a port is already in use, or prevents a filesystem from being unmounted, these commands can quickly reveal what is happening. Although their functions sometimes overlap, each command approaches troubleshooting differently. The ss command examines network sockets, lsof connects open files and network endpoints to running processes, and fuser identifies processes using a specific file, directory, mount point, or network port. Knowing how to combine these Linux troubleshooting commands can save…

High disk partition usage Linux systems can quickly become a critical issue, whether you are managing a production server, a virtual machine, or an HPC login node. When I investigate storage problems, I start by identifying which filesystem is full before narrowing the search to the directories and files responsible for the excessive disk usage. Over the years, I have found that Linux provides all the built-in tools needed to diagnose disk space problems without installing additional software. By combining commands such as df, du, find, and lsof, it is possible to identify storage bottlenecks efficiently and reduce the time…