What is journald on RHEL 8?

journald is a component of the systemd suite used in Red Hat Enterprise Linux (RHEL) and other modern Linux distributions. It serves as the system logging service, replacing older logging systems, such as syslog.

journald is the systemd journal daemon responsible for collecting and managing log messages from:

• The kernel
• System services
• Applications
• User processes

It stores these logs in a binary format rather than plain text, which allows for more structured and efficient querying.

We can check the daemon status by executing the following command:

systemctl status systemd-journald

Key Features

• Each log entry stores fields like timestamp, service name, process ID, user ID, and message priority for precise filtering and correlation.
• Flexible storage modes:
  • persistent: retains logs across reboots when /var/log/journal/ exists and is configured for it.
  • volatile: keeps logs in memory only (/run/log/journal/), cleared on reboot (default behavior)
  • auto: chooses persistent if possible, otherwise volatile.
• Access control: Full journal access requires root or membership in the systemd-journal group, protecting sensitive system messages.
• Configuration lives in /etc/systemd/journald.conf and controls storage, rotation, compression, and forwarding.

Interacting with the Journal

We can use the journalctl command for querying, filtering, and live-viewing logs. For example:

1- View all available logs:

journalctl

2- Follow logs live (like “tail -f”):

journalctl -f

3- Filter logs by service/daemon – in this case, filter by “httpd.service”:

journalctl -u httpd.service

4- Filter by time range:

journalctl --since "2025-07-01" --until "2025-07-02 08:00"

5- Show kernel messages only:

journalctl -k

6- Logs since last boot:

journalctl -b

7- Logs for previous boot:

journalctl -b -1

8- Filter logs by priority (e.g., errors):

journalctl -p err

9- Show the disk usage of the journal:

journalctl --disk-usage

10- Export all logs to a text file:

journalctl > /root/all-logs.txt

Configure journald for Persistent Storage and Limit the Disk Usage

As we mentioned earlier, by default, the journald does not store logs on disk. All the journald logs are stored in memory and are lost if the system reboots – all logs are kept under /run/log/journal/, it’s a temporary area that is lost with reboot.

So, to keep the journald logs persistently over reboots, follow the procedure below:

1- Create the persistent journal directory:

mkdir -p /var/log/journal

2- Edit the configuration file /etc/systemd/journald.conf.
Open the file in your editor:

vi /etc/systemd/journald.conf

Ensure these lines are present (uncomment if necessary):

Storage=persistent
SystemMaxUse=50M

3- Reload and restart journald:

systemctl daemon-reload
systemctl restart systemd-journald

4- Confirm the settings:

grep -E 'Storage|SystemMaxUse' /etc/systemd/journald.conf

journalctl --disk-usage

After restarting the daemon, we can confirm that the maximum value on disk is set to 50MB:

Generating Sample Error Messages and Filtering by Priority

We can filter logs by daemon using the option “-u <daemon_name>” and filter by priority using the option “-p <priority>”. Let’s provide an example of a situation where we need to filter errors from a specific daemon/service:

1- Create a dummy service that always fails.
Write a simple script at /usr/local/bin/dummy-error.sh:

cat << 'EOF' > /usr/local/bin/dummy-error.sh
#!/bin/bash
logger -p user.err "Dummy service error at $(date)"
exit 1
EOF

# Assigning execution permission for the script:
chmod +x /usr/local/bin/dummy-error.sh

2- Define the systemd unit in /etc/systemd/system/dummy-error.service:

cat << 'EOF' > /etc/systemd/system/dummy-error.service
[Unit]
Description=Dummy Error Service

[Service]
Type=oneshot
ExecStart=/usr/local/bin/dummy-error.sh

[Install]
WantedBy=multi-user.target
EOF

3- Start the service twice to generate errors:

systemctl daemon-reload
systemctl start dummy-error.service
systemctl start dummy-error.service

4- Filter journal entries at “err” priority or higher for this dummy service/daemon:

journalctl -u dummy-error.service -p err

Vacuum Older Logs

If necessary, you can vacuum older logs (logs older than a specified number of days):

1- Check the journal disk usage before cleaning the logs:

journalctl --disk-usage

2- Remove journal entries older than 7 days:

journalctl --vacuum-time=7d

3- Check how much space the journal now uses:

journalctl --disk-usage

Forward Journal Entries to rsyslog

The daemon journald does not push its messages into rsyslog by default. The default setting /etc/systemd/journald.conf is:

ForwardToSyslog=no

This means journal entries stay in the binary journal unless you explicitly opt in.

How does Forwarding work?

• journald writes to a Unix socket (/run/systemd/journal/syslog) when forwarding is enabled.
• rsyslog must be listening on that socket (it usually is on RHEL).
• Once enabled, every journal entry is handed off to rsyslog just like any other syslog source:

Enabling forwarding:

1- Open the journald config:

vi /etc/systemd/journald.conf

2- Set the forwarding flag to “yes”:

ForwardToSyslog=yes

3- Restart services:

systemctl restart systemd-journald
systemctl restart rsyslog

4- Test with logger:

logger -t JTEST "ForwardToSyslog is now enabled"
grep JTEST /var/log/messages

Note: If you’re planning to centralize logs via rsyslog or need text-based files for compliance, enabling ForwardToSyslog=yes is the way to bridge journald and rsyslog seamlessly.