Last Updated: 2024-11-21

Journalctl is a system-wide logging system (a more modern alternative to the role syslog often assumes)

Usage

• journalctl is a new tool with excellent autocomplete (with tab) support. Write journalctl and then tab to see all the possible filters. Then select one and tab again. For example, to filter by sending program sshd (which appears as an autocomplete option under _COMM=) would be journalctl _COMM=sshd

• it can also be fed a systemd unit (found via systemctl status) with journalctl -u nginx.service (i.e. with -u)

• to leave, press q

• to follow in real time journalctl -f.

• the data is available in json format - e.g. journalctl -u nginx -o json-pretty

• view all messages from all logs with plain old journalctl

Behind the scenes

It stores data in a binary format - the idea being to make it much faster to find particular messages. Unstructured text files from syslog are difficult to work with when large.

Default non-persistent

The default storage type in journald.conf is "auto". In this storage type, the journal logs may not be persistent and may not survive reboots.

Technically this is what happens:

"By default, the journal stores log data in /run/log/journal/. Since /run/ is volatile, log data is lost at reboot. To make the data persistent, it is sufficient to create /var/log/journal/ where systemd-journald will then store the data."

i.e. in "auto" mode you just have to create the needed directories for data to be persisted: mkdir -p /var/log/journal

Then restart the log using the systemctl interface to journald: systemctl restart systemd-journald.service

Comparison with syslog

• syslog is more of a convention (esp. in C) rather that something built into unix (like journalctl is)

• Be careful of duplication between syslog and systemd journal logging (if it is persistent)

• Journalctl does not seem to do forwarding to a centralized server. Therefore syslog is still the go-to for this use-case. You can configure journalctl to fwd to syslog

Data sources

It draws data from

• the socket at /run/systemd/journal/stdout for log data coming from systemd services
• /dev/kmsg - kernel log data. This is a character device. Multiple client processes can read it and all get the full data set (compare to /proc/kmsg - this is a pseudo FIFO that is read once. Therefore multiple readers each only get partial info)
• /dev/log - application data from syslog. It uses a symbolic link to /run/systemd/joural/dev-log behind the scenes

It then mixes all this data together

Resources

• https://unix.stackexchange.com/questions/205883/understand-logging-in-linux
• https://www.golinuxcloud.com/enable-persistent-logging-in-systemd-journald/
• https://unix.stackexchange.com/questions/159221/how-do-i-display-log-messages-from-previous-boots-under-centos-7