FoxuTech

Manage Linux logs with Systemd

motoskia
Manage Linux logs with Systemd

Manage Linux logs with Systemd

Systemd is a system and service manager for Linux. It’s become the de facto system management daemon in various Linux distributions in recent years. Systemd was first introduced in Fedora. Other distributions like Arch Linux, openSUSE, or CoreOS have already made it part of their operating systems. Red Hat Enterprise Linux (RHEL) and its downstream distros like CentOS started to use systemd natively from version 7. Another major distribution, Ubuntu—which had introduced another service management daemon called Upstart—started shipping with systemd from version 15.04.

The reason for this wide-scale adoption is the versatility of systemd. It manages not only daemons and processes in a Linux system, but also various resources like devices, sockets, and mount points. When the system boots, it does not load services sequentially like System V, which saves significant time at startup. Services are loaded in parallel, and a service waits until other required resources for it are also activated.

Systemd is backward compatible with predecessors like System V init and Upstart. That means any service still using older System V init scripts for starting will work under systemd, and you can use systemd commands like systemctl to start, stop, and check the service’s status. Another advantage of systemd is its ease of configuration. Systemd is controlled by unit files that are declarative in nature and easy to understand. This contrasts with System V where the application’s developer had to create complicated shell scripts for starting, stopping, or reloading the service.

As we’ll see later, systemd has a sophisticated logging service that can be used instead of the traditional syslog service. It can also be used to complement syslog.

Units and Targets

At the heart of systemd are unit files. A unit file is a plain text file that lives under the /lib/systemd/system directory and has a type associated with it. A unit file basically describes a resource and tells systemd how to activate that resource. The naming standard for a unit file is <resource_name>.<unit_type>. The different types of units include service, path, mount point, automount, swap, target, timer, device, and socket. So, we have unit files like cron.service, tmp.mount, syslog.socket, or graphical.target. For each service unit that’s enabled, a symbolic link to the unit file is placed under the /etc/systemd/system/<target>. wants/ directory.

A target unit is a special kind of unit file because it doesn’t represent a single resource; rather, it groups other units to bring the system to a particular state.

Target units in systemd loosely resemble run levels in System V in the sense that each target unit represents a particular system state. For example, the graphical.target unit represents a system that has booted in multi-user, graphical mode, similar to System V’s runlevel 5. Multi-user.target, on the other hand, is similar to runlevel 3 (multi-user, text mode with networking enabled). However, targets are also different from runlevels because in System V, a Linux box can exist in only one runlevel at any time. In systemd, target units are inclusive. A target unit can group other target units when it’s coming up—so it’s possible for a system to remain in more than one target. Going back to the graphical.target example, when the target comes up, it also activates multi-user.target.

Systemd Journal Basics

The journal is a component of systemd. It’s a centralized location for all messages logged by different components in a systemd-enabled Linux system. This includes kernel and boot messages, messages coming from syslog, or different services.

In traditional Linux, during the boot-up phase, different subsystems of the OS, or application daemons, would log all their message in different text files throughout the system. Each subsystem would log its messages with varying level of details. When troubleshooting, an administrator would often have to go through messages from multiple files within different time frames and correlate the entries. Journaling takes away this difficulty by recording both OS and application level messages in one place.

The journal is controlled by the systemd-journald daemon. It collects information from different sources and loads the messages into the journal.

Linux server performance Monitoring Tools – FoxuTech

What exactly is systemd?

systemd is a suite of computer programs aimed to manage and interconnect different parts of system together. It is designed as a replacement for SystemV and LSB-style startup scrips, which were prevalent since 1980s. systemd consists of these components:

This is a very high-level overview of systemd‘s architecture, but will suit us for now.

Journalctl: Viewing Logs

Logs are viewed using the journalctl utility. If the command journalctl is entered without any arguments, then the console will display the following enormous list:

The systemd journal is not a large text file. It’s a binary file maintained by the daemon. So, it can’t be opened with a text editor. As we’ll see later, the location and size of this binary file is controlled by the daemon’s configuration file. It doesn’t have to be persistent either; using configuration parameters, an administrator can turn off journaling altogether or keep it in memory so it’s volatile in nature. With in-memory journaling, systemd creates its journal files under the /run/log/journal directory. The directory is created if it doesn’t exist. With persistent storage, the journal is created under /var/log/journal directory; again, the directory is created by systemd if needed. If this directory is deleted for some reason, systemd-journald will not re-create it automatically; rather, it will write the logs under /run/log/journal in a non-persistent way. It will re-create the directory when the daemon is restarted.

Here is an example of the systemd journal:

 $ ls -l /var/log/journal
 drwxr-sr-x+ 2 root systemd-journal 12288 Jan 25 17:38 93c40120d70f45089c5bf370c8013f20

 $ ls -l /var/log/journal/fd8cf26e06e411e4a9d004010897bd01/
 -rw-r-----+ 1 root systemd-journal  8388608 Jan 26 20:20 system.journal
 -rw-r-----+ 1 root systemd-journal  8388608 Jan 26 11:09 user-1000.journal

Log rotation can be configured using the –vacuum-size and –vacuum-time option. The first sets a limit on the available disk space for saving logs (in our example, it’s 1 GB):

$ sudo journalctl --vacuum-size=2G

As soon as the log size exceeds this limit, unnecessary files will be automatically deleted. The –vacuum-time option works similarly. It establishes the period of time that logs will be saved for until being automatically deleted:

$ sudo journalctl --vacuum-time=1years

Configuring Log Rotation in Configuration Files

Log rotation can also be configured by modifying the configuration file /etc/systemd/journal.conf, which includes the following parameters:

Choosing an Output Format

Using the -o option, we can convert log data to any format, making parsing and further processing easier. For example:

$ journalctl  -u docker.service -o json
{"_SELINUX_CONTEXT":"unconfined\n","SYSLOG_FACILITY":"3","_CAP_EFFECTIVE":"1ffffffffff","__CURSOR":"s=c8d55d8ea5934cafb5b03d604c8058fd;i=4a8;b=01eedb599ce54>{"__REALTIME_TIMESTAMP":"1642396763714199","_COMM":"dockerd","_EXE":"/usr/bin/dockerd","_SYSTEMD_INVOCATION_ID":"f54cd2a91eac4ac09d999b634d8b191e","_CMDLINE>{"SYSLOG_IDENTIFIER":"dockerd","_SYSTEMD_INVOCATION_ID":"f54cd2a91eac4ac09d999b634d8b191e","_MACHINE_ID":"93c40120d70f45089c5bf370c8013f20","__MONOTONIC_TIM>

Json objects can be presented in a more structured and comprehensible way by specifying the format as json-pretty or json-sse:

$ journalctl -u docker.service -o json-pretty

JSON log data can also be presented in the following formats:

Journald Configuration

The main configuration file for systemd-journald is /etc/systemd/journald.conf. However, other packages can create their configuration files which can be under any of these directories with a .conf extension:

The main configuration file is read before any of the custom *.conf files. If there are custom configs present, they override the main configuration parameters.

If any of the values need to be changed, they have to be uncommented and the systemd-journald.service restarted.

 #  This file is part of systemd.##  systemd is free software; you can redistribute
 #  it and/or modify it under the terms of the GNU Lesser General Public License as
 #  published by the Free Software Foundation; either version 2.1 of the License, or
 #  (at your option) any later version.
 #
 # Entries in this file show the compile time defaults.
 # You can change settings by editing this file.
 # Defaults can be restored by simply deleting this file.
 #
 # See journald.conf(5) for details.
  
 [Journal]
  
 #Storage=auto
  
 #Compress=yes
  
 #Seal=yes
  
 #SplitMode=uid
  
 #SyncIntervalSec=5m
  
 #RateLimitInterval=30s
  
 #RateLimitBurst=1000
  
 ...

A brief description of some of the configuration parameters are shown below. The parameters relate to:

  1. Event message persistence
  2. Disk space management
  3. Writing to files, syslog servers, or other destinations

For a full list of parameters, see the official journald.conf documentation. Here are six parameters you might find useful:

ParameterPurpose and Possible Values
StorageThere are four possible values for it: ●      “none”: This effectively turns off journaling. Any log message received will be dropped. However, any redirection to console, syslog, or kernel log buffer would still be in effect. ●      “volatile”: Journal data is saved in memory and temporarily available under the /run/log/journal directory. The directory will be created if it does not exist. ●      “persistent”: Journal data is saved persistently on disk under the /var/log/journal directory. The directory will be created if it does not exist. If the disk volume is not accessible or writable, the files will be created under /run/log/journal. ●      “auto”: The storage mode is like persistent—data will be written to disk; however, if the /var/log/journal directory does not exist, it will be created under /run/log/journal.
CompressIf this parameter is enabled, data stored in the journal that is larger than a threshold will be compressed before being written to disk. The option is turned on by default and has a default threshold of 512 bytes. You can specify the threshold here, or set it to “true” to use the default.
SystemKeepFreeThis is one of several parameters that control how large the journal can grow up to. This parameter applies if systemd is saving journals under the /var/log/journal directory. It specifies how much disk space the systemd-journald daemon will leave for other applications in the file system where the journal is hosted. The default is 15%.
RuntimeKeepFreeThis is the same as SystemKeepFree, except this applies when the journaling storage option is set to “volatile”, meaning journal files are created under /run/log/journal.
ForwardToSyslogThis parameter specifies if log messages that are received by the systemd-journald daemon will also be forwarded to a syslog daemon. The default is yes, but if no process is reading off from the socket, nothing happens.
MaxLevelStoreThis parameter sets the maximum level of logs that are stored to disk, forwarded to syslog, the kernel log buffer, the console, or wall (if they are enabled). This parameter can take any of the following values:
●      0 or “emerg”
●      1 or “alert”
●      2 or “crit”
●      3 or “err”
●      4 or “warning”
●      5 or “notice”
●      6 or “info”
●      7 or “debug”
All messages equal or below the level specified will be stored on disk. The default value is “debug” which includes all log messages from “emerg” to “debug”.

Managing Journal Size

By default, systemd-journald ensures older journal records or journal files are deleted in order to keep a certain amount of disk space free. In the Linux Logging with Systemd section, we explain how to control the thresholds for deleting old log data with configuration parameters. We can also use journalctl to manage the size of the journal, which we’ll explain in more detail in this section.

To check how much disk space is currently taken up by the journal, use the –disk-usage parameter:

$ journalctl --disk-usage

Depending on the version of journalctl, the output can be similar to this:

Archived and active journals take up 896.2M in the file system.

You can also manage disk space taken up by systemd journal by fine-tuning these configuration parameters:

Deleting or Vacuuming Journals

To delete archived journal entries manually, you can use either the –vacuum-size or the –vacuum-time option. In the example below, we are deleting any archived journal files, so the journal size comes back to 200MB.

$ journalctl --vacuum-size=100M

The output should be something like this.

Vacuuming done, freed 784.1M of archived journals from /var/log/journal/93c40120d70f45089c5bf370c8013f20.

When using –vacuum-time option, archived journals older than the time specified will be deleted.

These two options are available from systemd version 218.

You can also use the –vacuum-files option, which deletes all but the specified number of journal files. For example, if you have 10 archived journal files and want to reduce these down to 2, you can do so by running the following command:

$ journalctl --vacuum-files=2

Note that this won’t delete any journal files that are actively in use.

Verifying Journals

To verify the journal for internal consistency, use the –verify option:

$ journalctl --verify

There will be a graphical progress bar as the check is done. Sometimes an alert message will be shown:

PASS: /var/log/journal/93c40120d70f45089c5bf370c8013f20/system.journal
 PASS: /var/log/journal/93c40120d70f45089c5bf370c8013f20/system.journal
 PASS: /var/log/journal/93c40120d70f45089c5bf370c8013f20/system@0005d654ad5bfe30-8e7b7a8ba6e8e91c.journal~
 PASS: /var/log/journal/93c40120d70f45089c5bf370c8013f20/user-1000@854681710d7e4bd7b72cd9d187080d7a-000000000000060d-0005d5c16bae7360.journal
 PASS: /var/log/journal/93c40120d70f45089c5bf370c8013f20/user-1000@0005d6512d29da99-86669a54cb3d4ec6.journal~
 PASS: /var/log/journal/93c40120d70f45089c5bf370c8013f20/system@0005d614d228ab15-57a6d2811f923302.journal~
 930f98: Invalid hash (3b28a2cca7997e7 vs. b4198a44bca6ad4c
 930f98: Invalid object contents: Bad message
 File corruption detected at /var/log/journal/93c40120d70f45089c5bf370c8013f20/system@0005d60414b0257e-c0937a379668661a.journal~:930f98 (of 16777216 bytes, 57%).
 FAIL: /var/log/journal/93c40120d70f45089c5bf370c8013f20/system@0005d60414b0257e-c0937a379668661a.journal~ (Bad message)
 PASS: /var/log/journal/93c40120d70f45089c5bf370c8013f20/system@0005d6511db6cd91-09c1187573da0a77.journal~
 PASS: /var/log/journal/93c40120d70f45089c5bf370c8013f20/user-1000@0005d5c16bae8053-1b21aa9351340399.journal~
 PASS: /var/log/journal/93c40120d70f45089c5bf370c8013f20/system@0005d629151537f3-3da5dc02d93a3652.journal~
 PASS: /var/log/journal/93c40120d70f45089c5bf370c8013f20/user-1000.journal
 PASS: /var/log/journal/93c40120d70f45089c5bf370c8013f20/system@c8d55d8ea5934cafb5b03d604c8058fd-0000000000000001-0005d5c04a6e7128.journal
 PASS: /var/log/journal/93c40120d70f45089c5bf370c8013f20/system@0005d666f215a68d-fbe7c60548c6d931.journal~
Exit mobile version