Log files in Linux

  • Log files can be described as records that are stored for administrative persons for keeping track and monitor system events
  • Log files can contain information of system, kernel, system services (daemons) and running programs
  • Usually log files can be divided in the following four categories in Linux:
    • Application logs
    • Event logs
    • Service logs
    • System logs
  • Log files should be monitored especially in server environment where the availability and operation of a service is critical
  • With log files system administrator can receive detailed information on server performance, error messages, security and other underlying issues
  • Linux log files are usually stored in /var/log directory

  • Below is the list of the most important log files found from /var/log directory in Ubuntu distribution:

    • syslog → generic system activity logs including application-related service errors and messages (critical system and kernel information is not stored here!)
    • auth.log → user authentication related information (monitor for example unauthorized login attempts)
    • kern.log → information logged by kernel (troubleshoot kernel related errors and warnings)

  • In addition to these listed log files there are also service specific logs

  • Below are couple of common services for Linux-based operating systems and log files for these services

    • mail.log → mail server related logs (information about postfix, smtpd, MailScanner etc. email related services installed and running on mail server)
    • httpd/ → this is a directory containing log files recorded by Apache web server (contains error_log for httpd and other system related errors and access_log for access requests to the server)
    • mysql.log → MySQL database server log file containing all debug, failure and success messages related to MySQL daemon

  • Examples of viewing log files in system:

  • Example 1: Print last five lines from authentication log file (auth.log)
testuser@ubuntu-pc:~$ cat /var/log/auth.log | tail -5
Oct 6 10:20:48 testuser login[1130]: pam_unix(login:session): session opened for user testuser by LOGIN (uid=0)
Oct 6 10:20:48 testuser systemd-logind[962]: New session 1 of user testuser.
Oct 6 10:20:48 testuser systemd: pam_unix(systemd-user:session): session opened for user testuser by (uid=0)
Oct 6 11:17:01 testuser CRON[29288]: pam_unix(cron:session): session opened for user root by (uid=0)
Oct 6 11:17:01 testuser CROM[29288]: pam_unix(cron:session): session closed for user root
  • Example 2: List all log entries for user testuser from auth.log file
testuser@ubuntu-pc:~$ cat /var/log/auth.log | grep "user testuser"
Oct 11 06:29:56 testuser su[11067]: pam_unix(su:session): session opened for user testuser by testuser(uid=1002)
Nov 20 11:57:34 testuser login[1025]: pam_unix(login:session): session opened for user testuser bu LOGIN(uid=0)
Nov 20 11:57:34 testuser systemd-logind[777]: New session 1 of user testuser.
Nov 20 11:47:34 testuser systemd: pam_unix(systemd-user:session): session opened for user testuser by (uid=0)
Dec 9 16:18:56 testuser login[958]: pam_unix(login:session):session opened for user testuser by LOGIN(uid=0)
Dec 9 16:18:56 testuser systemd-logind[807]: New session 1 of user testuser.
Dec 9 16:18:56 testuser systemd: pam_unix(systemd-user:session): session opened for user testuser by (uid=0)

Journald

  • Journald is included in systemd and is used for logging
  • Journald captures log data, records them, makes them available for search and remove log data when it expires and is no longer needed
  • Basically journald is a CLI tool for effectively access log data

  • Most important features in journald are the following:

    • Indexing: Binary storage for logs with indexed data (this makes data lookup a lot faster compared to plain text files)
    • Structured logging: Log data filtering for example with priority or date and time
    • Access control: Each regular Linux user sees his/her own log files only (root has access to all log files)
    • Automatic log rotation: Journald can be configured to delete unneccessary log files when disk space is low

  • Journald commands are run with journalctl

  • An example of running just the command journalctl is in the figure below

Journalctl command example

  • Like shown in the figure:

    • All log entries are listed from oldest to newest
    • Content is paged by less command meaning you can navigate with up and down arrow keys
    • Left and right arrow keys are used to access data going out of the screen border
    • Data format for each line is the following: timestamp (date and time) hostname program and its PID value message

  • Journalctl has some useful options for filtering the output:

    • last N lines → -n option
    • Monitor incoming new log lines → -f option
    • Data from newest to oldest → --reverse option
    • Skip paging (utilize grep instead for example) → --no-pager
    • Show entries within a specific time window → --since

  • Below are some examples of journalctl command:

  • Example 1: Show log entries from the last five minutes
testuser@ubuntu-PC:~$ journalctl --since "5 minutes ago"
-- Logs begin at Wed 2020-10-07 11:19:15 EEST, end at Wed 2020-10-07 13:12:54 EEST. --
Oct 07 13:11:12 ubuntu-PC systemd[841]: Started Application launched by gnome-shell.
Oct 07 13:11:21 ubuntu-PC dbus-daemon[471]: [system] Activating via systemd: service name='org.freedesktop.timedate1' unit='dbus-org.freedesktop.timedate1.service' requested b>
Oct 07 13:11:21 ubuntu-PC systemd[1]: Starting Time & Date Service...
Oct 07 13:11:21 ubuntu-PC dbus-daemon[471]: [system] Successfully activated service 'org.freedesktop.timedate1'
Oct 07 13:11:21 ubuntu-PC systemd[1]: Started Time & Date Service.
Oct 07 13:11:25 ubuntu-PC dbus-daemon[471]: [system] Activating via systemd: service name='org.freedesktop.locale1' unit='dbus-org.freedesktop.locale1.service' requested by ':>
Oct 07 13:11:25 ubuntu-PC systemd[1]: Starting Locale Service...
Oct 07 13:11:25 ubuntu-PC dbus-daemon[471]: [system] Successfully activated service 'org.freedesktop.locale1'
Oct 07 13:11:25 ubuntu-PC systemd[1]: Started Locale Service.
Oct 07 13:11:47 ubuntu-PC systemd[841]: gnome-launched-gnome-control-center.desktop-2182.scope: Succeeded.
Oct 07 13:11:51 ubuntu-PC systemd-resolved[434]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature le>
Oct 07 13:11:51 ubuntu-PC systemd[1]: systemd-timedated.service: Succeeded.
Oct 07 13:11:55 ubuntu-PC systemd[1]: systemd-localed.service: Succeeded.
Oct 07 13:12:54 ubuntu-PC PackageKit[1230]: daemon quit
Oct 07 13:12:54 ubuntu-PC systemd[1]: packagekit.service: Succeeded.
lines 1-16/16 (END)

-Example 2: List all system boots

testuser@ubuntu-PC:~$ journalctl --list-boots
-1 e9d1b3a823e94c82ac762c8c23444f26 Wed 2020-10-07 11:19:15 EEST—Wed 2020-10-07 13:06:02 EEST
 0 e14055576b6f41e8b4a0b48dd99c248e Wed 2020-10-07 13:07:35 EEST—Wed 2020-10-07 13:17:01 EEST