Logs (journalctl & logrotate)

核心 · Key Idea

In one line: modern Linux service logs default to systemd-journal (structured, indexed); legacy /var/log/*.log files still exist. journalctl -u service -f for live tail, logrotate to keep logs from filling the disk.

What it is#

/var/log/syslog           # legacy system log (Debian / Ubuntu)
/var/log/messages         # same on RHEL family
/var/log/auth.log         # login / sudo
/var/log/nginx/           # app-managed directory
journalctl -u nginx       # the same service log via systemd

systemd-journal stores logs structured (with unit / pid / boot id) — many distros still forward to syslog for dual storage.

Analogy#

打个比方 · Analogy

Old logs = a stack of handwritten diaries, archived by month. journal = an indexed digital filing cabinet — query by service, time, priority, or any field.

Key concepts#

UnitService unit

A service from systemd's view; corresponds to one log stream in journal.

PriorityPriority

0 emerg → 7 debug. `-p err` shows errors and above.

Boot IDBoot ID

`-b` shows current boot; `-b -1` previous boot (handy for 'why did it reboot').

logrotateLog rotation

Splits, compresses, and deletes old logs. Configs in `/etc/logrotate.d/*`.

Structured fieldsStructured Fields

`journalctl -o json`; filter by `_PID=` `_HOSTNAME=` etc.

Common commands#

# journalctl
journalctl -u nginx -f                   # follow
journalctl -u nginx --since "10 min ago"
journalctl -u nginx --since today
journalctl -u nginx -p err               # error and above
journalctl -k                            # kernel
journalctl --disk-usage                  # journal disk use
journalctl --vacuum-time=7d              # keep last 7 days
journalctl --vacuum-size=500M
 
# Legacy / app-written
tail -f /var/log/nginx/error.log
zcat /var/log/nginx/access.log.2.gz | grep '500 '
less /var/log/auth.log
 
# logrotate
logrotate -d /etc/logrotate.conf         # dry-run
logrotate -f /etc/logrotate.d/nginx      # force rotation

How it works#

The journal has its own space cap (SystemMaxUse, default 10 % of disk) — no logrotate needed for it.

Practical notes#

Services should log to stdout/stderr — systemd collects → journal. Don't write your own /var/log file (same rule for containers).
journal eating disk? journalctl --vacuum-size=500M immediately, or set SystemMaxUse= in /etc/systemd/journald.conf.

logrotate sample:

/var/log/nginx/*.log {
    daily
    rotate 14
    compress
    missingok
    notifempty
    sharedscripts
    postrotate
        nginx -s reopen
    endscript
}

Production pattern: app → stdout → journal → forward to central ELK / Loki; don't keep too much locally.
Beyond grep: rg is faster; lnav is great for multi-log visualization.

Easy confusions#

journal

Structured, query by unit / priority.
Auto-rotates with a size cap.

syslog / file logs

Plain text, needs logrotate.
Friendly to legacy tooling.