Subject: rsyslog: Please send user errors only to user.log and rotate that file based on size.
Date: Tue, 06 Jul 2021 02:51:01 -0700
Package: rsyslog
Version: 8.2102.0-2
Severity: important
Dear Maintainer,
My Debian box's hard disk filled up due to a single buggy user
application spewing messages at syslog. While I blame the application,
Debian's rsyslog should be more robust by default.
This is something I've seen happening more and more frequently.
A typical response people get when googling the problem is
"sudo rm /var/log/* and then set that up to run daily as a cronjob."
That's terrible advice and makes Debian seem shoddy.
Ideally, I'd like to see some sort of rate-limiter added to rsyslog.
In the meantime, here is a simple three step fix which would improve
stability for many people using Debian.
1. /etc/rsyslog.conf: add user.none to /var/log/syslog, debug, and messages.
That way, the spewage is limited to a single file, user.log.
2. /etc/logrotate.d/rsyslog: add "maxsize" so that log files that are
greater than a certain size are rotated even if their time
criteria are not due. I suggest "maxsize 1G" is reasonable for
most people and can be adjusted for those who want more.
3a. Move /etc/cron.daily/logrotate to /etc/cron.hourly/ so that file
sizes are checked more often. (This also fixes the "bug" where an
hourly entry in logrotate.conf only gets rotated daily.)
3b. /usr/lib/systemd/system/logrotate.timer: Likewise for systemd.
[Timer] [Timer]
OnCalendar=daily --> OnCalendar=hourly
AccuracySec=1h AccuracySec=1m
Persistent=true Persistent=true
Steps 1 and 2 ensure that the disk will not fill up and that important
system messages won't be rotated away too quickly based on file size.
Step 3 is necessary because the time in which a log file can fill up
/var is no longer measured in days. Currently, I have a single
chromium process sending over thirty thousand messages per second and
I doubt that's even close to the maximum possible.
Thank you.
*** Reporter, please consider answering these questions, where appropriate ***
* What led up to the situation?
* What exactly did you do (or not do) that was effective (or
ineffective)?
* What was the outcome of this action?
* What outcome did you expect instead?
*** End of the template - remove these template lines ***
-- System Information:
Debian Release: 11.0
APT prefers testing-security
APT policy: (500, 'testing-security'), (500, 'testing')
Architecture: amd64 (x86_64)
Foreign Architectures: i386
Kernel: Linux 5.10.0-7-amd64 (SMP w/48 CPU threads)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE not set
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled
Versions of packages rsyslog depends on:
ii init-system-helpers 1.60
ii libc6 2.31-12
ii libestr0 0.1.10-2.1+b1
ii libfastjson4 0.99.9-1
ii liblognorm5 2.0.5-1.1
ii libsystemd0 247.3-5
ii libuuid1 2.36.1-7
ii zlib1g 1:1.2.11.dfsg-2
Versions of packages rsyslog recommends:
ii logrotate 3.18.0-2
Versions of packages rsyslog suggests:
pn rsyslog-doc <none>
pn rsyslog-gssapi <none>
pn rsyslog-mongodb <none>
pn rsyslog-mysql | rsyslog-pgsql <none>
pn rsyslog-openssl | rsyslog-gnutls <none>
pn rsyslog-relp <none>
-- Configuration Files:
/etc/logrotate.d/rsyslog changed:
# Standard rsyslog log files. For syntax, see logrotate.conf(8) -*- conf -*-
/var/log/syslog
/var/log/mail.info
/var/log/mail.warn
/var/log/mail.err
/var/log/mail.log
/var/log/daemon.log
/var/log/user.log
/var/log/kern.log
/var/log/auth.log
/var/log/lpr.log
/var/log/cron.log
/var/log/debug
/var/log/messages
{
rotate 4
weekly
maxsize 1G
missingok
notifempty
compress
delaycompress
sharedscripts
postrotate
/usr/lib/rsyslog/rsyslog-rotate
endscript
}
/etc/rsyslog.conf changed:
# /etc/rsyslog.conf configuration file for rsyslog
#
# For more information install rsyslog-doc and see
# /usr/share/doc/rsyslog-doc/html/configuration/index.html
#################
#### MODULES ####
#################
module(load="imuxsock") # provides support for local system logging
module(load="imklog") # provides kernel logging support
#module(load="immark") # provides --MARK-- message capability
# provides UDP syslog reception
#module(load="imudp")
#input(type="imudp" port="514")
# provides TCP syslog reception
#module(load="imtcp")
#input(type="imtcp" port="514")
###########################
#### GLOBAL DIRECTIVES ####
###########################
#
# Use traditional timestamp format.
# To enable high precision timestamps, comment out the following line.
#
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
#
# Set the default permissions for all log files.
#
$FileOwner root
$FileGroup adm
$FileCreateMode 0640
$DirCreateMode 0755
$Umask 0022
#
# Where to place spool and state files
#
$WorkDirectory /var/spool/rsyslog
#
# Include all config files in /etc/rsyslog.d/
#
$IncludeConfig /etc/rsyslog.d/*.conf
###############
#### RULES ####
###############
#
# First some standard log files. Log by facility.
#
auth,authpriv.* /var/log/auth.log
*.*;auth,authpriv,user.none -/var/log/syslog
#cron.* /var/log/cron.log
daemon.* -/var/log/daemon.log
kern.* -/var/log/kern.log
lpr.* -/var/log/lpr.log
mail.* -/var/log/mail.log
user.* -/var/log/user.log
#
# Logging for the mail system. Split it up so that
# it is easy to write scripts to parse these files.
#
mail.info -/var/log/mail.info
mail.warn -/var/log/mail.warn
mail.err /var/log/mail.err
#
# Some "catch-all" log files.
#
*.=debug;\
auth,authpriv.none;\
mail,user.none -/var/log/debug
*.=info;*.=notice;*.=warn;\
auth,authpriv.none;\
cron,daemon.none;\
mail,user.none -/var/log/messages
#
# Emergencies are sent to everybody logged in.
#
*.emerg :omusrmsg:*
/usr/lib/systemd/system/logrotate.timer changed:
[Unit]
Description=Daily rotation of log files
Documentation=man:logrotate(8) man:logrotate.conf(5)
[Timer]
OnCalendar=hourly
AccuracySec=1m
Persistent=true
[Install]
WantedBy=timers.target
-- no debconf information
Acknowledgement sent
to Michael Biebl <[email protected]>:
Extra info received and forwarded to list.
(Tue, 06 Jul 2021 10:15:03 GMT) (full text, mbox, link).
Am 06.07.21 um 11:51 schrieb Ben Wong:
> Package: rsyslog
> Version: 8.2102.0-2
> Severity: important
>
> Dear Maintainer,
>
> My Debian box's hard disk filled up due to a single buggy user
> application spewing messages at syslog. While I blame the application,
> Debian's rsyslog should be more robust by default.
>
> This is something I've seen happening more and more frequently.
> A typical response people get when googling the problem is
> "sudo rm /var/log/* and then set that up to run daily as a cronjob."
> That's terrible advice and makes Debian seem shoddy.
>
> Ideally, I'd like to see some sort of rate-limiter added to rsyslog.
There is a rater limiter in imuxsock.
https://www.rsyslog.com/how-to-use-rate-limiting-in-rsyslog/
The problem is: a couple of GB of log data is something that a busy log
server can generate in a couple of minutes which perfectly valid.
On a laptop, a couple of GB of log data is probably unusual.
The problem is to pick a default which suits everyone, which is guess is
not possible.
> In the meantime, here is a simple three step fix which would improve
> stability for many people using Debian.
>
>
> 1. /etc/rsyslog.conf: add user.none to /var/log/syslog, debug, and messages.
> That way, the spewage is limited to a single file, user.log.
I'm not really sure we can win this game.
Next time, it's not a user application generating a lot of data but some
system service.
That said, there is certainly value in cleaning up the default Debian
rsyslog default config, see e.g.
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=508376https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=580552
to avoid duplication in log messages.
> 2. /etc/logrotate.d/rsyslog: add "maxsize" so that log files that are
> greater than a certain size are rotated even if their time
> criteria are not due. I suggest "maxsize 1G" is reasonable for
> most people and can be adjusted for those who want more.
Depends. I'd argue that the systemd journal nowadays is better suited
for end user systems anyway and syslog/rsyslog more targetted at the
enterprise area. So I don't think it makes sense to optimize for the former.
> 3a. Move /etc/cron.daily/logrotate to /etc/cron.hourly/ so that file
> sizes are checked more often. (This also fixes the "bug" where an
> hourly entry in logrotate.conf only gets rotated daily.)
>
> 3b. /usr/lib/systemd/system/logrotate.timer: Likewise for systemd.
>
> [Timer] [Timer]
> OnCalendar=daily --> OnCalendar=hourly
> AccuracySec=1h AccuracySec=1m
> Persistent=true Persistent=true
>
>
> Steps 1 and 2 ensure that the disk will not fill up and that important
> system messages won't be rotated away too quickly based on file size.
>
> Step 3 is necessary because the time in which a log file can fill up
> /var is no longer measured in days. Currently, I have a single
> chromium process sending over thirty thousand messages per second and
> I doubt that's even close to the maximum possible.
I don't think, changing the cron job to hourly is really going to help
if you have an application running amok. 1 hour is more then sufficient
to fill up your disk.
Fwiw, my plan is for bookworm to have rsyslog removed from a default
installation and rely on journald only (which does size-based log
truncation), as I think journald is a better fit for end-user systems.
rsyslog would then only be installed by those who explicitly need it
(those users will most likely will tweak its settings anyway to suit
their needs).
Regards,
Michael
Acknowledgement sent
to Michael Biebl <[email protected]>:
Extra info received and forwarded to list.
(Tue, 06 Jul 2021 10:18:02 GMT) (full text, mbox, link).
Am 06.07.21 um 11:51 schrieb Ben Wong:
> 2. /etc/logrotate.d/rsyslog: add "maxsize" so that log files that are
> greater than a certain size are rotated even if their time
> criteria are not due. I suggest "maxsize 1G" is reasonable for
> most people and can be adjusted for those who want more.
>
> 3a. Move /etc/cron.daily/logrotate to /etc/cron.hourly/ so that file
> sizes are checked more often. (This also fixes the "bug" where an
> hourly entry in logrotate.conf only gets rotated daily.)
This is basically a duplicate of
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=954069
Keep in mind that /etc/cron.daily/logrotate is not under the control of
the rsyslog package.
I'm inclined to merge your bug report accordingly.
Acknowledgement sent
to Michael Biebl <[email protected]>:
Extra info received and forwarded to list.
(Tue, 06 Jul 2021 10:57:03 GMT) (full text, mbox, link).
Am 06.07.21 um 12:08 schrieb Michael Biebl:
> Am 06.07.21 um 11:51 schrieb Ben Wong:
>> Ideally, I'd like to see some sort of rate-limiter added to rsyslog.
>
> There is a rater limiter in imuxsock.
> https://www.rsyslog.com/how-to-use-rate-limiting-in-rsyslog/
>
> The problem is: a couple of GB of log data is something that a busy log
> server can generate in a couple of minutes which perfectly valid.
>
> On a laptop, a couple of GB of log data is probably unusual.
>
> The problem is to pick a default which suits everyone, which is guess is
> not possible.
https://www.rsyslog.com/doc/v8-stable/configuration/modules/imuxsock.html#syssock-ratelimit-interval
Apparently this feature caused problems, so it was turned off again.
> SysSock.RateLimit.Interval
> type default max mandatory obsolete legacy directive
> integer 0 no $SystemLogRateLimitInterval
>
> Specifies the rate-limiting interval in seconds. Default value is 0, which turns off rate limiting. Set it to a number of seconds (5 recommended) to activate rate-limiting. The default of 0 has been chosen as people experienced problems with this feature activated by default. Now it needs an explicit opt-in by setting this parameter.
Debbugs is free software and licensed under the terms of the GNU General
Public License version 2. The current version can be obtained
from https://bugs.debian.org/debbugs-source/.