Package: rsyslog
Version: 8.2102.0-2+deb11u1
Severity: important
Tags: upstream
Hi,
We had a system fill up its root filesystem; after resolving that,
rsyslog would not start up, instead SEGVing every time.
strace showed that the SEGV was happening shortly after opening files
in /var/spool/rsyslog; moving those files away allowed rsyslog to
start up OK.
I attach a tarball of the offending files in case they help.
I assume that rsyslog is not adequately checking the spool files; I
think this is quite a significant problem, as it means log entries are
lost. Obviously they get lost when / is full, but this failure mode
means that even once that problem is fixed, rsyslog can't start up, so
log messages continue to be discarded. I considered setting serious
severity as a result, but I'll leave that to your discretion.
I think rsyslog should handle this better: either explicitly flagging
the offending file, or better by discarding a corrupt file and
recording the fact.
Thanks,
Matthew
-- System Information:
Debian Release: 11.6
APT prefers stable-updates
APT policy: (500, 'stable-updates'), (500, 'stable-security'), (500,
'stable-debug'), (500, 'stable')
Architecture: amd64 (x86_64)
Kernel: Linux 5.10.0-21-amd64 (SMP w/48 CPU threads)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE
not set
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
Versions of packages rsyslog depends on:
ii init-system-helpers 1.60
ii libc6 2.31-13+deb11u5
ii libestr0 0.1.10-2.1+b1
ii libfastjson4 0.99.9-1
ii liblognorm5 2.0.5-1.1
ii libsystemd0 247.3-7+deb11u1
ii libuuid1 2.36.1-8+deb11u1
ii zlib1g 1:1.2.11.dfsg-2+deb11u2
Versions of packages rsyslog recommends:
ii logrotate 3.18.0-2+deb11u1
Versions of packages rsyslog suggests:
pn rsyslog-doc <none>
ii rsyslog-gnutls 8.2102.0-2+deb11u1
pn rsyslog-gssapi <none>
pn rsyslog-mongodb <none>
pn rsyslog-mysql | rsyslog-pgsql <none>
pn rsyslog-relp <none>
-- no debconf information
Acknowledgement sent
to Michael Biebl <[email protected]>:
Extra info received and forwarded to list.
(Wed, 15 Feb 2023 17:18:02 GMT) (full text, mbox, link).
Control: tags -1 + moreinfo
Hi,
thanks for the bug report.
Am 15.02.23 um 17:16 schrieb Matthew Vernon:
> I attach a tarball of the offending files in case they help.
Can you share your rsyslog configuration and the output of
running "rsyslogd -d -n" ?
Regards,
Michael
Acknowledgement sent
to Michael Biebl <[email protected]>:
Extra info received and forwarded to list.
(Wed, 15 Feb 2023 18:30:03 GMT) (full text, mbox, link).
Am 15.02.23 um 18:16 schrieb Michael Biebl:
> Control: tags -1 + moreinfo
>
> Hi,
>
> thanks for the bug report.
>
> Am 15.02.23 um 17:16 schrieb Matthew Vernon:
>> I attach a tarball of the offending files in case they help.
>
> Can you share your rsyslog configuration and the output of
> running "rsyslogd -d -n" ?
I can try to reproduce the issue once I have the full rsyslog config.
Otherwise, a backtrace (with debug symbols) would be helpful as well.
Michael
Acknowledgement sent
to Michael Biebl <[email protected]>:
Extra info received and forwarded to list.
(Wed, 15 Feb 2023 19:15:02 GMT) (full text, mbox, link).
Am 15.02.23 um 19:27 schrieb Michael Biebl:
> Am 15.02.23 um 18:16 schrieb Michael Biebl:
>> Control: tags -1 + moreinfo
>>
>> Hi,
>>
>> thanks for the bug report.
>>
>> Am 15.02.23 um 17:16 schrieb Matthew Vernon:
>>> I attach a tarball of the offending files in case they help.
>>
>> Can you share your rsyslog configuration and the output of
>> running "rsyslogd -d -n" ?
>
> I can try to reproduce the issue once I have the full rsyslog config.
> Otherwise, a backtrace (with debug symbols) would be helpful as well.
I'm asking because if I can't reproduce the issue myself, I would kindly
ask you to try rsyslog from bpo and if it's still reproducible there,
file the issue directly upstream at
https://github.com/rsyslog/rsyslog/issues
It's likely that they will have further questions which are best
answered by you directly.
Michael
Acknowledgement sent
to Matthew Vernon <[email protected]>:
Extra info received and forwarded to list. Copy sent to Michael Biebl <[email protected]>.
(Thu, 16 Feb 2023 15:21:05 GMT) (full text, mbox, link).
Hi,
On 15/02/2023 18:27, Michael Biebl wrote:
> Am 15.02.23 um 18:16 schrieb Michael Biebl:
>> Control: tags -1 + moreinfo
>> Am 15.02.23 um 17:16 schrieb Matthew Vernon:
>>> I attach a tarball of the offending files in case they help.
>>
>> Can you share your rsyslog configuration and the output of
>> running "rsyslogd -d -n" ?
>
> I can try to reproduce the issue once I have the full rsyslog config.
> Otherwise, a backtrace (with debug symbols) would be helpful as well.
Here's a tarball of our rsyslog config (except the ssl keypair for
obvious reasons :-) ) - it's the contents of /etc/rsyslog.d and
/etc/rsyslog.lookup.d/
I've included everything for sake of completeness, but I suspect you
really only need 30-remote-syslog.conf (and maybe 40-swift.conf)
I hope this helps; the affected system is a production server where we
have limited redundancy, so I had to get it back into service ASAP.
Hopefully you can now reproduce this :)
Thanks,
Matthew
Acknowledgement sent
to Matthew Vernon <[email protected]>:
Extra info received and forwarded to list. Copy sent to Michael Biebl <[email protected]>.
(Thu, 16 Feb 2023 16:27:03 GMT) (full text, mbox, link).
Acknowledgement sent
to Michael Biebl <[email protected]>:
Extra info received and forwarded to list.
(Thu, 16 Feb 2023 20:51:03 GMT) (full text, mbox, link).
Am 16.02.23 um 16:18 schrieb Matthew Vernon:
> Hi,
>
> On 15/02/2023 18:27, Michael Biebl wrote:
>> Am 15.02.23 um 18:16 schrieb Michael Biebl:
>>> Control: tags -1 + moreinfo
>>> Am 15.02.23 um 17:16 schrieb Matthew Vernon:
>>>> I attach a tarball of the offending files in case they help.
>>>
>>> Can you share your rsyslog configuration and the output of
>>> running "rsyslogd -d -n" ?
>>
>> I can try to reproduce the issue once I have the full rsyslog config.
>> Otherwise, a backtrace (with debug symbols) would be helpful as well.
>
> Here's a tarball of our rsyslog config (except the ssl keypair for
> obvious reasons :-) ) - it's the contents of /etc/rsyslog.d and
> /etc/rsyslog.lookup.d/
>
> I've included everything for sake of completeness, but I suspect you
> really only need 30-remote-syslog.conf
The important bits were in 30-remote-syslog.conf indeed. With that the
issue was reproducible and I therefor forwarded this to upstream. See
https://github.com/rsyslog/rsyslog/issues/5085
I didn't explicitly ask you, if I could attach your config files/spool
files there, but I assumed as you attached it to the Debian bug tracker,
that this is ok. If not, please let me know.
Acknowledgement sent
to Matthew Vernon <[email protected]>:
Extra info received and forwarded to list. Copy sent to Michael Biebl <[email protected]>.
(Thu, 16 Feb 2023 20:57:03 GMT) (full text, mbox, link).
Subject: Re: Bug#1031360: rsyslog: SEGV on startup with truncated files in
spool
Date: Thu, 16 Feb 2023 20:54:41 +0000
Hi,
On 16/02/2023 20:48, Michael Biebl wrote:
> The important bits were in 30-remote-syslog.conf indeed. With that the
> issue was reproducible and I therefor forwarded this to upstream. See
> https://github.com/rsyslog/rsyslog/issues/5085
Great, thank you.
> I didn't explicitly ask you, if I could attach your config files/spool
> files there, but I assumed as you attached it to the Debian bug tracker,
> that this is ok. If not, please let me know.
Yes, that's fine.
Regards,
Matthew
Debbugs is free software and licensed under the terms of the GNU General
Public License version 2. The current version can be obtained
from https://bugs.debian.org/debbugs-source/.