There is still no way to filter application output at collection time, which can functionally make journald useless if you have an application that is too chatty, and will likely never be given that (AFAICT) there is hefty ideological opposition from the systems developers¹.
Isn't this something that, to a degree, might be mitigated by rate limiting journald has?
IMO Lennart is right in that collection should be optimized to a point where it is (almost) never the bottleneck. I, personally, would put a filtering program between the chatty application and journald.
- Legacy application, with no access to source code.
- Company that wrote the application hasn't existed for a decade.
- It logs several useless lines every second. This is non-configurable, and includes a timestamp which makes each linue distinct.
- It also logs some very important lines that _must_ be responded to.
I don't think this is a unique situation to find yourself in, and currently this application is permitted - explicitly - to spam the journal to a point that it is effectively useless. Yes, you can use a filtering application to sit in between systemd but that's a workaround and not an actual solution (as stated in the issue linked above):
- it could not be "plugged in" via overlay (i.e. modification of Exec... is needed - this gets ugly especially if you have ExecStartPre/ExecStopPost)
- SyslogIdentifier has to be set to produce proper name in journald
- The filter itself has to be quite reliable (to not hang) and a bit sophisticated to properly handle signals, process shutdown, low disk space etc.
What you need is a pipe through grep. It’s as reliable as it gets. I don’t understand, though, why your log filter should all of a sudden grow disk space monitoring and signal handling. This is what systemd and journald should be doing, and in their recent versions, they are quite capable.
How about to open this exceedly verbose legacy program with IDA and fill with NOPs all the places where this program logs something useless? I mean, this is an activity which really should be named as "hacking", in opposition to playing with init scripts.
I generally agree, but will say that’s not always easy if it’s something system level that’s being induced to be too chatty —- you’re stuck replacing the upstream unit’s Exec, which frankly kind of sucks. I’m also trusting the filter to be as reliable, which may or may not be true.
Rate limiting works, but it’s a pretty blunt instrument and can lead to losing your actually valuable entries vs capturing garbage.
¹https://github.com/systemd/systemd/issues/6432