Subject: linux-image-6.1.0-10-amd64: Fails to load kernel modules due to bpf/btf issue
Date: Tue, 01 Aug 2023 19:22:17 +1000
Package: linux-image-6.1.0-10-amd64
Severity: important
Tags: patch
Dear Maintainer,
Current kernel failed to load modules for MASQUERADE nat rules giving the following
in dmesg:
[37404.063872] BPF: [99720] ENUM STROY
[37404.064350] BPF: size=4 vlen=14
[37404.064699] BPF:
[37404.065000] BPF: Invalid name
[37404.065287] BPF:
[37404.065463] failed to validate module [nf_conntrack] BTF: -22
[37404.067137] missing module BTF, cannot register kfuncs
The following upstream patch fixes the issue:
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=3de4d22cc9ac7c9f38e10edcf54f9a8891a9c2aa
And it has been included in upstream stable 6.1 kernel releases since 6.1.39:
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/log/?h=linux-6.1.y&ofs=400
Please include in Bookworm as its currently preventing netavark from using iptables
to load the MASQUERADE module and, thus, preventing me from running containers.
Thanks. :)
PS: This, I think, is covered in bug 1003965 but my attempt to email into that bug appears
to have failed. If it ever succeeds apologies for the dupe report. :(
-- System Information:
Debian Release: 12.1
APT prefers stable-updates
APT policy: (500, 'stable-updates'), (500, 'stable-security'), (500, 'stable')
Architecture: amd64 (x86_64)
Locale: LANG=en_AU.UTF-8, LC_CTYPE=en_AU.UTF-8 (charmap=UTF-8), LANGUAGE=en_AU:en
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
Acknowledgement sent
to Diederik de Haas <[email protected]>:
Extra info received and forwarded to list. Copy sent to Debian Kernel Team <[email protected]>.
(Tue, 01 Aug 2023 11:21:03 GMT) (full text, mbox, link).
Control: reassign -1 src:linux 6.1.37-1
Control: tag -1 upstream fixed-upstream
On Tuesday, 1 August 2023 11:22:17 CEST AP wrote:
> Package: linux-image-6.1.0-10-amd64
> Severity: important
> Tags: patch
>
> Current kernel failed to load modules for MASQUERADE nat rules giving the
> following in dmesg:
>
> [37404.063872] BPF: [99720] ENUM STROY
> [37404.064350] BPF: size=4 vlen=14
> [37404.064699] BPF:
> [37404.065000] BPF: Invalid name
> [37404.065287] BPF:
> [37404.065463] failed to validate module [nf_conntrack] BTF: -22
> [37404.067137] missing module BTF, cannot register kfuncs
>
> The following upstream patch fixes the issue:
>
> https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=
> 3de4d22cc9ac7c9f38e10edcf54f9a8891a9c2aa
>
> And it has been included in upstream stable 6.1 kernel releases since
> 6.1.39:
>
> https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/log/?h=linu
> x-6.1.y&ofs=400
Awesome bug report, thanks!
I've extended the list to document other kernel series and their fixes:
master: 3de4d22cc9ac7c9f38e10edcf54f9a8891a9c2aa included in 6.5-rc1
6.4: 90b9e7202676742ede5f1e6c46000a53c6a6c4c0 included in 6.4.4
6.1: 6baa6e4836d75a5d693c18dbf79b0695f05697e9 included in 6.1.39
5.10: none, also not queued (yet on 2023-08-01). Doesn't have the commit
that it Fixes, so it may not have been in 5.10 to begin with
> PS: This, I think, is covered in bug 1003965 but my attempt to email into
> that bug appears to have failed.
It would seem so, but I have decided not to merge the bugs as that bug was
found in 5.15.5-2, but the 5.15 series has the same 'problem' as the 5.10
series: it doesn't have the commit that it fixes (in my initial search).
So _could_ it be that #1003965 and #1042237 are actually 2 different bugs?
Ben: Could you look at this and take appropriate action?
Cheers,
Diederik
Acknowledgement sent
to Salvatore Bonaccorso <[email protected]>:
Extra info received and forwarded to list. Copy sent to Debian Kernel Team <[email protected]>.
(Tue, 01 Aug 2023 11:48:03 GMT) (full text, mbox, link).
Subject: Re: Bug#1042815: linux-image-6.1.0-10-amd64: Fails to load kernel
modules due to bpf/btf issue
Date: Tue, 1 Aug 2023 13:44:26 +0200
Control: tags -1 + moreinfo
Hi,
On Tue, Aug 01, 2023 at 07:22:17PM +1000, AP wrote:
> Package: linux-image-6.1.0-10-amd64
> Severity: important
> Tags: patch
>
> Dear Maintainer,
>
> Current kernel failed to load modules for MASQUERADE nat rules giving the following
> in dmesg:
>
> [37404.063872] BPF: [99720] ENUM STROY
> [37404.064350] BPF: size=4 vlen=14
> [37404.064699] BPF:
> [37404.065000] BPF: Invalid name
> [37404.065287] BPF:
> [37404.065463] failed to validate module [nf_conntrack] BTF: -22
> [37404.067137] missing module BTF, cannot register kfuncs
>
> The following upstream patch fixes the issue:
>
> https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=3de4d22cc9ac7c9f38e10edcf54f9a8891a9c2aa
>
> And it has been included in upstream stable 6.1 kernel releases since 6.1.39:
>
> https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/log/?h=linux-6.1.y&ofs=400
>
> Please include in Bookworm as its currently preventing netavark from using iptables
> to load the MASQUERADE module and, thus, preventing me from running containers.
>
> Thanks. :)
>
> PS: This, I think, is covered in bug 1003965 but my attempt to email into that bug appears
> to have failed. If it ever succeeds apologies for the dupe report. :(
is the assumption that you updated from 6.1.37-1 or 6.1.38-1 to
6.1.38-2 (or any of the combinations which did not bump ABI) but did
not reboot before loading the module correct?
The particular bug is something which was introduced in 5.18-rc1, and
fixed 6.1.39 and 6.4.4 following the fixing in 6.5-rc1. But if the
above assumption is correct then you just missed the reboot after the
kernel update.
Regards,
Salvatore
Subject: Re: Bug#1042815: linux-image-6.1.0-10-amd64: Fails to load kernel
modules due to bpf/btf issue
Date: Tue, 1 Aug 2023 21:49:39 +1000
On Tue, Aug 01, 2023 at 01:44:26PM +0200, Salvatore Bonaccorso wrote:
> > [37404.063872] BPF: [99720] ENUM STROY
> > [37404.064350] BPF: size=4 vlen=14
> > [37404.064699] BPF:
> > [37404.065000] BPF: Invalid name
> > [37404.065287] BPF:
> > [37404.065463] failed to validate module [nf_conntrack] BTF: -22
> > [37404.067137] missing module BTF, cannot register kfuncs
...
> > PS: This, I think, is covered in bug 1003965 but my attempt to email into that bug appears
> > to have failed. If it ever succeeds apologies for the dupe report. :(
>
> is the assumption that you updated from 6.1.37-1 or 6.1.38-1 to
> 6.1.38-2 (or any of the combinations which did not bump ABI) but did
> not reboot before loading the module correct?
The dmesg report in the bug appeared to fit bar the "missing module" line
so I figured that was just missed or something.
For me, this is a minty fresh install that I'm creating on a fresh instance
with ansible and the container creation part of the playbook fails.
> The particular bug is something which was introduced in 5.18-rc1, and
> fixed 6.1.39 and 6.4.4 following the fixing in 6.5-rc1. But if the
> above assumption is correct then you just missed the reboot after the
> kernel update.
That bit doesn't fit me. As I said, minty fresh install where the playbook
for building the instance hasn't had a chance to finish.
Andrew
Subject: Re: Bug#1042815: linux-image-6.1.0-10-amd64: Fails to load kernel
modules due to bpf/btf issue
Date: Thu, 3 Aug 2023 12:28:00 +1000
On Tue, Aug 01, 2023 at 01:44:26PM +0200, Salvatore Bonaccorso wrote:
> > PS: This, I think, is covered in bug 1003965 but my attempt to email into that bug appears
> > to have failed. If it ever succeeds apologies for the dupe report. :(
>
> is the assumption that you updated from 6.1.37-1 or 6.1.38-1 to
> 6.1.38-2 (or any of the combinations which did not bump ABI) but did
> not reboot before loading the module correct?
Nope. You're right. This is what happens when you ansiblise everything.
The initial image is with -1, my playbook upgraded it to -2 and tried to use it
and it failed. Just did a manual test and modified my playbook to reboot and
tested that and it works fine after reboot.
So... my bad. Feeling a bit sheepish. Apologies.
Still... downloading and installing a new kernel shouldn't break a live system.
Andrew
Reply sent
to Salvatore Bonaccorso <[email protected]>:
You have taken responsibility.
(Thu, 20 Feb 2025 18:21:02 GMT) (full text, mbox, link).
Subject: Re: Bug#1042815: linux-image-6.1.0-10-amd64: Fails to load kernel
modules due to bpf/btf issue
Date: Thu, 20 Feb 2025 19:17:37 +0100
Hi,
On Thu, Aug 03, 2023 at 12:28:00PM +1000, AP wrote:
> On Tue, Aug 01, 2023 at 01:44:26PM +0200, Salvatore Bonaccorso wrote:
> > > PS: This, I think, is covered in bug 1003965 but my attempt to email into that bug appears
> > > to have failed. If it ever succeeds apologies for the dupe report. :(
> >
> > is the assumption that you updated from 6.1.37-1 or 6.1.38-1 to
> > 6.1.38-2 (or any of the combinations which did not bump ABI) but did
> > not reboot before loading the module correct?
>
> Nope. You're right. This is what happens when you ansiblise everything.
>
> The initial image is with -1, my playbook upgraded it to -2 and tried to use it
> and it failed. Just did a manual test and modified my playbook to reboot and
> tested that and it works fine after reboot.
>
> So... my bad. Feeling a bit sheepish. Apologies.
>
> Still... downloading and installing a new kernel shouldn't break a live system.
So I think we can close this bug now as in practice for stable we need
to bump ABI on each update (to actually suffice the requirements for
secure boot), and for trixie and above the issue is implicitly
resolved as well as there is no such thing anymore of maintaining
manually the ABI, but for each build the modules are signed with an
ephemeral key.
So closing the issue now.
Regards,
Salvatore
Debbugs is free software and licensed under the terms of the GNU General
Public License version 2. The current version can be obtained
from https://bugs.debian.org/debbugs-source/.