Subject: Bug#1084224: isenkram-cli: isenkram installs raspi-firmware on non-raspi i686 hardwareleading to an error

From: Christian Meyer <[email-masked]>
Date: Sun, 06 Oct 2024 22:21:57 +0200

Package: isenkram-cli
Version: 0.51
Severity: normal
X-Debbugs-Cc: [email-masked]

Dear Maintainer,

I used to run isenkram-autoinstall-firmware without any worries.
On my Lenovo ThinkPad 10 tablet (with Intel Atom processor) this installs raspi-firmware. This is unexpected.

This fails with 'raspi-firmware: missing /boot/firmware, did you forget to mount it?' even building a new initrd fails (with the same error)
Of course you could say: it's a bug in raspi-firmware (perhaps it is), but for me it's a problem with isenkram-cli, too.
To be honest, I never had agreed to install raspi-firmware, as my device isn't a raspi and I don't see, where isenkram 'believes' that I run a raspi.

This is my output of:
# isenkram-lookup 
beignet-opencl-icd
bluez
firmware-brcm80211
firmware-misc-nonfree
gkrellm-thinkbat
hdapsd
pidgin-blinklight
thinkfan
tlp
tp-smapi-dkms
tpb

The solution was pretty easy (apt purge raspi-firmware), but this doesn't solves the real problem.
What would be the correct firmware for my device?
How to automatically install the correct firmware (with a script, e.g. used in FAI)?

Thank you for your attention
Christian Meyer

-- System Information:
Debian Release: 12.7
  APT prefers stable-updates
  APT policy: (500, 'stable-updates'), (500, 'stable-security'), (500, 'proposed-updates'), (500, 'stable')
Architecture: i386 (i686)

Kernel: Linux 6.1.0-26-686 (SMP w/4 CPU threads; PREEMPT)
Locale: LANG=de_DE.UTF-8, LC_CTYPE=de_DE.UTF-8 (charmap=UTF-8), LANGUAGE not set
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages isenkram-cli depends on:
ii  appstream              0.16.1-2
ii  curl                   7.88.1-10+deb12u7
ii  debconf [debconf-2.0]  1.5.82
ii  gir1.2-appstream-1.0   0.16.1-2
ii  lsb-release            12.0-1
ii  python3                3.11.2-1+b1
ii  python3-apt            2.6.0
ii  python3-gi             3.42.2-3+b1
ii  usb.ids                2024.07.04-0+deb12u1

isenkram-cli recommends no packages.

isenkram-cli suggests no packages.

-- debconf information:
* isenkram/install_hw_packages: raspi-firmware

---

From: Petter Reinholdtsen <[email-masked]>
Date: Mon, 07 Oct 2024 07:23:33 +0200

[Christian Meyer]
> On my Lenovo ThinkPad 10 tablet (with Intel Atom processor) this
> installs raspi-firmware. This is unexpected.

Indeed.

Note, isenkram-lookup and isenkram-autoinstall-firmware use two
different approaches, so the output from the former can be ignored when
debugging isenkram-autoinstall-firmware.

The isenkram-autoinstall-firmware script fetches output from the kernel
log using dmesg, looking for lines containing "firmware: failed to
load", extracting the name of the file a kernel driver wanted to load.
This list of files are then looked up in a list of files and their
packages extracted from the Debian archive, using the same data set
searched by apt-file.

So to understand why isenkram-autoinstall-firmware want to install
raspi-firmware I recommend first running this command to locate the
drivers and their requested firmware files:

  sudo dmesg | grep "firmware: failed to load"

Next, for each requested file, run

  apt-file search {filename}

and see which package contain the firmware file in question.

To increase performance, and avoid a dependency on apt-file in
isenkram-cli, there is a cache of all firmware->package mappings in
/usr/share/isenkram/, this is extracted from the Debian archive when
isenkram is uploaded, and might be out of date if some firmware package
changed since the upload.  You might also want to look for the firmware
file listed there, if apt-file search turn up nothing.

-- 
Happy hacking
Petter Reinholdtsen

---

From: Petter Reinholdtsen <[email-masked]>
Date: Mon, 07 Oct 2024 07:59:10 +0200

Just for the record, it would be useful to have the output from the
problematic isenkram-autoinstall-firmware run.

Also, your email provider reject emails to [email-masked], claiming
"mailbox unavailable", so I assume my replies will not reach you.

-- 
Happy hacking
Petter Reinholdtsen

---

From: Chris Hofstaedtler <[email-masked]>
Date: Mon, 7 Oct 2024 18:06:03 +0200

> > installs raspi-firmware. This is unexpected.
[..]
>   apt-file search {filename}
[..]

Lenovo's support page for one version of the ThinkPad 10 [1] claims
that a BCM4356 wireless is part of the hardware.

I imagine the kernel driver will request one of the
brcmfmac43456-sdio* files as included in raspi-firmware, but
also present in firmware-brcm80211.

Chris

[1] https://pcsupport.lenovo.com/us/en/products/tablets/thinkpad-tablet-series/thinkpad-10-20e3-20e4/downloads/driver-list/component?name=Networking%3A%20Wireless%20LAN&id=E3519D23-890E-4DE1-9064-DE6E7DA2515B

---

From: Petter Reinholdtsen <[email-masked]>
Date: Mon, 07 Oct 2024 22:59:54 +0200

Thank you for the tip.  It definitely seem relevant to this problem.

[Chris Hofstaedtler]
> also present in firmware-brcm80211.

As far as I can tell, and <URL: https://bugs.debian.org/999485 > seem to
indicate the same, these files are not yet present in
firmware-brcm80211.

-- 
Happy hacking
Petter Reinholdtsen

---

From: [email-masked]
Date: Tue, 8 Oct 2024 01:52:31 +0200

Good evening,

sorry for my badly behaving freemail service, I think that should be fixed now (for a while).

On Mon, 07 Oct 2024 22:59:54 +0200 Petter Reinholdtsen <[email-masked]> wrote:

I did a
$ sudo dmesg | grep "firmware: failed to load" | cut -d\/ -f2 | cut -d\. -f1 | sort | uniq
BCM4324B5
brcmfmac43241b5-sdio

and then
$ apt-file search BCM4324B5
$ apt-file search brcmfmac43241b5-sdio
firmware-brcm80211: /lib/firmware/brcm/brcmfmac43241b5-sdio.bin

but that seems to be installed already:
# dpkg -l firmware-brcm80211
ii  firmware-brcm80211 20230210-5   all          Binary firmware for Broadcom/Cypress 802.11 wireless cards

> indicate the same, these files are not yet present in

This looks promising, so I had a look at /usr/share/isenkram as you told me:

# grep -r BCM4324B5 /usr/share/isenkram/

# grep -r brcmfmac43241b5 /usr/share/isenkram/
/usr/share/isenkram/Fw-Contents-all-bullseye-non-free:lib/firmware/brcm/brcmfmac43241b5-sdio.bin              non-free/kernel/firmware-brcm80211
/usr/share/isenkram/Fw-Contents-powerpc-sid-non-free:lib/firmware/brcm/brcmfmac43241b5-sdio.bin              non-free/kernel/firmware-brcm80211
/usr/share/isenkram/Fw-Contents-kfreebsd-amd64-sid-non-free:lib/firmware/brcm/brcmfmac43241b5-sdio.bin              non-free/kernel/firmware-brcm80211
/usr/share/isenkram/Fw-Contents-hurd-i386-sid-non-free:lib/firmware/brcm/brcmfmac43241b5-sdio.bin              non-free/kernel/firmware-brcm80211
/usr/share/isenkram/Fw-Contents-all-bookworm-non-free-firmware:lib/firmware/brcm/brcmfmac43241b5-sdio.bin              non-free-firmware/kernel/firmware-brcm80211
/usr/share/isenkram/Fw-Contents-kfreebsd-i386-sid-non-free:lib/firmware/brcm/brcmfmac43241b5-sdio.bin              non-free/kernel/firmware-brcm80211
/usr/share/isenkram/Fw-Contents-all-sid-non-free-firmware:lib/firmware/brcm/brcmfmac43241b5-sdio.bin              non-free-firmware/kernel/firmware-brcm80211
/usr/share/isenkram/Fw-Contents-mips-sid-non-free:lib/firmware/brcm/brcmfmac43241b5-sdio.bin              non-free/kernel/firmware-brcm80211

# grep -r raspi-firmware /usr/share/isenkram/
/usr/share/isenkram/Fw-Contents-arm64-bullseye-non-free:lib/firmware/brcm/brcmfmac43430-sdio.txt                non-free/misc/raspi-firmware
/usr/share/isenkram/Fw-Contents-arm64-bullseye-non-free:lib/firmware/brcm/brcmfmac43455-sdio.txt                non-free/misc/raspi-firmware
/usr/share/isenkram/Fw-Contents-arm64-bullseye-non-free:lib/firmware/brcm80211                                  non-free/misc/raspi-firmware
/usr/share/isenkram/Fw-Contents-all-bookworm-non-free-firmware:lib/firmware/brcm/brcmfmac43430-sdio.txt                non-free-firmware/misc/raspi-firmware
/usr/share/isenkram/Fw-Contents-all-bookworm-non-free-firmware:lib/firmware/brcm/brcmfmac43455-sdio.txt                non-free-firmware/misc/raspi-firmware
/usr/share/isenkram/Fw-Contents-all-bookworm-non-free-firmware:lib/firmware/brcm/brcmfmac43456-sdio.bin                non-free-firmware/misc/raspi-firmware
/usr/share/isenkram/Fw-Contents-all-bookworm-non-free-firmware:lib/firmware/brcm/brcmfmac43456-sdio.clm_blob           non-free-firmware/misc/raspi-firmware
/usr/share/isenkram/Fw-Contents-all-bookworm-non-free-firmware:lib/firmware/brcm/brcmfmac43456-sdio.txt                non-free-firmware/misc/raspi-firmware
/usr/share/isenkram/Fw-Contents-all-sid-non-free-firmware:lib/firmware/brcm/brcmfmac43430-sdio.txt                non-free-firmware/misc/raspi-firmware
/usr/share/isenkram/Fw-Contents-all-sid-non-free-firmware:lib/firmware/brcm/brcmfmac43455-sdio.txt                non-free-firmware/misc/raspi-firmware
/usr/share/isenkram/Fw-Contents-all-sid-non-free-firmware:lib/firmware/brcm/brcmfmac43456-sdio.bin                non-free-firmware/misc/raspi-firmware
/usr/share/isenkram/Fw-Contents-all-sid-non-free-firmware:lib/firmware/brcm/brcmfmac43456-sdio.clm_blob           non-free-firmware/misc/raspi-firmware
/usr/share/isenkram/Fw-Contents-all-sid-non-free-firmware:lib/firmware/brcm/brcmfmac43456-sdio.txt                non-free-firmware/misc/raspi-firmware
/usr/share/isenkram/Fw-Contents-armel-bullseye-non-free:lib/firmware/brcm/brcmfmac43430-sdio.txt                non-free/misc/raspi-firmware
/usr/share/isenkram/Fw-Contents-armel-bullseye-non-free:lib/firmware/brcm/brcmfmac43455-sdio.txt                non-free/misc/raspi-firmware
/usr/share/isenkram/Fw-Contents-armel-bullseye-non-free:lib/firmware/brcm80211                                  non-free/misc/raspi-firmware
/usr/share/isenkram/Fw-Contents-armhf-bullseye-non-free:lib/firmware/brcm/brcmfmac43430-sdio.txt                non-free/misc/raspi-firmware
/usr/share/isenkram/Fw-Contents-armhf-bullseye-non-free:lib/firmware/brcm/brcmfmac43455-sdio.txt                non-free/misc/raspi-firmware
/usr/share/isenkram/Fw-Contents-armhf-bullseye-non-free:lib/firmware/brcm80211                                  non-free/misc/raspi-firmware

So I tried to provide you with the output of isenkram-autoinstall-firmware.
Beside an 'unexpected operator' error in /usr/sbin/isenkram-autoinstall-firmware I found something different, but very interesting:

There is a difference when choosing LANG=C:
# LANG=C isenkram-autoinstall-firmware
it says: info: No new firmware package with requested firmware detected.

and without setting LANG (that means LANG=de_DE.UTF-8 in my case) there's raspi-firmware again.
See the complete output at: https://paste.debian.net/1331619

And there is another thing, I think I shoud mention:
I recently updated the system from bullseye to bookworm.

Thanks a lot
Christian

---

From: Petter Reinholdtsen <[email-masked]>
Date: Tue, 08 Oct 2024 16:05:01 +0200

> See the complete output at: https://paste.debian.net/1331619

To have it available also when the pastebin expire, I include the
content here:

# LANG=C isenkram-autoinstall-firmware
/usr/sbin/isenkram-autoinstall-firmware: 63: [: /lib/firmware/brcm/brcmfmac4330-sdio.Prowise-PT301.txt: unexpected operator
info: looking for firmware file brcm/brcmfmac43241b5-sdio.LENOVO-20C3S0HJ00.bin requested by kernel
info: looking for firmware file brcm/brcmfmac43241b5-sdio.LENOVO-20C3S0HJ00.bin requested by kernel
info: looking for firmware file brcm/brcmfmac43241b5-sdio.LENOVO-20C3S0HJ00.txt requested by kernel
info: looking for firmware file brcm/brcmfmac43241b5-sdio.LENOVO-20C3S0HJ00.txt requested by kernel
info: looking for firmware file brcm/brcmfmac43241b5-sdio.txt requested by kernel
info: looking for firmware file brcm/brcmfmac43241b5-sdio.txt requested by kernel
info: looking for firmware file brcm/brcmfmac43241b5-sdio.LENOVO-20C3S0HJ00.clm_blob requested by kernel
info: looking for firmware file brcm/brcmfmac43241b5-sdio.LENOVO-20C3S0HJ00.clm_blob requested by kernel
info: looking for firmware file brcm/brcmfmac43241b5-sdio.clm_blob requested by kernel
info: looking for firmware file brcm/brcmfmac43241b5-sdio.clm_blob requested by kernel
info: looking for firmware file brcm/BCM4324B5.hcd requested by kernel
info: looking for firmware file brcm/BCM4324B5.hcd requested by kernel
info: some kernel driver requested extra firmware files: brcm/brcmfmac*-pcie.*.clm_blob brcm/brcmfmac*-pcie.txt brcm/brcmfmac*-sdio.*.bin brcm/brcmfmac43430b0-sdio.bin brcm/brcmfmac43439-sdio.bin brcm/brcmfmac43439-sdio.clm_blob brcm/brcmfmac43456-sdio.bin brcm/brcmfmac4359-pcie.bin brcm/brcmfmac4359-sdio.bin brcm/brcmfmac4364-pcie.bin brcm/brcmfmac4365b-pcie.bin brcm/brcmfmac4365c-pcie.bin brcm/brcmfmac43752-sdio.bin brcm/brcmfmac43752-sdio.clm_blob brcm/brcmfmac4378b1-pcie.bin brcm/brcmfmac4378b1-pcie.clm_blob brcm/brcmfmac89459-pcie.bin rtl_bt/rtl8723b_config.bin rtl_bt/rtl8723bs_config.bin rtl_bt/rtl8723cs_cg_config.bin rtl_bt/rtl8723cs_cg_fw.bin rtl_bt/rtl8723cs_vf_config.bin rtl_bt/rtl8723cs_vf_fw.bin rtl_bt/rtl8723ds_config.bin rtl_bt/rtl8723ds_fw.bin rtl_bt/rtl8761a_config.bin rtl_bt/rtl8821cs_config.bin rtl_bt/rtl8821cs_fw.bin rtl_bt/rtl8851bu_config.bin rtl_bt/rtl8851bu_fw.bin brcm/brcmfmac43241b5-sdio.LENOVO-20C3S0HJ00.bin brcm/brcmfmac43241b5-sdio.LENOVO-20C3S0HJ00.bin brcm/brcmfmac43241b5-sdio.LENOVO-20C3S0HJ00.txt brcm/brcmfmac43241b5-sdio.LENOVO-20C3S0HJ00.txt brcm/brcmfmac43241b5-sdio.txt brcm/brcmfmac43241b5-sdio.txt brcm/brcmfmac43241b5-sdio.LENOVO-20C3S0HJ00.clm_blob brcm/brcmfmac43241b5-sdio.LENOVO-20C3S0HJ00.clm_blob brcm/brcmfmac43241b5-sdio.clm_blob brcm/brcmfmac43241b5-sdio.clm_blob brcm/BCM4324B5.hcd brcm/BCM4324B5.hcd
info: locating packages with the requested firmware files
info: determining whether enabling other components is required
info: No new firmware package with requested firmware detected.

# isenkram-autoinstall-firmware
/usr/sbin/isenkram-autoinstall-firmware: 63: [: /lib/firmware/brcm/brcmfmac4330-sdio.Prowise-PT301.txt: unexpected operator
info: looking for firmware file brcm/brcmfmac43241b5-sdio.LENOVO-20C3S0HJ00.bin requested by kernel
info: looking for firmware file brcm/brcmfmac43241b5-sdio.LENOVO-20C3S0HJ00.bin requested by kernel
info: looking for firmware file brcm/brcmfmac43241b5-sdio.LENOVO-20C3S0HJ00.txt requested by kernel
info: looking for firmware file brcm/brcmfmac43241b5-sdio.LENOVO-20C3S0HJ00.txt requested by kernel
info: looking for firmware file brcm/brcmfmac43241b5-sdio.txt requested by kernel
info: looking for firmware file brcm/brcmfmac43241b5-sdio.txt requested by kernel
info: looking for firmware file brcm/brcmfmac43241b5-sdio.LENOVO-20C3S0HJ00.clm_blob requested by kernel
info: looking for firmware file brcm/brcmfmac43241b5-sdio.LENOVO-20C3S0HJ00.clm_blob requested by kernel
info: looking for firmware file brcm/brcmfmac43241b5-sdio.clm_blob requested by kernel
info: looking for firmware file brcm/brcmfmac43241b5-sdio.clm_blob requested by kernel
info: looking for firmware file brcm/BCM4324B5.hcd requested by kernel
info: looking for firmware file brcm/BCM4324B5.hcd requested by kernel
info: some kernel driver requested extra firmware files: brcm/brcmfmac43430b0-sdio.bin brcm/brcmfmac43439-sdio.bin brcm/brcmfmac43439-sdio.clm_blob brcm/brcmfmac43456-sdio.bin brcm/brcmfmac4359-pcie.bin brcm/brcmfmac4359-sdio.bin brcm/brcmfmac4364-pcie.bin brcm/brcmfmac4365b-pcie.bin brcm/brcmfmac4365c-pcie.bin brcm/brcmfmac43752-sdio.bin brcm/brcmfmac43752-sdio.clm_blob brcm/brcmfmac4378b1-pcie.bin brcm/brcmfmac4378b1-pcie.clm_blob brcm/brcmfmac89459-pcie.bin brcm/brcmfmac*-pcie.*.clm_blob brcm/brcmfmac*-pcie.txt brcm/brcmfmac*-sdio.*.bin rtl_bt/rtl8723b_config.bin rtl_bt/rtl8723bs_config.bin rtl_bt/rtl8723cs_cg_config.bin rtl_bt/rtl8723cs_cg_fw.bin rtl_bt/rtl8723cs_vf_config.bin rtl_bt/rtl8723cs_vf_fw.bin rtl_bt/rtl8723ds_config.bin rtl_bt/rtl8723ds_fw.bin rtl_bt/rtl8761a_config.bin rtl_bt/rtl8821cs_config.bin rtl_bt/rtl8821cs_fw.bin rtl_bt/rtl8851bu_config.bin rtl_bt/rtl8851bu_fw.bin brcm/brcmfmac43241b5-sdio.LENOVO-20C3S0HJ00.bin brcm/brcmfmac43241b5-sdio.LENOVO-20C3S0HJ00.bin brcm/brcmfmac43241b5-sdio.LENOVO-20C3S0HJ00.txt brcm/brcmfmac43241b5-sdio.LENOVO-20C3S0HJ00.txt brcm/brcmfmac43241b5-sdio.txt brcm/brcmfmac43241b5-sdio.txt brcm/brcmfmac43241b5-sdio.LENOVO-20C3S0HJ00.clm_blob brcm/brcmfmac43241b5-sdio.LENOVO-20C3S0HJ00.clm_blob brcm/brcmfmac43241b5-sdio.clm_blob brcm/brcmfmac43241b5-sdio.clm_blob brcm/BCM4324B5.hcd brcm/BCM4324B5.hcd
info: locating packages with the requested firmware files
info: determining whether enabling other components is required
info: trying to install raspi-firmware
Vormals nicht ausgewähltes Paket raspi-firmware wird gewählt.
(Lese Datenbank ... 310874 Dateien und Verzeichnisse sind derzeit installiert.)
Vorbereitung zum Entpacken von .../raspi-firmware_1.20220830+ds-1_all.deb ...
Entpacken von raspi-firmware (1.20220830+ds-1) ...
raspi-firmware (1.20220830+ds-1) wird eingerichtet ...
Error: missing /boot/firmware, did you forget to mount it?
dpkg: Fehler beim Bearbeiten des Paketes raspi-firmware (--configure):
 »installiertes post-installation-Skript des Paketes raspi-firmware«-Unterprozess gab den Fehlerwert 1 zurück
Trigger für initramfs-tools (0.142+deb12u1) werden verarbeitet ...
update-initramfs: Generating /boot/initrd.img-6.1.0-26-686
I: The initramfs will attempt to resume from /dev/mmcblk2p5
I: (UUID=e0f5e93c-2c1e-43aa-93e6-0bd08f22b05e)
I: Set the RESUME variable to override this.
raspi-firmware: missing /boot/firmware, did you forget to mount it?
run-parts: /etc/initramfs/post-update.d//z50-raspi-firmware exited with return code 1
dpkg: Fehler beim Bearbeiten des Paketes initramfs-tools (--configure):
 »installiertes post-installation-Skript des Paketes initramfs-tools«-Unterprozess gab den Fehlerwert 1 zurück
Fehler traten auf beim Bearbeiten von:
 raspi-firmware
 initramfs-tools
E: Sub-process /usr/bin/dpkg returned an error code (1)

-- 
Happy hacking
Petter Reinholdtsen

---

From: Petter Reinholdtsen <[email-masked]>
Date: Tue, 08 Oct 2024 23:48:25 +0200

I believe part of the locale related problem is caused by wildcard paths
in some kernel module, and that the following patch should get rid of
that particular problem:

diff --git a/isenkram-autoinstall-firmware b/isenkram-autoinstall-firmware
index b3152e7..b30ca86 100755
--- a/isenkram-autoinstall-firmware
+++ b/isenkram-autoinstall-firmware
@@ -59,8 +59,8 @@ appstreamlookup() {
 }
 
 # Find firmware files requested by loaded kernel drivers.
-for fwfile in $(for module in $(awk '{print $1}' /proc/modules) ; do modinfo $module 2>/dev/null |awk '/^firmware:/ {print $2}'; done|sort -u); do
-    if [ ! -e /lib/firmware/$fwfile ] ; then
+for fwfile in $(for module in $(awk '{print $1}' /proc/modules) ; do modinfo $module 2>/dev/null |awk '/^firmware:/ {print $2}'; done| LC_ALL=C sort -u); do
+    if [ ! -e "/lib/firmware/$fwfile" ] ; then
        fwfiles="${fwfiles:+$fwfiles }$fwfile"
     fi
 done
@@ -70,7 +70,7 @@ done
 if dmesg >/dev/null 2>&1 ; then
     for fwfile in $(dmesg | sed -rn 's/^(\[[^]]*\] )?([^ ]+) [^ ]+: firmware: failed to load ([^ ]+) .*/\3/p'); do
        loginfo "looking for firmware file $fwfile requested by kernel"
-       if [ ! -e /lib/firmware/$fwfile ] ; then
+       if [ ! -e "/lib/firmware/$fwfile" ] ; then
             fwfiles="${fwfiles:+$fwfiles }$fwfile"
        fi
     done
@@ -145,7 +145,7 @@ done
 # Many different files can be required for a single hardware device,
 # plus several devices might require files from the same package, so
 # optimize!
-binpkginfos=$(echo "$binpkginfos" | tr ' ' '\n' | sort -u | xargs)
+binpkginfos=$(echo "$binpkginfos" | tr ' ' '\n' | LC_ALL=C sort -u | xargs)
 
 loginfo "determining whether enabling other components is required"
 binpkgs="${binpkgs:+$binpkgs }$(

Perhaps you can test it and see if the 'unexpected operator' message go
away, and the script behave the same for both LANG=C and
LANG=de_DE.UTF-8?  I suspect it will still try to install the
raspi-firmware package, though.

-- 
Happy hacking
Petter Reinholdtsen

---

From: Christian Meyer <[email-masked]>
Date: Thu, 10 Oct 2024 23:38:22 +0200

Am Dienstag, dem 08.10.2024 um 23:48 +0200 schrieb Petter Reinholdtsen:
> 
> I believe part of the locale related problem is caused by wildcard
> paths in some kernel module, and that the following patch should get
> rid of that particular problem:
> ...
> Perhaps you can test it and see if the 'unexpected operator' message
> go away, and the script behave the same for both LANG=C and
> LANG=de_DE.UTF-8?

Thank you, that was a good starting point.

1. I confirm that the 'unexpected operator' message went away, both for
LANG=C and LANG=de_DE.UTF-8. 

2. The different behaviour for different LANG's is unchanged.

3. I did some investigation and found that:
a) for LANG=de_DE.UTF-8:
fwfile: brcm/brcmfmac43241b5-sdio.clm_blob matches binpkginfo:
say: [none]
and
b) for LANG=C:
fwfile: brcm/brcmfmac43241b5-sdio.clm_blob matches binpkginfo:
firmware-realtek
firmware-misc-nonfree
firmware-netxen
firmware-samsung
firmware-bnx2x
firmware-siano
firmware-amd-graphics
firmware-ivtv
firmware-bnx2
firmware-netronome
firmware-libertas
firmware-linux-free
firmware-ti-connectivity
firmware-brcm80211
firmware-intel-sound
firmware-qlogic
firmware-qcom-soc
firmware-cavium
firmware-atheros
firmware-myricom
firmware-iwlwifi
firmware-ipw2x00
firmware-ath9k-htc

So the thing is:
appstreamcli has differnt outputs depending on LANG, and while you look
for lines with the word '/Package:/' with german LANG the same line
starts with the word "Paket".

I still was not able to figure out, why isenkramcli 'wants' to install
raspi-firmware and what happens with a second call of isenkram-
autoinstall-firmware, after raspi-firmware is installed successfully.

4. But I just added a LANG=C to appstreamlookup() and now it works
fine.

appstreamlookup() {
    fwfile="$1"
    LANG=C appstreamcli what-provides firmware:runtime "$fwfile" | \
        awk '/Package:/ { print $2}'
}

Thank you for your hint
Christian Meyer

---

From: Petter Reinholdtsen <[email-masked]>
Date: Fri, 11 Oct 2024 00:14:13 +0200

[Christian Meyer]
> starts with the word "Paket".

Aha, I did not know.  Good catch.

> autoinstall-firmware, after raspi-firmware is installed successfully.

As far as I can tell, isenkram-autoinstall-firmware want to install
raspi-firmware because the raspi-firmware package contain a firmware
file one of the drivers loaded on the Lenovo machine want.  See
<URL: https://bugs.debian.org/999485 > for a request to move the file
into a non-rpi related package.

Here is a new and adjusted patch, making sure appstreamcli uses the C
locale and trying to handle the wildcard firmware entries at least one
of your machines driver kernel module mention.  I am not sure if this
last change is a useful approach, but thought it best to give it a go.

diff --git a/isenkram-autoinstall-firmware b/isenkram-autoinstall-firmware
index b3152e7..5d55ff9 100755
--- a/isenkram-autoinstall-firmware
+++ b/isenkram-autoinstall-firmware
@@ -54,13 +54,13 @@ EOF
 
 appstreamlookup() {
     fwfile="$1"
-    appstreamcli what-provides firmware:runtime "$fwfile" | \
+    LC_ALL=C appstreamcli what-provides firmware:runtime "$fwfile" | \
        awk '/Package:/ { print $2}'
 }

-- 
Happy hacking
Petter Reinholdtsen

---

From: Christian Meyer <[email-masked]>
Date: Mon, 14 Oct 2024 15:53:46 +0200

Am Freitag, dem 11.10.2024 um 00:14 +0200 schrieb Petter Reinholdtsen:
> diff --git a/isenkram-autoinstall-firmware

Sorry, I don't know, how to apply your patch correctly and there has
been errors:

# patch --dry-run /usr/sbin/isenkram-autoinstall-firmware
./isenkram.patch 
checking file /usr/sbin/isenkram-autoinstall-firmware
Hunk #1 FAILED at 54.
Hunk #2 FAILED at 70.
Hunk #3 succeeded at 163 (offset 18 lines).
2 out of 3 hunks FAILED

Anyway:
I applied your changes by hand and I can confirm both problems are
gone:

# /usr/sbin/isenkram-autoinstall-firmware
info: looking for firmware file brcm/brcmfmac43241b5-sdio.LENOVO-
20C3S0HJ00.bin requested by kernel
info: looking for firmware file brcm/brcmfmac43241b5-sdio.LENOVO-
20C3S0HJ00.bin requested by kernel
info: looking for firmware file brcm/brcmfmac43241b5-sdio.LENOVO-
20C3S0HJ00.txt requested by kernel
info: looking for firmware file brcm/brcmfmac43241b5-sdio.LENOVO-
20C3S0HJ00.txt requested by kernel
info: looking for firmware file brcm/brcmfmac43241b5-sdio.txt requested
by kernel
info: looking for firmware file brcm/brcmfmac43241b5-sdio.txt requested
by kernel
info: looking for firmware file brcm/brcmfmac43241b5-sdio.LENOVO-
20C3S0HJ00.clm_blob requested by kernel
info: looking for firmware file brcm/brcmfmac43241b5-sdio.LENOVO-
20C3S0HJ00.clm_blob requested by kernel
info: looking for firmware file brcm/brcmfmac43241b5-sdio.clm_blob
requested by kernel
info: looking for firmware file brcm/brcmfmac43241b5-sdio.clm_blob
requested by kernel
info: looking for firmware file brcm/BCM4324B5.hcd requested by kernel
info: looking for firmware file brcm/BCM4324B5.hcd requested by kernel
info: some kernel driver requested extra firmware files:
brcm/brcmfmac*-pcie.*.bin brcm/brcmfmac*-pcie.*.clm_blob
brcm/brcmfmac*-pcie.*.txt brcm/brcmfmac*-pcie.txt brcm/brcmfmac*-
sdio.*.bin brcm/brcmfmac*-sdio.*.txt brcm/brcmfmac43430b0-sdio.bin
brcm/brcmfmac43439-sdio.bin brcm/brcmfmac43439-sdio.clm_blob
brcm/brcmfmac43456-sdio.bin brcm/brcmfmac4359-pcie.bin
brcm/brcmfmac4359-sdio.bin brcm/brcmfmac4364-pcie.bin
brcm/brcmfmac4365b-pcie.bin brcm/brcmfmac4365c-pcie.bin
brcm/brcmfmac43752-sdio.bin brcm/brcmfmac43752-sdio.clm_blob
brcm/brcmfmac4378b1-pcie.bin brcm/brcmfmac4378b1-pcie.clm_blob
brcm/brcmfmac89459-pcie.bin rtl_bt/rtl8723b_config.bin
rtl_bt/rtl8723bs_config.bin rtl_bt/rtl8723cs_cg_config.bin
rtl_bt/rtl8723cs_cg_fw.bin rtl_bt/rtl8723cs_vf_config.bin
rtl_bt/rtl8723cs_vf_fw.bin rtl_bt/rtl8723ds_config.bin
rtl_bt/rtl8723ds_fw.bin rtl_bt/rtl8761a_config.bin
rtl_bt/rtl8821cs_config.bin rtl_bt/rtl8821cs_fw.bin
rtl_bt/rtl8851bu_config.bin rtl_bt/rtl8851bu_fw.bin
brcm/brcmfmac43241b5-sdio.LENOVO-20C3S0HJ00.bin brcm/brcmfmac43241b5-
sdio.LENOVO-20C3S0HJ00.bin brcm/brcmfmac43241b5-sdio.LENOVO-
20C3S0HJ00.txt brcm/brcmfmac43241b5-sdio.LENOVO-20C3S0HJ00.txt
brcm/brcmfmac43241b5-sdio.txt brcm/brcmfmac43241b5-sdio.txt
brcm/brcmfmac43241b5-sdio.LENOVO-20C3S0HJ00.clm_blob
brcm/brcmfmac43241b5-sdio.LENOVO-20C3S0HJ00.clm_blob
brcm/brcmfmac43241b5-sdio.clm_blob brcm/brcmfmac43241b5-sdio.clm_blob
brcm/BCM4324B5.hcd brcm/BCM4324B5.hcd
info: locating packages with the requested firmware files
info: determining whether enabling other components is required
info: No new firmware package with requested firmware detected.

So since your syntax 'diff --git /usr/sbin/isenkram-autoinstall-
firmware.*' give an error on my system (unknown option »--git«), I did
a diff for myself to show you that I followed your changes
Some line numbers are different, but I think thats the difference
between debian stable and git development:

# diff /usr/sbin/isenkram-autoinstall-firmware.*
57c57
<     appstreamcli what-provides firmware:runtime "$fwfile" | \
---
>     LC_ALL=C appstreamcli what-provides firmware:runtime "$fwfile" |
\
62,63c62,63
< for fwfile in $(for module in $(awk '{print $1}' /proc/modules) ; do
modinfo $module 2>/dev/null |awk '/^firmware:/ {print $2}'; done|sort -
u); do
<     if [ ! -e /lib/firmware/$fwfile ] ; then
---
> for fwfile in $(for module in $(awk '{print $1}' /proc/modules) ; do
modinfo $module 2>/dev/null |awk '/^firmware:/ {print $2}'; done|
LC_ALL=C sort -u); do
>     if [ ! -e "/lib/firmware/$fwfile" ] ; then
73c73
< 	if [ ! -e /lib/firmware/$fwfile ] ; then
---
> 	if [ ! -e "/lib/firmware/$fwfile" ] ; then
166c166
< binpkginfos=$(echo "$binpkginfos" | tr ' ' '\n' | sort -u | xargs)
---
> binpkginfos=$(echo "$binpkginfos" | tr ' ' '\n' | LC_ALL=C sort -u |
xargs)

Thank you,
Christian

---

From: Petter Reinholdtsen <[email-masked]>
Date: Sun, 01 Dec 2024 23:25:47 +0100

[Christian Meyer]
> Sorry, I don't know, how to apply your patch correctly and there has

Ah, sorry about that.  It was related to my git repo, probably not
identical to the version you had.

> I applied your changes by hand and I can confirm both problems are

Good.

I've just uploaded a improved isenkram-autoinstall-firmware enforcing
the use of LC_ALL=C and quotes around firmware file names.  I did not
include the change below, as I do not quite understand what wildcard
firmware file entries are supposed to mean in kernel modules, and
decided to delay its inclusion until I do.

diff --git a/isenkram-autoinstall-firmware b/isenkram-autoinstall-firmware
index dd9df64..8bcd156 100755
--- a/isenkram-autoinstall-firmware
+++ b/isenkram-autoinstall-firmware
@@ -58,11 +58,14 @@ appstreamlookup() {
        awk '/Package:/ { print $2}'
 }
 
-# Find firmware files requested by loaded kernel drivers.
-for fwfile in $(for module in $(awk '{print $1}' /proc/modules) ; do modinfo $module 2>/dev/null |awk '/^firmware:/ {print $2}'; done | LC_ALL=C sort -u); do
-    if [ ! -e "/lib/firmware/$fwfile" ] ; then
-       fwfiles="${fwfiles:+$fwfiles }$fwfile"
-    fi
+# Find firmware files requested by loaded kernel drivers.  Some entries use
+# wildcard strings.
+for fwwildcard in $(for module in $(awk '{print $1}' /proc/modules) ; do modinfo $module 2>/dev/null |awk '/^firmware:/ {print $2}'; done | LC_ALL=C sort -u); do
+    for fwfile in $fwwildcard; do
+        if [ ! -e "/lib/firmware/$fwfile" ] ; then
+            fwfiles="${fwfiles:+$fwfiles }$fwfile"
+        fi
+    done
 done
 
 # Also look in dmesg for requested firmware for modules that refuse to

I do expect the uploaded change to fix the error messages you saw.  The
proposal of a rpi related package for your x86 machine need to be fixed
outside isenkram, as mentioned earlier.

-- 
Happy hacking
Petter Reinholdtsen

---

From: Petter Reinholdtsen <[email-masked]>
Date: Tue, 07 Jan 2025 12:19:22 +0100

[Cyril Brulebois]
> are you aware of something that changed in that regard?

Well, it has become more and more obvious that NEW processing is not
something to reliably wait for, as the pile of packages waiting in NEW
is slowly increasing and even fairly trivial and obviously free software
licensed packages has been waiting for a review for months.

Just for completeness sake, there seem to be a cluster of three issues
https://bugs.debian.org/999485, https://bugs.debian.org/1084224 and
https://bugs.debian.org/1064072, related to this firmware blob.  Perhaps
these should be merged and placed under one package, with affects to the
others?  Not sure where it best belong.

-- 
Happy hacking
Petter Reinholdtsen

---

From: Cyril Brulebois <[email-masked]>
Date: Tue, 7 Jan 2025 12:41:28 +0100

> licensed packages has been waiting for a review for months.

That does not match my experience, at all. And that's irrelevant to
implementing the best technical/legal solution for an issue that's not
exactly new (no pun intended).
-- 
Cyril Brulebois -- Debian Consultant @ DEBAMAX -- https://debamax.com/

---

From: Petter Reinholdtsen <[email-masked]>
Date: Tue, 07 Jan 2025 13:03:47 +0100

[Cyril Brulebois]
> That does not match my experience, at all.

Fortuntately we have measurements and graphs on
<URL: https://ftp-master.debian.org/stat.html > documenting the status,
so we do not need to depend on personal experience. :)

> And that's irrelevant to implementing the best technical/legal
> solution for an issue that's not exactly new (no pun intended).

Would be great if that was the case, but alas, there are several
suboptimal solutions implemented in Debian to avoid waiting for NEW
processing.  I too wish that would not be the case, but pragmatic
solutions this month beat waiting indefinitely for the perfect one.

-- 
Happy hacking
Petter Reinholdtsen

---

From: Cyril Brulebois <[email-masked]>
Date: Tue, 7 Jan 2025 13:59:27 +0100

> so we do not need to depend on personal experience. :)

Stats are just that, stats. They don't dictate what happens to a
specific package.

> solutions this month beat waiting indefinitely for the perfect one.

Surely the pragmatic solution would start with a package reaching NEW in
the first place, instead of badmouthing, complaining, and handwaving.

YMMV.