Debian Bug report logs - #1012649
software-properties-common: add-apt-repository fails: no valid OpenPGP data (with diagnosis and solution)

Reply or subscribe to this bug.

Display info messages

Message #5 received at [email protected] (full text, mbox, reply):

From: Ross Boylan <[email protected]>

To: Debian Bug Tracking System <[email protected]>

Subject: software-properties-common: add-apt-repository fails: no valid OpenPGP data (with diagnosis and solution)

Date: Fri, 10 Jun 2022 22:03:19 -0700

Package: software-properties-common
Version: 0.96.20.2-2.1
Severity: important
X-Debbugs-Cc: [email protected]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

This is a bit involved; it took several days of debugging to get to
the bottom of it.  After discussing severity I'll describe the initial
symptoms, then present the results of my analysis, and finally present
a possible fix.

**Summary**

When adding a key, add-apt-repository creates a number of keys and
keyrings.  At the last step it tries to import the wrong key and
fails; it should import the key it exported.

**Severity and Scope**

The package provides a single command, add-apt-repository, and it is
unable to add a repository for a ppa with Ubuntu's keyserver.  I don't
see anything that would limit the impact to the one ppa I tried to
add, or even to ppa's, though it may depend on the format of the downloaded
key.

It seems quite likely this is behind the unresolved bug 843946 (from 2016).


**Initial Symptoms**

My original goal was to build dosemu2 from source.  It is available in
Ubuntu but not Debian.
https://github.com/dosemu2/dosemu2/wiki/Building-From-Source describes
the process, the first step of which is to run the ci_preq.sh script
(https://github.com/dosemu2/dosemu2/blob/devel/ci_prereq.sh).  The
first line of the script is `sudo add-apt-repository -y ppa:dosemu2/ppa`.

I executed that command with additional debugging in a root shell,
without running the rest of the script.

<terminal>
# date; add-apt-repository -m ppa:dosemu2/ppa
Fri 03 Jun 2022 11:44:22 AM PDT

 More info: https://launchpad.net/~dosemu2/+archive/ubuntu/ppa
Press [ENTER] to continue or ctrl-c to cancel adding it

START SOURCES.LIST:
## most entries omitted to save space
# deb-src http://ppa.launchpad.net/dosemu2/ppa/ubuntu kinetic main

END SOURCES.LIST

gpg: keybox '/tmp/tmpcjw8ab77/pubring.gpg' created
gpg: keyserver receive failed: Server indicated a failure
</terminal>

This failure is likely unrelated to the subject of this bug.  The
failing command was on gpg --import from keyserver.ubuntu.com.  That
import was only erratically successful.  Here's one that worked:

 gpg --verbose --debug-level 7 --no-default-keyring --no-options --homedir
/tmp/tmpc20sx87y --secret-keyring /tmp/tmpc20sx87y/secring.gpg --keyring
/tmp/tmpc20sx87y/pubring.gpg --keyserver hkp://keyserver.ubuntu.com:80/ --recv
6D9CD73B401A130336ED0A56EBE1B5DED2AD45D6

Finally we arrive at the relevant problem for this bug.  The python
scripts of software-properties-common (invoked by add-apt-repository)
execute

'apt-key --keyring /etc/apt/trusted.gpg.d/dosemu2_ubuntu_ppa.gpg add
/tmp/tmpc20sx87y/pubring.gpg' which executes a series of steps, the last of
which is

gpg --ignore-time-conflict --no-options --no-default-keyring
            --homedir /tmp/apt-key-gpghome.xVHD35b6q9 -v
            --check-trustdb --keyring
            /tmp/apt-key-gpghome.xVHD35b6q9/empty.gpg
            --keyring '/etc/apt/trusted.gpg.d/dosemu2_ubuntu_ppa.gpg' "-v
- --batch --import /tmp/tmppr9n1vry/pubring.gpg"

(The directory names don't match previous command because I'm mixing different
runs).

The result:
gpg: no valid OpenPGP data found.
gpg: Total number processed: 0

The error was repeatable, assuming the keyserver was responsive.  It often
wasn't.

**Analysis**

Earlier steps export the key, but the step just above (with gpg) does
not use the exported key.  This is likely the result of the fact that
apt-key is invoked with pubring.gpg rather than the exported key.  In
fact, it doesn't seem *anything* uses the exported key.  I presume it
was exported to get it into a format that was usable.  This would mean
the failure arises from attempting to import a key in the wrong
format.  E.g., 852473 refers to this, in turn referencing 844724.

I've put a fuller analysis of the program flow below.

**Solution**

I imported the previously exported key, export-keyring.gpg, instead of
pubring.gpg.  It worked.

<terminal>
# date; gpg -v --ignore-time-conflict --no-options --no-default-keyring
- --homedir /tmp/apt-key-gpghome.xwyHmcOcK3 --no-auto-check-trustdb --trust-model
always --keyring /etc/apt/trusted.gpg.d/dosemu2_ubuntu_ppa.gpg -v --batch
- --import /tmp/tmpc20sx87y/export-keyring.gpg
Wed 08 Jun 2022 05:34:04 PM PDT
# off=0 ctb=99 tag=6 hlen=3 plen=525
:public key packet:
        version 4, algo 1, created 1475850017, expires 0
        pkey[0]: [4096 bits]
        pkey[1]: [17 bits]
        keyid: EBE1B5DED2AD45D6
# off=528 ctb=b4 tag=13 hlen=2 plen=25
:user ID packet: "Launchpad PPA for Dosemu2"
# off=555 ctb=89 tag=2 hlen=3 plen=568
:signature packet: algo 1, keyid EBE1B5DED2AD45D6
        version 4, created 1475850017, md5len 0, sigclass 0x13
        digest algo 2, begin of digest aa 4b
        hashed subpkt 2 len 4 (sig created 2016-10-07)
        hashed subpkt 27 len 1 (key flags: 03)
        hashed subpkt 11 len 5 (pref-sym-algos: 9 8 7 3 2)
        hashed subpkt 21 len 5 (pref-hash-algos: 8 2 9 10 11)
        hashed subpkt 22 len 3 (pref-zip-algos: 2 3 1)
        hashed subpkt 30 len 1 (features: 01)
        hashed subpkt 23 len 1 (keyserver preferences: 80)
        subpkt 16 len 8 (issuer key ID EBE1B5DED2AD45D6)
        data: [4095 bits]
gpg: pub  rsa4096/EBE1B5DED2AD45D6 2016-10-07  Launchpad PPA for Dosemu2
gpg: writing to '/etc/apt/trusted.gpg.d/dosemu2_ubuntu_ppa.gpg'
gpg: key EBE1B5DED2AD45D6: public key "Launchpad PPA for Dosemu2" imported
gpg: no running gpg-agent - starting '/usr/bin/gpg-agent'
gpg: waiting for the agent to come up ... (5s)
gpg: connection to agent established
gpg: Total number processed: 1
gpg:               imported: 1
</terminal>

As further confirmation, aptitude stopped complaining about an untrusted
repository after
this step.

**Comments**

Although the one substitution solved my immediate problem, there may
be other places in the code, supporting other operations, that require
a similar change.

The unreliability of the ubuntu keyserver seems like a problem, but
it's distinct from this one.

The code is extremely difficult to debug, and the failures difficult
to understand.  Starting with add-apt-repository, one has python code,
switches in the output stream (I guess to assure proper locale, but
the python debugger seemed allergic to it), invocation of
subprocesses.  apt-key continues, invoking gpg and other programs, and
even writing temporary shell scripts that invoke other temporary shell
scripts (gpg.1.sh invokes gpg.0.sh, both written by apt-key).  You
also have to guess whether -debug1 or -debug2 will be more useful for apt-key
since you can't do both--a fact that is not documented.  So when there
is a failure it is completely unclear what command failed or for what
reason.  Effective error reporting would need to flow all the way down to the
scripts
written by apt-key, since they are the ones that perform the actual commands.
add-apt-repository's -m switch threatens/promises --massive-debug, but fell
well short of that.

apt-key is going away; this package should not use it, as already noted in
995895.

**Appendix: Control flow**

Here are some of my notes on the flow of control from
add-apt-repository on down, including capturing the key commands that
were executed.  Indentation is nesting level, except for the actual shell
commands executed which tend to be flush left.  file:NN means the function
is on line NN of file. There are undoubtedly many mistakes.

Sequence of commands for add-apt-repository:
from addkey_func() at SoftwareProperties.py:691
  add_key ppa.py:384
    AddPPASigningKey ppa.py:160
    add_ppa_signing_key ppa.py:224
      note the cleanup function is defined inside this one
      get_ppa_info
        retrieves info from URL, including the signing fingerprint
           tmp_keyring_dir = tempfile.mkdtemp() ppa.py:247 makes temp dir
           self._recv_key actually gets key ->ppa.py:165
             verify_keyid_is_v4 just an integrity check
             call subprocess ppa.py:176
                gpg --recv  gives sources list
                ValueError: underlying buffer has been detached
                but then I can continue?!
                here's the command
gpg --no-default-keyring --no-options --homedir /tmp/tmpc20sx87y --secret-
keyring /tmp/tmpc20sx87y/secring.gpg --keyring /tmp/tmpc20sx87y/pubring.gpg
- --keyserver hkp://keyserver.ubuntu.com:80/ --recv
6D9CD73B401A130336ED0A56EBE1B5DED2AD45D6
which prints
gpg: key EBE1B5DED2AD45D6: "Launchpad PPA for Dosemu2" not changed
gpg: Total number processed: 1
gpg:              unchanged: 1

onwards
            self._export_key() ppa.py:257 -> 185
gpg --no-default-keyring --no-options --homedir /tmp/tmpc20sx87y --keyring
/tmp/tmpc20sx87y/pubring.gpg --output /tmp/tmpc20sx87y/export-keyring.gpg
- --export 6D9CD73B401A130336ED0A56EBE1B5DED2AD45D6
            asked if I wanted to overwrite an existing file
            self._verify_fingerprint() -> 211
              self._get_fingerprints -> 196
gpg --no-default-keyring --no-options --homedir /tmp/tmpc20sx87y --keyring
/tmp/tmpc20sx87y/export-keyring.gpg --fingerprint --batch --with-colons
    why are we verifying  a fingerprint we just got??
                subprocess.check_output() on the previous command,
            whose output was
'tru::1:1654359237:0:3:1:5\npub:-:4096:1:EBE1B5DED2AD45D6:1475850017:::-:::scSC::::::23::0:\nfpr:::::::::6D9CD73B401A130336ED0A56EBE1B5DED2AD45D6:\nuid:-::::1475850017::0BB83C1AF9D4B9B057D465F9DEEA0D44E3CF1CC5::Launchpad
PPA for Dosemu2::::::::::0:\n'
              fingerprint is extracted and returned
            back to line 267 pf add_ppa_signing_key which issues
'apt-key --keyring /etc/apt/trusted.gpg.d/dosemu2_ubuntu_ppa.gpg add
/tmp/tmpc20sx87y/pubring.gpg'
 which is the step that failed.  It does
             prepare_gpg_home makes tmp dir, e.g.
+ mktemp --directory --tmpdir apt-key-gpghome.XXXXXXXXXX
+ GPGHOMEDIR=/tmp/apt-key-gpghome.xVHD35b6q9
+ chmod 700 /tmp/apt-key-gpghome.xVHD35b6q9
and then
+ gpg --ignore-time-conflict --no-options --no-default-keyring
            --homedir /tmp/apt-key-gpghome.xVHD35b6q9 -v
            --check-trustdb --keyring
            /tmp/apt-key-gpghome.xVHD35b6q9/empty.gpg  (written to
            gpg.0.sh and exec with additional args --batch --import)
            There is no argument to --import and the comment saying this is
just to force creation of some files.
              warn_on_script_usage (apt-key deprecated)
              setup_merged_keyring
                dearmor file name same as original since .gpg
                create_new_keyring
                  if it doesn't exist make next file and set mode
                  /etc/apt/trusted.gpg.d/dosemu2_ubuntu_ppa.gpg
                  then make gpg.1.sh:
gpg.0.sh  --keyring '/etc/apt/trusted.gpg.d/dosemu2_ubuntu_ppa.gpg' "$@"
                  Note gpg.1.sh is not executed before function returns.
                  But GPG is redefined to invoke it.
                aptkey_execute then does execute:

+ sh /tmp/apt-key-gpghome.xVHD35b6q9/gpg.1.sh -v --batch --import
/tmp/tmppr9n1vry/pubring.gpg
which means (directory names inconsistent from multiple runs)
+ gpg --ignore-time-conflict --no-options --no-default-keyring
            --homedir /tmp/apt-key-gpghome.xVHD35b6q9 -v
            --check-trustdb --keyring
            /tmp/apt-key-gpghome.xVHD35b6q9/empty.gpg
            --keyring '/etc/apt/trusted.gpg.d/dosemu2_ubuntu_ppa.gpg' "-v
- --batch --import /tmp/tmppr9n1vry/pubring.gpg"
            Why did it create export-keyring.gpg and then not use it?  Why are
there 2 keyrings?
gpg: no valid OpenPGP data found.
gpg: Total number processed: 0



- -- System Information:
Debian Release: 11.3
  APT prefers stable-security
  APT policy: (500, 'stable-security'), (500, 'stable')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 5.10.0-14-amd64 (SMP w/8 CPU threads)
Kernel taint flags: TAINT_PROPRIETARY_MODULE, TAINT_OOT_MODULE,
TAINT_UNSIGNED_MODULE
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE not
set
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages software-properties-common depends on:
ii  ca-certificates              20210119
ii  gir1.2-glib-2.0              1.66.1-1+b1
ii  gir1.2-packagekitglib-1.0    1.2.2-2
ii  python-apt-common            2.2.1
ii  python3                      3.9.2-3
ii  python3-dbus                 1.2.16-5
ii  python3-gi                   3.38.0-2
ii  python3-software-properties  0.96.20.2-2.1

Versions of packages software-properties-common recommends:
ii  packagekit  1.2.2-2

software-properties-common suggests no packages.


-----BEGIN PGP SIGNATURE-----

iQFSBAEBCgA8FiEEreS674/HIyV9gBfdnAYPmOsbK2AFAmKkIgIeHHJvc3Nib3ls
YW5Ac3RhbmZvcmRhbHVtbmkub3JnAAoJEJwGD5jrGytgjsUH/03lTIvjoVmKSE55
R6rKEVhb5gHy4pRkycwse0+VTWTjD5o/21Ox2FSfSnbd2hVafNdpLE97V8BzqhPw
xhIzNGd/6KdppOjoW1Lq6+z3A75JqsJeYA6JCIH4E9VZ69oCUJQZqOT49OSy0mAR
hvGyP+qxoiz0AKC6kzcdPEd/dv5Rvp+zlMMgiWXN9S9UJilt3bzi2SxKaEprSAbI
R0aiEFzF3p7u57BgGcMXICPsJGvUeMYalSk5ghyPpIOZrQ476g9DCbCwN9K9xlWO
E2hFLnNdgQuYmXWh+XxLJmfUBr8/pj/htjKu3RLLrilFmLnAt37M6gC1pXSikyAZ
Q52pGfg=
=aIEH
-----END PGP SIGNATURE-----

Message #10 received at [email protected] (full text, mbox, reply):

From: Ross Boylan <[email protected]>

To: [email protected]

Subject: Re: Bug#1012649: Acknowledgement (software-properties-common: add-apt-repository fails: no valid OpenPGP data (with diagnosis and solution))

Date: Wed, 15 Jun 2022 22:15:47 -0700

[Message part 1 (text/plain, inline)]

Tags: + patch
thanks

Here's the small code change to fix the problem.  I built and
installed the patched version locally, and did 2 successful
add-apt-repository's to add ppa's.  Without the patch, every  attempt
failed (though admittedly each of those attempts was with a single ppa
and sometimes the problem seemed to be the keyserver was down).

I found only a single invocation of apt-key in ppa.py, and so I don't
think there are any other spots that need changing.

BTW, I don't know if my pseudo-headers will do the trick; I'm trying
to add a patch tag to the bug.

On Fri, Jun 10, 2022 at 10:27 PM Debian Bug Tracking System
<[email protected]> wrote:
>
> Thank you for filing a new Bug report with Debian.
>
> You can follow progress on this Bug here: 1012649: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1012649.
>
> This is an automatically generated reply to let you know your message
> has been received.
>
> Your message is being forwarded to the package maintainers and other
> interested parties for their attention; they will reply in due course.
>
> As you requested using X-Debbugs-CC, your message was also forwarded to
>   [email protected]
> (after having been given a Bug report number, if it did not have one).
>
> Your message has been sent to the package maintainer(s):
>  Julian Andres Klode <[email protected]>
>
> If you wish to submit further information on this problem, please
> send it to [email protected].
>
> Please do not send mail to [email protected] unless you wish
> to report a problem with the Bug-tracking system.
>
> --
> 1012649: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1012649
> Debian Bug Tracking System
> Contact [email protected] with problems

[rbkeyfix.patch (text/x-patch, attachment)]

Send a report that this bug log contains spam.