Debian Bug report logs - #1002056
ITP: zlib-ng -- optimized zlib compression library

Full log

🔗 View this message in rfc822 format

X-Loop: [email protected]
Subject: Bug#1002056: Supporting alternative zlib implementations
Reply-To: Fay Stegerman <[email protected]>, [email protected]
Resent-From: Fay Stegerman <[email protected]>
Resent-To: [email protected]
Resent-CC: [email protected], Guillem Jover <[email protected]>
X-Loop: [email protected]
Resent-Date: Wed, 25 Sep 2024 23:39:02 +0000
Resent-Message-ID: <[email protected]>
Resent-Sender: [email protected]
X-Debian-PR-Message: followup 1002056
X-Debian-PR-Package: wnpp
X-Debian-PR-Keywords: pending
References: <[email protected]> <[email protected]> <ZvM_jhwdb9eqGgH4@nihonium> <[email protected]> <[email protected]>
Received: via spool by [email protected] id=B1002056.17273073643803014
          (code B ref 1002056); Wed, 25 Sep 2024 23:39:02 +0000
Received: (at 1002056) by bugs.debian.org; 25 Sep 2024 23:36:04 +0000
X-Spam-Checker-Version: SpamAssassin 3.4.6-bugs.debian.org_2005_01_02
	(2021-04-09) on buxtehude.debian.org
X-Spam-Level: 
X-Spam-Status: No, score=-4.0 required=4.0 tests=BAYES_00,DKIM_SIGNED,
	DKIM_VALID,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no
	version=3.4.6-bugs.debian.org_2005_01_02
X-Spam-Bayes: score:0.0000 Tokens: new, 16; hammy, 150; neutral, 203; spammy,
	0. spammytokens: hammytokens:0.000-+--trixie, 0.000-+--afaik,
	0.000-+--UD:tracker.debian.org, 0.000-+--trackerdebianorg,
	0.000-+--tracker.debian.org
Received: from mx.kolabnow.com ([212.103.80.154]:48682)
	by buxtehude.debian.org with esmtps (TLS1.3:ECDHE_X25519__RSA_PSS_RSAE_SHA256__AES_256_GCM:256)
	(Exim 4.94.2)
	(envelope-from <[email protected]>)
	id 1stbYB-00FxJj-UC
	for [email protected]; Wed, 25 Sep 2024 23:36:04 +0000
Received: from localhost (unknown [127.0.0.1])
	by mx.kolabnow.com (Postfix) with ESMTP id 90B70D63E4;
	Thu, 26 Sep 2024 01:35:57 +0200 (CEST)
Authentication-Results: ext-mx-out013.mykolab.com (amavis);
 dkim=pass (2048-bit key) reason="pass (just generated, assumed good)"
 header.d=kolabnow.com
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kolabnow.com; h=
	in-reply-to:content-disposition:content-type:content-type
	:mime-version:references:message-id:subject:subject:from:from
	:date:date:received:received:received; s=dkim20240523; t=
	1727307346; x=1729121747; bh=vsNhBScUkeWjHTFBPQrLB8AK5nZjr31SaCP
	/kVhJKfY=; b=VQEpPupzxGSdEMA1CYg+zj/67zFa9Owz2SGV/3Ij/yW0imAI9ya
	gk4ULX8UWWSMjAa0C5ExPt4h/815LdIVXO+j6K0OOP+n8Sqg/sDm3k/38vKYuTJa
	ohYMGV9iPZxH2Io9/RWgJQf7TRtUZeugbLHFD7iR3uybK1IhqUgXq6fkuns6vGo5
	QopPHoc7+CSneQONMbldD8m6sLX7JUDjUFHGFiur0AVvdMoYDF/+Sw69D+PpYr2R
	bIwNErAdYuG6jOfQnK7n8j0g2IN8H7msmz/uiykjBsCeCVnTjQ3bDocx2WWBbj6G
	zQ5YRhntTnTwdRNe3SckWtu/Ja/uwPl8amA==
X-Virus-Scanned: amavis at mykolab.com
Received: from mx.kolabnow.com ([127.0.0.1])
 by localhost (ext-mx-out013.mykolab.com [127.0.0.1]) (amavis, port 10024)
 with ESMTP id MYVjsyaAnxuw; Thu, 26 Sep 2024 01:35:46 +0200 (CEST)
Received: from int-mx011.mykolab.com (unknown [10.9.13.11])
	by mx.kolabnow.com (Postfix) with ESMTPS id 38B71D63E3;
	Thu, 26 Sep 2024 01:35:45 +0200 (CEST)
Received: from ext-subm010.mykolab.com (unknown [10.9.6.10])
	by int-mx011.mykolab.com (Postfix) with ESMTPS id B4898308E786;
	Thu, 26 Sep 2024 01:35:45 +0200 (CEST)
Date: Thu, 26 Sep 2024 01:35:45 +0200
From: Fay Stegerman <[email protected]>
To: [email protected]
Cc: Sebastian Andrzej Siewior <[email protected]>,
	David Heidelberg <[email protected]>, [email protected]
Message-ID: <ZvSeUQyyMfrxeAG4@nihonium>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <[email protected]>

* Guillem Jover <[email protected]> [2024-09-25 01:55]:
> Hi!
>
> On Wed, 2024-09-25 at 00:39:10 +0200, Fay Stegerman wrote:
> > * Guillem Jover <[email protected]> [2024-09-24 17:45]:
> > > Personally, I think fully migrating from zlib to zlib-ng would sound
> > > great (even for trixie), but I guess we can take it slow if you do not
> > > feel confident or have concerns over this.
> >
> > As using an alternative zlib implementation could impact Reproducible Builds
> > [1], I would recommend taking that into consideration before deciding on this
> > kind of change.
>
> Ah, this is related to something I wanted to mention too and forgot.
>
> I don't think the specific case you mention is in itself a concern for
> Debian, because we only guarantee reproducibility given the same inputs,
> which includes the set of packages and their versions that were used
> when building the binaries. So if there was a switch, those would end up
> being recorded as well, and used when reproducing the outputs. And this
> could also happen with a newer version of zlib itself.
>
> The problem though is, that because the compressed stream is going to
> change, that can make certain test suites fail if we perform this
> switch, which I think would be the main fallout that we'd see from
> this and would need manual fixing, although I assume Fedora has probably
> handled most of these already. For example when I added explicit
> zlib-ng support to dpkg, I had to fix its test suite to parametrize
> sizes for test artifacts.

Whilst it indeed may not affect the reproducibility guarantees for Debian
packages themselves, it does affect being able to use a Debian system for
Reproducible Builds of other software for which the reference artefacts were
built with regular zlib and thus can no longer be reproduced on Debian if that
uses a different zlib implementation (so far I've only encountered the reverse,
which seems relatively rare -- for now).

For example, ZIP files or Android APKs built on a Debian system will have a
different compressed stream, like the test files you mention.  Which will likely
break Reproducible Builds tooling like apksigcopier [1] and
reproducible-apk-tools [2].

AFAIK all rebuilders (including my own [3]) for Android APKs use Debian base
systems, so this could cause quite a bit of breakage for Reproducible Builds
within that ecosystem, which is something I would like to avoid (or at least
have a decent workaround for -- e.g. being able to easily choose between
multiple zlib implementations during runtime in my Python tooling would be
great).

As you point out, we've been lucky that zlib has remained backwards-compatible
for a long time (even though it doesn't provide any guarantees of that AFAIK).
Which also makes me wonder how much more likely zlib-ng might be to produce
different compressed streams between different versions or using different
hardware (configurations).

There might also be issues with reproducibility of Debian packages themselves if
e.g. zlib-ng output can differ on different hardware (e.g. number of cores) even
with an otherwise identical build environment.  At the very least I think it
would be good to know how all this could be affected (and how likely things are
to remain as stable as zlib has been so far) before making a decision to switch.

> I think it would be pretty easy to at least see the extent of this
> fallout by performing a mass rebuild for packages build-depending
> on zlib1g-dev with a zlib-ng version.

- Fay

[1] https://tracker.debian.org/pkg/apksigcopier
[2] https://github.com/obfusk/reproducible-apk-tools
[3] https://github.com/obfusk/rbtlog

Send a report that this bug log contains spam.