Package: wnpp
Severity: wishlist
Owner: Guillem Jover <[email protected]>
X-Debbugs-Cc: [email protected], Mark Brown <[email protected]>
* Package name : zlib-ng
Version : 2.0.5
Upstream Author : zlib-ng Team
* URL : http://github.com/zlib-ng/zlib-ng
* License : Zlib, Zlib-RFC, CC-BY-3.0, CC-BY-4.0, Public-Domain
Programming Lang: C
Description : optimized zlib compression library
zlib-ng is a fork of the zlib library implementing the deflate compression
method found in gzip and PKZIP.
.
It includes and consolidates many optimizations found in alternative forks,
that have not been merged in the official zlib library.
I just discovered this project, and started packaging [P] it to be able
to play with a dpkg branch switching its zlib support to zlib-ng. The
speed up is quite significant, for example when packing the 0ad-data,
on my system it takes 5m50s~ with zlib and 4m30s~ with zlib-ng.
[P] https://git.hadrons.org/cgit/wip/debian/pkgs/zlib-ng.git/
The project has a compat mode which generates API "compatible" zlib
replacement libraries, but unfortunately it is stated not to guarantee
to be ABI compatible, so no compat packages or similar could be
produced right now as that could potentially break other packages. I've
filed a report [S] upstream to request a more usable shim instead.
[S] <https://github.com/zlib-ng/zlib-ng/issues/1081>
I'm still pondering whether to upload this, although the packing is
already done, but w/o the above mentioned shim its utility seems
restricted as most upstream projects use it via its compat mode,
instead of with its native API. But if that happens, I think it would
make sense to upload, as it's currently being embedded in several
upstream projects and even if dpkg would not switch to it, it would
still help with removing embedded code copies, and speeding up other
packages. Or make other RPFs such as #901490 (an alternative fork)
unnecessary.
Thanks,
Guillem
not that i expect you to have run extensive benchmarks or
anything, but how do you feel this compares to libdeflate? the
few comparisons i've seen suggest that they are (or at least
were) pretty much a wash, performance-wise.
On Mon, 2021-12-20 at 23:15:07 -0500, nick black wrote:
> not that i expect you to have run extensive benchmarks or
> anything, but how do you feel this compares to libdeflate?
I don't think it matters, because libdeflate does not support a
streaming API anyway, so it's of no use in many situations where
large files or streaming content needs to be dealt with. It also
does not provide a compatible zlib API, which means it cannot be
easily migrated to.
> the few comparisons i've seen suggest that they are (or at least
> were) pretty much a wash, performance-wise.
I think when I first noticed libdeflate being uploaded to Debian, I
took a look to also play with it for dpkg, but the above problems
meant that never got anywhere. From the benchmarks I've seen
libdeflate might be a bit faster, but then it would require tons more
memory to handle equivalent inputs and outputs.
The reasons I found the zlib-ng alternative interesting were because:
- the Intel fork is not going to be adding non-Intel optimizations,
- the Cloudflare fork does not look very lively,
- the zlib upstream does not look very active, and apparently outright
refuses contributions to the main code, and only accepts them into
its contrib/ directory,
OTOH the zlib-ng looks very lively, accepts contributions, has had its
code modernized and cleaned up, and seems to be performing rather well
in comparison on multiple architectures.
Thanks,
Guillem
On Tue, Dec 21, 2021 at 04:45:21AM +0100, Guillem Jover wrote:
> instead of with its native API. But if that happens, I think it would
> make sense to upload, as it's currently being embedded in several
> upstream projects and even if dpkg would not switch to it, it would
> still help with removing embedded code copies, and speeding up other
> packages. Or make other RPFs such as #901490 (an alternative fork)
> unnecessary.
The alternative fork is the big issue here - there's several different
zlib projects active which don't seem to be working towards each other
and there's none of them that is clearly going to be the one that will
become the default in future if it's not the original zlib. We should
just pick a new zlib if we want a new zlib rather than making users go
and figure out which they want.
Subject: Bug#1002056 in package zlib-ng marked as pending
Date: Fri, 13 Jan 2023 00:17:27 +0100
Control: tag 1002056 pending
Hi!
Bug #1002056 that you reported in package zlib-ng has been fixed
in the wip/debian/pkgs/zlib-ng.git git repository. You can see the changelog below,
and you can check the diff of the fix at:
https://git.hadrons.org/cgit/wip/debian/pkgs/zlib-ng.git/diff/?id=19382bf
---
commit 19382bf1f8369e87a4bc12b468cef0c4c447de1e (HEAD -> main)
Author: Guillem Jover <[email protected]>
Date: Tue Dec 21 02:22:23 2021 +0100
Initial packaging
diff --git a/debian/changelog b/debian/changelog
new file mode 100644
index 0000000..e4069a9
--- /dev/null
+++ b/debian/changelog
@@ -0,0 +1,5 @@
+zlib-ng (2.0.5-1) UNRELEASED; urgency=medium
+
+ * Initial release. (Closes: #1002056)
+
+ -- Guillem Jover <[email protected]> Mon, 20 Dec 2021 23:33:25 +0100
Hello,
I see you recently pushed some code into git, do you plan to push the
code also into Debian itself?
Thank you
David
--
David Heidelberg
Consultant Software Engineer
Hi!
On Sat, 2023-03-25 at 21:24:44 +0100, David Heidelberg wrote:
> I see you recently pushed some code into git, do you plan to push the code
> also into Debian itself?
Given that it conflicts with the zlib package, and I'm not sure it
makes sense to upload just the zlib-ng specific library without the
compatibility shims, I'm currently not planning on doing so, no. But
I should probably discuss this further with Mark Brown.
Thanks,
Guillem
Subject: Re: Bug#1002056: Fedora plans to transition to Zlib-ng
Date: Wed, 25 Oct 2023 23:17:06 +0200
Hi!
On Wed, 2023-10-25 at 14:13:54 -0300, Nelson A. de Oliveira wrote:
> JFTR, Fedora is planning to transition to Zlib-ng
>
> https://fedoraproject.org/wiki/Changes/ZlibNGTransition
Ah, thanks! I had in my mind getting back to this ITP, given that the
zlib-ng project has continued to gain traction and seems to have
consolidated most of the other forks around it.
So I'll draft another mail to Mark and probably to debian-devel to
discuss this.
Thanks,
Guillem
Subject: rust-gix-features seem to depend on zlib-ng
Date: Sat, 4 Nov 2023 20:30:19 +0100
Hi
Some rust code I'm trying to package seem to depend on zlib-ng, this
can maybe be patched out, but I thought I should ask about status on
this packaging effort before I undertake that work.
best regards
Alexander Kjäll
Subject: Re: Bug#1002056: Fedora plans to transition to Zlib-ng
Date: Mon, 6 Nov 2023 21:44:05 +0100
On 2023-10-25 23:17:06 [+0200], Guillem Jover wrote:
> Hi!
Hi,
> Ah, thanks! I had in my mind getting back to this ITP, given that the
> zlib-ng project has continued to gain traction and seems to have
> consolidated most of the other forks around it.
>
> So I'll draft another mail to Mark and probably to debian-devel to
> discuss this.
Do you want me to join your efforts? This looks interrestig. I may have
time ;)
> Thanks,
> Guillem
Sebastian
Hello,
I think it already makes sense to push zlib-ng and let it co-exist with
zlib since you can port your software directly to the zlib-ng, which I'm
currently doing for Mesa3D.
I dropped the zlib-ng sources into https://salsa.debian.org/dh/zlib-ng
feel free to force push there any Debian relevant changes.
After introducing the zlib-ng, we could continue to the second phase
migrating software still relying on zlib to zlib-ng compat layer.
What do you think?
David
On Mon, 6 Nov 2023 21:44:05 +0100 Sebastian Andrzej Siewior
<[email protected]> wrote:
> On 2023-10-25 23:17:06 [+0200], Guillem Jover wrote:
> > Hi!
> Hi,
>
> > Ah, thanks! I had in my mind getting back to this ITP, given that the
> > zlib-ng project has continued to gain traction and seems to have
> > consolidated most of the other forks around it.
> >
> > So I'll draft another mail to Mark and probably to debian-devel to
> > discuss this.
>
> Do you want me to join your efforts? This looks interrestig. I may have
> time ;)
>
> > Thanks,
> > Guillem
>
> Sebastian
>
>
--
David Heidelberg
Subject: Bug#1002056 in package zlib-ng marked as pending
Date: Thu, 2 May 2024 19:12:00 +0200 (CEST)
Control: tag 1002056 pending
Hi!
Bug #1002056 that you reported in package zlib-ng has been fixed
in the wip/debian/pkgs/zlib-ng.git git repository. You can see the changelog below,
and you can check the diff of the fix at:
https://git.hadrons.org/cgit/wip/debian/pkgs/zlib-ng.git/diff/?id=7a95e36
---
commit 7a95e36ae2813fde9aa6b732da9fcc1d9cd96386 (HEAD -> main)
Author: Guillem Jover <[email protected]>
Date: Tue Dec 21 02:22:23 2021 +0100
Initial packaging
diff --git a/debian/changelog b/debian/changelog
new file mode 100644
index 0000000..aa56259
--- /dev/null
+++ b/debian/changelog
@@ -0,0 +1,5 @@
+zlib-ng (2.1.6-1) UNRELEASED; urgency=medium
+
+ * Initial release. (Closes: #1002056)
+
+ -- Guillem Jover <[email protected]> Thu, 02 May 2024 18:48:28 +0200
Subject: Bug#1002056 in package zlib-ng marked as pending
Date: Thu, 2 May 2024 19:53:04 +0200 (CEST)
Control: tag 1002056 pending
Hi!
Bug #1002056 that you reported in package zlib-ng has been fixed
in the wip/debian/pkgs/zlib-ng.git git repository. You can see the changelog below,
and you can check the diff of the fix at:
https://git.hadrons.org/cgit/wip/debian/pkgs/zlib-ng.git/diff/?id=e0eed81
---
commit e0eed81519451f52595eac7f2cc2fc2e2fb84c89 (HEAD -> main)
Author: Guillem Jover <[email protected]>
Date: Tue Dec 21 02:22:23 2021 +0100
Initial packaging
diff --git a/debian/changelog b/debian/changelog
new file mode 100644
index 0000000..aa56259
--- /dev/null
+++ b/debian/changelog
@@ -0,0 +1,5 @@
+zlib-ng (2.1.6-1) UNRELEASED; urgency=medium
+
+ * Initial release. (Closes: #1002056)
+
+ -- Guillem Jover <[email protected]> Thu, 02 May 2024 18:48:28 +0200
A recurrning question with the zlib package in Debian is interest in the
various alternative zlib implementations that are out there. There was
a long period where upstream zlib development seemed very stalled,
during that period people who wanted improvements started forking their
own projects. The main ones I'm aware of are:
zlib-ng: https://github.com/zlib-ng/zlib-ng
chromium: https://chromium.googlesource.com/chromium/src/third_party/zlib
zlib-ng seems pretty healthy, the chromium fork is less generally active
but is used by Chrome/ChromeOS which is a big userbase.
The main thing people seem excited about is performance work for modern
platforms though both projects have been doing other work on the code.
Unfortunately it looks like there is little interest in bringing these
forks together in spite of zlib's upstream development having picked up
a bit again.
Fedora did a transition to zlib-ng relatively recently, in version 40:
https://fedoraproject.org/wiki/Changes/ZlibNGTransitionhttps://packages.fedoraproject.org/pkgs/zlib/zlib/https://packages.fedoraproject.org/pkgs/zlib-ng/zlib-ng/
In the past I've pushed back on doing anything here since zlib is
essential and it seemed better to be consistent over the ecosystem than
to use a more niche implementation, and some of the early optimisation
efforts had not worked well on CPUs other than their immediate targets.
However given the user feedback and looking at the Fedora experience I
think it might be time to reevaluate that.
Obviously it's far too late to do anything with the default for trixie,
we might want to evaluate doing something after the release but for now
it's too late.
There's been some ongoing discussion (which sadly I wasn't looped into
most of) of zlib-ng in WNPP:
https://bugs.debian.org/1002056
with some packaging done, but not AIUI building the zlib compatible ABI
for zlib-ng yet which would allow it to be used as a replacement.
Adding support for the compatible ABI allowing it to be an alternative
for standard zlib seems to me like an obvious step we could take, it
would need a lot of care given that zlib is essentially but would let
people get zlib-ng if they wanted, and if there are problems it can be
held in unstable (or experimental) to avoid impact on trixie. This
would allow people to kick the tires.
Does anyone have thoughts on this?
Subject: Re: Supporting alternative zlib implementations
Date: Tue, 24 Sep 2024 17:45:49 +0200
Hi!
On Tue, 2024-09-24 at 15:58:10 +0200, Mark Brown wrote:
> In the past I've pushed back on doing anything here since zlib is
> essential and it seemed better to be consistent over the ecosystem than
> to use a more niche implementation, and some of the early optimisation
> efforts had not worked well on CPUs other than their immediate targets.
> However given the user feedback and looking at the Fedora experience I
> think it might be time to reevaluate that.
Great! I'm happy to hear that.
> Obviously it's far too late to do anything with the default for trixie,
> we might want to evaluate doing something after the release but for now
> it's too late.
Personally I don't think it's too late, there should be several months
until the freeze, and I think if we wanted to switch we could perhaps
do a staged transition and see how it goes and only do the final
replacement if everything seems fine.
> There's been some ongoing discussion (which sadly I wasn't looped into
> most of) of zlib-ng in WNPP:
>
> https://bugs.debian.org/1002056
>
> with some packaging done, but not AIUI building the zlib compatible ABI
> for zlib-ng yet which would allow it to be used as a replacement.
> Adding support for the compatible ABI allowing it to be an alternative
> for standard zlib seems to me like an obvious step we could take, it
> would need a lot of care given that zlib is essentially but would let
> people get zlib-ng if they wanted, and if there are problems it can be
> held in unstable (or experimental) to avoid impact on trixie. This
> would allow people to kick the tires.
Sorry, I've been meaning to bring this up again to your attention,
given that as you mention zlib-ng has seen steady development and
buy in from the community at large. But at the same time, I've been
both a bit reluctant to upload anything to avoid the impression of
some kind of attempt to a hostile takeover, and to bring this up to
you as from your earlier push back I thought that would require some
(perhaps) exceeding changed circumstances. But given your mail, I'm
happy to work on this again and start with say uploading some initial
stuff into experimental for example, after this thread settles a bit?
(I'll start by refreshing the packaging first though.)
> Does anyone have thoughts on this?
Personally, I think fully migrating from zlib to zlib-ng would sound
great (even for trixie), but I guess we can take it slow if you do not
feel confident or have concerns over this.
Also if you'd prefer to take over the zlib-ng ITP, as a continuation
of zlib, that'd seem fine with me too.
Thanks,
Guillem
Subject: Bug#1002056 in package zlib-ng marked as pending
Date: Tue, 24 Sep 2024 18:29:49 +0200 (CEST)
Control: tag 1002056 pending
Hi!
Bug #1002056 that you reported in package zlib-ng has been fixed
in the wip/debian/pkgs/zlib-ng.git git repository. You can see the changelog below,
and you can check the diff of the fix at:
https://git.hadrons.org/cgit/wip/debian/pkgs/zlib-ng.git/diff/?id=c8d9119
---
commit c8d91194136159b6d32fa629396e3e2e2fe39234 (HEAD -> main)
Author: Guillem Jover <[email protected]>
Date: Tue Dec 21 02:22:23 2021 +0100
Initial packaging
diff --git a/debian/changelog b/debian/changelog
new file mode 100644
index 0000000..1c25ecd
--- /dev/null
+++ b/debian/changelog
@@ -0,0 +1,5 @@
+zlib-ng (2.2.2-1) UNRELEASED; urgency=medium
+
+ * Initial release. (Closes: #1002056)
+
+ -- Guillem Jover <[email protected]> Tue, 24 Sep 2024 18:18:29 +0200
Subject: Bug#1002056 in package zlib-ng marked as pending
Date: Tue, 24 Sep 2024 19:31:06 +0200 (CEST)
Control: tag 1002056 pending
Hi!
Bug #1002056 that you reported in package zlib-ng has been fixed
in the wip/debian/pkgs/zlib-ng.git git repository. You can see the changelog below,
and you can check the diff of the fix at:
https://git.hadrons.org/cgit/wip/debian/pkgs/zlib-ng.git/diff/?id=9decf65
---
commit 9decf6576c3296972479a7413b2379846755cd25 (HEAD -> main)
Author: Guillem Jover <[email protected]>
Date: Tue Dec 21 02:22:23 2021 +0100
Initial packaging
diff --git a/debian/changelog b/debian/changelog
new file mode 100644
index 0000000..1c25ecd
--- /dev/null
+++ b/debian/changelog
@@ -0,0 +1,5 @@
+zlib-ng (2.2.2-1) UNRELEASED; urgency=medium
+
+ * Initial release. (Closes: #1002056)
+
+ -- Guillem Jover <[email protected]> Tue, 24 Sep 2024 18:18:29 +0200
Subject: Re: Supporting alternative zlib implementations
Date: Wed, 25 Sep 2024 00:39:10 +0200
* Guillem Jover <[email protected]> [2024-09-24 17:45]:
[...]
> Personally, I think fully migrating from zlib to zlib-ng would sound
> great (even for trixie), but I guess we can take it slow if you do not
> feel confident or have concerns over this.
As using an alternative zlib implementation could impact Reproducible Builds
[1], I would recommend taking that into consideration before deciding on this
kind of change.
- Fay
[1] https://lists.reproducible-builds.org/pipermail/rb-general/2024-September/003526.html
Subject: Re: Supporting alternative zlib implementations
Date: Wed, 25 Sep 2024 01:55:17 +0200
Hi!
On Wed, 2024-09-25 at 00:39:10 +0200, Fay Stegerman wrote:
> * Guillem Jover <[email protected]> [2024-09-24 17:45]:
> > Personally, I think fully migrating from zlib to zlib-ng would sound
> > great (even for trixie), but I guess we can take it slow if you do not
> > feel confident or have concerns over this.
>
> As using an alternative zlib implementation could impact Reproducible Builds
> [1], I would recommend taking that into consideration before deciding on this
> kind of change.
Ah, this is related to something I wanted to mention too and forgot.
I don't think the specific case you mention is in itself a concern for
Debian, because we only guarantee reproducibility given the same inputs,
which includes the set of packages and their versions that were used
when building the binaries. So if there was a switch, those would end up
being recorded as well, and used when reproducing the outputs. And this
could also happen with a newer version of zlib itself.
The problem though is, that because the compressed stream is going to
change, that can make certain test suites fail if we perform this
switch, which I think would be the main fallout that we'd see from
this and would need manual fixing, although I assume Fedora has probably
handled most of these already. For example when I added explicit
zlib-ng support to dpkg, I had to fix its test suite to parametrize
sizes for test artifacts.
I think it would be pretty easy to at least see the extent of this
fallout by performing a mass rebuild for packages build-depending
on zlib1g-dev with a zlib-ng version.
Thanks,
Guillem
Subject: Re: Supporting alternative zlib implementations
Date: Wed, 25 Sep 2024 09:35:14 +0900
On Wed, Sep 25, 2024 at 01:55:17AM +0200, Guillem Jover wrote:
> Hi!
>
> On Wed, 2024-09-25 at 00:39:10 +0200, Fay Stegerman wrote:
> > * Guillem Jover <[email protected]> [2024-09-24 17:45]:
> > > Personally, I think fully migrating from zlib to zlib-ng would sound
> > > great (even for trixie), but I guess we can take it slow if you do not
> > > feel confident or have concerns over this.
> >
> > As using an alternative zlib implementation could impact Reproducible Builds
> > [1], I would recommend taking that into consideration before deciding on this
> > kind of change.
>
> Ah, this is related to something I wanted to mention too and forgot.
>
> I don't think the specific case you mention is in itself a concern for
> Debian, because we only guarantee reproducibility given the same inputs,
> which includes the set of packages and their versions that were used
> when building the binaries. So if there was a switch, those would end up
> being recorded as well, and used when reproducing the outputs. And this
> could also happen with a newer version of zlib itself.
>
> The problem though is, that because the compressed stream is going to
> change, that can make certain test suites fail if we perform this
> switch, which I think would be the main fallout that we'd see from
> this and would need manual fixing, although I assume Fedora has probably
> handled most of these already. For example when I added explicit
> zlib-ng support to dpkg, I had to fix its test suite to parametrize
> sizes for test artifacts.
As someone who recently tested a rust port of zlib-ng, another factor to
take into account is that while zlib-ng is faster, it also looks like it
compresses less at the same compression level.
Using higher compression levels with the same compression rate is also
usually faster too, but without touching the compression levels, you
end up with something that compresses in less time, but also with a
bigger output.
Mike
On Tue, Sep 24, 2024 at 05:45:49PM +0200, Guillem Jover wrote:
> On Tue, 2024-09-24 at 15:58:10 +0200, Mark Brown wrote:
> > Obviously it's far too late to do anything with the default for trixie,
> > we might want to evaluate doing something after the release but for now
> > it's too late.
> Personally I don't think it's too late, there should be several months
> until the freeze, and I think if we wanted to switch we could perhaps
> do a staged transition and see how it goes and only do the final
> replacement if everything seems fine.
We do OTOH package more software than most distros on more architectures
so we got a lot more exposure for testing coverage, and the revert would
involve switching the entire implementation which complicates things a
bit compared to a risky patch within a package. I'm not totally
opposed, and if everything goes smoothly we could definitely implement
it within the timeframe, but it feels like an impactful change to
introduce now not having considered it sooner.
> (perhaps) exceeding changed circumstances. But given your mail, I'm
> happy to work on this again and start with say uploading some initial
> stuff into experimental for example, after this thread settles a bit?
> (I'll start by refreshing the packaging first though.)
Sure.
> > Does anyone have thoughts on this?
> Personally, I think fully migrating from zlib to zlib-ng would sound
> great (even for trixie), but I guess we can take it slow if you do not
> feel confident or have concerns over this.
> Also if you'd prefer to take over the zlib-ng ITP, as a continuation
> of zlib, that'd seem fine with me too.
I'm fine with you carrying on with it (actually there is some slight
non-technical complication for me with doing it myself), or we could
also consider a packaging team. I think there was some other interest
in helping out but ICBW. If you're packaging it I'm also more confident
in letting you worry about how risky it is to transition and deal with
any fallout! :P
On Wed, Sep 25, 2024 at 01:55:17AM +0200, Guillem Jover wrote:
> The problem though is, that because the compressed stream is going to
> change, that can make certain test suites fail if we perform this
> switch, which I think would be the main fallout that we'd see from
> this and would need manual fixing, although I assume Fedora has probably
> handled most of these already. For example when I added explicit
> zlib-ng support to dpkg, I had to fix its test suite to parametrize
> sizes for test artifacts.
I guess this is also a risk for zlib upgrades, seems a bit fragile.
Subject: Re: Supporting alternative zlib implementations
Date: Thu, 26 Sep 2024 01:35:45 +0200
* Guillem Jover <[email protected]> [2024-09-25 01:55]:
> Hi!
>
> On Wed, 2024-09-25 at 00:39:10 +0200, Fay Stegerman wrote:
> > * Guillem Jover <[email protected]> [2024-09-24 17:45]:
> > > Personally, I think fully migrating from zlib to zlib-ng would sound
> > > great (even for trixie), but I guess we can take it slow if you do not
> > > feel confident or have concerns over this.
> >
> > As using an alternative zlib implementation could impact Reproducible Builds
> > [1], I would recommend taking that into consideration before deciding on this
> > kind of change.
>
> Ah, this is related to something I wanted to mention too and forgot.
>
> I don't think the specific case you mention is in itself a concern for
> Debian, because we only guarantee reproducibility given the same inputs,
> which includes the set of packages and their versions that were used
> when building the binaries. So if there was a switch, those would end up
> being recorded as well, and used when reproducing the outputs. And this
> could also happen with a newer version of zlib itself.
>
> The problem though is, that because the compressed stream is going to
> change, that can make certain test suites fail if we perform this
> switch, which I think would be the main fallout that we'd see from
> this and would need manual fixing, although I assume Fedora has probably
> handled most of these already. For example when I added explicit
> zlib-ng support to dpkg, I had to fix its test suite to parametrize
> sizes for test artifacts.
Whilst it indeed may not affect the reproducibility guarantees for Debian
packages themselves, it does affect being able to use a Debian system for
Reproducible Builds of other software for which the reference artefacts were
built with regular zlib and thus can no longer be reproduced on Debian if that
uses a different zlib implementation (so far I've only encountered the reverse,
which seems relatively rare -- for now).
For example, ZIP files or Android APKs built on a Debian system will have a
different compressed stream, like the test files you mention. Which will likely
break Reproducible Builds tooling like apksigcopier [1] and
reproducible-apk-tools [2].
AFAIK all rebuilders (including my own [3]) for Android APKs use Debian base
systems, so this could cause quite a bit of breakage for Reproducible Builds
within that ecosystem, which is something I would like to avoid (or at least
have a decent workaround for -- e.g. being able to easily choose between
multiple zlib implementations during runtime in my Python tooling would be
great).
As you point out, we've been lucky that zlib has remained backwards-compatible
for a long time (even though it doesn't provide any guarantees of that AFAIK).
Which also makes me wonder how much more likely zlib-ng might be to produce
different compressed streams between different versions or using different
hardware (configurations).
There might also be issues with reproducibility of Debian packages themselves if
e.g. zlib-ng output can differ on different hardware (e.g. number of cores) even
with an otherwise identical build environment. At the very least I think it
would be good to know how all this could be affected (and how likely things are
to remain as stable as zlib has been so far) before making a decision to switch.
> I think it would be pretty easy to at least see the extent of this
> fallout by performing a mass rebuild for packages build-depending
> on zlib1g-dev with a zlib-ng version.
- Fay
[1] https://tracker.debian.org/pkg/apksigcopier
[2] https://github.com/obfusk/reproducible-apk-tools
[3] https://github.com/obfusk/rbtlog
Subject: Re: Supporting alternative zlib implementations
Date: Thu, 3 Oct 2024 22:03:52 +0200
On 2024-09-26 01:35:45 [+0200], Fay Stegerman wrote:
> For example, ZIP files or Android APKs built on a Debian system will have a
> different compressed stream, like the test files you mention. Which will likely
> break Reproducible Builds tooling like apksigcopier [1] and
> reproducible-apk-tools [2].
wouldn't it work to compare the decompressed stream? Is an identical ZIP
file a requirement?
> There might also be issues with reproducibility of Debian packages themselves if
> e.g. zlib-ng output can differ on different hardware (e.g. number of cores) even
> with an otherwise identical build environment. At the very least I think it
> would be good to know how all this could be affected (and how likely things are
> to remain as stable as zlib has been so far) before making a decision to switch.
I don't know at this time. Maybe we could throw it into exp first and
evaluate the situtation.
> - Fay
Sebastian
Subject: Re: Supporting alternative zlib implementations
Date: Fri, 4 Oct 2024 00:17:33 +0200
* Sebastian Andrzej Siewior <[email protected]> [2024-10-03 22:03]:
> On 2024-09-26 01:35:45 [+0200], Fay Stegerman wrote:
> > For example, ZIP files or Android APKs built on a Debian system will have a
> > different compressed stream, like the test files you mention. Which will likely
> > break Reproducible Builds tooling like apksigcopier [1] and
> > reproducible-apk-tools [2].
>
> wouldn't it work to compare the decompressed stream? Is an identical ZIP
> file a requirement?
By definition a Reproducible Build means a bit-by-bit identical APK, including
the signature (which is why I built a tool to extract an existing signature and
use it as a build input instead of the private key). Which means you need
identical compressed data for Reproducible Builds.
Having identical uncompressed data gets you pretty close to the goals of RB, but
unpacking and/or skipping over signatures is very very hard to get right and
simply cannot provide the same guarantees as having two bitwise identical files.
And it's impossible to create an APK you can actually install if it's not
bit-by-bit identical as the signature would not be valid otherwise. So yes,
unfortunately an identical ZIP file is a requirement and comparing the
decompressed stream not an option, which is why this kind of change is not
something we can just consider an implementation detail or work around.
I wrote more about the very messy situation Fedora's switch to zlib-ng already
created for Android Reproducible Builds [1]. Which likely would have broken a
lot more reproducible Android apps already if Fedora's OpenJDK packages linked
against the system zlib like Debian's OpenJDK packages do (instead of using an
embedded copy of regular zlib).
- Fay
[1] https://lists.reproducible-builds.org/pipermail/rb-general/2024-September/003547.html
Subject: Re: Supporting alternative zlib implementations
Date: Fri, 4 Oct 2024 07:26:25 +0300
One minor moment: zlib-ng doesn't seem to be fully backward compatible.
E.g. Angie (nginx's fork with enhancements) is unable to perform gzip
compression [1] if built against zlib-ng.
It's highly likely that nginx is affected too.
[1] https://t.me/angie_support/4205
пт, 4 окт. 2024 г. в 03:13, Fay Stegerman <[email protected]>:
>
> * Sebastian Andrzej Siewior <[email protected]> [2024-10-03 22:03]:
> > On 2024-09-26 01:35:45 [+0200], Fay Stegerman wrote:
> > > For example, ZIP files or Android APKs built on a Debian system will have a
> > > different compressed stream, like the test files you mention. Which will likely
> > > break Reproducible Builds tooling like apksigcopier [1] and
> > > reproducible-apk-tools [2].
> >
> > wouldn't it work to compare the decompressed stream? Is an identical ZIP
> > file a requirement?
>
> By definition a Reproducible Build means a bit-by-bit identical APK, including
> the signature (which is why I built a tool to extract an existing signature and
> use it as a build input instead of the private key). Which means you need
> identical compressed data for Reproducible Builds.
>
> Having identical uncompressed data gets you pretty close to the goals of RB, but
> unpacking and/or skipping over signatures is very very hard to get right and
> simply cannot provide the same guarantees as having two bitwise identical files.
>
> And it's impossible to create an APK you can actually install if it's not
> bit-by-bit identical as the signature would not be valid otherwise. So yes,
> unfortunately an identical ZIP file is a requirement and comparing the
> decompressed stream not an option, which is why this kind of change is not
> something we can just consider an implementation detail or work around.
>
> I wrote more about the very messy situation Fedora's switch to zlib-ng already
> created for Android Reproducible Builds [1]. Which likely would have broken a
> lot more reproducible Android apps already if Fedora's OpenJDK packages linked
> against the system zlib like Debian's OpenJDK packages do (instead of using an
> embedded copy of regular zlib).
>
> - Fay
>
> [1] https://lists.reproducible-builds.org/pipermail/rb-general/2024-September/003547.html
>
--
SY,
Konstantin Demin
Subject: Bug#1002056 in package zlib-ng marked as pending
Date: Thu, 21 Nov 2024 22:27:24 +0100 (CET)
Control: tag 1002056 pending
Hi!
Bug #1002056 that you reported in package zlib-ng has been fixed
in the wip/debian/pkgs/zlib-ng.git git repository. You can see the changelog below,
and you can check the diff of the fix at:
https://git.hadrons.org/cgit/wip/debian/pkgs/zlib-ng.git/diff/?id=2815593
---
commit 2815593ee6b8bc252ea0d433f2c99b8e7e3abb1c (HEAD -> main)
Author: Guillem Jover <[email protected]>
Date: Tue Dec 21 02:22:23 2021 +0100
Initial packaging
diff --git a/debian/changelog b/debian/changelog
new file mode 100644
index 0000000..1c25ecd
--- /dev/null
+++ b/debian/changelog
@@ -0,0 +1,5 @@
+zlib-ng (2.2.2-1) UNRELEASED; urgency=medium
+
+ * Initial release. (Closes: #1002056)
+
+ -- Guillem Jover <[email protected]> Tue, 24 Sep 2024 18:18:29 +0200
Subject: [Summary]: Supporting alternative zlib implementations
Date: Fri, 22 Nov 2024 12:29:51 +0100
Hi!
[ I'll try to summarize the current discussion and status, what might
be blockers, and a potential incremental way forward. ]
On Wed, 2024-09-25 at 10:48:50 +0200, Mark Brown wrote:
> On Tue, Sep 24, 2024 at 05:45:49PM +0200, Guillem Jover wrote:
> > On Tue, 2024-09-24 at 15:58:10 +0200, Mark Brown wrote:
> > > Obviously it's far too late to do anything with the default for trixie,
> > > we might want to evaluate doing something after the release but for now
> > > it's too late.
>
> > Personally I don't think it's too late, there should be several months
> > until the freeze, and I think if we wanted to switch we could perhaps
> > do a staged transition and see how it goes and only do the final
> > replacement if everything seems fine.
>
> We do OTOH package more software than most distros on more architectures
> so we got a lot more exposure for testing coverage, and the revert would
> involve switching the entire implementation which complicates things a
> bit compared to a risky patch within a package. I'm not totally
> opposed, and if everything goes smoothly we could definitely implement
> it within the timeframe, but it feels like an impactful change to
> introduce now not having considered it sooner.
True, also two months have passed since (that's on me!). At this time,
I'm now not sure whether it is feasible to consider such a switch, even
if there was agreement to do it. As it is, I think there are too many
unknowns!
> > (perhaps) exceeding changed circumstances. But given your mail, I'm
> > happy to work on this again and start with say uploading some initial
> > stuff into experimental for example, after this thread settles a bit?
>
> > (I'll start by refreshing the packaging first though.)
>
> Sure.
I did that, and the current WIP zlib-ng packaging provides now two
builds, one with the new native zng_* API and another (tentatively)
with the compat API/ABI one in libz-dev and libz1 binary packages.
I've tentatively chosen those package names for the compat libraries
to avoid having to go through NEW multiple times (with the assumption
that we'd either go ahead with the switch or the packages could then
simply be dropped). I think this should initially only be uploaded to
experimental, to avoid getting packages built with either zlib or
zlib-ng. But depending on the outcome of this discussion, I think other
(probably better) options would be to perhaps name the compat packages
something like libz-ng-compat*, or drop them completely?
WIP package at <https://git.hadrons.org/cgit/wip/debian/pkgs/zlib-ng.git/>.
> > > Does anyone have thoughts on this?
>
> > Personally, I think fully migrating from zlib to zlib-ng would sound
> > great (even for trixie), but I guess we can take it slow if you do not
> > feel confident or have concerns over this.
>
> > Also if you'd prefer to take over the zlib-ng ITP, as a continuation
> > of zlib, that'd seem fine with me too.
>
> I'm fine with you carrying on with it (actually there is some slight
> non-technical complication for me with doing it myself), or we could
> also consider a packaging team. I think there was some other interest
> in helping out but ICBW. If you're packaging it I'm also more confident
> in letting you worry about how risky it is to transition and deal with
> any fallout! :P
Ok, so after the feedback on this thread, and Sebastian asking how we
can proceed, here are the concerns brought on this thread, along my
own and things I think we need to check or consider:
* There were concerns (from Fay) about whether given same input the
output changes per arch or hw setup, we'd need to check this; I'd
expect this not to be the case for different arches, but it might
be an issue with number of cores for example, but if either is true
this would be a serious blocker.
* I've had concerns both about providing the zlib compat API and the
native zlib-ng API in sid, and then getting a mess of packages
linking against (true) zlib and against (native) zlib-ng, or
packages relying on specific behaviors from either and breaking
when switching from (true) zlib to zlib-ng-compat or vice versa,
for example.
* There were concerns (from Fay) about the output stream changing due
to a potential implementation switch and that affecting external
reproducibility. Personally I think while I can see how this is
annoying for the involved parties, it's part of the "you need
the same tools to generate the same output" premise that we also
assume in Debian. I guess keeping both implementations around
indefinitely, I think, would make this less of an issue, with the
potential drawbacks mentioned in the previous point.
* There was a concern (from Konstantin) about at least one known
upstream (Angie) misbehaving with zlib-ng generated streams.
* There were concerns (from Mark) that even though projects like
Fedora have done such switch, we have way more packages and
architectures, so we might see more fallout that has not already
been handled.
* There was a concern (from Mike) about whether the performance gain
at the cost of stream size makes sense, given that the compression
level could be reduced instead to similar effect (?). I'm not sure
how these compare, so it would be interesting to analyze this,
because perhaps that's a less traumatic way to look at it (but that
might require redefining compression level semantics globally in
zlib, or patching users, with neither look very enticing options).
My perception from when I tested it is that the speed up was
significant enough and the size increase not so much, but… In any
case switching to zlib-ng upstream would also imply other benefits,
like (supposedly) a more responsive upstream with more frequent
releases, the new native API, and an implementation other
distributions are switching to.
* Some upstreams have started to use the zlib-ng native API, so
regardless of whether we plan a switch or not, I guess packaging
zlib-ng (w/ or w/ the compat API) might still make sense.
* To consider a switch we'd need to do a mass rebuild of the
archive. Ideally running autopkgtests and similar to exercise the
packages?
After having written the above, and if Mark agrees, I think I'd opt for
uploading zlib-ng to experimental, with the compat packages renamed to
libz-ng-compat* or similar (even if that implies later on another trip
through NEW if we want to perform a full switch), because that might
make it easier to move them to sid as a way less disruptive change,
even if we decide not to switch the default zlib implementation.
OTOH and unfortunately I don't think I'm currently prepared to drive any
of what I think might be required mass archive rebuilds and testing or
the analysis mentioned above.
Thanks,
Guillem
Subject: Re: [Summary]: Supporting alternative zlib implementations
Date: Fri, 22 Nov 2024 12:56:17 +0000
On Fri, 22 Nov 2024 at 12:29:51 +0100, Guillem Jover wrote:
> depending on the outcome of this discussion, I think other
> (probably better) options would be to perhaps name the compat packages
> something like libz-ng-compat*, or drop them completely?
You might find the history of the sdl12-compat source package interesting.
It's a reimplementation of the SDL 1.2 API/ABI in terms of SDL 2, and it
went through several stages:
- only SDL 2 available (as of bullseye);
- sdl12-compat available as non-default and outside the normal library
search path, but can take over from "classic" SDL 1.2 if explicitly
requested via LD_LIBRARY_PATH or by installing a extra "shim" package
(this state is what we shipped in bookworm);
- sdl12-compat takes over binary packages from "classic" SDL 1.2,
"classic" SDL 1.2 disappears, sdl12-compat's old binary packages become
transitional (this is the current state in trixie);
- in forky, we'll presumably drop the transitional packages
Having the second step made it a lot easier to check for regressions.
Switching from glibc's libcrypt to the external libxcrypt, and from IJG
libjpeg to libjpeg-turbo, both went through similar processes.
sdl12-compat is a bit different because it's *only* the
backwards-compatible API/ABI, and not the "native" API/ABI (but you could
think of libsdl2, in its own source package, as being the corresponding
"native" API/ABI).
libxcrypt has a "native" legacy-free API/ABI available, but in Debian we don't
seem to be compiling it at the moment, and we only have the glibc- and
musl-compatible backwards-compatible ABIs (but I think some other distros
like Arch have moved to the legacy-free ABI).
libjpeg-turbo has a "native" legacy-free API/ABI libturbojpeg* in parallel
with a backwards-compatible API/ABI resembling IJG libjpeg 6b, so in that
sense it seems the closest to what you're proposing for zlib-ng.
> * I've had concerns [...] about providing the zlib compat API and the
> native zlib-ng API in sid, and then getting a mess of packages
> linking against (true) zlib and against (native) zlib-ng
Is there any problem with that, other than duplicate functionality?
From the other things you've said about the native API, it sound as
though the symbols all have different names, so this is no worse than
the duplication you would get if you have indirect dependencies on both
zlib and libdeflate, or on both OpenSSL and GNUTLS; it isn't like trying
to link to both libreadline and libedit (which export essentially the
same symbols and so will collide badly), or more than one of GTK 2/3/4
(likewise).
smcv
Subject: Re: [Summary]: Supporting alternative zlib implementations
Date: Sat, 23 Nov 2024 00:05:48 +0100
On 2024-11-22 12:29:51 [+0100], Guillem Jover wrote:
> Hi!
Hi,
…
> WIP package at <https://git.hadrons.org/cgit/wip/debian/pkgs/zlib-ng.git/>.
just built that.
…
> * To consider a switch we'd need to do a mass rebuild of the
> archive. Ideally running autopkgtests and similar to exercise the
> packages?
So you upload it to experimental then autopkgtests should pick it up and
run against unstable. You should the results in
https://release.debian.org/britney/pseudo-excuses-experimental.html
> After having written the above, and if Mark agrees, I think I'd opt for
> uploading zlib-ng to experimental, with the compat packages renamed to
> libz-ng-compat* or similar (even if that implies later on another trip
> through NEW if we want to perform a full switch), because that might
> make it easier to move them to sid as a way less disruptive change,
> even if we decide not to switch the default zlib implementation.
>
> OTOH and unfortunately I don't think I'm currently prepared to drive any
> of what I think might be required mass archive rebuilds and testing or
> the analysis mentioned above.
I just kicked a rebuild locally with libz1 in the chroot. The proper
-dev gets installed if asked so that should work. Dose computed 38659
packages so this could a while.
> Thanks,
> Guillem
Sebastian
Subject: Re: [Summary]: Supporting alternative zlib implementations
Date: Sat, 23 Nov 2024 19:56:22 +0100
* Guillem Jover <[email protected]> [2024-11-22 12:29]:
[...]
> * There were concerns (from Fay) about the output stream changing due
> to a potential implementation switch and that affecting external
> reproducibility. Personally I think while I can see how this is
> annoying for the involved parties, it's part of the "you need
> the same tools to generate the same output" premise that we also
> assume in Debian. I guess keeping both implementations around
> indefinitely, I think, would make this less of an issue, with the
> potential drawbacks mentioned in the previous point.
[...]
As I reported to the Reproducible Builds mailing list in September, Fedora
switching to zlib-ng has created a very messy situation [1] for reproducibility
of Android APKs, which would have been far worse if not for the fact that
Fedora's own OpenJDK packages use a bundled zlib instead of linking against the
system zlib-ng.
Unfortunately, Reproducible Builds for Android APKs involve reproducing APKs
built by individual upstreams, not Debian buildds. Most upstreams don't use
Debian, many even use Windows. It's simply not feasible to use the exact same
build environment as upstream in a lot of cases, nor can we expect them to
switch to building on Debian to match the rebuilders.
So far, we have still been able get identical results 99% of the time because
the Android toolchain generates identical results on different platforms -- with
some exceptions, like newlines, that we have created workarounds for, though
those workarounds rely on being able to recreate a bitwise identical zlib output
stream.
I agree that it *should not* be Debian's responsibility to ensure compatibility
with Fedora/Windows/etc., but the reality is that if "you need the same tools to
generate the same output" -- which right now means using the same JDK and
Android toolchain but in 99% of cases doesn't require using the same OS since
everyone, including Google [2], standardised on zlib -- becomes "you cannot
reproduce APKs built on a OS other than Debian on Debian", that's not just
"annoying for the involved parties": it will effectively break the ability to
verify reproducibility of many Android apps.
Having both zlib implementations available and being able to choose between them
at runtime (perhaps using LD_LIBRARY_PATH in the tooling when needed) would help
a lot to at least allow creating workarounds, but that would still require
significant changes that someone needs to implement in order to unbreak
something that currently works fine.
- Fay
[1] https://lists.reproducible-builds.org/pipermail/rb-general/2024-September/003547.html
[2] https://android-developers.googleblog.com/2016/12/saving-data-reducing-the-size-of-app-updates-by-65-percent.html
Subject: Re: [Summary]: Supporting alternative zlib implementations
Date: Sat, 23 Nov 2024 20:56:46 +0100
* Fay Stegerman <[email protected]> [241123 19:58]:
> I agree that it *should not* be Debian's responsibility to ensure compatibility
> with Fedora/Windows/etc., but the reality is that if "you need the same tools to
> generate the same output" -- which right now means using the same JDK and
> Android toolchain but in 99% of cases doesn't require using the same OS since
> everyone, including Google [2], standardised on zlib -- becomes "you cannot
> reproduce APKs built on a OS other than Debian on Debian", that's not just
> "annoying for the involved parties": it will effectively break the ability to
> verify reproducibility of many Android apps.
Sorry, but what exactly are you saying here? That Debian should be
bound by the decision of a BigTech Corporate and by thousands of
individual Android developers, neither of which might be interested
in Debian?
To maybe make the argument the other way 'round: if Google switches
to zlib-ng tomorrow, should Debian be required to switch to zlib-ng?
I just don't see that's how it works.
Chris
Subject: Re: [Summary]: Supporting alternative zlib implementations
Date: Sun, 24 Nov 2024 02:06:39 +0100
* Chris Hofstaedtler <[email protected]> [2024-11-23 20:56]:
> * Fay Stegerman <[email protected]> [241123 19:58]:
> > I agree that it *should not* be Debian's responsibility to ensure compatibility
> > with Fedora/Windows/etc., but the reality is that if "you need the same tools to
> > generate the same output" -- which right now means using the same JDK and
> > Android toolchain but in 99% of cases doesn't require using the same OS since
> > everyone, including Google [2], standardised on zlib -- becomes "you cannot
> > reproduce APKs built on a OS other than Debian on Debian", that's not just
> > "annoying for the involved parties": it will effectively break the ability to
> > verify reproducibility of many Android apps.
>
> Sorry, but what exactly are you saying here? That Debian should be
> bound by the decision of a BigTech Corporate and by thousands of
> individual Android developers, neither of which might be interested
> in Debian?
> To maybe make the argument the other way 'round: if Google switches
> to zlib-ng tomorrow, should Debian be required to switch to zlib-ng?
What I'm saying is that this change will have consequences for downstreams using
Debian for Reproducible Builds. That includes e.g. hundreds of F-Droid apps,
which would no longer be able to get updates if Reproducible Builds break.
That's clearly not Debian's fault. And of course Debian isn't *required* to do
anything. I agree Debian should not be bound by the decisions of a "BigTech
Corporate" and thousands of individual Android developers. And I very much
dislike the fact that this matters.
But Debian's choices here do have consequences for downstreams, and I think
that's something we should take into account when reasonably possible. Similar
to how we didn't switch i386 to t64 to avoid breaking running existing legacy
binaries.
For example, if it can be made easy to install both and choose between zlib and
zlib-ng at runtime, so it's easy to build APKs using either zlib or zlib-ng as
needed, downstream breakage can be avoided. Considering whether that can
reasonably be done doesn't seem like an unreasonable request to me.
What I would like is to be able to continue to use Debian for Reproducible
Builds regardless of what Google does or doesn't do. Right now that means being
able to choose to keep using the original zlib for backwards compatibility. If
Google switched to zlib-ng I would be asking if Debian could provide a way to
opt in to using that instead.
- Fay
Subject: Re: [Summary]: Supporting alternative zlib implementations
Date: Sun, 24 Nov 2024 21:36:25 +0100
On 2024-11-23 00:05:48 [+0100], To [email protected] wrote:
> On 2024-11-22 12:29:51 [+0100], Guillem Jover wrote:
> > Hi!
> Hi,
Hi,
> > WIP package at <https://git.hadrons.org/cgit/wip/debian/pkgs/zlib-ng.git/>.
>
> just built that.
…
One thing I didn't debug and it might be expected: If I install
libz1_2.2.2-1_amd64.deb and then try to install build dependencies of
0ad via
apt build-dep 0ad
then it removes libz1 while doing so. sbuild aborts complains and
aborts. I worked around it by installing the lib + header files in the
chroot before building.
> I just kicked a rebuild locally with libz1 in the chroot. The proper
> -dev gets installed if asked so that should work. Dose computed 38659
> packages so this could a while.
ongoing… 59% left. In the meantime I uploaded what looks like possible
fallout to
https://people.debian.org/~bigeasy/zlib-ng/todo/
I moved cpp-httplib to done after filling #1088193 and
https://udd.debian.org/cgi-bin/[email protected]&tag=zlib-ng-ftbfs
lists it, too.
> > Thanks,
> > Guillem
Sebastian
Subject: Re: [Summary]: Supporting alternative zlib implementations
Date: Mon, 25 Nov 2024 22:03:32 +0100
On 2024-11-24 21:36:25 [+0100], To [email protected] wrote:
…
I've been looking at cdebootstrap. It is one of the failing. This
happens in the rules files:
| ( echo -n "misc:Built-Using="; dpkg-query -f='${source:Package} (=
| ${source:Version}), ' -W libc6-dev libdebian-installer4-dev
| libdebconfclient0-dev libbz2-dev liblzma-dev zlib1g-dev; echo ) >>
| debian/cdebootstrap-static.substvars
and then
| dpkg-gencontrol: warning: Depends field of package cdebootstrap-static: substitution variable ${shlibs:Depends} used, but is not defined
| dpkg-gencontrol: warning: can't parse dependency zlib1g-dev (= )
| dpkg-gencontrol: error: parsing package 'cdebootstrap-static' Built-Using field: bzip2 (= 1.0.8-6), glibc (= 2.40-4), cdebconf (= 0.273), libdebian-installer (= 0.125), xz-utils (= 5.6.3-1), zlib1g-dev (= ),
Since we don't have zlib1g-dev anymore, this goes boom. If we exclude
zlib1g-dev then it is not recorded. Can't we smart about this?
Sebastian
Subject: Re: [Summary]: Supporting alternative zlib implementations
Date: Thu, 28 Nov 2024 12:10:03 +0100
On 2024-11-25 22:03:32 [+0100], To [email protected] wrote:
> On 2024-11-24 21:36:25 [+0100], To [email protected] wrote:
> …
>
> I've been looking at cdebootstrap. It is one of the failing. This
> happens in the rules files:
> | ( echo -n "misc:Built-Using="; dpkg-query -f='${source:Package} (=
> | ${source:Version}), ' -W libc6-dev libdebian-installer4-dev
> | libdebconfclient0-dev libbz2-dev liblzma-dev zlib1g-dev; echo ) >>
> | debian/cdebootstrap-static.substvars
>
> and then
> | dpkg-gencontrol: warning: Depends field of package cdebootstrap-static: substitution variable ${shlibs:Depends} used, but is not defined
> | dpkg-gencontrol: warning: can't parse dependency zlib1g-dev (= )
> | dpkg-gencontrol: error: parsing package 'cdebootstrap-static' Built-Using field: bzip2 (= 1.0.8-6), glibc (= 2.40-4), cdebconf (= 0.273), libdebian-installer (= 0.125), xz-utils (= 5.6.3-1), zlib1g-dev (= ),
>
> Since we don't have zlib1g-dev anymore, this goes boom. If we exclude
> zlib1g-dev then it is not recorded. Can't we smart about this?
Rebuild done. Rebuild all packages as suggested by ratt for zlib.
Total : 38655
Passed : 37188
Failed with zlib-ng: 64
Failed always : 1401
The total amount of build logs is ~20GiB. I uploaded compressed logs of
the which failed with zlib-ng and overview to
https://people.debian.org/~bigeasy/zlib-ng/
Three packages failed due to the way Build-Using is generated. I moved
those to another folder (cdebootstrap, dar, sash).
Sebastian
On Fri, Nov 22, 2024 at 12:29:51PM +0100, Guillem Jover wrote:
> [ I'll try to summarize the current discussion and status, what might
> be blockers, and a potential incremental way forward. ]
You CCed some people but not me so I only saw this today...
> On Wed, 2024-09-25 at 10:48:50 +0200, Mark Brown wrote:
> > On Tue, Sep 24, 2024 at 05:45:49PM +0200, Guillem Jover wrote:
> > We do OTOH package more software than most distros on more architectures
> > so we got a lot more exposure for testing coverage, and the revert would
> > involve switching the entire implementation which complicates things a
> > bit compared to a risky patch within a package. I'm not totally
> > opposed, and if everything goes smoothly we could definitely implement
> > it within the timeframe, but it feels like an impactful change to
> > introduce now not having considered it sooner.
> True, also two months have passed since (that's on me!). At this time,
> I'm now not sure whether it is feasible to consider such a switch, even
> if there was agreement to do it. As it is, I think there are too many
> unknowns!
It does seem more safe to offer zlib-ng as an alternative at this
point...
> I did that, and the current WIP zlib-ng packaging provides now two
> builds, one with the new native zng_* API and another (tentatively)
> with the compat API/ABI one in libz-dev and libz1 binary packages.
> I've tentatively chosen those package names for the compat libraries
> to avoid having to go through NEW multiple times (with the assumption
> that we'd either go ahead with the switch or the packages could then
> simply be dropped). I think this should initially only be uploaded to
> experimental, to avoid getting packages built with either zlib or
> zlib-ng. But depending on the outcome of this discussion, I think other
> (probably better) options would be to perhaps name the compat packages
> something like libz-ng-compat*, or drop them completely?
The packages could have a different name and support diversion or
replacement of the zlib library packages? That'd let people use them
if they wanted to.
> * I've had concerns both about providing the zlib compat API and the
> native zlib-ng API in sid, and then getting a mess of packages
> linking against (true) zlib and against (native) zlib-ng, or
> packages relying on specific behaviors from either and breaking
> when switching from (true) zlib to zlib-ng-compat or vice versa,
> for example.
What are those concerns? Like Simon says the zlib and zlib-ng APIs are
just two separate APIs, they happen to have overlapping functionality
but while that might be a bit wasteful it's not obvious that there's any
actual problem.
> * There were concerns (from Fay) about the output stream changing due
> to a potential implementation switch and that affecting external
> reproducibility. Personally I think while I can see how this is
> annoying for the involved parties, it's part of the "you need
> the same tools to generate the same output" premise that we also
> assume in Debian. I guess keeping both implementations around
> indefinitely, I think, would make this less of an issue, with the
> potential drawbacks mentioned in the previous point.
Yes, I don't think this should be a blocker - this seems like it's on a
similar level to needing to use the same compilers.
> * There was a concern (from Mike) about whether the performance gain
> at the cost of stream size makes sense, given that the compression
> level could be reduced instead to similar effect (?). I'm not sure
> how these compare, so it would be interesting to analyze this,
> because perhaps that's a less traumatic way to look at it (but that
> might require redefining compression level semantics globally in
> zlib, or patching users, with neither look very enticing options).
> My perception from when I tested it is that the speed up was
> significant enough and the size increase not so much, but… In any
> case switching to zlib-ng upstream would also imply other benefits,
> like (supposedly) a more responsive upstream with more frequent
> releases, the new native API, and an implementation other
> distributions are switching to.
> * Some upstreams have started to use the zlib-ng native API, so
> regardless of whether we plan a switch or not, I guess packaging
> zlib-ng (w/ or w/ the compat API) might still make sense.
Yes, it seems clear that we should package zlib-ng and there's just a
question about handling of the compat interface it provides.
> After having written the above, and if Mark agrees, I think I'd opt for
> uploading zlib-ng to experimental, with the compat packages renamed to
> libz-ng-compat* or similar (even if that implies later on another trip
> through NEW if we want to perform a full switch), because that might
> make it easier to move them to sid as a way less disruptive change,
> even if we decide not to switch the default zlib implementation.
Yes, that sounds sensible to me.
Subject: Re: [Summary]: Supporting alternative zlib implementations
Date: Mon, 16 Dec 2024 23:38:38 +0100
* Guillem Jover <[email protected]> [2024-11-22 12:29]:
[...]
> * There were concerns (from Fay) about whether given same input the
> output changes per arch or hw setup, we'd need to check this; I'd
> expect this not to be the case for different arches, but it might
> be an issue with number of cores for example, but if either is true
> this would be a serious blocker.
> * There were concerns (from Fay) about the output stream changing due
> to a potential implementation switch and that affecting external
> reproducibility. Personally I think while I can see how this is
> annoying for the involved parties, it's part of the "you need
> the same tools to generate the same output" premise that we also
> assume in Debian. I guess keeping both implementations around
> indefinitely, I think, would make this less of an issue, with the
> potential drawbacks mentioned in the previous point.
[...]
I did some more testing with zlib-ng. With the original zlib, you will always
get an identical output stream given the same input stream and compressor
parameters (compression level being the only one that's commonly varied in ZIP
files). I expected that zlib-ng would often produce a different output steam
than the original, but what I found was a lot more non-deterministic than just
that.
With zlib-ng, feeding the data into the compressor in e.g. 1024-byte chunks
always gave me a different output stream than using 4096-byte chunks (at
compression level 6). In fact, every chunk size I tried gave a different
output. And that's with fixed size chunks, which is not a given if you're
handling e.g. a stream of input.
Even using the same buffer size, I cannot get an identical compressed output
stream with Python and Java any more, presumably because of subtle
implementation differences (in the stdlib code that ends up calling zlib to do
the compression) that do not affect zlib but clearly do affect zlib-ng.
Which makes zlib-ng unsuitable for use cases where you need to be able to create
an identical output stream without knowing exactly how the bytes were fed into
the zlib compressor (or simply have no way to control this). This fundamentally
breaks my tooling in ways I can't fix by using the same build environment.
Because programs that used to produce identical and deterministic output with
zlib no longer do with zlib-ng, despite using the exact same zlib-ng .so.
- Fay
Debbugs is free software and licensed under the terms of the GNU General
Public License version 2. The current version can be obtained
from https://bugs.debian.org/debbugs-source/.