Debian Bug report logs - #1002056
ITP: zlib-ng -- optimized zlib compression library

Full log

🔗 View this message in rfc822 format

X-Loop: [email protected]
Subject: Bug#1002056: [Summary]: Supporting alternative zlib implementations
Reply-To: Guillem Jover <[email protected]>, [email protected]
Resent-From: Guillem Jover <[email protected]>
Resent-To: [email protected]
Resent-CC: [email protected]
X-Loop: [email protected]
Resent-Date: Fri, 22 Nov 2024 11:33:02 +0000
Resent-Message-ID: <[email protected]>
Resent-Sender: [email protected]
X-Debian-PR-Message: followup 1002056
X-Debian-PR-Package: wnpp
X-Debian-PR-Keywords: pending
References: <[email protected]> <[email protected]> <[email protected]> <[email protected]>
Received: via spool by [email protected] id=B1002056.1732274994900934
          (code B ref 1002056); Fri, 22 Nov 2024 11:33:02 +0000
Received: (at 1002056) by bugs.debian.org; 22 Nov 2024 11:29:54 +0000
X-Spam-Checker-Version: SpamAssassin 3.4.6-bugs.debian.org_2005_01_02
	(2021-04-09) on buxtehude.debian.org
X-Spam-Level: 
X-Spam-Status: No, score=-107.3 required=4.0 tests=ALL_TRUSTED,BAYES_00,
	DKIMWL_WL_HIGH,DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,
	FOURLA,FROMDEVELOPER,SPF_HELO_NONE,SPF_NONE,USER_IN_DKIM_WELCOMELIST,
	USER_IN_DKIM_WHITELIST autolearn=ham autolearn_force=no
	version=3.4.6-bugs.debian.org_2005_01_02
X-Spam-Bayes: score:0.0000 Tokens: new, 17; hammy, 150; neutral, 393; spammy,
	0. spammytokens: hammytokens:0.000-+--trixie,
	0.000-+--Hx-spam-relays-external:36ff, 0.000-+--H*r:36ff,
	0.000-+--H*RT:sk:master., 0.000-+--H*RT:fe40
Received: from master.debian.org ([2001:41b8:202:deb:216:36ff:fe40:4001]:54518)
	from C=NA,ST=NA,L=Ankh Morpork,O=Debian SMTP,OU=Debian SMTP CA,CN=master.debian.org,[email protected] (verified)
	by buxtehude.debian.org with esmtps (TLS1.3:ECDHE_SECP256R1__RSA_PSS_RSAE_SHA256__AES_256_GCM:256)
	(Exim 4.94.2)
	(envelope-from <[email protected]>)
	id 1tERrG-003mN0-FW
	for [email protected]; Fri, 22 Nov 2024 11:29:54 +0000
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=debian.org;
	s=smtpauto.master; h=In-Reply-To:Content-Transfer-Encoding:Content-Type:
	MIME-Version:References:Message-ID:Subject:To:From:Date:Reply-To:Cc:
	Content-ID:Content-Description;
	bh=er6PgbG9NbSOR9r2chHyECxEaUOHh8MCHwtMbQm1FAY=; b=QDVJq9Na5SAYKBctIp+OHrgCzM
	xhNgX5y07A8nJNmbdlrR2HRlBefFVdIvNTA2bTrnoOM2in51wQfki5sxesZD0fT00DgV6hc1lBjBX
	RyswL5KTN/4ikdzD40rXqoQJbpkALvp+0Ze7s6mwdVCJMpdIe+1b9TssUzruNJZqfW2rj79bKtN9o
	WD5fSKEMa2EDB+m2VUnUHVvNPgKds5IkcTBaaXNYP2P97Ssm1jdMSYB0oe2g3l5lKn4SnrqIz9HZf
	caa6uV4mH89ifhx73BEEB/U35utgggVYEJtfFzgE7UnMr1P1MZUMd48KKpSNIUw2IbqeM88Gy5Yya
	4dobjuWA==;
Received: from guillem by master.debian.org with local (Exim 4.94.2)
	(envelope-from <[email protected]>)
	id 1tERrE-007ffH-NL; Fri, 22 Nov 2024 11:29:52 +0000
Date: Fri, 22 Nov 2024 12:29:51 +0100
From: Guillem Jover <[email protected]>
To: [email protected],
	Sebastian Andrzej Siewior <[email protected]>,
	David Heidelberg <[email protected]>, [email protected]
Message-ID: <[email protected]>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <[email protected]>

Hi!

[ I'll try to summarize the current discussion and status, what might
  be blockers, and a potential incremental way forward. ]

On Wed, 2024-09-25 at 10:48:50 +0200, Mark Brown wrote:
> On Tue, Sep 24, 2024 at 05:45:49PM +0200, Guillem Jover wrote:
> > On Tue, 2024-09-24 at 15:58:10 +0200, Mark Brown wrote:
> > > Obviously it's far too late to do anything with the default for trixie,
> > > we might want to evaluate doing something after the release but for now
> > > it's too late.  
> 
> > Personally I don't think it's too late, there should be several months
> > until the freeze, and I think if we wanted to switch we could perhaps
> > do a staged transition and see how it goes and only do the final
> > replacement if everything seems fine.
> 
> We do OTOH package more software than most distros on more architectures
> so we got a lot more exposure for testing coverage, and the revert would
> involve switching the entire implementation which complicates things a
> bit compared to a risky patch within a package.  I'm not totally
> opposed, and if everything goes smoothly we could definitely implement
> it within the timeframe, but it feels like an impactful change to
> introduce now not having considered it sooner.

True, also two months have passed since (that's on me!). At this time,
I'm now not sure whether it is feasible to consider such a switch, even
if there was agreement to do it. As it is, I think there are too many
unknowns!

> > (perhaps) exceeding changed circumstances. But given your mail, I'm
> > happy to work on this again and start with say uploading some initial
> > stuff into experimental for example, after this thread settles a bit?
> 
> > (I'll start by refreshing the packaging first though.)
> 
> Sure.

I did that, and the current WIP zlib-ng packaging provides now two
builds, one with the new native zng_* API and another (tentatively)
with the compat API/ABI one in libz-dev and libz1 binary packages.

I've tentatively chosen those package names for the compat libraries
to avoid having to go through NEW multiple times (with the assumption
that we'd either go ahead with the switch or the packages could then
simply be dropped). I think this should initially only be uploaded to
experimental, to avoid getting packages built with either zlib or
zlib-ng. But depending on the outcome of this discussion, I think other
(probably better) options would be to perhaps name the compat packages
something like libz-ng-compat*, or drop them completely?

WIP package at <https://git.hadrons.org/cgit/wip/debian/pkgs/zlib-ng.git/>.

> > > Does anyone have thoughts on this?
> 
> > Personally, I think fully migrating from zlib to zlib-ng would sound
> > great (even for trixie), but I guess we can take it slow if you do not
> > feel confident or have concerns over this.
> 
> > Also if you'd prefer to take over the zlib-ng ITP, as a continuation
> > of zlib, that'd seem fine with me too.
> 
> I'm fine with you carrying on with it (actually there is some slight
> non-technical complication for me with doing it myself), or we could
> also consider a packaging team.  I think there was some other interest
> in helping out but ICBW.  If you're packaging it I'm also more confident
> in letting you worry about how risky it is to transition and deal with
> any fallout!  :P

Ok, so after the feedback on this thread, and Sebastian asking how we
can proceed, here are the concerns brought on this thread, along my
own and things I think we need to check or consider:

  * There were concerns (from Fay) about whether given same input the
    output changes per arch or hw setup, we'd need to check this; I'd
    expect this not to be the case for different arches, but it might
    be an issue with number of cores for example, but if either is true
    this would be a serious blocker.
  * I've had concerns both about providing the zlib compat API and the
    native zlib-ng API in sid, and then getting a mess of packages
    linking against (true) zlib and against (native) zlib-ng, or
    packages relying on specific behaviors from either and breaking
    when switching from (true) zlib to zlib-ng-compat or vice versa,
    for example.
  * There were concerns (from Fay) about the output stream changing due
    to a potential implementation switch and that affecting external
    reproducibility. Personally I think while I can see how this is
    annoying for the involved parties, it's part of the "you need
    the same tools to generate the same output" premise that we also
    assume in Debian. I guess keeping both implementations around
    indefinitely, I think, would make this less of an issue, with the
    potential drawbacks mentioned in the previous point.
  * There was a concern (from Konstantin) about at least one known
    upstream (Angie) misbehaving with zlib-ng generated streams.
  * There were concerns (from Mark) that even though projects like
    Fedora have done such switch, we have way more packages and
    architectures, so we might see more fallout that has not already
    been handled.
  * There was a concern (from Mike) about whether the performance gain
    at the cost of stream size makes sense, given that the compression
    level could be reduced instead to similar effect (?). I'm not sure
    how these compare, so it would be interesting to analyze this,
    because perhaps that's a less traumatic way to look at it (but that
    might require redefining compression level semantics globally in
    zlib, or patching users, with neither look very enticing options).
    My perception from when I tested it is that the speed up was
    significant enough and the size increase not so much, but… In any
    case switching to zlib-ng upstream would also imply other benefits,
    like (supposedly) a more responsive upstream with more frequent
    releases, the new native API, and an implementation other
    distributions are switching to.
  * Some upstreams have started to use the zlib-ng native API, so
    regardless of whether we plan a switch or not, I guess packaging
    zlib-ng (w/ or w/ the compat API) might still make sense.
  * To consider a switch we'd need to do a mass rebuild of the
    archive. Ideally running autopkgtests and similar to exercise the
    packages?


After having written the above, and if Mark agrees, I think I'd opt for
uploading zlib-ng to experimental, with the compat packages renamed to
libz-ng-compat* or similar (even if that implies later on another trip
through NEW if we want to perform a full switch), because that might
make it easier to move them to sid as a way less disruptive change,
even if we decide not to switch the default zlib implementation.

OTOH and unfortunately I don't think I'm currently prepared to drive any
of what I think might be required mass archive rebuilds and testing or
the analysis mentioned above.

Thanks,
Guillem

Send a report that this bug log contains spam.