Debian Bug report logs - #865541
sbuild --apt-distupgrade should not remove build-essential

Reply or subscribe to this bug.

Display info messages

Message #5 received at [email protected] (full text, mbox, reply):

From: Raphaël Hertzog <[email protected]>

To: Debian Bug Tracking System <[email protected]>

Subject: sbuild --apt-distupgrade should not remove build-essential

Date: Thu, 22 Jun 2017 16:51:56 +0200

Package: sbuild
Version: 0.73.0-4
Severity: important
User: [email protected]
Usertags: origin-kali

The default configuration of sbuild includes a run of "apt-get
dist-upgrade" to upgrade the chroot. Unfortunately that run can break
the chroot in some cases. This happens from time to time in Kali because
we have forked linux (which builds linux-libc-dev) but not glibc (which
builds libc6-dev). libc6-dev embeds a >= dependency on linux-libc-dev on
the version on which it was built against and is thus non-installable in
Kali until we have updated the kernel...

But to update the kernel, we need a working sbuild... and it's no longer
working because it has done this:

+------------------------------------------------------------------------------+
| Update chroot                                                                |
+------------------------------------------------------------------------------+

Get:1 http://repo.kali.org/kali kali-dev InRelease [30.4 kB]
Get:2 http://repo.kali.org/kali kali-dev-only InRelease [23.1 kB]
Get:3 http://repo.kali.org/kali kali-dev/main Sources [11.4 MB]
Get:4 http://repo.kali.org/kali kali-dev/main i386 Packages [15.5 MB]
Get:5 http://repo.kali.org/kali kali-dev-only/main Sources [194 kB]
Get:6 http://repo.kali.org/kali kali-dev-only/main i386 Packages [212 kB]
Fetched 27.3 MB in 1s (13.8 MB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
Calculating upgrade...
The following packages will be REMOVED:
  build-essential g++ g++-6 libc6-dev libstdc++-6-dev
The following packages will be upgraded:
  libc-bin libc-dev-bin libc6 multiarch-support

[...]

I'm not sure what the best fix is. I don't know of any way to tell APT
to not remove build-essential while at the same time allowing to upgrade it.

Another solution is to run it with "-s" (simulate mode) and only run the
command if it doesn't remove build-essential.

Or another solution is possibly to first upgrade build-essential, put it on
hold and then run dist-upgrade.

Cheers,

-- System Information:
Debian Release: 9.0
  APT prefers oldoldstable
  APT policy: (500, 'oldoldstable'), (500, 'unstable'), (500, 'oldstable'), (1, 'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 4.9.0-3-amd64 (SMP w/4 CPU cores)
Locale: LANG=fr_FR.UTF-8, LC_CTYPE=fr_FR.UTF-8 (charmap=UTF-8), LANGUAGE=fr_FR.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages sbuild depends on:
ii  adduser         3.115
ii  libsbuild-perl  0.73.0-4
ii  perl            5.24.1-4

Versions of packages sbuild recommends:
ii  autopkgtest  4.4
ii  debootstrap  1.0.90
ii  schroot      1.6.10-4

Versions of packages sbuild suggests:
pn  deborphan  <none>
ii  kmod       24-1
ii  wget       1.19.1-3

-- no debconf information

Message #8 received at [email protected] (full text, mbox, reply):

From: Johannes Schauer <[email protected]>

To: [email protected], Raphaël Hertzog <[email protected]>

Subject: Re: [buildd-tools-devel] Bug#865541: sbuild --apt-distupgrade should not remove build-essential

Date: Fri, 23 Jun 2017 12:41:40 +0200

[Message part 1 (text/plain, inline)]

Hi Raphaël,

Quoting Raphaël Hertzog (2017-06-22 16:51:56)
> The default configuration of sbuild includes a run of "apt-get dist-upgrade"
> to upgrade the chroot. Unfortunately that run can break the chroot in some
> cases. This happens from time to time in Kali because we have forked linux
> (which builds linux-libc-dev) but not glibc (which builds libc6-dev).
> libc6-dev embeds a >= dependency on linux-libc-dev on the version on which it
> was built against and is thus non-installable in Kali until we have updated
> the kernel...
> 
> But to update the kernel, we need a working sbuild... and it's no longer
> working because it has done this:
> 
> +------------------------------------------------------------------------------+
> | Update chroot                                                                |
> +------------------------------------------------------------------------------+
> 
> Get:1 http://repo.kali.org/kali kali-dev InRelease [30.4 kB]
> Get:2 http://repo.kali.org/kali kali-dev-only InRelease [23.1 kB]
> Get:3 http://repo.kali.org/kali kali-dev/main Sources [11.4 MB]
> Get:4 http://repo.kali.org/kali kali-dev/main i386 Packages [15.5 MB]
> Get:5 http://repo.kali.org/kali kali-dev-only/main Sources [194 kB]
> Get:6 http://repo.kali.org/kali kali-dev-only/main i386 Packages [212 kB]
> Fetched 27.3 MB in 1s (13.8 MB/s)
> Reading package lists...
> Reading package lists...
> Building dependency tree...
> Reading state information...
> Calculating upgrade...
> The following packages will be REMOVED:
>   build-essential g++ g++-6 libc6-dev libstdc++-6-dev
> The following packages will be upgraded:
>   libc-bin libc-dev-bin libc6 multiarch-support
> 
> [...]
> 
> I'm not sure what the best fix is. I don't know of any way to tell APT
> to not remove build-essential while at the same time allowing to upgrade it.
> 
> Another solution is to run it with "-s" (simulate mode) and only run the
> command if it doesn't remove build-essential.
> 
> Or another solution is possibly to first upgrade build-essential, put it on
> hold and then run dist-upgrade.

either the subject is not well picked or I'm still misunderstanding the issue
here. As I understand it, the problem is not that sbuild removes
build-essential. I understand that what you would like sbuild to be able to do
is to ignore certain dependency relationships between packages. Is my
understanding correct?

In the mean time, why don't you run sbuild without running "apt-get
dist-upgrade"? You can do that by adding --no-apt-distupgrade to your sbuild
invocation or by adding

$apt_distupgrade = 0;

to your ~/.sbuildrc.

Could you expand on your issue so that I can understand what exactly it is that
sbuild is either doing wrong or what you would like sbuild to be able to do?

Thanks!

cheers, josch

[signature.asc (application/pgp-signature, inline)]

Message #13 received at [email protected] (full text, mbox, reply):

From: Raphael Hertzog <[email protected]>

To: Johannes Schauer <[email protected]>

Cc: [email protected]

Subject: Re: [buildd-tools-devel] Bug#865541: sbuild --apt-distupgrade should not remove build-essential

Date: Fri, 23 Jun 2017 15:17:50 +0200

Hi,

On Fri, 23 Jun 2017, Johannes Schauer wrote:
> > Or another solution is possibly to first upgrade build-essential, put it on
> > hold and then run dist-upgrade.
> 
> either the subject is not well picked or I'm still misunderstanding the issue
> here. As I understand it, the problem is not that sbuild removes
> build-essential. I understand that what you would like sbuild to be able to do
> is to ignore certain dependency relationships between packages. Is my
> understanding correct?

No, I just want the integrated "dist-upgrade" feature to not break the
build chroot. My definition of "breaking the build chroot" includes
"removing build-essential".

It happens that the repository that I was using had a broken libc6-dev
(until I updated linux-libc-dev to a newer version) but since libc6-dev
was already installed in the build chroot, sbuild should be able to build
the package anyway and just be happy to keep libc6 on hold instead of
force-upgrading it by removing build-essential.

> In the mean time, why don't you run sbuild without running "apt-get
> dist-upgrade"? You can do that by adding --no-apt-distupgrade to your sbuild
> invocation or by adding
> 
> $apt_distupgrade = 0;
> 
> to your ~/.sbuildrc.

This works only if you are able to anticipate the problem. Once libc6-dev
has been removed from the build chroot, you can't reinstall it as the version
available in the repository is broken. So you are stuck and must somehow
manually downgrade to another working version.

Note also that we are speaking of build daemons where our usual
interaction is just to upload a package... temporarily modifying the
sbuild configuration of the build daemon is painful.

> Could you expand on your issue so that I can understand what exactly it is that
> sbuild is either doing wrong or what you would like sbuild to be able to do?

I would "sbuild-update -d" to ensure that it doesn't remove
build-essential (assuming that's what sbuild calls when we use
--apt-distupgrade).

Cheers,
-- 
Raphaël Hertzog ◈ Debian Developer

Support Debian LTS: https://www.freexian.com/services/debian-lts.html
Learn to master Debian: https://debian-handbook.info/get/

Message #16 received at [email protected] (full text, mbox, reply):

From: Johannes Schauer <[email protected]>

To: Raphael Hertzog <[email protected]>

Cc: [email protected]

Subject: Re: [buildd-tools-devel] Bug#865541: sbuild --apt-distupgrade should not remove build-essential

Date: Sun, 25 Jun 2017 06:36:28 +0200

[Message part 1 (text/plain, inline)]

Hi Raphaël,

Quoting Raphael Hertzog (2017-06-23 15:17:50)
> No, I just want the integrated "dist-upgrade" feature to not break the
> build chroot. My definition of "breaking the build chroot" includes
> "removing build-essential".

right, that makes sense. Sbuild should never remove the build-essential package
from the chroot.

It should at least error out when it accidentally does remove it.

> It happens that the repository that I was using had a broken libc6-dev (until
> I updated linux-libc-dev to a newer version) but since libc6-dev was already
> installed in the build chroot, sbuild should be able to build the package
> anyway and just be happy to keep libc6 on hold instead of force-upgrading it
> by removing build-essential.

Yes, but by having the APT_DISTUPGRADE configuration option set to 1 (the
default), the user instructed sbuild to do a "apt-get distupgrade" before the
build. Imagine a user who not just implicitly (through the default) but
explicitly (for example via the ~/.sbuildrc) instructed sbuild to do a
distupgrade but then sbuild doesn't do that and some packages don't get
upgraded. I think as a user explicitly requiring a distupgrade I would expect
sbuild to fail if it cannot perform a distupgrade in a way that doesn't remove
an important package like build-essential. So I don't think it would be a good
solution to add code which by default does only part of a distupgrade.

> This works only if you are able to anticipate the problem. Once libc6-dev has
> been removed from the build chroot, you can't reinstall it as the version
> available in the repository is broken. So you are stuck and must somehow
> manually downgrade to another working version.
> 
> Note also that we are speaking of build daemons where our usual
> interaction is just to upload a package... temporarily modifying the
> sbuild configuration of the build daemon is painful.

Your problem sounds like a bootstrapping problem. You have a build-dependency
cycle between two source packages where each package needs each other to be
built. Thus, would a stage1 package built using build profiles not be able to
solve your situation?

Also, coming back to your first email (which I think I now understand better)

> This happens from time to time in Kali because we have forked linux (which
> builds linux-libc-dev) but not glibc (which builds libc6-dev).  libc6-dev
> embeds a >= dependency on linux-libc-dev on the version on which it was built
> against and is thus non-installable in Kali until we have updated the
> kernel...

So you are building src:glibc using a non-kali kernel?

Why is a >= dependency a problem? Isn't your fork of src:linux not having a
higher version number than the version it comes from?

> > Could you expand on your issue so that I can understand what exactly it is
> > that sbuild is either doing wrong or what you would like sbuild to be able
> > to do?
> 
> I would "sbuild-update -d" to ensure that it doesn't remove
> build-essential (assuming that's what sbuild calls when we use
> --apt-distupgrade).

Assuring to not remove build-essential is a good idea but it wouldn't solve
your problem if I would just ensure this by throwing an error if apt removes
build-essential because it wants to dist-upgrade.

I'm hesitant to add a --dont-distupgrade-if-removes-buildessential switch to
the command line options until I'm sure that there is no better way.

Thanks!

cheers, josch

[signature.asc (application/pgp-signature, inline)]

Message #21 received at [email protected] (full text, mbox, reply):

From: Raphael Hertzog <[email protected]>

To: Johannes Schauer <[email protected]>

Cc: [email protected]

Subject: Re: [buildd-tools-devel] Bug#865541: sbuild --apt-distupgrade should not remove build-essential

Date: Sun, 25 Jun 2017 22:32:41 +0200

Hi,

On Sun, 25 Jun 2017, Johannes Schauer wrote:
> > It happens that the repository that I was using had a broken libc6-dev (until
> > I updated linux-libc-dev to a newer version) but since libc6-dev was already
> > installed in the build chroot, sbuild should be able to build the package
> > anyway and just be happy to keep libc6 on hold instead of force-upgrading it
> > by removing build-essential.
> 
> Yes, but by having the APT_DISTUPGRADE configuration option set to 1 (the
> default), the user instructed sbuild to do a "apt-get distupgrade" before the
> build. Imagine a user who not just implicitly (through the default) but
> explicitly (for example via the ~/.sbuildrc) instructed sbuild to do a
> distupgrade but then sbuild doesn't do that and some packages don't get
> upgraded. I think as a user explicitly requiring a distupgrade I would expect
> sbuild to fail if it cannot perform a distupgrade in a way that doesn't remove
> an important package like build-essential. So I don't think it would be a good
> solution to add code which by default does only part of a distupgrade.

Then it would make sense to not have APT_DISTUPGRADE=1 by default. Having sbuild
refusing to build in that situation is not really helpful either.

To me it seems perfectly fine to just emit a warning saying that the
distupgrade has not been made because it would have broken the chroot.

> > Note also that we are speaking of build daemons where our usual
> > interaction is just to upload a package... temporarily modifying the
> > sbuild configuration of the build daemon is painful.
> 
> Your problem sounds like a bootstrapping problem. You have a build-dependency
> cycle between two source packages where each package needs each other to be
> built. Thus, would a stage1 package built using build profiles not be able to
> solve your situation?

Huh... not really, I think you misunderstood, cf below.

> > This happens from time to time in Kali because we have forked linux (which
> > builds linux-libc-dev) but not glibc (which builds libc6-dev).  libc6-dev
> > embeds a >= dependency on linux-libc-dev on the version on which it was built
> > against and is thus non-installable in Kali until we have updated the
> > kernel...
> 
> So you are building src:glibc using a non-kali kernel?

I'm not building src:glibc, I'm importing it from Debian in kali-dev and
the run I run britney to create kali-rolling to ensure that broken
dependencies are detected and are not part of kali-rolling.

> Why is a >= dependency a problem? Isn't your fork of src:linux not having a
> higher version number than the version it comes from?

My fork of linux has a higher version number than in Debian, except that
updating the fork takes a bit of time and when glibc and the kernel
migrate together in testing (that's what we track) I get the updated glibc
immediately (because it's not forked) and we update our out-of-sync
package like once a week (or sooner when we see it creates migrations
problems).

> > I would "sbuild-update -d" to ensure that it doesn't remove
> > build-essential (assuming that's what sbuild calls when we use
> > --apt-distupgrade).
> 
> Assuring to not remove build-essential is a good idea but it wouldn't solve
> your problem if I would just ensure this by throwing an error if apt removes
> build-essential because it wants to dist-upgrade.

Indeed. I don't see why a warning would not be sufficient. Or doing it in
two steps "apt-get install build-essential, apt-mark hold build-essential,
apt-get dist-upgrade"

Cheers,
-- 
Raphaël Hertzog ◈ Debian Developer

Support Debian LTS: https://www.freexian.com/services/debian-lts.html
Learn to master Debian: https://debian-handbook.info/get/

Message #24 received at [email protected] (full text, mbox, reply):

From: Johannes Schauer <[email protected]>

To: Raphael Hertzog <[email protected]>

Cc: [email protected]

Subject: Re: [buildd-tools-devel] Bug#865541: sbuild --apt-distupgrade should not remove build-essential

Date: Mon, 26 Jun 2017 11:14:52 +0200

[Message part 1 (text/plain, inline)]

Hi,

Quoting Raphael Hertzog (2017-06-25 22:32:41)
> > > It happens that the repository that I was using had a broken libc6-dev
> > > (until I updated linux-libc-dev to a newer version) but since libc6-dev
> > > was already installed in the build chroot, sbuild should be able to build
> > > the package anyway and just be happy to keep libc6 on hold instead of
> > > force-upgrading it by removing build-essential.
> > Yes, but by having the APT_DISTUPGRADE configuration option set to 1 (the
> > default), the user instructed sbuild to do a "apt-get distupgrade" before
> > the build. Imagine a user who not just implicitly (through the default) but
> > explicitly (for example via the ~/.sbuildrc) instructed sbuild to do a
> > distupgrade but then sbuild doesn't do that and some packages don't get
> > upgraded. I think as a user explicitly requiring a distupgrade I would
> > expect sbuild to fail if it cannot perform a distupgrade in a way that
> > doesn't remove an important package like build-essential. So I don't think
> > it would be a good solution to add code which by default does only part of
> > a distupgrade.
> Then it would make sense to not have APT_DISTUPGRADE=1 by default. Having
> sbuild refusing to build in that situation is not really helpful either.

Just to make sure I understand you: changing the default would probably not fix
your situation either?

> To me it seems perfectly fine to just emit a warning saying that the
> distupgrade has not been made because it would have broken the chroot.

Maybe to you it does but I do not want a tool that I told to do X to go on and
do it stuff even though it failed at doing X. If I tell a tool to do X and it
is unable to, then it should fail.

> > So you are building src:glibc using a non-kali kernel?
> I'm not building src:glibc, I'm importing it from Debian in kali-dev and the
> run I run britney to create kali-rolling to ensure that broken dependencies
> are detected and are not part of kali-rolling.

I assume kali-dev and kali-rolling are suite names?

> > Why is a >= dependency a problem? Isn't your fork of src:linux not having a
> > higher version number than the version it comes from?
> My fork of linux has a higher version number than in Debian, except that
> updating the fork takes a bit of time and when glibc and the kernel migrate
> together in testing (that's what we track) I get the updated glibc
> immediately (because it's not forked) and we update our out-of-sync package
> like once a week (or sooner when we see it creates migrations problems).

Ah, but because the glibc you imported was not built with your kernel the
binary packages that end up in your repos have the "wrong" versioned
dependencies.

> > > I would "sbuild-update -d" to ensure that it doesn't remove
> > > build-essential (assuming that's what sbuild calls when we use
> > > --apt-distupgrade).
> > Assuring to not remove build-essential is a good idea but it wouldn't solve
> > your problem if I would just ensure this by throwing an error if apt
> > removes build-essential because it wants to dist-upgrade.
> Indeed. I don't see why a warning would not be sufficient. Or doing it in two
> steps "apt-get install build-essential, apt-mark hold build-essential,
> apt-get dist-upgrade"

But if you put a hold on build-essential, then the distupgrade will not be able
to install the packages it needs to succeed.

I'm still not sure what sbuild is supposed to do here to solve this problem.

Having "apt-get distupgrade" either silently (with just a warning but without
killing sbuild) fail or not do the full upgrade seems wrong to me because if
the user requested a distupgrade and cannot get one, then sbuild should abort.
Otherwise that would be like the user requesting to do a archall-only build but
if it cannot sbuild will just issue a warning instead of failing.

Implementing a check whether build-essential is to be or was removed sounds
useful but it will not solve your problem.

Another option would be to add a command line option
--allow-unsuccessful-distupgrade which would not distupgrade if it breaks the
chroot. But sbuild already has so many command line options and special
exceptions that I'm hesitant to add another one.

Or maybe, since you already forked src:linux, maybe it would be possible to let
the linux-libc-dev package be built with the version number from testing?

If you are really set on letting sbuild silently fail if "apt-get upgrade"
fails you could instruct your buildds to not do the distupgrade by default but
you do it manually using --chroot-setup-commands. Using that you can then
handle the situation however you like.

And does the problem not ultimately come from how kali imports packages from
testing? Should you not rebuild all reverse dependencies of packages that you
fork to make sure that the source packages you offer have been built against
the right binary packages? So would the problem not also be solved if you would
also rebuild src:glibc?

cheers, josch

[signature.asc (application/pgp-signature, inline)]

Message #29 received at [email protected] (full text, mbox, reply):

From: Raphael Hertzog <[email protected]>

To: Johannes Schauer <[email protected]>

Cc: [email protected]

Subject: Re: [buildd-tools-devel] Bug#865541: sbuild --apt-distupgrade should not remove build-essential

Date: Mon, 26 Jun 2017 11:58:04 +0200

Hi,

On Mon, 26 Jun 2017, Johannes Schauer wrote:
> > Then it would make sense to not have APT_DISTUPGRADE=1 by default. Having
> > sbuild refusing to build in that situation is not really helpful either.
> 
> Just to make sure I understand you: changing the default would probably not fix
> your situation either?

It would because actually I pass "--apt-update --apt-upgrade" but not
"--apt-distupgrade" and I have no ~/.sbuildrc.

But then it would likely have to do something else to update the build
choot. That said I don't think that changing the default is the right
answer here.

> > To me it seems perfectly fine to just emit a warning saying that the
> > distupgrade has not been made because it would have broken the chroot.
> 
> Maybe to you it does but I do not want a tool that I told to do X to go on and
> do it stuff even though it failed at doing X. If I tell a tool to do X and it
> is unable to, then it should fail.

When I call "sbuild" I want a package build, it should not fail if it is
able to build the package.

If I call sbuild-update --dist-upgrade, I might understand that you would
like the command to fail.

> > I'm not building src:glibc, I'm importing it from Debian in kali-dev and the
> > run I run britney to create kali-rolling to ensure that broken dependencies
> > are detected and are not part of kali-rolling.
> 
> I assume kali-dev and kali-rolling are suite names?

Yes.

> Ah, but because the glibc you imported was not built with your kernel the
> binary packages that end up in your repos have the "wrong" versioned
> dependencies.

Yes. But that's the kind of temporary dependencies problems that happen
from time to time in unstable too when an architecture's buildd(s)
lag(s) behind.

> > > > I would "sbuild-update -d" to ensure that it doesn't remove
> > > > build-essential (assuming that's what sbuild calls when we use
> > > > --apt-distupgrade).
> > > Assuring to not remove build-essential is a good idea but it wouldn't solve
> > > your problem if I would just ensure this by throwing an error if apt
> > > removes build-essential because it wants to dist-upgrade.
> > Indeed. I don't see why a warning would not be sufficient. Or doing it in two
> > steps "apt-get install build-essential, apt-mark hold build-essential,
> > apt-get dist-upgrade"
> 
> But if you put a hold on build-essential, then the distupgrade will not be able
> to install the packages it needs to succeed.

I'm afraid we don't have the same definition of "succeed". It's normal for
a dist-upgrade to keep some packages on hold when doing otherwise would
remove packages that the user wants to keep.

For end users, this happens all the time in unstable during large
transitions. You don't want to get the latest perl until all the perl
modules that you have installed have been rebuilt.

For a build chroot, we want the opposite... as long as it doesn't break
the build chroot. If the new perl causes the removal of dpkg-dev, then
it should not be dist-upgraded. If it causes the removal of
liblocale-gettext-perl only, then it's fine.

> Having "apt-get distupgrade" either silently (with just a warning but without
> killing sbuild) fail or not do the full upgrade seems wrong to me because if
> the user requested a distupgrade and cannot get one, then sbuild should abort.

Then make the --apt-distupgrade a tri-state option:
never/always/when-possible and make it default to "when-possible".

But to me it seems like useless complexity when 99,9% of the users just
want the build to succeed with the freshest working build environment that
we can get in the current situation.

> Otherwise that would be like the user requesting to do a archall-only build but
> if it cannot sbuild will just issue a warning instead of failing.

I don't understand the analogy. The archall-only build is really about
what we want, if we can't get what we want, it should fail. The
--apt-distupgrade and not about what we want, it's about the environement,
it's more in the realm of the "how".

And while you interpret the option literaly as "the user wants a dist-upgrade"
I believe that it should be understood as "I want a package build in the
latest build environement available".

> Or maybe, since you already forked src:linux, maybe it would be possible to let
> the linux-libc-dev package be built with the version number from testing?

The problematic dependency is on libc6-dev and I don't control that one.

> If you are really set on letting sbuild silently fail if "apt-get upgrade"
> fails you could instruct your buildds to not do the distupgrade by default but
> you do it manually using --chroot-setup-commands. Using that you can then
> handle the situation however you like.

I have plenty of ways to hack a solution but if I file a bug report, it's
because I believe that it is a shortcoming that should be fixed for all,
not only for my specific case.

> And does the problem not ultimately come from how kali imports packages from
> testing? Should you not rebuild all reverse dependencies of packages that you
> fork to make sure that the source packages you offer have been built against
> the right binary packages? So would the problem not also be solved if you would
> also rebuild src:glibc?

Such additional work would solve this issue it would also multiply the
amount of work by ten (i.e. much more than having to deal with this
specific hiccup) and it would offer no benefits for the end users.

Cheers,
-- 
Raphaël Hertzog ◈ Debian Developer

Support Debian LTS: https://www.freexian.com/services/debian-lts.html
Learn to master Debian: https://debian-handbook.info/get/

Message #32 received at [email protected] (full text, mbox, reply):

From: Johannes Schauer <[email protected]>

To: Raphael Hertzog <[email protected]>

Cc: [email protected]

Subject: Re: [buildd-tools-devel] Bug#865541: sbuild --apt-distupgrade should not remove build-essential

Date: Mon, 26 Jun 2017 13:26:20 +0200

[Message part 1 (text/plain, inline)]

Hi,

Quoting Raphael Hertzog (2017-06-26 11:58:04)
> On Mon, 26 Jun 2017, Johannes Schauer wrote:
> > > Then it would make sense to not have APT_DISTUPGRADE=1 by default. Having
> > > sbuild refusing to build in that situation is not really helpful either.
> > Just to make sure I understand you: changing the default would probably not
> > fix your situation either?
> It would because actually I pass "--apt-update --apt-upgrade" but not
> "--apt-distupgrade" and I have no ~/.sbuildrc.

then why not pass --no-apt-distupgrade?

> But then it would likely have to do something else to update the build
> choot. That said I don't think that changing the default is the right
> answer here.

Yes, you would have to distupgrade the buildchroot outside of sbuild using
"sbuild-update" for example or using an external command hook like
--chroot-setup-commands.

> If I call sbuild-update --dist-upgrade, I might understand that you would
> like the command to fail.

Unfortunately, currently sbuild makes no distinction between how a
configuration variable was set, whether it was via the sbuildrc, a commandline
switch, an environment variable or whether the user changed nothing and it was
just the default.

Adding provisions that would let sbuild know who set a variable and then
letting sbuild behave differently depending on how the variable was set would
be a big effort.

> > > Indeed. I don't see why a warning would not be sufficient. Or doing it in
> > > two steps "apt-get install build-essential, apt-mark hold
> > > build-essential, apt-get dist-upgrade"
> > But if you put a hold on build-essential, then the distupgrade will not be
> > able to install the packages it needs to succeed.
> I'm afraid we don't have the same definition of "succeed". It's normal for a
> dist-upgrade to keep some packages on hold when doing otherwise would remove
> packages that the user wants to keep.
> 
> For end users, this happens all the time in unstable during large
> transitions. You don't want to get the latest perl until all the perl
> modules that you have installed have been rebuilt.
> 
> For a build chroot, we want the opposite... as long as it doesn't break
> the build chroot. If the new perl causes the removal of dpkg-dev, then
> it should not be dist-upgraded. If it causes the removal of
> liblocale-gettext-perl only, then it's fine.

Unfortunately, "apt-mark hold" has the nasty side effect that it not only stops
a package from being removed but it also stops a package from being upgraded.
So if a distupgrade were necessary to upgrade build-essential then we wouldn't
want it to be on hold or otherwise build-essential would not get upgraded. So I
don't think "apt-mark hold" is the right thing to do for sbuild in general.
(but it might be the right thing for you to do in which case you could use
--chroot-setup-commands)

> > Having "apt-get distupgrade" either silently (with just a warning but
> > without killing sbuild) fail or not do the full upgrade seems wrong to me
> > because if the user requested a distupgrade and cannot get one, then sbuild
> > should abort.
> Then make the --apt-distupgrade a tri-state option:
> never/always/when-possible and make it default to "when-possible".
> 
> But to me it seems like useless complexity when 99,9% of the users just
> want the build to succeed with the freshest working build environment that
> we can get in the current situation.

Do they? At least I would count myself to the other 0.1% but it's hard to find
out.

But making --apt-distupgrade a tristate option where in addition to
--no-apt-distupgrade we could have --maybe-apt-distupgrade would probably
indeed fix your issue.

> > Otherwise that would be like the user requesting to do a archall-only build
> > but if it cannot sbuild will just issue a warning instead of failing.
> 
> I don't understand the analogy. The archall-only build is really about
> what we want, if we can't get what we want, it should fail. The
> --apt-distupgrade and not about what we want, it's about the environement,
> it's more in the realm of the "how".
> 
> And while you interpret the option literaly as "the user wants a dist-upgrade"
> I believe that it should be understood as "I want a package build in the
> latest build environement available".

As explained above, with the way sbuild currently works there is no way for it
to know where the "I want a distupgrade" comes from. Whether it's from the
command line, the sbuildrc or whether it just picked the default. So with
things are as they are right now, having an option set can only be interpreted
as "the user wants it" independent of whether the user really manually set
something on the command line, made an entry in the sbuildrc or whether the
default was used and the user just "implicitly" chose a certain option.

> > If you are really set on letting sbuild silently fail if "apt-get upgrade"
> > fails you could instruct your buildds to not do the distupgrade by default
> > but you do it manually using --chroot-setup-commands. Using that you can
> > then handle the situation however you like.
> I have plenty of ways to hack a solution but if I file a bug report, it's
> because I believe that it is a shortcoming that should be fixed for all, not
> only for my specific case.

That's very much appreciated, thank you! I feel the same, which is why I try to
find a good solution together with you. Sorry if my arguing comes over as very
dismissive - it is not meant to be. I think much of my irritation just came
from me not exactly having understood your problem. But I think we are making
progress there. :)

> > And does the problem not ultimately come from how kali imports packages
> > from testing? Should you not rebuild all reverse dependencies of packages
> > that you fork to make sure that the source packages you offer have been
> > built against the right binary packages? So would the problem not also be
> > solved if you would also rebuild src:glibc?
> Such additional work would solve this issue it would also multiply the amount
> of work by ten (i.e. much more than having to deal with this specific hiccup)
> and it would offer no benefits for the end users.

Makes sense.

Thanks!

cheers, josch

[signature.asc (application/pgp-signature, inline)]

Message #37 received at [email protected] (full text, mbox, reply):

From: Raphael Hertzog <[email protected]>

To: Johannes Schauer <[email protected]>

Cc: [email protected]

Subject: Re: [buildd-tools-devel] Bug#865541: sbuild --apt-distupgrade should not remove build-essential

Date: Mon, 26 Jun 2017 15:00:28 +0200

Hi,

On Mon, 26 Jun 2017, Johannes Schauer wrote:
> > It would because actually I pass "--apt-update --apt-upgrade" but not
> > "--apt-distupgrade" and I have no ~/.sbuildrc.
> 
> then why not pass --no-apt-distupgrade?

Because when I wrote that script, I was probably not aware that
--apt-distupgrade was the default.

> > If I call sbuild-update --dist-upgrade, I might understand that you would
> > like the command to fail.
> 
> Unfortunately, currently sbuild makes no distinction between how a
> configuration variable was set, whether it was via the sbuildrc, a commandline
> switch, an environment variable or whether the user changed nothing and it was
> just the default.
> 
> Adding provisions that would let sbuild know who set a variable and then
> letting sbuild behave differently depending on how the variable was set would
> be a big effort.

I understand this. But you can react differently to the failure in the
various places where you are calling "$resolver->distupgrade".

> > For a build chroot, we want the opposite... as long as it doesn't break
> > the build chroot. If the new perl causes the removal of dpkg-dev, then
> > it should not be dist-upgraded. If it causes the removal of
> > liblocale-gettext-perl only, then it's fine.
> 
> Unfortunately, "apt-mark hold" has the nasty side effect that it not only stops
> a package from being removed but it also stops a package from being upgraded.

Hence we start with "apt-get install build-essential" which does upgrade
it before we mark it back on hold.

> > But to me it seems like useless complexity when 99,9% of the users just
> > want the build to succeed with the freshest working build environment that
> > we can get in the current situation.
> 
> Do they? At least I would count myself to the other 0.1% but it's hard to find
> out.

Yeah, I don't plan to run a survey on #debian-devel but that's my
feeling. :)

Cheers,
-- 
Raphaël Hertzog ◈ Debian Developer

Support Debian LTS: https://www.freexian.com/services/debian-lts.html
Learn to master Debian: https://debian-handbook.info/get/

Message #40 received at [email protected] (full text, mbox, reply):

From: Johannes Schauer <[email protected]>

To: Raphael Hertzog <[email protected]>

Cc: [email protected]

Subject: Re: [buildd-tools-devel] Bug#865541: sbuild --apt-distupgrade should not remove build-essential

Date: Mon, 26 Jun 2017 15:29:54 +0200

[Message part 1 (text/plain, inline)]

Hi,

Quoting Raphael Hertzog (2017-06-26 15:00:28)
> On Mon, 26 Jun 2017, Johannes Schauer wrote:
> > > It would because actually I pass "--apt-update --apt-upgrade" but not
> > > "--apt-distupgrade" and I have no ~/.sbuildrc.
> > then why not pass --no-apt-distupgrade?
> Because when I wrote that script, I was probably not aware that
> --apt-distupgrade was the default.

so does the existence of that option help your use-case somehow?

> > > If I call sbuild-update --dist-upgrade, I might understand that you would
> > > like the command to fail.
> > Unfortunately, currently sbuild makes no distinction between how a
> > configuration variable was set, whether it was via the sbuildrc, a
> > commandline switch, an environment variable or whether the user changed
> > nothing and it was just the default.
> > 
> > Adding provisions that would let sbuild know who set a variable and then
> > letting sbuild behave differently depending on how the variable was set would
> > be a big effort.
> 
> I understand this. But you can react differently to the failure in the
> various places where you are calling "$resolver->distupgrade".

What do you mean exactly here?

> > > For a build chroot, we want the opposite... as long as it doesn't break
> > > the build chroot. If the new perl causes the removal of dpkg-dev, then it
> > > should not be dist-upgraded. If it causes the removal of
> > > liblocale-gettext-perl only, then it's fine.
> > 
> > Unfortunately, "apt-mark hold" has the nasty side effect that it not only stops
> > a package from being removed but it also stops a package from being upgraded.
> Hence we start with "apt-get install build-essential" which does upgrade it
> before we mark it back on hold.

Ah right, you indeed said this in your first mail.

So how do you want to solve the issue now for kali? By just passing
--no-apt-distupgrade and upgrading the chroots manually from time to time like
the Debian and Ubuntu buildds do it?

Thanks!

cheers, josch

[signature.asc (application/pgp-signature, inline)]

Message #45 received at [email protected] (full text, mbox, reply):

From: Raphael Hertzog <[email protected]>

To: Johannes Schauer <[email protected]>

Cc: [email protected]

Subject: Re: [buildd-tools-devel] Bug#865541: sbuild --apt-distupgrade should not remove build-essential

Date: Mon, 26 Jun 2017 20:22:01 +0200

On Mon, 26 Jun 2017, Johannes Schauer wrote:
> > Because when I wrote that script, I was probably not aware that
> > --apt-distupgrade was the default.
> 
> so does the existence of that option help your use-case somehow?

I modified Kali's build script to pass it, yes. But my request remains.

> > I understand this. But you can react differently to the failure in the
> > various places where you are calling "$resolver->distupgrade".
> 
> What do you mean exactly here?

I mean that you can turn it into a warning in Sbuild/Build.pm and an
error into bin/sbuild-update.

> So how do you want to solve the issue now for kali? By just passing
> --no-apt-distupgrade and upgrading the chroots manually from time to time like
> the Debian and Ubuntu buildds do it?

That's what I will do, yes. But I believe that this should be avoidable
and letting sbuild dist-upgrade the chroot should have been enough and
should not have the current downsides.

Cheers,
-- 
Raphaël Hertzog ◈ Debian Developer

Support Debian LTS: https://www.freexian.com/services/debian-lts.html
Learn to master Debian: https://debian-handbook.info/get/

Message #48 received at [email protected] (full text, mbox, reply):

From: Johannes Schauer <[email protected]>

To: Raphael Hertzog <[email protected]>

Cc: [email protected]

Subject: Re: [buildd-tools-devel] Bug#865541: sbuild --apt-distupgrade should not remove build-essential

Date: Tue, 27 Jun 2017 06:56:29 +0200

[Message part 1 (text/plain, inline)]

Hi,

Quoting Raphael Hertzog (2017-06-26 20:22:01)
> On Mon, 26 Jun 2017, Johannes Schauer wrote:
> > > Because when I wrote that script, I was probably not aware that
> > > --apt-distupgrade was the default.
> > 
> > so does the existence of that option help your use-case somehow?
> 
> I modified Kali's build script to pass it, yes. But my request remains.
> 
> > > I understand this. But you can react differently to the failure in the
> > > various places where you are calling "$resolver->distupgrade".
> > 
> > What do you mean exactly here?
> 
> I mean that you can turn it into a warning in Sbuild/Build.pm and an
> error into bin/sbuild-update.
> 
> > So how do you want to solve the issue now for kali? By just passing
> > --no-apt-distupgrade and upgrading the chroots manually from time to time like
> > the Debian and Ubuntu buildds do it?
> 
> That's what I will do, yes. But I believe that this should be avoidable
> and letting sbuild dist-upgrade the chroot should have been enough and should
> not have the current downsides.

If you agree, then I will see this bugreport as a feature request to implement
the following:

Let APT_DISTUPGRADE be a tristate option: 0, 1, 2

In addition to --apt-distupgrade and --no-apt-distupgrade add
--maybe-apt-distupgrade.

In case of --apt-distupgrade, fail if sbuild removes build-essential.

In case of --maybe-apt-distupgrade, allow "partial" distupgrades by putting
build-essential on hold before distupgrading.

Please signal your agreement to this plan by lowering the severity of this bug
to wishlist.

Thanks!

cheers, josch

[signature.asc (application/pgp-signature, inline)]

Message #53 received at [email protected] (full text, mbox, reply):

From: Raphael Hertzog <[email protected]>

To: Johannes Schauer <[email protected]>

Cc: [email protected]

Subject: Re: [buildd-tools-devel] Bug#865541: sbuild --apt-distupgrade should not remove build-essential

Date: Tue, 27 Jun 2017 14:18:47 +0200

Control: severity -1 wishlist

On Tue, 27 Jun 2017, Johannes Schauer wrote:
> If you agree, then I will see this bugreport as a feature request to implement
> the following:
> 
> Let APT_DISTUPGRADE be a tristate option: 0, 1, 2
> 
> In addition to --apt-distupgrade and --no-apt-distupgrade add
> --maybe-apt-distupgrade.
> 
> In case of --apt-distupgrade, fail if sbuild removes build-essential.
> 
> In case of --maybe-apt-distupgrade, allow "partial" distupgrades by putting
> build-essential on hold before distupgrading.
> 
> Please signal your agreement to this plan by lowering the severity of this bug
> to wishlist.

Fine for me.

Cheers,
-- 
Raphaël Hertzog ◈ Debian Developer

Support Debian LTS: https://www.freexian.com/services/debian-lts.html
Learn to master Debian: https://debian-handbook.info/get/

Send a report that this bug log contains spam.