Subject: installation of crypt.h in the multiarch ___location breaks the GCC and
LLVM multilib builds
Date: Thu, 30 Jun 2022 14:10:17 +0200
Package: libcrypt-dev
Version: 4.4.28-1
Severity: serious
installation of crypt.h in the multiarch ___location breaks the GCC and LLVM
multilib builds.
For libsanitizer, crypt.h is needed to determine the size of a struct, the
library itself is not needed. Moving it to the MA ___location makes it unavailable
for the non-multilib builds.
Unfortunately the changelog doesn't mention anything why it was moved.
So either it should be moved back to /usr/include, or we need multilib builds
for libxcrypt.
Acknowledgement sent
to Marco d'Itri <[email protected]>:
Extra info received and forwarded to list. Copy sent to Marco d'Itri <[email protected]>.
(Thu, 30 Jun 2022 12:24:02 GMT) (full text, mbox, link).
On Jun 30, Matthias Klose <[email protected]> wrote:
> installation of crypt.h in the multiarch ___location breaks the GCC and LLVM
> multilib builds.
>
> For libsanitizer, crypt.h is needed to determine the size of a struct, the
> library itself is not needed. Moving it to the MA ___location makes it
> unavailable for the non-multilib builds.
>
> Unfortunately the changelog doesn't mention anything why it was moved.
>
> So either it should be moved back to /usr/include, or we need multilib
> builds for libxcrypt.
It was discussed in #1004102 (and is documented in the git commit),
where Helmut was positive that this would not cause any issues. Helmut?
(Why can't we retire multilib for good?)
--
ciao,
Marco
Acknowledgement sent
to Helmut Grohne <[email protected]>:
Extra info received and forwarded to list. Copy sent to Marco d'Itri <[email protected]>.
(Thu, 30 Jun 2022 18:21:03 GMT) (full text, mbox, link).
Control: clone -1 -2
Control: reassign -2 src:gcc-12
Control: tags -2 = ftbfs
Control: retitle -2 gcc-12 FTBFS in unstable on amd64: ../../src/gcc/config/i386/i386.h:2356:10: fatal error: insn-attr-common.h: No such file or directory
Control: tags -1 + moreinfo
Hi Matthias and Marco,
On Thu, Jun 30, 2022 at 02:21:22PM +0200, Marco d'Itri wrote:
> On Jun 30, Matthias Klose <[email protected]> wrote:
>
> > installation of crypt.h in the multiarch ___location breaks the GCC and LLVM
> > multilib builds.
This report seems needlessly imprecise to me. It is not exactly clear
whether you refer to building the gcc package here or whether you refer
to building something with gcc. I've tried building a small example
with -m32 on amd64. That happened to be successful. I also tried
building gcc-12 on amd64 in unstable (with the new version of libcrypt).
That happened to fail in the following way:
| /<<PKGBUILDDIR>>/build/./prev-gcc/xg++ -B/<<PKGBUILDDIR>>/build/./prev-gcc/ -B/usr/x86_64-linux-gnu/bin/ -nostdinc++ -B/<<PKGBUILDDIR>>/build/prev-x86_64-linux-gnu/libstdc++-v3/src/.libs -B/<<PKGBUILDDIR>>/build/prev-x86_64-linux-gnu/libstdc++-v3/libsupc++/.libs -I/<<PKGBUILDDIR>>/build/prev-x86_64-linux-gnu/libstdc++-v3/include/x86_64-linux-gnu -I/<<PKGBUILDDIR>>/build/prev-x86_64-linux-gnu/libstdc++-v3/include -I/<<PKGBUILDDIR>>/src/libstdc++-v3/libsupc++ -L/<<PKGBUILDDIR>>/build/prev-x86_64-linux-gnu/libstdc++-v3/src/.libs -L/<<PKGBUILDDIR>>/build/prev-x86_64-linux-gnu/libstdc++-v3/libsupc++/.libs -fno-PIE -c -g -g -O2 -fchecking=1 -DIN_GCC -fno-exceptions -fno-rtti -fasynchronous-unwind-tables -W -Wall -Wno-narrowing -Wwrite-strings -Wcast-qual -DHAVE_CONFIG_H -I. -Im2/gm2-gcc -I../../src/gcc -I../../src/gcc/m2/gm2-gcc -I../../src/gcc/../include -I../../src/gcc/../libcpp/include -I../../src/gcc/../libcody -I../../src/gcc/../libdecnumber -I../../src/gcc/../libdecnumber/bid -I../libdecnumber -I../../src/gcc/../libbacktrace -I. -Im2/gm2-gcc -I../../src/gcc -I../../src/gcc/m2/gm2-gcc -I../../src/gcc/../include -I../../src/gcc/../libcpp/include -I../../src/gcc/../libcody -I../../src/gcc/../libdecnumber -I../../src/gcc/../libdecnumber/bid -I../libdecnumber -I../../src/gcc/../libbacktrace ../../src/gcc/m2/gm2-gcc/m2convert.cc -o m2/gm2-gcc/m2convert.o
...
| In file included from ./tm.h:26,
| from ../../src/gcc/backend.h:28,
| from ../../src/gcc/m2/gm2-gcc/gcc-consolidation.h:26,
| from ../../src/gcc/m2/gm2-gcc/m2convert.cc:22:
| ../../src/gcc/config/i386/i386.h:2356:10: fatal error: insn-attr-common.h: No such file or directory
| 2356 | #include "insn-attr-common.h"
| | ^~~~~~~~~~~~~~~~~~~~
| compilation terminated.
| make[5]: *** [../../src/gcc/m2/Make-lang.in:600: m2/gm2-gcc/m2convert.o] Error 1
| make[5]: *** Waiting for unfinished jobs....
This looks unrelated to me given that no multilib flag is involved. On
the other hand, it talks about i386 in an amd64 build. I have no clue
how this failure could be attributed to crypt.h. It rather seems to be a
known issue and a classical FTBFS:
https://gcc.gnu.org/pipermail/gcc/2021-May/236101.html
I've attached the full build log to make report debuggable.
So I am unable to reproduce the reported issue with gcc.
> > For libsanitizer, crypt.h is needed to determine the size of a struct, the
> > library itself is not needed. Moving it to the MA ___location makes it
> > unavailable for the non-multilib builds.
Again, the lack of precision is not constructive. Linking to a failure
would help a lot here.
I assume that your statement about the MA ___location contains an
unintended negation. Is it correct that you meant to say that the MA
___location is unavailable to multilib (without "non-") builds? That would
make more sense to me.
> > Unfortunately the changelog doesn't mention anything why it was moved.
I am sorry for the lack of detail here. When moving headers the usual
reason is to avoid unpack errors. That's also the case here. crypt.h is
interpolated at build-time in an architecture-dependent way. Shipping it
on a non-multiarch path imposes a Multi-Arch: same violation.
> > So either it should be moved back to /usr/include, or we need multilib
> > builds for libxcrypt.
We cannot move it back for all architectures, because that would cause
unpack errors. In theory, we could move it back for multilib-enabled
architectures as long as none of them varies crypt.h (which actually
seems to be the case). I think this would be rather messy, but it seems
technically feasible and relatively simple.
That leaves the other option: adding multilib packages for libxcrypt.
I basically trust Matthias on the statement that something needs crypt.h
for some reason although I'd like to learn why precisely. In theory,
we'd like to make libcrypt-dev non-build-essential and if gcc requires
it, that's a no-brainer.
Assuming that, we basically have the two options above:
* Move crypt.h back for all multilib architectures only.
* Add multilib packages.
Marco, do you have any preference here?
> It was discussed in #1004102 (and is documented in the git commit),
> where Helmut was positive that this would not cause any issues. Helmut?
Indeed, I did not see this coming and I still am unable to reproduce the
reported problem.
> (Why can't we retire multilib for good?)
I would like to as well. A significant reason seems to be that doing so
would require cross-architecture dependencies to work with britney.
Helmut
Acknowledgement sent
to Marco d'Itri <[email protected]>:
Extra info received and forwarded to list.
(Sun, 03 Jul 2022 00:39:03 GMT) (full text, mbox, link).
On Jun 30, Helmut Grohne <[email protected]> wrote:
> Assuming that, we basically have the two options above:
> * Move crypt.h back for all multilib architectures only.
> * Add multilib packages.
>
> Marco, do you have any preference here?
I do not want to add any more complexity than what is strictly required
to support musl, which is not even a real port.
If moving crypt.h to the multiarch path only when building with musl is
a solution then let's do that.
--
ciao,
Marco
Acknowledgement sent
to Helmut Grohne <[email protected]>:
Extra info received and forwarded to list. Copy sent to Marco d'Itri <[email protected]>.
(Sun, 03 Jul 2022 09:15:03 GMT) (full text, mbox, link).
Hi Marco,
On Sun, Jul 03, 2022 at 02:37:27AM +0200, Marco d'Itri wrote:
> I do not want to add any more complexity than what is strictly required
> to support musl, which is not even a real port.
I note that this complexity is not due to musl. It is due to libcrypt2
regardless of the architecture. It just happens that musl is the first
user of libcrypt2. If we later decide to transition existing
architectures to libcrypt2, they'll be fully hit by this in the very
same way. So keep in mind: If you move to libcrypt2 you have a choice of
breaking multilib or multiarch or adding multlib packages.
> If moving crypt.h to the multiarch path only when building with musl is
> a solution then let's do that.
I'm attaching a patch for your convenience, but I'd prefer understanding
the actual problem before papering over it like this.
Helmut
Acknowledgement sent
to Helmut Grohne <[email protected]>:
Extra info received and forwarded to list. Copy sent to Marco d'Itri <[email protected]>.
(Mon, 04 Jul 2022 18:54:02 GMT) (full text, mbox, link).
Subject: Re: Bug#1014114: installation of crypt.h in the multiarch ___location
breaks the GCC and LLVM multilib builds
Date: Mon, 4 Jul 2022 20:51:05 +0200
Hi,
On Thu, Jun 30, 2022 at 05:15:32PM +0200, Helmut Grohne wrote:
> > > For libsanitizer, crypt.h is needed to determine the size of a struct, the
> > > library itself is not needed. Moving it to the MA ___location makes it
> > > unavailable for the non-multilib builds.
>
> Again, the lack of precision is not constructive. Linking to a failure
> would help a lot here.
I now understand that the libsanitizer aspect is separate to the
multilib aspect. Therefore, my proposed solution is a non-solution and
adding multilib packages is a non-solution as well. It is way worse and
completely messed up.
A gcc build (cross compiler stage3 or native) requires a target
architecture crypt.h. Trying to do without breaks libsanitizer (no
multilib involved). Example from
https://jenkins.debian.net/job/rebootstrap_ppc64_gcc12_nobiarch/5/consoleText
| /bin/bash ../libtool --tag=CXX --mode=compile /tmp/buildd/gcc3/gcc-12-12.1.0/build/./gcc/xgcc -shared-libgcc -B/tmp/buildd/gcc3/gcc-12-12.1.0/build/./gcc -nostdinc++ -L/tmp/buildd/gcc3/gcc-12-12.1.0/build/powerpc64-linux-gnu/libstdc++-v3/src -L/tmp/buildd/gcc3/gcc-12-12.1.0/build/powerpc64-linux-gnu/libstdc++-v3/src/.libs -L/tmp/buildd/gcc3/gcc-12-12.1.0/build/powerpc64-linux-gnu/libstdc++-v3/libsupc++/.libs -B/usr/powerpc64-linux-gnu/bin/ -B/usr/powerpc64-linux-gnu/lib/ -isystem /usr/powerpc64-linux-gnu/include -isystem /usr/powerpc64-linux-gnu/sys-include -isystem /tmp/buildd/gcc3/gcc-12-12.1.0/build/sys-include -D_GNU_SOURCE -D_DEBUG -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -DHAVE_RPC_XDR_H=0 -DHAVE_TIRPC_RPC_XDR_H=0 -I. -I../../../../src/libsanitizer/sanitizer_common -I.. -I ../../../../src/libsanitizer/include -I ../../../../src/libsanitizer -isystem ../../../../src/libsanitizer/include/system -Wall -W -Wno-unused-parameter -Wwrite-strings -pedantic -Wno-long-long -fPIC -fno-builtin -fno-exceptions -fno-rtti -fomit-frame-pointer -funwind-tables -fvisibility=hidden -Wno-variadic-macros -I../../libstdc++-v3/include -I../../libstdc++-v3/include/powerpc64-linux-gnu -I../../../../src/libsanitizer/../libstdc++-v3/libsupc++ -std=gnu++14 -DSANITIZER_LIBBACKTRACE -DSANITIZER_CP_DEMANGLE -I ../../../../src/libsanitizer/../libbacktrace -I ../libbacktrace -I ../../../../src/libsanitizer/../include -include ../../../../src/libsanitizer/libbacktrace/backtrace-rename.h -g -O2 -D_GNU_SOURCE -MT sanitizer_platform_limits_posix.lo -MD -MP -MF .deps/sanitizer_platform_limits_posix.Tpo -c -o sanitizer_platform_limits_posix.lo ../../../../src/libsanitizer/sanitizer_common/sanitizer_platform_limits_posix.cpp
| libtool: compile: /tmp/buildd/gcc3/gcc-12-12.1.0/build/./gcc/xgcc -shared-libgcc -B/tmp/buildd/gcc3/gcc-12-12.1.0/build/./gcc -nostdinc++ -L/tmp/buildd/gcc3/gcc-12-12.1.0/build/powerpc64-linux-gnu/libstdc++-v3/src -L/tmp/buildd/gcc3/gcc-12-12.1.0/build/powerpc64-linux-gnu/libstdc++-v3/src/.libs -L/tmp/buildd/gcc3/gcc-12-12.1.0/build/powerpc64-linux-gnu/libstdc++-v3/libsupc++/.libs -B/usr/powerpc64-linux-gnu/bin/ -B/usr/powerpc64-linux-gnu/lib/ -isystem /usr/powerpc64-linux-gnu/include -isystem /usr/powerpc64-linux-gnu/sys-include -isystem /tmp/buildd/gcc3/gcc-12-12.1.0/build/sys-include -D_GNU_SOURCE -D_DEBUG -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -DHAVE_RPC_XDR_H=0 -DHAVE_TIRPC_RPC_XDR_H=0 -I. -I../../../../src/libsanitizer/sanitizer_common -I.. -I ../../../../src/libsanitizer/include -I ../../../../src/libsanitizer -isystem ../../../../src/libsanitizer/include/system -Wall -W -Wno-unused-parameter -Wwrite-strings -pedantic -Wno-long-long -fPIC -fno-builtin -fno-exceptions -fno-rtti -fomit-frame-pointer -funwind-tables -fvisibility=hidden -Wno-variadic-macros -I../../libstdc++-v3/include -I../../libstdc++-v3/include/powerpc64-linux-gnu -I../../../../src/libsanitizer/../libstdc++-v3/libsupc++ -std=gnu++14 -DSANITIZER_LIBBACKTRACE -DSANITIZER_CP_DEMANGLE -I ../../../../src/libsanitizer/../libbacktrace -I ../libbacktrace -I ../../../../src/libsanitizer/../include -include ../../../../src/libsanitizer/libbacktrace/backtrace-rename.h -g -O2 -D_GNU_SOURCE -MT sanitizer_platform_limits_posix.lo -MD -MP -MF .deps/sanitizer_platform_limits_posix.Tpo -c ../../../../src/libsanitizer/sanitizer_common/sanitizer_platform_limits_posix.cpp -fPIC -DPIC -o .libs/sanitizer_platform_limits_posix.o
| ../../../../src/libsanitizer/sanitizer_common/sanitizer_platform_limits_posix.cpp:155:10: fatal error: crypt.h: No such file or directory
| 155 | #include <crypt.h>
| | ^~~~~~~~~
| compilation terminated.
| make[6]: *** [Makefile:616: sanitizer_platform_limits_posix.lo] Error 1
| make[6]: Leaving directory '/tmp/buildd/gcc3/gcc-12-12.1.0/build/powerpc64-linux-gnu/libsanitizer/sanitizer_common'
| make[5]: *** [Makefile:533: all-recursive] Error 1
| make[5]: Leaving directory '/tmp/buildd/gcc3/gcc-12-12.1.0/build/powerpc64-linux-gnu/libsanitizer'
| make[4]: *** [Makefile:420: all] Error 2
| make[4]: Leaving directory '/tmp/buildd/gcc3/gcc-12-12.1.0/build/powerpc64-linux-gnu/libsanitizer'
| make[3]: *** [Makefile:12936: all-target-libsanitizer] Error 2
| make[3]: Leaving directory '/tmp/buildd/gcc3/gcc-12-12.1.0/build'
| make[2]: *** [Makefile:1049: all] Error 2
| make[2]: Leaving directory '/tmp/buildd/gcc3/gcc-12-12.1.0/build'
| === TIME stamps/05-build-stamp ===
Given that crypt.h is architecture dependent, we'll have to build
crypt.h with a stage1 compiler or no C compiler at all. This is going to
be painful.
I've checked that this is the only place in gcc-12 that uses crypt.h and
that the only thing it uses is sizeof(struct crypt_data), which
theoretically is architecture-dependent, but practically is not. This is
why it has worked in the past. Note that this assumption was broken when
we used musl's crypt.h instead of libxcrypt's.
So in principle, we can stick to the workaround (multiarch only for
libcrypt2), but it comes at a cost:
* It encodes the assumption that sizeof(struct crypt_data) is
architecture-independent into libcrypt-dev.
* We will be unable to use any architecture that uses libcrypt2 as
build architecture for cross bootstraps. In other words, we cannot
bootstrap from musl.
* We can only have architectures that get crypt.h from libxcrypt.
In essence this is a cycle. gcc-12's libsanitizer requires crypt.h,
which is built from libxcrypt, which requires a C compiler, which is
built from gcc-12. It needs to be broken somehow - not least for native
bootstraps. It's been like that all the time and we used to break is
using the assumption that sizeof(struct crypt_data) is
architecture-independent.
The commit adding the crypt.h dependency is fairly recent:
| commit 3ca75cd55030a53a858bdbe9aba6d87f6b1e90b8
| Author: Martin Liska <[email protected]>
| Date: Tue Nov 5 14:54:57 2019 +0100
|
| Libsanitizer: merge from trunk with merge.sh.
|
| 2019-11-05 Martin Liska <[email protected]>
|
| * all source files: Merge from upstream r375507.
|
| From-SVN: r277834
I failed to figure out the commit messages of those subversion
revisions.
Any idea how to fix this? The TL;DR seems to be that either gcc-12 needs
to stop using crypt.h or we need a libxcrypt stage1 for building
crypt.h or crypt.h needs to become architecture-independent.
Short term, my workaround papers over the practical issues, but that's a
no-brainer long-term.
Helmut
Acknowledgement sent
to Helmut Grohne <[email protected]>:
Extra info received and forwarded to list. Copy sent to Marco d'Itri <[email protected]>.
(Tue, 05 Jul 2022 07:21:05 GMT) (full text, mbox, link).
Changed Bug title to 'bootstrap dependency cycle between gcc and libxcrypt: libsanitizer requires struct crypt_data from crypt.h' from 'installation of crypt.h in the multiarch ___location breaks the GCC and LLVM multilib builds'.
Request was from Helmut Grohne <[email protected]>
to [email protected].
(Tue, 05 Jul 2022 07:21:05 GMT) (full text, mbox, link).
Debbugs is free software and licensed under the terms of the GNU General
Public License version 2. The current version can be obtained
from https://bugs.debian.org/debbugs-source/.