Debian Bug report logs - #1094828
g++ internal compiler error on arm64 when compiling pytorch

Full log

Message #24 received at [email protected] (full text, mbox, reply):

Received: (at 1094828) by bugs.debian.org; 13 Feb 2025 20:41:51 +0000
From [email protected] Thu Feb 13 20:41:51 2025
X-Spam-Checker-Version: SpamAssassin 3.4.6-bugs.debian.org_2005_01_02
	(2021-04-09) on buxtehude.debian.org
X-Spam-Level: 
X-Spam-Status: No, score=-23.7 required=4.0 tests=BAYES_00,
	BODY_INCLUDES_CONTROL,BODY_INCLUDES_CONTROL_TAGS,FOURLA,FROMDEVELOPER,
	FVGT_m_MULTI_ODD,HAS_BUG_NUMBER,RCVD_IN_DNSWL_LOW,RCVD_IN_MSPIKE_H3,
	RCVD_IN_MSPIKE_WL,SPF_HELO_PASS,SPF_NONE autolearn=ham
	autolearn_force=no version=3.4.6-bugs.debian.org_2005_01_02
X-Spam-Bayes: score:0.0000 Tokens: new, 27; hammy, 150; neutral, 493; spammy,
	0. spammytokens: hammytokens:0.000-+--sk:buildd.,
	0.000-+--UD:buildd.debian.org, 0.000-+--buildddebianorg,
	0.000-+--buildd.debian.org, 0.000-+--cmake
Return-path: <[email protected]>
Received: from mx1.riseup.net ([198.252.153.129]:44584)
	by buxtehude.debian.org with esmtps (TLS1.3:ECDHE_X25519__RSA_PSS_RSAE_SHA256__AES_256_GCM:256)
	(Exim 4.94.2)
	(envelope-from <[email protected]>)
	id 1tig1v-00DtIn-3G
	for [email protected]; Thu, 13 Feb 2025 20:41:51 +0000
Received: from fews01-sea.riseup.net (fews01-sea-pn.riseup.net [10.0.1.109])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
	 key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256)
	(No client certificate requested)
	by mx1.riseup.net (Postfix) with ESMTPS id 4Yv6Y92By8zDq8y;
	Thu, 13 Feb 2025 20:41:49 +0000 (UTC)
X-Riseup-User-ID: 55C02C4681E49EFC8129779313C9673B00F1F5F0092CD9A91067BB537D507BAA
Received: from [127.0.0.1] (localhost [127.0.0.1])
	 by fews01-sea.riseup.net (Postfix) with ESMTPSA id 4Yv6Y62MpbzJsJT;
	Thu, 13 Feb 2025 20:41:46 +0000 (UTC)
Content-Type: multipart/mixed; boundary="------------GhpKOtGb0zUDkSbNswcc1Uzf"
Message-ID: <[email protected]>
Date: Thu, 13 Feb 2025 15:41:45 -0500
MIME-Version: 1.0
Subject: Re: Bug#1094828: g++ internal compiler error on arm64 when compiling
 pytorch
To: Matthias Klose <[email protected]>, [email protected]
References: <[email protected]>
 <[email protected]>
Content-Language: en-US
From: Mo Zhou <[email protected]>
In-Reply-To: <[email protected]>

[Message part 1 (text/plain, inline)]

Control: reopen -1

The problem persists, and the buildd is still failing on it several 
hours ago:
https://buildd.debian.org/status/fetch.php?pkg=pytorch&arch=arm64&ver=2.6.0%2Bdfsg-1%7Eexp1&stamp=1739418834&raw=0

This is a regression. Setting CC=gcc-13 and CXX=g++-13 in d/rules can avoid
this issue. The problem persists with GCC-15 in experimental.

    * the exact version of GCC;

g++ (Debian 14.2.0-17) 14.2.0

    * the system type;

arm64, on native arm64 machine.

    * the options given when GCC was configured/built;

Using built-in specs.
COLLECT_GCC=g++
COLLECT_LTO_WRAPPER=/usr/libexec/gcc/aarch64-linux-gnu/14/lto-wrapper
OFFLOAD_TARGET_NAMES=nvptx-none
OFFLOAD_TARGET_DEFAULT=1
Target: aarch64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Debian 
14.2.0-17' --with-bugurl=file:///usr/share/doc/gcc-14/README.Bugs 
--enable-languages=c,ada,c++,go,d,fortran,objc,obj-c++,m2,rust 
--prefix=/usr --with-gcc-major-version-only --program-suffix=-14 
--program-prefix=aarch64-linux-gnu- --enable-shared 
--enable-linker-build-id --libexecdir=/usr/libexec 
--without-included-gettext --enable-threads=posix --libdir=/usr/lib 
--enable-nls --enable-bootstrap --enable-clocale=gnu 
--enable-libstdcxx-debug --enable-libstdcxx-time=yes 
--with-default-libstdcxx-abi=new --enable-libstdcxx-backtrace 
--enable-gnu-unique-object --disable-libquadmath 
--disable-libquadmath-support --enable-plugin --enable-default-pie 
--with-system-zlib --enable-libphobos-checking=release 
--with-target-system-zlib=auto --enable-objc-gc=auto --enable-multiarch 
--enable-fix-cortex-a53-843419 --disable-werror 
--enable-offload-targets=nvptx-none=/build/reproducible-path/gcc-14-14.2.0/debian/tmp-nvptx/usr 
--enable-offload-defaulted --without-cuda-driver 
--enable-checking=release --build=aarch64-linux-gnu 
--host=aarch64-linux-gnu --target=aarch64-linux-gnu 
--with-build-config=bootstrap-lto-lean --enable-link-serialization=4
Thread model: posix
Supported LTO compression algorithms: zlib zstd
gcc version 14.2.0 (Debian 14.2.0-17)

    * the complete command line that triggers the bug;

/usr/bin/c++ -DAT_BUILD_ARM_VEC256_WITH_SLEEF -DAT_PER_OPERATOR_HEADERS 
-DBUILD_ONEDNN_GRAPH -DCAFFE2_BUILD_MAIN_LIB -DCAFFE2_PERF_WITH_SVE=1 
-DDNNL_EXPERIMENTAL_UKERNEL -DFLASHATTENTION_DISABLE_ALIBI 
-DFMT_HEADER_ONLY=1 -DHAVE_MALLOC_USABLE_SIZE=1 -DHAVE_MMAP=1 
-DHAVE_SHM_OPEN=1 -DHAVE_SHM_UNLINK=1 
-DMINIZ_DISABLE_ZIP_READER_CRC32_CHECKS -DONNXIFI_ENABLE_EXT=1 
-DONNX_ML=1 -DONNX_NAMESPACE=onnx -DTORCH_ENABLE_LLVM -DUSE_C10D_GLOO 
-DUSE_DISTRIBUTED -DUSE_EXTERNAL_MZCRC -DUSE_RPC -DUSE_TENSORPIPE 
-D_FILE_OFFSET_BITS=64 -Dtorch_cpu_EXPORTS 
-I/mnt/Debian/pytorch/build/aten/src -I/mnt/Debian/pytorch/aten/src 
-I/mnt/Debian/pytorch/build -I/mnt/Debian/pytorch 
-I/mnt/Debian/pytorch/cmake/../third_party/benchmark/include 
-I/usr/lib/llvm-19/include -I/mnt/Debian/pytorch/nlohmann 
-I/mnt/Debian/pytorch/torch/csrc/api 
-I/mnt/Debian/pytorch/torch/csrc/api/include 
-I/mnt/Debian/pytorch/caffe2/aten/src/TH 
-I/mnt/Debian/pytorch/build/caffe2/aten/src/TH 
-I/mnt/Debian/pytorch/build/caffe2/aten/src 
-I/mnt/Debian/pytorch/build/caffe2/../aten/src 
-I/mnt/Debian/pytorch/torch/csrc 
-I/mnt/Debian/pytorch/third_party/miniz-3.0.2 
-I/mnt/Debian/pytorch/debian/kineto/libkineto/include 
-I/mnt/Debian/pytorch/debian/kineto/libkineto/src 
-I/mnt/Debian/pytorch/third_party/cpp-httplib 
-I/mnt/Debian/pytorch/aten/src/ATen/.. -I/mnt/Debian/pytorch/c10/.. 
-I/mnt/Debian/pytorch/third_party/flatbuffers/include -isystem 
/mnt/Debian/pytorch/cmake/../third_party/googletest/googlemock/include 
-isystem 
/mnt/Debian/pytorch/cmake/../third_party/googletest/googletest/include 
-isystem /usr/include/eigen3 -isystem /usr/include/oneapi/dnnl -isystem 
/mnt/Debian/pytorch/INTERFACE -isystem 
/mnt/Debian/pytorch/third_party/nlohmann/include -isystem 
/mnt/Debian/pytorch/caffe2 -Wdate-time -D_FORTIFY_SOURCE=2 -g -O2 
-ffile-prefix-map=/mnt/Debian/pytorch=. -fstack-protector-strong 
-fstack-clash-protection -Wformat -Werror=format-security 
-mbranch-protection=standard -gsplit-dwarf -Wno-dangling-reference  
-I/usr -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden 
-DUSE_PTHREADPOOL -DUSE_KINETO -DLIBKINETO_NOCUPTI 
-DLIBKINETO_NOROCTRACER -DLIBKINETO_NOXPUPTI=ON 
-DAT_BUILD_ARM_VEC256_WITH_SLEEF -DUSE_XNNPACK 
-DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra 
-Werror=return-type -Werror=non-virtual-dtor 
-Werror=range-loop-construct -Werror=bool-operation -Wnarrowing 
-Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds 
-Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow 
-Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override 
-Wno-psabi -Wno-error=old-style-cast -Wno-missing-braces 
-fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable 
-Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math 
-Werror=format -Wno-error=dangling-reference -Wno-error=redundant-move 
-Wno-stringop-overflow -DHAVE_SVE_CPU_DEFINITION 
-DHAVE_SVE256_CPU_DEFINITION -O2 -g -DNDEBUG -std=gnu++17 -fPIC 
-DCAFFE2_USE_GLOO -D__NEON__ -Wall -Wextra -Wdeprecated 
-Wno-unused-parameter -Wno-missing-field-initializers -Wno-array-bounds 
-Wno-unknown-pragmas -Wno-strict-overflow -Wno-strict-aliasing 
-Wunused-function -Wunused-variable -Wunused-but-set-variable 
-Wno-maybe-uninitialized -fvisibility=hidden -O2 -fopenmp -O3  
-DCPU_CAPABILITY=DEFAULT -DCPU_CAPABILITY_DEFAULT -MD -MT 
caffe2/CMakeFiles/torch_cpu.dir/__/aten/src/ATen/native/cpu/Activation.cpp.DEFAULT.cpp.o 
-MF 
caffe2/CMakeFiles/torch_cpu.dir/__/aten/src/ATen/native/cpu/Activation.cpp.DEFAULT.cpp.o.d 
-o 
caffe2/CMakeFiles/torch_cpu.dir/__/aten/src/ATen/native/cpu/Activation.cpp.DEFAULT.cpp.o 
-c 
/mnt/Debian/pytorch/build/aten/src/ATen/native/cpu/Activation.cpp.DEFAULT.cpp 
-freport-bug -save-temps

    * the compiler output (error messages, warnings, etc.); and

root@01e269f40847 /m/D/p/build (master)# bash x.sh
during RTL pass: expand
In file included from 
/mnt/Debian/pytorch/aten/src/ATen/native/cpu/Activation.cpp:12,
                 from 
/mnt/Debian/pytorch/build/aten/src/ATen/native/cpu/Activation.cpp.DEFAULT.cpp:1:
/mnt/Debian/pytorch/aten/src/ATen/native/cpu/Activation.cpp: In lambda 
function:
/mnt/Debian/pytorch/aten/src/ATen/native/cpu/Activation.cpp:89:7: 
internal compiler error: Segmentation fault
   89 |       });
      |       ^
/mnt/Debian/pytorch/aten/src/ATen/Dispatch.h:202:7: note: in definition 
of macro ‘AT_DISPATCH_SWITCH’
  202 | __VA_ARGS__ \
      |       ^~~~~~~~~~~
/mnt/Debian/pytorch/aten/src/ATen/Dispatch.h:73:3: note: in expansion of 
macro ‘AT_PRIVATE_CASE_TYPE_USING_HINT’
   73 |   AT_PRIVATE_CASE_TYPE_USING_HINT(enum_type, scalar_t, __VA_ARGS__)
      |   ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/mnt/Debian/pytorch/aten/src/ATen/Dispatch.h:215:3: note: in expansion 
of macro ‘AT_DISPATCH_CASE’
  215 |   AT_DISPATCH_CASE(at::ScalarType::Double, __VA_ARGS__) \
      |   ^~~~~~~~~~~~~~~~
/mnt/Debian/pytorch/aten/src/ATen/Dispatch.h:219:34: note: in expansion 
of macro ‘AT_DISPATCH_CASE_FLOATING_TYPES’
  219 |   AT_DISPATCH_SWITCH(TYPE, NAME, 
AT_DISPATCH_CASE_FLOATING_TYPES(__VA_ARGS__))
      | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/mnt/Debian/pytorch/aten/src/ATen/native/cpu/Activation.cpp:65:5: note: 
in expansion of macro ‘AT_DISPATCH_FLOATING_TYPES’
   65 |     AT_DISPATCH_FLOATING_TYPES(input.scalar_type(), 
"log_sigmoid_cpu", [&] {
      |     ^~~~~~~~~~~~~~~~~~~~~~~~~~
0xd4f513 internal_error(char const*, ...)
        ???:0
0x135298f rebuild_jump_labels(rtx_insn*)
        ???:0
Please submit a full bug report, with preprocessed source (by using 
-freport-bug).
Please include the complete backtrace with any bug report.
See <file:///usr/share/doc/gcc-14/README.Bugs> for instructions.

    * the preprocessed file (*.i*) that triggers the bug, generated by
      adding -save-temps to the complete compilation command, or, in
      the case of a bug report for the GNAT front end, a complete set
      of source files (see below).

See attached.


On 1/31/25 10:19, Matthias Klose wrote:
> Control: tags -1 + moreinfo
>
> On 31.01.25 15:46, M. Zhou wrote:
>> Package: g++-14
>> Version: 14.2.0-14
>> Severity: important
>>
>> Dear maintainer,
>>
>> We noted that pytorch 2.6.0 FTBFS on arm64 due to an internal compiler
>> error from g++. The relevant part of error log reads:
>>
>> ```
>> Please submit a full bug report, with preprocessed source (by using 
>> -freport-bug).
>> Please include the complete backtrace with any bug report.
>> See <file:///usr/share/doc/gcc-14/README.Bugs> for instructions.
>> Preprocessed source stored into /tmp/ccE55iyB.out file, please attach 
>> this to your bugreport.
>
> please do so, follow the instruction in your included log
> please also check with gcc-15 from experimental.
>

[Activation.cpp.DEFAULT.cpp.ii.zst (application/octet-stream, attachment)]

Send a report that this bug log contains spam.