Acknowledgement sent
to Fab Stz <[email protected]>:
New Bug report received and forwarded. Copy sent to Devscripts Maintainers <[email protected]>.
(Tue, 05 Oct 2021 12:06:03 GMT) (full text, mbox, link).
Subject: mk-origtargz: repacking is slow when there are big folders Files-Excluded
Date: Tue, 05 Oct 2021 14:02:08 +0200
Package: devscripts
Version: 2.21.3
Severity: normal
Dear Maintainer,
mk-origtargz uses tar --delete to remove files listed in Files-Excluded.
However, when there are huge folders with many files in it, then it calls tar
--delete a huge amount of times.
This is because the size of the tar --delete command cannot contain all files
to exclude and then calls it as many times as needed.
When working on big archives this can take a very long time. For example I was
working on a 600MB archive (compressed) from which I wanted to removed ~2GB of
uncompressed data. This should lead to ~ 300MB archive in the end.
Since it takes forever with mk-origtargz I finally repacked it with a script
that uncompresses the tar, removes the files/folder listed in Files-Excluded,
and then creates a new archive.
In the documentation of --delete [1] it is already mentioned that "The
‘--delete’ operation can run very slowly"
Would it be possible to improve mk-origtargz so that it doesn't use tar
--delete but a faster alternative?
BTW, this also affects uscan which calls mk-origtargz.
[1] https://www.gnu.org/software/tar/manual/html_node/delete.html
-- Package-specific info:
--- /etc/devscripts.conf ---
Empty.
--- ~/.devscripts ---
Not present
-- System Information:
Debian Release: 11.0
APT prefers stable-updates
APT policy: (991, 'stable-updates'), (991, 'stable'), (95, 'testing'), (90,
'unstable')
Architecture: amd64 (x86_64)
Foreign Architectures: i386
Kernel: Linux 5.10.0-8-amd64 (SMP w/4 CPU threads)
Kernel taint flags: TAINT_PROPRIETARY_MODULE, TAINT_OOT_MODULE,
TAINT_UNSIGNED_MODULE
Locale: LANG=fr_FR.UTF-8, LC_CTYPE=fr_FR.UTF-8 (charmap=UTF-8),
LANGUAGE=fr:en_US
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled
Versions of packages devscripts depends on:
ii dpkg-dev 1.20.9
ii fakeroot 1.25.3-1.1
ii file 1:5.39-3
ii gnupg 2.2.27-2
ii gpgv 2.2.27-2
ii libc6 2.31-13
ii libfile-dirlist-perl 0.05-2
ii libfile-homedir-perl 1.006-1
ii libfile-touch-perl 0.11-1
ii libfile-which-perl 1.23-1
ii libipc-run-perl 20200505.0-1
ii libmoo-perl 2.004004-1
ii libwww-perl 6.52-1
ii patchutils 0.4.2-1
ii perl 5.32.1-4+deb11u1
ii python3 3.9.2-3
ii sensible-utils 0.0.14
ii wdiff 1.2.2-2+b1
Versions of packages devscripts recommends:
ii apt 2.2.4
ii curl 7.74.0-1.3+b1
ii dctrl-tools 2.24-3+b1
ii debian-keyring 2021.07.26
ii dput 1.1.0
ii equivs 2.3.1
ii libdistro-info-perl 1.0
ii libdpkg-perl 1.20.9
ii libencode-locale-perl 1.05-1.1
ii libgit-wrapper-perl 0.048-1
ii libgitlab-api-v4-perl 0.26-1
ii liblist-compare-perl 0.55-1
ii liblwp-protocol-https-perl 6.10-1
ii libsoap-lite-perl 1.27-1
ii libstring-shellquote-perl 1.04-1
ii libtry-tiny-perl 0.30-1
ii liburi-perl 5.08-1
ii licensecheck 3.1.1-2
ii lintian 2.104.0
ii man-db 2.9.4-2
ii patch 2.7.6-7
ii pristine-tar 1.49
ii python3-apt 2.2.1
ii python3-debian 0.1.39
ii python3-magic 2:0.4.20-3
ii python3-requests 2.25.1+dfsg-2
ii python3-unidiff 0.5.5-2
ii python3-xdg 0.27-2
ii strace 5.10-1
ii unzip 6.0-26
ii wget 1.21-1+b1
ii xz-utils 5.2.5-2
Versions of packages devscripts suggests:
pn adequate <none>
pn at <none>
ii autopkgtest 5.16
pn bls-standalone <none>
ii build-essential 12.9
pn check-all-the-things <none>
pn cvs-buildpackage <none>
ii debhelper 13.3.4
pn devscripts-el <none>
pn diffoscope <none>
pn disorderfs <none>
pn dose-extra <none>
pn duck <none>
pn faketime <none>
pn gnuplot <none>
pn how-can-i-help <none>
ii libauthen-sasl-perl 2.1600-1.1
pn libdbd-pg-perl <none>
ii libfile-desktopentry-perl 0.22-2
pn libnet-smtps-perl <none>
pn libterm-size-perl <none>
ii libtimedate-perl 2.3300-2
pn libyaml-syck-perl <none>
ii mailutils [mailx] 1:3.10-3+b1
pn mmdebstrap <none>
pn mozilla-devscripts <none>
pn mutt <none>
ii openssh-client [ssh-client] 1:8.4p1-5
pn piuparts <none>
ii postgresql-client-13 [postgresql-client] 13.3-1
pn pristine-lfs <none>
ii quilt 0.66-2.1
pn ratt <none>
pn reprotest <none>
pn svn-buildpackage <none>
pn w3m <none>
-- no debconf information
Acknowledgement sent
to Blair Noctis <[email protected]>:
Extra info received and forwarded to list. Copy sent to Devscripts Maintainers <[email protected]>.
(Thu, 30 Mar 2023 15:24:02 GMT) (full text, mbox, link).
On Tue, 05 Oct 2021 18:26:04 +0200 Yadd <[email protected]> wrote:
> tags 995770 + wontfix
> thanks
Hi Yadd,
Would you mind giving a little explanation on why? I'm considerably slowed down
by this when packaging rustc, a fairly large package, and frustrated by the fact
that it's single-threadly doing: read archive, update archive, reconstructing
the entirety of the file on disk due to the update, repeat.
--
Sdrager,
Blair Noctis
Acknowledgement sent
to Blair Noctis <[email protected]>:
Extra info received and forwarded to list. Copy sent to Devscripts Maintainers <[email protected]>.
(Sat, 11 Jan 2025 16:18:02 GMT) (full text, mbox, link).
Hi Yadd,
On Thu, 30 Mar 2023 23:05:51 +0800 Blair Noctis <[email protected]> wrote:
> On Tue, 05 Oct 2021 18:26:04 +0200 Yadd <[email protected]> wrote:
> > tags 995770 + wontfix
> > thanks
>
> Hi Yadd,
>
> Would you mind giving a little explanation on why? I'm considerably slowed down
> by this when packaging rustc, a fairly large package, and frustrated by the fact
> that it's single-threadly doing: read archive, update archive, reconstructing
> the entirety of the file on disk due to the update, repeat.
Would you kindly reconsider this? Our fellow rustc maintainer Fabian Grünbichler still suffers from this for hours on each rustc release, which needs many files excluded. It would immensely ease their work. Thank you!
--
Sdrager,
Blair Noctis
Debbugs is free software and licensed under the terms of the GNU General
Public License version 2. The current version can be obtained
from https://bugs.debian.org/debbugs-source/.