Debian Bug report logs - #831521
[uscan] add support to run custom mk-origtargz

Reply or subscribe to this bug.

Display info messages

Message #5 received at [email protected] (full text, mbox, reply):

From: Sergio Durigan Junior <[email protected]>

To: [email protected]

Subject: uscan: Implement new 'nomktar' option on watch file

Date: Sat, 16 Jul 2016 17:51:21 -0400

[Message part 1 (text/plain, inline)]

Package: devscripts
Tags: patch

Hi,

Rationale: <https://lists.debian.org/debian-mentors/2016/07/msg00389.html>

Some Debian maintainers have to deal with upstream projects that do not
offer tarballs for releases.  The modus operandi for working around this
"issue" is to provide your own replacement for the 'uupdate' script,
which should be responsible for manipulating the file(s) downloaded from
upstream and prepare a "fake" tarball to be provided to 'uupdate'.

This works, but has one drawback: there is no "easy" way to suppress the
'mk-origtargz' program to run.  And the problem with this is that
'mk-origtargz' expects a compressed tarball as its input, always.
Calling it in the situation described above leads to an error.  So, the
user is left with some non-ideal options:

1) Invoke 'uscan' passing '--no-symlink' as one of the parameters.  This
has the side-effect of skipping the call to 'mk-origtargz'.  The obvious
disadvantages are: (a) it is not obvious to the user that this option
solves the problem described above, and (b) the user will have to
remember to pass this option every time.

2) Hacking the 'uupdate'-replacement script in order to invoke it by
hand.  The obvious disadvantage is that the user won't be able to use
'uscan', and also will have to remember this every time.

Therefore, to fix this issue, I am proposing a new option, to be
provided via 'opts=' in the watch file, whose purpose is to suppress the
execution of 'mk-origtargz'.  Its effect will be similar to passing
'--no-symlink', although the user will still be able to provide
'--rename', for example.

Opinions?

Cheers,

-- 
Sergio
GPG key ID: 237A 54B1 0287 28BF 00EF  31F4 D0EB 7628 65FC 5E36
Please send encrypted e-mail if possible
http://sergiodj.net/

[0001-uscan-Implement-nomktar-option-to-allow-user-to-spec.patch (text/x-diff, inline)]

From b40dbd4f6b052f0753aede3822a372149d8d0c7f Mon Sep 17 00:00:00 2001
From: Sergio Durigan Junior <[email protected]>
Date: Sat, 16 Jul 2016 17:36:29 -0400
Subject: [PATCH] uscan: Implement 'nomktar' option to allow user to specify
 when she does not want mk-origtargz to be invoked.

---
 debian/changelog |  6 ++++++
 scripts/uscan.pl | 11 ++++++++++-
 2 files changed, 16 insertions(+), 1 deletion(-)

diff --git a/debian/changelog b/debian/changelog
index 3b9e5a0..3207485 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,8 +1,14 @@
 devscripts (2.16.7) UNRELEASED; urgency=medium
 
+  [ Paul Wise ]
   * grep-excuses:
     + Fix the script for the removal of testing.pl from release.debian.org
 
+  [ Sergio Durigan Junior ]
+  * uscan:
+    + Implement 'nomktar' option to allow user to specify when she does
+      not want mk-origtargz to be invoked.
+
  -- Paul Wise <[email protected]>  Fri, 15 Jul 2016 22:36:19 +0800
 
 devscripts (2.16.6) unstable; urgency=medium
diff --git a/scripts/uscan.pl b/scripts/uscan.pl
index 9403ba6..818292e 100755
--- a/scripts/uscan.pl
+++ b/scripts/uscan.pl
@@ -408,6 +408,13 @@ No signature available. (No warning.)
 
 Decompress compressed archive before the pgp/gpg signature verification.
 
+=item B<nomktar>
+
+Do not invoke B<mk-origtargz> after saving the downloaded file.  This
+is useful if you are dealing with files that are not tarballs, for
+example.  The side effect of this option is that no symlink will be
+created.
+
 =item B<bare>
 
 Disable all site specific special case code such as URL redirector uses and
@@ -2551,6 +2558,8 @@ sub process_watchline ($$$$$$)
 		    $options{'pgpmode'} = 'mangle';
 		} elsif ($opt =~ /^\s*oversionmangle\s*=\s*(.+?)\s*$/) {
 		    @{$options{'oversionmangle'}} = split /;/, $1;
+		} elsif ($opt =~ /^\s*nomktar\s*$/) {
+		    $options{'nomktar'} = 1;
 		} else {
 		    uscan_warn "unrecognised option $opt\n";
 		}
@@ -3700,7 +3709,7 @@ EOF
     my $mk_origtargz_out;
     my $path = "$destdir/$newfile_base";
     my $target = $newfile_base;
-    unless ($symlink eq "no") {
+    unless ($symlink eq "no" || $options{'nomktar'} ) {
 	my @cmd = ("mk-origtargz");
 	push @cmd, "--package", $pkg;
 	push @cmd, "--version", $common_mangled_newversion;
-- 
2.8.1

[signature.asc (application/pgp-signature, inline)]

Message #10 received at [email protected] (full text, mbox, reply):

From: Osamu Aoki <[email protected]>

To: Sergio Durigan Junior <[email protected]>, [email protected]

Cc: [email protected]

Subject: Re: Bug#831521: uscan: Implement new 'nomktar' option on watch file

Date: Mon, 18 Jul 2016 00:14:03 +0900

Hi,

On Sat, Jul 16, 2016 at 05:51:21PM -0400, Sergio Durigan Junior wrote:
> Package: devscripts
> Tags: patch
> 
> Hi,
> 
> Rationale: <https://lists.debian.org/debian-mentors/2016/07/msg00389.html>
> 
> Some Debian maintainers have to deal with upstream projects that do not
> offer tarballs for releases.  The modus operandi for working around this
> "issue" is to provide your own replacement for the 'uupdate' script,
> which should be responsible for manipulating the file(s) downloaded from
> upstream and prepare a "fake" tarball to be provided to 'uupdate'.
> 
> This works, but has one drawback: there is no "easy" way to suppress the
> 'mk-origtargz' program to run.  And the problem with this is that
> 'mk-origtargz' expects a compressed tarball as its input, always.
> Calling it in the situation described above leads to an error.  So, the
> user is left with some non-ideal options:
> 
> 1) Invoke 'uscan' passing '--no-symlink' as one of the parameters.  This
> has the side-effect of skipping the call to 'mk-origtargz'.  The obvious
> disadvantages are: (a) it is not obvious to the user that this option
> solves the problem described above, and (b) the user will have to
> remember to pass this option every time.
> 
> 2) Hacking the 'uupdate'-replacement script in order to invoke it by
> hand.  The obvious disadvantage is that the user won't be able to use
> 'uscan', and also will have to remember this every time.
> 
> Therefore, to fix this issue, I am proposing a new option, to be
> provided via 'opts=' in the watch file, whose purpose is to suppress the
> execution of 'mk-origtargz'.  Its effect will be similar to passing
> '--no-symlink', although the user will still be able to provide
> '--rename', for example.
> 
> Opinions?

This is certainly one way to solve your problem .... and I see some
people abused uupdate slot for this purpose.

But replacing uupdate (updating source tree from repackaged tarball and
old source tree) with a command for another purpose seems to complicate
situation.  We want to run your alternative command in place of
mk-origtargz.  How about ...  if debian/mk-mk-origtargz exists, use it
instead of mk-origtargz.

I know there are some resentments to run a script in debian/* but this
part is not executed when just scanning archive.  So situation is no
worse than uupdate situation.

Currently, mk-origtargz output is scanned to set some parameters so your
mk-origtargz may need to be written carefully.  That's the negative of
this approach but certainly this is another way to solve your problem.

By the way, can you tall me exact case you are thinking so we know we
are attacking the real problem (not just theoretical cases).

Note:

There is a question on how you call your alternative uupdate command
$cmd.  Currently, uscan invokes non-uupdate $cmd as:
 Version 4  : "$cmd --upstream-version <version>"
 Version 2,3: "$cmd --upstream-version <version> <path>"
 Version 1  : "$cmd <path> <version>"
This <path> is the full path filename of the file downloaded.

Maybe, I should also make version 4 to do the same as version 3 if $cmd
is not uupdate.  If $cmd is uupdate for version 4, it calls
 Version 4  : "uupdate --find --upstream-version <version>"
(This is because the <path> should be pointing the file in ../)

Anyway, these needs to be updated with documentation if this is
something needs to be done...

Osamu

Message #15 received at [email protected] (full text, mbox, reply):

From: Gianfranco Costamagna <[email protected]>

To: Osamu Aoki <[email protected]>, Sergio Durigan Junior <[email protected]>, "[email protected]" <[email protected]>

Subject: Re: Bug#831521: uscan: Implement new 'nomktar' option on watch file

Date: Mon, 18 Jul 2016 07:51:36 +0000 (UTC)

Hi Osamu,


>By the way, can you tall me exact case you are thinking so we know we
>are attacking the real problem (not just theoretical cases).


You can see an use case here [1] with another use case in one of my replies to
the thread, because I can't use uscan to download virtualbox-guest-additions
iso file.

[1] https://lists.debian.org/debian-mentors/2016/07/msg00389.html

hope this helps,

Gianfranco

Message #20 received at [email protected] (full text, mbox, reply):

From: Osamu Aoki <[email protected]>

To: [email protected], Sergio Durigan Junior <[email protected]>

Cc: Christopher Hoskin <[email protected]>

Subject: Custom repacking

Date: Mon, 28 Aug 2017 01:04:54 +0900

control: tags -1 - patch
control: severity -1 wishlist
control: retitle -1 implement capability to run custom mk-origtargz
control: severity 858319 wishlist
control: retitle 858319 implement capability to run custom mk-origtargz
control: merge 858319 831521

Hi,

Just because uupdate entry in watch file enables to run custom script,
abusing it is not good idea.

uscan deligates its key functions to:
 * repacking of upstream tarball/signature -- mk-origtargz
 * updating new sorce tree                 -- uupdate

So adding uscan a capability to run custom script in place of calling
mk-origtargz is the right approach.

That's exactly like repack.stub discussed in # 858319

I need to address these bugs without making overcomplicated uscan to do
more work.  Maybe if debian/mk-origtargz exists, it is used instead of
the default one.   That's simple.  This also enables if someone have
good script, I can merge into mk-origtargz.

Regards,

Osamu

Message #49 received at [email protected] (full text, mbox, reply):

From: Sergio Durigan Junior <[email protected]>

To: Osamu Aoki <[email protected]>

Cc: [email protected], Christopher Hoskin <[email protected]>

Subject: Re: Custom repacking

Date: Sun, 27 Aug 2017 17:24:43 -0400

[Message part 1 (text/plain, inline)]

On Sunday, August 27 2017, Osamu Aoki wrote:

> Hi,
>
> Just because uupdate entry in watch file enables to run custom script,
> abusing it is not good idea.
>
> uscan deligates its key functions to:
>  * repacking of upstream tarball/signature -- mk-origtargz
>  * updating new sorce tree                 -- uupdate
>
> So adding uscan a capability to run custom script in place of calling
> mk-origtargz is the right approach.
>
> That's exactly like repack.stub discussed in # 858319
>
> I need to address these bugs without making overcomplicated uscan to do
> more work.  Maybe if debian/mk-origtargz exists, it is used instead of
> the default one.   That's simple.  This also enables if someone have
> good script, I can merge into mk-origtargz.

Hi Osamu,

Thanks for following up this bug.  I confess I had forgotten about it!

Anyway, I took the liberty and implemented the following mostly-untested
patch to uscan which uses your idea and executes debian/mk-origtargz (if
it exists and is executable) instead of the system's mk-origtargz.

What do you thing?

-- 
Sergio
GPG key ID: 237A 54B1 0287 28BF 00EF  31F4 D0EB 7628 65FC 5E36
Please send encrypted e-mail if possible
http://sergiodj.net/

diff --git a/scripts/uscan.pl b/scripts/uscan.pl
index f871daf2..49a7acb2 100755
--- a/scripts/uscan.pl
+++ b/scripts/uscan.pl
@@ -60,8 +60,9 @@ than the last upstream version.
 =item * B<uscan> saves the downloaded tarball to the parent B<../> directory:
 I<< ../<upkg>-<uversion>.tar.gz >>
 
-=item * B<uscan> invokes B<mk-origtargz> to create the source tarball: I<<
-../<spkg>_<oversion>.orig.tar.gz >>
+=item * B<uscan> invokes B<mk-origtargz> to create the source tarball:
+I<< ../<spkg>_<oversion>.orig.tar.gz >> (if F<debian/mk-origtargz>
+exists and is executable, then execute it instead).
 
 =over
 
@@ -706,7 +707,8 @@ doesn't work.
 
 B<uscan> invokes B<mk-origtargz> to create the source tarball properly named
 for the source package with B<.orig.> (or B<< .orig-<component>. >> for the
-secondary tarballs) in its filename.
+secondary tarballs) in its filename.  If F<debian/mk-origtargz> exists and is
+executable, then B<uscan> invokes it instead.
 
 =over
 
@@ -3764,7 +3766,15 @@ EOF
     my $path = "$destdir/$newfile_base";
     my $target = $newfile_base;
     unless ($symlink eq "no") {
-	my @cmd = ("mk-origtargz");
+	my @cmd = ();
+	if (-x 'debian/mk-origtargz') {
+	    # If the user has specified a personalized 'mk-origtargz'
+	    # under debian/, we use it instead of the system script.
+	    push @cmd "debian/mk-origtargz";
+	    uscan_verbose "Using debian/mk-origtargz instead of mk-origtargz\n";
+	} else {
+	    push @cmd "mk-origtargz";
+	}
 	push @cmd, "--package", $pkg;
 	push @cmd, "--version", $common_mangled_newversion;
 	push @cmd, '--repack-suffix', $options{repacksuffix} if defined $options{repacksuffix};

[signature.asc (application/pgp-signature, inline)]

Message #54 received at [email protected] (full text, mbox, reply):

From: Osamu Aoki <[email protected]>

To: Sergio Durigan Junior <[email protected]>, [email protected]

Cc: Christopher Hoskin <[email protected]>

Subject: Re: Bug#831521: Custom repacking

Date: Mon, 28 Aug 2017 23:12:56 +0900

Hi,

On Sun, Aug 27, 2017 at 05:24:43PM -0400, Sergio Durigan Junior wrote:
> Anyway, I took the liberty and implemented the following mostly-untested
> patch to uscan which uses your idea and executes debian/mk-origtargz (if
> it exists and is executable) instead of the system's mk-origtargz.

Thanks 
 
> diff --git a/scripts/uscan.pl b/scripts/uscan.pl
> index f871daf2..49a7acb2 100755
> --- a/scripts/uscan.pl
> +++ b/scripts/uscan.pl
> @@ -60,8 +60,9 @@ than the last upstream version.
>  =item * B<uscan> saves the downloaded tarball to the parent B<../> directory:
>  I<< ../<upkg>-<uversion>.tar.gz >>
>  
> -=item * B<uscan> invokes B<mk-origtargz> to create the source tarball: I<<
> -../<spkg>_<oversion>.orig.tar.gz >>
> +=item * B<uscan> invokes B<mk-origtargz> to create the source tarball:
> +I<< ../<spkg>_<oversion>.orig.tar.gz >> (if F<debian/mk-origtargz>
> +exists and is executable, then execute it instead).
>  
>  =over
>  
> @@ -706,7 +707,8 @@ doesn't work.
>  
>  B<uscan> invokes B<mk-origtargz> to create the source tarball properly named
>  for the source package with B<.orig.> (or B<< .orig-<component>. >> for the
> -secondary tarballs) in its filename.
> +secondary tarballs) in its filename.  If F<debian/mk-origtargz> exists and is
> +executable, then B<uscan> invokes it instead.
>  
>  =over
>  
> @@ -3764,7 +3766,15 @@ EOF
>      my $path = "$destdir/$newfile_base";
>      my $target = $newfile_base;
>      unless ($symlink eq "no") {
> -	my @cmd = ("mk-origtargz");
> +	my @cmd = ();
> +	if (-x 'debian/mk-origtargz') {
> +	    # If the user has specified a personalized 'mk-origtargz'
> +	    # under debian/, we use it instead of the system script.
> +	    push @cmd "debian/mk-origtargz";
> +	    uscan_verbose "Using debian/mk-origtargz instead of mk-origtargz\n";
> +	} else {
> +	    push @cmd "mk-origtargz";
> +	}
>  	push @cmd, "--package", $pkg;
>  	push @cmd, "--version", $common_mangled_newversion;
>  	push @cmd, '--repack-suffix', $options{repacksuffix} if defined $options{repacksuffix};

Yes, this is what I am talking about.  Maybe, we should add
F<debian/uupdate> in the same way. 

Adding these features are easy but let's go slow.  Once we add any
features, we can't remove them easily.  Let's wait and check if there
are no objection or second thought.

Regards,

Osamu

Message #69 received at [email protected] (full text, mbox, reply):

From: Christopher Hoskin <[email protected]>

To: Debian Bug Tracking System <[email protected]>

Subject: Re: implement capability to run custom mk-origtargz

Date: Wed, 30 Aug 2017 20:22:01 +0100

Package: devscripts
Version: 2.17.6+deb9u1
Followup-For: Bug #831521

Thanks for looking into this. My second thought is that there's a lot of useful functionality in mk-origtargz (determining the correct form of the tarball from debain/changelog, converting from zip or xpi, excluding files based on patterns...) and anyone needing some custom functionality shouldn't need to reinvent this. In my case I needed to run a command on a number of files within the upstream tarball, and also exclude some files.

Now I could create a debian/mk-origtargz script which creates a temp directory, extracts the upstream tar ball to it, runs my commands on the extracted files, re-tars the temporary folder into an intermediate tarball, calls /usr/bin/mk-origtargz with the path of intermediate tarball as an argument and then removes the temporary folder and intermediate tarball, but that seems to duplicate a lot of what mk-origtargz already does.

I was wondering if it would be possible to insert some code around line 437 of mk-origtargz to check for the existance of a debian/modify-upstream script, and if it exists, call it with the path of the extracted files as the working directory?

Another possibility might be to split the mk-origtargz into an unpack and repack scripts, and re-implement mk-origtargz as a wrapper which calls the unpack and repack script. Anyone implementing a debian/mk-origtargz could then call the unpack script, do some custome logic and then call the repack script.

In the case of there being no upstream tarball, debian/mk-origtargz could assemble the upstream source into a temporary folder an then just call the repack script.

I haven't thought too hard about this, so these ideas may be flawed.

Christopher

Message #74 received at [email protected] (full text, mbox, reply):

From: Sergio Durigan Junior <[email protected]>

To: Osamu Aoki <[email protected]>

Cc: [email protected], Christopher Hoskin <[email protected]>

Subject: Re: Bug#831521: Custom repacking

Date: Wed, 30 Aug 2017 20:10:50 -0400

[Message part 1 (text/plain, inline)]

On Monday, August 28 2017, Osamu Aoki wrote:

>> diff --git a/scripts/uscan.pl b/scripts/uscan.pl
>> index f871daf2..49a7acb2 100755
>> --- a/scripts/uscan.pl
>> +++ b/scripts/uscan.pl
>> @@ -60,8 +60,9 @@ than the last upstream version.
>>  =item * B<uscan> saves the downloaded tarball to the parent B<../> directory:
>>  I<< ../<upkg>-<uversion>.tar.gz >>
>>  
>> -=item * B<uscan> invokes B<mk-origtargz> to create the source tarball: I<<
>> -../<spkg>_<oversion>.orig.tar.gz >>
>> +=item * B<uscan> invokes B<mk-origtargz> to create the source tarball:
>> +I<< ../<spkg>_<oversion>.orig.tar.gz >> (if F<debian/mk-origtargz>
>> +exists and is executable, then execute it instead).
>>  
>>  =over
>>  
>> @@ -706,7 +707,8 @@ doesn't work.
>>  
>>  B<uscan> invokes B<mk-origtargz> to create the source tarball properly named
>>  for the source package with B<.orig.> (or B<< .orig-<component>. >> for the
>> -secondary tarballs) in its filename.
>> +secondary tarballs) in its filename.  If F<debian/mk-origtargz> exists and is
>> +executable, then B<uscan> invokes it instead.
>>  
>>  =over
>>  
>> @@ -3764,7 +3766,15 @@ EOF
>>      my $path = "$destdir/$newfile_base";
>>      my $target = $newfile_base;
>>      unless ($symlink eq "no") {
>> -	my @cmd = ("mk-origtargz");
>> +	my @cmd = ();
>> +	if (-x 'debian/mk-origtargz') {
>> +	    # If the user has specified a personalized 'mk-origtargz'
>> +	    # under debian/, we use it instead of the system script.
>> +	    push @cmd "debian/mk-origtargz";
>> +	    uscan_verbose "Using debian/mk-origtargz instead of mk-origtargz\n";
>> +	} else {
>> +	    push @cmd "mk-origtargz";
>> +	}
>>  	push @cmd, "--package", $pkg;
>>  	push @cmd, "--version", $common_mangled_newversion;
>>  	push @cmd, '--repack-suffix', $options{repacksuffix} if defined $options{repacksuffix};
>
> Yes, this is what I am talking about.  Maybe, we should add
> F<debian/uupdate> in the same way. 
>
> Adding these features are easy but let's go slow.  Once we add any
> features, we can't remove them easily.  Let's wait and check if there
> are no objection or second thought.

Yeah, adding debian/update can be done later, prefereably in another bug
I think :-).

Thanks,

-- 
Sergio
GPG key ID: 237A 54B1 0287 28BF 00EF  31F4 D0EB 7628 65FC 5E36
Please send encrypted e-mail if possible
http://sergiodj.net/

[signature.asc (application/pgp-signature, inline)]

Message #79 received at [email protected] (full text, mbox, reply):

From: Christopher Hoskin <[email protected]>

To: Debian Bug Tracking System <[email protected]>

Subject: Re: implement capability to run custom mk-origtargz

Date: Thu, 31 Aug 2017 09:35:40 +0100

Package: devscripts
Version: 2.17.6+deb9u1
Followup-For: Bug #831521

Just thinking aloud here:

I wonder if it would make sense to create a debhelper plugin to add a get-orig-source sequence, with a sequence of commands, e.g. something like:

dh_uscan
dh_unpack
dh_repack
dh_exclude_files
dh_strip_icc_profiles

Packaging teams could then add commands to do common options e.g. the Java team might want a command to automatically exclude jar and war files. The commands could be overriden in debian/rules to cope with bespoke situations.

Just a thought.

Christopher

Send a report that this bug log contains spam.