Subject: uscan: Implement new 'nomktar' option on watch file

From: Sergio Durigan Junior <[email-masked]>
Date: Sat, 16 Jul 2016 17:51:21 -0400

Package: devscripts
Tags: patch

Hi,

Rationale: <https://lists.debian.org/debian-mentors/2016/07/msg00389.html>

Some Debian maintainers have to deal with upstream projects that do not
offer tarballs for releases.  The modus operandi for working around this
"issue" is to provide your own replacement for the 'uupdate' script,
which should be responsible for manipulating the file(s) downloaded from
upstream and prepare a "fake" tarball to be provided to 'uupdate'.

This works, but has one drawback: there is no "easy" way to suppress the
'mk-origtargz' program to run.  And the problem with this is that
'mk-origtargz' expects a compressed tarball as its input, always.
Calling it in the situation described above leads to an error.  So, the
user is left with some non-ideal options:

1) Invoke 'uscan' passing '--no-symlink' as one of the parameters.  This
has the side-effect of skipping the call to 'mk-origtargz'.  The obvious
disadvantages are: (a) it is not obvious to the user that this option
solves the problem described above, and (b) the user will have to
remember to pass this option every time.

2) Hacking the 'uupdate'-replacement script in order to invoke it by
hand.  The obvious disadvantage is that the user won't be able to use
'uscan', and also will have to remember this every time.

Therefore, to fix this issue, I am proposing a new option, to be
provided via 'opts=' in the watch file, whose purpose is to suppress the
execution of 'mk-origtargz'.  Its effect will be similar to passing
'--no-symlink', although the user will still be able to provide
'--rename', for example.

Opinions?

Cheers,

-- 
Sergio
GPG key ID: 237A 54B1 0287 28BF 00EF  31F4 D0EB 7628 65FC 5E36
Please send encrypted e-mail if possible
http://sergiodj.net/

From b40dbd4f6b052f0753aede3822a372149d8d0c7f Mon Sep 17 00:00:00 2001
From: Sergio Durigan Junior <[email-masked]>
Date: Sat, 16 Jul 2016 17:36:29 -0400
Subject: [PATCH] uscan: Implement 'nomktar' option to allow user to specify
 when she does not want mk-origtargz to be invoked.

---
 debian/changelog |  6 ++++++
 scripts/uscan.pl | 11 ++++++++++-
 2 files changed, 16 insertions(+), 1 deletion(-)

diff --git a/debian/changelog b/debian/changelog
index 3b9e5a0..3207485 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,8 +1,14 @@
 devscripts (2.16.7) UNRELEASED; urgency=medium
 
+  [ Paul Wise ]
   * grep-excuses:
     + Fix the script for the removal of testing.pl from release.debian.org
 
+  [ Sergio Durigan Junior ]
+  * uscan:
+    + Implement 'nomktar' option to allow user to specify when she does
+      not want mk-origtargz to be invoked.
+
  -- Paul Wise <[email-masked]>  Fri, 15 Jul 2016 22:36:19 +0800
 
 devscripts (2.16.6) unstable; urgency=medium
diff --git a/scripts/uscan.pl b/scripts/uscan.pl
index 9403ba6..818292e 100755
--- a/scripts/uscan.pl
+++ b/scripts/uscan.pl
@@ -408,6 +408,13 @@ No signature available. (No warning.)
 
 Decompress compressed archive before the pgp/gpg signature verification.
 
+=item B<nomktar>
+
+Do not invoke B<mk-origtargz> after saving the downloaded file.  This
+is useful if you are dealing with files that are not tarballs, for
+example.  The side effect of this option is that no symlink will be
+created.
+
 =item B<bare>
 
 Disable all site specific special case code such as URL redirector uses and
@@ -2551,6 +2558,8 @@ sub process_watchline ($$$$$$)
 		    $options{'pgpmode'} = 'mangle';
 		} elsif ($opt =~ /^\s*oversionmangle\s*=\s*(.+?)\s*$/) {
 		    @{$options{'oversionmangle'}} = split /;/, $1;
+		} elsif ($opt =~ /^\s*nomktar\s*$/) {
+		    $options{'nomktar'} = 1;
 		} else {
 		    uscan_warn "unrecognised option $opt\n";
 		}
@@ -3700,7 +3709,7 @@ EOF
     my $mk_origtargz_out;
     my $path = "$destdir/$newfile_base";
     my $target = $newfile_base;
-    unless ($symlink eq "no") {
+    unless ($symlink eq "no" || $options{'nomktar'} ) {
 	my @cmd = ("mk-origtargz");
 	push @cmd, "--package", $pkg;
 	push @cmd, "--version", $common_mangled_newversion;
-- 
2.8.1

---

From: Osamu Aoki <[email-masked]>
Date: Mon, 18 Jul 2016 00:14:03 +0900

Hi,

> '--no-symlink', although the user will still be able to provide

This is certainly one way to solve your problem .... and I see some
people abused uupdate slot for this purpose.

But replacing uupdate (updating source tree from repackaged tarball and
old source tree) with a command for another purpose seems to complicate
situation.  We want to run your alternative command in place of
mk-origtargz.  How about ...  if debian/mk-mk-origtargz exists, use it
instead of mk-origtargz.

I know there are some resentments to run a script in debian/* but this
part is not executed when just scanning archive.  So situation is no
worse than uupdate situation.

Currently, mk-origtargz output is scanned to set some parameters so your
mk-origtargz may need to be written carefully.  That's the negative of
this approach but certainly this is another way to solve your problem.

By the way, can you tall me exact case you are thinking so we know we
are attacking the real problem (not just theoretical cases).

Note:

There is a question on how you call your alternative uupdate command
$cmd.  Currently, uscan invokes non-uupdate $cmd as:
 Version 4  : "$cmd --upstream-version <version>"
 Version 2,3: "$cmd --upstream-version <version> <path>"
 Version 1  : "$cmd <path> <version>"
This <path> is the full path filename of the file downloaded.

Maybe, I should also make version 4 to do the same as version 3 if $cmd
is not uupdate.  If $cmd is uupdate for version 4, it calls
 Version 4  : "uupdate --find --upstream-version <version>"
(This is because the <path> should be pointing the file in ../)

Anyway, these needs to be updated with documentation if this is
something needs to be done...

Osamu

---

From: Gianfranco Costamagna <[email-masked]>
Date: Mon, 18 Jul 2016 07:51:36 +0000 (UTC)

Hi Osamu,

>are attacking the real problem (not just theoretical cases).

You can see an use case here [1] with another use case in one of my replies to
the thread, because I can't use uscan to download virtualbox-guest-additions
iso file.

[1] https://lists.debian.org/debian-mentors/2016/07/msg00389.html

hope this helps,

Gianfranco

---

From: Osamu Aoki <[email-masked]>
Date: Mon, 28 Aug 2017 01:04:54 +0900

control: tags -1 - patch
control: severity -1 wishlist
control: retitle -1 implement capability to run custom mk-origtargz
control: severity 858319 wishlist
control: retitle 858319 implement capability to run custom mk-origtargz
control: merge 858319 831521

Hi,

Just because uupdate entry in watch file enables to run custom script,
abusing it is not good idea.

uscan deligates its key functions to:
 * repacking of upstream tarball/signature -- mk-origtargz
 * updating new sorce tree                 -- uupdate

So adding uscan a capability to run custom script in place of calling
mk-origtargz is the right approach.

That's exactly like repack.stub discussed in # 858319

I need to address these bugs without making overcomplicated uscan to do
more work.  Maybe if debian/mk-origtargz exists, it is used instead of
the default one.   That's simple.  This also enables if someone have
good script, I can merge into mk-origtargz.

Regards,

Osamu

---

From: Sergio Durigan Junior <[email-masked]>
Date: Sun, 27 Aug 2017 17:24:43 -0400

Hi Osamu,

Thanks for following up this bug.  I confess I had forgotten about it!

Anyway, I took the liberty and implemented the following mostly-untested
patch to uscan which uses your idea and executes debian/mk-origtargz (if
it exists and is executable) instead of the system's mk-origtargz.

What do you thing?

diff --git a/scripts/uscan.pl b/scripts/uscan.pl
index f871daf2..49a7acb2 100755
--- a/scripts/uscan.pl
+++ b/scripts/uscan.pl
@@ -60,8 +60,9 @@ than the last upstream version.
 =item * B<uscan> saves the downloaded tarball to the parent B<../> directory:
 I<< ../<upkg>-<uversion>.tar.gz >>
 
-=item * B<uscan> invokes B<mk-origtargz> to create the source tarball: I<<
-../<spkg>_<oversion>.orig.tar.gz >>
+=item * B<uscan> invokes B<mk-origtargz> to create the source tarball:
+I<< ../<spkg>_<oversion>.orig.tar.gz >> (if F<debian/mk-origtargz>
+exists and is executable, then execute it instead).
 
 =over
 
@@ -706,7 +707,8 @@ doesn't work.
 
 B<uscan> invokes B<mk-origtargz> to create the source tarball properly named
 for the source package with B<.orig.> (or B<< .orig-<component>. >> for the
-secondary tarballs) in its filename.
+secondary tarballs) in its filename.  If F<debian/mk-origtargz> exists and is
+executable, then B<uscan> invokes it instead.
 
 =over
 
@@ -3764,7 +3766,15 @@ EOF
     my $path = "$destdir/$newfile_base";
     my $target = $newfile_base;
     unless ($symlink eq "no") {
-	my @cmd = ("mk-origtargz");
+	my @cmd = ();
+	if (-x 'debian/mk-origtargz') {
+	    # If the user has specified a personalized 'mk-origtargz'
+	    # under debian/, we use it instead of the system script.
+	    push @cmd "debian/mk-origtargz";
+	    uscan_verbose "Using debian/mk-origtargz instead of mk-origtargz\n";
+	} else {
+	    push @cmd "mk-origtargz";
+	}
 	push @cmd, "--package", $pkg;
 	push @cmd, "--version", $common_mangled_newversion;
 	push @cmd, '--repack-suffix', $options{repacksuffix} if defined $options{repacksuffix};

---

From: Osamu Aoki <[email-masked]>
Date: Mon, 28 Aug 2017 23:12:56 +0900

Hi,

> it exists and is executable) instead of the system's mk-origtargz.

Thanks 
 
>  	push @cmd, '--repack-suffix', $options{repacksuffix} if defined $options{repacksuffix};

Yes, this is what I am talking about.  Maybe, we should add
F<debian/uupdate> in the same way. 

Adding these features are easy but let's go slow.  Once we add any
features, we can't remove them easily.  Let's wait and check if there
are no objection or second thought.

Regards,

Osamu

---

From: Sergio Durigan Junior <[email-masked]>
Date: Wed, 30 Aug 2017 20:10:50 -0400

> are no objection or second thought.

Yeah, adding debian/update can be done later, prefereably in another bug
I think :-).

Thanks,