Debian Bug report logs - #994139
lintian: warning about superficial autopkgtests is counterproductive

Full log

🔗 View this message in rfc822 format

X-Loop: [email protected]
Subject: Bug#994139: lintian: warning about superficial autopkgtests is counterproductive
Reply-To: Simon McVittie <[email protected]>, [email protected]
Resent-From: Simon McVittie <[email protected]>
Resent-To: [email protected]
Resent-CC: Debian Lintian Maintainers <[email protected]>
X-Loop: [email protected]
Resent-Date: Sun, 12 Sep 2021 20:21:07 +0000
Resent-Message-ID: <[email protected]>
Resent-Sender: [email protected]
X-Debian-PR-Message: followup 994139
X-Debian-PR-Package: lintian
X-Debian-PR-Keywords: 
References: <[email protected]> <CAFHYt56hBQRMy-88+bN+JsEhxrp_RUXmrmA6jnnrNcuAOC+83Q@mail.gmail.com> <[email protected]>
X-Debian-PR-Source: lintian
Received: via spool by [email protected] id=B994139.163147786026169
          (code B ref 994139); Sun, 12 Sep 2021 20:21:07 +0000
Received: (at 994139) by bugs.debian.org; 12 Sep 2021 20:17:40 +0000
X-Spam-Checker-Version: SpamAssassin 3.4.2-bugs.debian.org_2005_01_02
	(2018-09-13) on buxtehude.debian.org
X-Spam-Level: 
X-Spam-Status: No, score=-20.3 required=4.0 tests=ALL_TRUSTED,BAYES_00,
	DKIMWL_WL_HIGH,DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,
	FOURLA,FROMDEVELOPER,HAS_BUG_NUMBER,SPF_HELO_NONE,SPF_NONE,TXREP
	autolearn=ham autolearn_force=no
	version=3.4.2-bugs.debian.org_2005_01_02
X-Spam-Bayes: score:0.0000 Tokens: new, 17; hammy, 150; neutral, 157; spammy,
	0. spammytokens: hammytokens:0.000-+--autopkgtest, 0.000-+--Lintian,
	0.000-+--Hx-spam-relays-external:36ff, 0.000-+--H*r:36ff,
	0.000-+--H*F:U*smcv
Received: from master.debian.org ([2001:41b8:202:deb:216:36ff:fe40:4001]:50850)
	from C=NA,ST=NA,L=Ankh Morpork,O=Debian SMTP,OU=Debian SMTP CA,CN=master.debian.org,[email protected] (verified)
	by buxtehude.debian.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256)
	(Exim 4.92)
	(envelope-from <[email protected]>)
	id 1mPVv1-0006nt-NV
	for [email protected]; Sun, 12 Sep 2021 20:17:39 +0000
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=debian.org;
	 s=smtpauto.master; h=In-Reply-To:Content-Type:MIME-Version:References:
	Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-Transfer-Encoding:
	Content-ID:Content-Description;
	bh=F3CH0bklL8Nujx5Axv8xdGy0aHqJwtV4AiAEibA63vo=; b=wZlSndwwTIWB2/mMcVR0ZOzTQF
	Lwq8E+H7hTPEeNdhin5ghvWQIY7pSxT5b7eFqrJMXWXjDOvtqg9RLv6pcEex+wrTN4BYES+G4qm8h
	R+YVIz0R7kDbzNpI0rtXhdpheS33OoPAgjNpUQbB5nYK4CaE020we0KTiDIMTvtWbyc7V6wzn4xoP
	1uB00PbbYqp2giCbQroMioclrr3VAyRwZdF6fwuwnFdwyB77pWar0gQBCE4ySjU0tSulo5An1Dzl5
	voFhrz7uFoImi8EPSNhF/b+DY/xuE9Jppul2v5MFE0pWjT2cLK+jCFXYxA008kil8p7j/6Jlnmg4m
	6rjLuMxA==;
Received: from smcv by master.debian.org with local (Exim 4.92)
	(envelope-from <[email protected]>)
	id 1mPVuy-0003kN-G8; Sun, 12 Sep 2021 20:17:36 +0000
Date: Sun, 12 Sep 2021 21:17:30 +0100
From: Simon McVittie <[email protected]>
To: Felix Lechner <[email protected]>
Cc: [email protected], Daniel Kahn Gillmor <[email protected]>,
	Paul Wise <[email protected]>, [email protected]
Message-ID: <[email protected]>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <CAFHYt56hBQRMy-88+bN+JsEhxrp_RUXmrmA6jnnrNcuAOC+83Q@mail.gmail.com>

On Sun, 12 Sep 2021 at 12:48:36 -0700, Felix Lechner wrote:
> On Sun, Sep 12, 2021 at 11:09 AM Simon McVittie <[email protected]> wrote:
> >
> > lintian has recently started emitting warnings for packages that
> > have autopkgtests, but only superficial autopkgtests.
> 
> The tag was implemented in response to Bug#932870. [1] It was
> originally suggested on OFTC#debci by dkg (who was copied here). I
> never heard back whether my implementation met the original intent.
> The tag originally had the info visibility, but was later upgraded to
> a warning. [2]

To be clear, I think this tag would have been fine as originally
implemented; it's the subsequent increase of severity from pedantic
to warning that I think is counterproductive.

I think a package with correctly-marked superficial autopkgtests,
like libsdl2-mixer 2.0.4+dfsg1-3, is better than a package with no
autopkgtests at all, like libsdl2-mixer 2.0.4+dfsg1-2. As a result,
to avoid giving maintainers wrong incentives, I think the Lintian tags
emitted for 2.0.4+dfsg1-3 should be of equal or lower severity.

> > the meaning of
> > testsuite-autopkgtest-missing might have changed at some point so that it
> > is only emitted for packages that have debian/tests/control, rather than
> > for all packages that lack autopkgtests?
> 
> Maybe a word is missing? The tag 'testsuite-autopkgtest-missing' was
> renamed to 'missing-tests-control' and is issued at the info level
> when sources do NOT ship debian/tests/control (or declare a predefined
> test suite in d/control, usually for teams). [3]

If that's the case, I would have expected it to be emitted for packages
that have absolutely no autopkgtest coverage, such as
gnome-shell-extension-caffeine. In the past, I think I saw
testsuite-autopkgtest-missing emitted for that package, but at the moment,
the replacement tag missing-tests-control is not emitted. Should it be?

The missing-tests-control description now says:

    The source package declares the generic <code>Testsuite: autopkgtest</code>
    field but provides no <code>debian/tests/control</code> file.

but packages like gnome-shell-extension-caffeine that don't have any
tests at all will not usually have a Testsuite field. If the intention
was for this tag to detect packages with zero test coverage, then it
should also be emitted for packages with no Testsuite.

GNOME shell extensions are probably a convenient example to use when
evaluating "you have no tests" tags, because they are unlikely to
get meaningful test coverage: they're essentially impossible to test
without using something like openQA, which would be a lot of effort
and inconveniently likely to fail as a result of changes that are
not actually bugs, and it isn't obvious how it would fit into the
autopkgtest framework even if someone has the time to implement it. I
think if we get test coverage for desktop environments, it would be more
likely to be a separate, parallel GUI-testing infrastructure alongside
autopkgtest.

    smcv

Send a report that this bug log contains spam.