Subject: Re: Bug#909439: autopkgtest/debci: misleading "Version" if a new version becomes available during testing

From: "Rebecca N. Palmer" <[email-masked]>
Date: Mon, 24 Sep 2018 21:51:04 +0100

On 23/09/2018 19:51, Paul Gevers wrote:
> Hi,
> 
> On 23-09-18 19:29, Rebecca N. Palmer wrote:
>>> What kind of logic do you have in mind?
>>
>> If any binary of the package under test is being installed and isn't the
>> same version as the source, abort the run as a tmpfail.  (At least in a
>> standard debci run, as some documented uses probably want to allow such
>> mismatches:
>> https://sources.debian.org/src/autopkgtest/5.5/doc/README.running-tests.rst/#L65
>> )
> 
> As I mentioned in my previous reply, I consider the current behavior a
> feature, so I don't like the logic you mention above.

What, if anything, do we actually disagree on?  I've already agreed that 
it should stay _possible_ to test non-matching source and binary 
versions: I just don't want this happening where it is unintended and 
confusing.

If the best place to fix this is outside autopkgtest, feel free to reassign.

> failing (or tmpfail, but I don't think I like that)

Why not?  I suggested tmpfail because this isn't an actual problem with 
the package under test, and the whole point of this bug is that debci 
shouldn't claim that it is.  (Debci fail = code 4, 6 or 12 
https://sources.debian.org/src/debci/1.13/bin/debci-generate-index/#L168 ).

However, if we're passing a requested version to autopkgtest we should 
probably check that it is updated when retrying a tmpfail, to avoid 
endless retry loops.

> debci could stop
> showing the version, but I don't think that is actually helpful

Agreed that generally hiding the version would be worse.  (Hiding the 
version / changing it to something like 'mixed-versions' *only* when 
these issues happen would be a valid solution, but may or may not be 
practical.)