Subject: Bug#909439: autopkgtest/debci: misleading "Version" if a new version becomes available during testing

From: Paul Gevers <[email-masked]>
Date: Tue, 25 Sep 2018 22:19:40 +0200

Hi Rebecca,

On 24-09-18 22:51, Rebecca N. Palmer wrote:
> On 23/09/2018 19:51, Paul Gevers wrote:
>> Hi,
>>
>> On 23-09-18 19:29, Rebecca N. Palmer wrote:
>>>> What kind of logic do you have in mind?
>>>
>>> If any binary of the package under test is being installed and isn't the
>>> same version as the source, abort the run as a tmpfail.  (At least in a
>>> standard debci run, as some documented uses probably want to allow such
>>> mismatches:
>>> https://sources.debian.org/src/autopkgtest/5.5/doc/README.running-tests.rst/#L65
>>>
>>> )
>>
>> As I mentioned in my previous reply, I consider the current behavior a
>> feature, so I don't like the logic you mention above.
> 
> What, if anything, do we actually disagree on?

Your quote mentions a different case than the one used by debci. I
concluded that you wanted to support that case to test non-matching
versions, but not the case that debci uses. I don't want to limit debci
to actually run these mismatching combinations.

> I've already agreed that
> it should stay _possible_ to test non-matching source and binary
> versions: I just don't want this happening where it is unintended and
> confusing.

Yes, but defining that is what we are trying to do now. I don't think
defining it unambiguously is easy.

> If the best place to fix this is outside autopkgtest, feel free to
> reassign.

Well, if we can clearly define what needs fixing and how, I bet it is
autopkgtest. And until we have defined what needs fixing, we can't even
reassign.

>> failing (or tmpfail, but I don't think I like that)
> 
> Why not?  I suggested tmpfail because this isn't an actual problem with
> the package under test, and the whole point of this bug is that debci
> shouldn't claim that it is.

Yes, but in which exact case do you want the tmpfail? That isn't clear
to me.

> (Debci fail = code 4, 6 or 12
> https://sources.debian.org/src/debci/1.13/bin/debci-generate-index/#L168 ).

Well, these just match autopkgtest, don't they?

> However, if we're passing a requested version to autopkgtest we should
> probably check that it is updated when retrying a tmpfail, to avoid
> endless retry loops.

Well, that exactly a reason why I don't like the tmpfail. I don't
remember if debci reschedules tmpfails in unstable, but britney2 (the
tests in testing) only reschedules tmpfails if it believes they are
still needed. So from britney2 side tmpfail is fine and probably good in
the case of version mismatch.

>> debci could stop
>> showing the version, but I don't think that is actually helpful
> 
> Agreed that generally hiding the version would be worse.  (Hiding the
> version / changing it to something like 'mixed-versions' *only* when
> these issues happen would be a valid solution, but may or may not be
> practical.)

debci can only do that if autopkgtest supports that.

Paul