Acknowledgement sent
to "Rebecca N. Palmer" <[email protected]>:
New Bug report received and forwarded. Copy sent to Debian CI team <[email protected]>.
(Sun, 23 Sep 2018 16:51:04 GMT) (full text, mbox, link).
Subject: autopkgtest/debci: misleading "Version" if a new version becomes
available during testing
Date: Sun, 23 Sep 2018 17:47:54 +0100
Package: autopkgtest
Version: 5.5
When a new version of the package under test enters the archive during a
test run, the tests towards the end of the run may use the binaries of
the new version, but debci lists it as a test of the old version.
If these tests fail because the new version contains a regression, this
gives a "fail" entry for the _old_ version, and hence the false
impression that the problem must be somewhere else.
Example: row 2018-09-20 11:04:29 of
https://ci.debian.net/packages/p/pymca/unstable/amd64/
(This is probably #909379 not a pymca regression, but this log isn't
proof it isn't)
Possibly related to #896023 / #902027, but those are about testing vs
unstable, while this is changes within a suite.
This is moderately unlikely, but not hugely so: for pymca (~15min of
tests and a big enough dependency chain that it gets tested ~2x/day),
~2% chance per upload.
I'd guess the best solution is to make this a tmpfail, as testing an
already-superseded version is probably a waste of resources.
Acknowledgement sent
to Paul Gevers <[email protected]>:
Extra info received and forwarded to list. Copy sent to Debian CI team <[email protected]>.
(Sun, 23 Sep 2018 17:09:02 GMT) (full text, mbox, link).
Hi Rebecca,
Thanks for reporting this issue. I didn't came across it yet.
On 23-09-18 18:47, Rebecca N. Palmer wrote:
> When a new version of the package under test enters the archive during a
> test run, the tests towards the end of the run may use the binaries of
> the new version, but debci lists it as a test of the old version.
A race condition indeed. It's worse than you describe, not all tests are
run with the same binaries.
> Possibly related to #896023 / #902027, but those are about testing vs
> unstable, while this is changes within a suite.
No, not related. That issue I don't consider a bug anymore, but an
autopkgtest feature. In the Debian archive testing case, we just need to
try harder to detect which version we want to test against. The old
behavior is currently not possible in our setup because we run with the
new --disable-apt-fallback option.
> This is moderately unlikely, but not hugely so: for pymca (~15min of
> tests and a big enough dependency chain that it gets tested ~2x/day),
> ~2% chance per upload.
>
> I'd guess the best solution is to make this a tmpfail, as testing an
> already-superseded version is probably a waste of resources.
What kind of logic do you have in mind?
Paul
Acknowledgement sent
to "Rebecca N. Palmer" <[email protected]>:
Extra info received and forwarded to list. Copy sent to Debian CI team <[email protected]>.
(Sun, 23 Sep 2018 17:33:04 GMT) (full text, mbox, link).
Subject: Re: Bug#909439: autopkgtest/debci: misleading "Version" if a new
version becomes available during testing
Date: Sun, 23 Sep 2018 18:29:52 +0100
On 23/09/2018 18:04, Paul Gevers wrote:
> I didn't came across it yet.
Probably because it's so rare, and easy not to notice if the status
didn't change.
> What kind of logic do you have in mind?
If any binary of the package under test is being installed and isn't the
same version as the source, abort the run as a tmpfail. (At least in a
standard debci run, as some documented uses probably want to allow such
mismatches:
https://sources.debian.org/src/autopkgtest/5.5/doc/README.running-tests.rst/#L65
)
I don't know if this is actually worth the effort; it also assumes
debci's displayed Version is the source version.
Acknowledgement sent
to Paul Gevers <[email protected]>:
Extra info received and forwarded to list. Copy sent to Debian CI team <[email protected]>.
(Sun, 23 Sep 2018 18:54:04 GMT) (full text, mbox, link).
Hi,
On 23-09-18 19:29, Rebecca N. Palmer wrote:
>> What kind of logic do you have in mind?
>
> If any binary of the package under test is being installed and isn't the
> same version as the source, abort the run as a tmpfail. (At least in a
> standard debci run, as some documented uses probably want to allow such
> mismatches:
> https://sources.debian.org/src/autopkgtest/5.5/doc/README.running-tests.rst/#L65
> )
As I mentioned in my previous reply, I consider the current behavior a
feature, so I don't like the logic you mention above. I have code laying
around for autopkgtest support of specifying the version one wants to
test. failing (or tmpfail, but I don't think I like that) in the case
where the version isn't matching the one requested makes sense to me.
> I don't know if this is actually worth the effort; it also assumes
> debci's displayed Version is the source version.
What autopkgtest outputs is the version of the package it took the test
from, which is still correct. It is however confusing in cases like this
(and in cases like those reported in bug #896023). debci just displays
whatever autopkgtest outputs here.
I guess the point is really that people, understandably, take the
version to mean something else than it really means as in most cases the
version is the same to what they believe it means. debci could stop
showing the version, but I don't think that is actually helpful except
in rare cases.
Paul
Acknowledgement sent
to "Rebecca N. Palmer" <[email protected]>:
Extra info received and forwarded to list. Copy sent to Debian CI team <[email protected]>.
(Mon, 24 Sep 2018 20:54:02 GMT) (full text, mbox, link).
Subject: Re: Bug#909439: autopkgtest/debci: misleading "Version" if a new
version becomes available during testing
Date: Mon, 24 Sep 2018 21:51:04 +0100
On 23/09/2018 19:51, Paul Gevers wrote:
> Hi,
>
> On 23-09-18 19:29, Rebecca N. Palmer wrote:
>>> What kind of logic do you have in mind?
>>
>> If any binary of the package under test is being installed and isn't the
>> same version as the source, abort the run as a tmpfail. (At least in a
>> standard debci run, as some documented uses probably want to allow such
>> mismatches:
>> https://sources.debian.org/src/autopkgtest/5.5/doc/README.running-tests.rst/#L65
>> )
>
> As I mentioned in my previous reply, I consider the current behavior a
> feature, so I don't like the logic you mention above.
What, if anything, do we actually disagree on? I've already agreed that
it should stay _possible_ to test non-matching source and binary
versions: I just don't want this happening where it is unintended and
confusing.
If the best place to fix this is outside autopkgtest, feel free to reassign.
> failing (or tmpfail, but I don't think I like that)
Why not? I suggested tmpfail because this isn't an actual problem with
the package under test, and the whole point of this bug is that debci
shouldn't claim that it is. (Debci fail = code 4, 6 or 12
https://sources.debian.org/src/debci/1.13/bin/debci-generate-index/#L168 ).
However, if we're passing a requested version to autopkgtest we should
probably check that it is updated when retrying a tmpfail, to avoid
endless retry loops.
> debci could stop
> showing the version, but I don't think that is actually helpful
Agreed that generally hiding the version would be worse. (Hiding the
version / changing it to something like 'mixed-versions' *only* when
these issues happen would be a valid solution, but may or may not be
practical.)
Acknowledgement sent
to Paul Gevers <[email protected]>:
Extra info received and forwarded to list. Copy sent to Debian CI team <[email protected]>.
(Tue, 25 Sep 2018 20:21:03 GMT) (full text, mbox, link).
Hi Rebecca,
On 24-09-18 22:51, Rebecca N. Palmer wrote:
> On 23/09/2018 19:51, Paul Gevers wrote:
>> Hi,
>>
>> On 23-09-18 19:29, Rebecca N. Palmer wrote:
>>>> What kind of logic do you have in mind?
>>>
>>> If any binary of the package under test is being installed and isn't the
>>> same version as the source, abort the run as a tmpfail. (At least in a
>>> standard debci run, as some documented uses probably want to allow such
>>> mismatches:
>>> https://sources.debian.org/src/autopkgtest/5.5/doc/README.running-tests.rst/#L65
>>>
>>> )
>>
>> As I mentioned in my previous reply, I consider the current behavior a
>> feature, so I don't like the logic you mention above.
>
> What, if anything, do we actually disagree on?
Your quote mentions a different case than the one used by debci. I
concluded that you wanted to support that case to test non-matching
versions, but not the case that debci uses. I don't want to limit debci
to actually run these mismatching combinations.
> I've already agreed that
> it should stay _possible_ to test non-matching source and binary
> versions: I just don't want this happening where it is unintended and
> confusing.
Yes, but defining that is what we are trying to do now. I don't think
defining it unambiguously is easy.
> If the best place to fix this is outside autopkgtest, feel free to
> reassign.
Well, if we can clearly define what needs fixing and how, I bet it is
autopkgtest. And until we have defined what needs fixing, we can't even
reassign.
>> failing (or tmpfail, but I don't think I like that)
>
> Why not? I suggested tmpfail because this isn't an actual problem with
> the package under test, and the whole point of this bug is that debci
> shouldn't claim that it is.
Yes, but in which exact case do you want the tmpfail? That isn't clear
to me.
> (Debci fail = code 4, 6 or 12
> https://sources.debian.org/src/debci/1.13/bin/debci-generate-index/#L168 ).
Well, these just match autopkgtest, don't they?
> However, if we're passing a requested version to autopkgtest we should
> probably check that it is updated when retrying a tmpfail, to avoid
> endless retry loops.
Well, that exactly a reason why I don't like the tmpfail. I don't
remember if debci reschedules tmpfails in unstable, but britney2 (the
tests in testing) only reschedules tmpfails if it believes they are
still needed. So from britney2 side tmpfail is fine and probably good in
the case of version mismatch.
>> debci could stop
>> showing the version, but I don't think that is actually helpful
>
> Agreed that generally hiding the version would be worse. (Hiding the
> version / changing it to something like 'mixed-versions' *only* when
> these issues happen would be a valid solution, but may or may not be
> practical.)
debci can only do that if autopkgtest supports that.
Paul
Acknowledgement sent
to "Rebecca N. Palmer" <[email protected]>:
Extra info received and forwarded to list. Copy sent to Debian CI team <[email protected]>.
(Sat, 23 Nov 2019 14:36:02 GMT) (full text, mbox, link).
https://ci.debian.net/data/autopkgtest/testing/amd64/p/python-feather-format/3483361/log.gz
appears to have the opposite problem: it finds the new source (-3), but
as this had not yet been built, it used the old binary (-2).
The testing excuses pages for both pandas and python-feather-format
currently list this as a test of -3, which it is not. The debci summary
page doesn't appear to list it at all - is that intended to fix this bug?
(It failed because -2 is uninstallable due to the Breaks: for #943925,
which is fixed in -3. A test of actual -3 has since been run, which
passed.)
Acknowledgement sent
to Paul Gevers <[email protected]>:
Extra info received and forwarded to list. Copy sent to Debian CI team <[email protected]>.
(Sat, 23 Nov 2019 20:39:06 GMT) (full text, mbox, link).
Hi Rebecca,
On 23-11-2019 15:33, Rebecca N. Palmer wrote:
> https://ci.debian.net/data/autopkgtest/testing/amd64/p/python-feather-format/3483361/log.gz
> appears to have the opposite problem: it finds the new source (-3), but
> as this had not yet been built, it used the old binary (-2).
The root cause here should be a mirror that is out-of-date, as britney
only schedules the tests when the build is done (so your statement isn't
correct). It's annoying that it can happen.
Further, this is not a huge issue (albeit confusing for people looking
into). It will be retried after one day and all should be fine.
It's the opposite that I worry about, a version X which passes while if
it would have correctly run with X+1 it would show a fail. So, a
regression that goes unnoticed due the issue you mention above.
> The testing excuses pages for both pandas and python-feather-format
> currently list this as a test of -3, which it is not. The debci summary
> page doesn't appear to list it at all - is that intended to fix this bug?
No, we changed debci a bit to only show "pure" runs in the overview
page, i.e. runs without pinned packages. Follow the link under "britney"
and you'll go to the page with all the runs scheduled by the migration
software.
> (It failed because -2 is uninstallable due to the Breaks: for #943925,
> which is fixed in -3. A test of actual -3 has since been run, which
> passed.)
Paul
Acknowledgement sent
to "Rebecca N. Palmer" <[email protected]>:
Extra info received and forwarded to list. Copy sent to Debian CI team <[email protected]>.
(Sat, 23 Nov 2019 21:03:06 GMT) (full text, mbox, link).
Subject: Re: Bug#909439: version mismatch, the other way
Date: Sat, 23 Nov 2019 20:45:50 +0000
On 23/11/2019 20:35, Paul Gevers wrote:
> Hi Rebecca,
>
> On 23-11-2019 15:33, Rebecca N. Palmer wrote:
>> https://ci.debian.net/data/autopkgtest/testing/amd64/p/python-feather-format/3483361/log.gz
>> appears to have the opposite problem: it finds the new source (-3), but
>> as this had not yet been built, it used the old binary (-2).
>
> The root cause here should be a mirror that is out-of-date, as britney
> only schedules the tests when the build is done (so your statement isn't
> correct).
The autopkgtest is timestamped 15:22, the amd64 build finish 15:47.
Maybe this test was triggered by something other than
python-feather-format itself?
> It will be retried after one day and all should be fine.
It now is.
Acknowledgement sent
to Paul Gevers <[email protected]>:
Extra info received and forwarded to list. Copy sent to Debian CI team <[email protected]>.
(Sat, 23 Nov 2019 21:27:03 GMT) (full text, mbox, link).
Hi Rebecca,
On 23-11-2019 21:45, Rebecca N. Palmer wrote:
> The autopkgtest is timestamped 15:22, the amd64 build finish 15:47.
> Maybe this test was triggered by something other than
> python-feather-format itself?
So it is, one can see that in the second line of the log:
--pin-packages=unstable=src:pandas,src:python-feather-format,src:python-skbio,src:statsmodels
The first package listed there is the triggering package. The rest come
from (versioned) Depends/Breaks/Conflicts relations.
Paul
Debbugs is free software and licensed under the terms of the GNU General
Public License version 2. The current version can be obtained
from https://bugs.debian.org/debbugs-source/.