branch xen-4.3-testing
xen branch xen-4.3-testing
job build-i386
test xen-build
Tree: qemuu git://xenbits.xen.org/staging/qemu-upstream-4.3-testing.git
Tree: xen git://xenbits.xen.org/xen.git
*** Found and reproduced problem changeset ***
Bug is in tree: xen git://xenbits.xen.org/xen.git
Bug introduced: 3dfb4018d4948588ff00e32f4ab12d4715bb8c5e
Bug not present: c8d233c644cbaba24b823ddd2394e4b4e07f7d9d
commit 3dfb4018d4948588ff00e32f4ab12d4715bb8c5e
Author: Ian Jackson <ian.jackson@eu.citrix.com>
Date: Mon Jul 1 16:51:43 2013 +0100
Config.mk: Update QEMU_TAG and QEMU_UPSTREAM_REVISION for 4.3
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
For bisection revision-tuple graph see:
http://www.chiark.greenend.org.uk/~xensrcts/results/bisect.xen-4.3-testing.build-i386.xen-build.html
Revision IDs in each graph node refer, respectively, to the Trees above.
----------------------------------------
Searching for failure / basis pass:
18321 fail [host=lace-bug] / 18261 [host=potato-beetle] 18368 ok.
Failure / basis pass flights: 18321 / 18368
(tree in basispass but not in latest: qemu)
Tree: qemuu git://xenbits.xen.org/staging/qemu-upstream-4.3-testing.git
Tree: xen git://xenbits.xen.org/xen.git
Latest 7483e7f15139603380c45ebcd8cc2a57dda5583c
f8cc9c2b713b1739b1d3d324716547fa639dce86
Basis pass 7483e7f15139603380c45ebcd8cc2a57dda5583c
c8d233c644cbaba24b823ddd2394e4b4e07f7d9d
Generating revisions with ./adhoc-revtuple-generator
git://xenbits.xen.org/staging/qemu-upstream-4.3-testing.git#7483e7f15139603380c45ebcd8cc2a57dda5583c-7483e7f15139603380c45ebcd8cc2a57dda5583c
git://xenbits.xen.org/xen.git#c8d233c644cbaba24b823ddd2394e4b4e07f7d9d-f8cc9c2b713b1739b1d3d324716547fa639dce86
using cache /export/home/osstest/repos/git-cache...
using cache /export/home/osstest/repos/git-cache...
locked cache /export/home/osstest/repos/git-cache...
processing ./cacheing-git clone --bare git://xenbits.xen.org/xen.git
/export/home/osstest/repos/xen...
Initialized empty Git repository in /export/home/osstest/repos/xen/
Initialized empty Git repository in /export/home/osstest/repos/xen/
updating cache /export/home/osstest/repos/git-cache xen...
using cache /export/home/osstest/repos/git-cache...
using cache /export/home/osstest/repos/git-cache...
locked cache /export/home/osstest/repos/git-cache...
processing ./cacheing-git clone --bare git://xenbits.xen.org/xen.git
/export/home/osstest/repos/xen...
Initialized empty Git repository in /export/home/osstest/repos/xen/
updating cache /export/home/osstest/repos/git-cache xen...
Loaded 1001 nodes in revision graph
Searching for test results:
18261 [host=potato-beetle]
18321 fail 7483e7f15139603380c45ebcd8cc2a57dda5583c
f8cc9c2b713b1739b1d3d324716547fa639dce86
18346 pass irrelevant
18348 fail 7483e7f15139603380c45ebcd8cc2a57dda5583c
f8cc9c2b713b1739b1d3d324716547fa639dce86
18349 pass irrelevant
18350 fail 7483e7f15139603380c45ebcd8cc2a57dda5583c
f8cc9c2b713b1739b1d3d324716547fa639dce86
18351 fail 7483e7f15139603380c45ebcd8cc2a57dda5583c
3dfb4018d4948588ff00e32f4ab12d4715bb8c5e
18352 pass irrelevant
18353 fail 7483e7f15139603380c45ebcd8cc2a57dda5583c
f8cc9c2b713b1739b1d3d324716547fa639dce86
18354 pass irrelevant
18357 fail 7483e7f15139603380c45ebcd8cc2a57dda5583c
f8cc9c2b713b1739b1d3d324716547fa639dce86
18358 pass 7483e7f15139603380c45ebcd8cc2a57dda5583c
c8d233c644cbaba24b823ddd2394e4b4e07f7d9d
18359 fail 7483e7f15139603380c45ebcd8cc2a57dda5583c
f8cc9c2b713b1739b1d3d324716547fa639dce86
18360 pass 7483e7f15139603380c45ebcd8cc2a57dda5583c
c8d233c644cbaba24b823ddd2394e4b4e07f7d9d
18361 fail 7483e7f15139603380c45ebcd8cc2a57dda5583c
f8cc9c2b713b1739b1d3d324716547fa639dce86
18363 fail 7483e7f15139603380c45ebcd8cc2a57dda5583c
3dfb4018d4948588ff00e32f4ab12d4715bb8c5e
18365 pass 7483e7f15139603380c45ebcd8cc2a57dda5583c
c8d233c644cbaba24b823ddd2394e4b4e07f7d9d
18366 fail 7483e7f15139603380c45ebcd8cc2a57dda5583c
3dfb4018d4948588ff00e32f4ab12d4715bb8c5e
18368 pass 7483e7f15139603380c45ebcd8cc2a57dda5583c
c8d233c644cbaba24b823ddd2394e4b4e07f7d9d
18370 fail 7483e7f15139603380c45ebcd8cc2a57dda5583c
3dfb4018d4948588ff00e32f4ab12d4715bb8c5e
Searching for interesting versions
Result found: flight 18358 (pass), for basis pass
Result found: flight 18359 (fail), for basis failure
Repro found: flight 18360 (pass), for basis pass
Repro found: flight 18361 (fail), for basis failure
0 revisions at 7483e7f15139603380c45ebcd8cc2a57dda5583c
c8d233c644cbaba24b823ddd2394e4b4e07f7d9d
No revisions left to test, checking graph state.
Result found: flight 18358 (pass), for last pass
Result found: flight 18363 (fail), for first failure
Repro found: flight 18365 (pass), for last pass
Repro found: flight 18366 (fail), for first failure
Repro found: flight 18368 (pass), for last pass
Repro found: flight 18370 (fail), for first failure
*** Found and reproduced problem changeset ***
Bug is in tree: xen git://xenbits.xen.org/xen.git
Bug introduced: 3dfb4018d4948588ff00e32f4ab12d4715bb8c5e
Bug not present: c8d233c644cbaba24b823ddd2394e4b4e07f7d9d
using cache /export/home/osstest/repos/git-cache...
using cache /export/home/osstest/repos/git-cache...
locked cache /export/home/osstest/repos/git-cache...
processing ./cacheing-git clone --bare git://xenbits.xen.org/xen.git
/export/home/osstest/repos/xen...
Initialized empty Git repository in /export/home/osstest/repos/xen/
updating cache /export/home/osstest/repos/git-cache xen...
commit 3dfb4018d4948588ff00e32f4ab12d4715bb8c5e
Author: Ian Jackson <ian.jackson@eu.citrix.com>
Date: Mon Jul 1 16:51:43 2013 +0100
Config.mk: Update QEMU_TAG and QEMU_UPSTREAM_REVISION for 4.3
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Revision graph left in
/home/xc_osstest/results/bisect.xen-4.3-testing.build-i386.xen-build.{dot,ps,png,html}.
----------------------------------------
18370: tolerable ALL FAIL
flight 18370 xen-4.3-testing real-bisect [real]
http://www.chiark.greenend.org.uk/~xensrcts/logs/18370/
Failures :-/ but no regressions.
Tests which did not succeed,
including tests which could not be run:
build-i386 4 xen-build fail baseline untested
jobs:
build-i386 fail
------------------------------------------------------------
sg-report-flight on woking.cam.xci-test.com
logs: /home/xc_osstest/logs
images: /home/xc_osstest/images
Logs, config files, etc. are available at
http://www.chiark.greenend.org.uk/~xensrcts/logs
Test harness code can be found at
http://xenbits.xensource.com/gitweb?p=osstest.git;a=summary
xen.org writes ("[xen-4.3-testing bisection] complete
build-i386"):> branch xen-4.3-testing
> xen branch xen-4.3-testing
> job build-i386
> test xen-build
>
> Tree: qemuu git://xenbits.xen.org/staging/qemu-upstream-4.3-testing.git
> Tree: xen git://xenbits.xen.org/xen.git
>
> *** Found and reproduced problem changeset ***
>
> Bug is in tree: xen git://xenbits.xen.org/xen.git
> Bug introduced: 3dfb4018d4948588ff00e32f4ab12d4715bb8c5e
> Bug not present: c8d233c644cbaba24b823ddd2394e4b4e07f7d9d
... > Config.mk: Update QEMU_TAG and QEMU_UPSTREAM_REVISION for 4.3
This is because
(a) qemu-xen-4.3-testing.git (the trad tree) had not had
git-update-server-info run in it since the xen-4.3.0 tag was
made.
(b) Config.mk specifies an http url (rather than a git url) for
the qemu trees.
I have fixed this and prodded the tester into starting another 4.3
test right away.
Post mortem:
The xen-4.3.0 tag in that tree was made some time last week. Running
g-u-s-i is in the checklist. I don''t know why the checklist
wasn''t
followed properly (by me). That was clearly a mistake. Sorry.
I think, though, that our release process has become quite complex and
single the ad-hoc checklist file we (mostly I) are using is creaking
under the strain rather.
ACTION: I think we need to write down our actual relase process from a
higher-level point of view, and have a separate file for technical
processes and runes at various stages.
The autotester didn''t detect this situation before the release for the
following reasons:
1. There was an intermittent infrastructure problem which broke the
autotester entirely for much of the last few weeks, for days at a
time. The cause of this infrastructure problem (an NFS lockup) has
not been identified. We hope to collect more data if it recurs.
ACTION: We intend to migrate the many services (not just osstest)
off the server which broke, into their own VMs. But this is
dependent on effort and hardware.
2. There was a problem with the way the autotester was trying to test
4.3 testing which was actually an overlooked hg-specific rune in a
slightly ad-hoc piece of machinery (which grabs the qemu trad
version out of a particular Xen''s Config.mk). For reasons which
still aren''t clear to me, this problem only manifested in the 4.3
testing branch, which was created only recently. (This bug is now
fixed.)
3. Because of problem (1) the cron mails reporting (2) didn''t arrive
for several days. So I didn''t become aware of it.
4. I was ill in bed on Monday.
For all of these reasons, there were no autotester test results of 4.3
until today. But we knew on Tuesday that the tests were missing.
I''m not sure why we didn''t detect this problem in any of the
ad-hoc
test builds we did after changing Config.mk.
I''m not sure all the test builds we did, but I know about the ones I
did as part of the tarball process. They wouldn''t detect this
situation because the tarball is made to include copies of the checked
out qemu code; thus a test build of the resulting tarball doesn''t need
to check out qemu. (And building the tarball itself is done from
runes in the checklist which do not involve cloning a fresh qemu tree,
so you wouldn''t see it then.)
If someone did an ad-hoc test build of changing Config.mk, it is
possible that this is because one of
(a) They didn''t clean their tree _entirely thoroughly_.
"make clean" is not sufficient because it doesn''t remove
cloned
subdirectories. "git-clean -x -d -f" is not sufficient because
it recurses into the cloned subdirectories, discovers that these
are separate git repositories, and stops.
"make distclean" plus "git-clean -x -d -f" should be
sufficient.
In my test just now it is for qemu, however there is a bug in
the build system which means that it leaves
tools/firmware/seabios-dir-remote/
This is not a particularly nice situation but arguably it is
something that someone doing a final test build would have to
know. ACTION: We should perhaps write it down. Where ?
(b) Their arrangements for building things involve using a local
.config (or similar) which specify explicitly which qemu to use.
Doing this for a final release check build after editing Config.mk
would arguably be simply a mistake.
(c) Their arrangements for building things involve using a local
.config or environment setting to tell the Xen build system to use
the git protocol rather than http.
It is not clear that we could regard this as unequivocally a
mistake.
Ian.
On 11/07/13 16:07, Ian Jackson wrote:> If someone did an ad-hoc test build of changing Config.mk, it is > possible that this is because one of > > (a) They didn''t clean their tree _entirely thoroughly_. > "make clean" is not sufficient because it doesn''t remove cloned > subdirectories. "git-clean -x -d -f" is not sufficient because > it recurses into the cloned subdirectories, discovers that these > are separate git repositories, and stops.If you specify -f twice it will override this behavior and delete cloned subdirectories. So "git clean -ffdx" should really really really clean it. -George
On 11/07/13 16:07, Ian Jackson wrote:> xen.org writes ("[xen-4.3-testing bisection] complete build-i386"): >> branch xen-4.3-testing >> xen branch xen-4.3-testing >> job build-i386 >> test xen-build >> >> Tree: qemuu git://xenbits.xen.org/staging/qemu-upstream-4.3-testing.git >> Tree: xen git://xenbits.xen.org/xen.git >> >> *** Found and reproduced problem changeset *** >> >> Bug is in tree: xen git://xenbits.xen.org/xen.git >> Bug introduced: 3dfb4018d4948588ff00e32f4ab12d4715bb8c5e >> Bug not present: c8d233c644cbaba24b823ddd2394e4b4e07f7d9d > ... >> Config.mk: Update QEMU_TAG and QEMU_UPSTREAM_REVISION for 4.3 > This is because > (a) qemu-xen-4.3-testing.git (the trad tree) had not had > git-update-server-info run in it since the xen-4.3.0 tag was > made. > (b) Config.mk specifies an http url (rather than a git url) for > the qemu trees. > > I have fixed this and prodded the tester into starting another 4.3 > test right away. > > > Post mortem: > > The xen-4.3.0 tag in that tree was made some time last week. Running > g-u-s-i is in the checklist. I don''t know why the checklist wasn''t > followed properly (by me). That was clearly a mistake. Sorry. > > I think, though, that our release process has become quite complex and > single the ad-hoc checklist file we (mostly I) are using is creaking > under the strain rather. > > ACTION: I think we need to write down our actual relase process from a > higher-level point of view, and have a separate file for technical > processes and runes at various stages.I''ve put up a skeleton of a release checklist here: http://wiki.xenproject.org/wiki/ReleaseChecklist A higher-level overview, plus maybe some "be aware of this" reminders before the step-by-step might be useful -- I might add those in tomorrow (or you can do it if you have a mind). I was also thinking it might be nice to have a text-based "to-do" for the different stages somewhere, so that whoever was doing it could copy the list to their personal to-do file, and delete the items as they did them (to make sure nothing got dropped). Not sure how useful that would be. -George