flight 12436 qemu-upstream-unstable real [real]
http://www.chiark.greenend.org.uk/~xensrcts/logs/12436/
Regressions :-(
Tests which did not succeed and are blocking,
including tests which could not be run:
test-amd64-amd64-xl-win 7 windows-install fail REGR. vs. 11890
Tests which are failing intermittently (not blocking):
test-amd64-i386-xl-credit2 7 debian-install fail pass in 12434
test-i386-i386-xl-winxpsp3 7 windows-install fail pass in 12434
Regressions which are regarded as allowable (not blocking):
test-amd64-i386-qemuu-rhel6hvm-amd 9 guest-start.2 fail blocked in 11890
Tests which did not succeed, but are not blocking:
test-amd64-amd64-xl-pcipt-intel 9 guest-start fail never pass
test-amd64-i386-rhel6hvm-intel 11 leak-check/check fail never pass
test-amd64-i386-rhel6hvm-amd 11 leak-check/check fail never pass
test-amd64-amd64-xl-winxpsp3 13 guest-stop fail never pass
test-i386-i386-xl-win 13 guest-stop fail never pass
test-amd64-i386-xl-win7-amd64 13 guest-stop fail never pass
test-amd64-i386-qemuu-rhel6hvm-intel 9 guest-start.2 fail never pass
test-amd64-amd64-xl-win7-amd64 13 guest-stop fail never pass
test-amd64-i386-xl-winxpsp3-vcpus1 13 guest-stop fail never pass
test-amd64-i386-win-vcpus1 16 leak-check/check fail never pass
test-amd64-i386-xl-win-vcpus1 13 guest-stop fail never pass
test-i386-i386-win 16 leak-check/check fail never pass
test-amd64-i386-win 16 leak-check/check fail never pass
test-amd64-i386-xend-winxpsp3 16 leak-check/check fail never pass
test-amd64-amd64-win 16 leak-check/check fail never pass
test-amd64-amd64-xl-qemuu-win7-amd64 7 windows-install fail never pass
test-i386-i386-xl-qemuu-winxpsp3 7 windows-install fail never pass
test-amd64-amd64-xl-qemuu-winxpsp3 7 windows-install fail never pass
test-i386-i386-xl-winxpsp3 13 guest-stop fail in 12434 never pass
version targeted for testing:
qemuu e1615984e765751b430f86be679c0b74b5d5cd15
baseline version:
qemuu 86a8d63bc11431509506b95c1481e1a023302cbc
------------------------------------------------------------
People who touched revisions under test:
Anthony PERARD <anthony.perard@citrix.com>
Stefano Stabellini <stefano.stabellini@eu.citrix.com>
------------------------------------------------------------
jobs:
build-amd64 pass
build-i386 pass
build-amd64-oldkern pass
build-i386-oldkern pass
build-amd64-pvops pass
build-i386-pvops pass
test-amd64-amd64-xl pass
test-amd64-i386-xl pass
test-i386-i386-xl pass
test-amd64-i386-rhel6hvm-amd fail
test-amd64-i386-qemuu-rhel6hvm-amd fail
test-amd64-amd64-xl-qemuu-win7-amd64 fail
test-amd64-amd64-xl-win7-amd64 fail
test-amd64-i386-xl-win7-amd64 fail
test-amd64-i386-xl-credit2 fail
test-amd64-amd64-xl-pcipt-intel fail
test-amd64-i386-rhel6hvm-intel fail
test-amd64-i386-qemuu-rhel6hvm-intel fail
test-amd64-i386-xl-multivcpu pass
test-amd64-amd64-pair pass
test-amd64-i386-pair pass
test-i386-i386-pair pass
test-amd64-amd64-xl-sedf-pin pass
test-amd64-amd64-pv pass
test-amd64-i386-pv pass
test-i386-i386-pv pass
test-amd64-amd64-xl-sedf pass
test-amd64-i386-win-vcpus1 fail
test-amd64-i386-xl-win-vcpus1 fail
test-amd64-i386-xl-winxpsp3-vcpus1 fail
test-amd64-amd64-win fail
test-amd64-i386-win fail
test-i386-i386-win fail
test-amd64-amd64-xl-win fail
test-i386-i386-xl-win fail
test-amd64-amd64-xl-qemuu-winxpsp3 fail
test-i386-i386-xl-qemuu-winxpsp3 fail
test-amd64-i386-xend-winxpsp3 fail
test-amd64-amd64-xl-winxpsp3 fail
test-i386-i386-xl-winxpsp3 fail
------------------------------------------------------------
sg-report-flight on woking.cam.xci-test.com
logs: /home/xc_osstest/logs
images: /home/xc_osstest/images
Logs, config files, etc. are available at
http://www.chiark.greenend.org.uk/~xensrcts/logs
Test harness code can be found at
http://xenbits.xensource.com/gitweb?p=osstest.git;a=summary
Not pushing.
------------------------------------------------------------
commit e1615984e765751b430f86be679c0b74b5d5cd15
Author: Anthony PERARD <anthony.perard@citrix.com>
Date: Wed Jan 25 12:36:06 2012 +0000
xen: do not allocate RAM during INMIGRATE runstate
Signed-off-by: Anthony PERARD <anthony.perard@citrix.com>
Signed-off-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
Ian Campbell
2012-Mar-26 09:57 UTC
Re: [qemu-upstream-unstable test] 12436: regressions - FAIL
On Mon, 2012-03-26 at 06:21 +0100, xen.org wrote:> flight 12436 qemu-upstream-unstable real [real] > http://www.chiark.greenend.org.uk/~xensrcts/logs/12436/ > > Regressions :-( > > Tests which did not succeed and are blocking, > including tests which could not be run: > test-amd64-amd64-xl-win 7 windows-install fail REGR. vs. 11890Is anyone looking into this failure? [...]> ------------------------------------------------------------ > commit e1615984e765751b430f86be679c0b74b5d5cd15 > Author: Anthony PERARD <anthony.perard@citrix.com> > Date: Wed Jan 25 12:36:06 2012 +0000 > > xen: do not allocate RAM during INMIGRATE runstate > > Signed-off-by: Anthony PERARD <anthony.perard@citrix.com> > Signed-off-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com> > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel
Ian Jackson
2012-Mar-27 15:54 UTC
Re: [qemu-upstream-unstable test] 12436: regressions - FAIL
Ian Campbell writes ("Re: [Xen-devel] [qemu-upstream-unstable test] 12436:
regressions - FAIL"):> On Mon, 2012-03-26 at 06:21 +0100, xen.org wrote:
> > flight 12436 qemu-upstream-unstable real [real]
> > http://www.chiark.greenend.org.uk/~xensrcts/logs/12436/
> >
> > Regressions :-(
> >
> > Tests which did not succeed and are blocking,
> > including tests which could not be run:
> > test-amd64-amd64-xl-win 7 windows-install fail REGR.
vs. 11890
>
> Is anyone looking into this failure?
NAFAIK.
This test failure is not related to the single commit made to the
qemu-upstream-unstable tree, since this test doesn''t actually even use
qemu-upstream-unstable. (It''s a bug in the test schedule generator
that it even considers this test relevant; it should not run it.)
I have taken a look at the logs. I think it is a genuine failure
showing up a real bug, but the bug is in xen-unstable or
qemu-xen-unstable or Jeremy''s Linux 2.6.32:
The test system decided in flight 12436 that the install had failed
because the guest did not, within the timeout, start listening on the
expected tcp port (which is set up by a guest agent whose installation
and startup is built into the image I''m using). The guest screenshot,
taken after the timeout via vnc, is a black screen. The guest
rebooted itself 18 minutes prior to the timeout; it had done a number
of reboots previously (as expected). The timeout is 7000 seconds
(1h56m40s).
I have searched the database for this test being run on this host
lake-frog (or its twin, fire-frog) with xen-unstable. The most recent
such run was in an adhoc test I ran (flight 11623) for which the main
logs have expired. The database (which does not expire) does record
that the whole ts-windows-install step succeeded (on lake-frog) and
took 3660 seconds[1].
This test has also been run on frogs with xen-4.0-testing but AFAICT
it always fails there, in a different manner, regardless of which
host, so this is probably not helpful.
The most recent test run for xen-unstable ran this test successfully
on moss-bug, where it took 6122s.
So it seems likely that the problem is that the HVM Windows
installation on lake-frog takes "too long". lake-frog is unusual
amongst my test machines; it''s much larger. It''s an AMD
machine with
32G of RAM, 24 cores over 2 sockets. Naively it might be expected to
be faster but perhaps the problem is that it''s too NUMA.
Regardless, though, I think 6000-odd seconds is indeed too long if
this step previously took 3660 seconds.
Ian.
[1] This 3660 seconds includes a number of activities which are not
included in the 7000 second timeout, such as the installation of rsync
on the test host, copying of the Windows install ISO image onto the
test host, and so forth.
Ian Campbell
2012-Mar-29 13:07 UTC
Re: [qemu-upstream-unstable test] 12436: regressions - FAIL
On Tue, 2012-03-27 at 16:54 +0100, Ian Jackson wrote:> Ian Campbell writes ("Re: [Xen-devel] [qemu-upstream-unstable test] 12436: regressions - FAIL"): > > On Mon, 2012-03-26 at 06:21 +0100, xen.org wrote: > > > flight 12436 qemu-upstream-unstable real [real] > > > http://www.chiark.greenend.org.uk/~xensrcts/logs/12436/ > > > > > > Regressions :-( > > > > > > Tests which did not succeed and are blocking, > > > including tests which could not be run: > > > test-amd64-amd64-xl-win 7 windows-install fail REGR. vs. 11890 > > > > Is anyone looking into this failure? > > NAFAIK. > > This test failure is not related to the single commit made to the > qemu-upstream-unstable tree, since this test doesn''t actually even use > qemu-upstream-unstable. (It''s a bug in the test schedule generator > that it even considers this test relevant; it should not run it.)So actually the upstream qemu is actually passing its testing fairly consistently? Assuming some of the other tests do actually test it.> I have taken a look at the logs. I think it is a genuine failure > showing up a real bug, but the bug is in xen-unstable or > qemu-xen-unstable or Jeremy''s Linux 2.6.32:Do we see run this sequence in the flights which are supposed to be testing those or only here? If we do run it elsewhere then we wouldn''t be papering over the issue by removing it from this test (where it doesn''t belong).> The test system decided in flight 12436 that the install had failed > because the guest did not, within the timeout, start listening on the > expected tcp port (which is set up by a guest agent whose installation > and startup is built into the image I''m using). The guest screenshot, > taken after the timeout via vnc, is a black screen. The guest > rebooted itself 18 minutes prior to the timeout; it had done a number > of reboots previously (as expected). The timeout is 7000 seconds > (1h56m40s). > > I have searched the database for this test being run on this host > lake-frog (or its twin, fire-frog) with xen-unstable. The most recent > such run was in an adhoc test I ran (flight 11623) for which the main > logs have expired. The database (which does not expire) does record > that the whole ts-windows-install step succeeded (on lake-frog) and > took 3660 seconds[1]. > > This test has also been run on frogs with xen-4.0-testing but AFAICT > it always fails there, in a different manner, regardless of which > host, so this is probably not helpful. > > The most recent test run for xen-unstable ran this test successfully > on moss-bug, where it took 6122s. > > So it seems likely that the problem is that the HVM Windows > installation on lake-frog takes "too long". lake-frog is unusual > amongst my test machines; it''s much larger. It''s an AMD machine with > 32G of RAM, 24 cores over 2 sockets. Naively it might be expected to > be faster but perhaps the problem is that it''s too NUMA. > > Regardless, though, I think 6000-odd seconds is indeed too long if > this step previously took 3660 seconds.Right.> > Ian. > > [1] This 3660 seconds includes a number of activities which are not > included in the 7000 second timeout, such as the installation of rsync > on the test host, copying of the Windows install ISO image onto the > test host, and so forth.
Ian Jackson
2012-Mar-29 14:10 UTC
Re: [qemu-upstream-unstable test] 12436: regressions - FAIL
Ian Campbell writes ("Re: [Xen-devel] [qemu-upstream-unstable test] 12436:
regressions - FAIL"):> On Tue, 2012-03-27 at 16:54 +0100, Ian Jackson wrote:
> > This test failure is not related to the single commit made to the
> > qemu-upstream-unstable tree, since this test doesn''t actually
even use
> > qemu-upstream-unstable. (It''s a bug in the test schedule
generator
> > that it even considers this test relevant; it should not run it.)
>
> So actually the upstream qemu is actually passing its testing fairly
> consistently? Assuming some of the other tests do actually test it.
No, that''s not a valid conclusion. The tests which do test upstream
qemu aren''t doing very well - but they aren''t blocking pushes
of
upstream qemu because the baseline is another version of upstream
qemu.
> > I have taken a look at the logs. I think it is a genuine failure
> > showing up a real bug, but the bug is in xen-unstable or
> > qemu-xen-unstable or Jeremy''s Linux 2.6.32:
>
> Do we see run this sequence in the flights which are supposed to be
> testing those or only here?
Others too.
> If we do run it elsewhere then we wouldn''t be papering over the
issue by
> removing it from this test (where it doesn''t belong).
Indeed. I plan to do this but I have a other (rather fiddly) stuff
queued up, related to testing other branches of upstream Linux, which
I''m trying to get through.
Ian.
Ian Campbell
2012-Mar-29 16:18 UTC
Re: [qemu-upstream-unstable test] 12436: regressions - FAIL
On Thu, 2012-03-29 at 15:10 +0100, Ian Jackson wrote:> Ian Campbell writes ("Re: [Xen-devel] [qemu-upstream-unstable test] 12436: regressions - FAIL"): > > On Tue, 2012-03-27 at 16:54 +0100, Ian Jackson wrote: > > > This test failure is not related to the single commit made to the > > > qemu-upstream-unstable tree, since this test doesn''t actually even use > > > qemu-upstream-unstable. (It''s a bug in the test schedule generator > > > that it even considers this test relevant; it should not run it.) > > > > So actually the upstream qemu is actually passing its testing fairly > > consistently? Assuming some of the other tests do actually test it. > > No, that''s not a valid conclusion. The tests which do test upstream > qemu aren''t doing very well - but they aren''t blocking pushes of > upstream qemu because the baseline is another version of upstream > qemu.Right, I somehow missed/blanked the massive list of failures in the next section of the original mail. Lots of them are the xl shutdown needing -F thing and some leak check failures though, the actual "look like qemu upstream" failures seem to be far fewer.> > > I have taken a look at the logs. I think it is a genuine failure > > > showing up a real bug, but the bug is in xen-unstable or > > > qemu-xen-unstable or Jeremy''s Linux 2.6.32: > > > > Do we see run this sequence in the flights which are supposed to be > > testing those or only here? > > Others too. > > > If we do run it elsewhere then we wouldn''t be papering over the issue by > > removing it from this test (where it doesn''t belong). > > Indeed. I plan to do this but I have a other (rather fiddly) stuff > queued up, related to testing other branches of upstream Linux, which > I''m trying to get through.Oh yes, that''s more important I think. Thanks. Ian.