flight 2045 xen-4.0-testing real
http://www.chiark.greenend.org.uk/~xensrcts/logs/2045/
Regressions :-(
tests which did not succeed:
test-amd64-amd64-pair <none executed> fail
test-amd64-amd64-xl 10 guest-stop fail never pass
test-amd64-i386-pair 12 guest-migrate/src_host/dst_host fail REGR. vs. 1993
test-amd64-i386-win 5 windows-install fail REGR. vs. 1993
test-amd64-i386-xl 10 guest-stop fail never pass
test-amd64-xcpkern-i386-xl 10 guest-stop fail never pass
test-i386-i386-pair 12 guest-migrate/src_host/dst_host fail never pass
test-i386-i386-win 5 windows-install fail REGR. vs. 1993
test-i386-i386-xl 10 guest-stop fail never pass
test-i386-xcpkern-i386-pair 12 guest-migrate/src_host/dst_host fail like 1993
test-i386-xcpkern-i386-xl 10 guest-stop fail never pass
version targeted for testing:
xen 7b8b976f534e
baseline version:
xen 8e8dd38374e9
jobs:
build-amd64 pass
build-amd64-oldkern pass
build-i386 pass
build-i386-oldkern pass
build-i386-xcpkern pass
test-amd64-amd64-pair fail
test-amd64-amd64-pv pass
test-amd64-amd64-win pass
test-amd64-amd64-xl fail
test-amd64-i386-pair fail
test-amd64-i386-pv pass
test-amd64-i386-win fail
test-amd64-i386-xl fail
test-amd64-xcpkern-i386-pair pass
test-amd64-xcpkern-i386-pv pass
test-amd64-xcpkern-i386-win pass
test-amd64-xcpkern-i386-xl fail
test-i386-i386-pair fail
test-i386-i386-pv pass
test-i386-i386-win fail
test-i386-i386-xl fail
test-i386-xcpkern-i386-pair fail
test-i386-xcpkern-i386-pv pass
test-i386-xcpkern-i386-win pass
test-i386-xcpkern-i386-xl fail
-------------------------------------------------------------------------------
build-amd64:
1 host-install(1) pass
2 host-build-prep pass
3 xen-build pass
linux e6b9b2cbca5093e8e38d
qemu 0a940d892e90c820567c
xen 21323:7b8b976f534e
-------------------------------------------------------------------------------
build-amd64-oldkern:
1 xen-build pass
linux 1025:042c8cd71081
qemu 0a940d892e90c820567c
xen 21323:7b8b976f534e
-------------------------------------------------------------------------------
build-i386:
1 host-install(1) pass
2 host-build-prep pass
3 xen-build pass
linux e6b9b2cbca5093e8e38d
qemu 0a940d892e90c820567c
xen 21323:7b8b976f534e
-------------------------------------------------------------------------------
build-i386-oldkern:
1 xen-build pass
linux 1025:042c8cd71081
qemu 0a940d892e90c820567c
xen 21323:7b8b976f534e
-------------------------------------------------------------------------------
build-i386-xcpkern:
1 kernel-build pass
linux 110879:32fc6955a6a5
pq_linux 154:61499f9e9a87
-------------------------------------------------------------------------------
test-amd64-amd64-pair:
1 xen-build-check(1) pass
2 host-install/src_host(2) pass
3 host-install/dst_host(3) pass
4 xen-install/src_host pass
5 xen-install/dst_host pass
6 capture-logs/src_host(6) pass
7 capture-logs/dst_host(7) pass
-------------------------------------------------------------------------------
test-amd64-amd64-pv:
1 xen-build-check(1) pass
2 host-install(2) pass
3 xen-install pass
4 xen-boot pass
5 debian-install pass
6 debian-fixup pass
7 guest-start pass
8 guest-saverestore pass
9 guest-localmigrate pass
10 guest-stop pass
11 capture-logs(11) pass
-------------------------------------------------------------------------------
test-amd64-amd64-win:
1 xen-build-check(1) pass
2 host-install(2) pass
3 xen-install pass
4 xen-boot pass
5 windows-install pass
6 guest-saverestore pass
7 guest-localmigrate pass
8 guest-stop pass
9 capture-logs(9) pass
-------------------------------------------------------------------------------
test-amd64-amd64-xl:
1 xen-build-check(1) pass
2 host-install(2) pass
3 xen-install pass
4 xen-boot pass
5 debian-install pass
6 debian-fixup pass
7 guest-start pass
8 guest-saverestore pass
9 guest-localmigrate pass
10 guest-stop fail
11 capture-logs(11) pass
-------------------------------------------------------------------------------
test-amd64-i386-pair:
1 xen-build-check(1) pass
2 host-install/src_host(2) pass
3 host-install/dst_host(3) pass
4 xen-install/src_host pass
5 xen-install/dst_host pass
6 xen-boot/src_host pass
7 xen-boot/dst_host pass
8 debian-install/dst_host pass
9 debian-fixup/dst_host pass
10 guests-nbd-mirror pass
11 guest-start pass
12 guest-migrate/src_host/dst_host fail
13 capture-logs/src_host(13) pass
14 capture-logs/dst_host(14) pass
-------------------------------------------------------------------------------
test-amd64-i386-pv:
1 xen-build-check(1) pass
2 host-install(2) pass
3 xen-install pass
4 xen-boot pass
5 debian-install pass
6 debian-fixup pass
7 guest-start pass
8 guest-saverestore pass
9 guest-localmigrate pass
10 guest-stop pass
11 capture-logs(11) pass
-------------------------------------------------------------------------------
test-amd64-i386-win:
1 xen-build-check(1) pass
2 host-install(2) pass
3 xen-install pass
4 xen-boot pass
5 windows-install fail
6 capture-logs(6) pass
-------------------------------------------------------------------------------
test-amd64-i386-xl:
1 xen-build-check(1) pass
2 host-install(2) pass
3 xen-install pass
4 xen-boot pass
5 debian-install pass
6 debian-fixup pass
7 guest-start pass
8 guest-saverestore pass
9 guest-localmigrate pass
10 guest-stop fail
11 capture-logs(11) pass
-------------------------------------------------------------------------------
test-amd64-xcpkern-i386-pair:
1 xen-build-check(1) pass
2 host-install/src_host(2) pass
3 host-install/dst_host(3) pass
4 xen-install/src_host pass
5 xen-install/dst_host pass
6 xen-boot/src_host pass
7 xen-boot/dst_host pass
8 debian-install/dst_host pass
9 debian-fixup/dst_host pass
10 guests-nbd-mirror pass
11 guest-start pass
12 guest-migrate/src_host/dst_host pass
13 guest-migrate/dst_host/src_host pass
14 capture-logs/src_host(14) pass
15 capture-logs/dst_host(15) pass
-------------------------------------------------------------------------------
test-amd64-xcpkern-i386-pv:
1 xen-build-check(1) pass
2 host-install(2) pass
3 xen-install pass
4 xen-boot pass
5 debian-install pass
6 debian-fixup pass
7 guest-start pass
8 guest-saverestore pass
9 guest-localmigrate pass
10 guest-stop pass
11 capture-logs(11) pass
-------------------------------------------------------------------------------
test-amd64-xcpkern-i386-win:
1 xen-build-check(1) pass
2 host-install(2) pass
3 xen-install pass
4 xen-boot pass
5 windows-install pass
6 guest-saverestore pass
7 guest-localmigrate pass
8 guest-stop pass
9 capture-logs(9) pass
-------------------------------------------------------------------------------
test-amd64-xcpkern-i386-xl:
1 xen-build-check(1) pass
2 host-install(2) pass
3 xen-install pass
4 xen-boot pass
5 debian-install pass
6 debian-fixup pass
7 guest-start pass
8 guest-saverestore pass
9 guest-localmigrate pass
10 guest-stop fail
11 capture-logs(11) pass
-------------------------------------------------------------------------------
test-i386-i386-pair:
1 xen-build-check(1) pass
2 host-install/src_host(2) pass
3 host-install/dst_host(3) pass
4 xen-install/src_host pass
5 xen-install/dst_host pass
6 xen-boot/src_host pass
7 xen-boot/dst_host pass
8 debian-install/dst_host pass
9 debian-fixup/dst_host pass
10 guests-nbd-mirror pass
11 guest-start pass
12 guest-migrate/src_host/dst_host fail
13 capture-logs/src_host(13) pass
14 capture-logs/dst_host(14) pass
-------------------------------------------------------------------------------
test-i386-i386-pv:
1 xen-build-check(1) pass
2 host-install(2) pass
3 xen-install pass
4 xen-boot pass
5 debian-install pass
6 debian-fixup pass
7 guest-start pass
8 guest-saverestore pass
9 guest-localmigrate pass
10 guest-stop pass
11 capture-logs(11) pass
-------------------------------------------------------------------------------
test-i386-i386-win:
1 xen-build-check(1) pass
2 host-install(2) pass
3 xen-install pass
4 xen-boot pass
5 windows-install fail
6 capture-logs(6) pass
-------------------------------------------------------------------------------
test-i386-i386-xl:
1 xen-build-check(1) pass
2 host-install(2) pass
3 xen-install pass
4 xen-boot pass
5 debian-install pass
6 debian-fixup pass
7 guest-start pass
8 guest-saverestore pass
9 guest-localmigrate pass
10 guest-stop fail
11 capture-logs(11) pass
-------------------------------------------------------------------------------
test-i386-xcpkern-i386-pair:
1 xen-build-check(1) pass
2 host-install/src_host(2) pass
3 host-install/dst_host(3) pass
4 xen-install/src_host pass
5 xen-install/dst_host pass
6 xen-boot/src_host pass
7 xen-boot/dst_host pass
8 debian-install/dst_host pass
9 debian-fixup/dst_host pass
10 guests-nbd-mirror pass
11 guest-start pass
12 guest-migrate/src_host/dst_host fail
13 capture-logs/src_host(13) pass
14 capture-logs/dst_host(14) pass
-------------------------------------------------------------------------------
test-i386-xcpkern-i386-pv:
1 xen-build-check(1) pass
2 host-install(2) pass
3 xen-install pass
4 xen-boot pass
5 debian-install pass
6 debian-fixup pass
7 guest-start pass
8 guest-saverestore pass
9 guest-localmigrate pass
10 guest-stop pass
11 capture-logs(11) pass
-------------------------------------------------------------------------------
test-i386-xcpkern-i386-win:
1 xen-build-check(1) pass
2 host-install(2) pass
3 xen-install pass
4 xen-boot pass
5 windows-install pass
6 guest-saverestore pass
7 guest-localmigrate pass
8 guest-stop pass
9 capture-logs(9) pass
-------------------------------------------------------------------------------
test-i386-xcpkern-i386-xl:
1 xen-build-check(1) pass
2 host-install(2) pass
3 xen-install pass
4 xen-boot pass
5 debian-install pass
6 debian-fixup pass
7 guest-start pass
8 guest-saverestore pass
9 guest-localmigrate pass
10 guest-stop fail
11 capture-logs(11) pass
------------------------------------------------------------
sg-report-flight on woking.cam.xci-test.com
logs: /home/xc_osstest/logs
images: /home/xc_osstest/images
Logs, config files, etc. are available at
http://www.chiark.greenend.org.uk/~xensrcts/logs
Test harness code can be found at
http://xenbits.xensource.com/gitweb?p=osstest.git;a=summary
Not pushing.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Pasi Kärkkäinen
2010-Aug-25 15:40 UTC
Re: [Xen-devel] [xen-4.0-testing test] 2045: regressions - FAIL
On Wed, Aug 25, 2010 at 03:21:50PM +0100, xen.org wrote:> flight 2045 xen-4.0-testing real > http://www.chiark.greenend.org.uk/~xensrcts/logs/2045/ > > Regressions :-( > > tests which did not succeed: > test-amd64-amd64-pair <none executed> fail > test-amd64-amd64-xl 10 guest-stop fail never pass > test-amd64-i386-pair 12 guest-migrate/src_host/dst_host fail REGR. vs. 1993 > test-amd64-i386-win 5 windows-install fail REGR. vs. 1993 > test-amd64-i386-xl 10 guest-stop fail never pass > test-amd64-xcpkern-i386-xl 10 guest-stop fail never pass > test-i386-i386-pair 12 guest-migrate/src_host/dst_host fail never pass > test-i386-i386-win 5 windows-install fail REGR. vs. 1993 > test-i386-i386-xl 10 guest-stop fail never pass > test-i386-xcpkern-i386-pair 12 guest-migrate/src_host/dst_host fail like 1993 > test-i386-xcpkern-i386-xl 10 guest-stop fail never pass >So ok, half of these failures are xl related.. How about the other failures? -- Pasi _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ian Jackson
2010-Aug-25 18:25 UTC
Re: [Xen-devel] [xen-4.0-testing test] 2045: regressions - FAIL
Pasi Kärkkäinen writes ("Re: [Xen-devel] [xen-4.0-testing test] 2045:
regressions - FAIL"):> So ok, half of these failures are xl related..
> How about the other failures?
These failures are as follows:
test-amd64-i386-pair 12 guest-migrate/src_host/dst_host fail REGR. vs. 1993
It seems that PV migration of a domain from one host to another
doesn''t work properly. Looking at the logs everything is absolutely
fine except that the domU isn''t responding to arps and pings for at
least the 20s that the test system waits 20s after migration.
test-amd64-i386-win 5 windows-install fail REGR. vs. 1993
test-i386-i386-win 5 windows-install fail REGR. vs. 1993
These were HVM installs of Windows. The timed out after 7000s.
Looking at the logs, the test harness tried to collect various
information including screenshots, but this generally failed because
the dom0 was running very very slowly. For example,
ssh root@<dom0> brctl show
took 7 secons to run; and
ssh root@<dom0> xenstore-read <some thing to find VNC port>
took more than 10 seconds which made the test harness time it out.
I have seen these symptoms myself: dom0 can become very unresponsive
indeed, sometimes locking up (or at least, not listening to the
network) for tens of seconds at a time.
test-amd64-amd64-xl 10 guest-stop fail never pass
test-amd64-i386-xl 10 guest-stop fail never pass
test-amd64-xcpkern-i386-xl 10 guest-stop fail never pass
test-i386-i386-xl 10 guest-stop fail never pass
test-i386-xcpkern-i386-xl 10 guest-stop fail never pass
This is because "xl shutdown -w" is not implemented.
test-i386-xcpkern-i386-pair 12 guest-migrate/src_host/dst_host fail like 1993
test-i386-i386-pair 12 guest-migrate/src_host/dst_host fail never pass
I haven''t looked into these.
test-amd64-amd64-pair <none executed> fail
This is a malfunction in the test infrastructure.
Ian.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Ian Campbell
2010-Aug-26 07:06 UTC
Re: [Xen-devel] [xen-4.0-testing test] 2045: regressions - FAIL
On Wed, 2010-08-25 at 19:25 +0100, Ian Jackson wrote:> > > test-amd64-i386-pair 12 guest-migrate/src_host/dst_host fail REGR. > vs. 1993 > > It seems that PV migration of a domain from one host to another > doesn''t work properly. Looking at the logs everything is absolutely > fine except that the domU isn''t responding to arps and pings for at > least the 20s that the test system waits 20s after migration.This is with a PVops domU kernel? If so then I''m surprised this is a regression rather than a never passed. Until very recently a pvops kernel would not even attempt to send a gratuitous ARP after a migration, which could lead to 20-30s timeouts like this. (I suppose it might spuriously pass if a test run got very lucky?) This was fixed in v2.6.36-rc1 and was backported to stable kernel v2.6.32.19 and various v2.6.(>32).y as well. Even with the fix in place the gratuitous ARP behaviour is disabled by default so you need to enable the net.ipv4.conf.<dev>.arp_notify sysctl for any device you want to send the notifications. When I was testing I did this by adding net.ipv4.conf.default.arp_notify = 1 to /etc/sysctl.conf and that seemed to do the trick. (remember that we are testing 2.6.32.18 at the moment until 2.6.32.21 is out) Ian. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ian Jackson
2010-Aug-31 14:35 UTC
Re: [Xen-devel] [xen-4.0-testing test] 2045: regressions - FAIL
Ian Campbell writes ("Re: [Xen-devel] [xen-4.0-testing test] 2045:
regressions - FAIL"):> This is with a PVops domU kernel? If so then I''m surprised this is
a
> regression rather than a never passed. Until very recently a pvops
> kernel would not even attempt to send a gratuitous ARP after a
> migration, which could lead to 20-30s timeouts like this. (I suppose it
> might spuriously pass if a test run got very lucky?)
That''s quite likely. arp timeouts are typically a few minutes so you
have maybe a 10% chance of getting lucky. One pass in the past and
the tests will consider it a pass.
> Even with the fix in place the gratuitous ARP behaviour is disabled by
> default so you need to enable the net.ipv4.conf.<dev>.arp_notify
sysctl
> for any device you want to send the notifications. When I was testing I
> did this by adding
> net.ipv4.conf.default.arp_notify = 1
> to /etc/sysctl.conf and that seemed to do the trick.
I think this is a bug. I think the default should be for something to
send this gratuitous arp and the most logical answer in the PV or
PV-on-HVM case is the domU.
However: all of this doesn''t apply to the test network used for these
tests. There is a daemon on that network which sends broadcast ARP
who-has packets for every IP address at very short intervals (5s or
so) and the reply will cause the switches to update their ideas of
MAC<->port mapping.
Ian.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Ian Campbell
2010-Aug-31 15:05 UTC
Re: [Xen-devel] [xen-4.0-testing test] 2045: regressions - FAIL
On Tue, 2010-08-31 at 15:35 +0100, Ian Jackson wrote:> > > Even with the fix in place the gratuitous ARP behaviour is disabled > by > > default so you need to enable the net.ipv4.conf.<dev>.arp_notify > sysctl > > for any device you want to send the notifications. When I was > testing I > > did this by adding > > net.ipv4.conf.default.arp_notify = 1 > > to /etc/sysctl.conf and that seemed to do the trick. > > I think this is a bug. I think the default should be for something to > send this gratuitous arp and the most logical answer in the PV or > PV-on-HVM case is the domU.This is the upstream default, not something we control directly from netfront etc.> However: all of this doesn''t apply to the test network used for these > tests. There is a daemon on that network which sends broadcast ARP > who-has packets for every IP address at very short intervals (5s or > so) and the reply will cause the switches to update their ideas of > MAC<->port mapping.Is this daemon specifically working around this issue or is there some other purpose which I can''t quite think of? Ian. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ian Jackson
2010-Aug-31 15:10 UTC
Re: [Xen-devel] [xen-4.0-testing test] 2045: regressions - FAIL
Ian Campbell writes ("Re: [Xen-devel] [xen-4.0-testing test] 2045:
regressions - FAIL"):> Is this daemon specifically working around this issue or is there some
> other purpose which I can''t quite think of?
It''s there for a related test system on the same network to identify
which machines are up and down and what IP addresses they are
currently trying to use. That turns out to be quite useful.
Ian.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Ian Jackson
2010-Aug-31 18:11 UTC
Re: [Xen-devel] [xen-4.0-testing test] 2045: regressions - FAIL
Ian Campbell writes ("Re: [Xen-devel] [xen-4.0-testing test] 2045:
regressions - FAIL"):> On Tue, 2010-08-31 at 15:35 +0100, Ian Jackson wrote:
> > [Ian C:]
> > > did this by adding
> > > net.ipv4.conf.default.arp_notify = 1
> > > to /etc/sysctl.conf and that seemed to do the trick.
> >
> > I think this is a bug. I think the default should be for something to
> > send this gratuitous arp and the most logical answer in the PV or
> > PV-on-HVM case is the domU.
>
> This is the upstream default, not something we control directly from
> netfront etc.
Um, I''m not sure I follow. The situation when we need to do
gratuitous arp is after save/restore or migration. This is somewhat
analogous to hibernation/suspend, except that hibernation/suspend
pretty much assumes that the system was previously not running, not
previously on a different switch port etc.
Perhaps netfront needs to do something special.
Ian.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Ian Campbell
2010-Aug-31 18:49 UTC
Re: [Xen-devel] [xen-4.0-testing test] 2045: regressions - FAIL
On Tue, 2010-08-31 at 19:11 +0100, Ian Jackson wrote:> Ian Campbell writes ("Re: [Xen-devel] [xen-4.0-testing test] 2045: regressions - FAIL"): > > On Tue, 2010-08-31 at 15:35 +0100, Ian Jackson wrote: > > > [Ian C:] > > > > did this by adding > > > > net.ipv4.conf.default.arp_notify = 1 > > > > to /etc/sysctl.conf and that seemed to do the trick. > > > > > > I think this is a bug. I think the default should be for something to > > > send this gratuitous arp and the most logical answer in the PV or > > > PV-on-HVM case is the domU. > > > > This is the upstream default, not something we control directly from > > netfront etc. > > Um, I''m not sure I follow. The situation when we need to do > gratuitous arp is after save/restore or migration. This is somewhat > analogous to hibernation/suspend, except that hibernation/suspend > pretty much assumes that the system was previously not running, not > previously on a different switch port etc. > > Perhaps netfront needs to do something special.When netfront was upstreamed we were asked to remove the explicit gratuitous ARP sending code from the driver and to instead use the network stack provided functionality -- this is the arp_notify hooks which we already call as appropriate from netfront but as I said it is disabled upstream by default. It''s possible that someone should have a go at upstreaming a patch to enable arp_notify by default or to make notifications triggered by netif_notify_peers() not gate on the arp_notify sysctl or something. like that. I don''t think just enabling arp_notify by default will fly -- there are other triggers for that path such as interfaces coming up. Ian. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jan Beulich
2010-Sep-17 14:09 UTC
Re: [Xen-devel] [xen-4.0-testing test] 2045: regressions - FAIL
>>> On 31.08.10 at 17:05, Ian Campbell <Ian.Campbell@eu.citrix.com> wrote: > On Tue, 2010-08-31 at 15:35 +0100, Ian Jackson wrote: >> >> > Even with the fix in place the gratuitous ARP behaviour is disabled >> by >> > default so you need to enable the net.ipv4.conf.<dev>.arp_notify >> sysctl >> > for any device you want to send the notifications. When I was >> testing I >> > did this by adding >> > net.ipv4.conf.default.arp_notify = 1 >> > to /etc/sysctl.conf and that seemed to do the trick. >> >> I think this is a bug. I think the default should be for something to >> send this gratuitous arp and the most logical answer in the PV or >> PV-on-HVM case is the domU. > > This is the upstream default, not something we control directly from > netfront etc.Wouldn''t it seem possible/reasonable to force this on from netfront (via IN_DEV_CONF_SET()) at least until upstream possibly decides to not let the ARP_NOTIFY sysctl control the sending of an ARP explicitly requested through netif_notify_peers()? The perhaps undesirable effect of this (as well as setting the sysctl through /etc/sysctl.conf) is that it doesn''t only control the ARP we want sent, but also one when bringing the interface up or changing its address, so I''d still favor moving the NETDEV_NOTIFY_PEERS case past the "if (IN_DEV_ARP_NOTIFY(in_dev))" in inetdev_event(). Have you got any feedback from Linux folks on such a potential change? Jan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel