flight 2045 xen-4.0-testing real http://www.chiark.greenend.org.uk/~xensrcts/logs/2045/ Regressions :-( tests which did not succeed: test-amd64-amd64-pair <none executed> fail test-amd64-amd64-xl 10 guest-stop fail never pass test-amd64-i386-pair 12 guest-migrate/src_host/dst_host fail REGR. vs. 1993 test-amd64-i386-win 5 windows-install fail REGR. vs. 1993 test-amd64-i386-xl 10 guest-stop fail never pass test-amd64-xcpkern-i386-xl 10 guest-stop fail never pass test-i386-i386-pair 12 guest-migrate/src_host/dst_host fail never pass test-i386-i386-win 5 windows-install fail REGR. vs. 1993 test-i386-i386-xl 10 guest-stop fail never pass test-i386-xcpkern-i386-pair 12 guest-migrate/src_host/dst_host fail like 1993 test-i386-xcpkern-i386-xl 10 guest-stop fail never pass version targeted for testing: xen 7b8b976f534e baseline version: xen 8e8dd38374e9 jobs: build-amd64 pass build-amd64-oldkern pass build-i386 pass build-i386-oldkern pass build-i386-xcpkern pass test-amd64-amd64-pair fail test-amd64-amd64-pv pass test-amd64-amd64-win pass test-amd64-amd64-xl fail test-amd64-i386-pair fail test-amd64-i386-pv pass test-amd64-i386-win fail test-amd64-i386-xl fail test-amd64-xcpkern-i386-pair pass test-amd64-xcpkern-i386-pv pass test-amd64-xcpkern-i386-win pass test-amd64-xcpkern-i386-xl fail test-i386-i386-pair fail test-i386-i386-pv pass test-i386-i386-win fail test-i386-i386-xl fail test-i386-xcpkern-i386-pair fail test-i386-xcpkern-i386-pv pass test-i386-xcpkern-i386-win pass test-i386-xcpkern-i386-xl fail ------------------------------------------------------------------------------- build-amd64: 1 host-install(1) pass 2 host-build-prep pass 3 xen-build pass linux e6b9b2cbca5093e8e38d qemu 0a940d892e90c820567c xen 21323:7b8b976f534e ------------------------------------------------------------------------------- build-amd64-oldkern: 1 xen-build pass linux 1025:042c8cd71081 qemu 0a940d892e90c820567c xen 21323:7b8b976f534e ------------------------------------------------------------------------------- build-i386: 1 host-install(1) pass 2 host-build-prep pass 3 xen-build pass linux e6b9b2cbca5093e8e38d qemu 0a940d892e90c820567c xen 21323:7b8b976f534e ------------------------------------------------------------------------------- build-i386-oldkern: 1 xen-build pass linux 1025:042c8cd71081 qemu 0a940d892e90c820567c xen 21323:7b8b976f534e ------------------------------------------------------------------------------- build-i386-xcpkern: 1 kernel-build pass linux 110879:32fc6955a6a5 pq_linux 154:61499f9e9a87 ------------------------------------------------------------------------------- test-amd64-amd64-pair: 1 xen-build-check(1) pass 2 host-install/src_host(2) pass 3 host-install/dst_host(3) pass 4 xen-install/src_host pass 5 xen-install/dst_host pass 6 capture-logs/src_host(6) pass 7 capture-logs/dst_host(7) pass ------------------------------------------------------------------------------- test-amd64-amd64-pv: 1 xen-build-check(1) pass 2 host-install(2) pass 3 xen-install pass 4 xen-boot pass 5 debian-install pass 6 debian-fixup pass 7 guest-start pass 8 guest-saverestore pass 9 guest-localmigrate pass 10 guest-stop pass 11 capture-logs(11) pass ------------------------------------------------------------------------------- test-amd64-amd64-win: 1 xen-build-check(1) pass 2 host-install(2) pass 3 xen-install pass 4 xen-boot pass 5 windows-install pass 6 guest-saverestore pass 7 guest-localmigrate pass 8 guest-stop pass 9 capture-logs(9) pass ------------------------------------------------------------------------------- test-amd64-amd64-xl: 1 xen-build-check(1) pass 2 host-install(2) pass 3 xen-install pass 4 xen-boot pass 5 debian-install pass 6 debian-fixup pass 7 guest-start pass 8 guest-saverestore pass 9 guest-localmigrate pass 10 guest-stop fail 11 capture-logs(11) pass ------------------------------------------------------------------------------- test-amd64-i386-pair: 1 xen-build-check(1) pass 2 host-install/src_host(2) pass 3 host-install/dst_host(3) pass 4 xen-install/src_host pass 5 xen-install/dst_host pass 6 xen-boot/src_host pass 7 xen-boot/dst_host pass 8 debian-install/dst_host pass 9 debian-fixup/dst_host pass 10 guests-nbd-mirror pass 11 guest-start pass 12 guest-migrate/src_host/dst_host fail 13 capture-logs/src_host(13) pass 14 capture-logs/dst_host(14) pass ------------------------------------------------------------------------------- test-amd64-i386-pv: 1 xen-build-check(1) pass 2 host-install(2) pass 3 xen-install pass 4 xen-boot pass 5 debian-install pass 6 debian-fixup pass 7 guest-start pass 8 guest-saverestore pass 9 guest-localmigrate pass 10 guest-stop pass 11 capture-logs(11) pass ------------------------------------------------------------------------------- test-amd64-i386-win: 1 xen-build-check(1) pass 2 host-install(2) pass 3 xen-install pass 4 xen-boot pass 5 windows-install fail 6 capture-logs(6) pass ------------------------------------------------------------------------------- test-amd64-i386-xl: 1 xen-build-check(1) pass 2 host-install(2) pass 3 xen-install pass 4 xen-boot pass 5 debian-install pass 6 debian-fixup pass 7 guest-start pass 8 guest-saverestore pass 9 guest-localmigrate pass 10 guest-stop fail 11 capture-logs(11) pass ------------------------------------------------------------------------------- test-amd64-xcpkern-i386-pair: 1 xen-build-check(1) pass 2 host-install/src_host(2) pass 3 host-install/dst_host(3) pass 4 xen-install/src_host pass 5 xen-install/dst_host pass 6 xen-boot/src_host pass 7 xen-boot/dst_host pass 8 debian-install/dst_host pass 9 debian-fixup/dst_host pass 10 guests-nbd-mirror pass 11 guest-start pass 12 guest-migrate/src_host/dst_host pass 13 guest-migrate/dst_host/src_host pass 14 capture-logs/src_host(14) pass 15 capture-logs/dst_host(15) pass ------------------------------------------------------------------------------- test-amd64-xcpkern-i386-pv: 1 xen-build-check(1) pass 2 host-install(2) pass 3 xen-install pass 4 xen-boot pass 5 debian-install pass 6 debian-fixup pass 7 guest-start pass 8 guest-saverestore pass 9 guest-localmigrate pass 10 guest-stop pass 11 capture-logs(11) pass ------------------------------------------------------------------------------- test-amd64-xcpkern-i386-win: 1 xen-build-check(1) pass 2 host-install(2) pass 3 xen-install pass 4 xen-boot pass 5 windows-install pass 6 guest-saverestore pass 7 guest-localmigrate pass 8 guest-stop pass 9 capture-logs(9) pass ------------------------------------------------------------------------------- test-amd64-xcpkern-i386-xl: 1 xen-build-check(1) pass 2 host-install(2) pass 3 xen-install pass 4 xen-boot pass 5 debian-install pass 6 debian-fixup pass 7 guest-start pass 8 guest-saverestore pass 9 guest-localmigrate pass 10 guest-stop fail 11 capture-logs(11) pass ------------------------------------------------------------------------------- test-i386-i386-pair: 1 xen-build-check(1) pass 2 host-install/src_host(2) pass 3 host-install/dst_host(3) pass 4 xen-install/src_host pass 5 xen-install/dst_host pass 6 xen-boot/src_host pass 7 xen-boot/dst_host pass 8 debian-install/dst_host pass 9 debian-fixup/dst_host pass 10 guests-nbd-mirror pass 11 guest-start pass 12 guest-migrate/src_host/dst_host fail 13 capture-logs/src_host(13) pass 14 capture-logs/dst_host(14) pass ------------------------------------------------------------------------------- test-i386-i386-pv: 1 xen-build-check(1) pass 2 host-install(2) pass 3 xen-install pass 4 xen-boot pass 5 debian-install pass 6 debian-fixup pass 7 guest-start pass 8 guest-saverestore pass 9 guest-localmigrate pass 10 guest-stop pass 11 capture-logs(11) pass ------------------------------------------------------------------------------- test-i386-i386-win: 1 xen-build-check(1) pass 2 host-install(2) pass 3 xen-install pass 4 xen-boot pass 5 windows-install fail 6 capture-logs(6) pass ------------------------------------------------------------------------------- test-i386-i386-xl: 1 xen-build-check(1) pass 2 host-install(2) pass 3 xen-install pass 4 xen-boot pass 5 debian-install pass 6 debian-fixup pass 7 guest-start pass 8 guest-saverestore pass 9 guest-localmigrate pass 10 guest-stop fail 11 capture-logs(11) pass ------------------------------------------------------------------------------- test-i386-xcpkern-i386-pair: 1 xen-build-check(1) pass 2 host-install/src_host(2) pass 3 host-install/dst_host(3) pass 4 xen-install/src_host pass 5 xen-install/dst_host pass 6 xen-boot/src_host pass 7 xen-boot/dst_host pass 8 debian-install/dst_host pass 9 debian-fixup/dst_host pass 10 guests-nbd-mirror pass 11 guest-start pass 12 guest-migrate/src_host/dst_host fail 13 capture-logs/src_host(13) pass 14 capture-logs/dst_host(14) pass ------------------------------------------------------------------------------- test-i386-xcpkern-i386-pv: 1 xen-build-check(1) pass 2 host-install(2) pass 3 xen-install pass 4 xen-boot pass 5 debian-install pass 6 debian-fixup pass 7 guest-start pass 8 guest-saverestore pass 9 guest-localmigrate pass 10 guest-stop pass 11 capture-logs(11) pass ------------------------------------------------------------------------------- test-i386-xcpkern-i386-win: 1 xen-build-check(1) pass 2 host-install(2) pass 3 xen-install pass 4 xen-boot pass 5 windows-install pass 6 guest-saverestore pass 7 guest-localmigrate pass 8 guest-stop pass 9 capture-logs(9) pass ------------------------------------------------------------------------------- test-i386-xcpkern-i386-xl: 1 xen-build-check(1) pass 2 host-install(2) pass 3 xen-install pass 4 xen-boot pass 5 debian-install pass 6 debian-fixup pass 7 guest-start pass 8 guest-saverestore pass 9 guest-localmigrate pass 10 guest-stop fail 11 capture-logs(11) pass ------------------------------------------------------------ sg-report-flight on woking.cam.xci-test.com logs: /home/xc_osstest/logs images: /home/xc_osstest/images Logs, config files, etc. are available at http://www.chiark.greenend.org.uk/~xensrcts/logs Test harness code can be found at http://xenbits.xensource.com/gitweb?p=osstest.git;a=summary Not pushing. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Pasi Kärkkäinen
2010-Aug-25 15:40 UTC
Re: [Xen-devel] [xen-4.0-testing test] 2045: regressions - FAIL
On Wed, Aug 25, 2010 at 03:21:50PM +0100, xen.org wrote:> flight 2045 xen-4.0-testing real > http://www.chiark.greenend.org.uk/~xensrcts/logs/2045/ > > Regressions :-( > > tests which did not succeed: > test-amd64-amd64-pair <none executed> fail > test-amd64-amd64-xl 10 guest-stop fail never pass > test-amd64-i386-pair 12 guest-migrate/src_host/dst_host fail REGR. vs. 1993 > test-amd64-i386-win 5 windows-install fail REGR. vs. 1993 > test-amd64-i386-xl 10 guest-stop fail never pass > test-amd64-xcpkern-i386-xl 10 guest-stop fail never pass > test-i386-i386-pair 12 guest-migrate/src_host/dst_host fail never pass > test-i386-i386-win 5 windows-install fail REGR. vs. 1993 > test-i386-i386-xl 10 guest-stop fail never pass > test-i386-xcpkern-i386-pair 12 guest-migrate/src_host/dst_host fail like 1993 > test-i386-xcpkern-i386-xl 10 guest-stop fail never pass >So ok, half of these failures are xl related.. How about the other failures? -- Pasi _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ian Jackson
2010-Aug-25 18:25 UTC
Re: [Xen-devel] [xen-4.0-testing test] 2045: regressions - FAIL
Pasi Kärkkäinen writes ("Re: [Xen-devel] [xen-4.0-testing test] 2045: regressions - FAIL"):> So ok, half of these failures are xl related.. > How about the other failures?These failures are as follows: test-amd64-i386-pair 12 guest-migrate/src_host/dst_host fail REGR. vs. 1993 It seems that PV migration of a domain from one host to another doesn''t work properly. Looking at the logs everything is absolutely fine except that the domU isn''t responding to arps and pings for at least the 20s that the test system waits 20s after migration. test-amd64-i386-win 5 windows-install fail REGR. vs. 1993 test-i386-i386-win 5 windows-install fail REGR. vs. 1993 These were HVM installs of Windows. The timed out after 7000s. Looking at the logs, the test harness tried to collect various information including screenshots, but this generally failed because the dom0 was running very very slowly. For example, ssh root@<dom0> brctl show took 7 secons to run; and ssh root@<dom0> xenstore-read <some thing to find VNC port> took more than 10 seconds which made the test harness time it out. I have seen these symptoms myself: dom0 can become very unresponsive indeed, sometimes locking up (or at least, not listening to the network) for tens of seconds at a time. test-amd64-amd64-xl 10 guest-stop fail never pass test-amd64-i386-xl 10 guest-stop fail never pass test-amd64-xcpkern-i386-xl 10 guest-stop fail never pass test-i386-i386-xl 10 guest-stop fail never pass test-i386-xcpkern-i386-xl 10 guest-stop fail never pass This is because "xl shutdown -w" is not implemented. test-i386-xcpkern-i386-pair 12 guest-migrate/src_host/dst_host fail like 1993 test-i386-i386-pair 12 guest-migrate/src_host/dst_host fail never pass I haven''t looked into these. test-amd64-amd64-pair <none executed> fail This is a malfunction in the test infrastructure. Ian. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ian Campbell
2010-Aug-26 07:06 UTC
Re: [Xen-devel] [xen-4.0-testing test] 2045: regressions - FAIL
On Wed, 2010-08-25 at 19:25 +0100, Ian Jackson wrote:> > > test-amd64-i386-pair 12 guest-migrate/src_host/dst_host fail REGR. > vs. 1993 > > It seems that PV migration of a domain from one host to another > doesn''t work properly. Looking at the logs everything is absolutely > fine except that the domU isn''t responding to arps and pings for at > least the 20s that the test system waits 20s after migration.This is with a PVops domU kernel? If so then I''m surprised this is a regression rather than a never passed. Until very recently a pvops kernel would not even attempt to send a gratuitous ARP after a migration, which could lead to 20-30s timeouts like this. (I suppose it might spuriously pass if a test run got very lucky?) This was fixed in v2.6.36-rc1 and was backported to stable kernel v2.6.32.19 and various v2.6.(>32).y as well. Even with the fix in place the gratuitous ARP behaviour is disabled by default so you need to enable the net.ipv4.conf.<dev>.arp_notify sysctl for any device you want to send the notifications. When I was testing I did this by adding net.ipv4.conf.default.arp_notify = 1 to /etc/sysctl.conf and that seemed to do the trick. (remember that we are testing 2.6.32.18 at the moment until 2.6.32.21 is out) Ian. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ian Jackson
2010-Aug-31 14:35 UTC
Re: [Xen-devel] [xen-4.0-testing test] 2045: regressions - FAIL
Ian Campbell writes ("Re: [Xen-devel] [xen-4.0-testing test] 2045: regressions - FAIL"):> This is with a PVops domU kernel? If so then I''m surprised this is a > regression rather than a never passed. Until very recently a pvops > kernel would not even attempt to send a gratuitous ARP after a > migration, which could lead to 20-30s timeouts like this. (I suppose it > might spuriously pass if a test run got very lucky?)That''s quite likely. arp timeouts are typically a few minutes so you have maybe a 10% chance of getting lucky. One pass in the past and the tests will consider it a pass.> Even with the fix in place the gratuitous ARP behaviour is disabled by > default so you need to enable the net.ipv4.conf.<dev>.arp_notify sysctl > for any device you want to send the notifications. When I was testing I > did this by adding > net.ipv4.conf.default.arp_notify = 1 > to /etc/sysctl.conf and that seemed to do the trick.I think this is a bug. I think the default should be for something to send this gratuitous arp and the most logical answer in the PV or PV-on-HVM case is the domU. However: all of this doesn''t apply to the test network used for these tests. There is a daemon on that network which sends broadcast ARP who-has packets for every IP address at very short intervals (5s or so) and the reply will cause the switches to update their ideas of MAC<->port mapping. Ian. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ian Campbell
2010-Aug-31 15:05 UTC
Re: [Xen-devel] [xen-4.0-testing test] 2045: regressions - FAIL
On Tue, 2010-08-31 at 15:35 +0100, Ian Jackson wrote:> > > Even with the fix in place the gratuitous ARP behaviour is disabled > by > > default so you need to enable the net.ipv4.conf.<dev>.arp_notify > sysctl > > for any device you want to send the notifications. When I was > testing I > > did this by adding > > net.ipv4.conf.default.arp_notify = 1 > > to /etc/sysctl.conf and that seemed to do the trick. > > I think this is a bug. I think the default should be for something to > send this gratuitous arp and the most logical answer in the PV or > PV-on-HVM case is the domU.This is the upstream default, not something we control directly from netfront etc.> However: all of this doesn''t apply to the test network used for these > tests. There is a daemon on that network which sends broadcast ARP > who-has packets for every IP address at very short intervals (5s or > so) and the reply will cause the switches to update their ideas of > MAC<->port mapping.Is this daemon specifically working around this issue or is there some other purpose which I can''t quite think of? Ian. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ian Jackson
2010-Aug-31 15:10 UTC
Re: [Xen-devel] [xen-4.0-testing test] 2045: regressions - FAIL
Ian Campbell writes ("Re: [Xen-devel] [xen-4.0-testing test] 2045: regressions - FAIL"):> Is this daemon specifically working around this issue or is there some > other purpose which I can''t quite think of?It''s there for a related test system on the same network to identify which machines are up and down and what IP addresses they are currently trying to use. That turns out to be quite useful. Ian. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ian Jackson
2010-Aug-31 18:11 UTC
Re: [Xen-devel] [xen-4.0-testing test] 2045: regressions - FAIL
Ian Campbell writes ("Re: [Xen-devel] [xen-4.0-testing test] 2045: regressions - FAIL"):> On Tue, 2010-08-31 at 15:35 +0100, Ian Jackson wrote: > > [Ian C:] > > > did this by adding > > > net.ipv4.conf.default.arp_notify = 1 > > > to /etc/sysctl.conf and that seemed to do the trick. > > > > I think this is a bug. I think the default should be for something to > > send this gratuitous arp and the most logical answer in the PV or > > PV-on-HVM case is the domU. > > This is the upstream default, not something we control directly from > netfront etc.Um, I''m not sure I follow. The situation when we need to do gratuitous arp is after save/restore or migration. This is somewhat analogous to hibernation/suspend, except that hibernation/suspend pretty much assumes that the system was previously not running, not previously on a different switch port etc. Perhaps netfront needs to do something special. Ian. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ian Campbell
2010-Aug-31 18:49 UTC
Re: [Xen-devel] [xen-4.0-testing test] 2045: regressions - FAIL
On Tue, 2010-08-31 at 19:11 +0100, Ian Jackson wrote:> Ian Campbell writes ("Re: [Xen-devel] [xen-4.0-testing test] 2045: regressions - FAIL"): > > On Tue, 2010-08-31 at 15:35 +0100, Ian Jackson wrote: > > > [Ian C:] > > > > did this by adding > > > > net.ipv4.conf.default.arp_notify = 1 > > > > to /etc/sysctl.conf and that seemed to do the trick. > > > > > > I think this is a bug. I think the default should be for something to > > > send this gratuitous arp and the most logical answer in the PV or > > > PV-on-HVM case is the domU. > > > > This is the upstream default, not something we control directly from > > netfront etc. > > Um, I''m not sure I follow. The situation when we need to do > gratuitous arp is after save/restore or migration. This is somewhat > analogous to hibernation/suspend, except that hibernation/suspend > pretty much assumes that the system was previously not running, not > previously on a different switch port etc. > > Perhaps netfront needs to do something special.When netfront was upstreamed we were asked to remove the explicit gratuitous ARP sending code from the driver and to instead use the network stack provided functionality -- this is the arp_notify hooks which we already call as appropriate from netfront but as I said it is disabled upstream by default. It''s possible that someone should have a go at upstreaming a patch to enable arp_notify by default or to make notifications triggered by netif_notify_peers() not gate on the arp_notify sysctl or something. like that. I don''t think just enabling arp_notify by default will fly -- there are other triggers for that path such as interfaces coming up. Ian. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jan Beulich
2010-Sep-17 14:09 UTC
Re: [Xen-devel] [xen-4.0-testing test] 2045: regressions - FAIL
>>> On 31.08.10 at 17:05, Ian Campbell <Ian.Campbell@eu.citrix.com> wrote: > On Tue, 2010-08-31 at 15:35 +0100, Ian Jackson wrote: >> >> > Even with the fix in place the gratuitous ARP behaviour is disabled >> by >> > default so you need to enable the net.ipv4.conf.<dev>.arp_notify >> sysctl >> > for any device you want to send the notifications. When I was >> testing I >> > did this by adding >> > net.ipv4.conf.default.arp_notify = 1 >> > to /etc/sysctl.conf and that seemed to do the trick. >> >> I think this is a bug. I think the default should be for something to >> send this gratuitous arp and the most logical answer in the PV or >> PV-on-HVM case is the domU. > > This is the upstream default, not something we control directly from > netfront etc.Wouldn''t it seem possible/reasonable to force this on from netfront (via IN_DEV_CONF_SET()) at least until upstream possibly decides to not let the ARP_NOTIFY sysctl control the sending of an ARP explicitly requested through netif_notify_peers()? The perhaps undesirable effect of this (as well as setting the sysctl through /etc/sysctl.conf) is that it doesn''t only control the ARP we want sent, but also one when bringing the interface up or changing its address, so I''d still favor moving the NETDEV_NOTIFY_PEERS case past the "if (IN_DEV_ARP_NOTIFY(in_dev))" in inetdev_event(). Have you got any feedback from Linux folks on such a potential change? Jan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel