Xen devel - Jan 2011 - xen 4.1 rc2 test report ( 5new issues found)

Hi, all
	According to the xen 4.1 rc1 test result, we did a round bug verification
against rc2. 7 issues get fixed on rc2, 5 new issues were reported. We still
have 22 open issues.

Version info
Changeset: 22812:73b3debb90cf
Pvops dom0: 75cc13f5aa29b4f3227d269ca165dfa8937c94fe

New issues(5)
1. [vt-d] Xen panic on PAE when enable iommu
http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1728
2. Xen complains msi error when startup
http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1782
3. xl does not check the memory size of guest
http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1729
4. "xl vcpu-set" causes dom0 crash or panic  
http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1730
5. "xl vcpu-list" does not response after run vcpu-pin
http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1731

Fixed issues(7):
1. windows up guest boot fail
http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1704
2. [vt-d] Can not detach the device which was assigned statically
http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1717 
3. guest shows white screen when boot guest with NIC assigned
http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1712 
4. memory corruption was reported by "xl" with device pass-throu
http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1713 
5. [vt-d]Guest only reveal one device if static assign two or more devices to
guest
http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1710
6. Guest does not disappear after poweroff it
http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1720 
7. Too many error information showed when destory an inexistent guest
http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1714

Bug list:
Vt-d ( 8 bugs)
1. ubuntu PAE SMP guest has network problem with NIC assigned (Community)
http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1709
2. [VT-d] xen panic on function do_IRQ after many times NIC pass-throu (Intel)
http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1706
3. [VT-D]run guest with NIC assigned will cause system hang sometimes under PAE
on Sandy bridge (Intel)
http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1725
4.[vt-d] dom0 igb driver is too old to support 4-port	(Intel)
http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1708
5.[VT-d] xen panic when run guest with NIC assigned sometimes (Intel)
http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=24
6.[vt-d] xl command does not response after passthrou IGD card (Intel)
http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1723
7.[vt-d] fail to get IP address after hotplug VF for 300 times (Intel)
http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1722
8.[vt-d] Xen panic on PAE when enable iommu
http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1728

RAS (1 bug)
1. System hang when running cpu offline (Intel)
http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1654

ACPI (1 bug)
1. System cann''t resume after do suspend (Intel)
http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1707

Save/Restore(1 bug)
1. RHEL6 guest fail to do save/restore (Community)
http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1716

xl command(5 bugs)
1. xl does not check the duplicated configure file and image file (Community)
http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1711
2. Guest network broken after do SAVE/RESTOR with xl (Community)
http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1703
3. xl does not check the memory size of guest(Community)
http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1729 
4. "xl vcpu-set" causes dom0 crash or panic (Community)
http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1730 
5. "xl vcpu-list" does not response after run vcpu-pin(Community)
http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1731 

Hypervisor(3 bugs)
1. Only two 1GB-pages be allocated to a 10GBs memory guest (Intel)
http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1721
3. Dom0 crashes on Core2 when dom0_mem is no more than 1972MB (Community)
http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1726
3. Xen complains msi error when startup
http://otc-qa.sh.intel.com/xen/bugzilla/show_bug.cgi?id=1829

Performance(1 bug)
1. guest boot very slowly without limit dom0 cpu number on EX (Intel)
http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1719

X2APIC (1 bug)
1. Fail to bootup sandy bridge under PAE with x2apic enabled (Intel)
http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1718

Guest OS (1 bug)
1. guest with vnif assigned fail to bootup when disable apic (Community)
http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1692

Thanks & Regards,
Shaohui




_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Fri, 28 Jan 2011, Zheng, Shaohui wrote:
> Save/Restore(1 bug)
> 1. RHEL6 guest fail to do save/restore (Community)
> http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1716
the bug has been filed on red hat''s bugzilla
> 
> xl command(5 bugs)
> 1. xl does not check the duplicated configure file and image file
(Community)
> http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1711
patches are coming to fix this

> 2. Guest network broken after do SAVE/RESTOR with xl (Community)
> http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1703
cannot reproduce this bug

> 3. xl does not check the memory size of guest(Community)
> http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1729 
I am not entirely sure we should try to solve this bug, after all we
should always try to do what the user asks us to do, even if it doesn''t
make sense :)

> 4. "xl vcpu-set" causes dom0 crash or panic (Community)
> http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1730 
The repro steps say: "xl vcpu-set 0 16" but the vcpu-set commands
actually takes 3 arguments:

Usage: xl [-v] vcpu-pin <Domain> <VCPU|all> <CPUs|all>

In any case the bug looks like a dom0 kernel bug more than anything
else...

> 5. "xl vcpu-list" does not response after run vcpu-pin(Community)
> http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1731 
 
Are use sure this is not just because the system has become very very
slow because 5 vcpus are running in a single pcpu?
Because I tried the very same steps reported above and it seems to work
for me.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Thanks & Regards,
Shaohui

> -----Original Message-----
> From: Stefano Stabellini [mailto:stefano.stabellini@eu.citrix.com]
> Sent: Saturday, January 29, 2011 12:55 AM
> To: Zheng, Shaohui
> Cc: xen-devel@lists.xensource.com
> Subject: Re: [Xen-devel] xen 4.1 rc2 test report ( 5new issues found)
> 
> On Fri, 28 Jan 2011, Zheng, Shaohui wrote:
> > Save/Restore(1 bug)
> > 1. RHEL6 guest fail to do save/restore (Community)
> > http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1716
> 
> the bug has been filed on red hat''s bugzilla
> 
Let ''s track it in xen''s bugzilla, too.
> >
> > xl command(5 bugs)
> > 1. xl does not check the duplicated configure file and image file
(Community)
> > http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1711
> 
> patches are coming to fix this
> 
> 
> > 2. Guest network broken after do SAVE/RESTOR with xl (Community)
> > http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1703
> 
> cannot reproduce this bug
> 
After save/restore guest OS, We can not ping any address, we need to run
dhclient to re-fetch an IP address, and the network works again.
The expected result is that guest network still working after restore it.
> 
> > 3. xl does not check the memory size of guest(Community)
> > http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1729
> 
> I am not entirely sure we should try to solve this bug, after all we
> should always try to do what the user asks us to do, even if it
doesn''t
> make sense :)
> 
It is a critical issue in fact. the request to create a very large guest will
fail obviously, but the system status already becomes abnormal. It prints a lot
error information continuously.

(XEN) memory.c:133:d0 Could not allocate order=0 extent: id=0 memflags=0 (0 of
512)
(XEN) memory.c:133:d0 Could not allocate order=0 extent: id=0 memflags=0 (0 of
512)

And We cannot create any guest any more except reboot the system.
> 
> > 4. "xl vcpu-set" causes dom0 crash or panic (Community)
> > http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1730
> 
> The repro steps say: "xl vcpu-set 0 16" but the vcpu-set commands
> actually takes 3 arguments:
> 
> Usage: xl [-v] vcpu-pin <Domain> <VCPU|all> <CPUs|all>
> 
> In any case the bug looks like a dom0 kernel bug more than anything
> else...
> 
You are showing the Usage of "xl vcpu-pin", not "xl
vcpu-set",  "xl vcpu-set" takes 2 arguments.

[root@vt-nhm7 ~]# xl help vcpu-set
Usage: xl [-v] vcpu-set <Domain> <vCPUs>

Set the number of active VCPUs allowed for the domain.

> > 5. "xl vcpu-list" does not response after run
vcpu-pin(Community)
> > http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1731
> 
> Are use sure this is not just because the system has become very very
> slow because 5 vcpus are running in a single pcpu?
> Because I tried the very same steps reported above and it seems to work
> for me.
It is NOT because we bind too much vcpu to the a single pcpu.  "xm
vcpu-list" can list the vcpus which does not do the binding only, but for
the binded vcpu, even though we bind only one vcpu to a pcpu, "xl
vcpu-list" command does not return, either. We need to type
"CTRL-C" to terminate it.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Sun, 2011-01-30 at 02:27 +0000, Zheng, Shaohui wrote:
> > > 2. Guest network broken after do SAVE/RESTOR with xl (Community)
> > > http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1703
> > 
> > cannot reproduce this bug
> > 
> After save/restore guest OS, We can not ping any address, we need to run
dhclient to re-fetch an IP address, and the network works again.
> The expected result is that guest network still working after restore it.
Sounds like another redhat distro bug. The guest kernel must be
deliberately nuking the IP information on save/restore but then not
doing anything to trigger the network scripts to re-run dhcp. If that is
indeed the case then there''s not much we can do about that on the xen
side.

Gianni


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Mon, 31 Jan 2011, Gianni Tedesco wrote:
> On Sun, 2011-01-30 at 02:27 +0000, Zheng, Shaohui wrote:
> > > > 2. Guest network broken after do SAVE/RESTOR with xl
(Community)
> > > > http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1703
> > > 
> > > cannot reproduce this bug
> > > 
> > After save/restore guest OS, We can not ping any address, we need to
run dhclient to re-fetch an IP address, and the network works again.
> > The expected result is that guest network still working after restore
it.
> 
> Sounds like another redhat distro bug. The guest kernel must be
> deliberately nuking the IP information on save/restore but then not
> doing anything to trigger the network scripts to re-run dhcp. If that is
> indeed the case then there''s not much we can do about that on the
xen
> side.
I cannot repro the issue on debian.
Is anybody else running redhat based guests?

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Sun, 30 Jan 2011, Zheng, Shaohui wrote:
> > -----Original Message-----
> > From: Stefano Stabellini [mailto:stefano.stabellini@eu.citrix.com]
> > Sent: Saturday, January 29, 2011 12:55 AM
> > To: Zheng, Shaohui
> > Cc: xen-devel@lists.xensource.com
> > Subject: Re: [Xen-devel] xen 4.1 rc2 test report ( 5new issues found)
> > 
> > On Fri, 28 Jan 2011, Zheng, Shaohui wrote:
> > > Save/Restore(1 bug)
> > > 1. RHEL6 guest fail to do save/restore (Community)
> > > http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1716
> > 
> > the bug has been filed on red hat''s bugzilla
> > 
> Let ''s track it in xen''s bugzilla, too.
> 
Yep, good idea.

> > 
> > > 3. xl does not check the memory size of guest(Community)
> > > http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1729
> > 
> > I am not entirely sure we should try to solve this bug, after all we
> > should always try to do what the user asks us to do, even if it
doesn''t
> > make sense :)
> > 
> 
> It is a critical issue in fact. the request to create a very large guest
will fail obviously, but the system status already becomes abnormal. It prints a
lot error information continuously.
> 
> (XEN) memory.c:133:d0 Could not allocate order=0 extent: id=0 memflags=0 (0
of 512)
> (XEN) memory.c:133:d0 Could not allocate order=0 extent: id=0 memflags=0 (0
of 512)
> 
> And We cannot create any guest any more except reboot the system.
> 
I see. That is definitely a bug and I can reproduce it.
I don''t think xl is doing anything wrong here, because it is correctly
reporting an error and cleaning up the domain.
The problem is that xen keeps printing

"memory.c:133:d0 Could not allocate order=0 extent: id=0 memflags=0 (0 of
512)"

even after xl returned (!!!).
Could it be an problem caused by the hypercall continuation (CC''ing
Keir and Tim)?

> > 
> > > 4. "xl vcpu-set" causes dom0 crash or panic (Community)
> > > http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1730
> > 
> > The repro steps say: "xl vcpu-set 0 16" but the vcpu-set
commands
> > actually takes 3 arguments:
> > 
> > Usage: xl [-v] vcpu-pin <Domain> <VCPU|all>
<CPUs|all>
> > 
> > In any case the bug looks like a dom0 kernel bug more than anything
> > else...
> > 
> 
> You are showing the Usage of "xl vcpu-pin", not "xl
vcpu-set",  "xl vcpu-set" takes 2 arguments.
> 
> [root@vt-nhm7 ~]# xl help vcpu-set
> Usage: xl [-v] vcpu-set <Domain> <vCPUs>
> 
> Set the number of active VCPUs allowed for the domain.
> 
Ooops, my mistake :)
vcpu-set only writes to xenstore, so this is defenitely a dom0 kernel
bug (CC''ing Jeremy).

> 
> > > 5. "xl vcpu-list" does not response after run
vcpu-pin(Community)
> > > http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1731
> > 
> > Are use sure this is not just because the system has become very very
> > slow because 5 vcpus are running in a single pcpu?
> > Because I tried the very same steps reported above and it seems to
work
> > for me.
> 
> It is NOT because we bind too much vcpu to the a single pcpu.  "xm
vcpu-list" can list the vcpus which does not do the binding only, but for
the binded vcpu, even though we bind only one vcpu to a pcpu, "xl
vcpu-list" command does not return, either. We need to type
"CTRL-C" to terminate it.
> 
Strange. Could you run xl vcpu-list with strace?
Do you have any more logging (xen or dom0 serial)?
I suspect there might be a bug underneath...

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel