Ed Smith
2006-Oct-03 20:05 UTC
[Xen-devel] Testing status of HVM (Intel VT) on 64bit XEN unstable c/s 11686
Summary:
Changeset 11686
- NEW: Domain crash running ltp "mtest01 -p80" on 32bit SMP
HVM guest and then XEN locked up requiring a hard reset:
domain_crash_sync called from private.h:495 (failure.8)
- XEN crash running ltp "mtest01 -p80" on 32bit SMP HVM guest:
BUG at multi.c:2967 from sh_page_fault__shadow_3_guest_3 (failure.3)
Test Configuration:
Dell Precision WorkStation 380, Dual Core, 2GB, 3 SATA (Intel VT)
64bit XEN RELEASE Build Hypervisor on a RHEL4U2 64bit root (/dev/sda)
32bit fully virtualized (HVM) guest RHEL4U2 256MB (/dev/sdb)
pae=1(smp) pae=0(up), acpi=1, apic=1
kernargs noapic
64bit fully virtualized (HVM) guest RHEL4U2 256MB (/dev/sdc)
pae=1, acpi=1, apic=1
kernargs noapic
Boot Tests:
Boot a fully virtualized (HVM) guest to the login prompt
Results are marked Pass|Fail where (n) points to a failure description
Regression Tests:
852 tests (851 ltp tests and one 30 minute user load test)
Tests are marked #Pass/#Fail where (n) points to a failure description
XEN 64bit 2 CPU Hypervisor (booted smp):
----------------------------------------------------------------------
| XEN | Guest Kernel (SMP kernels booted with 2 CPUs) |
| Changeset|-----------------------------------------------------------|
| | 32bit UP | 32bit SMP | 64bit UP | 64bit SMP |
| |--------------|--------------|--------------|--------------|
| | Boot | Test | Boot | Test | Boot | Test | Boot | Test |
|----------|------|-------|------|-------|------|-------|------|-------|
| 11686 | Pass | | Pass | 850/2 | Pass | | Pass | 852/0 |
| | | | |(1,3,8)| | | | |
|----------|------|-------|------|-------|------|-------|------|-------|
| 11655 | Pass | | Pass | 850/2 | Fail | | Fail | |
| | | | | (1,4) | (7) | | (7) | |
|----------|------|-------|------|-------|------|-------|------|-------|
| 11616 | Pass | | Fail | | Fail | | Fail | |
| | | | (6) | | (5) | | (5) | |
|----------|------|-------|------|-------|------|-------|------|-------|
| 11600 | Pass | | Pass | 851/1 | Pass | | Pass | 852/0 |
| | | | |(1,4) | | | | |
|----------|------|-------|------|-------|------|-------|------|-------|
| 11486 | Pass | | Pass | 851/1 | Pass | | Pass | 852/0 |
| | | | |(1,2,3)| | | | |
----------------------------------------------------------------------
Multiple Guest Boot Test
Test is a 30 minute user load on both Guests
XEN 64bit 2 CPU Hypervisor (booted smp):
--------------------------------------------
| XEN | Guest Kernel |
| Changeset|---------------------------------|
| | 32bit 1CPU UP | 32bit 2CPU SMP |
| | 64bit 1CPU UP | 64bit 2CPU SMP |
| |----------------|----------------|
| | Boot | Test | Boot | Test |
|----------|------|---------|------|---------|
| 11686 | Pass | Pass | Pass | Pass |
| | | | | |
|----------|------|---------|------|---------|
| 11655 | Fail | | Fail | |
| | (7) | | (7) | |
|----------|------|---------|------|---------|
| 11616 | Fail | | Fail | |
| | (5) | | (5) | |
|----------|------|---------|------|---------|
| 11600 | Pass | Pass | Pass | Pass |
| | | | | |
|----------|------|---------|------|---------|
| 11486 | Pass | Pass | Pass | Pass |
| | | | | |
--------------------------------------------
Failures:
1. BUG 666: 32bit guests fail ltp gettimeofday02
and nanosleep01/02 with clock problems
2. XEN crash on an xm destroy of a dead guest
32bit SMP HVM guest:
Fatal page fault - put_page_from_l1e+0x85/0x140
3. XEN crash running ltp "mtest01 -p80" on 32bit SMP HVM guest:
BUG at multi.c:2967 from sh_page_fault__shadow_3_guest_3
4. XEN crash running ltp "mtest01 -p80" on 32bit SMP HVM guest:
BUG at multi.c:4082 from sh_clear_shadow_entry__shadow_3_guest_3
5. FIXED: 64bit UP and SMP guests crash domain on boot
domain_crash_sync called from vmx.c:2268
6. FIXED: 32bit SMP HVM Guests hang on boot:
"Uncompressing Linux... OK booting the kernel"
7. FIXED: 64bit UP/SMP HVM Guests hang on boot:
"Decompressing Linux... done booting the kernel"
8. NEW: Domain crash running ltp "mtest01 -p80" on 32bit SMP
HVM guest and then XEN locked up requiring a hard reset:
domain_crash_sync called from private.h:495
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Tim Deegan
2006-Oct-04 13:12 UTC
Re: [Xen-devel] Testing status of HVM (Intel VT) on 64bit XEN unstable c/s 11686
Hi Ed, At 16:05 -0400 on 03 Oct (1159891531), Ed Smith wrote:> - NEW: Domain crash running ltp "mtest01 -p80" on 32bit SMP > HVM guest and then XEN locked up requiring a hard reset: > domain_crash_sync called from private.h:495 (failure.8) > - XEN crash running ltp "mtest01 -p80" on 32bit SMP HVM guest: > BUG at multi.c:2967 from sh_page_fault__shadow_3_guest_3 (failure.3)We''re having some difficulty reproducing these failures. Are you running full ltp tests or just the mtest01 binary? Are you running anything else in the VM at the time? How often do the failures occur? Was there a difference between the tests which caused the two failures or did they happen on different runs of the same test? Cheers, Tim. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ed Smith
2006-Oct-04 13:22 UTC
RE: [Xen-devel] Testing status of HVM (Intel VT) on 64bit XEN unstable c/s 11686
These bugs happen every time for me. To reproduce the bug I do the following: - Boot a release build of XEN - Launch a RHEL4U2-32bit HVM guest, 256MB, 2CPU SMP kernel, 4GB swap file, local SATA disk - ssh into the guest and run the following test: # while [ 1 ]; do mtest01 -v -p80; done - Within 1 or 2 passes of the test I hit one of the several crashes reported. - Nothing else is running on the guest and this is the only guest running on XEN. Hope this helps, Ed -----Original Message----- From: Tim Deegan [mailto:Tim.Deegan@xensource.com] Sent: Wednesday, October 04, 2006 9:13 AM To: Ed Smith Cc: Xen Devel Subject: Re: [Xen-devel] Testing status of HVM (Intel VT) on 64bit XEN unstable c/s 11686 Hi Ed, At 16:05 -0400 on 03 Oct (1159891531), Ed Smith wrote:> - NEW: Domain crash running ltp "mtest01 -p80" on 32bit SMP > HVM guest and then XEN locked up requiring a hard reset: > domain_crash_sync called from private.h:495 (failure.8) > - XEN crash running ltp "mtest01 -p80" on 32bit SMP HVM guest: > BUG at multi.c:2967 from sh_page_fault__shadow_3_guest_3 (failure.3)We''re having some difficulty reproducing these failures. Are you running full ltp tests or just the mtest01 binary? Are you running anything else in the VM at the time? How often do the failures occur? Was there a difference between the tests which caused the two failures or did they happen on different runs of the same test? Cheers, Tim. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Tim Deegan
2006-Oct-09 16:04 UTC
Re: [Xen-devel] Testing status of HVM (Intel VT) on 64bit XEN unstable c/s 11686
Hi Ed, Changeset 11729:6e932f32662c of xen-unstable should fix these crashes. Cheers, Tim. At 09:22 -0400 on 04 Oct (1159953769), Ed Smith wrote:> These bugs happen every time for me. To reproduce the bug I do the > following: > > - Boot a release build of XEN > - Launch a RHEL4U2-32bit HVM guest, 256MB, 2CPU SMP kernel, 4GB swap > file, local SATA disk > - ssh into the guest and run the following test: > # while [ 1 ]; do mtest01 -v -p80; done > - Within 1 or 2 passes of the test I hit one of the several crashes > reported. > - Nothing else is running on the guest and this is the only guest > running on XEN. > > Hope this helps, > Ed > > > -----Original Message----- > From: Tim Deegan [mailto:Tim.Deegan@xensource.com] > Sent: Wednesday, October 04, 2006 9:13 AM > To: Ed Smith > Cc: Xen Devel > Subject: Re: [Xen-devel] Testing status of HVM (Intel VT) on 64bit XEN > unstable c/s 11686 > > Hi Ed, > > At 16:05 -0400 on 03 Oct (1159891531), Ed Smith wrote: > > - NEW: Domain crash running ltp "mtest01 -p80" on 32bit SMP > > HVM guest and then XEN locked up requiring a hard reset: > > domain_crash_sync called from private.h:495 (failure.8) > > - XEN crash running ltp "mtest01 -p80" on 32bit SMP HVM guest: > > BUG at multi.c:2967 from sh_page_fault__shadow_3_guest_3 (failure.3) > > We''re having some difficulty reproducing these failures. Are you > running full ltp tests or just the mtest01 binary? Are you running > anything else in the VM at the time? How often do the failures occur? > Was there a difference between the tests which caused the two failures > or did they happen on different runs of the same test? > > Cheers, > > Tim._______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ed Smith
2006-Oct-10 13:17 UTC
RE: [Xen-devel] Testing status of HVM (Intel VT) on 64bit XEN unstable c/s 11686
Thanks Tim, I''m testing with c/s 11731 now and it looks good. I''ll be posting my test report shortly. Ed -----Original Message----- From: Tim Deegan [mailto:Tim.Deegan@xensource.com] Sent: Monday, October 09, 2006 12:05 PM To: Ed Smith Cc: Xen Devel Subject: Re: [Xen-devel] Testing status of HVM (Intel VT) on 64bit XEN unstable c/s 11686 Hi Ed, Changeset 11729:6e932f32662c of xen-unstable should fix these crashes. Cheers, Tim. At 09:22 -0400 on 04 Oct (1159953769), Ed Smith wrote:> These bugs happen every time for me. To reproduce the bug I do the > following: > > - Boot a release build of XEN > - Launch a RHEL4U2-32bit HVM guest, 256MB, 2CPU SMP kernel, 4GB swap > file, local SATA disk > - ssh into the guest and run the following test: > # while [ 1 ]; do mtest01 -v -p80; done > - Within 1 or 2 passes of the test I hit one of the several crashes > reported. > - Nothing else is running on the guest and this is the only guest > running on XEN. > > Hope this helps, > Ed > > > -----Original Message----- > From: Tim Deegan [mailto:Tim.Deegan@xensource.com] > Sent: Wednesday, October 04, 2006 9:13 AM > To: Ed Smith > Cc: Xen Devel > Subject: Re: [Xen-devel] Testing status of HVM (Intel VT) on 64bit XEN> unstable c/s 11686 > > Hi Ed, > > At 16:05 -0400 on 03 Oct (1159891531), Ed Smith wrote: > > - NEW: Domain crash running ltp "mtest01 -p80" on 32bit SMP > > HVM guest and then XEN locked up requiring a hard reset: > > domain_crash_sync called from private.h:495 (failure.8) > > - XEN crash running ltp "mtest01 -p80" on 32bit SMP HVM guest: > > BUG at multi.c:2967 from sh_page_fault__shadow_3_guest_3 > > (failure.3) > > We''re having some difficulty reproducing these failures. Are you > running full ltp tests or just the mtest01 binary? Are you running > anything else in the VM at the time? How often do the failures occur? > Was there a difference between the tests which caused the two failures> or did they happen on different runs of the same test? > > Cheers, > > Tim._______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ed Smith
2006-Oct-10 14:44 UTC
Re: [Xen-devel] Testing status of HVM (Intel VT) on 64bit XEN unstable c/s 11686
OK it took a lot longer to reproduce but I did hit the Error: shadow lock held by shadow_update_cr3 crash running mtest01 on a RHEL4U2-32bit SMP HVM guest. I was unable to reproduce any of the other shadow crashes. Details in my next test report post for c/s 11731. Ed Ed Smith wrote:> Thanks Tim, > > I''m testing with c/s 11731 now and it looks good. I''ll be posting my > test report shortly. > > Ed > > -----Original Message----- > From: Tim Deegan [mailto:Tim.Deegan@xensource.com] > Sent: Monday, October 09, 2006 12:05 PM > To: Ed Smith > Cc: Xen Devel > Subject: Re: [Xen-devel] Testing status of HVM (Intel VT) on 64bit XEN > unstable c/s 11686 > > Hi Ed, > > Changeset 11729:6e932f32662c of xen-unstable should fix these crashes. > > Cheers, > > Tim. > > At 09:22 -0400 on 04 Oct (1159953769), Ed Smith wrote: >> These bugs happen every time for me. To reproduce the bug I do the >> following: >> >> - Boot a release build of XEN >> - Launch a RHEL4U2-32bit HVM guest, 256MB, 2CPU SMP kernel, 4GB swap >> file, local SATA disk >> - ssh into the guest and run the following test: >> # while [ 1 ]; do mtest01 -v -p80; done >> - Within 1 or 2 passes of the test I hit one of the several crashes >> reported. >> - Nothing else is running on the guest and this is the only guest >> running on XEN. >> >> Hope this helps, >> Ed >> >> >> -----Original Message----- >> From: Tim Deegan [mailto:Tim.Deegan@xensource.com] >> Sent: Wednesday, October 04, 2006 9:13 AM >> To: Ed Smith >> Cc: Xen Devel >> Subject: Re: [Xen-devel] Testing status of HVM (Intel VT) on 64bit XEN > >> unstable c/s 11686 >> >> Hi Ed, >> >> At 16:05 -0400 on 03 Oct (1159891531), Ed Smith wrote: >>> - NEW: Domain crash running ltp "mtest01 -p80" on 32bit SMP >>> HVM guest and then XEN locked up requiring a hard reset: >>> domain_crash_sync called from private.h:495 (failure.8) >>> - XEN crash running ltp "mtest01 -p80" on 32bit SMP HVM guest: >>> BUG at multi.c:2967 from sh_page_fault__shadow_3_guest_3 >>> (failure.3) >> We''re having some difficulty reproducing these failures. Are you >> running full ltp tests or just the mtest01 binary? Are you running >> anything else in the VM at the time? How often do the failures occur? >> Was there a difference between the tests which caused the two failures > >> or did they happen on different runs of the same test? >> >> Cheers, >> >> Tim._______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel