thr3ads.net - Xen users - [Xen-users] OpenSUSE 11.2 Xen HVM guest crash causing BUG() in dom0 [Jan 2010]

If this information is useful, please help other people find it:
Share via:

Pasi Kärkkäinen

2010-Jan-23 14:49 UTC

[Xen-users] OpenSUSE 11.2 Xen HVM guest crash causing BUG() in dom0

Hello,

I was talking on irc with an user who had problems with HVM guests 
(freebsd freenas) on OpenSUSE 11.2 Xen 3.4.1.

Andy/Jan: Have you guys seen this before? Is it normal to have 
this kind of BUG()s in dom0, when a guest crashes? 

dom0 kernel: 2.6.31.8-0.1-xen x86_64
Xen: Xen-3.4.1_19718_04-2.1 x86_64

So basicly HVM guest crashes, and then dom0 gets BUG().

Also reported here:
https://bugzilla.novell.com/show_bug.cgi?id=573305

First:
(XEN) Domain 4 reported crashed by domain 0 on cpu#0:

then lots of:

(XEN) domain_crash called from p2m.c:1091
(XEN) p2m_pod_demand_populate: Out of populate-on-demand memory!

And then:

[ 5293.824815] BUG: soft lockup - CPU#0 stuck for 61s! [qemu-dm:6405]
[ 5293.824815] Modules linked in: tun nf_conntrack_ipv4 nf_defrag_ipv4 xt_state
nf_conntrack xt_physdev iptable_filter ip_tables x_tables netbk blkbk
blkback_pagemap blktap xenbus_be edd nls_utf8 cifs i915 drm i2c_algo_bit video
output bridge stp llc fuse loop dm_mod usb_storage 3c59x 8250_pnp 8250_pci
intel_agp pcspkr heci(C) serio_raw iTCO_wdt wmi sr_mod sg iTCO_vendor_support
8250 agpgart container button i2c_i801 serial_core i2c_core usbhid hid raid456
raid6_pq async_xor async_memcpy async_tx xor raid1 raid0 uhci_hcd ehci_hcd
xenblk cdrom xennet fan processor ide_pci_generic ide_core ata_generic thermal
thermal_sys hwmon
[ 5293.824815] CPU 0:
[ 5293.824815] Modules linked in: tun nf_conntrack_ipv4 nf_defrag_ipv4 xt_state
nf_conntrack xt_physdev iptable_filter ip_tables x_tables netbk blkbk
blkback_pagemap blktap xenbus_be edd nls_utf8 cifs i915 drm i2c_algo_bit video
output bridge stp llc fuse loop dm_mod usb_storage 3c59x 8250_pnp 8250_pci
intel_agp pcspkr heci(C) serio_raw iTCO_wdt wmi sr_mod sg iTCO_vendor_support
8250 agpgart container button i2c_i801 serial_core i2c_core usbhid hid raid456
raid6_pq async_xor async_memcpy async_tx xor raid1 raid0 uhci_hcd ehci_hcd
xenblk cdrom xennet fan processor ide_pci_generic ide_core ata_generic thermal
thermal_sys hwmon
[ 5293.824815] Pid: 6405, comm: qemu-dm Tainted: G         C 2.6.31.8-0.1-xen #1
7484A8U
[ 5293.824815] RIP: e030:[<ffffffff8000802a>]  [<ffffffff8000802a>]
0xffffffff8000802a
[ 5293.824815] RSP: e02b:ffff880028791cc0  EFLAGS: 00000246
[ 5293.824815] RAX: 00000000ffffffea RBX: 8000000000000427 RCX: ffffffff8000802a
[ 5293.824815] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88002f4e5000
[ 5293.824815] RBP: ffff880028791d38 R08: 0000000000000000 R09: ffff880028791d00
[ 5293.824815] R10: 0000000000000004 R11: 0000000000000246 R12: ffff88002f4e5010
[ 5293.824815] R13: 00007f2512e9d000 R14: 00000000000016f6 R15: ffff88002f4e5000
[ 5293.824815] FS:  00007f251ede76f0(0000) GS:ffffc90000000000(0000)
knlGS:0000000000000000
[ 5293.824815] CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 5293.824815] CR2: 00007fd847703000 CR3: 000000001bd9d000 CR4: 0000000000002660
[ 5293.824815] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 5293.824815] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 5293.824815] Call Trace:
[ 5293.824815]  [<ffffffff80026759>] __direct_remap_pfn_range+0x1b9/0x200
[ 5293.824815]  [<ffffffff80026813>] direct_remap_pfn_range+0x43/0x50
[ 5293.824815]  [<ffffffff80311a08>] privcmd_ioctl+0x618/0x740
[ 5293.824815]  [<ffffffff80180772>] proc_reg_unlocked_ioctl+0x92/0x160
[ 5293.824815]  [<ffffffff8012b510>] vfs_ioctl+0x30/0xd0
[ 5293.824815]  [<ffffffff8012b6f0>] do_vfs_ioctl+0x90/0x430
[ 5293.824815]  [<ffffffff8012bb29>] sys_ioctl+0x99/0xb0
[ 5293.824815]  [<ffffffff8000c868>] system_call_fastpath+0x16/0x1b
[ 5293.824815]  [<00007f251d26d7e7>] 0x7f251d26d7e7

Full log here:
http://pastebin.com/m24b0e01

-- Pasi


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Andrew Lyon

2010-Jan-23 17:04 UTC

head link

[Xen-users] Re: OpenSUSE 11.2 Xen HVM guest crash causing BUG() in dom0

Pasi,

I''m not sure if that kernel rpm version includes the swiotlb fixes
that made 2.6.31 reliable on all of my xen systems, it would be a good
idea if the user tries the latest kotd rpm for kernel-xen as that
should be up to date with the git tree which definatly has the
necessary fixes.

Since Jan fixed the swiotlb problems I''ve not had a single crash with
.31 and I''ve tested it very thoroughly so I am surprised if there are
still undiscovered bugs.

Andy

On 23/01/2010, Pasi Kärkkäinen <pasik@iki.fi>
wrote:> Hello,
>
> I was talking on irc with an user who had problems with HVM guests
> (freebsd freenas) on OpenSUSE 11.2 Xen 3.4.1.
>
> Andy/Jan: Have you guys seen this before? Is it normal to have
> this kind of BUG()s in dom0, when a guest crashes?
>
> dom0 kernel: 2.6.31.8-0.1-xen x86_64
> Xen: Xen-3.4.1_19718_04-2.1 x86_64
>
> So basicly HVM guest crashes, and then dom0 gets BUG().
>
> Also reported here:
> https://bugzilla.novell.com/show_bug.cgi?id=573305
>
> First:
> (XEN) Domain 4 reported crashed by domain 0 on cpu#0:
>
> then lots of:
>
> (XEN) domain_crash called from p2m.c:1091
> (XEN) p2m_pod_demand_populate: Out of populate-on-demand memory!
>
> And then:
>
> [ 5293.824815] BUG: soft lockup - CPU#0 stuck for 61s! [qemu-dm:6405]
> [ 5293.824815] Modules linked in: tun nf_conntrack_ipv4 nf_defrag_ipv4
> xt_state nf_conntrack xt_physdev iptable_filter ip_tables x_tables netbk
> blkbk blkback_pagemap blktap xenbus_be edd nls_utf8 cifs i915 drm
> i2c_algo_bit video output bridge stp llc fuse loop dm_mod usb_storage 3c59x
> 8250_pnp 8250_pci intel_agp pcspkr heci(C) serio_raw iTCO_wdt wmi sr_mod sg
> iTCO_vendor_support 8250 agpgart container button i2c_i801 serial_core
> i2c_core usbhid hid raid456 raid6_pq async_xor async_memcpy async_tx xor
> raid1 raid0 uhci_hcd ehci_hcd xenblk cdrom xennet fan processor
> ide_pci_generic ide_core ata_generic thermal thermal_sys hwmon
> [ 5293.824815] CPU 0:
> [ 5293.824815] Modules linked in: tun nf_conntrack_ipv4 nf_defrag_ipv4
> xt_state nf_conntrack xt_physdev iptable_filter ip_tables x_tables netbk
> blkbk blkback_pagemap blktap xenbus_be edd nls_utf8 cifs i915 drm
> i2c_algo_bit video output bridge stp llc fuse loop dm_mod usb_storage 3c59x
> 8250_pnp 8250_pci intel_agp pcspkr heci(C) serio_raw iTCO_wdt wmi sr_mod sg
> iTCO_vendor_support 8250 agpgart container button i2c_i801 serial_core
> i2c_core usbhid hid raid456 raid6_pq async_xor async_memcpy async_tx xor
> raid1 raid0 uhci_hcd ehci_hcd xenblk cdrom xennet fan processor
> ide_pci_generic ide_core ata_generic thermal thermal_sys hwmon
> [ 5293.824815] Pid: 6405, comm: qemu-dm Tainted: G         C
> 2.6.31.8-0.1-xen #1 7484A8U
> [ 5293.824815] RIP: e030:[<ffffffff8000802a>] 
[<ffffffff8000802a>]
> 0xffffffff8000802a
> [ 5293.824815] RSP: e02b:ffff880028791cc0  EFLAGS: 00000246
> [ 5293.824815] RAX: 00000000ffffffea RBX: 8000000000000427 RCX:
> ffffffff8000802a
> [ 5293.824815] RDX: 0000000000000000 RSI: 0000000000000001 RDI:
> ffff88002f4e5000
> [ 5293.824815] RBP: ffff880028791d38 R08: 0000000000000000 R09:
> ffff880028791d00
> [ 5293.824815] R10: 0000000000000004 R11: 0000000000000246 R12:
> ffff88002f4e5010
> [ 5293.824815] R13: 00007f2512e9d000 R14: 00000000000016f6 R15:
> ffff88002f4e5000
> [ 5293.824815] FS:  00007f251ede76f0(0000) GS:ffffc90000000000(0000)
> knlGS:0000000000000000
> [ 5293.824815] CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 5293.824815] CR2: 00007fd847703000 CR3: 000000001bd9d000 CR4:
> 0000000000002660
> [ 5293.824815] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> 0000000000000000
> [ 5293.824815] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
> 0000000000000400
> [ 5293.824815] Call Trace:
> [ 5293.824815]  [<ffffffff80026759>]
__direct_remap_pfn_range+0x1b9/0x200
> [ 5293.824815]  [<ffffffff80026813>] direct_remap_pfn_range+0x43/0x50
> [ 5293.824815]  [<ffffffff80311a08>] privcmd_ioctl+0x618/0x740
> [ 5293.824815]  [<ffffffff80180772>]
proc_reg_unlocked_ioctl+0x92/0x160
> [ 5293.824815]  [<ffffffff8012b510>] vfs_ioctl+0x30/0xd0
> [ 5293.824815]  [<ffffffff8012b6f0>] do_vfs_ioctl+0x90/0x430
> [ 5293.824815]  [<ffffffff8012bb29>] sys_ioctl+0x99/0xb0
> [ 5293.824815]  [<ffffffff8000c868>] system_call_fastpath+0x16/0x1b
> [ 5293.824815]  [<00007f251d26d7e7>] 0x7f251d26d7e7
>
> Full log here:
> http://pastebin.com/m24b0e01
>
> -- Pasi
>
>
-- 
Sent from my mobile device

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Pasi Kärkkäinen

2010-Jan-23 17:33 UTC

head link

[Xen-users] Re: OpenSUSE 11.2 Xen HVM guest crash causing BUG() in dom0

On Sat, Jan 23, 2010 at 05:04:37PM +0000, Andrew Lyon
wrote:> Pasi,
> 
> I''m not sure if that kernel rpm version includes the swiotlb fixes
> that made 2.6.31 reliable on all of my xen systems, it would be a good
> idea if the user tries the latest kotd rpm for kernel-xen as that
> should be up to date with the git tree which definatly has the
> necessary fixes.
> 
ftp://ftp.suse.com/pub/projects/kernel/kotd/master/x86_64/
Seems to have 2.6.32 based kernels.

Do you know if there are newer 2.6.31 based? 
2.6.31.8-0.1-xen in OpenSUSE 11.2 seems to be dated 18 Dec 2009.
> Since Jan fixed the swiotlb problems I''ve not had a single crash
with
> .31 and I''ve tested it very thoroughly so I am surprised if there
are
> still undiscovered bugs.
> 
Yeah.. that''s what I said too :)

-- Pasi
> Andy
> 
> On 23/01/2010, Pasi Kärkkäinen <pasik@iki.fi> wrote:
> > Hello,
> >
> > I was talking on irc with an user who had problems with HVM guests
> > (freebsd freenas) on OpenSUSE 11.2 Xen 3.4.1.
> >
> > Andy/Jan: Have you guys seen this before? Is it normal to have
> > this kind of BUG()s in dom0, when a guest crashes?
> >
> > dom0 kernel: 2.6.31.8-0.1-xen x86_64
> > Xen: Xen-3.4.1_19718_04-2.1 x86_64
> >
> > So basicly HVM guest crashes, and then dom0 gets BUG().
> >
> > Also reported here:
> > https://bugzilla.novell.com/show_bug.cgi?id=573305
> >
> > First:
> > (XEN) Domain 4 reported crashed by domain 0 on cpu#0:
> >
> > then lots of:
> >
> > (XEN) domain_crash called from p2m.c:1091
> > (XEN) p2m_pod_demand_populate: Out of populate-on-demand memory!
> >
> > And then:
> >
> > [ 5293.824815] BUG: soft lockup - CPU#0 stuck for 61s! [qemu-dm:6405]
> > [ 5293.824815] Modules linked in: tun nf_conntrack_ipv4 nf_defrag_ipv4
> > xt_state nf_conntrack xt_physdev iptable_filter ip_tables x_tables
netbk
> > blkbk blkback_pagemap blktap xenbus_be edd nls_utf8 cifs i915 drm
> > i2c_algo_bit video output bridge stp llc fuse loop dm_mod usb_storage
3c59x
> > 8250_pnp 8250_pci intel_agp pcspkr heci(C) serio_raw iTCO_wdt wmi
sr_mod sg
> > iTCO_vendor_support 8250 agpgart container button i2c_i801 serial_core
> > i2c_core usbhid hid raid456 raid6_pq async_xor async_memcpy async_tx
xor
> > raid1 raid0 uhci_hcd ehci_hcd xenblk cdrom xennet fan processor
> > ide_pci_generic ide_core ata_generic thermal thermal_sys hwmon
> > [ 5293.824815] CPU 0:
> > [ 5293.824815] Modules linked in: tun nf_conntrack_ipv4 nf_defrag_ipv4
> > xt_state nf_conntrack xt_physdev iptable_filter ip_tables x_tables
netbk
> > blkbk blkback_pagemap blktap xenbus_be edd nls_utf8 cifs i915 drm
> > i2c_algo_bit video output bridge stp llc fuse loop dm_mod usb_storage
3c59x
> > 8250_pnp 8250_pci intel_agp pcspkr heci(C) serio_raw iTCO_wdt wmi
sr_mod sg
> > iTCO_vendor_support 8250 agpgart container button i2c_i801 serial_core
> > i2c_core usbhid hid raid456 raid6_pq async_xor async_memcpy async_tx
xor
> > raid1 raid0 uhci_hcd ehci_hcd xenblk cdrom xennet fan processor
> > ide_pci_generic ide_core ata_generic thermal thermal_sys hwmon
> > [ 5293.824815] Pid: 6405, comm: qemu-dm Tainted: G         C
> > 2.6.31.8-0.1-xen #1 7484A8U
> > [ 5293.824815] RIP: e030:[<ffffffff8000802a>] 
[<ffffffff8000802a>]
> > 0xffffffff8000802a
> > [ 5293.824815] RSP: e02b:ffff880028791cc0  EFLAGS: 00000246
> > [ 5293.824815] RAX: 00000000ffffffea RBX: 8000000000000427 RCX:
> > ffffffff8000802a
> > [ 5293.824815] RDX: 0000000000000000 RSI: 0000000000000001 RDI:
> > ffff88002f4e5000
> > [ 5293.824815] RBP: ffff880028791d38 R08: 0000000000000000 R09:
> > ffff880028791d00
> > [ 5293.824815] R10: 0000000000000004 R11: 0000000000000246 R12:
> > ffff88002f4e5010
> > [ 5293.824815] R13: 00007f2512e9d000 R14: 00000000000016f6 R15:
> > ffff88002f4e5000
> > [ 5293.824815] FS:  00007f251ede76f0(0000) GS:ffffc90000000000(0000)
> > knlGS:0000000000000000
> > [ 5293.824815] CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
> > [ 5293.824815] CR2: 00007fd847703000 CR3: 000000001bd9d000 CR4:
> > 0000000000002660
> > [ 5293.824815] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> > 0000000000000000
> > [ 5293.824815] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
> > 0000000000000400
> > [ 5293.824815] Call Trace:
> > [ 5293.824815]  [<ffffffff80026759>]
__direct_remap_pfn_range+0x1b9/0x200
> > [ 5293.824815]  [<ffffffff80026813>]
direct_remap_pfn_range+0x43/0x50
> > [ 5293.824815]  [<ffffffff80311a08>] privcmd_ioctl+0x618/0x740
> > [ 5293.824815]  [<ffffffff80180772>]
proc_reg_unlocked_ioctl+0x92/0x160
> > [ 5293.824815]  [<ffffffff8012b510>] vfs_ioctl+0x30/0xd0
> > [ 5293.824815]  [<ffffffff8012b6f0>] do_vfs_ioctl+0x90/0x430
> > [ 5293.824815]  [<ffffffff8012bb29>] sys_ioctl+0x99/0xb0
> > [ 5293.824815]  [<ffffffff8000c868>]
system_call_fastpath+0x16/0x1b
> > [ 5293.824815]  [<00007f251d26d7e7>] 0x7f251d26d7e7
> >
> > Full log here:
> > http://pastebin.com/m24b0e01
> >
> > -- Pasi
> >
> >
> 
> -- 
> Sent from my mobile device
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

mail ignored

2010-Jan-23 17:41 UTC

head link

Re: [Xen-users] Re: OpenSUSE 11.2 Xen HVM guest crash causing BUG() in dom0

On Sat, Jan 23, 2010 at 9:33 AM, Pasi Kärkkäinen <pasik@iki.fi>
wrote:> Do you know if there are newer 2.6.31 based?
> 2.6.31.8-0.1-xen in OpenSUSE 11.2 seems to be dated 18 Dec 2009.
-> ftp://ftp.suse.com/pub/projects/kernel/kotd/openSUSE-11.2/x86_64/

has,

  kernel-xen-2.6.31.12-0.0.0.7.1bdde44.x86_64.rpm

etc ...

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Pasi Kärkkäinen

2010-Jan-23 17:46 UTC

head link

Re: [Xen-users] Re: OpenSUSE 11.2 Xen HVM guest crash causing BUG() in dom0

On Sat, Jan 23, 2010 at 09:41:08AM -0800, mail ignored
wrote:> On Sat, Jan 23, 2010 at 9:33 AM, Pasi Kärkkäinen <pasik@iki.fi>
wrote:
> > Do you know if there are newer 2.6.31 based?
> > 2.6.31.8-0.1-xen in OpenSUSE 11.2 seems to be dated 18 Dec 2009.
> 
> -> ftp://ftp.suse.com/pub/projects/kernel/kotd/openSUSE-11.2/x86_64/
> 
> has,
> 
>   kernel-xen-2.6.31.12-0.0.0.7.1bdde44.x86_64.rpm
> 
> etc ...
>
Doh. Should have noticed that.
Thanks. Hopefully he''ll try that kernel..

-- Pasi


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Mike Fenton

2010-Jan-23 18:44 UTC

head link

[Xen-users] Re: OpenSUSE 11.2 Xen HVM guest crash causing BUG() in dom0

Hi Andrew,

I just updated to the latest kotd with no luck. Xen is producing the same
error.

On Sat, Jan 23, 2010 at 12:33 PM, Pasi Kärkkäinen <pasik@iki.fi> wrote:
> On Sat, Jan 23, 2010 at 05:04:37PM +0000, Andrew Lyon wrote:
> > Pasi,
> >
> > I''m not sure if that kernel rpm version includes the swiotlb
fixes
> > that made 2.6.31 reliable on all of my xen systems, it would be a good
> > idea if the user tries the latest kotd rpm for kernel-xen as that
> > should be up to date with the git tree which definatly has the
> > necessary fixes.
> >
>
> ftp://ftp.suse.com/pub/projects/kernel/kotd/master/x86_64/
> Seems to have 2.6.32 based kernels.
>
> Do you know if there are newer 2.6.31 based?
> 2.6.31.8-0.1-xen in OpenSUSE 11.2 seems to be dated 18 Dec 2009.
>
> > Since Jan fixed the swiotlb problems I''ve not had a single
crash with
> > .31 and I''ve tested it very thoroughly so I am surprised if
there are
> > still undiscovered bugs.
> >
>
> Yeah.. that''s what I said too :)
>
> -- Pasi
>
> > Andy
> >
> > On 23/01/2010, Pasi Kärkkäinen <pasik@iki.fi> wrote:
> > > Hello,
> > >
> > > I was talking on irc with an user who had problems with HVM
guests
> > > (freebsd freenas) on OpenSUSE 11.2 Xen 3.4.1.
> > >
> > > Andy/Jan: Have you guys seen this before? Is it normal to have
> > > this kind of BUG()s in dom0, when a guest crashes?
> > >
> > > dom0 kernel: 2.6.31.8-0.1-xen x86_64
> > > Xen: Xen-3.4.1_19718_04-2.1 x86_64
> > >
> > > So basicly HVM guest crashes, and then dom0 gets BUG().
> > >
> > > Also reported here:
> > > https://bugzilla.novell.com/show_bug.cgi?id=573305
> > >
> > > First:
> > > (XEN) Domain 4 reported crashed by domain 0 on cpu#0:
> > >
> > > then lots of:
> > >
> > > (XEN) domain_crash called from p2m.c:1091
> > > (XEN) p2m_pod_demand_populate: Out of populate-on-demand memory!
> > >
> > > And then:
> > >
> > > [ 5293.824815] BUG: soft lockup - CPU#0 stuck for 61s!
[qemu-dm:6405]
> > > [ 5293.824815] Modules linked in: tun nf_conntrack_ipv4
nf_defrag_ipv4
> > > xt_state nf_conntrack xt_physdev iptable_filter ip_tables
x_tables
> netbk
> > > blkbk blkback_pagemap blktap xenbus_be edd nls_utf8 cifs i915 drm
> > > i2c_algo_bit video output bridge stp llc fuse loop dm_mod
usb_storage
> 3c59x
> > > 8250_pnp 8250_pci intel_agp pcspkr heci(C) serio_raw iTCO_wdt wmi
> sr_mod sg
> > > iTCO_vendor_support 8250 agpgart container button i2c_i801
serial_core
> > > i2c_core usbhid hid raid456 raid6_pq async_xor async_memcpy
async_tx
> xor
> > > raid1 raid0 uhci_hcd ehci_hcd xenblk cdrom xennet fan processor
> > > ide_pci_generic ide_core ata_generic thermal thermal_sys hwmon
> > > [ 5293.824815] CPU 0:
> > > [ 5293.824815] Modules linked in: tun nf_conntrack_ipv4
nf_defrag_ipv4
> > > xt_state nf_conntrack xt_physdev iptable_filter ip_tables
x_tables
> netbk
> > > blkbk blkback_pagemap blktap xenbus_be edd nls_utf8 cifs i915 drm
> > > i2c_algo_bit video output bridge stp llc fuse loop dm_mod
usb_storage
> 3c59x
> > > 8250_pnp 8250_pci intel_agp pcspkr heci(C) serio_raw iTCO_wdt wmi
> sr_mod sg
> > > iTCO_vendor_support 8250 agpgart container button i2c_i801
serial_core
> > > i2c_core usbhid hid raid456 raid6_pq async_xor async_memcpy
async_tx
> xor
> > > raid1 raid0 uhci_hcd ehci_hcd xenblk cdrom xennet fan processor
> > > ide_pci_generic ide_core ata_generic thermal thermal_sys hwmon
> > > [ 5293.824815] Pid: 6405, comm: qemu-dm Tainted: G         C
> > > 2.6.31.8-0.1-xen #1 7484A8U
> > > [ 5293.824815] RIP: e030:[<ffffffff8000802a>] 
[<ffffffff8000802a>]
> > > 0xffffffff8000802a
> > > [ 5293.824815] RSP: e02b:ffff880028791cc0  EFLAGS: 00000246
> > > [ 5293.824815] RAX: 00000000ffffffea RBX: 8000000000000427 RCX:
> > > ffffffff8000802a
> > > [ 5293.824815] RDX: 0000000000000000 RSI: 0000000000000001 RDI:
> > > ffff88002f4e5000
> > > [ 5293.824815] RBP: ffff880028791d38 R08: 0000000000000000 R09:
> > > ffff880028791d00
> > > [ 5293.824815] R10: 0000000000000004 R11: 0000000000000246 R12:
> > > ffff88002f4e5010
> > > [ 5293.824815] R13: 00007f2512e9d000 R14: 00000000000016f6 R15:
> > > ffff88002f4e5000
> > > [ 5293.824815] FS:  00007f251ede76f0(0000)
GS:ffffc90000000000(0000)
> > > knlGS:0000000000000000
> > > [ 5293.824815] CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
> > > [ 5293.824815] CR2: 00007fd847703000 CR3: 000000001bd9d000 CR4:
> > > 0000000000002660
> > > [ 5293.824815] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> > > 0000000000000000
> > > [ 5293.824815] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
> > > 0000000000000400
> > > [ 5293.824815] Call Trace:
> > > [ 5293.824815]  [<ffffffff80026759>]
> __direct_remap_pfn_range+0x1b9/0x200
> > > [ 5293.824815]  [<ffffffff80026813>]
direct_remap_pfn_range+0x43/0x50
> > > [ 5293.824815]  [<ffffffff80311a08>]
privcmd_ioctl+0x618/0x740
> > > [ 5293.824815]  [<ffffffff80180772>]
proc_reg_unlocked_ioctl+0x92/0x160
> > > [ 5293.824815]  [<ffffffff8012b510>] vfs_ioctl+0x30/0xd0
> > > [ 5293.824815]  [<ffffffff8012b6f0>]
do_vfs_ioctl+0x90/0x430
> > > [ 5293.824815]  [<ffffffff8012bb29>] sys_ioctl+0x99/0xb0
> > > [ 5293.824815]  [<ffffffff8000c868>]
system_call_fastpath+0x16/0x1b
> > > [ 5293.824815]  [<00007f251d26d7e7>] 0x7f251d26d7e7
> > >
> > > Full log here:
> > > http://pastebin.com/m24b0e01
> > >
> > > -- Pasi
> > >
> > >
> >
> > --
> > Sent from my mobile device
>

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Pasi Kärkkäinen

2010-Jan-23 19:14 UTC

head link

[Xen-users] Re: OpenSUSE 11.2 Xen HVM guest crash causing BUG() in dom0

On Sat, Jan 23, 2010 at 01:44:56PM -0500, Mike Fenton
wrote:>    Hi Andrew,
> 
>    I just updated to the latest kotd with no luck. Xen is producing the
same
>    error.
> 
Just to be more clear, Mike used 2.6.32-41.1 x86_64 kernel from 
http://download.opensuse.org/repositories/Kernel:/HEAD/openSUSE_11.2/x86_64/

and it seems he doesn''t get BUG() and traceback in dom0 anymore when 
HVM guest crashes:

http://pastebin.com/m5aff9158

(XEN) Domain 3 reported crashed by domain 0 on cpu#0:

lots of these still though:

(XEN) domain_crash called from p2m.c:1091
(XEN) p2m_pod_demand_populate: Out of populate-on-demand memory!

and then

[  233.171451] BUG: soft lockup - CPU#0 stuck for 61s! [qemu-dm:3722]

So I guess that''s OK now..

-- Pasi

>    On Sat, Jan 23, 2010 at 12:33 PM, Pasi Kärkkäinen
<[1]pasik@iki.fi> wrote:
> 
>      On Sat, Jan 23, 2010 at 05:04:37PM +0000, Andrew Lyon wrote:
>      > Pasi,
>      >
>      > I''m not sure if that kernel rpm version includes the
swiotlb fixes
>      > that made 2.6.31 reliable on all of my xen systems, it would be a
good
>      > idea if the user tries the latest kotd rpm for kernel-xen as that
>      > should be up to date with the git tree which definatly has the
>      > necessary fixes.
>      >
> 
>      [2]ftp://ftp.suse.com/pub/projects/kernel/kotd/master/x86_64/
>      Seems to have 2.6.32 based kernels.
> 
>      Do you know if there are newer 2.6.31 based?
>      2.6.31.8-0.1-xen in OpenSUSE 11.2 seems to be dated 18 Dec 2009.
>      > Since Jan fixed the swiotlb problems I''ve not had a
single crash with
>      > .31 and I''ve tested it very thoroughly so I am surprised
if there are
>      > still undiscovered bugs.
>      >
> 
>      Yeah.. that''s what I said too :)
>      -- Pasi
>      > Andy
>      >
>      > On 23/01/2010, Pasi Kärkkäinen <[3]pasik@iki.fi> wrote:
>      > > Hello,
>      > >
>      > > I was talking on irc with an user who had problems with HVM
guests
>      > > (freebsd freenas) on OpenSUSE 11.2 Xen 3.4.1.
>      > >
>      > > Andy/Jan: Have you guys seen this before? Is it normal to
have
>      > > this kind of BUG()s in dom0, when a guest crashes?
>      > >
>      > > dom0 kernel: 2.6.31.8-0.1-xen x86_64
>      > > Xen: Xen-3.4.1_19718_04-2.1 x86_64
>      > >
>      > > So basicly HVM guest crashes, and then dom0 gets BUG().
>      > >
>      > > Also reported here:
>      > > [4]https://bugzilla.novell.com/show_bug.cgi?id=573305
>      > >
>      > > First:
>      > > (XEN) Domain 4 reported crashed by domain 0 on cpu#0:
>      > >
>      > > then lots of:
>      > >
>      > > (XEN) domain_crash called from p2m.c:1091
>      > > (XEN) p2m_pod_demand_populate: Out of populate-on-demand
memory!
>      > >
>      > > And then:
>      > >
>      > > [ 5293.824815] BUG: soft lockup - CPU#0 stuck for 61s!
>      [qemu-dm:6405]
>      > > [ 5293.824815] Modules linked in: tun nf_conntrack_ipv4
>      nf_defrag_ipv4
>      > > xt_state nf_conntrack xt_physdev iptable_filter ip_tables
x_tables
>      netbk
>      > > blkbk blkback_pagemap blktap xenbus_be edd nls_utf8 cifs
i915 drm
>      > > i2c_algo_bit video output bridge stp llc fuse loop dm_mod
>      usb_storage 3c59x
>      > > 8250_pnp 8250_pci intel_agp pcspkr heci(C) serio_raw
iTCO_wdt wmi
>      sr_mod sg
>      > > iTCO_vendor_support 8250 agpgart container button i2c_i801
>      serial_core
>      > > i2c_core usbhid hid raid456 raid6_pq async_xor async_memcpy
async_tx
>      xor
>      > > raid1 raid0 uhci_hcd ehci_hcd xenblk cdrom xennet fan
processor
>      > > ide_pci_generic ide_core ata_generic thermal thermal_sys
hwmon
>      > > [ 5293.824815] CPU 0:
>      > > [ 5293.824815] Modules linked in: tun nf_conntrack_ipv4
>      nf_defrag_ipv4
>      > > xt_state nf_conntrack xt_physdev iptable_filter ip_tables
x_tables
>      netbk
>      > > blkbk blkback_pagemap blktap xenbus_be edd nls_utf8 cifs
i915 drm
>      > > i2c_algo_bit video output bridge stp llc fuse loop dm_mod
>      usb_storage 3c59x
>      > > 8250_pnp 8250_pci intel_agp pcspkr heci(C) serio_raw
iTCO_wdt wmi
>      sr_mod sg
>      > > iTCO_vendor_support 8250 agpgart container button i2c_i801
>      serial_core
>      > > i2c_core usbhid hid raid456 raid6_pq async_xor async_memcpy
async_tx
>      xor
>      > > raid1 raid0 uhci_hcd ehci_hcd xenblk cdrom xennet fan
processor
>      > > ide_pci_generic ide_core ata_generic thermal thermal_sys
hwmon
>      > > [ 5293.824815] Pid: 6405, comm: qemu-dm Tainted: G         C
>      > > 2.6.31.8-0.1-xen #1 7484A8U
>      > > [ 5293.824815] RIP: e030:[<ffffffff8000802a>] 
[<ffffffff8000802a>]
>      > > 0xffffffff8000802a
>      > > [ 5293.824815] RSP: e02b:ffff880028791cc0  EFLAGS: 00000246
>      > > [ 5293.824815] RAX: 00000000ffffffea RBX: 8000000000000427
RCX:
>      > > ffffffff8000802a
>      > > [ 5293.824815] RDX: 0000000000000000 RSI: 0000000000000001
RDI:
>      > > ffff88002f4e5000
>      > > [ 5293.824815] RBP: ffff880028791d38 R08: 0000000000000000
R09:
>      > > ffff880028791d00
>      > > [ 5293.824815] R10: 0000000000000004 R11: 0000000000000246
R12:
>      > > ffff88002f4e5010
>      > > [ 5293.824815] R13: 00007f2512e9d000 R14: 00000000000016f6
R15:
>      > > ffff88002f4e5000
>      > > [ 5293.824815] FS:  00007f251ede76f0(0000)
GS:ffffc90000000000(0000)
>      > > knlGS:0000000000000000
>      > > [ 5293.824815] CS:  e033 DS: 0000 ES: 0000 CR0:
000000008005003b
>      > > [ 5293.824815] CR2: 00007fd847703000 CR3: 000000001bd9d000
CR4:
>      > > 0000000000002660
>      > > [ 5293.824815] DR0: 0000000000000000 DR1: 0000000000000000
DR2:
>      > > 0000000000000000
>      > > [ 5293.824815] DR3: 0000000000000000 DR6: 00000000ffff0ff0
DR7:
>      > > 0000000000000400
>      > > [ 5293.824815] Call Trace:
>      > > [ 5293.824815]  [<ffffffff80026759>]
>      __direct_remap_pfn_range+0x1b9/0x200
>      > > [ 5293.824815]  [<ffffffff80026813>]
>      direct_remap_pfn_range+0x43/0x50
>      > > [ 5293.824815]  [<ffffffff80311a08>]
privcmd_ioctl+0x618/0x740
>      > > [ 5293.824815]  [<ffffffff80180772>]
>      proc_reg_unlocked_ioctl+0x92/0x160
>      > > [ 5293.824815]  [<ffffffff8012b510>]
vfs_ioctl+0x30/0xd0
>      > > [ 5293.824815]  [<ffffffff8012b6f0>]
do_vfs_ioctl+0x90/0x430
>      > > [ 5293.824815]  [<ffffffff8012bb29>]
sys_ioctl+0x99/0xb0
>      > > [ 5293.824815]  [<ffffffff8000c868>]
system_call_fastpath+0x16/0x1b
>      > > [ 5293.824815]  [<00007f251d26d7e7>] 0x7f251d26d7e7
>      > >
>      > > Full log here:
>      > > [5]http://pastebin.com/m24b0e01
>      > >
>      > > -- Pasi
>      > >
>      > >
>      >
>      > --
>      > Sent from my mobile device
> 
> References
> 
>    Visible links
>    1. mailto:pasik@iki.fi
>    2. ftp://ftp.suse.com/pub/projects/kernel/kotd/master/x86_64/
>    3. mailto:pasik@iki.fi
>    4. https://bugzilla.novell.com/show_bug.cgi?id=573305
>    5. http://pastebin.com/m24b0e01
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Jan Beulich

2010-Jan-25 08:30 UTC

head link

[Xen-users] Re: OpenSUSE 11.2 Xen HVM guest crash causing BUG() in dom0

>>> Pasi Kärkkäinen<pasik@iki.fi> 23.01.10 20:14 >>>
>Just to be more clear, Mike used 2.6.32-41.1 x86_64 kernel from 
>http://download.opensuse.org/repositories/Kernel:/HEAD/openSUSE_11.2/x86_64/
>
>and it seems he doesn''t get BUG() and traceback in dom0 anymore
when
>HVM guest crashes:
>
>http://pastebin.com/m5aff9158 
>
>(XEN) Domain 3 reported crashed by domain 0 on cpu#0:
>
>lots of these still though:
>
>(XEN) domain_crash called from p2m.c:1091
>(XEN) p2m_pod_demand_populate: Out of populate-on-demand memory!
>
>and then
>
>[  233.171451] BUG: soft lockup - CPU#0 stuck for 61s! [qemu-dm:3722]
>
>So I guess that''s OK now..
Hmm, OK is probably not the right statement with there still being
lockups. However, I take your wording as the lockups not being
connected to the domain crashes anymore. Disabling C-state
management (cpuidle=0 or max_cstate=0 on the Xen command line)
might be worth trying (based on recent observations reported on
xen-devel).

Of course, fixing the underlying config problem might help the user
even more: Apparently they configure the guest with mem!=maxmem,
and either the balloon driver is not being loaded at all or isn''t
getting
its target set low enough (we''re apparently having the latter problem
in SLE11SP1, so it''s not impossible for it to also exist in 11.2).

Jan

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Pasi Kärkkäinen

2010-Jan-25 08:47 UTC

head link

[Xen-users] Re: OpenSUSE 11.2 Xen HVM guest crash causing BUG() in dom0

On Mon, Jan 25, 2010 at 08:30:33AM +0000, Jan Beulich
wrote:> >>> Pasi Kärkkäinen<pasik@iki.fi> 23.01.10 20:14
>>>
> >Just to be more clear, Mike used 2.6.32-41.1 x86_64 kernel from 
>
>http://download.opensuse.org/repositories/Kernel:/HEAD/openSUSE_11.2/x86_64/
> >
> >and it seems he doesn''t get BUG() and traceback in dom0
anymore when
> >HVM guest crashes:
> >
> >http://pastebin.com/m5aff9158 
> >
> >(XEN) Domain 3 reported crashed by domain 0 on cpu#0:
> >
> >lots of these still though:
> >
> >(XEN) domain_crash called from p2m.c:1091
> >(XEN) p2m_pod_demand_populate: Out of populate-on-demand memory!
> >
> >and then
> >
> >[  233.171451] BUG: soft lockup - CPU#0 stuck for 61s! [qemu-dm:3722]
> >
> >So I guess that''s OK now..
> 
> Hmm, OK is probably not the right statement with there still being
> lockups. However, I take your wording as the lockups not being
> connected to the domain crashes anymore. 
>
Sorry, I wrote it a bit wrong. The lockup still happens when the HVM guest
crashes,
but now there was not traceback anymore.
> Disabling C-state management (cpuidle=0 or max_cstate=0 on the Xen command
line)
> might be worth trying (based on recent observations reported on
> xen-devel).
> 
Hopefully Mike can try it..
> Of course, fixing the underlying config problem might help the user
> even more: Apparently they configure the guest with mem!=maxmem,
> and either the balloon driver is not being loaded at all or isn''t
getting
> its target set low enough (we''re apparently having the latter
problem
> in SLE11SP1, so it''s not impossible for it to also exist in 11.2).
>
Ok. The HVM guest was Freenas (FreeBSD), so I don''t know if it even has
balloon driver available..

-- Pasi


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Jan Beulich

2010-Jan-25 08:54 UTC

head link

[Xen-users] Re: OpenSUSE 11.2 Xen HVM guest crash causing BUG() in dom0

>>> Pasi Kärkkäinen<pasik@iki.fi> 25.01.10 09:47 >>>
>Ok. The HVM guest was Freenas (FreeBSD), so I don''t know if it even
has
>balloon driver available..
If it doesn''t it''s a mistake to configure mem!=maxmem.

Jan

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Mike Fenton

2010-Jan-25 13:59 UTC

head link

[Xen-users] Re: OpenSUSE 11.2 Xen HVM guest crash causing BUG() in dom0

I do not have the FreeNAS HVM guest anymore. I deleted the guest I rebuilt
with paravirtulization using OpenFiler.

Does mem!=maxmem mean memory shouldn''t equal maxmem? So for instance I
would
have something like I have below?

memory=512
maxmemory=768

On Mon, Jan 25, 2010 at 3:54 AM, Jan Beulich <JBeulich@novell.com> wrote:
> >>> Pasi Kärkkäinen<pasik@iki.fi> 25.01.10 09:47
>>>
> >Ok. The HVM guest was Freenas (FreeBSD), so I don''t know if it
even has
> >balloon driver available..
>
> If it doesn''t it''s a mistake to configure mem!=maxmem.
>
> Jan
>

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Jan Beulich

2010-Jan-25 15:18 UTC

head link

[Xen-users] Re: OpenSUSE 11.2 Xen HVM guest crash causing BUG() in dom0

>>> Mike Fenton <mfenton1@gmail.com> 25.01.10 14:59 >>>
>Does mem!=maxmem mean memory shouldn''t equal maxmem? So for
instance I would
>have something like I have below?
No, the other way around: memory should equal maxmem if you do not
have a balloon driver in the guest.

Jan


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Xen users - Jan 2010 - OpenSUSE 11.2 Xen HVM guest crash causing BUG() in dom0

[Xen-users] OpenSUSE 11.2 Xen HVM guest crash causing BUG() in dom0

[Xen-users] Re: OpenSUSE 11.2 Xen HVM guest crash causing BUG() in dom0

[Xen-users] Re: OpenSUSE 11.2 Xen HVM guest crash causing BUG() in dom0

Re: [Xen-users] Re: OpenSUSE 11.2 Xen HVM guest crash causing BUG() in dom0

Re: [Xen-users] Re: OpenSUSE 11.2 Xen HVM guest crash causing BUG() in dom0

[Xen-users] Re: OpenSUSE 11.2 Xen HVM guest crash causing BUG() in dom0

[Xen-users] Re: OpenSUSE 11.2 Xen HVM guest crash causing BUG() in dom0

[Xen-users] Re: OpenSUSE 11.2 Xen HVM guest crash causing BUG() in dom0

[Xen-users] Re: OpenSUSE 11.2 Xen HVM guest crash causing BUG() in dom0

[Xen-users] Re: OpenSUSE 11.2 Xen HVM guest crash causing BUG() in dom0

[Xen-users] Re: OpenSUSE 11.2 Xen HVM guest crash causing BUG() in dom0

[Xen-users] Re: OpenSUSE 11.2 Xen HVM guest crash causing BUG() in dom0