thr3ads.net - Xen devel - [Xen-devel] Tapdisk failures / kernel general protection fault at xen 4.0.2rc3 / kernel pvops 2.6.32.36 [Apr 2011]

If this information is useful, please help other people find it:
Share via:

Gerd Jakobovitsch

2011-Apr-13 21:02 UTC

[Xen-devel] Tapdisk failures / kernel general protection fault at xen 4.0.2rc3 / kernel pvops 2.6.32.36

I''m trying to run several VMs (linux hvm, with tapdisk:aio disks at a 
storage over nfs) on a CentOS system, using the up-to-date version of 
xen 4.0 / kernel pvops 2.6.32.x stable. With a configuration without 
(most of) debug activated, I can start several instances - I''m running
7
of them - but shortly afterwards the system stops responding. I can''t 
find any information on this.

Activating several debug configuration items, among them 
DEBUG_PAGEALLOC, I get an exception as soon as I try to start up a VM. 
The system reboots.

Below the log from /var/log/messages:

Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[4988]: Created 
/dev/xen/blktap-2/control device
Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[4988]: Created 
/dev/xen/blktap-2/blktap0 device
Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[4988]: Created 
/dev/xen/blktap-2/tapdev0 device
Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[4988]: new interface: ring: 251, 
device: 253, minor: 0
Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[4988]: I/O queue driver: lio
Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[4988]: block-aio 
open(''/storage5_nfs/3/CD996633-linux-centos-5-64b-base-rip-sx-7253/hda'')
Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[4988]: 
open(/storage5_nfs/3/CD996633-linux-centos-5-64b-base-rip-sx-7253/hda) 
with O_DIRECT
Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[4988]: Image size:       pre 
sector_shift  [134217728]   post sector_shift [262144]
Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[4988]: opened image 
/storage5_nfs/3/CD996633-linux-centos-5-64b-base-rip-sx-7253/hda (1 
users, state: 0x00000001, type: 0)
Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[4988]: VBD CHAIN:
Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[4988]: 
/storage5_nfs/3/CD996633-linux-centos-5-64b-base-rip-sx-7253/hda: 0
Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.158549] block tda: 
sector-size: 512 capacity: 262144
Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.200514] general protection 
fault: 0000 [#1] SMP DEBUG_PAGEALLOC
Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.200703] last sysfs file: 
/sys/block/tda/removable
Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.200761] CPU 0
Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.200847] Modules linked in: 
bridge stp bonding bnx2i libiscsi scsi_transport_iscsi cnic uio bnx2 
megaraid_sas
Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.201363] Pid: 4988, comm: 
tapdisk2 Not tainted 2.6.32.36 #3 PowerEdge M610
Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.201424] RIP: 
e030:[<ffffffff812b9c24>]  [<ffffffff812b9c24>] 
blktap_device_end_request+0x49/0x5e
Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.201543] RSP: 
e02b:ffff88006a7f7cd8  EFLAGS: 00010046
Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.201600] RAX: 
6b6b6b6b6b6b6b6b RBX: ffff88006a6fc000 RCX: ffff88006a7f7c38
Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.201662] RDX: 
0000000000000000 RSI: 0000000000000000 RDI: ffff88006a5c3500
Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.201723] RBP: 
ffff88006a7f7cf8 R08: ffffffff818383c0 R09: ffff88006a7f7c38
Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.201784] R10: 
0000000000000000 R11: ffff88007b697b18 R12: ffff88007b697b18
Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.201845] R13: 
ffff88006a5c3360 R14: 0000000000000000 R15: ffff88006a5c3370
Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.201910] FS:  
00007f50a9445730(0000) GS:ffff8800280c7000(0000) knlGS:0000000000000000
Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.201974] CS:  e033 DS: 0000 
ES: 0000 CR0: 000000008005003b
Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.202032] CR2: 
00007fb35d12e6e8 CR3: 000000006a4ce000 CR4: 0000000000002660
Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.202093] DR0: 
0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.202154] DR3: 
0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.202436] Process tapdisk2 
(pid: 4988, threadinfo ffff88006a7f6000, task ffff88006b5a0000)
Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.202941] Stack:
Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.203206]  ffff88006b5a0000 
0000000000000000 0000000000000000 0000000000000000
Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.203609] <0> 
ffff88006a7f7e88 ffffffff812b9416 ffff88006a6c80f8 0000000100000000
Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.204310] <0> 
00000000ffffffff ffff88006a5c3360 000000017edd7ab0 0000000000000000
Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.205284] Call Trace:
Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.205553]  
[<ffffffff812b9416>] blktap_ring_ioctl+0x183/0x2d8
Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.205838]  
[<ffffffff81209a64>] ? inode_has_perm+0xa1/0xb3
Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.206120]  
[<ffffffff8157641f>] ? _spin_unlock+0x26/0x2a
Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.206400]  
[<ffffffff81126ff9>] ? aio_read_evt+0x56/0xe0
Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.206678]  
[<ffffffff81127071>] ? aio_read_evt+0xce/0xe0
Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.206957]  
[<ffffffff8124f5c1>] ? _raw_spin_lock+0x77/0x12d
Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.207236]  
[<ffffffff81209bf8>] ? file_has_perm+0xb4/0xc6
Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.207516]  
[<ffffffff8110464e>] vfs_ioctl+0x5e/0x77
Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.207793]  
[<ffffffff81104b63>] do_vfs_ioctl+0x484/0x4d5
Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.208069]  
[<ffffffff81104c0b>] sys_ioctl+0x57/0x7a
Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.208346]  
[<ffffffff81012cc2>] system_call_fastpath+0x16/0x1b
Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.208621] Code: 89 de 4c 89 
ef e8 60 f4 ff ff 49 8b 44 24 40 48 8b b8 90 04 00 00 e8 41 c9 2b 00 44 
89 f6 4c 89 e7 e8 39 fc ff ff 49 8b 44 24 40 <48> 8b b8 90 04 00 00 e8 
66 c7 2b 00 5b 41 5c 41 5d 41 5e c9 c3
Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.211986] RIP  
[<ffffffff812b9c24>] blktap_device_end_request+0x49/0x5e
Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.212306]  RSP
<ffff88006a7f7cd8>
Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.212579] ---[ end trace 
b97070122f44735d ]---
Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[5009]: Created 
/dev/xen/blktap-2/blktap1 device
Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[5009]: Created 
/dev/xen/blktap-2/tapdev1 device
Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[5009]: new interface: ring: 251, 
device: 253, minor: 1
Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[5009]: I/O queue driver: lio
Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[5009]: block-aio 
open(''/storage5_nfs/3/CD996633-linux-centos-5-64b-base-rip-sx-7253/xvda'')
Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[5009]: 
open(/storage5_nfs/3/CD996633-linux-centos-5-64b-base-rip-sx-7253/xvda) 
with O_DIRECT
Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[5009]: Image size:       pre 
sector_shift  [10737418240]         post sector_shift [20971520]
Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[5009]: opened image 
/storage5_nfs/3/CD996633-linux-centos-5-64b-base-rip-sx-7253/xvda (1 
users, state: 0x00000001, type: 0)
Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[5009]: VBD CHAIN:
Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[5009]: 
/storage5_nfs/3/CD996633-linux-centos-5-64b-base-rip-sx-7253/xvda: 0
Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.317931] block tdb: 
sector-size: 512 capacity: 20971520



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Konrad Rzeszutek Wilk

2011-Apr-14 13:15 UTC

head link

Re: [Xen-devel] Tapdisk failures / kernel general protection fault at xen 4.0.2rc3 / kernel pvops 2.6.32.36

On Wed, Apr 13, 2011 at 06:02:13PM -0300, Gerd Jakobovitsch
wrote:> I''m trying to run several VMs (linux hvm, with tapdisk:aio disks
at
> a storage over nfs) on a CentOS system, using the up-to-date version
> of xen 4.0 / kernel pvops 2.6.32.x stable. With a configuration
> without (most of) debug activated, I can start several instances -
> I''m running 7 of them - but shortly afterwards the system stops
> responding. I can''t find any information on this.
First time I see it.> 
> Activating several debug configuration items, among them
> DEBUG_PAGEALLOC, I get an exception as soon as I try to start up a
> VM. The system reboots.
Oooh, and is the log below from that situation?

Daniel, any thoughs?> 
> Below the log from /var/log/messages:
> 
> Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[4988]: Created
> /dev/xen/blktap-2/control device
> Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[4988]: Created
> /dev/xen/blktap-2/blktap0 device
> Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[4988]: Created
> /dev/xen/blktap-2/tapdev0 device
> Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[4988]: new interface: ring:
> 251, device: 253, minor: 0
> Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[4988]: I/O queue driver: lio
> Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[4988]: block-aio
open(''/storage5_nfs/3/CD996633-linux-centos-5-64b-base-rip-sx-7253/hda'')
> Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[4988]:
open(/storage5_nfs/3/CD996633-linux-centos-5-64b-base-rip-sx-7253/hda)
> with O_DIRECT
> Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[4988]: Image size:       pre
> sector_shift  [134217728]   post sector_shift [262144]
> Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[4988]: opened image
> /storage5_nfs/3/CD996633-linux-centos-5-64b-base-rip-sx-7253/hda (1
> users, state: 0x00000001, type: 0)
> Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[4988]: VBD CHAIN:
> Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[4988]:
> /storage5_nfs/3/CD996633-linux-centos-5-64b-base-rip-sx-7253/hda: 0
> Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.158549] block tda:
> sector-size: 512 capacity: 262144
> Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.200514] general
> protection fault: 0000 [#1] SMP DEBUG_PAGEALLOC
> Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.200703] last sysfs
> file: /sys/block/tda/removable
> Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.200761] CPU 0
> Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.200847] Modules linked
> in: bridge stp bonding bnx2i libiscsi scsi_transport_iscsi cnic uio
> bnx2 megaraid_sas
> Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.201363] Pid: 4988,
> comm: tapdisk2 Not tainted 2.6.32.36 #3 PowerEdge M610
> Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.201424] RIP:
> e030:[<ffffffff812b9c24>]  [<ffffffff812b9c24>]
> blktap_device_end_request+0x49/0x5e
> Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.201543] RSP:
> e02b:ffff88006a7f7cd8  EFLAGS: 00010046
> Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.201600] RAX:
> 6b6b6b6b6b6b6b6b RBX: ffff88006a6fc000 RCX: ffff88006a7f7c38
> Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.201662] RDX:
> 0000000000000000 RSI: 0000000000000000 RDI: ffff88006a5c3500
> Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.201723] RBP:
> ffff88006a7f7cf8 R08: ffffffff818383c0 R09: ffff88006a7f7c38
> Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.201784] R10:
> 0000000000000000 R11: ffff88007b697b18 R12: ffff88007b697b18
> Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.201845] R13:
> ffff88006a5c3360 R14: 0000000000000000 R15: ffff88006a5c3370
> Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.201910] FS:
> 00007f50a9445730(0000) GS:ffff8800280c7000(0000)
> knlGS:0000000000000000
> Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.201974] CS:  e033 DS:
> 0000 ES: 0000 CR0: 000000008005003b
> Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.202032] CR2:
> 00007fb35d12e6e8 CR3: 000000006a4ce000 CR4: 0000000000002660
> Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.202093] DR0:
> 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.202154] DR3:
> 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.202436] Process
> tapdisk2 (pid: 4988, threadinfo ffff88006a7f6000, task
> ffff88006b5a0000)
> Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.202941] Stack:
> Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.203206]
> ffff88006b5a0000 0000000000000000 0000000000000000 0000000000000000
> Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.203609] <0>
> ffff88006a7f7e88 ffffffff812b9416 ffff88006a6c80f8 0000000100000000
> Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.204310] <0>
> 00000000ffffffff ffff88006a5c3360 000000017edd7ab0 0000000000000000
> Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.205284] Call Trace:
> Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.205553]
> [<ffffffff812b9416>] blktap_ring_ioctl+0x183/0x2d8
> Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.205838]
> [<ffffffff81209a64>] ? inode_has_perm+0xa1/0xb3
> Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.206120]
> [<ffffffff8157641f>] ? _spin_unlock+0x26/0x2a
> Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.206400]
> [<ffffffff81126ff9>] ? aio_read_evt+0x56/0xe0
> Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.206678]
> [<ffffffff81127071>] ? aio_read_evt+0xce/0xe0
> Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.206957]
> [<ffffffff8124f5c1>] ? _raw_spin_lock+0x77/0x12d
> Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.207236]
> [<ffffffff81209bf8>] ? file_has_perm+0xb4/0xc6
> Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.207516]
> [<ffffffff8110464e>] vfs_ioctl+0x5e/0x77
> Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.207793]
> [<ffffffff81104b63>] do_vfs_ioctl+0x484/0x4d5
> Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.208069]
> [<ffffffff81104c0b>] sys_ioctl+0x57/0x7a
> Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.208346]
> [<ffffffff81012cc2>] system_call_fastpath+0x16/0x1b
> Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.208621] Code: 89 de 4c
> 89 ef e8 60 f4 ff ff 49 8b 44 24 40 48 8b b8 90 04 00 00 e8 41 c9 2b
> 00 44 89 f6 4c 89 e7 e8 39 fc ff ff 49 8b 44 24 40 <48> 8b b8 90 04
> 00 00 e8 66 c7 2b 00 5b 41 5c 41 5d 41 5e c9 c3
> Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.211986] RIP
> [<ffffffff812b9c24>] blktap_device_end_request+0x49/0x5e
> Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.212306]  RSP
<ffff88006a7f7cd8>
> Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.212579] ---[ end trace
> b97070122f44735d ]---
> Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[5009]: Created
> /dev/xen/blktap-2/blktap1 device
> Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[5009]: Created
> /dev/xen/blktap-2/tapdev1 device
> Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[5009]: new interface: ring:
> 251, device: 253, minor: 1
> Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[5009]: I/O queue driver: lio
> Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[5009]: block-aio
open(''/storage5_nfs/3/CD996633-linux-centos-5-64b-base-rip-sx-7253/xvda'')
> Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[5009]:
open(/storage5_nfs/3/CD996633-linux-centos-5-64b-base-rip-sx-7253/xvda)
> with O_DIRECT
> Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[5009]: Image size:       pre
> sector_shift  [10737418240]         post sector_shift [20971520]
> Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[5009]: opened image
> /storage5_nfs/3/CD996633-linux-centos-5-64b-base-rip-sx-7253/xvda (1
> users, state: 0x00000001, type: 0)
> Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[5009]: VBD CHAIN:
> Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[5009]:
> /storage5_nfs/3/CD996633-linux-centos-5-64b-base-rip-sx-7253/xvda: 0
> Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.317931] block tdb:
> sector-size: 512 capacity: 20971520
> 
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Daniel Stodden

2011-Apr-14 16:38 UTC

head link

Re: [Xen-devel] Tapdisk failures / kernel general protection fault at xen 4.0.2rc3 / kernel pvops 2.6.32.36

On Thu, 2011-04-14 at 09:15 -0400, Konrad Rzeszutek Wilk
wrote:> On Wed, Apr 13, 2011 at 06:02:13PM -0300, Gerd Jakobovitsch wrote:
> > I''m trying to run several VMs (linux hvm, with tapdisk:aio
disks at
> > a storage over nfs) on a CentOS system, using the up-to-date version
> > of xen 4.0 / kernel pvops 2.6.32.x stable. With a configuration
> > without (most of) debug activated, I can start several instances -
> > I''m running 7 of them - but shortly afterwards the system
stops
> > responding. I can''t find any information on this.
> 
> First time I see it.
> > 
> > Activating several debug configuration items, among them
> > DEBUG_PAGEALLOC, I get an exception as soon as I try to start up a
> > VM. The system reboots.
> 
> Oooh, and is the log below from that situation?
> 
> Daniel, any thoughs?
---
	  Unmap pages from the kernel linear mapping after free_pages().
	  This results in a large slowdown, but helps to find certain types
	  of memory corruption.

Stunning. Our I/O page allocator is a sort of twisted mempool. Unless
the allocation is explicitly modified in sysfs/, everything should stay
pinned. We might be just tripping over debug code alone, but I didn''t
figure it out yet.

Daniel
> > 
> > Below the log from /var/log/messages:
> > 
> > Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[4988]: Created
> > /dev/xen/blktap-2/control device
> > Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[4988]: Created
> > /dev/xen/blktap-2/blktap0 device
> > Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[4988]: Created
> > /dev/xen/blktap-2/tapdev0 device
> > Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[4988]: new interface: ring:
> > 251, device: 253, minor: 0
> > Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[4988]: I/O queue driver: lio
> > Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[4988]: block-aio
open(''/storage5_nfs/3/CD996633-linux-centos-5-64b-base-rip-sx-7253/hda'')
> > Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[4988]:
open(/storage5_nfs/3/CD996633-linux-centos-5-64b-base-rip-sx-7253/hda)
> > with O_DIRECT
> > Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[4988]: Image size:       pre
> > sector_shift  [134217728]   post sector_shift [262144]
> > Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[4988]: opened image
> > /storage5_nfs/3/CD996633-linux-centos-5-64b-base-rip-sx-7253/hda (1
> > users, state: 0x00000001, type: 0)
> > Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[4988]: VBD CHAIN:
> > Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[4988]:
> > /storage5_nfs/3/CD996633-linux-centos-5-64b-base-rip-sx-7253/hda: 0
> > Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.158549] block tda:
> > sector-size: 512 capacity: 262144
> > Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.200514] general
> > protection fault: 0000 [#1] SMP DEBUG_PAGEALLOC
> > Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.200703] last sysfs
> > file: /sys/block/tda/removable
> > Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.200761] CPU 0
> > Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.200847] Modules linked
> > in: bridge stp bonding bnx2i libiscsi scsi_transport_iscsi cnic uio
> > bnx2 megaraid_sas
> > Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.201363] Pid: 4988,
> > comm: tapdisk2 Not tainted 2.6.32.36 #3 PowerEdge M610
> > Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.201424] RIP:
> > e030:[<ffffffff812b9c24>]  [<ffffffff812b9c24>]
> > blktap_device_end_request+0x49/0x5e
> > Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.201543] RSP:
> > e02b:ffff88006a7f7cd8  EFLAGS: 00010046
> > Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.201600] RAX:
> > 6b6b6b6b6b6b6b6b RBX: ffff88006a6fc000 RCX: ffff88006a7f7c38
> > Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.201662] RDX:
> > 0000000000000000 RSI: 0000000000000000 RDI: ffff88006a5c3500
> > Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.201723] RBP:
> > ffff88006a7f7cf8 R08: ffffffff818383c0 R09: ffff88006a7f7c38
> > Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.201784] R10:
> > 0000000000000000 R11: ffff88007b697b18 R12: ffff88007b697b18
> > Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.201845] R13:
> > ffff88006a5c3360 R14: 0000000000000000 R15: ffff88006a5c3370
> > Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.201910] FS:
> > 00007f50a9445730(0000) GS:ffff8800280c7000(0000)
> > knlGS:0000000000000000
> > Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.201974] CS:  e033 DS:
> > 0000 ES: 0000 CR0: 000000008005003b
> > Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.202032] CR2:
> > 00007fb35d12e6e8 CR3: 000000006a4ce000 CR4: 0000000000002660
> > Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.202093] DR0:
> > 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> > Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.202154] DR3:
> > 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> > Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.202436] Process
> > tapdisk2 (pid: 4988, threadinfo ffff88006a7f6000, task
> > ffff88006b5a0000)
> > Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.202941] Stack:
> > Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.203206]
> > ffff88006b5a0000 0000000000000000 0000000000000000 0000000000000000
> > Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.203609] <0>
> > ffff88006a7f7e88 ffffffff812b9416 ffff88006a6c80f8 0000000100000000
> > Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.204310] <0>
> > 00000000ffffffff ffff88006a5c3360 000000017edd7ab0 0000000000000000
> > Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.205284] Call Trace:
> > Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.205553]
> > [<ffffffff812b9416>] blktap_ring_ioctl+0x183/0x2d8
> > Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.205838]
> > [<ffffffff81209a64>] ? inode_has_perm+0xa1/0xb3
> > Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.206120]
> > [<ffffffff8157641f>] ? _spin_unlock+0x26/0x2a
> > Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.206400]
> > [<ffffffff81126ff9>] ? aio_read_evt+0x56/0xe0
> > Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.206678]
> > [<ffffffff81127071>] ? aio_read_evt+0xce/0xe0
> > Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.206957]
> > [<ffffffff8124f5c1>] ? _raw_spin_lock+0x77/0x12d
> > Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.207236]
> > [<ffffffff81209bf8>] ? file_has_perm+0xb4/0xc6
> > Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.207516]
> > [<ffffffff8110464e>] vfs_ioctl+0x5e/0x77
> > Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.207793]
> > [<ffffffff81104b63>] do_vfs_ioctl+0x484/0x4d5
> > Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.208069]
> > [<ffffffff81104c0b>] sys_ioctl+0x57/0x7a
> > Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.208346]
> > [<ffffffff81012cc2>] system_call_fastpath+0x16/0x1b
> > Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.208621] Code: 89 de 4c
> > 89 ef e8 60 f4 ff ff 49 8b 44 24 40 48 8b b8 90 04 00 00 e8 41 c9 2b
> > 00 44 89 f6 4c 89 e7 e8 39 fc ff ff 49 8b 44 24 40 <48> 8b b8 90
04
> > 00 00 e8 66 c7 2b 00 5b 41 5c 41 5d 41 5e c9 c3
> > Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.211986] RIP
> > [<ffffffff812b9c24>] blktap_device_end_request+0x49/0x5e
> > Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.212306]  RSP
<ffff88006a7f7cd8>
> > Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.212579] ---[ end trace
> > b97070122f44735d ]---
> > Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[5009]: Created
> > /dev/xen/blktap-2/blktap1 device
> > Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[5009]: Created
> > /dev/xen/blktap-2/tapdev1 device
> > Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[5009]: new interface: ring:
> > 251, device: 253, minor: 1
> > Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[5009]: I/O queue driver: lio
> > Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[5009]: block-aio
open(''/storage5_nfs/3/CD996633-linux-centos-5-64b-base-rip-sx-7253/xvda'')
> > Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[5009]:
open(/storage5_nfs/3/CD996633-linux-centos-5-64b-base-rip-sx-7253/xvda)
> > with O_DIRECT
> > Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[5009]: Image size:       pre
> > sector_shift  [10737418240]         post sector_shift [20971520]
> > Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[5009]: opened image
> > /storage5_nfs/3/CD996633-linux-centos-5-64b-base-rip-sx-7253/xvda (1
> > users, state: 0x00000001, type: 0)
> > Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[5009]: VBD CHAIN:
> > Apr 13 17:47:23 r2b16ch2x28p2 tapdisk2[5009]:
> > /storage5_nfs/3/CD996633-linux-centos-5-64b-base-rip-sx-7253/xvda: 0
> > Apr 13 17:47:23 r2b16ch2x28p2 kernel: [  179.317931] block tdb:
> > sector-size: 512 capacity: 20971520
> > 
> > 
> > 
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xensource.com
> > http://lists.xensource.com/xen-devel


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Daniel Stodden

2011-Apr-14 17:32 UTC

head link

Re: [Xen-devel] Tapdisk failures / kernel general protection fault at xen 4.0.2rc3 / kernel pvops 2.6.32.36

On Thu, 2011-04-14 at 12:38 -0400, Daniel Stodden wrote:> On Thu, 2011-04-14 at 09:15 -0400, Konrad Rzeszutek Wilk wrote:
> > On Wed, Apr 13, 2011 at 06:02:13PM -0300, Gerd Jakobovitsch wrote:
> > > I''m trying to run several VMs (linux hvm, with
tapdisk:aio disks at
> > > a storage over nfs) on a CentOS system, using the up-to-date
version
> > > of xen 4.0 / kernel pvops 2.6.32.x stable. With a configuration
> > > without (most of) debug activated, I can start several instances
-
> > > I''m running 7 of them - but shortly afterwards the
system stops
> > > responding. I can''t find any information on this.
> > 
> > First time I see it.
> > > 
> > > Activating several debug configuration items, among them
> > > DEBUG_PAGEALLOC, I get an exception as soon as I try to start up
a
> > > VM. The system reboots.
> > 
> > Oooh, and is the log below from that situation?
> > 
> > Daniel, any thoughs?
> 
> ---
> 	  Unmap pages from the kernel linear mapping after free_pages().
> 	  This results in a large slowdown, but helps to find certain types
> 	  of memory corruption.
> 
> Stunning. Our I/O page allocator is a sort of twisted mempool. Unless
> the allocation is explicitly modified in sysfs/, everything should stay
> pinned. We might be just tripping over debug code alone, but I
didn''t
> figure it out yet.
Stuff worked fine with 2.6.32.32, but gets immediately reproducible
with .32.36.

Daniel


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Daniel Stodden

2011-Apr-14 17:43 UTC

head link

Re: [Xen-devel] Tapdisk failures / kernel general protection fault at xen 4.0.2rc3 / kernel pvops 2.6.32.36

On Thu, 2011-04-14 at 12:38 -0400, Daniel Stodden wrote:> On Thu, 2011-04-14 at 09:15 -0400, Konrad Rzeszutek Wilk wrote:
> > On Wed, Apr 13, 2011 at 06:02:13PM -0300, Gerd Jakobovitsch wrote:
> > > I''m trying to run several VMs (linux hvm, with
tapdisk:aio disks at
> > > a storage over nfs) on a CentOS system, using the up-to-date
version
> > > of xen 4.0 / kernel pvops 2.6.32.x stable. With a configuration
> > > without (most of) debug activated, I can start several instances
-
> > > I''m running 7 of them - but shortly afterwards the
system stops
> > > responding. I can''t find any information on this.
> > 
> > First time I see it.
> > > 
> > > Activating several debug configuration items, among them
> > > DEBUG_PAGEALLOC, I get an exception as soon as I try to start up
a
> > > VM. The system reboots.
> > 
> > Oooh, and is the log below from that situation?
> > 
> > Daniel, any thoughs?
> 
> ---
> 	  Unmap pages from the kernel linear mapping after free_pages().
> 	  This results in a large slowdown, but helps to find certain types
> 	  of memory corruption.
> 
> Stunning. Our I/O page allocator is a sort of twisted mempool. Unless
> the allocation is explicitly modified in sysfs/, everything should stay
> pinned. We might be just tripping over debug code alone, but I
didn''t
> figure it out yet.
Ah, that''s just missing Dominic''s spinlock fix. 

http://xenbits.xen.org/gitweb/?p=people/dstodden/linux.git;a=commit;h=a765257af7e28c41bd776c3e03615539597eb592

Daniel


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Gerd Jakobovitsch

2011-Apr-14 18:05 UTC

head link

Re: [Xen-devel] Tapdisk failures / kernel general protection fault at xen 4.0.2rc3 / kernel pvops 2.6.32.36

Hello Daniel:

I applied the patch and the bug at VM startup was solved. Thank you for 
your help.

Regards

Gerd

On 04/14/2011 02:43 PM, Daniel Stodden wrote:> On Thu, 2011-04-14 at 12:38 -0400, Daniel Stodden wrote:
>> On Thu, 2011-04-14 at 09:15 -0400, Konrad Rzeszutek Wilk wrote:
>>> On Wed, Apr 13, 2011 at 06:02:13PM -0300, Gerd Jakobovitsch wrote:
>>>> I''m trying to run several VMs (linux hvm, with
tapdisk:aio disks at
>>>> a storage over nfs) on a CentOS system, using the up-to-date
version
>>>> of xen 4.0 / kernel pvops 2.6.32.x stable. With a configuration
>>>> without (most of) debug activated, I can start several
instances -
>>>> I''m running 7 of them - but shortly afterwards the
system stops
>>>> responding. I can''t find any information on this.
>>> First time I see it.
>>>> Activating several debug configuration items, among them
>>>> DEBUG_PAGEALLOC, I get an exception as soon as I try to start
up a
>>>> VM. The system reboots.
>>> Oooh, and is the log below from that situation?
>>>
>>> Daniel, any thoughs?
>> ---
>> 	  Unmap pages from the kernel linear mapping after free_pages().
>> 	  This results in a large slowdown, but helps to find certain types
>> 	  of memory corruption.
>>
>> Stunning. Our I/O page allocator is a sort of twisted mempool. Unless
>> the allocation is explicitly modified in sysfs/, everything should stay
>> pinned. We might be just tripping over debug code alone, but I
didn''t
>> figure it out yet.
> Ah, that''s just missing Dominic''s spinlock fix.
>
>
http://xenbits.xen.org/gitweb/?p=people/dstodden/linux.git;a=commit;h=a765257af7e28c41bd776c3e03615539597eb592
>
> Daniel
>


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Daniel Stodden

2011-Apr-14 23:42 UTC

head link

Re: [Xen-devel] Tapdisk failures / kernel general protection fault at xen 4.0.2rc3 / kernel pvops 2.6.32.36

Jeremy,

it wants a manual patch into drivers/xen/blktap/device.c because it''s
rolled into that combo of renaming plus overall feature upgrades I have
in that series. 

It''s the only pending bugfix, quite an obvious one actually.
It''s been
rare enough unless provoked like Gerd did, but we found it first in XCP
so it actually tends to happen.

Can you just pick it, and push through to stable straight away?

On Thu, 2011-04-14 at 14:05 -0400, Gerd Jakobovitsch
wrote:> Hello Daniel:
> 
> I applied the patch and the bug at VM startup was solved. Thank you for 
> your help.
> 
> Regards
> 
> Gerd
> > Ah, that''s just missing Dominic''s spinlock fix.
> >
> >
http://xenbits.xen.org/gitweb/?p=people/dstodden/linux.git;a=commit;h=a765257af7e28c41bd776c3e03615539597eb592
Thanks very much.

Daniel

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Gerd Jakobovitsch

2011-Apr-15 14:21 UTC

head link

[Xen-devel] xen 4.0.2rc3/kernel 2.6.32.36: BUG: unable to handle kernel paging request

Reporting a new bug that appeared during stress tests. The scenario is 
the same as reported below, with patches applied:

On 04/14/2011 10:15 AM, Konrad Rzeszutek Wilk wrote:> On Wed, Apr 13, 2011 at 06:02:13PM -0300, Gerd Jakobovitsch wrote:
>> I''m trying to run several VMs (linux hvm, with tapdisk:aio
disks at
>> a storage over nfs) on a CentOS system, using the up-to-date version
>> of xen 4.0 / kernel pvops 2.6.32.x stable. With a configuration
>> without (most of) debug activated, I can start several instances -
>> I''m running 7 of them - but shortly afterwards the system
stops
>> responding. I can''t find any information on this.
> First time I see it.
>> Activating several debug configuration items, among them
>> DEBUG_PAGEALLOC, I get an exception as soon as I try to start up a
>> VM. The system reboots.
>>With the debug information still set, I''m running 42 VMs - mixed Linux 
(several distros) and Windows, most of them running benchmarks for CPU 
and disk usage. After roughly 15 hours, a bug message appeared at dmesg. 
It affected xm commands - it seems to be related to a specific VM - but 
xl commands still work. VMs are running.

# xm list
Error: (5, ''Input/output error, while reading 
/local/domain/33/console/vnc-port'')
Usage: xm list [options] [Domain, ...]

After killing the VM that reported error, xm commands are working again.

The BUG message at dmesg:

[66007.135552] BUG: unable to handle kernel paging request at 
ffff8800004ca458
[66007.135567] IP: [<ffffffff8100d4ae>] xen_set_pte+0x3e/0x4b
[66007.135580] PGD 1002067 PUD 1006067 PMD 2d78067 PTE 100000004ca025
[66007.135675] Oops: 0003 [#1] SMP DEBUG_PAGEALLOC
[66007.135686] last sysfs file: 
/sys/class/net/virtbr/bridge/topology_change_detected
[66007.135693] CPU 4
[66007.135698] Modules linked in: arptable_filter arp_tables bridge stp 
bonding bnx2i libiscsi scsi_transport_iscsi cnic uio bnx2 megaraid_sas
[66007.135729] Pid: 683, comm: pageattr-test Not tainted 2.6.32.36 #7 
PowerEdge M610
[66007.135735] RIP: e030:[<ffffffff8100d4ae>]  [<ffffffff8100d4ae>] 
xen_set_pte+0x3e/0x4b
[66007.135746] RSP: e02b:ffff88007c8edbb0  EFLAGS: 00010202
[66007.135751] RAX: 0000000000e32cb6 RBX: 0000000000e32cb6 RCX: 
0000000000000001
[66007.135757] RDX: 0000000000000000 RSI: 8010000800569267 RDI: 
ffff8800004ca458
[66007.135764] RBP: ffff88007c8edbd0 R08: 0000000000000001 R09: 
0000000000000000
[66007.135770] R10: ffffffff818385f8 R11: ffffffff818385e0 R12: 
8010000800569267
[66007.135776] R13: ffff8800004ca458 R14: 8010000416569067 R15: 
8010000800569267
[66007.135786] FS:  00007f0eeede66e0(0000) GS:ffff88002813f000(0000) 
knlGS:0000000000000000
[66007.135792] CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
[66007.135797] CR2: ffff8800004ca458 CR3: 000000007b663000 CR4: 
0000000000002660
[66007.135804] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 
0000000000000000
[66007.135810] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 
0000000000000400
[66007.135816] Process pageattr-test (pid: 683, threadinfo 
ffff88007c8ec000, task ffff88007e4ce480)
[66007.135822] Stack:
[66007.135825]  0000000000000000 8010000004569067 0000000000004569 
ffff88007c8edd20
[66007.135835] <0> ffff88007c8edbe0 ffffffff81034740 ffff88007c8edbf0 
ffffffff8103474d
[66007.135848] <0> ffff88007c8edcf0 ffffffff81034e77 000000017c8edc40 
ffffffff818385e0
[66007.135860] Call Trace:
[66007.135868]  [<ffffffff81034740>] set_pte+0x17/0x1b
[66007.135875]  [<ffffffff8103474d>] set_pte_atomic+0x9/0xb
[66007.135882]  [<ffffffff81034e77>]
__change_page_attr_set_clr+0x186/0x82d
[66007.135936]  [<ffffffff8124f4a0>] ? _raw_spin_unlock+0xab/0xb1
[66007.135951]  [<ffffffff8157641f>] ? _spin_unlock+0x26/0x2a
[66007.135961]  [<ffffffff810e587d>] ? vm_unmap_aliases+0x151/0x160
[66007.135969]  [<ffffffff81035695>] change_page_attr_set_clr+0x177/0x360
[66007.135976]  [<ffffffff8103597a>] change_page_attr_set+0x27/0x29
[66007.135983]  [<ffffffff810348e2>] ? pte_flags+0x9/0x18
[66007.135990]  [<ffffffff81035c01>] do_pageattr_test+0x285/0x4b1
[66007.135998]  [<ffffffff8103597c>] ? do_pageattr_test+0x0/0x4b1
[66007.136097]  [<ffffffff8106a9c3>] kthread+0x69/0x71
[66007.136105]  [<ffffffff81013daa>] child_rip+0xa/0x20
[66007.136112]  [<ffffffff81012ee6>] ? int_ret_from_sys_call+0x7/0x1b
[66007.136119]  [<ffffffff81013726>] ? retint_restore_args+0x5/0x6
[66007.136127]  [<ffffffff81013da0>] ? child_rip+0x0/0x20
[66007.136131] Code: e8 3c ff ff ff ff 05 b6 5c 94 00 e8 31 ff ff ff 8b 
1d b3 5c 94 00 e8 a2 23 02 00 ff c8 0f 94 c0 0f b6 c0 01 d8 89 05 9e 5c 
94 00 <4d> 89 65 00 41 59 5b 41 5c 41 5d c9 c3 55 48 89 e5 53 89 fb 48
[66007.136273] RIP  [<ffffffff8100d4ae>] xen_set_pte+0x3e/0x4b
[66007.136281]  RSP <ffff88007c8edbb0>
[66007.136285] CR2: ffff8800004ca458
[66007.136574] ---[ end trace 4e200a271895cc90 ]---

Attached errors registered in xm dmesg and xend.log.



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Pasi Kärkkäinen

2011-Apr-15 15:44 UTC

head link

Re: [Xen-devel] xen 4.0.2rc3/kernel 2.6.32.36: BUG: unable to handle kernel paging request

On Fri, Apr 15, 2011 at 11:21:40AM -0300, Gerd Jakobovitsch
wrote:> Reporting a new bug that appeared during stress tests. The scenario is  
> the same as reported below, with patches applied:
>
Hello,

Can you clarify which exact kernel version/changeset you used for this test?

Does your kernel include the latest bugfix (from 3 days ago)?
http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=commit;h=ae333e97552c81ab10395ad1ffc6d6daaadb144a

changelog:
http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=shortlog;h=xen/stable-2.6.32.x

-- Pasi

> On 04/14/2011 10:15 AM, Konrad Rzeszutek Wilk wrote:
>> On Wed, Apr 13, 2011 at 06:02:13PM -0300, Gerd Jakobovitsch wrote:
>>> I''m trying to run several VMs (linux hvm, with tapdisk:aio
disks at
>>> a storage over nfs) on a CentOS system, using the up-to-date
version
>>> of xen 4.0 / kernel pvops 2.6.32.x stable. With a configuration
>>> without (most of) debug activated, I can start several instances -
>>> I''m running 7 of them - but shortly afterwards the system
stops
>>> responding. I can''t find any information on this.
>> First time I see it.
>>> Activating several debug configuration items, among them
>>> DEBUG_PAGEALLOC, I get an exception as soon as I try to start up a
>>> VM. The system reboots.
>>>
> With the debug information still set, I''m running 42 VMs - mixed
Linux
> (several distros) and Windows, most of them running benchmarks for CPU  
> and disk usage. After roughly 15 hours, a bug message appeared at dmesg.  
> It affected xm commands - it seems to be related to a specific VM - but  
> xl commands still work. VMs are running.
>
> # xm list
> Error: (5, ''Input/output error, while reading  
> /local/domain/33/console/vnc-port'')
> Usage: xm list [options] [Domain, ...]
>
> After killing the VM that reported error, xm commands are working again.
>
> The BUG message at dmesg:
>
> [66007.135552] BUG: unable to handle kernel paging request at  
> ffff8800004ca458
> [66007.135567] IP: [<ffffffff8100d4ae>] xen_set_pte+0x3e/0x4b
> [66007.135580] PGD 1002067 PUD 1006067 PMD 2d78067 PTE 100000004ca025
> [66007.135675] Oops: 0003 [#1] SMP DEBUG_PAGEALLOC
> [66007.135686] last sysfs file:  
> /sys/class/net/virtbr/bridge/topology_change_detected
> [66007.135693] CPU 4
> [66007.135698] Modules linked in: arptable_filter arp_tables bridge stp  
> bonding bnx2i libiscsi scsi_transport_iscsi cnic uio bnx2 megaraid_sas
> [66007.135729] Pid: 683, comm: pageattr-test Not tainted 2.6.32.36 #7  
> PowerEdge M610
> [66007.135735] RIP: e030:[<ffffffff8100d4ae>] 
[<ffffffff8100d4ae>]
> xen_set_pte+0x3e/0x4b
> [66007.135746] RSP: e02b:ffff88007c8edbb0  EFLAGS: 00010202
> [66007.135751] RAX: 0000000000e32cb6 RBX: 0000000000e32cb6 RCX:  
> 0000000000000001
> [66007.135757] RDX: 0000000000000000 RSI: 8010000800569267 RDI:  
> ffff8800004ca458
> [66007.135764] RBP: ffff88007c8edbd0 R08: 0000000000000001 R09:  
> 0000000000000000
> [66007.135770] R10: ffffffff818385f8 R11: ffffffff818385e0 R12:  
> 8010000800569267
> [66007.135776] R13: ffff8800004ca458 R14: 8010000416569067 R15:  
> 8010000800569267
> [66007.135786] FS:  00007f0eeede66e0(0000) GS:ffff88002813f000(0000)  
> knlGS:0000000000000000
> [66007.135792] CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
> [66007.135797] CR2: ffff8800004ca458 CR3: 000000007b663000 CR4:  
> 0000000000002660
> [66007.135804] DR0: 0000000000000000 DR1: 0000000000000000 DR2:  
> 0000000000000000
> [66007.135810] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:  
> 0000000000000400
> [66007.135816] Process pageattr-test (pid: 683, threadinfo  
> ffff88007c8ec000, task ffff88007e4ce480)
> [66007.135822] Stack:
> [66007.135825]  0000000000000000 8010000004569067 0000000000004569  
> ffff88007c8edd20
> [66007.135835] <0> ffff88007c8edbe0 ffffffff81034740 ffff88007c8edbf0
> ffffffff8103474d
> [66007.135848] <0> ffff88007c8edcf0 ffffffff81034e77 000000017c8edc40
> ffffffff818385e0
> [66007.135860] Call Trace:
> [66007.135868]  [<ffffffff81034740>] set_pte+0x17/0x1b
> [66007.135875]  [<ffffffff8103474d>] set_pte_atomic+0x9/0xb
> [66007.135882]  [<ffffffff81034e77>]
__change_page_attr_set_clr+0x186/0x82d
> [66007.135936]  [<ffffffff8124f4a0>] ? _raw_spin_unlock+0xab/0xb1
> [66007.135951]  [<ffffffff8157641f>] ? _spin_unlock+0x26/0x2a
> [66007.135961]  [<ffffffff810e587d>] ? vm_unmap_aliases+0x151/0x160
> [66007.135969]  [<ffffffff81035695>]
change_page_attr_set_clr+0x177/0x360
> [66007.135976]  [<ffffffff8103597a>] change_page_attr_set+0x27/0x29
> [66007.135983]  [<ffffffff810348e2>] ? pte_flags+0x9/0x18
> [66007.135990]  [<ffffffff81035c01>] do_pageattr_test+0x285/0x4b1
> [66007.135998]  [<ffffffff8103597c>] ? do_pageattr_test+0x0/0x4b1
> [66007.136097]  [<ffffffff8106a9c3>] kthread+0x69/0x71
> [66007.136105]  [<ffffffff81013daa>] child_rip+0xa/0x20
> [66007.136112]  [<ffffffff81012ee6>] ? int_ret_from_sys_call+0x7/0x1b
> [66007.136119]  [<ffffffff81013726>] ? retint_restore_args+0x5/0x6
> [66007.136127]  [<ffffffff81013da0>] ? child_rip+0x0/0x20
> [66007.136131] Code: e8 3c ff ff ff ff 05 b6 5c 94 00 e8 31 ff ff ff 8b  
> 1d b3 5c 94 00 e8 a2 23 02 00 ff c8 0f 94 c0 0f b6 c0 01 d8 89 05 9e 5c  
> 94 00 <4d> 89 65 00 41 59 5b 41 5c 41 5d c9 c3 55 48 89 e5 53 89 fb
48
> [66007.136273] RIP  [<ffffffff8100d4ae>] xen_set_pte+0x3e/0x4b
> [66007.136281]  RSP <ffff88007c8edbb0>
> [66007.136285] CR2: ffff8800004ca458
> [66007.136574] ---[ end trace 4e200a271895cc90 ]---
>
> Attached errors registered in xm dmesg and xend.log.
> [2011-04-15 05:32:06 4415] ERROR (xmlrpclib2:181) Internal error handling
xend.domain
> Traceback (most recent call last):
>   File
"/usr/lib64/python2.4/site-packages/xen/util/xmlrpclib2.py", line 134,
in _marshaled_dispatch
>     response = self._dispatch(method, params)
>   File "/usr/lib64/python2.4/SimpleXMLRPCServer.py", line 406, in
_dispatch
>     return func(*params)
>   File
"/usr/lib64/python2.4/site-packages/xen/xend/server/XMLRPCServer.py",
line 58, in domain
>     info = lookup(domid)
>   File
"/usr/lib64/python2.4/site-packages/xen/xend/server/XMLRPCServer.py",
line 50, in lookup
>     info = XendDomain.instance().domain_lookup(domid)
>   File
"/usr/lib64/python2.4/site-packages/xen/xend/XendDomain.py", line 543,
in domain_lookup
>     self._refresh(refresh_shutdown = False)
>   File
"/usr/lib64/python2.4/site-packages/xen/xend/XendDomain.py", line 417,
in _refresh
>     self._refreshTxn(txn, refresh_shutdown)
>   File
"/usr/lib64/python2.4/site-packages/xen/xend/XendDomain.py", line 430,
in _refreshTxn
>     self.domains[domid].update(dom, refresh_shutdown, transaction)
>   File
"/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line
3523, in update
>     self._update_consoles(transaction)
>   File
"/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line
1656, in _update_consoles
>     vnc_port = self.readDomTxn(transaction,
''console/vnc-port'')
>   File
"/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line
1720, in readDomTxn
>     return transaction.read(*paths)
>   File
"/usr/lib64/python2.4/site-packages/xen/xend/xenstore/xstransact.py",
line 72, in read
>     return self._read(args[0])
>   File
"/usr/lib64/python2.4/site-packages/xen/xend/xenstore/xstransact.py",
line 59, in _read
>     ''%s, while reading %s'' % (ex.args[1], path))
> RuntimeError: (5, ''Input/output error, while reading
/local/domain/33/console/vnc-port'')
> [2011-04-15 05:32:12 4415] ERROR (xmlrpclib2:181) Internal error handling
xend.domain
> Traceback (most recent call last):
>   File
"/usr/lib64/python2.4/site-packages/xen/util/xmlrpclib2.py", line 134,
in _marshaled_dispatch
>     response = self._dispatch(method, params)
>   File "/usr/lib64/python2.4/SimpleXMLRPCServer.py", line 406, in
_dispatch
>     return func(*params)
>   File
"/usr/lib64/python2.4/site-packages/xen/xend/server/XMLRPCServer.py",
line 58, in domain
>     info = lookup(domid)
>   File
"/usr/lib64/python2.4/site-packages/xen/xend/server/XMLRPCServer.py",
line 50, in lookup
>     info = XendDomain.instance().domain_lookup(domid)
>   File
"/usr/lib64/python2.4/site-packages/xen/xend/XendDomain.py", line 543,
in domain_lookup
>     self._refresh(refresh_shutdown = False)
>   File
"/usr/lib64/python2.4/site-packages/xen/xend/XendDomain.py", line 417,
in _refresh
>     self._refreshTxn(txn, refresh_shutdown)
>   File
"/usr/lib64/python2.4/site-packages/xen/xend/XendDomain.py", line 430,
in _refreshTxn
>     self.domains[domid].update(dom, refresh_shutdown, transaction)
>   File
"/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line
3523, in update
>     self._update_consoles(transaction)
>   File
"/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line
1656, in _update_consoles
>     vnc_port = self.readDomTxn(transaction,
''console/vnc-port'')
>   File
"/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line
1720, in readDomTxn
>     return transaction.read(*paths)
>   File
"/usr/lib64/python2.4/site-packages/xen/xend/xenstore/xstransact.py",
line 72, in read
>     return self._read(args[0])
>   File
"/usr/lib64/python2.4/site-packages/xen/xend/xenstore/xstransact.py",
line 59, in _read
>     ''%s, while reading %s'' % (ex.args[1], path))
> RuntimeError: (5, ''Input/output error, while reading
/local/domain/33/console/vnc-port'')
> [2011-04-15 05:32:17 4415] ERROR (xmlrpclib2:181) Internal error handling
xend.domain
> Traceback (most recent call last):
>   File
"/usr/lib64/python2.4/site-packages/xen/util/xmlrpclib2.py", line 134,
in _marshaled_dispatch
>     response = self._dispatch(method, params)
>   File "/usr/lib64/python2.4/SimpleXMLRPCServer.py", line 406, in
_dispatch
>     return func(*params)
>   File
"/usr/lib64/python2.4/site-packages/xen/xend/server/XMLRPCServer.py",
line 58, in domain
>     info = lookup(domid)
>   File
"/usr/lib64/python2.4/site-packages/xen/xend/server/XMLRPCServer.py",
line 50, in lookup
>     info = XendDomain.instance().domain_lookup(domid)
>   File
"/usr/lib64/python2.4/site-packages/xen/xend/XendDomain.py", line 543,
in domain_lookup
>     self._refresh(refresh_shutdown = False)
>   File
"/usr/lib64/python2.4/site-packages/xen/xend/XendDomain.py", line 417,
in _refresh
>     self._refreshTxn(txn, refresh_shutdown)
>   File
"/usr/lib64/python2.4/site-packages/xen/xend/XendDomain.py", line 430,
in _refreshTxn
>     self.domains[domid].update(dom, refresh_shutdown, transaction)
>   File
"/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line
3523, in update
>     self._update_consoles(transaction)
>   File
"/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line
1656, in _update_consoles
>     vnc_port = self.readDomTxn(transaction,
''console/vnc-port'')
>   File
"/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line
1720, in readDomTxn
>     return transaction.read(*paths)
>   File
"/usr/lib64/python2.4/site-packages/xen/xend/xenstore/xstransact.py",
line 72, in read
>     return self._read(args[0])
>   File
"/usr/lib64/python2.4/site-packages/xen/xend/xenstore/xstransact.py",
line 59, in _read
>     ''%s, while reading %s'' % (ex.args[1], path))
> 
> (XEN) HVM43: Booting from 0000:7c00
> (XEN) HVM44: Press F12 for boot menu.
> (XEN) HVM44: 
> (XEN) HVM44: Booting from Hard Disk...
> (XEN) HVM44: Booting from 0000:7c00
> (XEN) HVM44: int13_harddisk: function 42. LBA out of range
> (XEN) stdvga.c:151:d43 leaving stdvga
> (XEN) stdvga.c:147:d43 entering stdvga and caching modes
> (XEN) irq.c:243: Dom43 PCI link 0 changed 5 -> 0
> (XEN) irq.c:243: Dom43 PCI link 1 changed 10 -> 0
> (XEN) irq.c:243: Dom43 PCI link 2 changed 11 -> 0
> (XEN) irq.c:243: Dom43 PCI link 3 changed 5 -> 0
> (XEN) stdvga.c:151:d43 leaving stdvga
> (XEN) stdvga.c:151:d42 leaving stdvga
> (XEN) mm.c:3617:d0 PTE entry 81244b for address ffff88007caa0000
doesn''t match frame 621c4b
> (XEN) mm.c:3617:d0 PTE entry 812bed for address ffff88007caa1000
doesn''t match frame 6217ed
> (XEN) mm.c:3617:d0 PTE entry 812426 for address ffff88007caa2000
doesn''t match frame 621c26
> (XEN) mm.c:3617:d0 PTE entry 812590 for address ffff88007caa3000
doesn''t match frame 621d90
> (XEN) mm.c:3617:d0 PTE entry 812591 for address ffff88007caa4000
doesn''t match frame 621d91
> (XEN) mm.c:3617:d0 PTE entry 812492 for address ffff88007caa5000
doesn''t match frame 621c92
> (XEN) mm.c:3617:d0 PTE entry 812493 for address ffff88007caa6000
doesn''t match frame 621c93
> (XEN) mm.c:3617:d0 PTE entry 8124ca for address ffff88007caa7000
doesn''t match frame 621cca
> (XEN) mm.c:3617:d0 PTE entry 8124cb for address ffff88007caa8000
doesn''t match frame 621ccb
> (XEN) mm.c:3617:d0 PTE entry 80e8b0 for address ffff88007caa9000
doesn''t match frame 6258b0
> (XEN) mm.c:3617:d0 PTE entry 80e8b1 for address ffff88007caaa000
doesn''t match frame 6258b1
> (XEN) mm.c:3617:d0 PTE entry 803fe2 for address ffff88007caab000
doesn''t match frame 5cfde2
> (XEN) mm.c:3617:d0 PTE entry 80b3fd for address ffff88007ca99000
doesn''t match frame 46b1fd
> (XEN) mm.c:3617:d0 PTE entry 80b3fe for address ffff88007ca9a000
doesn''t match frame 46b1fe
> (XEN) mm.c:3617:d0 PTE entry 80ac10 for address ffff88007ca9b000
doesn''t match frame 46b610
> (XEN) mm.c:3617:d0 PTE entry 80c01e for address ffff88007ca9c000
doesn''t match frame 46a21e
> (XEN) mm.c:3617:d0 PTE entry 80ab16 for address ffff88007ca9d000
doesn''t match frame 46b916
> (XEN) mm.c:3617:d0 PTE entry 80b0f6 for address ffff88007ca9e000
doesn''t match frame 46b2f6
> (XEN) mm.c:3617:d0 PTE entry 809c8c for address ffff88007ca9f000
doesn''t match frame 46c68c
> (XEN) mm.c:3617:d0 PTE entry 8017e0 for address ffff88007cb9f000
doesn''t match frame 5b19e0
> (XEN) mm.c:3617:d0 PTE entry 8017e1 for address ffff88007cba0000
doesn''t match frame 5b19e1
> (XEN) mm.c:3617:d0 PTE entry 801fc4 for address ffff88007cba1000
doesn''t match frame 5b11c4
> (XEN) mm.c:3617:d0 PTE entry 801fc5 for address ffff88007cba2000
doesn''t match frame 5b11c5
> (XEN) mm.c:3617:d0 PTE entry 801f2a for address ffff88007cba3000
doesn''t match frame 5b112a
> (XEN) mm.c:3617:d0 PTE entry 801f2b for address ffff88007cba4000
doesn''t match frame 5b112b
> (XEN) mm.c:3617:d0 PTE entry 801ee8 for address ffff88007cba5000
doesn''t match frame 5b10e8
> (XEN) mm.c:3617:d0 PTE entry 801ee9 for address ffff88007cba6000
doesn''t match frame 5b10e9
> (XEN) mm.c:3617:d0 PTE entry 803fde for address ffff88007cba7000
doesn''t match frame 5af1de
> (XEN) mm.c:3617:d0 PTE entry 8120ec for address ffff88007cb9e000
doesn''t match frame 9660ec
> (XEN) mm.c:3617:d0 PTE entry 8120ed for address ffff88007cb9f000
doesn''t match frame 9660ed
> (XEN) mm.c:3617:d0 PTE entry 810e0a for address ffff88007cba0000
doesn''t match frame 96720a
> (XEN) mm.c:3617:d0 PTE entry 810e0b for address ffff88007cba1000
doesn''t match frame 96720b
> (XEN) mm.c:3617:d0 PTE entry 81080e for address ffff88007cba2000
doesn''t match frame 96780e
> (XEN) mm.c:3617:d0 PTE entry 81080f for address ffff88007cba3000
doesn''t match frame 96780f
> (XEN) mm.c:3617:d0 PTE entry 80ef3e for address ffff88007cba4000
doesn''t match frame 96933e
> (XEN) mm.c:3617:d0 PTE entry 802fd7 for address ffff88007c433000
doesn''t match frame 2111d7
> (XEN) mm.c:3617:d0 PTE entry 805131 for address ffff88007c434000
doesn''t match frame 20ef31
> (XEN) mm.c:3617:d0 PTE entry 80c1ea for address ffff88007c435000
doesn''t match frame 207fea
> (XEN) mm.c:3617:d0 PTE entry 814ae9 for address ffff88007c9b0000
doesn''t match frame 8866e9
> (XEN) mm.c:3617:d0 PTE entry 8113b9 for address ffff88007ca20000
doesn''t match frame 9277b9
> (XEN) mm.c:3617:d0 PTE entry 808ab2 for address ffff88007ca21000
doesn''t match frame 92feb2
> (XEN) mm.c:3617:d0 PTE entry 803bb3 for address ffff88007ca22000
doesn''t match frame 934fb3
> (XEN) mm.c:3617:d0 PTE entry 814747 for address ffff88007ca23000
doesn''t match frame 924347
> (XEN) mm.c:3617:d0 PTE entry 819873 for address ffff88007ca24000
doesn''t match frame 91f073
> (XEN) mm.c:3617:d0 PTE entry 80302c for address ffff88007ca25000
doesn''t match frame 93582c
> (XEN) mm.c:3617:d0 PTE entry 81b75c for address ffff88007ca26000
doesn''t match frame 91d35c
> (XEN) mm.c:3617:d0 PTE entry c5fd0b for address ffff88007ca86000
doesn''t match frame 95830b
> (XEN) mm.c:3617:d0 PTE entry 81a093 for address ffff88007ca87000
doesn''t match frame 9bde93
> (XEN) mm.c:3617:d0 PTE entry 815f11 for address ffff88007ca88000
doesn''t match frame 942111
> (XEN) mm.c:3617:d0 PTE entry 815bc5 for address ffff88007ca89000
doesn''t match frame 9425c5
> (XEN) mm.c:3617:d0 PTE entry 816875 for address ffff88007cb9b000
doesn''t match frame 59c675
> (XEN) mm.c:3617:d0 PTE entry 8158a8 for address ffff88007cb9c000
doesn''t match frame 59d6a8
> (XEN) mm.c:3617:d0 PTE entry 82fa1c for address ffff88007c926000
doesn''t match frame 5e321c
> (XEN) mm.c:3617:d0 PTE entry 82655e for address ffff88007c927000
doesn''t match frame 5ec95e
> (XEN) mm.c:3617:d0 PTE entry c5ce72 for address ffff88007c926000
doesn''t match frame 95b072
> (XEN) mm.c:3617:d0 PTE entry c5cea9 for address ffff88007c927000
doesn''t match frame 95b0a9
> (XEN) mm.c:2399:d0 Bad type (saw 1c00000000000002 != exp 7000000000000000)
for mfn 800569 (pfn 4569)
> (XEN) mm.c:878:d0 Error getting mfn 800569 (pfn 4569) from L1 entry
8010000800569267 for l1e_owner=0, pg_owner=0
> (XEN) mm.c:4664:d0 ptwr_emulate: could not get_page_from_l1e()
> (XEN) mm.c:3617:d0 PTE entry 80e284 for address ffff88007ca89000
doesn''t match frame 468084
> (XEN) mm.c:3617:d0 PTE entry 817918 for address ffff88007ca8a000
doesn''t match frame 47f118
> (XEN) common.c:3722:d0 tracking VRAM f0000 - f0160
> (XEN) common.c:3722:d0 tracking VRAM f0000 - f0160
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Gerd Jakobovitsch

2011-Apr-15 15:47 UTC

head link

Re: [Xen-devel] xen 4.0.2rc3/kernel 2.6.32.36: BUG: unable to handle kernel paging request

Hello Pasi:

Yes, the kernel includes the last bugfix, and additionally the patch 
Daniel Stodden indicated yesterday:

http://xenbits.xen.org/gitweb/?p=people/dstodden/linux.git;a=commit;h=a765257af7e28c41bd776c3e03615539597eb592



On 04/15/2011 12:44 PM, Pasi Kärkkäinen wrote:> On Fri, Apr 15, 2011 at 11:21:40AM -0300, Gerd Jakobovitsch wrote:
>> Reporting a new bug that appeared during stress tests. The scenario is
>> the same as reported below, with patches applied:
>>
> Hello,
>
> Can you clarify which exact kernel version/changeset you used for this
test?
>
> Does your kernel include the latest bugfix (from 3 days ago)?
>
http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=commit;h=ae333e97552c81ab10395ad1ffc6d6daaadb144a
>
> changelog:
>
http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=shortlog;h=xen/stable-2.6.32.x
>
> -- Pasi
>
>
>> On 04/14/2011 10:15 AM, Konrad Rzeszutek Wilk wrote:
>>> On Wed, Apr 13, 2011 at 06:02:13PM -0300, Gerd Jakobovitsch wrote:
>>>> I''m trying to run several VMs (linux hvm, with
tapdisk:aio disks at
>>>> a storage over nfs) on a CentOS system, using the up-to-date
version
>>>> of xen 4.0 / kernel pvops 2.6.32.x stable. With a configuration
>>>> without (most of) debug activated, I can start several
instances -
>>>> I''m running 7 of them - but shortly afterwards the
system stops
>>>> responding. I can''t find any information on this.
>>> First time I see it.
>>>> Activating several debug configuration items, among them
>>>> DEBUG_PAGEALLOC, I get an exception as soon as I try to start
up a
>>>> VM. The system reboots.
>>>>
>> With the debug information still set, I''m running 42 VMs -
mixed Linux
>> (several distros) and Windows, most of them running benchmarks for CPU
>> and disk usage. After roughly 15 hours, a bug message appeared at
dmesg.
>> It affected xm commands - it seems to be related to a specific VM - but
>> xl commands still work. VMs are running.
>>
>> # xm list
>> Error: (5, ''Input/output error, while reading
>> /local/domain/33/console/vnc-port'')
>> Usage: xm list [options] [Domain, ...]
>>
>> After killing the VM that reported error, xm commands are working
again.
>>
>> The BUG message at dmesg:
>>
>> [66007.135552] BUG: unable to handle kernel paging request at
>> ffff8800004ca458
>> [66007.135567] IP: [<ffffffff8100d4ae>] xen_set_pte+0x3e/0x4b
>> [66007.135580] PGD 1002067 PUD 1006067 PMD 2d78067 PTE 100000004ca025
>> [66007.135675] Oops: 0003 [#1] SMP DEBUG_PAGEALLOC
>> [66007.135686] last sysfs file:
>> /sys/class/net/virtbr/bridge/topology_change_detected
>> [66007.135693] CPU 4
>> [66007.135698] Modules linked in: arptable_filter arp_tables bridge stp
>> bonding bnx2i libiscsi scsi_transport_iscsi cnic uio bnx2 megaraid_sas
>> [66007.135729] Pid: 683, comm: pageattr-test Not tainted 2.6.32.36 #7
>> PowerEdge M610
>> [66007.135735] RIP: e030:[<ffffffff8100d4ae>] 
[<ffffffff8100d4ae>]
>> xen_set_pte+0x3e/0x4b
>> [66007.135746] RSP: e02b:ffff88007c8edbb0  EFLAGS: 00010202
>> [66007.135751] RAX: 0000000000e32cb6 RBX: 0000000000e32cb6 RCX:
>> 0000000000000001
>> [66007.135757] RDX: 0000000000000000 RSI: 8010000800569267 RDI:
>> ffff8800004ca458
>> [66007.135764] RBP: ffff88007c8edbd0 R08: 0000000000000001 R09:
>> 0000000000000000
>> [66007.135770] R10: ffffffff818385f8 R11: ffffffff818385e0 R12:
>> 8010000800569267
>> [66007.135776] R13: ffff8800004ca458 R14: 8010000416569067 R15:
>> 8010000800569267
>> [66007.135786] FS:  00007f0eeede66e0(0000) GS:ffff88002813f000(0000)
>> knlGS:0000000000000000
>> [66007.135792] CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
>> [66007.135797] CR2: ffff8800004ca458 CR3: 000000007b663000 CR4:
>> 0000000000002660
>> [66007.135804] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
>> 0000000000000000
>> [66007.135810] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
>> 0000000000000400
>> [66007.135816] Process pageattr-test (pid: 683, threadinfo
>> ffff88007c8ec000, task ffff88007e4ce480)
>> [66007.135822] Stack:
>> [66007.135825]  0000000000000000 8010000004569067 0000000000004569
>> ffff88007c8edd20
>> [66007.135835]<0>  ffff88007c8edbe0 ffffffff81034740
ffff88007c8edbf0
>> ffffffff8103474d
>> [66007.135848]<0>  ffff88007c8edcf0 ffffffff81034e77
000000017c8edc40
>> ffffffff818385e0
>> [66007.135860] Call Trace:
>> [66007.135868]  [<ffffffff81034740>] set_pte+0x17/0x1b
>> [66007.135875]  [<ffffffff8103474d>] set_pte_atomic+0x9/0xb
>> [66007.135882]  [<ffffffff81034e77>]
__change_page_attr_set_clr+0x186/0x82d
>> [66007.135936]  [<ffffffff8124f4a0>] ? _raw_spin_unlock+0xab/0xb1
>> [66007.135951]  [<ffffffff8157641f>] ? _spin_unlock+0x26/0x2a
>> [66007.135961]  [<ffffffff810e587d>] ?
vm_unmap_aliases+0x151/0x160
>> [66007.135969]  [<ffffffff81035695>]
change_page_attr_set_clr+0x177/0x360
>> [66007.135976]  [<ffffffff8103597a>]
change_page_attr_set+0x27/0x29
>> [66007.135983]  [<ffffffff810348e2>] ? pte_flags+0x9/0x18
>> [66007.135990]  [<ffffffff81035c01>] do_pageattr_test+0x285/0x4b1
>> [66007.135998]  [<ffffffff8103597c>] ? do_pageattr_test+0x0/0x4b1
>> [66007.136097]  [<ffffffff8106a9c3>] kthread+0x69/0x71
>> [66007.136105]  [<ffffffff81013daa>] child_rip+0xa/0x20
>> [66007.136112]  [<ffffffff81012ee6>] ?
int_ret_from_sys_call+0x7/0x1b
>> [66007.136119]  [<ffffffff81013726>] ?
retint_restore_args+0x5/0x6
>> [66007.136127]  [<ffffffff81013da0>] ? child_rip+0x0/0x20
>> [66007.136131] Code: e8 3c ff ff ff ff 05 b6 5c 94 00 e8 31 ff ff ff 8b
>> 1d b3 5c 94 00 e8 a2 23 02 00 ff c8 0f 94 c0 0f b6 c0 01 d8 89 05 9e 5c
>> 94 00<4d>  89 65 00 41 59 5b 41 5c 41 5d c9 c3 55 48 89 e5 53 89
fb 48
>> [66007.136273] RIP  [<ffffffff8100d4ae>] xen_set_pte+0x3e/0x4b
>> [66007.136281]  RSP<ffff88007c8edbb0>
>> [66007.136285] CR2: ffff8800004ca458
>> [66007.136574] ---[ end trace 4e200a271895cc90 ]---
>>
>> Attached errors registered in xm dmesg and xend.log.
>> [2011-04-15 05:32:06 4415] ERROR (xmlrpclib2:181) Internal error
handling xend.domain
>> Traceback (most recent call last):
>>    File
"/usr/lib64/python2.4/site-packages/xen/util/xmlrpclib2.py", line 134,
in _marshaled_dispatch
>>      response = self._dispatch(method, params)
>>    File "/usr/lib64/python2.4/SimpleXMLRPCServer.py", line
406, in _dispatch
>>      return func(*params)
>>    File
"/usr/lib64/python2.4/site-packages/xen/xend/server/XMLRPCServer.py",
line 58, in domain
>>      info = lookup(domid)
>>    File
"/usr/lib64/python2.4/site-packages/xen/xend/server/XMLRPCServer.py",
line 50, in lookup
>>      info = XendDomain.instance().domain_lookup(domid)
>>    File
"/usr/lib64/python2.4/site-packages/xen/xend/XendDomain.py", line 543,
in domain_lookup
>>      self._refresh(refresh_shutdown = False)
>>    File
"/usr/lib64/python2.4/site-packages/xen/xend/XendDomain.py", line 417,
in _refresh
>>      self._refreshTxn(txn, refresh_shutdown)
>>    File
"/usr/lib64/python2.4/site-packages/xen/xend/XendDomain.py", line 430,
in _refreshTxn
>>      self.domains[domid].update(dom, refresh_shutdown, transaction)
>>    File
"/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line
3523, in update
>>      self._update_consoles(transaction)
>>    File
"/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line
1656, in _update_consoles
>>      vnc_port = self.readDomTxn(transaction,
''console/vnc-port'')
>>    File
"/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line
1720, in readDomTxn
>>      return transaction.read(*paths)
>>    File
"/usr/lib64/python2.4/site-packages/xen/xend/xenstore/xstransact.py",
line 72, in read
>>      return self._read(args[0])
>>    File
"/usr/lib64/python2.4/site-packages/xen/xend/xenstore/xstransact.py",
line 59, in _read
>>      ''%s, while reading %s'' % (ex.args[1], path))
>> RuntimeError: (5, ''Input/output error, while reading
/local/domain/33/console/vnc-port'')
>> [2011-04-15 05:32:12 4415] ERROR (xmlrpclib2:181) Internal error
handling xend.domain
>> Traceback (most recent call last):
>>    File
"/usr/lib64/python2.4/site-packages/xen/util/xmlrpclib2.py", line 134,
in _marshaled_dispatch
>>      response = self._dispatch(method, params)
>>    File "/usr/lib64/python2.4/SimpleXMLRPCServer.py", line
406, in _dispatch
>>      return func(*params)
>>    File
"/usr/lib64/python2.4/site-packages/xen/xend/server/XMLRPCServer.py",
line 58, in domain
>>      info = lookup(domid)
>>    File
"/usr/lib64/python2.4/site-packages/xen/xend/server/XMLRPCServer.py",
line 50, in lookup
>>      info = XendDomain.instance().domain_lookup(domid)
>>    File
"/usr/lib64/python2.4/site-packages/xen/xend/XendDomain.py", line 543,
in domain_lookup
>>      self._refresh(refresh_shutdown = False)
>>    File
"/usr/lib64/python2.4/site-packages/xen/xend/XendDomain.py", line 417,
in _refresh
>>      self._refreshTxn(txn, refresh_shutdown)
>>    File
"/usr/lib64/python2.4/site-packages/xen/xend/XendDomain.py", line 430,
in _refreshTxn
>>      self.domains[domid].update(dom, refresh_shutdown, transaction)
>>    File
"/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line
3523, in update
>>      self._update_consoles(transaction)
>>    File
"/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line
1656, in _update_consoles
>>      vnc_port = self.readDomTxn(transaction,
''console/vnc-port'')
>>    File
"/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line
1720, in readDomTxn
>>      return transaction.read(*paths)
>>    File
"/usr/lib64/python2.4/site-packages/xen/xend/xenstore/xstransact.py",
line 72, in read
>>      return self._read(args[0])
>>    File
"/usr/lib64/python2.4/site-packages/xen/xend/xenstore/xstransact.py",
line 59, in _read
>>      ''%s, while reading %s'' % (ex.args[1], path))
>> RuntimeError: (5, ''Input/output error, while reading
/local/domain/33/console/vnc-port'')
>> [2011-04-15 05:32:17 4415] ERROR (xmlrpclib2:181) Internal error
handling xend.domain
>> Traceback (most recent call last):
>>    File
"/usr/lib64/python2.4/site-packages/xen/util/xmlrpclib2.py", line 134,
in _marshaled_dispatch
>>      response = self._dispatch(method, params)
>>    File "/usr/lib64/python2.4/SimpleXMLRPCServer.py", line
406, in _dispatch
>>      return func(*params)
>>    File
"/usr/lib64/python2.4/site-packages/xen/xend/server/XMLRPCServer.py",
line 58, in domain
>>      info = lookup(domid)
>>    File
"/usr/lib64/python2.4/site-packages/xen/xend/server/XMLRPCServer.py",
line 50, in lookup
>>      info = XendDomain.instance().domain_lookup(domid)
>>    File
"/usr/lib64/python2.4/site-packages/xen/xend/XendDomain.py", line 543,
in domain_lookup
>>      self._refresh(refresh_shutdown = False)
>>    File
"/usr/lib64/python2.4/site-packages/xen/xend/XendDomain.py", line 417,
in _refresh
>>      self._refreshTxn(txn, refresh_shutdown)
>>    File
"/usr/lib64/python2.4/site-packages/xen/xend/XendDomain.py", line 430,
in _refreshTxn
>>      self.domains[domid].update(dom, refresh_shutdown, transaction)
>>    File
"/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line
3523, in update
>>      self._update_consoles(transaction)
>>    File
"/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line
1656, in _update_consoles
>>      vnc_port = self.readDomTxn(transaction,
''console/vnc-port'')
>>    File
"/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line
1720, in readDomTxn
>>      return transaction.read(*paths)
>>    File
"/usr/lib64/python2.4/site-packages/xen/xend/xenstore/xstransact.py",
line 72, in read
>>      return self._read(args[0])
>>    File
"/usr/lib64/python2.4/site-packages/xen/xend/xenstore/xstransact.py",
line 59, in _read
>>      ''%s, while reading %s'' % (ex.args[1], path))
>>
>> (XEN) HVM43: Booting from 0000:7c00
>> (XEN) HVM44: Press F12 for boot menu.
>> (XEN) HVM44:
>> (XEN) HVM44: Booting from Hard Disk...
>> (XEN) HVM44: Booting from 0000:7c00
>> (XEN) HVM44: int13_harddisk: function 42. LBA out of range
>> (XEN) stdvga.c:151:d43 leaving stdvga
>> (XEN) stdvga.c:147:d43 entering stdvga and caching modes
>> (XEN) irq.c:243: Dom43 PCI link 0 changed 5 ->  0
>> (XEN) irq.c:243: Dom43 PCI link 1 changed 10 ->  0
>> (XEN) irq.c:243: Dom43 PCI link 2 changed 11 ->  0
>> (XEN) irq.c:243: Dom43 PCI link 3 changed 5 ->  0
>> (XEN) stdvga.c:151:d43 leaving stdvga
>> (XEN) stdvga.c:151:d42 leaving stdvga
>> (XEN) mm.c:3617:d0 PTE entry 81244b for address ffff88007caa0000
doesn''t match frame 621c4b
>> (XEN) mm.c:3617:d0 PTE entry 812bed for address ffff88007caa1000
doesn''t match frame 6217ed
>> (XEN) mm.c:3617:d0 PTE entry 812426 for address ffff88007caa2000
doesn''t match frame 621c26
>> (XEN) mm.c:3617:d0 PTE entry 812590 for address ffff88007caa3000
doesn''t match frame 621d90
>> (XEN) mm.c:3617:d0 PTE entry 812591 for address ffff88007caa4000
doesn''t match frame 621d91
>> (XEN) mm.c:3617:d0 PTE entry 812492 for address ffff88007caa5000
doesn''t match frame 621c92
>> (XEN) mm.c:3617:d0 PTE entry 812493 for address ffff88007caa6000
doesn''t match frame 621c93
>> (XEN) mm.c:3617:d0 PTE entry 8124ca for address ffff88007caa7000
doesn''t match frame 621cca
>> (XEN) mm.c:3617:d0 PTE entry 8124cb for address ffff88007caa8000
doesn''t match frame 621ccb
>> (XEN) mm.c:3617:d0 PTE entry 80e8b0 for address ffff88007caa9000
doesn''t match frame 6258b0
>> (XEN) mm.c:3617:d0 PTE entry 80e8b1 for address ffff88007caaa000
doesn''t match frame 6258b1
>> (XEN) mm.c:3617:d0 PTE entry 803fe2 for address ffff88007caab000
doesn''t match frame 5cfde2
>> (XEN) mm.c:3617:d0 PTE entry 80b3fd for address ffff88007ca99000
doesn''t match frame 46b1fd
>> (XEN) mm.c:3617:d0 PTE entry 80b3fe for address ffff88007ca9a000
doesn''t match frame 46b1fe
>> (XEN) mm.c:3617:d0 PTE entry 80ac10 for address ffff88007ca9b000
doesn''t match frame 46b610
>> (XEN) mm.c:3617:d0 PTE entry 80c01e for address ffff88007ca9c000
doesn''t match frame 46a21e
>> (XEN) mm.c:3617:d0 PTE entry 80ab16 for address ffff88007ca9d000
doesn''t match frame 46b916
>> (XEN) mm.c:3617:d0 PTE entry 80b0f6 for address ffff88007ca9e000
doesn''t match frame 46b2f6
>> (XEN) mm.c:3617:d0 PTE entry 809c8c for address ffff88007ca9f000
doesn''t match frame 46c68c
>> (XEN) mm.c:3617:d0 PTE entry 8017e0 for address ffff88007cb9f000
doesn''t match frame 5b19e0
>> (XEN) mm.c:3617:d0 PTE entry 8017e1 for address ffff88007cba0000
doesn''t match frame 5b19e1
>> (XEN) mm.c:3617:d0 PTE entry 801fc4 for address ffff88007cba1000
doesn''t match frame 5b11c4
>> (XEN) mm.c:3617:d0 PTE entry 801fc5 for address ffff88007cba2000
doesn''t match frame 5b11c5
>> (XEN) mm.c:3617:d0 PTE entry 801f2a for address ffff88007cba3000
doesn''t match frame 5b112a
>> (XEN) mm.c:3617:d0 PTE entry 801f2b for address ffff88007cba4000
doesn''t match frame 5b112b
>> (XEN) mm.c:3617:d0 PTE entry 801ee8 for address ffff88007cba5000
doesn''t match frame 5b10e8
>> (XEN) mm.c:3617:d0 PTE entry 801ee9 for address ffff88007cba6000
doesn''t match frame 5b10e9
>> (XEN) mm.c:3617:d0 PTE entry 803fde for address ffff88007cba7000
doesn''t match frame 5af1de
>> (XEN) mm.c:3617:d0 PTE entry 8120ec for address ffff88007cb9e000
doesn''t match frame 9660ec
>> (XEN) mm.c:3617:d0 PTE entry 8120ed for address ffff88007cb9f000
doesn''t match frame 9660ed
>> (XEN) mm.c:3617:d0 PTE entry 810e0a for address ffff88007cba0000
doesn''t match frame 96720a
>> (XEN) mm.c:3617:d0 PTE entry 810e0b for address ffff88007cba1000
doesn''t match frame 96720b
>> (XEN) mm.c:3617:d0 PTE entry 81080e for address ffff88007cba2000
doesn''t match frame 96780e
>> (XEN) mm.c:3617:d0 PTE entry 81080f for address ffff88007cba3000
doesn''t match frame 96780f
>> (XEN) mm.c:3617:d0 PTE entry 80ef3e for address ffff88007cba4000
doesn''t match frame 96933e
>> (XEN) mm.c:3617:d0 PTE entry 802fd7 for address ffff88007c433000
doesn''t match frame 2111d7
>> (XEN) mm.c:3617:d0 PTE entry 805131 for address ffff88007c434000
doesn''t match frame 20ef31
>> (XEN) mm.c:3617:d0 PTE entry 80c1ea for address ffff88007c435000
doesn''t match frame 207fea
>> (XEN) mm.c:3617:d0 PTE entry 814ae9 for address ffff88007c9b0000
doesn''t match frame 8866e9
>> (XEN) mm.c:3617:d0 PTE entry 8113b9 for address ffff88007ca20000
doesn''t match frame 9277b9
>> (XEN) mm.c:3617:d0 PTE entry 808ab2 for address ffff88007ca21000
doesn''t match frame 92feb2
>> (XEN) mm.c:3617:d0 PTE entry 803bb3 for address ffff88007ca22000
doesn''t match frame 934fb3
>> (XEN) mm.c:3617:d0 PTE entry 814747 for address ffff88007ca23000
doesn''t match frame 924347
>> (XEN) mm.c:3617:d0 PTE entry 819873 for address ffff88007ca24000
doesn''t match frame 91f073
>> (XEN) mm.c:3617:d0 PTE entry 80302c for address ffff88007ca25000
doesn''t match frame 93582c
>> (XEN) mm.c:3617:d0 PTE entry 81b75c for address ffff88007ca26000
doesn''t match frame 91d35c
>> (XEN) mm.c:3617:d0 PTE entry c5fd0b for address ffff88007ca86000
doesn''t match frame 95830b
>> (XEN) mm.c:3617:d0 PTE entry 81a093 for address ffff88007ca87000
doesn''t match frame 9bde93
>> (XEN) mm.c:3617:d0 PTE entry 815f11 for address ffff88007ca88000
doesn''t match frame 942111
>> (XEN) mm.c:3617:d0 PTE entry 815bc5 for address ffff88007ca89000
doesn''t match frame 9425c5
>> (XEN) mm.c:3617:d0 PTE entry 816875 for address ffff88007cb9b000
doesn''t match frame 59c675
>> (XEN) mm.c:3617:d0 PTE entry 8158a8 for address ffff88007cb9c000
doesn''t match frame 59d6a8
>> (XEN) mm.c:3617:d0 PTE entry 82fa1c for address ffff88007c926000
doesn''t match frame 5e321c
>> (XEN) mm.c:3617:d0 PTE entry 82655e for address ffff88007c927000
doesn''t match frame 5ec95e
>> (XEN) mm.c:3617:d0 PTE entry c5ce72 for address ffff88007c926000
doesn''t match frame 95b072
>> (XEN) mm.c:3617:d0 PTE entry c5cea9 for address ffff88007c927000
doesn''t match frame 95b0a9
>> (XEN) mm.c:2399:d0 Bad type (saw 1c00000000000002 != exp
7000000000000000) for mfn 800569 (pfn 4569)
>> (XEN) mm.c:878:d0 Error getting mfn 800569 (pfn 4569) from L1 entry
8010000800569267 for l1e_owner=0, pg_owner=0
>> (XEN) mm.c:4664:d0 ptwr_emulate: could not get_page_from_l1e()
>> (XEN) mm.c:3617:d0 PTE entry 80e284 for address ffff88007ca89000
doesn''t match frame 468084
>> (XEN) mm.c:3617:d0 PTE entry 817918 for address ffff88007ca8a000
doesn''t match frame 47f118
>> (XEN) common.c:3722:d0 tracking VRAM f0000 - f0160
>> (XEN) common.c:3722:d0 tracking VRAM f0000 - f0160
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
-- 

*Gerd Jakobovitsch
Engenheiro de Produto **
---------------------------------------------------------* *
**ALOG Data Centers do Brasil**
**Excelência em Projetos de Hosting*
Rua Dr. Miguel Couto, 58 -- 01008-010 -- São Paulo - SP
Telefone: (11) 3524-4970 / (11) 7152-0815
*http://www.alog.com.br* <http://www.alog.com.br/>


*"Como estão nossos serviços? Clique aqui 
<https://www.surveymonkey.com/s/faleconosco> e nos conte. Queremos 
escutar a sua opinião!"*







_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Konrad Rzeszutek Wilk

2011-Apr-18 21:05 UTC

head link

Re: [Xen-devel] xen 4.0.2rc3/kernel 2.6.32.36: BUG: unable to handle kernel paging request

On Fri, Apr 15, 2011 at 12:47:51PM -0300, Gerd Jakobovitsch
wrote:> Hello Pasi:
> 
> Yes, the kernel includes the last bugfix, and additionally the patch
> Daniel Stodden indicated yesterday:
> >>The BUG message at dmesg:
.. snip..> >>
> >>[66007.135552] BUG: unable to handle kernel paging request at
> >>ffff8800004ca458
> >>[66007.135567] IP: [<ffffffff8100d4ae>] xen_set_pte+0x3e/0x4b
> >>[66007.135580] PGD 1002067 PUD 1006067 PMD 2d78067 PTE
100000004ca025
> >>[66007.135675] Oops: 0003 [#1] SMP DEBUG_PAGEALLOC
.. snip..> >>[66007.135860] Call Trace:
> >>[66007.135868]  [<ffffffff81034740>] set_pte+0x17/0x1b
> >>[66007.135875]  [<ffffffff8103474d>] set_pte_atomic+0x9/0xb
> >>[66007.135882]  [<ffffffff81034e77>]
__change_page_attr_set_clr+0x186/0x82d
> >>[66007.135936]  [<ffffffff8124f4a0>] ?
_raw_spin_unlock+0xab/0xb1
> >>[66007.135951]  [<ffffffff8157641f>] ? _spin_unlock+0x26/0x2a
> >>[66007.135961]  [<ffffffff810e587d>] ?
vm_unmap_aliases+0x151/0x160
> >>[66007.135969]  [<ffffffff81035695>]
change_page_attr_set_clr+0x177/0x360
> >>[66007.135976]  [<ffffffff8103597a>]
change_page_attr_set+0x27/0x29
> >>[66007.135983]  [<ffffffff810348e2>] ? pte_flags+0x9/0x18
> >>[66007.135990]  [<ffffffff81035c01>]
do_pageattr_test+0x285/0x4b1
> >>[66007.135998]  [<ffffffff8103597c>] ?
do_pageattr_test+0x0/0x4b1
> >>[66007.136097]  [<ffffffff8106a9c3>] kthread+0x69/0x71
> >>[66007.136105]  [<ffffffff81013daa>] child_rip+0xa/0x20
> >>[66007.136112]  [<ffffffff81012ee6>] ?
int_ret_from_sys_call+0x7/0x1b
> >>[66007.136119]  [<ffffffff81013726>] ?
retint_restore_args+0x5/0x6
> >>[66007.136127]  [<ffffffff81013da0>] ? child_rip+0x0/0x20
.. snip..> >>(XEN) HVM43: Booting from 0000:7c00
> >>(XEN) HVM44: Press F12 for boot menu.
> >>(XEN) HVM44:
> >>(XEN) HVM44: Booting from Hard Disk...
> >>(XEN) HVM44: Booting from 0000:7c00
> >>(XEN) HVM44: int13_harddisk: function 42. LBA out of range
> >>(XEN) stdvga.c:151:d43 leaving stdvga
> >>(XEN) stdvga.c:147:d43 entering stdvga and caching modes
> >>(XEN) irq.c:243: Dom43 PCI link 0 changed 5 ->  0
> >>(XEN) irq.c:243: Dom43 PCI link 1 changed 10 ->  0
> >>(XEN) irq.c:243: Dom43 PCI link 2 changed 11 ->  0
> >>(XEN) irq.c:243: Dom43 PCI link 3 changed 5 ->  0
> >>(XEN) stdvga.c:151:d43 leaving stdvga
> >>(XEN) stdvga.c:151:d42 leaving stdvga
> >>(XEN) mm.c:3617:d0 PTE entry 81244b for address ffff88007caa0000
doesn''t match frame 621c4b
> >>(XEN) mm.c:3617:d0 PTE entry 812bed for address ffff88007caa1000
doesn''t match frame 6217ed
> >>(XEN) mm.c:3617:d0 PTE entry 812426 for address ffff88007caa2000
doesn''t match frame 621c26
> >>(XEN) mm.c:3617:d0 PTE entry 812590 for address ffff88007caa3000
doesn''t match frame 621d90
> >>(XEN) mm.c:3617:d0 PTE entry 812591 for address ffff88007caa4000
doesn''t match frame 621d91
> >>(XEN) mm.c:3617:d0 PTE entry 812492 for address ffff88007caa5000
doesn''t match frame 621c92
> >>(XEN) mm.c:3617:d0 PTE entry 812493 for address ffff88007caa6000
doesn''t match frame 621c93
> >>(XEN) mm.c:3617:d0 PTE entry 8124ca for address ffff88007caa7000
doesn''t match frame 621cca
> >>(XEN) mm.c:3617:d0 PTE entry 8124cb for address ffff88007caa8000
doesn''t match frame 621ccb
> >>(XEN) mm.c:3617:d0 PTE entry 80e8b0 for address ffff88007caa9000
doesn''t match frame 6258b0
> >>(XEN) mm.c:3617:d0 PTE entry 80e8b1 for address ffff88007caaa000
doesn''t match frame 6258b1
> >>(XEN) mm.c:3617:d0 PTE entry 803fe2 for address ffff88007caab000
doesn''t match frame 5cfde2
> >>(XEN) mm.c:3617:d0 PTE entry 80b3fd for address ffff88007ca99000
doesn''t match frame 46b1fd
> >>(XEN) mm.c:3617:d0 PTE entry 80b3fe for address ffff88007ca9a000
doesn''t match frame 46b1fe
> >>(XEN) mm.c:3617:d0 PTE entry 80ac10 for address ffff88007ca9b000
doesn''t match frame 46b610
> >>(XEN) mm.c:3617:d0 PTE entry 80c01e for address ffff88007ca9c000
doesn''t match frame 46a21e
> >>(XEN) mm.c:3617:d0 PTE entry 80ab16 for address ffff88007ca9d000
doesn''t match frame 46b916
> >>(XEN) mm.c:3617:d0 PTE entry 80b0f6 for address ffff88007ca9e000
doesn''t match frame 46b2f6
> >>(XEN) mm.c:3617:d0 PTE entry 809c8c for address ffff88007ca9f000
doesn''t match frame 46c68c
> >>(XEN) mm.c:3617:d0 PTE entry 8017e0 for address ffff88007cb9f000
doesn''t match frame 5b19e0
> >>(XEN) mm.c:3617:d0 PTE entry 8017e1 for address ffff88007cba0000
doesn''t match frame 5b19e1
> >>(XEN) mm.c:3617:d0 PTE entry 801fc4 for address ffff88007cba1000
doesn''t match frame 5b11c4
> >>(XEN) mm.c:3617:d0 PTE entry 801fc5 for address ffff88007cba2000
doesn''t match frame 5b11c5
> >>(XEN) mm.c:3617:d0 PTE entry 801f2a for address ffff88007cba3000
doesn''t match frame 5b112a
> >>(XEN) mm.c:3617:d0 PTE entry 801f2b for address ffff88007cba4000
doesn''t match frame 5b112b
> >>(XEN) mm.c:3617:d0 PTE entry 801ee8 for address ffff88007cba5000
doesn''t match frame 5b10e8
> >>(XEN) mm.c:3617:d0 PTE entry 801ee9 for address ffff88007cba6000
doesn''t match frame 5b10e9
> >>(XEN) mm.c:3617:d0 PTE entry 803fde for address ffff88007cba7000
doesn''t match frame 5af1de
> >>(XEN) mm.c:3617:d0 PTE entry 8120ec for address ffff88007cb9e000
doesn''t match frame 9660ec
> >>(XEN) mm.c:3617:d0 PTE entry 8120ed for address ffff88007cb9f000
doesn''t match frame 9660ed
> >>(XEN) mm.c:3617:d0 PTE entry 810e0a for address ffff88007cba0000
doesn''t match frame 96720a
> >>(XEN) mm.c:3617:d0 PTE entry 810e0b for address ffff88007cba1000
doesn''t match frame 96720b
> >>(XEN) mm.c:3617:d0 PTE entry 81080e for address ffff88007cba2000
doesn''t match frame 96780e
> >>(XEN) mm.c:3617:d0 PTE entry 81080f for address ffff88007cba3000
doesn''t match frame 96780f
> >>(XEN) mm.c:3617:d0 PTE entry 80ef3e for address ffff88007cba4000
doesn''t match frame 96933e
> >>(XEN) mm.c:3617:d0 PTE entry 802fd7 for address ffff88007c433000
doesn''t match frame 2111d7
> >>(XEN) mm.c:3617:d0 PTE entry 805131 for address ffff88007c434000
doesn''t match frame 20ef31
> >>(XEN) mm.c:3617:d0 PTE entry 80c1ea for address ffff88007c435000
doesn''t match frame 207fea
> >>(XEN) mm.c:3617:d0 PTE entry 814ae9 for address ffff88007c9b0000
doesn''t match frame 8866e9
> >>(XEN) mm.c:3617:d0 PTE entry 8113b9 for address ffff88007ca20000
doesn''t match frame 9277b9
> >>(XEN) mm.c:3617:d0 PTE entry 808ab2 for address ffff88007ca21000
doesn''t match frame 92feb2
> >>(XEN) mm.c:3617:d0 PTE entry 803bb3 for address ffff88007ca22000
doesn''t match frame 934fb3
> >>(XEN) mm.c:3617:d0 PTE entry 814747 for address ffff88007ca23000
doesn''t match frame 924347
> >>(XEN) mm.c:3617:d0 PTE entry 819873 for address ffff88007ca24000
doesn''t match frame 91f073
> >>(XEN) mm.c:3617:d0 PTE entry 80302c for address ffff88007ca25000
doesn''t match frame 93582c
> >>(XEN) mm.c:3617:d0 PTE entry 81b75c for address ffff88007ca26000
doesn''t match frame 91d35c
> >>(XEN) mm.c:3617:d0 PTE entry c5fd0b for address ffff88007ca86000
doesn''t match frame 95830b
> >>(XEN) mm.c:3617:d0 PTE entry 81a093 for address ffff88007ca87000
doesn''t match frame 9bde93
> >>(XEN) mm.c:3617:d0 PTE entry 815f11 for address ffff88007ca88000
doesn''t match frame 942111
> >>(XEN) mm.c:3617:d0 PTE entry 815bc5 for address ffff88007ca89000
doesn''t match frame 9425c5
> >>(XEN) mm.c:3617:d0 PTE entry 816875 for address ffff88007cb9b000
doesn''t match frame 59c675
> >>(XEN) mm.c:3617:d0 PTE entry 8158a8 for address ffff88007cb9c000
doesn''t match frame 59d6a8
> >>(XEN) mm.c:3617:d0 PTE entry 82fa1c for address ffff88007c926000
doesn''t match frame 5e321c
> >>(XEN) mm.c:3617:d0 PTE entry 82655e for address ffff88007c927000
doesn''t match frame 5ec95e
> >>(XEN) mm.c:3617:d0 PTE entry c5ce72 for address ffff88007c926000
doesn''t match frame 95b072
> >>(XEN) mm.c:3617:d0 PTE entry c5cea9 for address ffff88007c927000
doesn''t match frame 95b0a9
> >>(XEN) mm.c:2399:d0 Bad type (saw 1c00000000000002 != exp
7000000000000000) for mfn 800569 (pfn 4569)
> >>(XEN) mm.c:878:d0 Error getting mfn 800569 (pfn 4569) from L1 entry
8010000800569267 for l1e_owner=0, pg_owner=0
> >>(XEN) mm.c:4664:d0 ptwr_emulate: could not get_page_from_l1e()
> >>(XEN) mm.c:3617:d0 PTE entry 80e284 for address ffff88007ca89000
doesn''t match frame 468084
> >>(XEN) mm.c:3617:d0 PTE entry 817918 for address ffff88007ca8a000
doesn''t match frame 47f118
> >>(XEN) common.c:3722:d0 tracking VRAM f0000 - f0160
> >>(XEN) common.c:3722:d0 tracking VRAM f0000 - f0160
So to reproduce this, what should I be doing? Can you give me a rough idea of
what
your guests are doing, how big the NFS files are, etc?


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Xen devel - Apr 2011 - Tapdisk failures / kernel general protection fault at xen 4.0.2rc3 / kernel pvops 2.6.32.36

[Xen-devel] Tapdisk failures / kernel general protection fault at xen 4.0.2rc3 / kernel pvops 2.6.32.36

Re: [Xen-devel] Tapdisk failures / kernel general protection fault at xen 4.0.2rc3 / kernel pvops 2.6.32.36

Re: [Xen-devel] Tapdisk failures / kernel general protection fault at xen 4.0.2rc3 / kernel pvops 2.6.32.36

Re: [Xen-devel] Tapdisk failures / kernel general protection fault at xen 4.0.2rc3 / kernel pvops 2.6.32.36

Re: [Xen-devel] Tapdisk failures / kernel general protection fault at xen 4.0.2rc3 / kernel pvops 2.6.32.36

Re: [Xen-devel] Tapdisk failures / kernel general protection fault at xen 4.0.2rc3 / kernel pvops 2.6.32.36

Re: [Xen-devel] Tapdisk failures / kernel general protection fault at xen 4.0.2rc3 / kernel pvops 2.6.32.36

[Xen-devel] xen 4.0.2rc3/kernel 2.6.32.36: BUG: unable to handle kernel paging request

Re: [Xen-devel] xen 4.0.2rc3/kernel 2.6.32.36: BUG: unable to handle kernel paging request

Re: [Xen-devel] xen 4.0.2rc3/kernel 2.6.32.36: BUG: unable to handle kernel paging request

Re: [Xen-devel] xen 4.0.2rc3/kernel 2.6.32.36: BUG: unable to handle kernel paging request