Xen devel - Feb 2011 - Crash when reading /proc/kcore, kernel 2.6.32 xen/stable-2.6.32.x

I''m using kernel 2.6.32 xen/stable-2.6.32.x from Jeremy''s git
tree.
While testing zfs, I noticed that this kernel would crash every time
/proc/kcore is read. Something like this is enough to reproduce it:

# dd if=/proc/kcore of=/dev/null bs=1M count=1
BUG: unable to handle kernel paging request at ffff9d5555555000
IP: [<ffffffff81037ee0>] kern_addr_valid+0x44/0xcc
PGD 0
Oops: 0000 [#1] SMP
last sysfs file: /sys/devices/system/cpu/cpu1/cache/index2/shared_cpu_map
CPU 1
Modules linked in: xen_netback blktap xen_blkback blkback_pagemap
ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE
iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state
nf_conntrack ipt_REJECT xt_tcpudp iptable_filter ip_tables x_tables
autofs4 hidp rfcomm l2cap bluetooth rfkill lzfs zfs(P) zcommon(P)
znvpair(P) zavl(P) zlib_deflate zunicode(P) spl lockd sunrpc 8021q
garp bridge stp llc ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core
ib_addr iscsi_tcp bnx2i cnic uio ipv6 cxgb3i cxgb3 mdio libiscsi_tcp
libiscsi scsi_transport_iscsi video output sbs sbshc parport_pc lp
parport ipmi_devintf bnx2 radeon snd_seq_dummy ttm drm_kms_helper drm
i2c_algo_bit i2c_core ipmi_si ipmi_msghandler hpwdt snd_seq_oss
snd_seq_midi_event serio_raw snd_seq snd_seq_device snd_pcm_oss
snd_mixer_oss snd_pcm i5k_amb iTCO_wdt snd_timer i5000_edac
iTCO_vendor_support edac_core snd shpchp soundcore snd_page_alloc
pcspkr qla2xxx scsi_transport_fc scsi_tgt cciss uhci_hcd ohci_hcd
ehci_hcd [last unloaded: freq_table]
Pid: 8944, comm: dd Tainted: P           2.6.32.27-1.pv_ops.el5.fanxen
#1 ProLiant BL460c G1
RIP: e030:[<ffffffff81037ee0>]  [<ffffffff81037ee0>]
kern_addr_valid+0x44/0xcc
RSP: e02b:ffff880038a1dc20  EFLAGS: 00010286
RAX: ffff9d5555555000 RBX: ffff800000000000 RCX: ffffffff81918440
RDX: ffff880000000000 RSI: 0000000000000000 RDI: 00000002afffd067
RBP: ffff880038a1dc28 R08: ffff880038a1ddd8 R09: ffff880038a1de08
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000001000
R13: ffff800000000000 R14: 0000000000002000 R15: 00000000000fe000
FS:  00007f31849dd6e0(0000) GS:ffff880028055000(0000) knlGS:0000000000000000
CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: ffff9d5555555000 CR3: 0000000038ab3000 CR4: 0000000000002660
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process dd (pid: 8944, threadinfo ffff880038a1c000, task ffff880036b981c0)
Stack:
 ffff8800327d44a8 ffff880038a1dec8 ffffffff81170afb ffff880038a1df48
<0> 00007f31848dd000 0000000000002000 0000000000002000 0000000000000000
<0> 0000000000000000 0000000000000000 0000000000000000 0000000000000000
Call Trace:
 [<ffffffff81170afb>] read_kcore+0x4bb/0x574
 [<ffffffff8100ee27>] ? xen_set_pte_at+0xd1/0xef
 [<ffffffff81167cdf>] proc_reg_read+0x76/0x90
 [<ffffffff81118e69>] vfs_read+0xad/0x107
 [<ffffffff8111991d>] sys_read+0x4c/0x70
 [<ffffffff81012db2>] system_call_fastpath+0x16/0x1b
Code: f8 fd 0f 86 a6 00 00 00 48 c1 ef 24 81 e7 f8 0f 00 00 48 03 3d
9a be 64 00 48 83 3f 00 0f 84 8b 00 00 00 48 89 de e8 24 fa ff ff <48>
83 38 00 48 89 c7 74 7a 48 89 de e8 d7 f9 ff ff 48 8b 38 48
RIP  [<ffffffff81037ee0>] kern_addr_valid+0x44/0xcc
 RSP <ffff880038a1dc20>
CR2: ffff9d5555555000
---[ end trace 037fec8dd189bf20 ]---
Kernel panic - not syncing: Fatal exception
Pid: 8944, comm: dd Tainted: P      D    2.6.32.27-1.pv_ops.el5.fanxen #1
Call Trace:
 [<ffffffff810300ef>] ? gdb_regs_to_pt_regs+0x83/0xb6
 [<ffffffff81058b6a>] panic+0xa5/0x164
 [<ffffffff81430035>] ? agp_amd64_probe+0x570/0x584
 [<ffffffff8100f2b1>] ? xen_force_evtchn_callback+0xd/0xf
 [<ffffffff8100fa32>] ? check_events+0x12/0x20
 [<ffffffff810300ef>] ? gdb_regs_to_pt_regs+0x83/0xb6
 [<ffffffff81058837>] ? print_oops_end_marker+0x23/0x25
 [<ffffffff810300ef>] ? gdb_regs_to_pt_regs+0x83/0xb6
 [<ffffffff8143eb66>] oops_end+0xb7/0xc7
 [<ffffffff810387ac>] no_context+0x205/0x214
 [<ffffffff810300ef>] ? gdb_regs_to_pt_regs+0x83/0xb6
 [<ffffffff810f1372>] ? handle_mm_fault+0x312/0x776
 [<ffffffff810389ff>] __bad_area_nosemaphore+0x183/0x1a6
 [<ffffffff81038aac>] bad_area_nosemaphore+0x13/0x15
 [<ffffffff8143ffdd>] do_page_fault+0x147/0x26c
 [<ffffffff8143e075>] page_fault+0x25/0x30
 [<ffffffff81037ee0>] ? kern_addr_valid+0x44/0xcc
 [<ffffffff81170afb>] read_kcore+0x4bb/0x574
 [<ffffffff8100ee27>] ? xen_set_pte_at+0xd1/0xef
 [<ffffffff81167cdf>] proc_reg_read+0x76/0x90
 [<ffffffff81118e69>] vfs_read+0xad/0x107
 [<ffffffff8111991d>] sys_read+0x4c/0x70
 [<ffffffff81012db2>] system_call_fastpath+0x16/0x1b
(XEN) Debugging connection not set up.
(XEN) Domain 0 crashed: rebooting machine in 5 seconds.

Is this a known issue?

-- 
Fajar

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Fri, Feb 18, 2011 at 10:30:51AM +0700, Fajar A. Nugraha
wrote:
> I''m using kernel 2.6.32 xen/stable-2.6.32.x from Jeremy''s
git tree.
> While testing zfs, I noticed that this kernel would crash every time
> /proc/kcore is read. Something like this is enough to reproduce it:
Yikes!
Interestingly enough, the 2.6.38-rc5 does not have this problem.
> 
> # dd if=/proc/kcore of=/dev/null bs=1M count=1
> BUG: unable to handle kernel paging request at ffff9d5555555000
> IP: [<ffffffff81037ee0>] kern_addr_valid+0x44/0xcc
> PGD 0
> Oops: 0000 [#1] SMP
> last sysfs file: /sys/devices/system/cpu/cpu1/cache/index2/shared_cpu_map
> CPU 1
> Modules linked in: xen_netback blktap xen_blkback blkback_pagemap
> ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE
> iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state
> nf_conntrack ipt_REJECT xt_tcpudp iptable_filter ip_tables x_tables
> autofs4 hidp rfcomm l2cap bluetooth rfkill lzfs zfs(P) zcommon(P)
> znvpair(P) zavl(P) zlib_deflate zunicode(P) spl lockd sunrpc 8021q
> garp bridge stp llc ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core
> ib_addr iscsi_tcp bnx2i cnic uio ipv6 cxgb3i cxgb3 mdio libiscsi_tcp
> libiscsi scsi_transport_iscsi video output sbs sbshc parport_pc lp
> parport ipmi_devintf bnx2 radeon snd_seq_dummy ttm drm_kms_helper drm
> i2c_algo_bit i2c_core ipmi_si ipmi_msghandler hpwdt snd_seq_oss
> snd_seq_midi_event serio_raw snd_seq snd_seq_device snd_pcm_oss
> snd_mixer_oss snd_pcm i5k_amb iTCO_wdt snd_timer i5000_edac
> iTCO_vendor_support edac_core snd shpchp soundcore snd_page_alloc
> pcspkr qla2xxx scsi_transport_fc scsi_tgt cciss uhci_hcd ohci_hcd
> ehci_hcd [last unloaded: freq_table]
> Pid: 8944, comm: dd Tainted: P           2.6.32.27-1.pv_ops.el5.fanxen
> #1 ProLiant BL460c G1
> RIP: e030:[<ffffffff81037ee0>]  [<ffffffff81037ee0>]
kern_addr_valid+0x44/0xcc
> RSP: e02b:ffff880038a1dc20  EFLAGS: 00010286
> RAX: ffff9d5555555000 RBX: ffff800000000000 RCX: ffffffff81918440
> RDX: ffff880000000000 RSI: 0000000000000000 RDI: 00000002afffd067
> RBP: ffff880038a1dc28 R08: ffff880038a1ddd8 R09: ffff880038a1de08
> R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000001000
> R13: ffff800000000000 R14: 0000000000002000 R15: 00000000000fe000
> FS:  00007f31849dd6e0(0000) GS:ffff880028055000(0000)
knlGS:0000000000000000
> CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
> CR2: ffff9d5555555000 CR3: 0000000038ab3000 CR4: 0000000000002660
> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Process dd (pid: 8944, threadinfo ffff880038a1c000, task ffff880036b981c0)
> Stack:
>  ffff8800327d44a8 ffff880038a1dec8 ffffffff81170afb ffff880038a1df48
> <0> 00007f31848dd000 0000000000002000 0000000000002000
0000000000000000
> <0> 0000000000000000 0000000000000000 0000000000000000
0000000000000000
> Call Trace:
>  [<ffffffff81170afb>] read_kcore+0x4bb/0x574
>  [<ffffffff8100ee27>] ? xen_set_pte_at+0xd1/0xef
>  [<ffffffff81167cdf>] proc_reg_read+0x76/0x90
>  [<ffffffff81118e69>] vfs_read+0xad/0x107
>  [<ffffffff8111991d>] sys_read+0x4c/0x70
>  [<ffffffff81012db2>] system_call_fastpath+0x16/0x1b
> Code: f8 fd 0f 86 a6 00 00 00 48 c1 ef 24 81 e7 f8 0f 00 00 48 03 3d
> 9a be 64 00 48 83 3f 00 0f 84 8b 00 00 00 48 89 de e8 24 fa ff ff
<48>
> 83 38 00 48 89 c7 74 7a 48 89 de e8 d7 f9 ff ff 48 8b 38 48
> RIP  [<ffffffff81037ee0>] kern_addr_valid+0x44/0xcc
>  RSP <ffff880038a1dc20>
> CR2: ffff9d5555555000
> ---[ end trace 037fec8dd189bf20 ]---
> Kernel panic - not syncing: Fatal exception
> Pid: 8944, comm: dd Tainted: P      D    2.6.32.27-1.pv_ops.el5.fanxen #1
> Call Trace:
>  [<ffffffff810300ef>] ? gdb_regs_to_pt_regs+0x83/0xb6
>  [<ffffffff81058b6a>] panic+0xa5/0x164
>  [<ffffffff81430035>] ? agp_amd64_probe+0x570/0x584
>  [<ffffffff8100f2b1>] ? xen_force_evtchn_callback+0xd/0xf
>  [<ffffffff8100fa32>] ? check_events+0x12/0x20
>  [<ffffffff810300ef>] ? gdb_regs_to_pt_regs+0x83/0xb6
>  [<ffffffff81058837>] ? print_oops_end_marker+0x23/0x25
>  [<ffffffff810300ef>] ? gdb_regs_to_pt_regs+0x83/0xb6
>  [<ffffffff8143eb66>] oops_end+0xb7/0xc7
>  [<ffffffff810387ac>] no_context+0x205/0x214
>  [<ffffffff810300ef>] ? gdb_regs_to_pt_regs+0x83/0xb6
>  [<ffffffff810f1372>] ? handle_mm_fault+0x312/0x776
>  [<ffffffff810389ff>] __bad_area_nosemaphore+0x183/0x1a6
>  [<ffffffff81038aac>] bad_area_nosemaphore+0x13/0x15
>  [<ffffffff8143ffdd>] do_page_fault+0x147/0x26c
>  [<ffffffff8143e075>] page_fault+0x25/0x30
>  [<ffffffff81037ee0>] ? kern_addr_valid+0x44/0xcc
>  [<ffffffff81170afb>] read_kcore+0x4bb/0x574
>  [<ffffffff8100ee27>] ? xen_set_pte_at+0xd1/0xef
>  [<ffffffff81167cdf>] proc_reg_read+0x76/0x90
>  [<ffffffff81118e69>] vfs_read+0xad/0x107
>  [<ffffffff8111991d>] sys_read+0x4c/0x70
>  [<ffffffff81012db2>] system_call_fastpath+0x16/0x1b
> (XEN) Debugging connection not set up.
> (XEN) Domain 0 crashed: rebooting machine in 5 seconds.
> 
> Is this a known issue?
Nope. First tiem I see it.
> 
> -- 
> Fajar
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Fri, Feb 18, 2011 at 10:30 PM, Konrad Rzeszutek Wilk
<konrad.wilk@oracle.com> wrote:
> On Fri, Feb 18, 2011 at 10:30:51AM +0700, Fajar A. Nugraha wrote:
>> I''m using kernel 2.6.32 xen/stable-2.6.32.x from
Jeremy''s git tree.
>> While testing zfs, I noticed that this kernel would crash every time
>> /proc/kcore is read. Something like this is enough to reproduce it:
>
> Yikes!
> Interestingly enough, the 2.6.38-rc5 does not have this problem.
Unfortunately zfs module can only be built on kernels 2.6.32 - 36

Can 2.6.38 currently be considered stable enough to replace 2.6.32 as
dom0 kernel?

-- 
Fajar

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Fri, Feb 18, 2011 at 11:05:30PM +0700, Fajar A. Nugraha
wrote:
> On Fri, Feb 18, 2011 at 10:30 PM, Konrad Rzeszutek Wilk
> <konrad.wilk@oracle.com> wrote:
> > On Fri, Feb 18, 2011 at 10:30:51AM +0700, Fajar A. Nugraha wrote:
> >> I''m using kernel 2.6.32 xen/stable-2.6.32.x from
Jeremy''s git tree.
> >> While testing zfs, I noticed that this kernel would crash every
time
> >> /proc/kcore is read. Something like this is enough to reproduce
it:
> >
> > Yikes!
> > Interestingly enough, the 2.6.38-rc5 does not have this problem.
> 
> Unfortunately zfs module can only be built on kernels 2.6.32 - 36
> 
> Can 2.6.38 currently be considered stable enough to replace 2.6.32 as
> dom0 kernel?
No.
> 
> -- 
> Fajar
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel