Vivien Bernet-Rollande
2010-Nov-10 11:28 UTC
[Xen-devel] Infiniband from userland in dom0, process killed, bad pagetable
Hello all.
I''m new to Xen or kernel development.
I''m trying to fix the following issue :
- I''m in dom0, I don''t care about having IB in domU.
- Hardware is mellanox, driver is ib_mthca + libmthca
- IB from the kernel seems to work fine.
- However, when a process tries to access the hardware from userland
(through libmthca) it is killed by the kernel. The following message is sent
to the console :
Message from syslogd@x-dev-4 at Nov 10 14:00:42 ...
kernel:Bad pagetable: 000f [#1] SMP
Message from syslogd@x-dev-4 at Nov 10 14:00:42 ...
kernel:last sysfs file:
/sys/devices/pci0000:00/0000:00:02.0/0000:02:00.0/infiniband/mthca0/node_guid
Dmesg shows :
client: Corrupted page table at address 7ff332e74020
PGD 21abc067 PUD 4a9b4067 PMD 4c018067 PTE fffffffffffff237
Bad pagetable: 000f [#3] SMP
last sysfs file:
/sys/devices/pci0000:00/0000:00:02.0/0000:02:00.0/infiniband/mthca0/node_guid
CPU 0
Modules linked in: bridge stp llc sunrpc rdma_ucm ib_sdp rdma_cm iw_cm
ib_addr ib_ipoib ib_cm ib_sa ipv6 ib_uverbs ib_umad iw_nes libcrc32c
iw_cxgb3 cxgb3 mlx4_ib mlx4_en mlx4_core xen_netback xen_blkback
blkback_pagemap xen_gntdev xen_evtchn xenfs ib_mthca snd_hda_codec_realtek
snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device ib_mad snd_pcm
tg3 hp_wmi ppdev snd_timer rfkill k8temp ib_core snd edac_core parport_pc
soundcore i2c_piix4 snd_page_alloc wmi edac_mce_amd shpchp parport serio_raw
xfs exportfs pata_acpi ata_generic dm_multipath pata_atiixp radeon ttm
drm_kms_helper drm i2c_algo_bit i2c_core [last unloaded: scsi_wait_scan]
Pid: 3869, comm: client Tainted: G D W
2.6.32.21-167.xendom0.fc12.x86_64 #1 HP Compaq dc5750 Microtower
RIP: e033:[<00007ff332653d3f>] [<00007ff332653d3f>] 0x7ff332653d3f
RSP: e02b:00007fff029299a8 EFLAGS: 00010206
RAX: 0000000086000212 RBX: 0000000000401568 RCX: 0000000000000001
RDX: 00007ff332e74000 RSI: 0000000000000000 RDI: 0000000001d10b70
RBP: 00007fff029299c0 R08: 0000000000000000 R09: 0000000000000008
R10: 0000000000000001 R11: 00000030b1eecea0 R12: 0000000000400a90
R13: 00007fff02929cc0 R14: 0000000000000000 R15: 0000000000000000
FS: 00007ff332e6a700(0000) GS:ffff880003ea2000(0000) knlGS:0000000000000000
CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007ff332e74020 CR3: 0000000011a08000 CR4: 0000000000000660
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff4ff0 DR7: 0000000000000400
Process client (pid: 3869, threadinfo ffff880014d64000, task
ffff88005a531760)
RIP [<00007ff332653d3f>] 0x7ff332653d3f
RSP <00007fff029299a8>
---[ end trace a7919e7f17c0a729 ]---
xm dmesg :
(XEN) d0:v0: reserved bit in page table (ec=000F)
(XEN) Pagetable walk from 00007f9478d3c020:
(XEN) L4[0x0ff] = 00000000690c8067 000000000000d550
(XEN) L3[0x051] = 00000000496a8067 000000000002cf30
(XEN) L2[0x1c6] = 000000004989c067 000000000002cd24
(XEN) L1[0x13c] = fffff7fffffff237 ffffffffffffffff
(XEN) ----[ Xen-4.0.1 x86_64 debug=n Not tainted ]----
(XEN) CPU: 0
(XEN) RIP: e033:[<00007f947851bd3f>]
(XEN) RFLAGS: 0000000000010206 EM: 0 CONTEXT: pv guest
(XEN) rax: 0000000086000312 rbx: 0000000000401568 rcx: 0000000000000001
(XEN) rdx: 00007f9478d3c000 rsi: 0000000000000000 rdi: 0000000002429b70
(XEN) rbp: 00007fffbb95c3e0 rsp: 00007fffbb95c3c8 r8: 0000000000000000
(XEN) r9: 0000000000000008 r10: 0000000000000001 r11: 00000030b1eecea0
(XEN) r12: 0000000000400a90 r13: 00007fffbb95c6e0 r14: 0000000000000000
(XEN) r15: 0000000000000000 cr0: 000000008005003b cr4: 00000000000006f0
(XEN) cr3: 0000000043848000 cr2: 00007f9478d3c020
(XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: e02b cs: e033
(XEN) Guest stack trace from rsp=00007fffbb95c3c8:
(XEN) 0000000000400bea 00000000bb95c6e0 0000000002429b70 00007fffbb95c600
(XEN) 0000000000400f74 00007fffbb95c6e8 0000000203d8f538 0000000200000000
(XEN) 0000000000000001 0000000000000000 0000000000000000 0000000000000000
(XEN) 0000000000000000 0000000002429ad0 00000030b1a00784 00000030b1a00380
(XEN) 00000030b1c1fa00 00007fffbb95c5b0 0000000000000000 00000030b1c1f0e8
(XEN) 0000000000000000 00007f9478d33078 00007f9478d3db30 0000000000000000
(XEN) 0000000000000000 0000000000000000 0000000000000000 00000001000007f0
(XEN) 00000030b1c1f678 0000000000000000 0000000000000000 0000000000000000
(XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000001
(XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN) 0000000000000000 0000000000000000 00007fffbb95c5d0 0000000000000000
(XEN) 0000000000000000 0000000000000000 00000000004005ba 000000000242d1b0
(XEN) 0000000002430120 00007fffbb95c700 00007fffbb95c700 0000000000000004
(XEN) 00000000024303c0 0000000002429980 00000000024299d0 0000000002429b70
(XEN) 0000000000401455 0000000002429ad0 0000000000401410 0000000000000000
(XEN) 0000000000400a90 00007fffbb95c6e0 0000000000000000 0000000000000000
(XEN) 00000030b1e1eb1d 0000000000000000 00007fffbb95c6e8 0000000200000000
(XEN) 0000000000400c68 0000000000000000 36674056a3d08701 0000000000400a90
As the logs say, there''s a 1 in a reserved bit in a page table entry,
to
which a write access is performed in user mode ("ec=000F").
Now, this error looks a lot like a few of those :
http://wiki.xensource.com/xenwiki/XenPVOPSDRM so I expect the problem is
similar : physical addresses used are not the true physical addresses, and
bad things happen.
BTW, I am aware of the xen-smartio repository, and will use it as an
example. However, I will have to identify the changes it made and port from
2.6.18 xenlinux to something more recent.
My main question is the following : how is it possible to end up with an
invalid page table ? I would expect that such a bug would trigger a
segmentation violation. But since I suppose all page descriptors are
initialized by the kernel or Xen, how does a 1 end up in a reserved field ?
Or did I miss something ?
-- Vivien Bernet-Rollande
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel