Sander Eikelenboom
2010-Mar-11 10:44 UTC
[Xen-devel] [pvops-2.6.31.12 ]Fatal DMA error! Please use ''swiotlb=force'' kernel BUG at drivers/pci/xen-iommu.c:227! invalid opcode: 0000 [#1] SMP
Hi Konrad , On this machine i''m using - xen-3.4.2 - pvops 2.6.31.6 dom0 kernel - debian stock 2.6.26.2-xen kernels - All are 64bit, total system mem is 8GB, no hardware iommu present. I''m passing through several pci devices with great succes (worked stable for several months) - Soundcard - 1 PCI and 1 PCI-e USB 2.0 cards used for videograbbing Now i have tried to passthrough a third USB card, (PCI-e, USB 3.0 in this case), with a new pvops 2.6.31.12 kernel. But although i specified swiotlb=force, something seems to go wrong with DMA mapping, dmesg from the domU shows the error below. The complete dmesg output is attached. [ 1.139471] EXT3-fs: mounted filesystem with writeback data mode. [ 1.275690] stty used greatest stack depth: 4472 bytes left [ 1.532874] udevd version 125 started [ 1.623120] usb usb1: uevent [ 1.623154] usb 1-0:1.0: uevent [ 2.068935] hub 1-0:1.0: state 7 ports 4 chg 0000 evt 0010 [ 2.068952] hub 1-0:1.0: port 4, status 0101, change 0001, 12 Mb/s [ 2.172056] hub 1-0:1.0: debounce: port 4: total 100ms stable 100ms status 0x101 [ 2.274050] usb 1-4: new high speed USB device using xhci_hcd and address 0 [ 2.285064] dma address needs mapping [ 2.285081] mask: 0xffffffff [ 2.285082] address: [0x13654d1e0] [ 2.285092] Fatal DMA error! Please use ''swiotlb=force'' [ 2.285115] ------------[ cut here ]------------ [ 2.285122] kernel BUG at drivers/pci/xen-iommu.c:227! [ 2.285130] invalid opcode: 0000 [#1] SMP [ 2.285139] last sysfs file: /sys/devices/virtual/net/lo/operstate [ 2.285147] CPU 0 [ 2.285153] Modules linked in: [last unloaded: scsi_wait_scan] [ 2.285165] Pid: 254, comm: khubd Tainted: G W 2.6.31.12 #2 [ 2.285173] RIP: e030:[<ffffffff81206f5f>] [<ffffffff81206f5f>] xen_map_page+0x91/0xa0 [ 2.285192] RSP: e02b:ffff88000fcef960 EFLAGS: 00010286 [ 2.285199] RAX: 0000000000000041 RBX: 00000000ffffffff RCX: 00000000ffff0064 [ 2.285207] RDX: 0000000000007474 RSI: ffffffff815400d1 RDI: ffff88000fce02a0 [ 2.285216] RBP: ffff88000fcef980 R08: 0000000000000000 R09: ffff88000fc09000 [ 2.285224] R10: 0000000000000000 R11: ffff88000fc03200 R12: 000000013654d1e0 [ 2.285232] R13: ffff88000fe04880 R14: 0000000000000001 R15: 0000000000000000 [ 2.285244] FS: 00007fb57f4a0770(0000) GS:ffffc90000000000(0000) knlGS:0000000000000000 [ 2.285254] CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b [ 2.285262] CR2: 00007fb57f4a4000 CR3: 000000000f59d000 CR4: 0000000000000660 [ 2.285270] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2.285279] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 2.285288] Process khubd (pid: 254, threadinfo ffff88000fcee000, task ffff88000fcf6d60) [ 2.285298] Stack: [ 2.285303] 0000000000000100 ffffffff81206ece ffff88000fe04880 ffff88000e88f1e0 [ 2.285315] <0> ffff88000fcef9c0 ffffffff8135c5ab 0000000000000008 ffff88000fcefb50 [ 2.285331] <0> 0000000000000200 ffff88000e4f8f00 ffff88000fcff000 0000000000000010 [ 2.285350] Call Trace: [ 2.285358] [<ffffffff81206ece>] ? xen_map_page+0x0/0xa0 [ 2.285368] [<ffffffff8135c5ab>] dma_map_single_attrs+0x7d/0x97 [ 2.285378] [<ffffffff8135cdda>] usb_hcd_submit_urb+0xaa/0x9b8 [ 2.285388] [<ffffffff8100ed52>] ? check_events+0x12/0x20 [ 2.285398] [<ffffffff8100ed3f>] ? xen_restore_fl_direct_end+0x0/0x1 [ 2.285407] [<ffffffff8100c109>] ? xen_mc_flush+0x162/0x187 [ 2.285416] [<ffffffff8100ed3f>] ? xen_restore_fl_direct_end+0x0/0x1 [ 2.285426] [<ffffffff810bb600>] ? shmem_delete_inode+0xe0/0xff [ 2.285437] [<ffffffff8100b485>] ? xen_end_context_switch+0x19/0x1d [ 2.285447] [<ffffffff81011688>] ? __switch_to+0x18d/0x2d9 [ 2.285456] [<ffffffff8135dcb2>] usb_submit_urb+0x306/0x323 [ 2.285465] [<ffffffff8100ed52>] ? check_events+0x12/0x20 [ 2.285474] [<ffffffff8135efad>] usb_start_wait_urb+0x55/0x165 [ 2.285485] [<ffffffff810dda6f>] ? __kmalloc+0x134/0x146 [ 2.285494] [<ffffffff8135e04a>] ? usb_init_urb+0x22/0x33 [ 2.285502] [<ffffffff8135f300>] usb_control_msg+0x115/0x137 [ 2.285512] [<ffffffff8100ed52>] ? check_events+0x12/0x20 [ 2.285520] [<ffffffff813603b0>] usb_get_descriptor+0x71/0xa1 [ 2.285530] [<ffffffff810dcc69>] ? kmem_cache_alloc+0x8e/0xef [ 2.285539] [<ffffffff81360478>] usb_get_device_descriptor+0x98/0xc4 [ 2.285549] [<ffffffff81357bdd>] hub_port_init+0x5cc/0x773 [ 2.285558] [<ffffffff8135a99d>] hub_thread+0xbff/0x145a [ 2.285568] [<ffffffff8104005e>] ? dequeue_entity+0x1b/0x1df [ 2.285578] [<ffffffff81061ba4>] ? autoremove_wake_function+0x0/0x38 [ 2.285588] [<ffffffff81359d9e>] ? hub_thread+0x0/0x145a [ 2.285596] [<ffffffff81061810>] kthread+0x8f/0x97 [ 2.285605] [<ffffffff81013caa>] child_rip+0xa/0x20 [ 2.285613] [<ffffffff81012e67>] ? int_ret_from_sys_call+0x7/0x1b [ 2.285622] [<ffffffff8101361d>] ? retint_restore_args+0x5/0x6 [ 2.285631] [<ffffffff81013ca0>] ? child_rip+0x0/0x20 [ 2.285638] Code: 70 81 31 c0 e8 72 cb 33 00 4c 89 e2 48 89 de 48 c7 c7 c1 d0 70 81 31 c0 e8 5e cb 33 00 48 c7 c7 e3 d0 70 81 31 c0 e8 50 cb 33 00 <0f> 0b eb fe 5a 5b 4c 89 e0 41 5c 41 5d c9 c3 55 48 89 f8 25 ff [ 2.285745] RIP [<ffffffff81206f5f>] xen_map_page+0x91/0xa0 [ 2.285755] RSP <ffff88000fcef960> [ 2.285763] ---[ end trace 4eaa2a86a8e2da24 ]--- [ 2.285779] khubd used greatest stack depth: 3688 bytes left [ 2.541731] Adding 524280k swap on /dev/xvda1. Priority:-1 extents:1 across:524280k SS [ 2.588323] EXT3 FS on xvda2, internal journal [ 13.628035] eth0: no IPv6 routers present tv2:~# -- Best regards, Sander mailto:linux@eikelenboom.it _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Konrad Rzeszutek Wilk
2010-Mar-11 20:35 UTC
Re: [Xen-devel] [pvops-2.6.31.12 ]Fatal DMA error! Please use ''swiotlb=force'' kernel BUG at drivers/pci/xen-iommu.c:227! invalid opcode: 0000 [#1] SMP
On Thu, Mar 11, 2010 at 11:44:53AM +0100, Sander Eikelenboom wrote:> Hi Konrad , > > On this machine i''m using > - xen-3.4.2 > - pvops 2.6.31.6 dom0 kernel > - debian stock 2.6.26.2-xen kernels > - All are 64bit, total system mem is 8GB, no hardware iommu present. > > I''m passing through several pci devices with great succes (worked stable for several months) > - Soundcard > - 1 PCI and 1 PCI-e USB 2.0 cards used for videograbbing > > > > Now i have tried to passthrough a third USB card, (PCI-e, USB 3.0 in this case), with a new pvops 2.6.31.12 kernel.Hmm.. That is mostly unexpected.> > But although i specified swiotlb=force, something seems to go wrong with DMA mapping, dmesg from the domU shows the error below. > The complete dmesg output is attached. > > > [ 1.139471] EXT3-fs: mounted filesystem with writeback data mode. > [ 1.275690] stty used greatest stack depth: 4472 bytes left > [ 1.532874] udevd version 125 started > [ 1.623120] usb usb1: uevent > [ 1.623154] usb 1-0:1.0: uevent > [ 2.068935] hub 1-0:1.0: state 7 ports 4 chg 0000 evt 0010 > [ 2.068952] hub 1-0:1.0: port 4, status 0101, change 0001, 12 Mb/s > [ 2.172056] hub 1-0:1.0: debounce: port 4: total 100ms stable 100ms status 0x101 > [ 2.274050] usb 1-4: new high speed USB device using xhci_hcd and address 0 > [ 2.285064] dma address needs mapping > [ 2.285081] mask: 0xffffffff > [ 2.285082] address: [0x13654d1e0] > [ 2.285092] Fatal DMA error! Please use ''swiotlb=force'' > [ 2.285115] ------------[ cut here ]------------ > [ 2.285122] kernel BUG at drivers/pci/xen-iommu.c:227!That shouldn''t be there anymore. It looks like the SWIOTLB is initialized, _but_ the other less useful IOMMU is used instead. Let me dig in this and come back to you next week. In the internim, 2.6.32 (xen/next) should be working much better - try that. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Konrad Rzeszutek Wilk
2010-Mar-22 19:10 UTC
Re: [Xen-devel] [pvops-2.6.31.12 ]Fatal DMA error! Please use ''swiotlb=force'' kernel BUG at drivers/pci/xen-iommu.c:227! invalid opcode: 0000 [#1] SMP
On Thu, Mar 11, 2010 at 11:44:53AM +0100, Sander Eikelenboom wrote:> Hi Konrad , > > On this machine i''m using > - xen-3.4.2 > - pvops 2.6.31.6 dom0 kernel > - debian stock 2.6.26.2-xen kernels > - All are 64bit, total system mem is 8GB, no hardware iommu present. > > I''m passing through several pci devices with great succes (worked stable for several months) > - Soundcard > - 1 PCI and 1 PCI-e USB 2.0 cards used for videograbbing > > > > Now i have tried to passthrough a third USB card, (PCI-e, USB 3.0 in this case), with a new pvops 2.6.31.12 kernel. > > But although i specified swiotlb=force, something seems to go wrong with DMA mapping, dmesg from the domU shows the error below. > The complete dmesg output is attached.Ugh. There is one more parameter you need to add, that is ''iommu=soft''. That ''use swiotlb=force'' nees to be fixed. Thanks for spotting it. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel