Hello, I have a system running ubuntu12.10 with xen 4.1.3, which I believe was recently upgraded from xen 4.1.2 via the normal ubuntu ''apt-get upgrade'' path. 2 AMD opteron 6134 cpus, supermicro H8DG6 motherboard. Recently I noticed that AMD-Vi gets disabled at boot, which didn''t happen before. From ''xm dmesg'': (XEN) Command line: placeholder amd-iommu-debug iommu=verbose iommu=1 apic=debug iommu_inclusive_mapping=1 [...] (XEN) IVHD Error: Conflicting IO-APIC 0x0 entries (XEN) AMD-Vi: Error initialization (XEN) I/O virtualisation disabled I noticed it when getting an error upon trying to passthrough a PCI device. A month ago PCI passthru worked, but I haven''t tested in between so I cannot say exactly when the problem arose, if it has to do with the transit from 4.1.2 to 4.1.3 or not. IOMMU is enabled in BIOS, of course. Could it be a xen bug (I know that my system is somewhat unusual), a old BIOS bug interacting with newer xen or an aqcuired hardware error? I''ll of course provide more logs if needed. Thanks, Andreas ADDENDUM: Some additional debugging I did (maybe helpful, maybe not): If I boot a baremetal linux3.5 kernel, IOMMU seems to enable as it should, but I get an "AMD-Vi: Completion-Wait loop timed out" message in dmesg followed by a soft CPU lockup: [ 1.152261] AMD-Vi: Enabling IOMMU at 0000:00:00.2 cap 0x40 [ 1.226282] AMD-Vi: Enabling IOMMU at 0000:40:00.2 cap 0x40 [ 1.333560] AMD-Vi: Completion-Wait loop timed out [ 1.440835] AMD-Vi: Completion-Wait loop timed out (last line repeated x 100ish) [ 28.425464] BUG: soft lockup - CPU#11 stuck for 22s! [swapper/0:1] [ 28.425467] Modules linked in: [ 28.425471] CPU 11 [ 28.425472] Modules linked in: [ 28.425476] [ 28.425479] Pid: 1, comm: swapper/0 Not tainted 3.5.0-23-generic #35-Ubuntu Supermicro H8DG6/H8DGi/H8DG6/H8DGi [ 28.425486] RIP: 0010:[<ffffffff8101aff6>] [<ffffffff8101aff6>] native_read_tsc+0x6/0x20 [ 28.425497] RSP: 0000:ffff880234855d10 EFLAGS: 00000246 [ 28.425501] RAX: 00000000226cded8 RBX: 00000000ffffffff RCX: 000000000118c4ae [ 28.425504] RDX: 0000000000000025 RSI: 0000000000000286 RDI: 00000000000008ea [ 28.425507] RBP: ffff880234855d10 R08: 000000000000040a R09: 0000000000001ff0 [ 28.425511] R10: 0720072007200720 R11: 0720072007200720 R12: ffffffff81e49c23 [ 28.425514] R13: 0000000000000246 R14: ffff880234855cf0 R15: 0000000000000006 [ 28.425517] FS: 0000000000000000(0000) GS:ffff88083fcc0000(0000) knlGS:0000000000000000 [ 28.425521] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 28.425524] CR2: 0000000000000000 CR3: 0000000001c0b000 CR4: 00000000000007e0 [ 28.425527] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 28.425531] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 28.425534] Process swapper/0 (pid: 1, threadinfo ffff880234854000, task ffff880234858000) [ 28.425538] Stack: [ 28.425540] ffff880234855d40 ffffffff8133599a 0000000000010ef6 ffff880234855d98 [ 28.425547] 0000000000000286 0000000000002000 ffff880234855d50 ffffffff813358cc [ 28.425553] ffff880234855d70 ffffffff8154a7c5 ffff880434e4c800 ffff880434e4c814 [ 28.425559] Call Trace: [ 28.425566] [<ffffffff8133599a>] delay_tsc+0x4a/0x80 [ 28.425570] [<ffffffff813358cc>] __const_udelay+0x2c/0x30 [ 28.425577] [<ffffffff8154a7c5>] wait_on_sem+0x35/0x70 [ 28.425582] [<ffffffff8154b493>] iommu_queue_command_sync+0x83/0x140 [ 28.425587] [<ffffffff8154b563>] iommu_queue_command+0x13/0x20 [ 28.425592] [<ffffffff8154dd37>] iommu_flush_all_caches+0xd7/0x100 [ 28.425597] [<ffffffff8154ed90>] enable_iommus+0x250/0x360 [ 28.425605] [<ffffffff81cfc17e>] ? memblock_find_dma_reserve+0x13d/0x13d [ 28.425611] [<ffffffff81d328ce>] amd_iommu_init_hardware+0x1f7/0x231 [ 28.425616] [<ffffffff81d32913>] amd_iommu_init+0xb/0x8b [ 28.425620] [<ffffffff81cfc191>] pci_iommu_init+0x13/0x3e [ 28.425627] [<ffffffff8100212a>] do_one_initcall+0x12a/0x180 [ 28.425631] [<ffffffff81cf3d3a>] kernel_init+0x140/0x1c9 [ 28.425635] [<ffffffff81cf3588>] ? loglevel+0x31/0x31 [ 28.425642] [<ffffffff8168ce64>] kernel_thread_helper+0x4/0x10 [ 28.425646] [<ffffffff81cf3bfa>] ? start_kernel+0x3d2/0x3d2 [ 28.425650] [<ffffffff8168ce60>] ? gs_change+0x13/0x13 [ 28.425652] Code: c3 0f 1f 40 00 55 89 f8 48 89 e5 e6 70 e4 71 5d c3 0f 1f 40 00 55 89 f0 4 8 89 e5 e6 70 89 f8 e6 71 5d c3 66 90 55 48 89 e5 0f 31 <89> c0 48 c1 e2 20 48 09 c2 48 89 d0 5d c3 66 66 66 2e 0f 1f 84 [ 28.458326] AMD-Vi: Completion-Wait loop timed out [ 28.565522] AMD-Vi: Completion-Wait loop timed out [ 28.672707] AMD-Vi: Completion-Wait loop timed out [ 28.779909] AMD-Vi: Completion-Wait loop timed out [ 28.886441] AMD-Vi: Completion-Wait loop timed out [ 28.886491] pci 0000:00:00.2: irq 40 for MSI/MSI-X [ 28.886676] pci 0000:40:00.2: irq 41 for MSI/MSI-X [ 28.896427] AMD-Vi: Lazy IO/TLB flushing enabled (boot continues as normal) This may point towards either hardware or bios. But I do not know if this would have happened before I started getting problems with xen. _______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users
Ian Campbell
2013-Feb-14 10:03 UTC
Re: IVHD error, AMD-Vi gets disabled (4.1.3-3ubuntu1.3)
On Wed, 2013-02-13 at 14:17 +0000, Andreas Falck wrote:> > Could it be a xen bug (I know that my system is somewhat unusual), a > old BIOS bug interacting with newer xen or an aqcuired hardware error? > I''ll of course provide more logs if needed.Almost certainly the middle one IMHO, as bugs get fixed in Xen it becomes less tolerant of dubious BIOS tables. Ian.
Hans Mueller
2013-Feb-15 23:43 UTC
Re: IVHD error, AMD-Vi gets disabled (4.1.3-3ubuntu1.3)
On Wednesday, 13. February 2013 15:17:54 Andreas Falck wrote:> Hello, > > I have a system running ubuntu12.10 with xen 4.1.3, which I believe was > recently upgraded from xen 4.1.2 via the normal ubuntu ''apt-get upgrade'' > path. 2 AMD opteron 6134 cpus, supermicro H8DG6 motherboard. > > Recently I noticed that AMD-Vi gets disabled at boot, which didn''t happen > before. > > From ''xm dmesg'': > > (XEN) Command line: placeholder amd-iommu-debug iommu=verbose iommu=1 > apic=debug iommu_inclusive_mapping=1 > [...] > (XEN) IVHD Error: Conflicting IO-APIC 0x0 entries > (XEN) AMD-Vi: Error initialization > (XEN) I/O virtualisation disabledlooks like related to the XSA-36 http://lists.xen.org/archives/html/xen-announce/2013-02/msg00002.html
Andreas Falck
2013-Feb-16 07:58 UTC
Re: IVHD error, AMD-Vi gets disabled (4.1.3-3ubuntu1.3)
Ian and Hans are probably right. Adding the "iommu=amd-iommu-global-intremap" option to xen enables the IOMMU with warnings. A BIOS update did NOT work however, to the latest one available from Supermicro (R 3.0, Bios Date 09/10/2012). This would suggest that Supermicro mainboard models H8DG6 / H8DGi / H8DG6-F / H8DGi-F are broken in this respect, at least with current available BIOSes. I will stick with the insecure workaround, but I''ll notify Supermicro about the problem. Thanks 2013/2/14 Ian Campbell>On Wed, 2013-02-13 at 14:17 +0000, Andreas Falck wrote: >> >> Could it be a xen bug (I know that my system is somewhat unusual), a >> old BIOS bug interacting with newer xen or an aqcuired hardware error? >> I''ll of course provide more logs if needed.>Almost certainly the middle one IMHO, as bugs get fixed in Xen it >becomes less tolerant of dubious BIOS tables.2013/2/16 Hans Mueller>On Wednesday, 13. February 2013 15:17:54 Andreas Falck wrote: >> [...] >> (XEN) IVHD Error: Conflicting IO-APIC 0x0 entries >> (XEN) AMD-Vi: Error initialization >> (XEN) I/O virtualisation disabled>looks like related to the XSA-36 >http://lists.xen.org/archives/html/xen-announce/2013-02/msg00002.html<http://lists.xen.org/archives/html/xen-announce/2013-02/msg00002.html>_______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users
Andreas Falck
2013-May-01 12:25 UTC
Re: IVHD error, AMD-Vi gets disabled (4.1.3-3ubuntu1.3)
Just to follow up on this issue, Supermicro support went on the problem quite quickly and emailed me an updated bios which made iommu work without interrupt remapping. I informed them on the success, however, they have not yet updated the official bios. So it might help to contact them, if you experience the same problem. Now I have working iommu:s on a Supermicro H8DG6, with recent xen, without need for global interrupt remapping. Regards, Andreas 2013/2/16 Andreas Falck <falck.andreas.lists@gmail.com>> Ian and Hans are probably right. Adding the > "iommu=amd-iommu-global-intremap" option to xen enables the IOMMU with > warnings. > > A BIOS update did NOT work however, to the latest one available from > Supermicro (R 3.0, Bios Date 09/10/2012). This would suggest that > Supermicro mainboard models H8DG6 / H8DGi / H8DG6-F / H8DGi-F are broken in > this respect, at least with current available BIOSes. > > I will stick with the insecure workaround, but I''ll notify Supermicro > about the problem. > > Thanks > > 2013/2/14 Ian Campbell > >On Wed, 2013-02-13 at 14:17 +0000, Andreas Falck wrote: > >> > >> Could it be a xen bug (I know that my system is somewhat unusual), a > >> old BIOS bug interacting with newer xen or an aqcuired hardware error? > >> I''ll of course provide more logs if needed. > > >Almost certainly the middle one IMHO, as bugs get fixed in Xen it > >becomes less tolerant of dubious BIOS tables. > > 2013/2/16 Hans Mueller > > >On Wednesday, 13. February 2013 15:17:54 Andreas Falck wrote: > >> [...] > >> (XEN) IVHD Error: Conflicting IO-APIC 0x0 entries > >> (XEN) AMD-Vi: Error initialization > >> (XEN) I/O virtualisation disabled > > >looks like related to the XSA-36 > >http://lists.xen.org/archives/html/xen-announce/2013-02/msg00002.html<http://lists.xen.org/archives/html/xen-announce/2013-02/msg00002.html> > >_______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users