Xen users - Jan 2013 - Xen 4.2.1 boot failure with IOMMU enabled

Hi all

I have a problem with enabling IOMMU on Xen 4.2.1. When I enable it in BIOS
and in grub.conf using iommu=1 kernel option, my machine cannot boot.
I get a following error on serial console:

(XEN) ****************************************
(XEN) Panic on CPU 0:
(XEN) Xen BUG at pci_amd_iommu.c:35
(XEN) ****************************************
(XEN)
(XEN) Reboot in five seconds...

Error says it is a bug, but maybe there is a workaround for it.

I run CentOS 6.3 and use dom0 kernel 3.7.1 and Xen from
http://au1.mirror.crc.id.au/repo/el6/x86_64/ repository,
but I also tried with other 3.x and 2.6.32 kernels and different Xen
builds with no luck.

GRUB entry:
 title CentOS Xen kernel IOMMU serial console (3.7.1-3.el6xen.x86_64)
        root (hd0,0)
        kernel /xen.gz dom0_mem=1G,max:1G dom0_max_vcpus=1
dom0_vcpus_pin iommu=verbose loglvl=all guest_loglvl=all iommu=1
com1=38400,8n1 console=com1
        module /vmlinuz-3.7.1-3.el6xen.x86_64 ro
root=/dev/mapper/vg_titan_raid5-lv_titan_root
rd_LVM_LV=vg_titan_raid5/lv_titan_root rd_NO_LUKS LANG=en_US.UTF-8
rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=auto
rd_LVM_LV=vg_titan_raid5/lv_titan_swap  KEYBOARDTYPE=pc KEYTABLE=us
rd_NO_DM console=hvc0 earlyprintk=xen nomodeset
        module /initramfs-3.7.1-3.el6xen.x86_64.img

Hardware info:
 Motherboard: M4A89TD PRO USB3 (AMD 890FX chipset)
 CPU: AMD Phenom II X6 1045T

Software info:
 OS: CentOS 6.3 64bit
 Xen: 4.2.1
 BIOS version: 3029 (up to date)

Detailed information:
 Full serial output: http://pastebin.com/raw.php?i=K1DuhDcj
 xl info (when booting with iommu=0): http://pastebin.com/raw.php?i=jU7bEFrN
 lspci -vvv: http://pastebin.com/raw.php?i=3wpKPQT9
 dmidecode: http://pastebin.com/raw.php?i=7wEcTXzr
 kernel config: http://pastebin.com/raw.php?i=zYgGZ84f

Please help
povder

Hello Povder,

On Jan 4, 2013, at 1:33 PM, povder <povder@gmail.com> wrote:
> Hi all
>
> I have a problem with enabling IOMMU on Xen 4.2.1. When I enable it in BIOS
> and in grub.conf using iommu=1 kernel option, my machine cannot boot.
> I get a following error on serial console:
>
> (XEN) ****************************************
> (XEN) Panic on CPU 0:
> (XEN) Xen BUG at pci_amd_iommu.c:35
> (XEN) ****************************************
> (XEN)
> (XEN) Reboot in five seconds...
>
> Error says it is a bug, but maybe there is a workaround for it.
<snip>
> Hardware info:
> Motherboard: M4A89TD PRO USB3 (AMD 890FX chipset)
> CPU: AMD Phenom II X6 1045T
>
> Software info:
> OS: CentOS 6.3 64bit
> Xen: 4.2.1
> BIOS version: 3029 (up to date)
<snip>

It''s a shot in the dark, but in my experiences with an MSI board
running the same chipset... Well, really almost every board with that
chipset... IOMMU support wasn''t quite ready "out of the
gate," and was
usually added and adjusted over time.  In my case, IOMMU had been
added with a BIOS version one or two steps before the version that
shipped with my board.  Between that initial version and the one I
had, they managed to break IOMMU in such a way that it worked with
neither Xen nor ESXi.  That took days to figure out :P

A search of the mailing list brought up a thread from June of 2010,
stating that IOMMU was reported working on the board you have.  You
might want to try a BIOS version that was current at that time, and
work your way forward until it breaks.  The latest BIOS I saw from
that period was 0901.  You could try that and work your way back.

That''s the best suggestion I can come up with ;)

Best Regards,
Andrew Bobulsky

> A search of the mailing list brought up a thread from June of 2010,
> stating that IOMMU was reported working on the board you have.  You
> might want to try a BIOS version that was current at that time, and
> work your way forward until it breaks.  The latest BIOS I saw from
> that period was 0901.  You could try that and work your way back.
I tried to downgrade BIOS to 0901 version but no matter how hard i tried
various BIOS update utilities won''t allow me to downgrade to 0901.
I even took the risk of cutting off power during flashing 3029 version
to install 0901 from BIOS recovery mode but even in recovery mode
I couldn''t flash 0901 - I got message "Image is outdated".

I managed to downgrade to various different versions though (starting
from v1006), and I still cannot boot with IOMMU enabled, so I switched
back to v3029.

Can I be sure that Xen error message
"(XEN) Xen BUG at pci_amd_iommu.c:35" is caused by a buggy BIOS?

Hello Povder,

On Jan 13, 2013, at 5:57 AM, povder <povder@gmail.com> wrote:

A search of the mailing list brought up a thread from June of 2010,

stating that IOMMU was reported working on the board you have.  You

might want to try a BIOS version that was current at that time, and

work your way forward until it breaks.  The latest BIOS I saw from

that period was 0901.  You could try that and work your way back.


I tried to downgrade BIOS to 0901 version but no matter how hard i tried
various BIOS update utilities won''t allow me to downgrade to 0901.
I even took the risk of cutting off power during flashing 3029 version
to install 0901 from BIOS recovery mode but even in recovery mode
I couldn''t flash 0901 - I got message "Image is outdated".


I managed to downgrade to various different versions though (starting
from v1006), and I still cannot boot with IOMMU enabled, so I switched
back to v3029.


I''m sorry you weren''t able to do that downgrade, but it
appears that''s
actually a good thing.... It looks like the reason you were able to flash
1006 but nothing prior is because that''s the first BIOS to support the
CPU
you have.  At least it prevented you from having to dig a CPU out of a
different computer to fix that problem :P

Can I be sure that Xen error message
"(XEN) Xen BUG at pci_amd_iommu.c:35" is caused by a buggy BIOS?


Unfortunately, no.  I only offered it as a suggestion because it''s the
best
one I''ve got based on my previous experience with 890FX boards.  If you
can''t chase down any better leads on what could potentially be the
cause of
this issue, you *could *grab an older CPU and work the BIOS back down to
0901 to see what happens.  That''s more of an academic endeavor than
anything else though, as I''m sure you have a 6-core CPU for a reason,
of
course.

If someone else here isn''t able to help---and I''d wager that
the lack of
replies to this thread are what that means---you could try asking on the
Xen-Devel mailing list.  The folks there should be more in tune with what,
precisely, the error messages really mean.  Though I''d suggest giving
it a
day or two just in case that''s off-topic... It is the weekend, after
all,
and someone may yet have an idea here :)

Cheers,
Andrew Bobulsky


_______________________________________________
Xen-users mailing list
Xen-users@lists.xen.org
http://lists.xen.org/xen-users