Xen devel - Jul 2011 - IOMMU issue on HP DL385g7 servers - workaround_bogus_bios / workaround_bug_bios doesn''t help

Hello,

I have cretaed wheezy xen 4.1 debian live image (s. the link below to 
it) but even I tried different boot options like "iommu=1 | 
workaround_bios_bug | workaround_bogus_bios" I am getting such entry 
"(XEN) AMD-Vi: IOMMU not found!" in the log (s. for more details in 
http://www.it-infrastrukturen.com/fileadmin/linux/paste/xm_dmesg.xen41_on_DL385g7_NOK.txt
)

link to xen 4.0 (squeeze) and xen 4.1 (wheezy) debian live images - 
http://www.it-infrastrukturen.com/fileadmin/linux/debian-live-xen/README.xen

do you have any idea how to solve this issue? - thank you in advance for 
any hints (different one than killing the hardware vendor (HP) ;) ... 
for the missing IOMMU BIOS option

Linux xen41dom0 2.6.39-2-amd64 #1 SMP Wed Jun 8 11:01:04 UTC 2011 x86_64 
GNU/Linux

Thank you in advance for any hints.

Regards, Mark

-- 
ms@it-infrastrukturen.org


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Thank you for your message Konrad.
>  Which link is that for creating a live image with Xen?
Sorry for confusion - currently it is only a link to a file with 
instruction where to get last images (one for sqeeze and one for wheezy) 
from and some additional information.
http://www.it-infrastrukturen.com/fileadmin/linux/debian-live-xen/README.xen
# ---

rsync -avP
rsync://www.it-infrastrukturen.ch/ftp/xen40-squeeze-amd64-live-gnome-binary-hybrid.iso
.

# ---

As I need HVM with IOMMU support to passthrough some PCI-devices to domU 
domains, I am looking for a workaround for missing IOMMU-flag support in 
BIOS of HP DL385g7. Server hardware from other vendors can work 
correctly. For this reason I have also created a xen live image for 
wheezy (the available one with kernel 2.6.39-2 amd64 and non booting one 
with 3.0-rc6 (non available yet for downloads)), however there are some 
kernel 3.0-rc6 issues with aufs support so the wheezy image doesnt boot 
as expected with kernel 3.0-rc6)

I use debian live builder to create these live images. If you need some 
detailed instruction how to create such images with debian live please 
let me know. It is a bunch of config files, some hook scripts, local 
packages (like firmware-bnx2 ... you need for HP DL servers) and xen 
PVM/HVM config examples I copy into the live builder dir structure after 
running "lb config ... " and before running "lb-build ... "
>  Did you try to follow the ParaVirt Wiki for the troubleshooting options?
Like
>  ratching up the debug options?
I am not sure what wiki (URL) do you mean exactly. I have read a lot of 
different wiki-pages.
I will check for it.

Regards,
Mark

-- 
ms@it-infrastrukturen.org



Am 14.07.2011 15:34, schrieb Konrad Rzeszutek Wilk:
> On Wed, Jul 13, 2011 at 04:05:54PM +0200, Mark Schneider wrote:
>    
>> Hello,
>>
>> I have cretaed wheezy xen 4.1 debian live image (s. the link below
>> to it) but even I tried different boot options like "iommu=1 |
>>      
> Which link is that for creating a live image with Xen?
>
>    
>> workaround_bios_bug | workaround_bogus_bios" I am getting such
entry
>> "(XEN) AMD-Vi: IOMMU not found!" in the log (s. for more
details in
http://www.it-infrastrukturen.com/fileadmin/linux/paste/xm_dmesg.xen41_on_DL385g7_NOK.txt
>> )
>>
>> link to xen 4.0 (squeeze) and xen 4.1 (wheezy) debian live images -
http://www.it-infrastrukturen.com/fileadmin/linux/debian-live-xen/README.xen
>>
>> do you have any idea how to solve this issue? - thank you in advance
>> for any hints (different one than killing the hardware vendor (HP)
>> ;) ... for the missing IOMMU BIOS option
>>      
> Did you try to follow the ParaVirt Wiki for the troubleshooting options?
Like
> ratching up the debug options?
>    
>> Linux xen41dom0 2.6.39-2-amd64 #1 SMP Wed Jun 8 11:01:04 UTC 2011
>> x86_64 GNU/Linux
>>
>> Thank you in advance for any hints.
>>
>> Regards, Mark
>>
>> -- 
>> ms@it-infrastrukturen.org
>>
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
>>      
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Wed, Jul 13, 2011 at 04:05:54PM +0200, Mark Schneider
wrote:
> Hello,
> 
> I have cretaed wheezy xen 4.1 debian live image (s. the link below
> to it) but even I tried different boot options like "iommu=1 |
Which link is that for creating a live image with Xen?
> workaround_bios_bug | workaround_bogus_bios" I am getting such entry
> "(XEN) AMD-Vi: IOMMU not found!" in the log (s. for more details
in
http://www.it-infrastrukturen.com/fileadmin/linux/paste/xm_dmesg.xen41_on_DL385g7_NOK.txt
> )
> 
> link to xen 4.0 (squeeze) and xen 4.1 (wheezy) debian live images -
http://www.it-infrastrukturen.com/fileadmin/linux/debian-live-xen/README.xen
> 
> do you have any idea how to solve this issue? - thank you in advance
> for any hints (different one than killing the hardware vendor (HP)
> ;) ... for the missing IOMMU BIOS option
Did you try to follow the ParaVirt Wiki for the troubleshooting options? Like
ratching up the debug options?
> 
> Linux xen41dom0 2.6.39-2-amd64 #1 SMP Wed Jun 8 11:01:04 UTC 2011
> x86_64 GNU/Linux
> 
> Thank you in advance for any hints.
> 
> Regards, Mark
> 
> -- 
> ms@it-infrastrukturen.org
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> > Did you try to follow the ParaVirt Wiki for the troubleshooting
options? Like
> > ratching up the debug options?
> 
> I am not sure what wiki (URL) do you mean exactly. I have read a lot
> of different wiki-pages.
> I will check for it.
http://wiki.xensource.com/xenwiki/XenParavirtOps

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Wednesday 13 July 2011 16:05:54 Mark Schneider wrote:
> Hello,
>
> I have cretaed wheezy xen 4.1 debian live image (s. the link below to
> it) but even I tried different boot options like "iommu=1 |
> workaround_bios_bug | workaround_bogus_bios" I am getting such entry
> "(XEN) AMD-Vi: IOMMU not found!" in the log (s. for more details
in
> http://www.it-infrastrukturen.com/fileadmin/linux/paste/xm_dmesg.xen41_on_D
>L385g7_NOK.txt )
>
> link to xen 4.0 (squeeze) and xen 4.1 (wheezy) debian live images -
> http://www.it-infrastrukturen.com/fileadmin/linux/debian-live-xen/README.xe
>n
>
> do you have any idea how to solve this issue? - thank you in advance for
> any hints (different one than killing the hardware vendor (HP) ;) ...
> for the missing IOMMU BIOS option
>
> Linux xen41dom0 2.6.39-2-amd64 #1 SMP Wed Jun 8 11:01:04 UTC 2011 x86_64
> GNU/Linux
>
> Thank you in advance for any hints.
>
> Regards, Mark
Hi,
workaround_bios_bug  and workaround_bogus_bios only works for vtd not amd.
You should have IVRS table build by BIOS to enable amd iommu. It could look 
like this:
(XEN) ACPI: IVRS AFE7C1ED, 00C8 (r1  AMD     RD890S  6040000 AMD         0)
Thanks,
Wei



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Am 14.07.2011 18:27, schrieb Wei Wang2:
> Wednesday 13 July 2011 16:05:54 Mark Schneider wrote:
>
> I have cretaed wheezy xen 4.1 debian live image (s. the link below to
> it) but even I tried different boot options like "iommu=1 |
> workaround_bios_bug | workaround_bogus_bios" I am getting such entry
> "(XEN) AMD-Vi: IOMMU not found!" in the log (s. for more details
in
> http://www.it-infrastrukturen.com/fileadmin/linux/paste/xm_dmesg.xen41_on_D
> L385g7_NOK.txt )
>
> link to xen 4.0 (squeeze) and xen 4.1 (wheezy) debian live images -
> http://www.it-infrastrukturen.com/fileadmin/linux/debian-live-xen/README.xe
> n
>
> do you have any idea how to solve this issue? - thank you in advance for
> any hints (different one than killing the hardware vendor (HP) ;) ...
> for the missing IOMMU BIOS option
>
> Linux xen41dom0 2.6.39-2-amd64 #1 SMP Wed Jun 8 11:01:04 UTC 2011 x86_64
> GNU/Linux
>
> Thank you in advance for any hints.
>
> Regards, Mark
>    
> Hi,
> workaround_bios_bug  and workaround_bogus_bios only works for vtd not amd.
> You should have IVRS table build by BIOS to enable amd iommu. It could look
> like this:
> (XEN) ACPI: IVRS AFE7C1ED, 00C8 (r1  AMD     RD890S  6040000 AMD         0)
> Thanks,
> Wei
>    
Thanks a lot for your hints Wei.

All ACPI entries I see in "xm dmesg" are only like below. I
didn''t find
any IVRS entry
# --
(XEN) Xen-e820 RAM map:
(XEN)  0000000000000000 - 000000000009e800 (usable)
(XEN)  000000000009e800 - 00000000000a0000 (reserved)
(XEN)  00000000000f0000 - 0000000000100000 (reserved)
(XEN)  0000000000100000 - 00000000bfc1c000 (usable)
(XEN)  00000000bfc1c000 - 00000000bfc4c000 (ACPI data)
(XEN)  00000000bfc4c000 - 00000000bfc4d000 (usable)
(XEN)  00000000bfc4d000 - 00000000d0000000 (reserved)
(XEN)  00000000fec00000 - 00000000fee10000 (reserved)
(XEN)  00000000ff800000 - 0000000100000000 (reserved)
(XEN)  0000000100000000 - 000000043ffff000 (usable)
(XEN) ACPI: RSDP 000F4EE0, 0024 (r2 HP    )
(XEN) ACPI: XSDT BFC1D3C0, 00A4 (r1 HP     ProLiant        2   Ò^D     162E)
(XEN) ACPI: FACP BFC1D4C0, 00F4 (r3 HP     ProLiant        2   Ò^D     162E)
(XEN) ACPI: DSDT BFC1D5C0, CCBB (r1 HP         DSDT        1 INTL 20061109)
(XEN) ACPI: FACS BFC1C140, 0040
(XEN) ACPI: SPCR BFC1C180, 0050 (r1 HP     SPCRRBSU        1   Ò^D     162E)
(XEN) ACPI: MCFG BFC1C200, 003C (r1 HP     ProLiant        1             0)
(XEN) ACPI: HPET BFC1C240, 0038 (r1 HP     ProLiant        2   Ò^D     162E)
(XEN) ACPI: SPMI BFC1C280, 0040 (r5 HP     ProLiant        1   Ò^D     162E)
(XEN) ACPI: ERST BFC1C2C0, 01D0 (r1 HP     ProLiant        1   Ò^D     162E)
(XEN) ACPI: APIC BFC1C4C0, 011E (r1 HP     ProLiant        2             0)
(XEN) ACPI: SRAT BFC1C7C0, 02A0 (r2 AMD    F10             1 AMD         1)
(XEN) ACPI: FFFF BFC1CFC0, 0176 (r1 HP     ProLiant        1   Ò^D     162E)
(XEN) ACPI: BERT BFC1D140, 0030 (r1 HP     ProLiant        1   Ò^D     162E)
(XEN) ACPI: HEST BFC1D180, 018C (r1 HP     ProLiant        1   Ò^D     162E)
(XEN) ACPI: FFFF BFC1D340, 0064 (r2 HP     ProLiant        2   Ò^D     162E)
(XEN) ACPI: SSDT BFC2A280, 0126 (r3     HP  PCIE1X8        2 INTL 20090625)
(XEN) ACPI: SSDT BFC2A3C0, 024A (r3     HP PCIE2N3X        2 INTL 20090625)
(XEN) ACPI: SSDT BFC2A640, 0125 (r3     HP  CRSPCI0        2   HP        1)
(XEN) ACPI: SSDT BFC2A780, 0377 (r1     HP     pmab        1 INTL 20090625)
.
.
(XEN) Initing memory sharing.
(XEN) AMD-Vi: IOMMU not found!
(XEN) I/O virtualisation disabled
(XEN) ENABLING IO-APIC IRQs
(XEN)  -> Using new ACK method
(XEN) Platform timer is 14.318MHz HPET
(XEN) Allocated console ring of 64 KiB.
(XEN) HVM: ASIDs enabled.
(XEN) SVM: Supported advanced features:
(XEN)  - Nested Page Tables (NPT)
(XEN)  - Last Branch Record (LBR) Virtualisation
(XEN)  - Next-RIP Saved on #VMEXIT
(XEN)  - Pause-Intercept Filter
(XEN) HVM: SVM enabled
(XEN) HVM: Hardware Assisted Paging detected.
(XEN) Brought up 24 CPUs
(XEN) Xenoprofile: AMD IBS detected (0x0000001f)
# ---

It looks like HP DL385g7 BIOS doesn''t offer such IVRS table. Maybe
there
is a kind of workaround to use the IOMMU support (890FX chipset / 6176SE 
CPUs support it) as it is required for "passing through" PCI devices
to
domu domains.

I have tested xen enabled kernels 2.6.32-5, 2.6.39-2 and 3.0.0-rc7 (with 
aufs support)
# ---
root@xen41dom0:~# uname -a
Linux xen41dom0 3.0.0-rc7 #2 SMP Fri Jul 15 02:10:42 CEST 2011 x86_64 
GNU/Linux


I will upload today the last test xen live iso image (s. below) for 
downloads.
(xen41-wheezy-kernel3-amd64-live-gnome-binary-hybrid.iso)

Please take a look for details at: 
http://www.it-infrastrukturen.com/fileadmin/linux/debian-live-xen/README.xen

In the (selfmade) used kernel  3.0.0-rc7 packages I have activated 
IOMMU-debugging however I have to look for a debug howto.

Thank you in advance for your time.

Regards,
Mark

-- 
ms@it-infrastrukturen.org





_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel