thr3ads.net - Xen users - [Xen-users] Unable to pass device to IOMMU [Sep 2010]

If this information is useful, please help other people find it:
Share via:

Michael Maher

2010-Sep-16 17:44 UTC

[Xen-users] Unable to pass device to IOMMU

Hello,

 

I''m hoping someone may be able to push us in the right direction.
I''m trying
to get one of our products to work with the current branch of Xen
(4.0/4.0.0) but I''m hitting a problem.

 

We are currently using Xen 3.3.2 on a Sun Blade 600 chassis (with x6270
blades) this works great.

 

But when we upgrade to Xen 4.0/4.0.1 we get the following message when
starting the DomU:

                Device Cannot be passed to IOMMU

 

I understand from the release note that there were some changes made to the
IOMMU/pciback pass-through in this release, does anyone have any details as
to what might be going wrong?

 

We''re using SLES11/sp1 and I''ve tried with the Novell build
and compiled it
myself. I also have the latest firmware for the chassis and blade from Sun,
all of which show the same problem.

 

Has anyone seen to same thing?

Any help would be greatly appreciated,

 

 

Many Thanks,

Michael

Michael Maher

Adax Europe Ltd

mmaher@adax.co.uk



_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Michael Maher

2010-Sep-21 19:05 UTC

head link

[Xen-users] Re: Unable to pass device to IOMMU

Hello,

I was wondering if anybody had any insight they could lend us on this?

Your help would be very much appreciated,


Many Thanks,
Michael




On Thu, 2010-09-16 at 18:44 +0100, Michael Maher wrote:> Hello,
> 
>  
> 
> I’m hoping someone may be able to push us in the right direction. I’m
> trying to get one of our products to work with the current branch of
> Xen (4.0/4.0.0) but I’m hitting a problem.
> 
>  
> 
> We are currently using Xen 3.3.2 on a Sun Blade 600 chassis (with
> x6270 blades) this works great.
> 
>  
> 
> But when we upgrade to Xen 4.0/4.0.1 we get the following message when
> starting the DomU:
> 
>                 Device Cannot be passed to IOMMU
> 
>  
> 
> I understand from the release note that there were some changes made
> to the IOMMU/pciback pass-through in this release, does anyone have
> any details as to what might be going wrong?
> 
>  
> 
> We’re using SLES11/sp1 and I’ve tried with the Novell build and
> compiled it myself. I also have the latest firmware for the chassis
> and blade from Sun, all of which show the same problem.
> 
>  
> 
> Has anyone seen to same thing?
> 
> Any help would be greatly appreciated,
> 
>  
> 
>  
> 
> Many Thanks,
> 
> Michael
> 
> 
> Michael Maher
> 
> Adax Europe Ltd
> 
> mmaher@adax.co.uk
> 
> 

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Satya Prasad Nemana

2010-Sep-22 08:33 UTC

head link

[Xen-users] Unable to pass device to IOMMU

Hi

I''m hoping someone may be able to push us in the right direction.
I''m trying to get one of our products to work with the current branch
of
Xen (4.0/4.0.0) but I''m hitting a problem.
We are currently using Xen 3.3.2 on a Sun Blade 600 chassis (with x6270 
blades) this works great.
But when we upgrade to Xen 4.0/4.0.1 we get the following message when 
starting the DomU:

                 Device Cannot be passed to IOMMU

I understand from the release note that there were some changes made to 
the IOMMU/pciback pass-through in this release, does anyone have any 
details as to what might be going wrong?
We''re using SLES11/sp1 and I''ve tried with the Novell build
and compiled
it myself. I also have the latest firmware for the chassis and blade 
from Sun, all of which show the same problem.

Has anyone seen to same thing?

Any help would be greatly appreciated

Thanks in advance for your help.

Best Regards,
Satya



_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Todd Deshane

2010-Sep-23 13:23 UTC

head link

Re: [Xen-users] Unable to pass device to IOMMU

Hi Michael,

On Thu, Sep 16, 2010 at 1:44 PM, Michael Maher <mmaher@adax.co.uk>
wrote:> Hello,
>
>
>
> I’m hoping someone may be able to push us in the right direction. I’m
trying
> to get one of our products to work with the current branch of Xen
> (4.0/4.0.0) but I’m hitting a problem.
>
>
>
> We are currently using Xen 3.3.2 on a Sun Blade 600 chassis (with x6270
> blades) this works great.
>
>
>
> But when we upgrade to Xen 4.0/4.0.1 we get the following message when
> starting the DomU:
>
>                 Device Cannot be passed to IOMMU
>
>
>
> I understand from the release note that there were some changes made to the
> IOMMU/pciback pass-through in this release, does anyone have any details as
> to what might be going wrong?
>
>
>
> We’re using SLES11/sp1 and I’ve tried with the Novell build and compiled it
> myself. I also have the latest firmware for the chassis and blade from Sun,
> all of which show the same problem.
>
Can you provide more detailed information? See for example a recent
post on PCI Passthrough and/or search xen.markmail.org for PCI
Passthrough and IOMMU.

Did it work before? You are implying that some change made it not
work. If you have evidence that is the case, the devel list is the
place to let them know and they can try to walk you through what
changed and try to solve it that way.

Hope that helps,
Todd

-- 
Todd Deshane
http://todddeshane.net
http://runningxen.com

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Michael Maher

2010-Sep-24 16:31 UTC

head link

Re: [Xen-users] Unable to pass device to IOMMU

Hi Todd,

Thanks for getting back to me. The situation is that it works with Xen 
3.3.2 but not with Xen 4.0.1, exactly the same setup and configuration.

I have just seen another post very similar to this too. What I was 
expecting was that there was a subtle config change between the versions 
that I might have missed, but after re-reading the documentation I
can''t
see anything.

I was really just wanting to see how the handling of the PCI pass-back 
had changed between the two version, which might push me in the right 
direction to be looking in. Any ideas?


Many Thanks,
Michael



On 23/09/2010 14:23, Todd Deshane wrote:> Hi Michael,
>
> On Thu, Sep 16, 2010 at 1:44 PM, Michael Maher<mmaher@adax.co.uk> 
wrote:
>> Hello,
>>
>>
>>
>> I’m hoping someone may be able to push us in the right direction. I’m
trying
>> to get one of our products to work with the current branch of Xen
>> (4.0/4.0.0) but I’m hitting a problem.
>>
>>
>>
>> We are currently using Xen 3.3.2 on a Sun Blade 600 chassis (with x6270
>> blades) this works great.
>>
>>
>>
>> But when we upgrade to Xen 4.0/4.0.1 we get the following message when
>> starting the DomU:
>>
>>                  Device Cannot be passed to IOMMU
>>
>>
>>
>> I understand from the release note that there were some changes made to
the
>> IOMMU/pciback pass-through in this release, does anyone have any
details as
>> to what might be going wrong?
>>
>>
>>
>> We’re using SLES11/sp1 and I’ve tried with the Novell build and
compiled it
>> myself. I also have the latest firmware for the chassis and blade from
Sun,
>> all of which show the same problem.
>>
> Can you provide more detailed information? See for example a recent
> post on PCI Passthrough and/or search xen.markmail.org for PCI
> Passthrough and IOMMU.
>
> Did it work before? You are implying that some change made it not
> work. If you have evidence that is the case, the devel list is the
> place to let them know and they can try to walk you through what
> changed and try to solve it that way.
>
> Hope that helps,
> Todd
>
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Adam Moss

2010-Oct-21 15:16 UTC

head link

[Xen-users] Re: Unable to pass device to IOMMU

Hi all,

I have worked with Michael on this problem, here are some further 
details that may help troubleshoot the issue:

The error seen when starting DomU is in fact "Failed to assign device to 
IOMMU".

We''re using the following system setup for our Xen environment:

SunBlade 6000 chassis
SunBlade X6270 blade
SuSE ES11 SP1 64bit Dom0 (Xen 4.0.0 - Kernel 2.6.32.12-0.7-xen )
SuSE ES11 SP1 64bit DomU

The lspci -vt output at Dom0 looks like this:

###
-[0000:00]-+-00.0  Intel Corporation 5520 I/O Hub to ESI Port
            +-01.0-[0000:01-06]----00.0  LSI Logic / Symbios Logic 
SAS1068E PCI-Express Fusion-MPT SAS
            +-03.0-[0000:07-0c]--
            +-05.0-[0000:0d-12]----00.0-[0000:0e]----00.0  Freescale 
Semiconductor Inc Device 0091
            +-07.0-[0000:13-18]--
            +-09.0-[0000:19-1e]--
            +-13.0  Intel Corporation 5520/5500/X58 I/O Hub I/OxAPIC 
Interrupt Controller
            +-14.0  Intel Corporation 5520/5500/X58 I/O Hub System 
Management Registers
            +-14.1  Intel Corporation 5520/5500/X58 I/O Hub GPIO and 
Scratch Pad Registers
            +-14.2  Intel Corporation 5520/5500/X58 I/O Hub Control 
Status and RAS Registers
            +-14.3  Intel Corporation 5520/5500/X58 I/O Hub Throttle 
Registers
            +-16.0  Intel Corporation 5520/5500/X58 Chipset QuickData 
Technology Device
            +-16.1  Intel Corporation 5520/5500/X58 Chipset QuickData 
Technology Device
            +-16.2  Intel Corporation 5520/5500/X58 Chipset QuickData 
Technology Device
            +-16.3  Intel Corporation 5520/5500/X58 Chipset QuickData 
Technology Device
            +-16.4  Intel Corporation 5520/5500/X58 Chipset QuickData 
Technology Device
            +-16.5  Intel Corporation 5520/5500/X58 Chipset QuickData 
Technology Device
            +-16.6  Intel Corporation 5520/5500/X58 Chipset QuickData 
Technology Device
            +-16.7  Intel Corporation 5520/5500/X58 Chipset QuickData 
Technology Device
            +-1a.0  Intel Corporation 82801JI (ICH10 Family) USB UHCI 
Controller #4
            +-1a.1  Intel Corporation 82801JI (ICH10 Family) USB UHCI 
Controller #5
            +-1a.2  Intel Corporation 82801JI (ICH10 Family) USB UHCI 
Controller #6
            +-1a.7  Intel Corporation 82801JI (ICH10 Family) USB2 EHCI 
Controller #2
            +-1c.0-[0000:1f]--+-00.0  Intel Corporation 82575EB Gigabit 
Network Connection
            |                 \-00.1  Intel Corporation 82575EB Gigabit 
Network Connection
            +-1d.0  Intel Corporation 82801JI (ICH10 Family) USB UHCI 
Controller #1
            +-1d.1  Intel Corporation 82801JI (ICH10 Family) USB UHCI 
Controller #2
            +-1d.2  Intel Corporation 82801JI (ICH10 Family) USB UHCI 
Controller #3
            +-1d.7  Intel Corporation 82801JI (ICH10 Family) USB2 EHCI 
Controller #1
            +-1e.0-[0000:20]----05.0  ASPEED Technology, Inc. ASPEED 
Graphics Family
            +-1f.0  Intel Corporation 82801JIR (ICH10R) LPC Interface 
Controller
            +-1f.2  Intel Corporation 82801JI (ICH10 Family) SATA AHCI 
Controller
            \-1f.3  Intel Corporation 82801JI (ICH10 Family) SMBus 
Controller
###

The following commands are used to hide the single device (0000:0e:00.0) 
we want to pass back to DomU:

###
modprobe pciback

echo -n "0000:0e:00.0" > /sys/bus/pci/drivers/pciback/new_slot
echo -n "0000:0e:00.0" > /sys/bus/pci/drivers/pciback/bind
###

After this the device is shown correctly in the assignable devices list.

###
SunBlade2:/etc/xen/vm # xm pci-list-assignable-devices
0000:0e:00.0
###

The DomU is then started/created using the following configuration file:

###
import os, re
arch = os.uname()[4]
if re.search(''64'', arch):
     arch_libdir = ''lib64''
else:
     arch_libdir = ''lib''
kernel          = "/usr/lib/xen/boot/hvmloader"
builder         = ''hvm''
device_model    = ''/usr/'' + arch_libdir +
''/xen/bin/qemu-dm''
name            = "node1"
vcpus           =  1
memory          =  1024
pae             =  1
acpi            =  1
apic            =  1

boot            = "dc"
vif             =[''mac=00:11:22:33:00:04, bridge=br0,
model=e1000'']
disk            =[''phy:sda5,hda,w'']

on_poweroff     = ''destroy''
on_reboot       = ''restart''
on_crash        = ''restart''

snapshot        =  0
sdl             =  0
opengl          =  1
vnc             =  1
vnclisten       = "127.0.0.1"
vncdisplay      =  4
vncunused       =  0
vncpasswd       = ''''
nographic       =  0
stdvga          =  0
serial          = ''pty''

pci           = [''0000:0e:00.0'']
###

After running the create command the "failed to assign to IOMMU" error
is seen:

###
SunBlade2:/etc/xen/vm # xm create node1.conf
Using config file "./node1.conf".
Error: Failed to assign device to IOMMU 
(0000:0e:00.0@100,msitranslate=1,power_mgmt=0)
###

We have enabled the debug traces (iommu=verbose loglvl=all 
guest_loglvl=all) which show the following at the point of failure 
(Please ignore the time stamps):

/var/log/messages
--------------------------

Mar 10 02:57:05 SunBlade2 kernel: [ 1495.751322] 
(cdrom_add_media_watch() 
file=/usr/src/packages/BUILD/kernel-xen-2.6.32.12/linux-2.6.32/drivers/xen/blkback/cdrom.c,
line=108) nodename:backend/vbd/2/768
Mar 10 02:57:05 SunBlade2 kernel: [ 1495.751327] (cdrom_is_type() 
file=/usr/src/packages/BUILD/kernel-xen-2.6.32.12/linux-2.6.32/drivers/xen/blkback/cdrom.c,
line=95) type:0
Mar 10 02:57:05 SunBlade2 kernel: [ 1495.936015] pcieport 0000:00:05.0: 
AER: Uncorrected (Non-Fatal) error received: id=0028
Mar 10 02:57:05 SunBlade2 kernel: [ 1496.023732] pcieport 0000:00:05.0: 
PCIE Bus Error: severity=Uncorrected (Non-Fatal), type=Transaction 
Layer, id=0028(Requester ID)
Mar 10 02:57:05 SunBlade2 kernel: [ 1496.023737] pcieport 
0000:00:05.0:   device [8086:340c] error status/mask=00004000/00000000
Mar 10 02:57:05 SunBlade2 kernel: [ 1496.023742] pcieport 
0000:00:05.0:    [14] Completion Timeout     (First)
Mar 10 02:57:05 SunBlade2 kernel: [ 1496.024193] pcieport 0000:00:05.0: 
broadcast error_detected message
Mar 10 02:57:05 SunBlade2 kernel: [ 1496.024198] pciback 0000:0e:00.0: 
pciback device is not found/assigned
Mar 10 02:57:05 SunBlade2 kernel: [ 1496.024201] pcieport 0000:00:05.0: 
broadcast mmio_enabled message
Mar 10 02:57:05 SunBlade2 kernel: [ 1496.024203] pciback 0000:0e:00.0: 
pciback device is not found/assigned
Mar 10 02:57:05 SunBlade2 kernel: [ 1496.024205] pcieport 0000:00:05.0: 
broadcast resume message
Mar 10 02:57:05 SunBlade2 kernel: [ 1496.024208] pciback 0000:0e:00.0: 
pciback device is not found/assigned
Mar 10 02:57:05 SunBlade2 kernel: [ 1496.024213] pcieport 0000:00:05.0: 
AER driver successfully recovered
Mar 10 02:57:05 SunBlade2 kernel: [ 1496.024216] pcieport 0000:00:05.0: 
AER: Uncorrected (Non-Fatal) error received: id=0028
Mar 10 02:57:05 SunBlade2 kernel: [ 1496.024305] pcieport 0000:00:05.0: 
can''t find device of ID0028
Mar 10 02:57:05 SunBlade2 kernel: [ 1496.024307] pcieport 0000:00:05.0: 
AER: Uncorrected (Non-Fatal) error received: id=0028
Mar 10 02:57:05 SunBlade2 kernel: [ 1496.024395] pcieport 0000:00:05.0: 
can''t find device of ID0028
Mar 10 02:57:05 SunBlade2 kernel: [ 1495.587191] device tap2.0 entered 
promiscuous mode
Mar 10 02:57:05 SunBlade2 kernel: [ 1495.587205] br0: port 3(tap2.0) 
entering forwarding state
Mar 10 02:57:05 SunBlade2 kernel: [ 1495.771897] br0: port 3(tap2.0) 
entering disabled state
Mar 10 02:57:06 SunBlade2 kernel: [ 1495.959814] device tap2.0 left 
promiscuous mode

/var/log/xen/xend.log
----------------------------

[2011-03-10 02:58:42 5613] DEBUG (XendDomainInfo:825) 
XendDomainInfo.hvm_pci_device_insert_dev: 
0000:0e:00.0@100,msitranslate=1,power_mgmt=0
[2011-03-10 02:58:42 5613] ERROR (XendDomainInfo:2957) 
XendDomainInfo.initDomain: exception occurred
Traceback (most recent call last):
File "/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py", 
line 2949, in _initDomain
     self._createDevices()
   File
"/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py",
line 2440, in _createDevices
     self.pci_device_configure_boot()
   File
"/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py",
line 641, in pci_device_configure_boot
     self.pci_device_configure(dev_sxp, first_dev = first)
   File
"/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py",
line 934, in pci_device_configure
     vdevfn = self.hvm_pci_device_insert(dev_config)
   File
"/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py",
line 800, in hvm_pci_device_insert
     return self.hvm_pci_device_insert_dev(new_dev)
   File
"/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py",
line 828, in hvm_pci_device_insert_dev
     raise VmError("Failed to assign device to IOMMU (%s)" % bdf_str)
VmError: Failed to assign device to IOMMU 
(0000:0e:00.0@100,msitranslate=1,power_mgmt=0)
[2011-03-10 02:58:42 5613] ERROR (XendDomainInfo:486) VM start failed
Traceback (most recent call last):
   File
"/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py",
line 472, in start
     XendTask.log_progress(31, 60, self._initDomain)
   File "/usr/lib64/python2.6/site-packages/xen/xend/XendTask.py",
line
209, in log_progress
     retval = func(*args, **kwds)
   File
"/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py",
line 2960, in _initDomain
     raise exn
VmError: Failed to assign device to IOMMU 
(0000:0e:00.0@100,msitranslate=1,power_mgmt=0)


xm dmes
-------------

(XEN) *** Serial input -> DOM0 (type ''CTRL-a'' three times
to switch
input to Xen)
(XEN) Freed 184kB init memory.
(XEN) mtrr: type mismatch for fb000000,800000 old: write-back new: 
write-combining
(XEN) [VT-D]iommu.c:1461: d0:PCI: unmap bdf = e:0.0
(XEN) [VT-D]iommu.c:1332: d1:PCI: map bdf = e:0.0
(XEN) domctl.c:853:d0 XEN_DOMCTL_assign_device: assign device (e:0.0) 
failed
(XEN) event_channel.c:192:d0 EVTCHNOP failure: domain 1, error -22
(XEN) [VT-D]iommu.c:1461: d0:PCI: unmap bdf = e:0.0
(XEN) [VT-D]iommu.c:1332: d2:PCI: map bdf = e:0.0
(XEN) domctl.c:853:d0 XEN_DOMCTL_assign_device: assign device (e:0.0) 
failed
(XEN) [VT-D]iommu.c:1461: d0:PCI: unmap bdf = e:0.0
(XEN) [VT-D]iommu.c:1332: d3:PCI: map bdf = e:0.0
(XEN) domctl.c:853:d0 XEN_DOMCTL_assign_device: assign device (e:0.0) 
failed
(XEN) [VT-D]iommu.c:1461: d0:PCI: unmap bdf = e:0.0
(XEN) [VT-D]iommu.c:1332: d4:PCI: map bdf = e:0.0
(XEN) domctl.c:853:d0 XEN_DOMCTL_assign_device: assign device (e:0.0) 
failed

The full trace (xm dmes, xend.log) output and other relevant information 
from the system is attached.

Things to note
==========
1) The problem still occurs if we try and hotplug the device using "xm 
pci-attach ..."
2) If the same hardware and software configuration is used with Xen 
3.3.1 at Dom0 (SLES11) the problem is not seen.
3) We have compiled Xen 4.0.1 for SLES11 SP1 at Dom0 and still see the 
same problem.
4) We have tried passing back the PLX bridge device "0d:00.0"
associated
with the device as well as the device itself however we then see the 
error "Error: pci: PCI Backend and pci-stub don''t own device 
0000:0d:00.0" during DomU startup even though 0d:00.0 is shown in the 
assignable device list.

Looking at recent posts this problems looks very similar to the one in 
subject "PCI Passthrough Problems/Questions" posted today.

Any pointers at this stage would be great.
Thanks,
Adam

----------
Adam Moss

Adax Europe Ltd
Email : Adam.Moss@adax.co.uk

On 21/09/2010 20:05, Michael Maher wrote:> Hello,
>
> I was wondering if anybody had any insight they could lend us on this?
>
> Your help would be very much appreciated,
>
>
> Many Thanks,
> Michael
>
>
>
>
> On Thu, 2010-09-16 at 18:44 +0100, Michael Maher wrote:
>> Hello,
>>
>>
>>
>> I’m hoping someone may be able to push us in the right direction. I’m
>> trying to get one of our products to work with the current branch of
>> Xen (4.0/4.0.0) but I’m hitting a problem.
>>
>>
>>
>> We are currently using Xen 3.3.2 on a Sun Blade 600 chassis (with
>> x6270 blades) this works great.
>>
>>
>>
>> But when we upgrade to Xen 4.0/4.0.1 we get the following message when
>> starting the DomU:
>>
>>                  Device Cannot be passed to IOMMU
>>
>>
>>
>> I understand from the release note that there were some changes made
>> to the IOMMU/pciback pass-through in this release, does anyone have
>> any details as to what might be going wrong?
>>
>>
>>
>> We’re using SLES11/sp1 and I’ve tried with the Novell build and
>> compiled it myself. I also have the latest firmware for the chassis
>> and blade from Sun, all of which show the same problem.
>>
>>
>>
>> Has anyone seen to same thing?
>>
>> Any help would be greatly appreciated,
>>
>>
>>
>>
>>
>> Many Thanks,
>>
>> Michael
>>
>>
>> Michael Maher
>>
>> Adax Europe Ltd
>>
>> mmaher@adax.co.uk
>>
>>


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Reasonably Related Threads

Search for more reasonably related threads

Xen users - Sep 2010 - Unable to pass device to IOMMU

[Xen-users] Unable to pass device to IOMMU

[Xen-users] Re: Unable to pass device to IOMMU

[Xen-users] Unable to pass device to IOMMU

Re: [Xen-users] Unable to pass device to IOMMU

Re: [Xen-users] Unable to pass device to IOMMU

[Xen-users] Re: Unable to pass device to IOMMU

Reasonably Related Threads