thr3ads.net - Xen users - [Xen-users] VT-d passthrough

If this information is useful, please help other people find it:
Share via:

Thomas Prause

2010-Mar-04 21:47 UTC

[Xen-users] VT-d passthrough - NMI crashes system

I want to pass some FC adapters to the domU following the steps
described in the VTdHowTo -> http://wiki.xensource.com/xenwiki/VTdHowTo
using XEN 3.3.1 (SLES 11)

dom0 seems to be configured properly:
# xm pci-list-assignable-devices
0000:15:00.0
0000:24:00.0
0000:1f:00.0
0000:1a:00.0
#

But after starting a domU with one adapter assigned it does not show up
to lspci in the domU. Even ''xm pci-list-assignable-devices''
does not
show this one adapter any longer.

After some seconds or minutes the whole system crashes, tries to reboot,
fails with NMI errors in the BIOS log and finally stops in the BIOS with
an error that 3 consecutive boot failures have occured and now the
default settings will be used :-o

System can then be booted and the log of the dom0 shows:

Mar  2 12:52:27 manni logger: /etc/xen/scripts/block: Writing
backend/vbd/1/5632/hotplug-status connected to xenstore.
Mar  2 12:52:27 manni kernel: vbd vbd-1-5632: 2 creating vbd structure
Mar  2 12:52:36 manni kernel: vif1.0: no IPv6 routers present
Mar  2 12:52:41 manni kernel: blkback: ring-ref 8, event-channel 8,
protocol 1 (x86_64-abi)
Mar  2 12:52:41 manni kernel: blkback: ring-ref 9, event-channel 9,
protocol 1 (x86_64-abi)
Mar  2 12:52:46 manni kernel: Uhhuh. NMI received for unknown reason 00.
Mar  2 12:52:46 manni kernel: Do you have a strange power saving mode
enabled?
Mar  2 12:52:46 manni kernel: Dazed and confused, but trying to continue
Mar  2 12:53:33 manni sshd[7298]: Accepted keyboard-interactive/pam for
root from xx.xx.xx.xx port 42896 ssh2
Mar  3 16:17:26 manni syslog-ng[3708]: syslog-ng starting up;
version=''2.0.9''

I assumed some sort of IRQ conflict and checked:
# dmesg | grep GSI
IOAPIC[0]: apic_id 8, version 0, address 0xfec00000, GSI 0-23
IOAPIC[1]: apic_id 9, version 0, address 0xfec80000, GSI 24-47
pci 0000:00:01.0: PCI INT A -> GSI 28 (level, low) -> IRQ 28
pci 0000:00:02.0: PCI INT A -> GSI 29 (level, low) -> IRQ 29
pci 0000:00:03.0: PCI INT A -> GSI 24 (level, low) -> IRQ 24
pci 0000:00:05.0: PCI INT A -> GSI 26 (level, low) -> IRQ 26
pci 0000:00:07.0: PCI INT A -> GSI 30 (level, low) -> IRQ 30
pci 0000:00:09.0: PCI INT A -> GSI 32 (level, low) -> IRQ 32
pci 0000:00:1c.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
pci 0000:00:1c.4: PCI INT A -> GSI 16 (level, low) -> IRQ 16
pci 0000:06:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
ata_piix 0000:00:1f.2: PCI INT A -> GSI 16 (level, low) -> IRQ 16
ata_piix 0000:00:1f.5: PCI INT C -> GSI 21 (level, low) -> IRQ 21
megaraid_sas 0000:01:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
pciback 0000:24:00.0: PCI INT A -> GSI 32 (level, low) -> IRQ 32
pciback 0000:1f:00.0: PCI INT A -> GSI 30 (level, low) -> IRQ 30
pciback 0000:1a:00.0: PCI INT A -> GSI 26 (level, low) -> IRQ 26
pciback 0000:15:00.0: PCI INT A -> GSI 24 (level, low) -> IRQ 24
 sda:<6>ehci_hcd 0000:00:1a.7: PCI INT C -> GSI 19 (level, low) ->
IRQ 19
ehci_hcd 0000:00:1d.7: PCI INT A -> GSI 17 (level, low) -> IRQ 17
uhci_hcd 0000:00:1a.0: PCI INT A -> GSI 17 (level, low) -> IRQ 17
uhci_hcd 0000:00:1a.1: PCI INT B -> GSI 18 (level, low) -> IRQ 18
uhci_hcd 0000:00:1d.0: PCI INT A -> GSI 17 (level, low) -> IRQ 17
uhci_hcd 0000:00:1d.1: PCI INT B -> GSI 18 (level, low) -> IRQ 18
uhci_hcd 0000:00:1d.2: PCI INT C -> GSI 19 (level, low) -> IRQ 19
bnx2 0000:0b:00.0: PCI INT A -> GSI 28 (level, low) -> IRQ 28
bnx2 0000:0b:00.1: PCI INT B -> GSI 40 (level, low) -> IRQ 40
i801_smbus 0000:00:1f.3: PCI INT B -> GSI 22 (level, low) -> IRQ 22
#

The IRQs for the FC adapters are used by other devices. These are PCI
bridges:
# lspci
00:00.0 Host bridge: Intel Corporation X58 I/O Hub to ESI Port (rev 13)
00:01.0 PCI bridge: Intel Corporation X58 I/O Hub PCI Express Root Port
1 (rev 13)
00:02.0 PCI bridge: Intel Corporation X58 I/O Hub PCI Express Root Port
2 (rev 13)
00:03.0 PCI bridge: Intel Corporation X58 I/O Hub PCI Express Root Port
3 (rev 13)
00:05.0 PCI bridge: Intel Corporation X58 I/O Hub PCI Express Root Port
5 (rev 13)
00:07.0 PCI bridge: Intel Corporation X58 I/O Hub PCI Express Root Port
7 (rev 13)
00:09.0 PCI bridge: Intel Corporation X58 I/O Hub PCI Express Root Port
9 (rev 13)

So I tried to passthrough the corresponding PCI with the FC adapter. But
this failed as well:
 # cat /sys/bus/pci/drivers/pciback/slots
0000:15:00.0
0000:1a:00.0
0000:1f:00.0
0000:24:00.0
0000:00:03.0
 # xm create /etc/xen/vm/sid-sles11 pci=00:00:03.0 pci=00:15:00.0
Using config file "/etc/xen/vm/sid-sles11".
Error: pci: PCI Backend does not own device 0000:00:03.0
See the pciback.hide kernel command-line parameter or
bind your slot/device to the PCI backend using sysfs
 #

Am I on the wrong track with the PCI bridge? Or is this some sort of
hardware issue (IBM X3650 M2 with latest firmware)?

Any thoughts are highly appreciated.
Thanks.

Thomas

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Pasi Kärkkäinen

2010-Mar-05 07:32 UTC

head link

Re: [Xen-users] VT-d passthrough - NMI crashes system

On Thu, Mar 04, 2010 at 10:47:09PM +0100, Thomas Prause
wrote:> I want to pass some FC adapters to the domU following the steps
> described in the VTdHowTo -> http://wiki.xensource.com/xenwiki/VTdHowTo
> using XEN 3.3.1 (SLES 11)
> 
iirc SLES11 has also Xen 3.4 available.. try updating to it? Also make 
sure you''re running the latest dom0 kernel.

--  Pasi
> dom0 seems to be configured properly:
> # xm pci-list-assignable-devices
> 0000:15:00.0
> 0000:24:00.0
> 0000:1f:00.0
> 0000:1a:00.0
> #
> 
> But after starting a domU with one adapter assigned it does not show up
> to lspci in the domU. Even ''xm
pci-list-assignable-devices'' does not
> show this one adapter any longer.
> 
> After some seconds or minutes the whole system crashes, tries to reboot,
> fails with NMI errors in the BIOS log and finally stops in the BIOS with
> an error that 3 consecutive boot failures have occured and now the
> default settings will be used :-o
> 
> System can then be booted and the log of the dom0 shows:
> 
> Mar  2 12:52:27 manni logger: /etc/xen/scripts/block: Writing
> backend/vbd/1/5632/hotplug-status connected to xenstore.
> Mar  2 12:52:27 manni kernel: vbd vbd-1-5632: 2 creating vbd structure
> Mar  2 12:52:36 manni kernel: vif1.0: no IPv6 routers present
> Mar  2 12:52:41 manni kernel: blkback: ring-ref 8, event-channel 8,
> protocol 1 (x86_64-abi)
> Mar  2 12:52:41 manni kernel: blkback: ring-ref 9, event-channel 9,
> protocol 1 (x86_64-abi)
> Mar  2 12:52:46 manni kernel: Uhhuh. NMI received for unknown reason 00.
> Mar  2 12:52:46 manni kernel: Do you have a strange power saving mode
> enabled?
> Mar  2 12:52:46 manni kernel: Dazed and confused, but trying to continue
> Mar  2 12:53:33 manni sshd[7298]: Accepted keyboard-interactive/pam for
> root from xx.xx.xx.xx port 42896 ssh2
> Mar  3 16:17:26 manni syslog-ng[3708]: syslog-ng starting up;
> version=''2.0.9''
> 
> I assumed some sort of IRQ conflict and checked:
> # dmesg | grep GSI
> IOAPIC[0]: apic_id 8, version 0, address 0xfec00000, GSI 0-23
> IOAPIC[1]: apic_id 9, version 0, address 0xfec80000, GSI 24-47
> pci 0000:00:01.0: PCI INT A -> GSI 28 (level, low) -> IRQ 28
> pci 0000:00:02.0: PCI INT A -> GSI 29 (level, low) -> IRQ 29
> pci 0000:00:03.0: PCI INT A -> GSI 24 (level, low) -> IRQ 24
> pci 0000:00:05.0: PCI INT A -> GSI 26 (level, low) -> IRQ 26
> pci 0000:00:07.0: PCI INT A -> GSI 30 (level, low) -> IRQ 30
> pci 0000:00:09.0: PCI INT A -> GSI 32 (level, low) -> IRQ 32
> pci 0000:00:1c.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
> pci 0000:00:1c.4: PCI INT A -> GSI 16 (level, low) -> IRQ 16
> pci 0000:06:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
> ata_piix 0000:00:1f.2: PCI INT A -> GSI 16 (level, low) -> IRQ 16
> ata_piix 0000:00:1f.5: PCI INT C -> GSI 21 (level, low) -> IRQ 21
> megaraid_sas 0000:01:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
> pciback 0000:24:00.0: PCI INT A -> GSI 32 (level, low) -> IRQ 32
> pciback 0000:1f:00.0: PCI INT A -> GSI 30 (level, low) -> IRQ 30
> pciback 0000:1a:00.0: PCI INT A -> GSI 26 (level, low) -> IRQ 26
> pciback 0000:15:00.0: PCI INT A -> GSI 24 (level, low) -> IRQ 24
>  sda:<6>ehci_hcd 0000:00:1a.7: PCI INT C -> GSI 19 (level, low)
-> IRQ 19
> ehci_hcd 0000:00:1d.7: PCI INT A -> GSI 17 (level, low) -> IRQ 17
> uhci_hcd 0000:00:1a.0: PCI INT A -> GSI 17 (level, low) -> IRQ 17
> uhci_hcd 0000:00:1a.1: PCI INT B -> GSI 18 (level, low) -> IRQ 18
> uhci_hcd 0000:00:1d.0: PCI INT A -> GSI 17 (level, low) -> IRQ 17
> uhci_hcd 0000:00:1d.1: PCI INT B -> GSI 18 (level, low) -> IRQ 18
> uhci_hcd 0000:00:1d.2: PCI INT C -> GSI 19 (level, low) -> IRQ 19
> bnx2 0000:0b:00.0: PCI INT A -> GSI 28 (level, low) -> IRQ 28
> bnx2 0000:0b:00.1: PCI INT B -> GSI 40 (level, low) -> IRQ 40
> i801_smbus 0000:00:1f.3: PCI INT B -> GSI 22 (level, low) -> IRQ 22
> #
> 
> The IRQs for the FC adapters are used by other devices. These are PCI
> bridges:
> # lspci
> 00:00.0 Host bridge: Intel Corporation X58 I/O Hub to ESI Port (rev 13)
> 00:01.0 PCI bridge: Intel Corporation X58 I/O Hub PCI Express Root Port
> 1 (rev 13)
> 00:02.0 PCI bridge: Intel Corporation X58 I/O Hub PCI Express Root Port
> 2 (rev 13)
> 00:03.0 PCI bridge: Intel Corporation X58 I/O Hub PCI Express Root Port
> 3 (rev 13)
> 00:05.0 PCI bridge: Intel Corporation X58 I/O Hub PCI Express Root Port
> 5 (rev 13)
> 00:07.0 PCI bridge: Intel Corporation X58 I/O Hub PCI Express Root Port
> 7 (rev 13)
> 00:09.0 PCI bridge: Intel Corporation X58 I/O Hub PCI Express Root Port
> 9 (rev 13)
> 
> So I tried to passthrough the corresponding PCI with the FC adapter. But
> this failed as well:
>  # cat /sys/bus/pci/drivers/pciback/slots
> 0000:15:00.0
> 0000:1a:00.0
> 0000:1f:00.0
> 0000:24:00.0
> 0000:00:03.0
>  # xm create /etc/xen/vm/sid-sles11 pci=00:00:03.0 pci=00:15:00.0
> Using config file "/etc/xen/vm/sid-sles11".
> Error: pci: PCI Backend does not own device 0000:00:03.0
> See the pciback.hide kernel command-line parameter or
> bind your slot/device to the PCI backend using sysfs
>  #
> 
> Am I on the wrong track with the PCI bridge? Or is this some sort of
> hardware issue (IBM X3650 M2 with latest firmware)?
> 
> Any thoughts are highly appreciated.
> Thanks.
> 
> Thomas
> 
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Thomas Prause

2010-Mar-05 23:01 UTC

head link

Re: [Xen-users] VT-d passthrough - NMI crashes system

>> I want to pass some FC adapters to the domU following the steps
>> described in the VTdHowTo ->
http://wiki.xensource.com/xenwiki/VTdHowTo
>> using XEN 3.3.1 (SLES 11)
>>
> 
> iirc SLES11 has also Xen 3.4 available.. try updating to it? Also make 
> sure you''re running the latest dom0 kernel.
> 
The SLES 11 update sources do not provide XEN 3.4. But I found some
patches for the XEN 3.3.1 from Novell, After installing these patches
the domU can see the FC adapter. But still the whole system crashes with
the NMI error in the log of dom0.

Is there any indication that XEN 3.4 would fix this issue or is this
more likely a flaw in the hardware/BIOS?

Do I need to pass through the PCI bridge as well?

Thanks for your response
Thomas

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Thomas Prause

2010-Mar-09 22:29 UTC

head link

Re: [Xen-users] VT-d passthrough - NMI crashes system

>> I want to pass some FC adapters to the domU following the steps
>> described in the VTdHowTo ->
http://wiki.xensource.com/xenwiki/VTdHowTo
>> using XEN 3.3.1 (SLES 11)
>>
> 
> iirc SLES11 has also Xen 3.4 available.. try updating to it? Also make 
> sure you''re running the latest dom0 kernel.
> 
> --  Pasi
> 
Compiled XEN 3.4.2 from source and booted with the SLES 11 dom0 kernel
(version by Novel 2.6.27.45-0.1-xen) -> still the same crash.
What do you mean by latest dom0 kernel?
The "2.6.18 updated for Xen 3.4.2"?

Thanks,
Thomas

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Pasi Kärkkäinen

2010-Mar-10 06:26 UTC

head link

Re: [Xen-users] VT-d passthrough - NMI crashes system

On Tue, Mar 09, 2010 at 11:29:49PM +0100, Thomas Prause
wrote:> >> I want to pass some FC adapters to the domU following the steps
> >> described in the VTdHowTo ->
http://wiki.xensource.com/xenwiki/VTdHowTo
> >> using XEN 3.3.1 (SLES 11)
> >>
> > 
> > iirc SLES11 has also Xen 3.4 available.. try updating to it? Also make
> > sure you''re running the latest dom0 kernel.
> > 
> > --  Pasi
> > 
> 
> Compiled XEN 3.4.2 from source and booted with the SLES 11 dom0 kernel
> (version by Novel 2.6.27.45-0.1-xen) -> still the same crash.
>
Ok :( Did you already report this to novell? 
> What do you mean by latest dom0 kernel?
>
The latest version/update from SLES11.
> The "2.6.18 updated for Xen 3.4.2"?
> 
Well you could try grabbing latest linux-2.6.18-xen from
http://xenbits.xen.org/linux-2.6.18-xen.hg and using that to see 
if it makes any difference..

-- Pasi


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Thomas Prause

2010-Apr-09 22:09 UTC

head link

Re: [Xen-users] VT-d passthrough - NMI crashes system

Pasi Kärkkäinen schrieb:> On Tue, Mar 09, 2010 at 11:29:49PM +0100, Thomas Prause wrote:
>>>> I want to pass some FC adapters to the domU following the steps
>>>> described in the VTdHowTo ->
http://wiki.xensource.com/xenwiki/VTdHowTo
>>>> using XEN 3.3.1 (SLES 11)
>>>>
>>> iirc SLES11 has also Xen 3.4 available.. try updating to it? Also
make
>>> sure you''re running the latest dom0 kernel.
>>>
>>> --  Pasi
>>>
>> Compiled XEN 3.4.2 from source and booted with the SLES 11 dom0 kernel
>> (version by Novel 2.6.27.45-0.1-xen) -> still the same crash.
>>
> 
> Ok :( Did you already report this to novell? 
> 
>> What do you mean by latest dom0 kernel?
>>
> 
> The latest version/update from SLES11.
> 
>> The "2.6.18 updated for Xen 3.4.2"?
>>
> 
> Well you could try grabbing latest linux-2.6.18-xen from
> http://xenbits.xen.org/linux-2.6.18-xen.hg and using that to see 
> if it makes any difference..
> 
Sorry for the delayed response. I adressed this issue with Novell
and IBM. It turned out out to be a bug in the firmware of the
FC adapters. Upgrading the firmware from version 2.50A6 to 2.82A3
did fix it. Just in case sombody else has the same issue: the
adapters are Emulex P/N 42C2069.

Thanks,
Thomas

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Xen users - Mar 2010 - VT-d passthrough - NMI crashes system

[Xen-users] VT-d passthrough - NMI crashes system

Re: [Xen-users] VT-d passthrough - NMI crashes system

Re: [Xen-users] VT-d passthrough - NMI crashes system

Re: [Xen-users] VT-d passthrough - NMI crashes system

Re: [Xen-users] VT-d passthrough - NMI crashes system

Re: [Xen-users] VT-d passthrough - NMI crashes system