thr3ads.net - Xen devel - [Xen-devel] pciback bug "secondary bus reset failed" [May 2010]

If this information is useful, please help other people find it:
Share via:

Alex Zeffertt

2010-May-24 15:53 UTC

[Xen-devel] pciback bug "secondary bus reset failed"

Hi,

I''m trying to passthrough a PCI network card to a domU guest, but
I''m hitting a
bug that prevents the IRQ being unbound from dom0.

Here are the steps I am taking:

  * configure Xen to pass $pci (0000:06:00.0) through to guest VM
  * unbind from current driver in dom0 and rebind to pciback:

      echo -n $pci > /sys/bus/pci/devices/$pci/driver/unbind
      modprobe pciback
      echo -n $pci > /sys/bus/pci/drivers/pciback/new_slot
      echo -n $pci > /sys/bus/pci/drivers/pciback/bind

  * start guest VM

I then get the following on the dom0 console

   pciback 0000:06:00.0: secondary bus reset failed for device - all functions \
   need to be co-assigned - err: 6
   pciback 0000:06:00.0: FLR functionality not supported; attempts to use \
   secondary bus reset unsuccessful;
   pciback 0000:06:00.0: FLR not performed for device

I don''t understand the above, but the upshot is that
pirq_guest_unbind() does
not get called.  Later when the guest does an "ifconfig up" the IRQ
gets sent to
dom0 instead of the domU, and this causes the following to appear on the dom0 
console:

   irq 21: nobody cared (try booting with the "irqpoll" option)
   Pid: 0, comm: swapper Tainted: G         
2.6.27.42-0.1.1.xs5.6.0.44.111158xen
    #1
    [<c0154517>] __report_bad_irq+0x27/0x90
    [<c015487c>] note_interrupt+0x2fc/0x330
    [<f009e92d>] ? usb_hcd_irq+0x4d/0xe0 [usbcore]
    [<c0153951>] ? handle_IRQ_event+0x31/0x90
    [<c0155204>] handle_level_irq+0xe4/0x110
    [<c0107733>] do_IRQ+0x43/0x90

I have ioapic_ack=old on the Xen command line.  This causes Xen to EOI the 
interrupt straight away instead of waiting for a guest to EOI and forwarding 
that on to the IO APIC.  If I remove this then since no guest handles the 
interrupt, all lower priority IRQs get blocked forever and I lose access to my 
hard disk!

Any help would be much appreciated.

Regards,

Alex Zeffertt

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Pasi Kärkkäinen

2010-May-24 16:30 UTC

head link

Re: [Xen-devel] pciback bug "secondary bus reset failed"

On Mon, May 24, 2010 at 04:53:01PM +0100, Alex Zeffertt
wrote:> Hi,
>
> I''m trying to passthrough a PCI network card to a domU guest, but
I''m
> hitting a bug that prevents the IRQ being unbound from dom0.
>
> Here are the steps I am taking:
>
>  * configure Xen to pass $pci (0000:06:00.0) through to guest VM
>  * unbind from current driver in dom0 and rebind to pciback:
>
>      echo -n $pci > /sys/bus/pci/devices/$pci/driver/unbind
>      modprobe pciback
>      echo -n $pci > /sys/bus/pci/drivers/pciback/new_slot
>      echo -n $pci > /sys/bus/pci/drivers/pciback/bind
>
>  * start guest VM
>
> I then get the following on the dom0 console
>
>   pciback 0000:06:00.0: secondary bus reset failed for device - all
functions \
>   need to be co-assigned - err: 6
>   pciback 0000:06:00.0: FLR functionality not supported; attempts to use \
>   secondary bus reset unsuccessful;
>   pciback 0000:06:00.0: FLR not performed for device
>
Please check this wiki page:
http://wiki.xensource.com/xenwiki/XenPCIpassthrough

It has a chapter about the "all functions need to be co-assigned"
error..

-- Pasi
> I don''t understand the above, but the upshot is that
pirq_guest_unbind()
> does not get called.  Later when the guest does an "ifconfig up"
the IRQ
> gets sent to dom0 instead of the domU, and this causes the following to 
> appear on the dom0 console:
>
>   irq 21: nobody cared (try booting with the "irqpoll" option)
>   Pid: 0, comm: swapper Tainted: G         
2.6.27.42-0.1.1.xs5.6.0.44.111158xen
>    #1
>    [<c0154517>] __report_bad_irq+0x27/0x90
>    [<c015487c>] note_interrupt+0x2fc/0x330
>    [<f009e92d>] ? usb_hcd_irq+0x4d/0xe0 [usbcore]
>    [<c0153951>] ? handle_IRQ_event+0x31/0x90
>    [<c0155204>] handle_level_irq+0xe4/0x110
>    [<c0107733>] do_IRQ+0x43/0x90
>
> I have ioapic_ack=old on the Xen command line.  This causes Xen to EOI 
> the interrupt straight away instead of waiting for a guest to EOI and 
> forwarding that on to the IO APIC.  If I remove this then since no guest 
> handles the interrupt, all lower priority IRQs get blocked forever and I 
> lose access to my hard disk!
>
> Any help would be much appreciated.
>
> Regards,
>
> Alex Zeffertt
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Alex Zeffertt

2010-May-24 16:53 UTC

head link

Re: [Xen-devel] pciback bug "secondary bus reset failed"

Pasi Kärkkäinen wrote:> On Mon, May 24, 2010 at 04:53:01PM +0100, Alex Zeffertt wrote:
>> Hi,
>>
>> I''m trying to passthrough a PCI network card to a domU guest,
but I''m
>> hitting a bug that prevents the IRQ being unbound from dom0.
>>
>> Here are the steps I am taking:
>>
>>  * configure Xen to pass $pci (0000:06:00.0) through to guest VM
>>  * unbind from current driver in dom0 and rebind to pciback:
>>
>>      echo -n $pci > /sys/bus/pci/devices/$pci/driver/unbind
>>      modprobe pciback
>>      echo -n $pci > /sys/bus/pci/drivers/pciback/new_slot
>>      echo -n $pci > /sys/bus/pci/drivers/pciback/bind
>>
>>  * start guest VM
>>
>> I then get the following on the dom0 console
>>
>>   pciback 0000:06:00.0: secondary bus reset failed for device - all
functions \
>>   need to be co-assigned - err: 6
>>   pciback 0000:06:00.0: FLR functionality not supported; attempts to
use \
>>   secondary bus reset unsuccessful;
>>   pciback 0000:06:00.0: FLR not performed for device
>>
> 
> Please check this wiki page:
> http://wiki.xensource.com/xenwiki/XenPCIpassthrough
> 
> It has a chapter about the "all functions need to be co-assigned"
error..
> 
> -- Pasi
Hi Pasi,

Thanks for your reply.  I think I understand what is happening now.  My card is 
too old to have the Function Level Reset functionality.  This leaves the 
following options:

  1. Assign all the cards that share the PCI bus (0000:06) to the same guest. 
This enables the FLR functionality to be replaced by the bridge''s
"Secondary Bus
Reset" functionality

  - or -

  2. Apply the patch referred to on 
http://wiki.xensource.com/xenwiki/XenPCIpassthrough to bypass the FLR

If I can''t do number #1 because some cards are in use by dom0 then I
guess #2 is
the only option.  But does #2 require that dom0 has never accessed the card? 
Also, what if you reboot the domU - does the fact that you can''t do an
FLR on
the card mean that there will be unpredictable results the 2nd time domU tries 
to initialise it?

Thanks again,

Alex

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Weidong Han

2010-May-25 01:03 UTC

head link

Re: [Xen-devel] pciback bug "secondary bus reset failed"

Alex Zeffertt wrote:> Pasi Kärkkäinen wrote:
>   
>> On Mon, May 24, 2010 at 04:53:01PM +0100, Alex Zeffertt wrote:
>>     
>>> Hi,
>>>
>>> I''m trying to passthrough a PCI network card to a domU
guest, but I''m
>>> hitting a bug that prevents the IRQ being unbound from dom0.
>>>
>>> Here are the steps I am taking:
>>>
>>>  * configure Xen to pass $pci (0000:06:00.0) through to guest VM
>>>  * unbind from current driver in dom0 and rebind to pciback:
>>>
>>>      echo -n $pci > /sys/bus/pci/devices/$pci/driver/unbind
>>>      modprobe pciback
>>>      echo -n $pci > /sys/bus/pci/drivers/pciback/new_slot
>>>      echo -n $pci > /sys/bus/pci/drivers/pciback/bind
>>>
>>>  * start guest VM
>>>
>>> I then get the following on the dom0 console
>>>
>>>   pciback 0000:06:00.0: secondary bus reset failed for device - all
functions \
>>>   need to be co-assigned - err: 6
>>>   pciback 0000:06:00.0: FLR functionality not supported; attempts
to use \
>>>   secondary bus reset unsuccessful;
>>>   pciback 0000:06:00.0: FLR not performed for device
>>>
>>>       
>> Please check this wiki page:
>> http://wiki.xensource.com/xenwiki/XenPCIpassthrough
>>
>> It has a chapter about the "all functions need to be
co-assigned" error..
>>
>> -- Pasi
>>     
>
> Hi Pasi,
>
> Thanks for your reply.  I think I understand what is happening now.  My
card is
> too old to have the Function Level Reset functionality.  This leaves the 
> following options:
>
>
>   1. Assign all the cards that share the PCI bus (0000:06) to the same
guest.
> This enables the FLR functionality to be replaced by the bridge''s
"Secondary Bus
> Reset" functionality
>
>   - or -
>
>   2. Apply the patch referred to on 
> http://wiki.xensource.com/xenwiki/XenPCIpassthrough to bypass the FLR
>
>
> If I can''t do number #1 because some cards are in use by dom0 then
I guess #2 is
> the only option.  But does #2 require that dom0 has never accessed the
card?
>   No. Dom0 still can access the card. Dom0 just doesn''t perform FLR on it
to avoid your issue.> Also, what if you reboot the domU - does the fact that you can''t
do an FLR on
> the card mean that there will be unpredictable results the 2nd time domU
tries
> to initialise it?
>   Yes. FLR is required by pci passthrough. your NIC only can be 
co-assigned (#1). #2 is just a temporary workaround for your try.

Regards,
Weidong> Thanks again,
>
> Alex
>
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>   

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Alex Zeffertt

2010-May-25 08:42 UTC

head link

Re: [Xen-devel] pciback bug "secondary bus reset failed"

Weidong Han wrote:> Alex Zeffertt wrote:
>> Also, what if you reboot the domU - does the fact that you
can''t do an FLR on
>> the card mean that there will be unpredictable results the 2nd time
domU tries
>> to initialise it?
>>   
> Yes. FLR is required by pci passthrough. your NIC only can be 
> co-assigned (#1). #2 is just a temporary workaround for your try.
> 
Is there a way of finding out from a running system whether a PCI card supports 
Feature Level Reset?

Thanks,

Alex

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Ian Campbell

2010-May-25 08:50 UTC

head link

Re: [Xen-devel] pciback bug "secondary bus reset failed"

On Tue, 2010-05-25 at 09:42 +0100, Alex Zeffertt wrote:> Weidong Han wrote:
> > Alex Zeffertt wrote:
> >> Also, what if you reboot the domU - does the fact that you
can''t do an FLR on
> >> the card mean that there will be unpredictable results the 2nd
time domU tries
> >> to initialise it?
> >>   
> > Yes. FLR is required by pci passthrough. your NIC only can be 
> > co-assigned (#1). #2 is just a temporary workaround for your try.
> > 
> 
> Is there a way of finding out from a running system whether a PCI card
supports
> Feature Level Reset?
I think "lspci -vv" will include "FLReset+" in the DevCap
field.

Ian.


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Xen devel - May 2010 - pciback bug "secondary bus reset failed"

[Xen-devel] pciback bug "secondary bus reset failed"

Re: [Xen-devel] pciback bug "secondary bus reset failed"

Re: [Xen-devel] pciback bug "secondary bus reset failed"

Re: [Xen-devel] pciback bug "secondary bus reset failed"

Re: [Xen-devel] pciback bug "secondary bus reset failed"

Re: [Xen-devel] pciback bug "secondary bus reset failed"