Xen devel - Oct 2009 - Non-page aligned MMIO BAR error on dom-0 on Xen HVM setup

Hi,
I am using Neterion x3100  multi-function NIC adapter on stable Xen(2.6.18)
in HVM mode. I think the problem I have (explained below) is a general one
and has got nothing to do with neterion card in specific.
I am trying to assign one of the functions on the adapter to a guest OS(FC11
64 bit) in pass-through mode. When I try to start the guest OS (using xm
create), the dom-0 throws  the error saying the pass-through device has a
"non-page aligned MMIO BAR" and cannot boot. The function infact has a
BAR
(bar 4) whose size is 256 bytes and the resources are mapped on a
page-aligned boundary. THe guest cannot be booted for some reason because of
this.

To get around with this, I used the following kernel command line option:

reassign_resources  guestdev=<b:d.f>,<b:d,f>...

The BDFs of all the devices whose resources we want to re-assign to a
page-boundary are given in guestdev parameter above. As these neterion
functions are under a bridge, the bridge''s BDF is also given in the
guestdev
option (without the bridge''s BDF, I had issues with loading the
neterion
driver on guest OS).

With this kernel option, I am able to boot the guest OS but I see a
different problem in the config space of the device in dom-0. The BAR
regions are shown up as "disabled" in lspci output shown below.
Without this
option, the BARs are enabled.

01:00.2 Ethernet controller: Neterion Inc. X3100 Series 10 Gigabit Ethernet
PCIe (rev 01)
        Subsystem: Neterion Inc. X3110 Single Port
10GBase-CR
        Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
        Interrupt: pin A routed to IRQ
16
        Region 0: Memory at f9000000 (64-bit, prefetchable)
*[disabled]*[size=8M]
        Region 2: Memory at fc404000 (64-bit, prefetchable)
*[disabled]*[size=4K]
        Region 4: Memory at fc405000 (64-bit, prefetchable)
*[disabled]*[size=256]
        [virtual] Expansion ROM at fc100000 *[disabled]*[size=512K]

What this "disabled" means is the resources are not treated as
"memory
mapped resources" at all (as bit-2 in COMMAND register is not set in config
space).
Surprisingly, on the guest OS, the "disabled" flag is not seen on the
BAR
regions. I can successfully load the driver and do some data flow on
pass-through device on the guest.

Things seem fine so far despite the problem above on dom-0.

But, when I trigger an FLR (function level reset) on the pass-through
device/function on dom-0 (not on dom-u), I have problems on guest OS. The
"disabled" bit appears in the lspci output on the device in the guest.
The
driver is not able to do any memory-mapped reads/writes and hence I am not
able to load the driver. I triggered the FLR using following command:
# setpci -s 01:00.2 CAP_EXP+8.w=0xa814
(Before triggering FLR, value at CAP_EXP+8.w is 0x2814. As we have to set
last bit in PCI Express capability for FLR, value used in above command is
0xa814).

Does anybody have similar issues? Is "reassign_resources" the correct
kernel
option to use? If not, how do I get around the "non-page alignment
problem"

Any bit of help will be really helpful.

Thanks,
PurnaChandar M


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 15/10/2009 08:30, "Purna Chandar"
<purnachandar.midi@oneconvergence.com>
wrote:
> But, when I trigger an FLR (function level reset) on the pass-through
> device/function on dom-0 (not on dom-u), I have problems on guest OS. The
> "disabled" bit appears in the lspci output on the device in the
guest. The
> driver is not able to do any memory-mapped reads/writes and hence I am not
> able to load the driver. I triggered the FLR using following command:
> # setpci -s 01:00.2 CAP_EXP+8.w=0xa814
> (Before triggering FLR, value at CAP_EXP+8.w is 0x2814. As we have to set
last
> bit in PCI Express capability for FLR, value used in above command is
0xa814).
Why would you do that? Do you do it while the guest is running and owns the
device?

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi Keir,
Thanks for the response. Yes I wanted to do it while the guest is running. I
wanted to see if a FLR on dom-0 stops any data traffic from guest OS or not.
I also wanted to see whether the device gets operational on guest OS after
an interface down (ifconfig down) or reload of the driver in worst-case.

Thanks,
PurnaChandar M

2009/10/15 Keir Fraser <keir.fraser@eu.citrix.com>
> On 15/10/2009 08:30, "Purna Chandar"
<purnachandar.midi@oneconvergence.com
> >
> wrote:
>
> > But, when I trigger an FLR (function level reset) on the pass-through
> > device/function on dom-0 (not on dom-u), I have problems on guest OS.
The
> > "disabled" bit appears in the lspci output on the device in
the guest.
> The
> > driver is not able to do any memory-mapped reads/writes and hence I am
> not
> > able to load the driver. I triggered the FLR using following command:
> > # setpci -s 01:00.2 CAP_EXP+8.w=0xa814
> > (Before triggering FLR, value at CAP_EXP+8.w is 0x2814. As we have to
set
> last
> > bit in PCI Express capability for FLR, value used in above command is
> 0xa814).
>
> Why would you do that? Do you do it while the guest is running and owns the
> device?
>
>  -- Keir
>
>
>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Thu, Oct 15, 2009 at 01:00:31PM +0530, Purna Chandar
wrote:
> Hi,
> I am using Neterion x3100  multi-function NIC adapter on stable Xen(2.6.18)
> in HVM mode. I think the problem I have (explained below) is a general one
> and has got nothing to do with neterion card in specific.
> I am trying to assign one of the functions on the adapter to a guest
OS(FC11
> 64 bit) in pass-through mode. When I try to start the guest OS (using xm
> create), the dom-0 throws  the error saying the pass-through device has a
> "non-page aligned MMIO BAR" and cannot boot. The function infact
has a BAR
> (bar 4) whose size is 256 bytes and the resources are mapped on a
> page-aligned boundary. THe guest cannot be booted for some reason because
of
> this.
> 
> To get around with this, I used the following kernel command line option:
> 
> reassign_resources  guestdev=<b:d.f>,<b:d,f>...
> 
> The BDFs of all the devices whose resources we want to re-assign to a
> page-boundary are given in guestdev parameter above. As these neterion
> functions are under a bridge, the bridge''s BDF is also given in
the guestdev
> option (without the bridge''s BDF, I had issues with loading the
neterion
> driver on guest OS).
> 
> With this kernel option, I am able to boot the guest OS but I see a
> different problem in the config space of the device in dom-0. The BAR
> regions are shown up as "disabled" in lspci output shown below.
Without this
> option, the BARs are enabled.
> 
> 01:00.2 Ethernet controller: Neterion Inc. X3100 Series 10 Gigabit Ethernet
> PCIe (rev 01)
>         Subsystem: Neterion Inc. X3110 Single Port
> 10GBase-CR
>         Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
> Stepping- SERR- FastB2B- DisINTx-
>         Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>         Interrupt: pin A routed to IRQ
> 16
>         Region 0: Memory at f9000000 (64-bit, prefetchable)
> *[disabled]*[size=8M]
>         Region 2: Memory at fc404000 (64-bit, prefetchable)
> *[disabled]*[size=4K]
>         Region 4: Memory at fc405000 (64-bit, prefetchable)
> *[disabled]*[size=256]
>         [virtual] Expansion ROM at fc100000 *[disabled]*[size=512K]
> 
> What this "disabled" means is the resources are not treated as
"memory
> mapped resources" at all (as bit-2 in COMMAND register is not set in
config
> space).
> Surprisingly, on the guest OS, the "disabled" flag is not seen on
the BAR
> regions. I can successfully load the driver and do some data flow on
> pass-through device on the guest.
> 
> Things seem fine so far despite the problem above on dom-0.
> 
> But, when I trigger an FLR (function level reset) on the pass-through
> device/function on dom-0 (not on dom-u), I have problems on guest OS. The
> "disabled" bit appears in the lspci output on the device in the
guest. The
> driver is not able to do any memory-mapped reads/writes and hence I am not
> able to load the driver. I triggered the FLR using following command:
Does it change the state in the dom0? It is still disabled, right?

What if you manually set the bit-2 of the command register in either locations
(dom0 or domU).
> # setpci -s 01:00.2 CAP_EXP+8.w=0xa814
> (Before triggering FLR, value at CAP_EXP+8.w is 0x2814. As we have to set
> last bit in PCI Express capability for FLR, value used in above command is
> 0xa814).
> 
> Does anybody have similar issues? Is "reassign_resources" the
correct kernel
> option to use? If not, how do I get around the "non-page alignment
problem"
The reassign_resources is the fix for non-page aligned BARs in the older
kernels. In the newer ones... still working on that.

The BAR is not page-aligned on baremetal either, right?

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel