Xen devel - Oct 2008 - xen kernel 2.6.18 bug with a d-link DFE 580TX - pci hide

Hi,

i have tested the following setup:

System: gentoo_2008, xen-sources-2.6.18-r12, x86_64

1. Hide one port from the d-link quad nic via pci.hide in dom0.

module: sundance

grub.conf: pciback.hide=(06:07.0)

2. Use it within a domU.

domu.conf: pci = [ ''06:07.0'' ]

Tools:
------
Gnu C                  4.1.2
Gnu make               3.81
binutils               Binutils
util-linux             2.13.1.1
mount                  2.13.1.1
module-init-tools      3.4
e2fsprogs              1.40.9
xfsprogs               2.9.7
Linux C Library        6.1
Dynamic linker (ldd)   2.6.1
Procps                 3.2.7
Net-tools              1.60
Kbd                    1.13
Sh-utils               6.10
udev                   124
Modules Loaded         sundance

Result see attachment.

I have tested the same setup with the original xen
linux-2.6.18-xen-3.3.0 upstream kernel and it crashed too. With the
vanilla 2.6.27 kernel it doesn''t crashed, but the device is not
available in the booted domU.

Is there any chance to get such a setup working?
Do you need more informations?

TIA
Michael


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 25/10/08 21:05, "Michael Kapp" <tquarkk@isis-it.de> wrote:
> ----------- [cut here ] --------- [please bite here ] ---------
> Kernel BUG at arch/x86_64/kernel/../../i386/kernel/pci-dma-xen.c:377
What''s at line 377 in that source file in your kernel source tree?

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 26/10/08 08:40, "Michael Kapp" <tquarkk@isis-it.de> wrote:
> Keir Fraser wrote:
>> On 25/10/08 21:05, "Michael Kapp" <tquarkk@isis-it.de>
wrote:
>> 
>>> ----------- [cut here ] --------- [please bite here ] ---------
>>> Kernel BUG at
arch/x86_64/kernel/../../i386/kernel/pci-dma-xen.c:377
>> 
>> What''s at line 377 in that source file in your kernel source
tree?
> 
> See attachment of the specific part from pci-dma-xen.c.
Add swiotlb=force to your domain''s kernel command line.

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Keir Fraser wrote:
> On 25/10/08 21:05, "Michael Kapp" <tquarkk@isis-it.de>
wrote:
> 
>> ----------- [cut here ] --------- [please bite here ] ---------
>> Kernel BUG at arch/x86_64/kernel/../../i386/kernel/pci-dma-xen.c:377
> 
> What''s at line 377 in that source file in your kernel source tree?
See attachment of the specific part from pci-dma-xen.c.
>  -- Keir
Regards,
Michael




_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Keir Fraser wrote:
> On 26/10/08 08:40, "Michael Kapp" <tquarkk@isis-it.de>
wrote:
> 
>> Keir Fraser wrote:
>>> On 25/10/08 21:05, "Michael Kapp"
<tquarkk@isis-it.de> wrote:
>>>
>>>> ----------- [cut here ] --------- [please bite here ] ---------
>>>> Kernel BUG at
arch/x86_64/kernel/../../i386/kernel/pci-dma-xen.c:377
>>> What''s at line 377 in that source file in your kernel
source tree?
>> See attachment of the specific part from pci-dma-xen.c.
> 
> Add swiotlb=force to your domain''s kernel command line.
I set swiotlb=force in the domU conf, that doesn''t fix it - also with
noirqdebug.

Michael


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 26/10/08 08:58, "Michael Kapp" <tquarkk@isis-it.de> wrote:
>> Add swiotlb=force to your domain''s kernel command line.
> 
> I set swiotlb=force in the domU conf, that doesn''t fix it - also
with
> noirqdebug.
That should be impossible since the line you are bugging on will not be
executed if the swiotlb has been successfully enabled. Do you get a line in
dmesg stating "Software IO TLB enabled"? If not then is CONFIG_SWIOTLB
enabled in your kernel config (should happen by default). The file
lib/swiotlb-xen.c should be getting built and you''ll see there is code
near
the top of that file to parse swiotlb=force which sets swiotlb_force and
should cause swiotlb_init() to do the right thing (which should get called
via the path mem_init->pci_iommu_alloc->pci_swiotlb_init->swiotlb_init.

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Keir Fraser wrote:
> On 26/10/08 08:58, "Michael Kapp" <tquarkk@isis-it.de>
wrote:
> 
>>> Add swiotlb=force to your domain''s kernel command line.
>> I set swiotlb=force in the domU conf, that doesn''t fix it -
also with
>> noirqdebug.
> 
> That should be impossible since the line you are bugging on will not be
> executed if the swiotlb has been successfully enabled. Do you get a line in
> dmesg stating "Software IO TLB enabled"? If not then is
CONFIG_SWIOTLB
> enabled in your kernel config (should happen by default). The file
> lib/swiotlb-xen.c should be getting built and you''ll see there is
code near
> the top of that file to parse swiotlb=force which sets swiotlb_force and
> should cause swiotlb_init() to do the right thing (which should get called
> via the path
mem_init->pci_iommu_alloc->pci_swiotlb_init->swiotlb_init.
yeah - your right! My domU config was wrong, with swiotlb=force it''s
working, thanks!

Can you tell me what this parameter is doing in detail, or where i can
find docs about it?

Regards,
Michael

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 26/10/08 10:48, "Michael Kapp" <tquarkk@isis-it.de> wrote:
>>>> Add swiotlb=force to your domain''s kernel command
line.
>>> I set swiotlb=force in the domU conf, that doesn''t fix it
- also with
>>> noirqdebug.
>> 
>> That should be impossible since the line you are bugging on will not be
>> executed if the swiotlb has been successfully enabled. Do you get a
line in
>> dmesg stating "Software IO TLB enabled"? If not then is
CONFIG_SWIOTLB
>> enabled in your kernel config (should happen by default). The file
>> lib/swiotlb-xen.c should be getting built and you''ll see there
is code near
>> the top of that file to parse swiotlb=force which sets swiotlb_force
and
>> should cause swiotlb_init() to do the right thing (which should get
called
>> via the path
mem_init->pci_iommu_alloc->pci_swiotlb_init->swiotlb_init.
> 
> yeah - your right! My domU config was wrong, with swiotlb=force
it''s
> working, thanks!
> 
> Can you tell me what this parameter is doing in detail, or where i can
> find docs about it?
The network card is only capable of 32-bit DMA. So for addresses above 4GB a
bounce buffer is required (DMA to/from bounce buffer; then copy to/from
proper high-memory buffer using host CPU). The swiotlb is basically a
pre-allocated bounce-buffer area. Because it is pre-allocated it is a waste
of memory if it isn''t used and, since domUs do not normally perform
real
physical I/O directly, we therefore by default only create a swiotlb for
dom0. You then have to override on the cmdline if this default doesn''t
actually work (as in this case).

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Keir Fraser wrote:
>> Can you tell me what this parameter is doing in detail, or where i can
>> find docs about it?
> 
> The network card is only capable of 32-bit DMA. So for addresses above 4GB
a
> bounce buffer is required (DMA to/from bounce buffer; then copy to/from
> proper high-memory buffer using host CPU). The swiotlb is basically a
> pre-allocated bounce-buffer area. Because it is pre-allocated it is a waste
> of memory if it isn''t used and, since domUs do not normally
perform real
> physical I/O directly, we therefore by default only create a swiotlb for
> dom0. You then have to override on the cmdline if this default
doesn''t
> actually work (as in this case).
...and why is it not possible (from the dom0 point of view) to detect
this behaviour while starting the domU, reading it''s conf and stop the
boot process with a warning?

Michael

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 26/10/08 19:17, "Michael Kapp" <tquarkk@isis-it.de> wrote:
>> The network card is only capable of 32-bit DMA. So for addresses above
4GB a
>> bounce buffer is required (DMA to/from bounce buffer; then copy to/from
>> proper high-memory buffer using host CPU). The swiotlb is basically a
>> pre-allocated bounce-buffer area. Because it is pre-allocated it is a
waste
>> of memory if it isn''t used and, since domUs do not normally
perform real
>> physical I/O directly, we therefore by default only create a swiotlb
for
>> dom0. You then have to override on the cmdline if this default
doesn''t
>> actually work (as in this case).
> 
> ...and why is it not possible (from the dom0 point of view) to detect
> this behaviour while starting the domU, reading it''s conf and stop
the
> boot process with a warning?
IOMMU_BUG_ON() is supposed to printk a suitable warnign before actually
crashing. Not sure why that didn''t work for you.

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel