Xen devel - Nov 2007 - Massive problems with ''PCI-DMA: Out of SW-IOMMU space for XXX bytes [..]'' with 3Ware controller

Hi list, 

this days I''ve tried to update a 32bit XEN 3.0.4 system (kernel 2.6.16)
to
XEN 3.1.2 on AMD64 (kernel 2.6.18). The system has 2GB RAM and a 3ware
9500-12 SATA raid controller. 

The old system worked fine for about half a year now. 
The new system boots fine so far but when doing some "bigger" changes
to the
filesystem I get a lot of this errors:
> PCI-DMA: Out of SW-IOMMU space for 40960 bytes at device 0000:03:00.0
> 3w-9xxx: scsi0: ERROR: (0x06:0x001C): Failed to map scatter gather list.
(03:00.0 is the 3Ware Controller)

I''ve increased the ''swiotlb'' kernel parameter in
steps from 32, 64, 128, 256
up to 512MB. The only effect is, that with 512MB my dom0 uses over 600MB of
RAM (compared to ~50MB with XEN 3.0.4 on 32bit) and more fs-activity can be
done before the error arises. 
I''s very easy to reproduce, just do a ''dd if=/dev/zero
of=/root/testfile''.
After a short while (some seconds) the error arises. 

The funny thing is: I''ve tried this dom0 on my workstation before
installing
it on a production server. My workstation has an nForce SATA controller and
here I don''t have any problems - even without setting the
''swiotlb'' kernel
paramter. 

It looks like there are some problems in conjunction with the 3Ware
controller that have not been there on XEN 3.0.4 on 2.6.16 (32bit). 

The complete piece of the logfile can be found here:
http://nopaste.biz/21780
And this is just from writing a 5MB initramfs to an ext2 filesytem (which
had some corruption afterwards). 

Thank''s for your help!

Regards, 
Markus Schuster


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

What happens if you run the old kernel (32-bit 2.6.16) on new Xen? What
happens if you run a new (2.6.18) 32-bit kernel on new Xen? Do these tests
without inflating the swiotlb size on the command line. Or what about new
2.6.18 kernel on old Xen?

(You can run 32-bit kernel on 64-bit Xen in 3.1.x, but not in 3.0.x, by the
way.)

 -- Keir

On 25/11/07 16:03, "Markus Schuster" <ma.schuster@gmx.de> wrote:
> Hi list, 
> 
> this days I''ve tried to update a 32bit XEN 3.0.4 system (kernel
2.6.16) to
> XEN 3.1.2 on AMD64 (kernel 2.6.18). The system has 2GB RAM and a 3ware
> 9500-12 SATA raid controller.
> 
> The old system worked fine for about half a year now.
> The new system boots fine so far but when doing some "bigger"
changes to the
> filesystem I get a lot of this errors:
>> PCI-DMA: Out of SW-IOMMU space for 40960 bytes at device 0000:03:00.0
>> 3w-9xxx: scsi0: ERROR: (0x06:0x001C): Failed to map scatter gather
list.
> (03:00.0 is the 3Ware Controller)
> 
> I''ve increased the ''swiotlb'' kernel parameter in
steps from 32, 64, 128, 256
> up to 512MB. The only effect is, that with 512MB my dom0 uses over 600MB of
> RAM (compared to ~50MB with XEN 3.0.4 on 32bit) and more fs-activity can be
> done before the error arises.
> I''s very easy to reproduce, just do a ''dd if=/dev/zero
of=/root/testfile''.
> After a short while (some seconds) the error arises.
> 
> The funny thing is: I''ve tried this dom0 on my workstation before
installing
> it on a production server. My workstation has an nForce SATA controller and
> here I don''t have any problems - even without setting the
''swiotlb'' kernel
> paramter. 
> 
> It looks like there are some problems in conjunction with the 3Ware
> controller that have not been there on XEN 3.0.4 on 2.6.16 (32bit).
> 
> The complete piece of the logfile can be found here:
> http://nopaste.biz/21780
> And this is just from writing a 5MB initramfs to an ext2 filesytem (which
> had some corruption afterwards).
> 
> Thank''s for your help!
> 
> Regards, 
> Markus Schuster
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Keir Fraser wrote:

Before I start testing, I have some questions:
> What happens if you run the old kernel (32-bit 2.6.16) on new Xen? 
I could simply exchange the Xen hypervisor on my old system, but
1. Will the old xend work with the new hypervisor? I don''t want to
further
touch the old (stable) system. An exchange of the hypervisor is not the
problem, as it''s just one file, but the complete Xen userland...
2. Will a 32bit xend work with a 64bit hypervisor?

> What happens if you run a new (2.6.18) 32-bit kernel on new Xen? 
Ok, I''ll have to do some compiling before I can say anything on this (I
only
have 64bit packages (Xen and kernel)). 
And I have to install a test-32bit-dom0. 

> Or what about new 2.6.18 kernel on old Xen? 
You mean the 32bit version of 2.6.18? Have to do some compiling...

Regards, 
Markus Schuster


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

You have to run old Xen with old tools, and new Xen with new tools. However,
the dom0 kernels should be interchangeable (with the exception that old
64-bit Xen cannot run a 32-bit dom0).

 -- Keir

On 25/11/07 21:08, "Markus Schuster" <ma.schuster@gmx.de> wrote:
> Keir Fraser wrote:
> 
> Before I start testing, I have some questions:
> 
>> What happens if you run the old kernel (32-bit 2.6.16) on new Xen?
> 
> I could simply exchange the Xen hypervisor on my old system, but
> 1. Will the old xend work with the new hypervisor? I don''t want to
further
> touch the old (stable) system. An exchange of the hypervisor is not the
> problem, as it''s just one file, but the complete Xen userland...
> 2. Will a 32bit xend work with a 64bit hypervisor?
> 
> 
>> What happens if you run a new (2.6.18) 32-bit kernel on new Xen?
> 
> Ok, I''ll have to do some compiling before I can say anything on
this (I only
> have 64bit packages (Xen and kernel)).
> And I have to install a test-32bit-dom0.
> 
> 
>> Or what about new 2.6.18 kernel on old Xen?
> 
> You mean the 32bit version of 2.6.18? Have to do some compiling...
> 
> Regards, 
> Markus Schuster
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Keir Fraser wrote:
> You have to run old Xen with old tools, and new Xen with new tools.
> However, the dom0 kernels should be interchangeable (with the exception
> that old 64-bit Xen cannot run a 32-bit dom0).
OK, then I''ll have to install a new 32bit dom0 for testing purposes. 
Compilation of the kernel for 32bit is currently running - Xen is the next
in the queue - but I will need some days to provide reasonable results. 

The good thing is: I''ll have Xen 3.1.2 Debian packages for i386
afterwards :)

Regards, 
Markus


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Markus Schuster wrote:
> PCI-DMA: Out of SW-IOMMU space for 40960 bytes at device 0000:03:00.0
> 3w-9xxx: scsi0: ERROR: (0x06:0x001C): Failed to map scatter gather list.
> (03:00.0 is the 3Ware Controller)
Here''s my "me too" data point.

I''m also seeing this on Xen 3.1.2, 64bit Xen, PAE Dom0 (2.6.18).  For 
me, it only happens if Dom0 is low on memory and into swap quite a bit. 
  Compiling a kernel with -j4 or so, with only 256M allocated to Dom0 
will trigger it.  I tried a number of swiotlb settings, none eliminate 
the problem.

-Chris


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi, 

I finally have results for you:

The one you already know:
64bit Xen 3.1.2
64bit 2.6.18
=> FAIL

Here my tested results:
32bit Xen 3.1.2 PAE
32bit 2.6.18 PAE
=> FAIL

64bit Xen 3.1.2
32bit 2.6.18 PAE
=> FAIL

32bit Xen 3.1.2 NON-PAE
32bit 2.6.16 NON-PAE
=> OK (I''ve written over 15GB to disk without problems)

I had to use a NON-PAE version of Xen for testing my old working 2.6.16
kernel, ''cause it doesn''t have PAE enabled (I thought it would
have, but
well..). 
But all 3.1.2 Xen binaries have been built from the same source. 

One interesting fact: The server mainboard has some onboard SATA ports. If I
connect a hard disc to this ports, I can write to them without problems.
Just in conjunction with the 3Ware I suffer from this problem. 

The bug seems to be in the kernel part, not the hypervisor itself, as my old
kernel works without problems. 

Regards, 
Markus


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Markus Schuster wrote:
> The bug seems to be in the kernel part, not the hypervisor itself, as my
old
> kernel works without problems. 
I hit this again over the weekend, causing massive file system damage to 
dom0.

I''d love to get this fixed so we can deploy 3.1.2 -- What can we do to 
help you guys further debug this?  I''ve got plenty of boxes with this 
hardware setup to test with.

Much appreciated,
-Chris

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel