Xen users - Dec 2007 - About swiotlb and " ...PCI-DMA: Out of SW-IOMMU..." error

Hi all,

I posted the following message in the devel list without successful, expect here
have a little more luck ;)

Just add that testing the same setup on a VMware (I mean, not in a physical
hardware) all go fine.

Hi all,

Sorry write to this list, here is the only place with a thread and a
solution to this problem.

At
http://lists.xensource.com/archives/html/xen-devel/2007-09/msg00140.html
you can read

“
No, you simply don’t have a big enough swiotlb. The default is 2MB for a
machine with less than 2GB of RAM, or 64MB for >2GB systems.
“

All seen that for that guy that problem get solved changing the swiotlb,
but for me and lot of people
(https://bugs.launchpad.net/ubuntu/+source/xen-meta/+bug/162147) not,
can any help us to sort up this bug?

In resume the problem is that after try to create the first DomU lots of
lines come to the console like this one:

“[ 5345.211424] PCI-DMA: Out of SW-IOMMU sapce for 65536 bytes at device
0000:03:01.0”

and in dmesg appear lot of line like:

“
[ 5345.302257] (scsi4:A:0:0): data overrun detected in Data-out phase.
Tag == 0x2.
[ 5345.302263] (scsi4:A:0:0): Have seen Data Phase. Length = 0. NumSGs 0. 
”

Thanks a lot in advanced, I''m trying to deploy a Xen solution in my
University and is important for me solve this bug.

Salute
Frank Abel





__________________________________________

Participe en Universidad 2008.
11 al 15 de febrero del 2008.
Palacio de las Convenciones, Ciudad de la Habana, Cuba
http://www.universidad2008.cu

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

I have the same problem with xen 3.1 and adding swiotlb=128 to my grub conf 
just caused xen to crash/panic.  Not only that but this problem killed my 
dom0 root partition forcing all of the data into /lost+found which is oh so 
much fun.  The only solution i''ve found to stopping this from happening
is
to force dom0 to use less than 2GB of ram.  In my case i forced my dom0 to 
use 1024MB of ram leaving the rest (around 15GB) on vacation.  Not a big 
deal since before 3.1 i always forced dom0 to use only 1024MB of ram anyway 
leaving the rest for the domU''s.

I''m running 3ware 9560 controllers btw too if it matters.

~Shaun



_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Add that my system have just 512Mb of ram. How do you exactly force dom0
use less than 2GB of ram? with the dom0-min-mem var in
the /etc/xen/xend-config.sxp conf file?

Salute
Frank Abel

On Thu, 2007-12-06 at 12:24 -0800, Shaun R. wrote:
> I have the same problem with xen 3.1 and adding swiotlb=128 to my grub conf
> just caused xen to crash/panic.  Not only that but this problem killed my 
> dom0 root partition forcing all of the data into /lost+found which is oh so
> much fun.  The only solution i''ve found to stopping this from
happening is
> to force dom0 to use less than 2GB of ram.  In my case i forced my dom0 to 
> use 1024MB of ram leaving the rest (around 15GB) on vacation.  Not a big 
> deal since before 3.1 i always forced dom0 to use only 1024MB of ram anyway
> leaving the rest for the domU''s.
> 
> I''m running 3ware 9560 controllers btw too if it matters.
> 
> ~Shaun
> 
> 
> 
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users
> 


__________________________________________

Participe en Universidad 2008.
11 al 15 de febrero del 2008.
Palacio de las Convenciones, Ciudad de la Habana, Cuba
http://www.universidad2008.cu

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

add dom0_mem=1024000 to the line with the xen.gz on it, that would allocate 
1024MB of ram...  If you only have 512 though total on your system you 
technically shouldnt be hitting this error according to that other post.  I 
dont really understand whats happening but this has to be a bug considering 
it doesnt happen with a vanilla kernel.....


~Shaun


"Frank Abel Cancio Bello" <frankabel@tesla.cujae.edu.cu> wrote
in message
news:1196979626.5691.13.camel@miriam...
> Add that my system have just 512Mb of ram. How do you exactly force dom0
> use less than 2GB of ram? with the dom0-min-mem var in
> the /etc/xen/xend-config.sxp conf file?
>
> Salute
> Frank Abel
>
> On Thu, 2007-12-06 at 12:24 -0800, Shaun R. wrote:
>> I have the same problem with xen 3.1 and adding swiotlb=128 to my grub 
>> conf
>> just caused xen to crash/panic.  Not only that but this problem killed
my
>> dom0 root partition forcing all of the data into /lost+found which is
oh
>> so
>> much fun.  The only solution i''ve found to stopping this from
happening
>> is
>> to force dom0 to use less than 2GB of ram.  In my case i forced my dom0
>> to
>> use 1024MB of ram leaving the rest (around 15GB) on vacation.  Not a
big
>> deal since before 3.1 i always forced dom0 to use only 1024MB of ram 
>> anyway
>> leaving the rest for the domU''s.
>>
>> I''m running 3ware 9560 controllers btw too if it matters.
>>
>> ~Shaun
>>
>>
>>
>> _______________________________________________
>> Xen-users mailing list
>> Xen-users@lists.xensource.com
>> http://lists.xensource.com/xen-users
>>
>
>
>
> __________________________________________
>
> Participe en Universidad 2008.
> 11 al 15 de febrero del 2008.
> Palacio de las Convenciones, Ciudad de la Habana, Cuba
> http://www.universidad2008.cu 


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Shaun R. wrote:
> [..]
>  The only solution i''ve found to stopping this from happening is
> to force dom0 to use less than 2GB of ram.  In my case i forced my dom0 to
> use 1024MB of ram leaving the rest (around 15GB) on vacation.  
> 
> I''m running 3ware 9560 controllers btw too if it matters.
I''ve had a discussion (subject: "Massive problems with
''PCI-DMA: Out of
SW-IOMMU space for XXX bytes [..]'' with 3Ware controller") with
Keir Fraser
on xen-devel this days because of this problem and for /me dom0 memory size
does not matter (I''ve tried to not restrict dom0 (so it can use almost
all
of my 2GB) and to set it to only 128MB - no difference). 
btw: I own a 3ware 9500 - so we use the same driver - 3w_9xxx - maybe here
the problem starts...

Here is a quote from an e-mail Keir sent me off-list:
> Sounds like the 2.6.18 kernel is screwed then. Perhaps the
> 2.6.18 driver leaks swiotlb space... I haven''t got many spare
cycles right
> now to look into this. Probably the next step though might be to dump the
> swiotlb state when this bug-out happens, and see if the state looks like a
> leak has been happening. A leak makes sense given that you seem to run out
> of space eventually regardless of swiotlb size.
For what I can say for now is that it does not happen on my old 2.6.16
kernel patched for Xen 3.0.4. With a 2.6.18 patched for Xen 3.1.2 it
happens. Will do some additional tests this days (if time permits). 

Regards, 
Markus


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

But I haven''t a 3ware on my server and I have the same problem.

Salute
Frank Abel


On Sun, 2007-12-09 at 03:31 +0100, Markus Schuster
wrote:
> Shaun R. wrote:
> > [..]
> >  The only solution i''ve found to stopping this from happening
is
> > to force dom0 to use less than 2GB of ram.  In my case i forced my
dom0 to
> > use 1024MB of ram leaving the rest (around 15GB) on vacation.  
> > 
> > I''m running 3ware 9560 controllers btw too if it matters.
> 
> I''ve had a discussion (subject: "Massive problems with
''PCI-DMA: Out of
> SW-IOMMU space for XXX bytes [..]'' with 3Ware controller")
with Keir Fraser
> on xen-devel this days because of this problem and for /me dom0 memory size
> does not matter (I''ve tried to not restrict dom0 (so it can use
almost all
> of my 2GB) and to set it to only 128MB - no difference). 
> btw: I own a 3ware 9500 - so we use the same driver - 3w_9xxx - maybe here
> the problem starts...
> 
> Here is a quote from an e-mail Keir sent me off-list:
> 
> > Sounds like the 2.6.18 kernel is screwed then. Perhaps the
> > 2.6.18 driver leaks swiotlb space... I haven''t got many spare
cycles right
> > now to look into this. Probably the next step though might be to dump
the
> > swiotlb state when this bug-out happens, and see if the state looks
like a
> > leak has been happening. A leak makes sense given that you seem to run
out
> > of space eventually regardless of swiotlb size.
> 
> For what I can say for now is that it does not happen on my old 2.6.16
> kernel patched for Xen 3.0.4. With a 2.6.18 patched for Xen 3.1.2 it
> happens. Will do some additional tests this days (if time permits). 
> 
> Regards, 
> Markus
> 
> 
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users
> 


__________________________________________

Participe en Universidad 2008.
11 al 15 de febrero del 2008.
Palacio de las Convenciones, Ciudad de la Habana, Cuba
http://www.universidad2008.cu

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> btw: I own a 3ware 9500 - so we use the same driver - 3w_9xxx - maybe here
> the problem starts...
I run the 9560 controller and i patch is a newer driver provided by 3ware, i 
had to do this because 2.6.18 doesnt have support for the 9560 controllers. 
There are some problems with Xen and 3wares drivers.  I also had a ML 
discussion going a while back because the latest driver for the 3ware 
controller would cause a kernel panic only with a Xen patched kernel, with a 
vanilla kernel it would work fine.  Keir also talked with me alittle about 
this but didnt really seam interested to hear or look into it futher. 
I''m
sure they are busy and i''m not positive that it was a problem with Xen
it
could have been a problem with the 3ware code.  It was only happening with 
the xen patches though so i figured they would want to have a look, 3ware is 
a popular controller these days...

~Shaun 



_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Hi,

as i''m going to install a bunch of new servers with xen 3.1.2,
i''m really
interested in which setup this Out of SW-IOMMU error occurs.

Recent posts on xen-devel and xen-user are not really clear about the used
hardware, the xen version and the dom0 kernel.

To collect some statements:
- It occurs when using 3ware controller on xen 3.1.2 with bundled 2.6.18
- It occurs when using more than 2 G Ram (in dom0 ?)
- It has to do with the used memory model (maybe only on 64bit xen ?)
- One could ommit the error by limiting dom0 max_mem
- It occurs when a pciback.hide()/pci=[] routed physical device is used
  by buggy domU drivers (seen on the list w/. Infiniband card)

My favorite installation would be xen 3.1.2 64bit w/ bundled (and slightly
patched to support areca raid) 2.6.18 64bit dom0, running 32bit PAE
domU''s without any physical device access.
The machines are dual Quad Xeons equipped w/ 32G Ram and areca raid controller.

As I do not want to waste time building packages just to end up with this
strange error, could you please shed some light on me?

Thanks in advance

Stephan



Shaun R. schrieb:
>> btw: I own a 3ware 9500 - so we use the same driver - 3w_9xxx - maybe
>> here
>> the problem starts...
> 
> I run the 9560 controller and i patch is a newer driver provided by
> 3ware, i had to do this because 2.6.18 doesnt have support for the 9560
> controllers. There are some problems with Xen and 3wares drivers.  I
> also had a ML discussion going a while back because the latest driver
> for the 3ware controller would cause a kernel panic only with a Xen
> patched kernel, with a vanilla kernel it would work fine.  Keir also
> talked with me alittle about this but didnt really seam interested to
> hear or look into it futher.  I''m sure they are busy and
i''m not
> positive that it was a problem with Xen it could have been a problem
> with the 3ware code.  It was only happening with the xen patches though
> so i figured they would want to have a look, 3ware is a popular
> controller these days...
> 
> ~Shaun
> 


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Stephan Seitz wrote:
> Hi,
>
> as i''m going to install a bunch of new servers with xen 3.1.2,
i''m really
> interested in which setup this Out of SW-IOMMU error occurs.
>
> Recent posts on xen-devel and xen-user are not really clear about the used
> hardware, the xen version and the dom0 kernel.
>
> To collect some statements:
> - It occurs when using 3ware controller on xen 3.1.2 with bundled 2.6.18
> - It occurs when using more than 2 G Ram (in dom0 ?)
> - It has to do with the used memory model (maybe only on 64bit xen ?)
> - One could ommit the error by limiting dom0 max_mem
> - It occurs when a pciback.hide()/pci=[] routed physical device is used
>   by buggy domU drivers (seen on the list w/. Infiniband card)
>
> My favorite installation would be xen 3.1.2 64bit w/ bundled (and slightly
> patched to support areca raid) 2.6.18 64bit dom0, running 32bit PAE
> domU''s without any physical device access.
> The machines are dual Quad Xeons equipped w/ 32G Ram and areca raid
controller.
>
> As I do not want to waste time building packages just to end up with this
> strange error, could you please shed some light on me?
>
> Thanks in advance
>
> Stephan
>
>   
Just wonder if anyone has found any solution yet.

I am seeing this problem on machines that have Dell PERC/4 or MegaRaid
SCSI controller. The machines have kernel 2.6.18 with 32bit non-pae Xen
3.1.2. Both machines have 2G ram, with 256MB allocated to dom0.

This error does not occur with 2.6.16. Also running Xen 3.1.2 with
2.6.16 seems working fine.

Simon

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users