Xen devel - Oct 2007 - DL385 G2 - PCI-DMA: Random memory would be DMAed

Mark suggested that I post this to the xen-devel list (I had orginally posted 
to xen-users):

Original post:

 > I am attempting to run Xen 3.1.1 on a HP DL385 G2 box (opteron based
 > system), using Gentoo.  The machine will boot, and I can login and edit
 > files etc, but as soon as I do anything more intensive such as copying
 > files off a cdrom (making me think it is a cciss driver problem) or
running
 > make, I get the following :
 >
 > 	kernel panic not syncing: PCI-DMA: Random memory would be DMAed
 >
 > Searching on this doesn''t come up with much except the source
code where
 > this error message is written out.  The comment there indicates there is
an
 > IOMMU problem, but I am not sure where to go from there. I am pretty sure
 > it is not a hardware issue since, when booting with a non-xen kernel and
 > the same file system, no problem.  Anyone have any ideas or run into this
 > problem before?
 >
 > As a side note, I have two of these boxes which are identical, and I have
 > successfully installed XenExpress on one of them, in the hopes of finding
 > something there.


To answer Mark''s question:

 > What kernel are you using?

from gentoo: xen-sources-2.6.20-r6

Regarding the full error message, all I am getting is:

	kernel panic not syncing: PCI-DMA: Random memory would be DMAed

since the machine is rebooting after that.  There maybe somewhere else to get 
more detail, but beyond dmesg or messages I am not sure where else to look.



	Zak

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Mark suggested that I post this to the xen-devel list (I had orginally posted 
to xen-users):

Original post:

 > I am attempting to run Xen 3.1.1 on a HP DL385 G2 box (opteron based
 > system), using Gentoo.  The machine will boot, and I can login and edit
 > files etc, but as soon as I do anything more intensive such as copying
 > files off a cdrom (making me think it is a cciss driver problem) or
running
 > make, I get the following :
 >
 >     kernel panic not syncing: PCI-DMA: Random memory would be DMAed
 >
 > Searching on this doesn''t come up with much except the source
code where
 > this error message is written out.  The comment there indicates there is
an
 > IOMMU problem, but I am not sure where to go from there. I am pretty sure
 > it is not a hardware issue since, when booting with a non-xen kernel and
 > the same file system, no problem.  Anyone have any ideas or run into this
 > problem before?
 >
 > As a side note, I have two of these boxes which are identical, and I have
 > successfully installed XenExpress on one of them, in the hopes of finding
 > something there.


To answer Mark''s question:

 > What kernel are you using?

from gentoo: xen-sources-2.6.20-r6

Regarding the full error message, all I am getting is:

     kernel panic not syncing: PCI-DMA: Random memory would be DMAed

since the machine is rebooting after that.  There maybe somewhere else to get 
more detail, but beyond dmesg or messages I am not sure where else to look.



     Zak

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> To answer Mark''s question:
> 
>  > What kernel are you using?
> 
> from gentoo: xen-sources-2.6.20-r6
> 
> Regarding the full error message, all I am getting is:
> 
>      kernel panic not syncing: PCI-DMA: Random memory would be DMAed
> 
> since the machine is rebooting after that.  There maybe 
> somewhere else to get more detail, but beyond dmesg or
> messages I am not sure where else to look.
For some reason, your kernel is running with pci-gart.c
enabled, and dom0 is trying to allocate DMA through the
AGP GART in your AMD64 processor''s Northbridge.  I 
didn''t think Xen supported pci-gart, but maybe I missed
a changeset.

At any rate, passing iommu=soft should disable the pci-gart
and allow you to use bounce buffering.  Hopefully, that
should clear up your problem.

-Mark Langsdorf
Operating System Research Center
AMD



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Thanks for chiming in Mark, I was at a loss to suggest something specific!
> For some reason, your kernel is running with pci-gart.c
> enabled, and dom0 is trying to allocate DMA through the
> AGP GART in your AMD64 processor''s Northbridge.  I
> didn''t think Xen supported pci-gart, but maybe I missed
> a changeset.
Maybe a Gentoo-specific patch?
> At any rate, passing iommu=soft should disable the pci-gart
> and allow you to use bounce buffering.  Hopefully, that
> should clear up your problem.
If it does fix it, I''d suggest reporting it to the package maintainer.

Cheers,
Mark

-- 
Dave: Just a question. What use is a unicyle with no seat?  And no pedals!
Mark: To answer a question with a question: What use is a skateboard?
Dave: Skateboards have wheels.
Mark: My wheel has a wheel!

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

>  > I am attempting to run Xen 3.1.1 on a HP DL385 G2 box (opteron
based
>  > system), using Gentoo.  The machine will boot, and I can login and
edit
>  > files etc, but as soon as I do anything more intensive such as
copying
>  > files off a cdrom (making me think it is a cciss driver problem) or
> running
>  > make, I get the following :
>  >
>  >     kernel panic not syncing: PCI-DMA: Random memory would be DMAed
>  >
FWIW, I am running Xen 3.1.0 with a Debian Etch Dom0 on a DL385G2 in a
production environment without any issues at all...

James

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 18/10/07 20:05, "Zachary Richmond" <zak@ihsdoor.com> wrote:
> Regarding the full error message, all I am getting is:
> 
> kernel panic not syncing: PCI-DMA: Random memory would be DMAed
> 
> since the machine is rebooting after that.  There maybe somewhere else to
get
> more detail, but beyond dmesg or messages I am not sure where else to look.
You should also get a line immediately before that saying:
"PCI-DMA: Out of SW-IOMMU space for xxx bytes at device xxx"

This message makes it clear that your swiotlb is not big enough. The default
size is 2MB if your machine has less than 2GB RAM. Otherwise the default is
64MB. You can increase the swiotlb size by, for example, adding
''swiotlb=128'' as a boot parameter to your dom0 kernel (the
example here
would set the swiotlb size to be 128MB).

Setting the parameter too small will make your system crash (as you have
seen). Setting it bigger than necessary will waste memory. The optimum
setting will depend on your system.

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Keir Fraser wrote:
> On 18/10/07 20:05, "Zachary Richmond" <zak@ihsdoor.com>
wrote:
> 
>> Regarding the full error message, all I am getting is:
>>
>> kernel panic not syncing: PCI-DMA: Random memory would be DMAed
>>
>> since the machine is rebooting after that.  There maybe somewhere else
to get
>> more detail, but beyond dmesg or messages I am not sure where else to
look.
> 
> You should also get a line immediately before that saying:
> "PCI-DMA: Out of SW-IOMMU space for xxx bytes at device xxx"
> 
> This message makes it clear that your swiotlb is not big enough. The
default
> size is 2MB if your machine has less than 2GB RAM. Otherwise the default is
> 64MB. You can increase the swiotlb size by, for example, adding
> ''swiotlb=128'' as a boot parameter to your dom0 kernel
(the example here
> would set the swiotlb size to be 128MB).
> 
> Setting the parameter too small will make your system crash (as you have
> seen). Setting it bigger than necessary will waste memory. The optimum
> setting will depend on your system.
> 
>  -- Keir
> 
Thanks for the tip.  If Mark''s suggestion doesn''t work
I''ll have to give this a
try as well, but I wasn''t receiving that error message.  When I looked
at the
source code it seems like I should be getting that error message as well, but 
it is a printk call which I think is sent to dmesg which is overwritten at the 
next boot.  I guess I could boot off a cd or into a different file system to 
preserve dmesg if the problem persists.

	Zak

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Langsdorf, Mark wrote:
>> To answer Mark''s question:
>>
>>  > What kernel are you using?
>>
>> from gentoo: xen-sources-2.6.20-r6
>>
>> Regarding the full error message, all I am getting is:
>>
>>      kernel panic not syncing: PCI-DMA: Random memory would be DMAed
>>
>> since the machine is rebooting after that.  There maybe 
>> somewhere else to get more detail, but beyond dmesg or
>> messages I am not sure where else to look.
> 
> For some reason, your kernel is running with pci-gart.c
> enabled, and dom0 is trying to allocate DMA through the
> AGP GART in your AMD64 processor''s Northbridge.  I 
> didn''t think Xen supported pci-gart, but maybe I missed
> a changeset.
> 
> At any rate, passing iommu=soft should disable the pci-gart
> and allow you to use bounce buffering.  Hopefully, that
> should clear up your problem.
> 
> -Mark Langsdorf
> Operating System Research Center
> AMD
> 
> 
> 
Unfortunately I am still having the problem.  Having booted off a cd to 
preserve log files a quick grep on dmesg shows:

	PCI-DMA: Using software bounce buffering for IO (SWIOTLB)

So it is enabled, but still the kernel panic.  Any other ideas on where to poke 
around?

	Thanks,

	Zak





_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 19/10/07 20:29, "Zachary Richmond" <admin.ihsdoor@gmail.com>
wrote:
> Unfortunately I am still having the problem.  Having booted off a cd to
> preserve log files a quick grep on dmesg shows:
> 
> PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
> 
> So it is enabled, but still the kernel panic.  Any other ideas on where to
> poke 
> around?
Have you increased the swiotlb size yet?

 K.



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Keir Fraser wrote:
> 
> 
> On 19/10/07 20:29, "Zachary Richmond"
<admin.ihsdoor@gmail.com> wrote:
> 
>> Unfortunately I am still having the problem.  Having booted off a cd to
>> preserve log files a quick grep on dmesg shows:
>>
>> PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
>>
>> So it is enabled, but still the kernel panic.  Any other ideas on where
to
>> poke 
>> around?
> 
> Have you increased the swiotlb size yet?
> 
>  K.
> 
No, but my system on has 2GB for now.  I am however trying it now.

	Zak


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 19/10/07 20:56, "Zachary Richmond" <admin.ihsdoor@gmail.com>
wrote:
>> Have you increased the swiotlb size yet?
>> 
>>  K.
>> 
> 
> No, but my system on has 2GB for now.  I am however trying it now.
Hence you only have a 2MB swiotlb by default. If your block driver cannot
address 2GB (and some controllers can''t) then you''ll be
hitting the swiotlb
hard. Try swiotlb=16.

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Zachary Richmond wrote:
> Keir Fraser wrote:
>>
>>
>> On 19/10/07 20:29, "Zachary Richmond"
<admin.ihsdoor@gmail.com> wrote:
>>
>>> Unfortunately I am still having the problem.  Having booted off a
cd to
>>> preserve log files a quick grep on dmesg shows:
>>>
>>> PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
>>>
>>> So it is enabled, but still the kernel panic.  Any other ideas on 
>>> where to
>>> poke around?
>>
>> Have you increased the swiotlb size yet?
>>
>>  K.
>>
> 
> No, but my system on has 2GB for now.  I am however trying it now.
> 
>     Zak
> 
I set swiotlb=64, rebooted.  As a quick test, I am now joyfully compiling a 
kernel (after a clean) while running on my dom0 kernel.  I''ll have to
do a few
more tests, but since I am running gentoo I will have a bit more compiling to 
do.  My guess is that the default swiotlb value is something else.  Per Mark 
Williamson''s suggestion earlier I''ll have to let the package
maintainer know
about this, and likely add this to http://gentoo-wiki.com/HOWTO_Xen_and_Gentoo.

 > Try swiotlb=16.

For now I''ll probably keep it at 64 for now since I will be adding more
memory
later.  In the meantime I''ll have to read up more on the various
parameters to
be able to fine tune the systems.


  Thanks for all the help,

	Zak


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel