Xen devel - Jan 2010 - pvops dom0: no sound after boot; possibly caused by swiotlb

Software: xen 3.4.1, lastest xen/master (version 2.6.31.6), both 32bit
Hardware: Intel Core2Duo System
          4GB Ram
          Realtek ALC888 soundchip

Initial Symptoms:

When playing audio in dom0 there are just "knock" sounds.

After rmmod the kernel-module for the soundcard (snd-hda-intel) and oss modules
(snd-seq-oss and snd-seq-pcm) and inserting them with modprobe again makes the
sound work. Doing that only with snd-hda-intel doesnt help.

I compiled the sound-modules directly into the kernel but that didnt changed
anything.

This problem doesnt appear with the gentoo-dom0 patches for kernel 2.6.31 so it
looks like a pvops dom0 problem.

Strangely that problem doesnt appear on another system with the same xen-version
and the exactly same kernel. Main-difference is that the other system is a
2-core AMD-system with 2 GB Ram and a different soundcard.

But starting the domU with only 2 GB didnt made any difference.


Final findings:

It finally turns out that when the sound-modules are loaded after a pv-domU is
started the sound in domU works fine from the beginning.

As i suspected a problem in the memory-layout that got "fixed" by the
start of a PV-domU i started the domU with different memory-sizes and found out
that the sound works fine if the domU is started with at least 66 MB. Everything
under that and there is no sound (even the knock-sound is not there)

The first thing i found that had could have to do with the 66 MB was the 64 MB
swiotlb buffer. To check that this is really the problem i changed the code in
arch/x86/xen/pci-swiotlb.c and lowered the allocated buffer to 32MB. After that
change the sound worked from the beginning when the domU was started with less
then 66 MB.


Further investigations:
>From here i dont know at the moment how to investigate that problem further
myself.
Which logs should i post that could help to find the problem?

What further steps could/should i do to investigate that myself?

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 26/01/2010 00:40, "Ronny.Hegewald@online.de"
<Ronny.Hegewald@online.de>
wrote:
> When playing audio in dom0 there are just "knock" sounds.
Your bug report seems to talk about dom0 and then switches seamlessly to
talking about sound and swiotlbs in domUs. It''s unclear.

 -- Keir
> After rmmod the kernel-module for the soundcard (snd-hda-intel) and oss
> modules (snd-seq-oss and snd-seq-pcm) and inserting them with modprobe
again
> makes the sound work. Doing that only with snd-hda-intel doesnt help.
> 
> I compiled the sound-modules directly into the kernel but that didnt
changed
> anything.
> 
> This problem doesnt appear with the gentoo-dom0 patches for kernel 2.6.31
so
> it looks like a pvops dom0 problem.
> 
> Strangely that problem doesnt appear on another system with the same
> xen-version and the exactly same kernel. Main-difference is that the other
> system is a 2-core AMD-system with 2 GB Ram and a different soundcard.
> 
> But starting the domU with only 2 GB didnt made any difference.


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Tue, Jan 26, 2010 at 01:40:12AM +0100, Ronny.Hegewald@online.de
wrote:
> Software: xen 3.4.1, lastest xen/master (version 2.6.31.6), both 32bit
> Hardware: Intel Core2Duo System
>           4GB Ram
>           Realtek ALC888 soundchip
Motherboard?
> 
> Initial Symptoms:
> 
> When playing audio in dom0 there are just "knock" sounds.
> 
> After rmmod the kernel-module for the soundcard (snd-hda-intel) and oss
modules (snd-seq-oss and snd-seq-pcm) and inserting them with modprobe again
makes the sound work. Doing that only with snd-hda-intel doesnt help.
> 
> I compiled the sound-modules directly into the kernel but that didnt
changed anything.
> 
> This problem doesnt appear with the gentoo-dom0 patches for kernel 2.6.31
so it looks like a pvops dom0 problem.
> 
> Strangely that problem doesnt appear on another system with the same
xen-version and the exactly same kernel. Main-difference is that the other
system is a 2-core AMD-system with 2 GB Ram and a different soundcard.
If you have 2G in your Intel box does that make the problem go
away?
> 
> But starting the domU with only 2 GB didnt made any difference.
> 
> 
> Final findings:
> 
> It finally turns out that when the sound-modules are loaded after a pv-domU
is started the sound in domU works fine from the beginning.
> 
> As i suspected a problem in the memory-layout that got "fixed" by
the start of a PV-domU i started the domU with different memory-sizes and found
out that the sound works fine if the domU is started with at least 66 MB.
Everything under that and there is no sound (even the knock-sound is not there)
> 
> The first thing i found that had could have to do with the 66 MB was the 64
MB swiotlb buffer. To check that this is really the problem i changed the code
in arch/x86/xen/pci-swiotlb.c and lowered the allocated buffer to 32MB. After
that change the sound worked from the beginning when the domU was started with
less then 66 MB.
You could also pass in ''swiotlb=16384'' as boot-up parameter-
that would change it to 32MB.
> 
> 
> Further investigations:
> 
> >From here i dont know at the moment how to investigate that problem
further myself.
> 
> Which logs should i post that could help to find the problem?
I am confused.

What I think you are saying is that, when you perform these steps:

1). Boot up machine, have swiotlb=16384 set (or just hand-coded
swiotlb.c to have a default size of 32MB).

2). Start a DomU with more than 66MB allocated to it.

3). Load the sound modules in Dom0

4). Play music/sound/etc in Dom0, the sound works fine.

What I am curious is your E820 table. That is the first thing Xen
prints. You can get that by doing ''xm dmesg''

Also include your ''dmesg'' output for good measure. 

Lastly, try adding this line to your Xen line: dom0_mem=max:2GB and
don''t do any of the above mentioned hacks. Just start the sound module
and see if it works.
> 
> What further steps could/should i do to investigate that myself?
Well, since you are volunteering. The problem sounds like the sound card
allocates a buffer from the region above 4GB and tries to DMA to it.
Keep in mind that on most machines, when you have 4GB, 768 MB of it are offset
past the 4GB mark. You have these two mega regions: 0-3.3GB, 4GB-4.7GB,
the 3.3GB to 4GB is called the PCI hole.

Back to the sound card. The sound card can (I think) only DMA up to 4GB,
so when it tries to fetch data from above 4GB it ends up truncating the
physical address down to 32-bit and fetches data from somewhere else. So
you are probably listening to binary code being played :-)

Having a DomU start makes Xen shrink Dom0 by a certain size and the DMA
window for the sound is moved "down" below the 4GB mark (that I think
is
the problem you are encountering). This should NOT happen if the sound
card driver is using DMA libraries to allocate that buffer (ie,
dma_alloc_coherent, dma_alloc_free, etc). But it might be using
''vmalloc_32'' which on virtualized environments is not
guaranteed to give
a swatch of memory below 4GB. Here are the steps to investigate:
look in the sound card drivers and see where and how it allocates the buffer.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

>> BTW, what is the name of the driver in the source code? 
Its the driver under sound/pci/hda. In hda_intel.c are all the important calls
for DMA and where the coherent_dma_mask is set.
>What is your sound-driver detecting the card as? As 64-bit or 32-bit or
>none of those?
As 64-bit.
>On pvops we can get away from calling dma_alloc_coherent b/c we have
>this piece of logic to determine where the driver can DMA from:
>
>604         if (hwdev != NULL && hwdev->coherent_dma_mask)
>605                 mask = hwdev->coherent_dma_mask;
>606         else
>607                 mask = DMA_BIT_MASK(32);
...
>So the ''xen_swiotlb_alloc_coherent'' checks if you have the
coherent DMA
>mask and if not, assumes you have a driver that can only access up to
>4GB. While the bare-metal assumes that if the driver doesn''t have
that
>mask , it checks the gfp_t flag and if it has __GFP_DMA make the mask
>24-bit, otherwise 32-bit.
But thats not quite all whats dma_alloc_coherent does. As it only returns a
32-bit variable all coherent_dma_mask over 32-bit get casted down. This way
bare-metal makes sure that the dma-mask is never over 32-bit.

Or are you are saying that when the hardware supports 64 bit and has set the
coherent_dma_mask accordingly and dom0 is 32-bit that the allocation of DMA
after the 4 GB should work fine? Because thats the assumption i see in the
pvops-code.

And from what i understood so far the DMA memory should be allocated preferably
in the 24-bit address space or max. 32 bit address space, at least in a 32-bit
kernel.
>The only difference here is that under pvops we behave badly with
>devices that have GFP_DMA set and don''t have the coherent_dma_mask
>(which it does not seem to be the case?).
As i understand it the opposite is the case. If coherent_dma_mask is not set
xen_swiotlb_alloc_coherent sets it to 32-bit. That should work usually (except
the device needs the dma-memory in the 24-bit space).

The problem-case is that the coherent_dma_mask is set. So pvops-dom0 just uses
this value, when bare-metal makes sure that it cant be over 32bit by calling
dma_alloc_coherent_mask.
>So is your sound-driver not detecting the card properly and not setting
>the coherent_dma_mask and/or dma_mask?
>From what i have seen the driver works correct. It checks a register of the
soundcard if it supports 64 bit and sets the coherent_dma_mask to 64bit, else to
32bit.
As my soundcard says that it supports the 64bit the mask is set accordingly.

I added debug-messages in the code to be sure about that, when i researched the
issue.
>Can you print out both of those entries when the sound driver
>calls the ''xen_swiotlb_alloc_coherent'' (without setting
the flags to 32
>forcefully?)
The value of the coherent_dma_mask in xen_swiotlb_alloc_coherent was
0xFFFFFFFFFFFFFFFF when i didnt set 32bit forcefully.

Until now i only checked the coherent_dma_mask flag because as i understand it,
thats the value that is used when coherent dma memory is requested. And i never
saw dma_mask used.

But i can send the values of dma_mask tomorrow if they are useful in that case.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> But thats not quite all whats dma_alloc_coherent does. As it only returns a
32-bit variable all coherent_dma_mask over 32-bit get casted down. This way
bare-metal makes sure that the dma-mask is never over 32-bit.
Ooooh. I completly failed to notice that your dom0 was 32-bit.

But having that there would make the mask always be below
4GB, irregardless if the dom0 is 32 or 64-bit. Which is
exactly what it does on bare-metal. Hmmm, ok: 

http://kerneltrap.org/mailarchive/git-commits-head/2008/10/28/3841954

shows what made that happen.

I am bit worried on the casting - it does not seem to have been the
purpose of that change to utilize that, but that is a upstream problem.

I am curious - if you dom0 is 64-bit, does the sound card
work?
> 
> Or are you are saying that when the hardware supports 64 bit and has set
the coherent_dma_mask accordingly and dom0 is 32-bit that the allocation of DMA
after the 4 GB should work fine? Because thats the assumption i see in the
pvops-code.
Yes. It should have worked fine.
> 
> And from what i understood so far the DMA memory should be allocated
preferably in the 24-bit address space or max. 32 bit address space, at least in
a 32-bit kernel.
> 
> >The only difference here is that under pvops we behave badly with
> >devices that have GFP_DMA set and don''t have the
coherent_dma_mask
> >(which it does not seem to be the case?).
> 
> As i understand it the opposite is the case. If coherent_dma_mask is not
set xen_swiotlb_alloc_coherent sets it to 32-bit. That should work usually
(except the device needs the dma-memory in the 24-bit space).
> 
You are right.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Tue, Feb 02, 2010 at 07:31:16PM -0500, Konrad Rzeszutek Wilk
wrote:
> > But thats not quite all whats dma_alloc_coherent does. As it only
returns a 32-bit variable all coherent_dma_mask over 32-bit get casted down.
This way bare-metal makes sure that the dma-mask is never over 32-bit.
> 
> Ooooh. I completly failed to notice that your dom0 was 32-bit.
> 
> But having that there would make the mask always be below
> 4GB, irregardless if the dom0 is 32 or 64-bit. Which is
<sigh> That is actually incorrect. Looking at the
dma_alloc_coherent_mask I missed the ''!'' and thought it would
set it
irregardless of what the previous value was. That is not the case.

So your fix is good and it should be put in pv-ops. Thanks for tracking
down the problem and fixing it.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel