Xen devel - Feb 2010 - Re: pvops dom0: no sound after boot; possibly causedby swiotlb

>Well, since you are volunteering. The problem sounds like the sound card
>allocates a buffer from the region above 4GB and tries to DMA to it.
>Keep in mind that on most machines, when you have 4GB, 768 MB of it are 
>offset
>past the 4GB mark. You have these two mega regions: 0-3.3GB, 4GB-4.7GB,
>the 3.3GB to 4GB is called the PCI hole.
I added some log-messages in the sound-driver and indeed, it allocates memory
from above 4GB.

On bare metal and with dom0 based on the forward-ported xen-patches from gentoo
the dma-memory is allocated in the 4GB range.

When i checked the dma-memory allocation in the driver i noticed that depending
if the sound-card reports that it is 64bit-compatible the driver sets the
dma_mask to 64, else to 32 (by calling pci_set_consistent_dma_mask).

So i changed the driver that it always sets the dma-mask to 32 - and the sound
worked without the need to start a domU.

Which left the question why it didnt worked out of the box on pvops-dom0.

So i followed the code for the dma-allocation from the sound-driver and found
the problem in the xen-swiotlb code in arch/x86/xen/pci-swiotlb.

Bare metal and the forward-ported dom0 both call dma_alloc_coherent_mask if they
try to allocate coherent dma-memory - pvops dom0 doesnt. This call sets the
DMA-mask to 24 bit when __GFP_DMA is set, else to 32-bit.

I guess this is the right fix for pvops-dom0 too? I will sent a proper patch for
that soon.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Mon, Feb 01, 2010 at 09:54:02PM +0100, Ronny.Hegewald@online.de
wrote:
> >Well, since you are volunteering. The problem sounds like the sound
card
> >allocates a buffer from the region above 4GB and tries to DMA to it.
> >Keep in mind that on most machines, when you have 4GB, 768 MB of it are
> >offset
> >past the 4GB mark. You have these two mega regions: 0-3.3GB, 4GB-4.7GB,
> >the 3.3GB to 4GB is called the PCI hole.
> 
> I added some log-messages in the sound-driver and indeed, it allocates
memory from above 4GB.
> 
> On bare metal and with dom0 based on the forward-ported xen-patches from
gentoo the dma-memory is allocated in the 4GB range.
> 
> When i checked the dma-memory allocation in the driver i noticed that
depending if the sound-card reports that it is 64bit-compatible the driver sets
the dma_mask to 64, else to 32 (by calling pci_set_consistent_dma_mask).
> 
> So i changed the driver that it always sets the dma-mask to 32 - and the
sound worked without the need to start a domU.
Good detective job. BTW, what is the name of the driver in the source
code?
> 
> Which left the question why it didnt worked out of the box on pvops-dom0.
> 
> So i followed the code for the dma-allocation from the sound-driver and
found the problem in the xen-swiotlb code in arch/x86/xen/pci-swiotlb.
> 
> Bare metal and the forward-ported dom0 both call dma_alloc_coherent_mask if
they try to allocate coherent dma-memory - pvops dom0 doesnt. This call sets the
DMA-mask to 24 bit when __GFP_DMA is set, else to 32-bit.
What is your sound-driver detecting the card as? As 64-bit or 32-bit or
none of those?

On pvops we can get away from calling dma_alloc_coherent b/c we have
this piece of logic to determine where the driver can DMA from:

604         if (hwdev != NULL && hwdev->coherent_dma_mask)
605                 mask = hwdev->coherent_dma_mask;
606         else
607                 mask = DMA_BIT_MASK(32);

You mentioned that the sound driver calls the pci_set_consistent_dma_mask. That
function is the one that sets the coherent_dma_mask:

2045 int
2046 pci_set_consistent_dma_mask(struct pci_dev *dev, u64 mask)
2047 {
2048         if (!pci_dma_supported(dev, mask))
2049                 return -EIO;
2050
2051         dev->dev.coherent_dma_mask = mask;

So the ''xen_swiotlb_alloc_coherent'' checks if you have the
coherent DMA
mask and if not, assumes you have a driver that can only access up to
4GB. While the bare-metal assumes that if the driver doesn''t have that
mask , it checks the gfp_t flag and if it has __GFP_DMA make the mask
24-bit, otherwise 32-bit.

The only difference here is that under pvops we behave badly with
devices that have GFP_DMA set and don''t have the coherent_dma_mask
(which it does not seem to be the case?).

So is your sound-driver not detecting the card properly and not setting
the coherent_dma_mask and/or dma_mask?

Can you print out both of those entries when the sound driver
calls the ''xen_swiotlb_alloc_coherent'' (without setting the
flags to 32 forcefully?)


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel