Xen devel - Nov 2006 - [Patch] Constrain DMADOM/swiotlb allocations to 1G phys to fix b44 NIC driver

Hi all,

The b44 driver currently won''t load on Xen kernels with 1G memory or
more.  It tries to set a pci dma mask of 0x3ffffff (ie. it can only
address 1G physical memory), but the swiotlb cannot deal with this ---
swiotlb memory is allocated from the hypervisor''s DMADOM heap, which is
restricted to the bottom 2GB of physical memory.  So the pci dma mask
cannot be set and the driver load fails with

        b44 0000:03:00.0: No usable DMA configuration, aborting.

It turns out that this 2GB / 31-bit constraint is magically hardcoded in
a couple of places rather than being read correctly from config.h.  So
this patch fixes that; changes config.h to specify a 1GB / 30-bit
constraint for the DMADOM heap; and changes the x86/x86_64 swiotlb to
request a 30-bit constraint for allocations of the swiotlb memory.

With this in place, b44 works.  (Red Hat bugzilla #208242).

--Stephen



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi,

On Tue, 2006-11-28 at 15:46 +0000, Stephen C. Tweedie wrote:
> It turns out that this 2GB / 31-bit constraint is magically hardcoded in
> a couple of places rather than being read correctly from config.h.  So
> this patch fixes that; changes config.h to specify a 1GB / 30-bit
> constraint for the DMADOM heap; and changes the x86/x86_64 swiotlb to
> request a 30-bit constraint for allocations of the swiotlb memory.
The 30 bit constraint is enough to let b44 work.  But there are a number
of sound cards which require a 28-bit DMA mask (256MB max physical
addressable memory).  1GB seems reasonably safe, but I''m much less
comfortable with restricting DMADOM to 256MB by default.  Does anyone
have any strong feelings about shifting this?  What other users of
DMADOM are there that might overflow if we restrict it even further?

Cheers,
 Stephen



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

How much should we care about Broadcom 4400? Wasn''t it mainly used in
old
notebooks?

 -- Keir

On 28/11/06 15:46, "Stephen C. Tweedie" <sct@redhat.com> wrote:
> Hi all,
> 
> The b44 driver currently won''t load on Xen kernels with 1G memory
or
> more.  It tries to set a pci dma mask of 0x3ffffff (ie. it can only
> address 1G physical memory), but the swiotlb cannot deal with this ---
> swiotlb memory is allocated from the hypervisor''s DMADOM heap,
which is
> restricted to the bottom 2GB of physical memory.  So the pci dma mask
> cannot be set and the driver load fails with
> 
>         b44 0000:03:00.0: No usable DMA configuration, aborting.
> 
> It turns out that this 2GB / 31-bit constraint is magically hardcoded in
> a couple of places rather than being read correctly from config.h.  So
> this patch fixes that; changes config.h to specify a 1GB / 30-bit
> constraint for the DMADOM heap; and changes the x86/x86_64 swiotlb to
> request a 30-bit constraint for allocations of the swiotlb memory.
> 
> With this in place, b44 works.  (Red Hat bugzilla #208242).
> 
> --Stephen
> 
> # HG changeset patch
> # User Stephen Tweedie <sct@redhat.com>
> # Date 1164722972 0
> # Node ID 6cbddc31f52d0bad2aa4b13452a4092dfeada47f
> # Parent  0af1ba62a14b4eaf1c0687d855edc7307edb21ba
> Clean up the DMADOM physical upper limit handling to properly honour
> settings in config.h.
> 
> Set the upper limit for DMADOM and swiotlb allocations to be 30 bits
> (1G) instead of 31 bits, to allow the b44 NIC driver to work.
> 
> Signed-off-by: Stephen Tweedie <sct@redhat.com>
> 
> diff -r 0af1ba62a14b -r 6cbddc31f52d
> linux-2.6-xen-sparse/arch/i386/kernel/swiotlb.c
> --- a/linux-2.6-xen-sparse/arch/i386/kernel/swiotlb.c Tue Nov 28 11:47:28
2006
> +0000
> +++ b/linux-2.6-xen-sparse/arch/i386/kernel/swiotlb.c Tue Nov 28 14:09:32
2006
> +0000
> @@ -47,8 +47,8 @@ EXPORT_SYMBOL(swiotlb);
>   */
>  #define IO_TLB_SHIFT 11
>  
> -/* Width of DMA addresses in the IO TLB. 31 bits is an aacraid limitation.
*/
> -#define IO_TLB_DMA_BITS 31
> +/* Width of DMA addresses in the IO TLB. 30 bits is a b44 limitation. */
> +#define IO_TLB_DMA_BITS 30
>  
>  static int swiotlb_force;
>  static char *iotlb_virt_start;
> diff -r 0af1ba62a14b -r 6cbddc31f52d xen/common/memory.c
> --- a/xen/common/memory.c Tue Nov 28 11:47:28 2006 +0000
> +++ b/xen/common/memory.c Tue Nov 28 14:09:32 2006 +0000
> @@ -328,7 +328,7 @@ static long memory_exchange(XEN_GUEST_HA
>           (exch.out.address_bits <
>            (get_order_from_pages(max_page) + PAGE_SHIFT)) )
>      {
> -        if ( exch.out.address_bits < 31 )
> +        if ( exch.out.address_bits < MAX_DMADOM_BITS )
>          {
>              rc = -ENOMEM;
>              goto fail_early;
> @@ -541,7 +541,7 @@ long do_memory_op(unsigned long cmd, XEN
>               (reservation.address_bits <
>                (get_order_from_pages(max_page) + PAGE_SHIFT)) )
>          {
> -            if ( reservation.address_bits < 31 )
> +            if ( reservation.address_bits < MAX_DMADOM_BITS )
>                  return start_extent;
>              args.memflags = MEMF_dma;
>          }
> diff -r 0af1ba62a14b -r 6cbddc31f52d xen/include/asm-ia64/config.h
> --- a/xen/include/asm-ia64/config.h Tue Nov 28 11:47:28 2006 +0000
> +++ b/xen/include/asm-ia64/config.h Tue Nov 28 14:09:32 2006 +0000
> @@ -41,7 +41,9 @@
>  #define CONFIG_IOSAPIC
>  #define supervisor_mode_kernel (0)
>  
> -#define MAX_DMADOM_PFN (0x7FFFFFFFUL >> PAGE_SHIFT) /* 31
addressable bits */
> +#define MAX_DMADOM_BITS 30
> +#define MAX_DMADOM_MASK ((1UL << MAX_DMADOM_BITS) - 1)
> +#define MAX_DMADOM_PFN  (MAX_DMADOM_MASK >> PAGE_SHIFT)
>  
>  /* If PERFC is used, include privop maps.  */
>  #ifdef PERF_COUNTERS
> diff -r 0af1ba62a14b -r 6cbddc31f52d xen/include/asm-x86/config.h
> --- a/xen/include/asm-x86/config.h Tue Nov 28 11:47:28 2006 +0000
> +++ b/xen/include/asm-x86/config.h Tue Nov 28 14:09:32 2006 +0000
> @@ -82,7 +82,9 @@
>  /* Debug stack is restricted to 8kB by guard pages. */
>  #define DEBUG_STACK_SIZE 8192
>  
> -#define MAX_DMADOM_PFN 0x7FFFFUL /* 31 addressable bits */
> +#define MAX_DMADOM_BITS 30
> +#define MAX_DMADOM_MASK ((1UL << MAX_DMADOM_BITS) - 1)
> +#define MAX_DMADOM_PFN  (MAX_DMADOM_MASK >> PAGE_SHIFT)
>  
>  #ifndef __ASSEMBLY__
>  extern unsigned long _end; /* standard ELF symbol */
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 28/11/06 15:49, "Stephen C. Tweedie" <sct@redhat.com> wrote:
>> It turns out that this 2GB / 31-bit constraint is magically hardcoded
in
>> a couple of places rather than being read correctly from config.h.  So
>> this patch fixes that; changes config.h to specify a 1GB / 30-bit
>> constraint for the DMADOM heap; and changes the x86/x86_64 swiotlb to
>> request a 30-bit constraint for allocations of the swiotlb memory.
> 
> The 30 bit constraint is enough to let b44 work.  But there are a number
> of sound cards which require a 28-bit DMA mask (256MB max physical
> addressable memory).  1GB seems reasonably safe, but I''m much less
> comfortable with restricting DMADOM to 256MB by default.  Does anyone
> have any strong feelings about shifting this?  What other users of
> DMADOM are there that might overflow if we restrict it even further?
We could make bit-width of the DMA allocator in Xen a boot parameter. And we
could make the bit-width of the SWIOTLB allocations a Linux boot parameter.
Then those that care can do the necessary cmdline hacking.

I''d be tempted to recommend this for b44 users too, and leave the
default
bit-width as 31. I''m not convinced there can be that many who are using
Xen.
But maybe the chipset is more common than I think.

 -- Keir


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Tue, 28 Nov 2006 15:54:10 +0000
Keir Fraser <keir@xensource.com> wrote:
> How much should we care about Broadcom 4400? Wasn''t it mainly used
in old
> notebooks?
The same problem is found with other devices including some current
wireless cards.

Alan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Tue, Nov 28, 2006 at 04:31:33PM +0000, Alan wrote:
> > > constraint for the DMADOM heap; and changes the x86/x86_64
swiotlb to
> > > request a 30-bit constraint for allocations of the swiotlb
memory.
> 
> (The hardware pseudo-iommu on the AMD can''t handle this either as
it
> can only appear in the PCI window)
Assuming you mean GART, Xen doesn''t support it at the moment (more
accurately, dom0 cannot run with GART enabled on Xen). There were
patches for this from AMD, but they weren''t merged AFAICR.

Cheers,
Muli


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> > constraint for the DMADOM heap; and changes the x86/x86_64 swiotlb to
> > request a 30-bit constraint for allocations of the swiotlb memory.
(The hardware pseudo-iommu on the AMD can''t handle this either as it
can
only appear in the PCI window)
> The 30 bit constraint is enough to let b44 work.  But there are a number
> of sound cards which require a 28-bit DMA mask (256MB max physical
> addressable memory).  1GB seems reasonably safe, but I''m much less
> comfortable with restricting DMADOM to 256MB by default.  Does anyone
> have any strong feelings about shifting this?  What other users of
> DMADOM are there that might overflow if we restrict it even further?
For modern hardware the constraints are primarily the 30/31bit ones found
on some devices that know too much about Windows. The older sound cards
with 28bit limits like the ESS Maestro series are very much "old tech"
now. 

Alan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 28/11/06 16:31, "Alan" <alan@lxorguk.ukuu.org.uk> wrote:
>> The 30 bit constraint is enough to let b44 work.  But there are a
number
>> of sound cards which require a 28-bit DMA mask (256MB max physical
>> addressable memory).  1GB seems reasonably safe, but I''m much
less
>> comfortable with restricting DMADOM to 256MB by default.  Does anyone
>> have any strong feelings about shifting this?  What other users of
>> DMADOM are there that might overflow if we restrict it even further?
> 
> For modern hardware the constraints are primarily the 30/31bit ones found
> on some devices that know too much about Windows. The older sound cards
> with 28bit limits like the ESS Maestro series are very much "old
tech"
> now. 
Then it sounds like the restriction to a 1GB DMA pool is a reasonable
one-off change rather than the thin end of a wedge.

 -- Keir


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> On Tue, Nov 28, 2006 at 04:31:33PM +0000, Alan wrote:
> > > > constraint for the DMADOM heap; and changes the x86/x86_64
swiotlb to
> > > > request a 30-bit constraint for allocations of the swiotlb
memory.
> >
> > (The hardware pseudo-iommu on the AMD can''t handle this
either as it
> > can only appear in the PCI window)
> 
> Assuming you mean GART, Xen doesn''t support it at the moment (more
> accurately, dom0 cannot run with GART enabled on Xen). There were
> patches for this from AMD, but they weren''t merged AFAICR.
It would be good to see those patches floated on the list again, along
with some benchmarks to show that they beat bounce buffers.

Thanks,
Ian

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel