Xen devel - Apr 2007 - Swiotlb

While writing a driver for a device doing lots of DMA I''ve hit an
"swiotlb_full()" problem.  This surprised me somewhat as I
wouldn''t have
expected to need the use of the software TLB - it''s a 64 bit capable
device on a server with only 2 GB of RAM, and so I''d have expected to
be
using a hardware TLB.  Is this a peculiarity of Xen, or should I be
right to be surprised?

I expect I can increase the size of the swiotlb to avoid this problem,
but thought I''d check that there isn''t something more
fundamentally
wrong first.

Thanks

Kieran


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 5/4/07 15:23, "Kieran Mansley" <kmansley@solarflare.com>
wrote:
> While writing a driver for a device doing lots of DMA I''ve hit an
> "swiotlb_full()" problem.  This surprised me somewhat as I
wouldn''t have
> expected to need the use of the software TLB - it''s a 64 bit
capable
> device on a server with only 2 GB of RAM, and so I''d have expected
to be
> using a hardware TLB.  Is this a peculiarity of Xen, or should I be
> right to be surprised?
> 
> I expect I can increase the size of the swiotlb to avoid this problem,
> but thought I''d check that there isn''t something more
fundamentally
> wrong first.
Well, I would agree that it sounds odd. You should definitely investigate --
you don''t want to be invoking the swiotlb on the data path for a modern
high-speed device.

 -- Keir


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Thu, 2007-04-05 at 16:10 +0100, Keir Fraser wrote:
> On 5/4/07 15:23, "Kieran Mansley" <kmansley@solarflare.com>
wrote:
> > While writing a driver for a device doing lots of DMA I''ve
hit an
> > "swiotlb_full()" problem.  This surprised me somewhat as I
wouldn''t have
> > expected to need the use of the software TLB - it''s a 64 bit
capable
> > device on a server with only 2 GB of RAM, and so I''d have
expected to be
> > using a hardware TLB.  Is this a peculiarity of Xen, or should I be
> > right to be surprised?
> > 
> > I expect I can increase the size of the swiotlb to avoid this problem,
> > but thought I''d check that there isn''t something
more fundamentally
> > wrong first.
> 
> Well, I would agree that it sounds odd. You should definitely investigate
--
> you don''t want to be invoking the swiotlb on the data path for a
modern
> high-speed device.
It''s not quite on the data path - it''s hitting this problem
when
allocating buffers for network access, which typically will come from a
preallocated pool rather than being on demand, so perhaps it''s not a
big
problem.

I''ve had a quick look at what''s going on.  The call into
swiotlb comes
from dma_map_single() (which is I think the one defined in
arch/i386/kernel/pci-dma-xen.c).  This happens because the swiotlb
global is set to 1 in swiotlb_init().  It looks like this is unavoidable
as there''s a comment there saying:
        /* Domain 0 always has a swiotlb. */

I could undefine CONFIG_SWIOTLB or set swiotlb_force to -1 and so avoid
the whole thing, but I''m not sure that would be a good thing - some
other drivers might depend on it.

I guess what I''m surprised about is that there''s no check to
see if
swiotlb is necessary on any of the above paths.  Perhaps I should be
calling something other than dma_map_single()?  Or perhaps this is just
the way it''s supposed to be and I need to allocate a bigger swiotlb?
Any suggestions welcome.

Kieran


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Tue, Apr 10, 2007 at 09:41:55AM +0100, Kieran Mansley wrote:
> It''s not quite on the data path - it''s hitting this
problem when
> allocating buffers for network access, which typically will come
> from a preallocated pool rather than being on demand, so perhaps
> it''s not a big problem.
> 
> I''ve had a quick look at what''s going on.  The call into
swiotlb comes
> from dma_map_single() (which is I think the one defined in
> arch/i386/kernel/pci-dma-xen.c).  This happens because the swiotlb
> global is set to 1 in swiotlb_init().  It looks like this is unavoidable
> as there''s a comment there saying:
>         /* Domain 0 always has a swiotlb. */
> 
> I could undefine CONFIG_SWIOTLB or set swiotlb_force to -1 and so
> avoid the whole thing, but I''m not sure that would be a good thing
-
> some other drivers might depend on it.
swiotlb is used to give you machine-contigous buffers if your buffers
aren''t machine-contigous. Just because it''s enabled
doesn''t mean it
will be used - there''s a check whether it''s needed in every
map/alloc
call and if it isn''t it''s not used. If all your buffers are
machine-contigous you shouldn''t ever need to use any space in the
swiotlb aperture. If they aren''t, that''s probably the real
problem.

Cheers,
Muli



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel