While writing a driver for a device doing lots of DMA I''ve hit an "swiotlb_full()" problem. This surprised me somewhat as I wouldn''t have expected to need the use of the software TLB - it''s a 64 bit capable device on a server with only 2 GB of RAM, and so I''d have expected to be using a hardware TLB. Is this a peculiarity of Xen, or should I be right to be surprised? I expect I can increase the size of the swiotlb to avoid this problem, but thought I''d check that there isn''t something more fundamentally wrong first. Thanks Kieran _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 5/4/07 15:23, "Kieran Mansley" <kmansley@solarflare.com> wrote:> While writing a driver for a device doing lots of DMA I''ve hit an > "swiotlb_full()" problem. This surprised me somewhat as I wouldn''t have > expected to need the use of the software TLB - it''s a 64 bit capable > device on a server with only 2 GB of RAM, and so I''d have expected to be > using a hardware TLB. Is this a peculiarity of Xen, or should I be > right to be surprised? > > I expect I can increase the size of the swiotlb to avoid this problem, > but thought I''d check that there isn''t something more fundamentally > wrong first.Well, I would agree that it sounds odd. You should definitely investigate -- you don''t want to be invoking the swiotlb on the data path for a modern high-speed device. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Thu, 2007-04-05 at 16:10 +0100, Keir Fraser wrote:> On 5/4/07 15:23, "Kieran Mansley" <kmansley@solarflare.com> wrote: > > While writing a driver for a device doing lots of DMA I''ve hit an > > "swiotlb_full()" problem. This surprised me somewhat as I wouldn''t have > > expected to need the use of the software TLB - it''s a 64 bit capable > > device on a server with only 2 GB of RAM, and so I''d have expected to be > > using a hardware TLB. Is this a peculiarity of Xen, or should I be > > right to be surprised? > > > > I expect I can increase the size of the swiotlb to avoid this problem, > > but thought I''d check that there isn''t something more fundamentally > > wrong first. > > Well, I would agree that it sounds odd. You should definitely investigate -- > you don''t want to be invoking the swiotlb on the data path for a modern > high-speed device.It''s not quite on the data path - it''s hitting this problem when allocating buffers for network access, which typically will come from a preallocated pool rather than being on demand, so perhaps it''s not a big problem. I''ve had a quick look at what''s going on. The call into swiotlb comes from dma_map_single() (which is I think the one defined in arch/i386/kernel/pci-dma-xen.c). This happens because the swiotlb global is set to 1 in swiotlb_init(). It looks like this is unavoidable as there''s a comment there saying: /* Domain 0 always has a swiotlb. */ I could undefine CONFIG_SWIOTLB or set swiotlb_force to -1 and so avoid the whole thing, but I''m not sure that would be a good thing - some other drivers might depend on it. I guess what I''m surprised about is that there''s no check to see if swiotlb is necessary on any of the above paths. Perhaps I should be calling something other than dma_map_single()? Or perhaps this is just the way it''s supposed to be and I need to allocate a bigger swiotlb? Any suggestions welcome. Kieran _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Tue, Apr 10, 2007 at 09:41:55AM +0100, Kieran Mansley wrote:> It''s not quite on the data path - it''s hitting this problem when > allocating buffers for network access, which typically will come > from a preallocated pool rather than being on demand, so perhaps > it''s not a big problem. > > I''ve had a quick look at what''s going on. The call into swiotlb comes > from dma_map_single() (which is I think the one defined in > arch/i386/kernel/pci-dma-xen.c). This happens because the swiotlb > global is set to 1 in swiotlb_init(). It looks like this is unavoidable > as there''s a comment there saying: > /* Domain 0 always has a swiotlb. */ > > I could undefine CONFIG_SWIOTLB or set swiotlb_force to -1 and so > avoid the whole thing, but I''m not sure that would be a good thing - > some other drivers might depend on it.swiotlb is used to give you machine-contigous buffers if your buffers aren''t machine-contigous. Just because it''s enabled doesn''t mean it will be used - there''s a check whether it''s needed in every map/alloc call and if it isn''t it''s not used. If all your buffers are machine-contigous you shouldn''t ever need to use any space in the swiotlb aperture. If they aren''t, that''s probably the real problem. Cheers, Muli _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Maybe Matching Threads
- [PATCH 35/35] x86: Don't panic if can not alloc buffer for swiotlb
- [PATCH 35/35] x86: Don't panic if can not alloc buffer for swiotlb
- blkif_map error starting fourth guest domain
- [PATCH 04/10] s390/mm: force swiotlb for protected virtualization
- [PATCH 04/10] s390/mm: force swiotlb for protected virtualization