Here''s the latest USB code. This patch is much less crufty than the one I sent out a week ago since I just finished the refactoring which was in-progress in the last patch. In this patch, the inter-domain communication endpoint object (xenidc_endpoint) is theoretically code-complete but is substantially unexecuted so I wouldn''t recommend trying to use it unless you want to help test it. FIXMEs in the patch indicate the work remaining to complete the USB driver which include: 1) Implement the xenidc_rbr_provider_pool and the xenidc_rbr_mapper_pool objects which are used to implement the bulk data transfer between the FE and BE based on the grant-tables API. 2) Update the usbback_driver_port state machine to handle the cn and dn stimuli. 3) Some minor bits and pieces here and there. I also need to hook up the claim and release port interfaces to xenbus. And of course, lots of testing. Enjoy. -- Harry Butterworth <harry@hebutterworth.freeserve.co.uk> _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Sat, 2005-11-05 at 00:50 +0000, Mark Williamson wrote:> > Most of the difficulty is in working with xenstore, which is deeply > > inconvenient as a tool for sequencing dependent operations. That''s what > > resulted in the state machine I posted to the list as a ps file. You > > could probably hack this state machine into the existing drivers with > > only a small amount of additional code but it will just make the drivers > > even more ugly and incomprehensible than they are already. > > Yes, right now I''m working with Xenstore too. The updates I mentioned are > supposed to tackle this problem by providing a better API, automating the > channel setup state machine for the majority of users (i.e. all devices). I > believe a some of this code is fully completed and awaiting push to the main > tree once it''s been reviewed by a few people.Really? I haven''t seen any discussion about this on the list. Or any patches for review for that matter. Another private Cambridge clique?> > > The main reason for me to write the API was to have something concrete > > to code the USB driver against whilst the Xen infrastructure was in > > flux---previously the two changes to xend and the change to grant-tables > > had caused me a huge amount of rework. The API allowed me to complete > > the driver whilst the infrastructure was stabilising. > > Surely you still had to update the code just as often? I guess having it > separated would at least ease the update burden. I''ve rewritten my code > several times for the control plane also; it''s tough working out-of-tree for > long periods based on the unstable tree.No, after I stopped trying to sync up with the xenbus-of-the-week API I just defined the API I wanted, coded my driver to it and it''s only been in the last few weeks that I''ve been implementing the API. Thankfully the xenbus and grant-table interfaces haven''t changed enough to completely break me since then. And, yes, you''ll find it is much easier to update the code when it isn''t all munged together in one big lump.> > > Rather than remove the xenidc stuff, I think it would be much more > > useful to pick up the xenidc code for general use, base all new drivers > > on it and eventually port the block and net drivers to it. This would > > allow you to change the infrastructure underneath all new drivers > > relatively easily and I would expect it to save you a significant amount > > of effort when implementing new drivers, when optimizing the code and, > > in the future, if you extend the system over the network or do any fancy > > page sharing. > > I really would have liked to see a more abstract interface to communications, > such as the one you propose. However, it is not going to happen for the 3.0 > release and that''s when the team promised the interfaces will be frozen. If > it can''t be compatible with the existing interfaces, that''s going to make it > much less likely that anyone will update the existing drivers and then > maintain two separate interdomain interfaces for each. > > I think the project is likely to follow the path of least resistance and go > with a "xenstore connection setup library" that replaces most of the > duplicate code but retain the old datapath API. I get the impression this > decision has been taken already.This doesn''t seem to have been discussed on the list either.> > When I''ve got the code up and working and have split the patch up into a > > series of manageable chunks you should judge it on technical merit. I > > would expect that peer review will find ways to improve the > > implementation and perhaps make it fit better with Xen but I''m confident > > that the basic design is reasonable. > > I agree, but regardless of the goals of the basic design it''s going to be > harder to get a merge for a driver that implements its own complex > abstraction layer than for one which uses the Xen APIs directly. I''d like to > have a nicer driver API but if that isn''t accepted then we''re probably better > with the drivers as small as possible. > > It''s an unfortunate situation; what you propose is a nice way of doing things > but doesn''t appear to be gaining much traction yet. If the USB driver is > going to be predicated on it I''d suggest having a clear plan about how you''re > going to get the generic IDC code into the main tree - I''m not sure whether > rolling it in with the USB driver itself will be accepted.Current plan: write good quality generic code which is reusable and useful to other drivers. Document and break patch down into manageable chunks. Get in to the tree on technical merit. In any case, I can always change the implementation of the xenidc code to base it on whatever API you provide. This is what it was designed for. So at least I won''t have to rewrite the driver again.> > Cheers, > Mark >-- Harry Butterworth <harry@hebutterworth.freeserve.co.uk> _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Harry Butterworth wrote:> On Sat, 2005-11-05 at 00:50 +0000, Mark Williamson wrote:>>Yes, right now I''m working with Xenstore too. The updates I mentioned are >>supposed to tackle this problem by providing a better API, automating the >>channel setup state machine for the majority of users (i.e. all devices). I >>believe a some of this code is fully completed and awaiting push to the main >>tree once it''s been reviewed by a few people.Hi Mark, I didn''t see the original email that Harry quoted, but could you elaborate on that? Are you proposing a new API? Why only a few people? Any reason not on the list? Rusty had asked a question regarding what''s in 3.0: http://marc.theaimsgroup.com/?l=xen-devel&m=113038760216447&w=2 and I guess I''ll second that..>>I really would have liked to see a more abstract interface to communications, >>such as the one you propose. However, it is not going to happen for the 3.0 >>release and that''s when the team promised the interfaces will be frozen. If >>it can''t be compatible with the existing interfaces, that''s going to make it >>much less likely that anyone will update the existing drivers and then >>maintain two separate interdomain interfaces for each. >> >>I think the project is likely to follow the path of least resistance and go >>with a "xenstore connection setup library" that replaces most of the >>duplicate code but retain the old datapath API. I get the impression this >>decision has been taken already.Any reason for this to not be shared on xen-devel? thanks, Nivedita _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 5 Nov 2005, at 04:08, Nivedita Singhvi wrote:>>> I really would have liked to see a more abstract interface to >>> communications, such as the one you propose. However, it is not >>> going to happen for the 3.0 release and that''s when the team >>> promised the interfaces will be frozen. If it can''t be compatible >>> with the existing interfaces, that''s going to make it much less >>> likely that anyone will update the existing drivers and then >>> maintain two separate interdomain interfaces for each. >>> >>> I think the project is likely to follow the path of least resistance >>> and go with a "xenstore connection setup library" that replaces most >>> of the duplicate code but retain the old datapath API. I get the >>> impression this decision has been taken already. > > Any reason for this to not be shared on xen-devel?All that''s being done is the simplest possible bug fixing and refactoring to get stable drivers for a 3.0.0 release asap. xenidc is possibly a suitable alternative for future releases, after we have had time to look at it, integrate it with all the split drivers, and fix bug fallout. Given the size and complexity of the patch, I think it would be very high risk to commit to doing that in the next couple of weeks. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Sat, 2005-11-05 at 10:32 +0000, Keir Fraser wrote:> All that''s being done is the simplest possible bug fixing and > refactoring to get stable drivers for a 3.0.0 release asap. xenidc is > possibly a suitable alternative for future releases, after we have had > time to look at it, integrate it with all the split drivers, and fix > bug fallout. Given the size and complexity of the patch, I think it > would be very high risk to commit to doing that in the next couple of > weeks.Good. This seems entirely reasonable to me. So I''ll release the USB driver against xenidc as an example and document and break down the xenidc patch (and run it through a reformatter) so it can be reviewed. If you like xenidc you can keep it, if not I can factor out the code later. Easy come, easy go :-)> > -- Keir > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel >-- Harry Butterworth <harry@hebutterworth.freeserve.co.uk> _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> > Yes, right now I''m working with Xenstore too. The updates I mentioned > > are supposed to tackle this problem by providing a better API, automating > > the channel setup state machine for the majority of users (i.e. all > > devices). I believe a some of this code is fully completed and awaiting > > push to the main tree once it''s been reviewed by a few people. > > Really? I haven''t seen any discussion about this on the list. Or any > patches for review for that matter. Another private Cambridge clique?I believe I mentioned it to you in a previous e-mail. But no, it hasn''t been on the public list; I found out about it by chance and was trying to keep you informed. It''s not my place to publically announce other people''s work unless it''s relevant to an on-list discussion. That said, it does seem like a logical next step to progress to.> No, after I stopped trying to sync up with the xenbus-of-the-week API I > just defined the API I wanted, coded my driver to it and it''s only been > in the last few weeks that I''ve been implementing the API. Thankfully > the xenbus and grant-table interfaces haven''t changed enough to > completely break me since then. And, yes, you''ll find it is much easier > to update the code when it isn''t all munged together in one big lump.I separated the control code from the data code within my driver, so I just have to rewrite that, rather than the entire driver.> > I think the project is likely to follow the path of least resistance and > > go with a "xenstore connection setup library" that replaces most of the > > duplicate code but retain the old datapath API. I get the impression > > this decision has been taken already. > > This doesn''t seem to have been discussed on the list either.It''s the same thing I mentioned above...> Current plan: write good quality generic code which is reusable and > useful to other drivers. Document and break patch down into manageable > chunks. Get in to the tree on technical merit.I''ll look forward to future patches. Cheers, Mark _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> I didn''t see the original email that Harry quoted, but could > you elaborate on that? Are you proposing a new API?My understanding is that a new Xenbus "library" has been developed that''ll automate the state machines involved in connecting a device channel. This''ll hopefully eliminate some of the "cut and paste" code that''s in all the drivers. The data-plane code and the actual Xenstore setup protocols will remain the same. I believe the code exists (at least partially) but that''s all I know. I''m afraid it''s not my area... Somebody else may be more qualified to comment.> Rusty had asked a question regarding what''s in 3.0: > http://marc.theaimsgroup.com/?l=xen-devel&m=113038760216447&w=2 > > and I guess I''ll second that..There''s a TODO list on the Wiki which should give some idea. Beyond that, it''s hard to know exactly. I''m not directly working on the release code this time round.> >>I think the project is likely to follow the path of least resistance and > >> go with a "xenstore connection setup library" that replaces most of the > >> duplicate code but retain the old datapath API. I get the impression > >> this decision has been taken already. > > Any reason for this to not be shared on xen-devel?In case there''s any confusion again, this "library" is the same code referred to above. I didn''t post on xen-devel because it was speculative, and intended to be shared in private discussion with Harry. Cheers, Mark _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser wrote:> All that''s being done is the simplest possible bug fixing and > refactoring to get stable drivers for a 3.0.0 release asap. xenidc is > possibly a suitable alternative for future releases, after we have had > time to look at it, integrate it with all the split drivers, and fix bug > fallout. Given the size and complexity of the patch, I think it would be > very high risk to commit to doing that in the next couple of weeks.Thanks for the clarification, Keir, good to know! thanks, Nivedita _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Here''s the latest USB code. I''ve spent a few days testing and fixing bugs and it''s now possible to create and mount a filesystem on a USB key in the FE using this code. I''m in the middle of debugging a problem where dom0 reboots when flushing a big file to the USB key. I didn''t manage to find a serial cable for a serial console this evening so I don''t have much to go on yet. Since last week I''ve hooked the claim port call into xenbus such that the configured port is claimed when the device is probed. At the moment the code will generate a 7 port virtual USB hub in the FE for each configured backend device. It would probably be better to either put multiple devices on each FE hub or have single port hubs in the front end. I think I prefer the latter approach. The code doesn''t currently support hot-reconfiguration of the mapping between BE usb ports and FE domains but does support hot plug and unplug of USB devices into configured ports. I also fixed a few FIXMEs in the code including checking that port numbering was consistently one-based and checking that URB serialisation was maintained correctly. Remaining work: o - more testing, fix dom0 reboot issue ASAP. o - reformat to kernel coding style o - split patch up into manageable chunks for review and patch submission o - run the USB code by the usb mailing list o - fix remaining FIXMEs in code, error codes in particular. o - more comments and API documentation If anyone is interested in trying this out the steps are as follows: Apply the patch to a recent xen-unstable. make world Choose to build the usb frontend and backend drivers either as loadable modules or in kernel (both seem to work equally well). Do not enable tracing in the xenidc code or the usb FE or BE (there''s too much tracing). Configure a usb device in a FE domain config script: usb = [''path=1''] This means attach whatever device is in the BE USB port specified by the path (i.e. port 1 in this case) to the FE domain. This will create a USB HUB in the FE domain and whatever device is attached to the BE port will get attached to port one of the FE hub. Boot the FE domain. Load the modules (unless compiled into the kernel). Plug a USB device in. You are supposed to be able to load and unload modules and plug/unplug USB devices however you like but unloading the backend module sometimes hangs when unregistering a watch. I think this is likely to be the deadlock in xenstore that Rusty described. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> I''ve spent a few days testing and fixing bugs and it''s now > possible to create and mount a filesystem on a USB key in the > FE using this code.Excellent, sounds promising.> I''m in the middle of debugging a problem where dom0 reboots > when flushing a big file to the USB key. I didn''t manage to > find a serial cable for a serial console this evening so I > don''t have much to go on yet.You could try ''noreboot'' and stick to a text mode. It''ll be good to get USB functionality back in the tree. Thanks, Ian _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Harry Butterworth
2005-Nov-13 18:28 UTC
RE: [Xen-devel] USB virt status --- Help please!!!
> > I''m in the middle of debugging a problem where dom0 reboots > > when flushing a big file to the USB key. I didn''t manage to > > find a serial cable for a serial console this evening so I > > don''t have much to go on yet. > > You could try ''noreboot'' and stick to a text mode.Thanks, yes this worked. It turns out that dom0 is bombing out in dma_map_single (arch/xen/i386/kernel/pci-dma.c) with a failure of range_straddles_page_boundary and if I use swiotlb=force on the command line as suggested by the printk then it all seems to work fine. Looking at the kernel documentation for the DMA api I get the impression that dma_map_single ought to work across page boundaries provided the pages are contiguous in physical memory. Given that the code seems to work writing and verifying large random files with swiotlb=force I think I''m probably not getting the addresses wrong and I''m assuming that if the memory was good enough for the USB driver in the FE it ought to be good enough for the USB driver in the BE. So, to me this looks like a bug in the xen implementation of dma_map_single which I think ought to not fail just because of crossing a page boundary but only if the memory is discontiguous. But, I don''t really know enough about the memory model to know if this is right. So, I need to know what I ought to be doing with the USB memory mapping. Currently, I''m taking the FE pointer to the memory buffer and creating a grant reference for each page spanned by the request then mapping them into the BE address space in an empty page range taken from the balloon driver. This is the same as the block driver where I copied the code from. Then I pass the address of the start of the buffer in the empty page range in the URB to the usb core. Is this the right approach? Is the check in dma_map_single overzealous? Harry. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 13 Nov 2005, at 18:28, Harry Butterworth wrote:> Is this the right approach? Is the check in dma_map_single overzealous?When running on Xen, just because Linux has allocated adjacent pages from its ''physical'' memory map, doesn''t mean they really are physically contiguous. For I/O we have to go to extra effort to really allocate truly contiguous multi-page extents. So your code is correct. If you can preallocate buffers then you could do that with dma_alloc_coherent and that would guarantee contigous buffers.... -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Mon, 2005-11-14 at 09:56 +0000, Keir Fraser wrote:> On 13 Nov 2005, at 18:28, Harry Butterworth wrote: > > > Is this the right approach? Is the check in dma_map_single overzealous? > > When running on Xen, just because Linux has allocated adjacent pages > from its ''physical'' memory map, doesn''t mean they really are physically > contiguous. For I/O we have to go to extra effort to really allocate > truly contiguous multi-page extents. > > So your code is correct. If you can preallocate buffers then you could > do that with dma_alloc_coherent and that would guarantee contigous > buffers....I''m given the buffers by the assorted USB drivers. I could stage the data using memcpy into my own buffer allocated using dma_alloc_coherent. Or should I go through all the USB drivers and change the buffer allocation?> > -- Keir > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Mon, 2005-11-14 at 10:12 +0000, harry wrote:> On Mon, 2005-11-14 at 09:56 +0000, Keir Fraser wrote: > > On 13 Nov 2005, at 18:28, Harry Butterworth wrote: > > > > > Is this the right approach? Is the check in dma_map_single overzealous? > > > > When running on Xen, just because Linux has allocated adjacent pages > > from its ''physical'' memory map, doesn''t mean they really are physically > > contiguous. For I/O we have to go to extra effort to really allocate > > truly contiguous multi-page extents. > > > > So your code is correct. If you can preallocate buffers then you could > > do that with dma_alloc_coherent and that would guarantee contigous > > buffers.... > > I''m given the buffers by the assorted USB drivers. I could stage the > data using memcpy into my own buffer allocated using dma_alloc_coherent. > Or should I go through all the USB drivers and change the buffer > allocation?It looks like there is an operation usb_buffer_alloc which I can implement to give the right kind of buffers to clients. Any buffers not allocated from here can be staged. Should hopefully be OK. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 14 Nov 2005, at 10:12, harry wrote:>> When running on Xen, just because Linux has allocated adjacent pages >> from its ''physical'' memory map, doesn''t mean they really are >> physically >> contiguous. For I/O we have to go to extra effort to really allocate >> truly contiguous multi-page extents. >> >> So your code is correct. If you can preallocate buffers then you could >> do that with dma_alloc_coherent and that would guarantee contigous >> buffers.... > > I''m given the buffers by the assorted USB drivers. I could stage the > data using memcpy into my own buffer allocated using > dma_alloc_coherent. > Or should I go through all the USB drivers and change the buffer > allocation?Your first approach would be equivalent to just using swiotlb. But it is easy to implement and at least avoids ''use swiotlb=force'' crashes. One possibility is to switch to having swiotlb default to ''on'' and then need to force it off. The second approach would never get past the kernel maintainers. This is a very xen-specific problem and the maintainers won''t allow us to solve it by hacking at generic driver code. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Stephen C. Tweedie
2005-Nov-14 17:30 UTC
Re: [Xen-devel] USB virt status --- Help please!!!
Hi, On Mon, 2005-11-14 at 10:33 +0000, Keir Fraser wrote:> Your first approach would be equivalent to just using swiotlb. But it > is easy to implement and at least avoids ''use swiotlb=force'' crashes. > One possibility is to switch to having swiotlb default to ''on'' and then > need to force it off. > > The second approach would never get past the kernel maintainers. This > is a very xen-specific problem and the maintainers won''t allow us to > solve it by hacking at generic driver code.Having swiotlb the default would also have dealt with the network DMA problem we had recently, with pci_map_single() refusing to work over page boundaries. That got fixed that by adding an arch-specific __alloc_skb which avoided crossing page boundaries by disabling CONFIG_DEBUG_SLAB for the skb caches. But that fix is also something that I doubt maintainers will allow to go upstream. I wonder if we''ll find enough such special cases that we really want to have a swiotlb available, just in case, at all times? In cases where we expect most IO not to need it, we can reduce the default size of it at boot time. swiotlb_map_single() already tests address_needs_mapping() as almost the first thing it does, so hopefully the fast path for cases where the swiotlb is present but not needed will be good enough to have it always available. --Stephen _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 14 Nov 2005, at 17:30, Stephen C. Tweedie wrote:> Having swiotlb the default would also have dealt with the network DMA > problem we had recently, with pci_map_single() refusing to work over > page boundaries. > > That got fixed that by adding an arch-specific __alloc_skb which > avoided > crossing page boundaries by disabling CONFIG_DEBUG_SLAB for the skb > caches. But that fix is also something that I doubt maintainers will > allow to go upstream. I wonder if we''ll find enough such special cases > that we really want to have a swiotlb available, just in case, at all > times?swiotlb performance is pretty poor since it involves memcpy. The alloc_skb fix was pretty clean, if we''re allowed an arch hook for alloc_skb. We didn;t need to hack the slab allocator itself. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Stephen C. Tweedie
2005-Nov-14 18:50 UTC
Re: [Xen-devel] USB virt status --- Help please!!!
Hi, On Mon, 2005-11-14 at 18:42 +0000, Keir Fraser wrote:> > That got fixed that by adding an arch-specific __alloc_skb which > > avoided > > crossing page boundaries by disabling CONFIG_DEBUG_SLAB for the skb > > caches. But that fix is also something that I doubt maintainers will > > allow to go upstream. I wonder if we''ll find enough such special cases > > that we really want to have a swiotlb available, just in case, at all > > times? > > swiotlb performance is pretty poor since it involves memcpy. The > alloc_skb fix was pretty clean, if we''re allowed an arch hook for > alloc_skb. We didn;t need to hack the slab allocator itself.There are two different questions --- whether we should rely on swiotlb in the general case, and whether we should have it available just-in-case for edge cases. I''m not suggesting that we fix the networking problem via swiotlb. But that problem does make me wonder what other edge-cases remain lurking that may bite users later; and if they exist, panicing the kernel when we get an IO we can''t translate directly is a lot worse than falling back to a slow swiotlb path. The fact is, on 2.6 the slab allocator can hand out objects (even sub-pagesized ones) that cross page boundaries. If some driver happens to allocate its own objects from a slab cache and run pci_map_single() on them, it works fine on 2.6 mainline but breaks on Xen w/o swiotlb. The alloc_skb is just avoiding one special case of this. It''s an important special case, sure, but to be robust, might we not want to have a minimal swiotlb cache available at all times as fallback? --Stephen _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Harry Butterworth
2005-Nov-14 20:36 UTC
Re: [Xen-devel] USB virt status --- Help please!!!
On Mon, 2005-11-14 at 13:50 -0500, Stephen C. Tweedie wrote:> Hi, > > On Mon, 2005-11-14 at 18:42 +0000, Keir Fraser wrote: > > > > That got fixed that by adding an arch-specific __alloc_skb which > > > avoided > > > crossing page boundaries by disabling CONFIG_DEBUG_SLAB for the skb > > > caches. But that fix is also something that I doubt maintainers will > > > allow to go upstream. I wonder if we''ll find enough such special cases > > > that we really want to have a swiotlb available, just in case, at all > > > times? > > > > swiotlb performance is pretty poor since it involves memcpy. The > > alloc_skb fix was pretty clean, if we''re allowed an arch hook for > > alloc_skb. We didn;t need to hack the slab allocator itself. > > There are two different questions --- whether we should rely on swiotlb > in the general case, and whether we should have it available > just-in-case for edge cases. > > I''m not suggesting that we fix the networking problem via swiotlb. But > that problem does make me wonder what other edge-cases remain lurking > that may bite users later; and if they exist, panicing the kernel when > we get an IO we can''t translate directly is a lot worse than falling > back to a slow swiotlb path. > > The fact is, on 2.6 the slab allocator can hand out objects (even > sub-pagesized ones) that cross page boundaries. If some driver happens > to allocate its own objects from a slab cache and run pci_map_single() > on them, it works fine on 2.6 mainline but breaks on Xen w/o swiotlb. > > The alloc_skb is just avoiding one special case of this. It''s an > important special case, sure, but to be robust, might we not want to > have a minimal swiotlb cache available at all times as fallback? > > --StephenI agree: there has to be code in the BE somewhere to check that the pages used by the FE are good otherwise there is a big security hole: any FE can toast dom0 by invoking the BUG in pci_map_single. Having swiotlb on by default would be convenient because it would implement the checking once for all the drivers that need it. If it''s off, I''m going to at least have to implement extra checking in the BE to avoid the security hole and my own staging if I want to guarantee that the USB driver will work reliably. So, I''d like it to be on as a fallback. -- Harry Butterworth <harry@hebutterworth.freeserve.co.uk> _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Mon, Nov 14, 2005 at 08:36:07PM +0000, Harry Butterworth wrote:> So, I''d like it to be on as a fallback.I''ve been pondering an extension of my dma_ops patch[0] to allow setting the dma_ops dynamically on a per device / subsystem, somewhat similar to how ppc does it. The objective would be to use nommu where possible and safe, and fallback to swiotlb for devices or subsystems that aren''t safe otherwise. So for net and block we could have nommu and fall back to swiotlb for e.g. USB. What do people think about this approach? Cheers, Muli -- Muli Ben-Yehuda http://www.mulix.org | http://mulix.livejournal.com/ _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Mon, Nov 14, 2005 at 01:50:39PM -0500, Stephen C. Tweedie wrote:> The alloc_skb is just avoiding one special case of this. It''s an > important special case, sure, but to be robust, might we not want to > have a minimal swiotlb cache available at all times as fallback?I think we certainly do. swiotlb will already avoid bouncing where possible via the checks for address_needs_mapping(). Is swiotlb prohibitively expensive even when no bouncing is necessary? if yes, we could either trim it down or just use it selectively as I outlined earlier on a per bus / device / subsystem basis. Cheers, Muli -- Muli Ben-Yehuda http://www.mulix.org | http://mulix.livejournal.com/ _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 14 Nov 2005, at 21:46, Muli Ben-Yehuda wrote:>> The alloc_skb is just avoiding one special case of this. It''s an >> important special case, sure, but to be robust, might we not want to >> have a minimal swiotlb cache available at all times as fallback? > > I think we certainly do. swiotlb will already avoid bouncing where > possible via the checks for address_needs_mapping(). Is swiotlb > prohibitively expensive even when no bouncing is necessary? if yes, we > could either trim it down or just use it selectively as I outlined > earlier on a per bus / device / subsystem basis.Not expensive in time, but it does waste memory if it''s unused. Currently we allocate a 64MB aperture if the machine has memory mapped above 2GB (this catches devices like aacraid that can only address 31 bits). I could also allocate an aperture of, say, 2MB for smaller systems. There is of course a tension between not wasting memory yet having a big enough aperture that the system will not run out of iommu space and crash even when stressed. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Tue, Nov 15, 2005 at 10:23:40AM +0000, Keir Fraser wrote:> Not expensive in time, but it does waste memory if it''s unused. > Currently we allocate a 64MB aperture if the machine has memory mapped > above 2GB (this catches devices like aacraid that can only address 31 > bits). I could also allocate an aperture of, say, 2MB for smaller > systems. There is of course a tension between not wasting memory yet > having a big enough aperture that the system will not run out of iommu > space and crash even when stressed.Since Andi''s ZONE_DMA32 patches went in last night, how about foregoing the up-front allocation of bounce buffers and allocating them as necessary from ZONE_DMA32? once we have that, we should be able to switch to swiotlb by default. Cheers, Muli -- Muli Ben-Yehuda http://www.mulix.org | http://mulix.livejournal.com/ _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel