David Muench
2005-Jul-07 16:07 UTC
[Xen-devel] Re: [ivtv-devel] Problems loading ivtv in Xen - DMA issues?
Hi Keir, I recieved this reply to my request for help with ivtv on Xen on the ivtv-devel list. I don''t feel very well qualified to respond to Andrew. Can I ask what your thoughts are? Am I just out of luck? Thanks, Dave On 7/7/05, Andrew May <acmay@acmay.homeip.net> wrote:> On Wed, 2005-07-06 at 10:20 -0400, David Muench wrote: > > "I think the ivtv driver is probably not calculating dma addresses in > > the way that xen requires. On native Linux, if you allocate a > > multi-page chunk of physical memory, you can pass the start address of > > that buffer to hardware and it can dma the entire buffer given just > > that address. In Xen, because we give guests ''pseudo-physical'' memory, > > that physical buffer may not be really physically contiguous. So we > > need drivers to dma_alloc_coherent or pci_alloc_consistent the memory > > they will use for dma --- we modified those functions to ensure they > > return suitable contiguous physical memory." > > The Xen people seem to be a bit off here. You may want to look at > linux/Documentation/DMA-mapping.txt. > > But alloc_coherent and alloc_consistent are for uncached mem for DMA and > that is very different than normal mem for DMA. That could really slow > things down when getting the data out of buffers latter. > > pci_map_single is correct for doing bulk dma to normal kmalloc data. > > The coherent/consistent mem doesn''t make much sense on x86 stuff, but > for other archs like some PPC ones without cache snooping mem it becomes > a big issue. The CPU and the DMA device cannot write to the same > cacheline of data without causing havoc. The CPU would work with > the data in cache, but the DMA would go directly to DRAM. When the > cache flush does happen it would overwrite half the work done. > > This only comes into play on a smaller ring of descriptors for most HW. > > If the bulk of the data is only touched by one device at a time, then > the map_single function just needs to do a cache flush before the > transfer happens along with the other work. > > I am much more familar with network drivers where it is common for one > consistent set of mem for the descriptors, but all the packets do just > the map_single(). But since jumbo frames aren''t typical yet, the packets > all fit in one page. > > So it looks like the Xen people have some work to do if they really want > to support any type of DMA device. > > > > > ------------------------------------------------------- > SF.Net email is sponsored by: Discover Easy Linux Migration Strategies > from IBM. Find simple to follow Roadmaps, straightforward articles, > informative Webcasts and more! Get everything you need to get up to > speed, fast. http://ads.osdn.com/?ad_id=7477&alloc_id=16492&op=click > _______________________________________________ > ivtv-devel mailing list > ivtv-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/ivtv-devel >-- David Muench - davemuench@gmail.com Jabber ID: dave@jabber.wasteland.org _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2005-Jul-07 17:22 UTC
[Xen-devel] Re: [ivtv-devel] Problems loading ivtv in Xen - DMA issues?
On 7 Jul 2005, at 17:07, David Muench wrote:> I recieved this reply to my request for help with ivtv on Xen on the > ivtv-devel list. I don''t feel very well qualified to respond to > Andrew. Can I ask what your thoughts are? Am I just out of luck?Andrew''s simply pointing out that the ivtv driver already uses the dma-mapping interface properly. It''s good to have that confirmed. :-) However, looking back at XenLinux''s implementation of {dma,pci}_map_single, it is broken for multi-page buffers. It''s surprising noone has hit a problem before now, but most devices other than video-capture cards you deal with only up to a page at a time. So this is a Xen bug: I''ll find some time to fix this and get back to you. Are you using Xen 2.0 or 3.0 (unstable)? -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
David Muench
2005-Jul-07 17:34 UTC
[Xen-devel] Re: [ivtv-devel] Problems loading ivtv in Xen - DMA issues?
On 7/7/05, Keir Fraser <Keir.Fraser@cl.cam.ac.uk> wrote:> > On 7 Jul 2005, at 17:07, David Muench wrote: > > > I recieved this reply to my request for help with ivtv on Xen on the > > ivtv-devel list. I don''t feel very well qualified to respond to > > Andrew. Can I ask what your thoughts are? Am I just out of luck? > > Andrew''s simply pointing out that the ivtv driver already uses the > dma-mapping interface properly. It''s good to have that confirmed. :-) > > However, looking back at XenLinux''s implementation of > {dma,pci}_map_single, it is broken for multi-page buffers. It''s > surprising noone has hit a problem before now, but most devices other > than video-capture cards you deal with only up to a page at a time. > > So this is a Xen bug: I''ll find some time to fix this and get back to > you. Are you using Xen 2.0 or 3.0 (unstable)?That''s great news, I thought I was up a creek. I''m using 2.0.6. Thanks again, I really appreciate your help. Dave -- David Muench - davemuench@gmail.com Jabber ID: dave@jabber.wasteland.org _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Andrew May
2005-Jul-08 04:01 UTC
[Xen-devel] Re: [ivtv-devel] Problems loading ivtv in Xen - DMA issues?
On Thu, 2005-07-07 at 13:34 -0400, David Muench wrote:> On 7/7/05, Keir Fraser <Keir.Fraser@cl.cam.ac.uk> wrote: > > > > On 7 Jul 2005, at 17:07, David Muench wrote: > > > > > I recieved this reply to my request for help with ivtv on Xen on the > > > ivtv-devel list. I don''t feel very well qualified to respond to > > > Andrew. Can I ask what your thoughts are? Am I just out of luck? > > > > Andrew''s simply pointing out that the ivtv driver already uses the > > dma-mapping interface properly. It''s good to have that confirmed. :-)I am not sure I can really confirm that, it was more the suggestion of pci_alloc_consistent that looked very wrong. I only did a quick check in ivtv-queue.c and most of the stuff looks in order. But since I don''t really know the real flow of data it is hard to say for sure. But given that the driver works for me it must be correct. :> _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2005-Jul-08 13:05 UTC
[Xen-devel] Re: [ivtv-devel] Problems loading ivtv in Xen - DMA issues?
On 7 Jul 2005, at 18:34, David Muench wrote:>> So this is a Xen bug: I''ll find some time to fix this and get back to >> you. Are you using Xen 2.0 or 3.0 (unstable)? > > That''s great news, I thought I was up a creek. I''m using 2.0.6. Thanks > again, I really appreciate your help.I''ve pushed the fix to our xen-2.0-testing hg repository on xenbits.xensource.com. If you don''t want to use hg, I''ve also put a source tarball at http://www.cl.cam.ac.uk/~kaf24/xen-2.0-testing.tar.gz (you''ll still need to build it yourself though). Please let me know if this fixes your problem -- if so then I''ll also apply the fix to our unstable repository. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
David Muench
2005-Jul-08 15:38 UTC
[Xen-devel] Re: [ivtv-devel] Problems loading ivtv in Xen - DMA issues?
On 7/8/05, Keir Fraser <Keir.Fraser@cl.cam.ac.uk> wrote:> > I''ve pushed the fix to our xen-2.0-testing hg repository on > xenbits.xensource.com. If you don''t want to use hg, I''ve also put a > source tarball at http://www.cl.cam.ac.uk/~kaf24/xen-2.0-testing.tar.gz > (you''ll still need to build it yourself though). > > Please let me know if this fixes your problem -- if so then I''ll also > apply the fix to our unstable repository.Hi Keir, Looks like it didn''t quite work. I started a capture and then ended it 20 seconds later, and I still have a zero byte file. Ivtv has some new messages in dmesg though: saa7115: decoder disable output saa7115: decoder set input (8) saa7115: now setting S-Video input ivtv: Setting audio matrix to input 1, output 1 saa7115: decoder enable output saa7115: decoder disable output saa7115: decoder enable output ivtv: DEC: REG_ENCSG1LEN wait failed ivtv: DMA Registers State: xfer: 0x00000002, state: 0x00000001 dec_addr: 0x00000000 enc_addr: 0x44360060 control: 0x00000003 ivtv: DMA DEC Buffers: 0x00000000:0x00000000:0x00000000 ivtv: DMA ENC Buffers: 0x00000000:0x00000000:0x00000000 0x00000000:0x00000000:0x00000000 0x00000000:0x00000000:0x00000000 0x00000000:0x00000000:0x00000000 0x00000000:0x00000000:0x00000000 0x00000000:0x00000000:0x00000000 0x00000000:0x00000000:0x00000000 0x00000000:0x00000000:0x00000000 ivtv: ENC: REG_DMAXFER wait failed ivtv: DMA Registers State: xfer: 0x00000002, state: 0x00000001 dec_addr: 0x00000000 enc_addr: 0x44360060 control: 0x00000003 ivtv: DMA DEC Buffers: 0x00000000:0x00000000:0x00000000 ivtv: DMA ENC Buffers: 0x00000000:0x00000000:0x00000000 0x00000000:0x00000000:0x00000000 0x00000000:0x00000000:0x00000000 0x00000000:0x00000000:0x00000000 0x00000000:0x00000000:0x00000000 0x00000000:0x00000000:0x00000000 0x00000000:0x00000000:0x00000000 0x00000000:0x00000000:0x00000000 ivtv: ENC: User stopped capture. ivtv: ENC: DMA still Pending while stoping capture! ivtv: ENC: REG_DMASTATUS2 wait failed ivtv: DMA Registers State: xfer: 0x00000002, state: 0x00000001 dec_addr: 0x00000000 enc_addr: 0x44360060 control: 0x00000003 ivtv: DMA DEC Buffers: 0x00000000:0x00000000:0x00000000 ivtv: DMA ENC Buffers: 0x00000000:0x00000000:0x00000000 0x00000000:0x00000000:0x00000000 0x00000000:0x00000000:0x00000000 0x00000000:0x00000000:0x00000000 0x00000000:0x00000000:0x00000000 0x00000000:0x00000000:0x00000000 0x00000000:0x00000000:0x00000000 0x00000000:0x00000000:0x00000000 Thanks, Dave -- David Muench - davemuench@gmail.com Jabber ID: dave@jabber.wasteland.org _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2005-Jul-08 16:50 UTC
[Xen-devel] Re: [ivtv-devel] Problems loading ivtv in Xen - DMA issues?
On 8 Jul 2005, at 16:38, David Muench wrote:> Hi Keir, > > Looks like it didn''t quite work. I started a capture and then ended it > 20 seconds later, and I still have a zero byte file. Ivtv has some new > messages in dmesg though:Well, it''s not clear whether this is a bug in Xen or the ivtv driver. :-) Basically, my fix for dma_map_single() introduces bounce buffers. These should work okay if: (a) I haven''t screwed up the bounce-buffer code; and (b) The ivtv driver is scrupulously careful in its use of the streaming DMA interface. For example, it is not good for the driver to read/write the buffer without calling unmap, or sync_for_cpu. I see that the driver does do the right thing at least most of the time but, of course, if there are any bugs they won''t actually cause problems on most architectures (i386 in particular). OTOH, the bounce buffer code is not exactly thoroughly tested, so it is feasible the bug lies there. Although the implementation is pretty simple. I think the bottom line is that this will need fixing by someone with the hardware -- probably by adding tracing to the map/sync functions to check that they are used correctly, and to sanity-check the bounce buffer code. Some more detail on how the bounce buffers are implemented. It''s very similar to what arch/arm does for these streaming DMA calls: * dma_map_single() allocates a physically contiguous bounce buffer and, if dir!=DMA_FROM_DEVICE, copies the host buffer into the bounce buffer. * dma_unmap_single() copies bounce -> host (if dir!=DMA_TO_DEVICE) and then frees the bounce buffer * dma_sync_single_to_{cpu,device}() copy to/from the bounce buffer as appropriate, if teh direction argument is in the matching direction. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2005-Jul-11 10:10 UTC
[Xen-devel] Re: [ivtv-devel] Problems loading ivtv in Xen - DMA issues?
On 8 Jul 2005, at 16:38, David Muench wrote:> Looks like it didn''t quite work. I started a capture and then ended it > 20 seconds later, and I still have a zero byte file. Ivtv has some new > messages in dmesg though:One last thing you might want to try is to edit linux-2.6.11/arch/xen/i386/kernel/pci-dma.c. There you will see that there are four separate uses of memcpy(), each conditional on a test of the form ''if (direction != DMA_xxx)''. You *might* have more success if you remove the direction-conditional tests and always memcpy. Apart from that, this needs someone with hardware and expertise, and who cares about both ivtv and xen, to wade in. :-) -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
David Muench
2005-Jul-11 13:08 UTC
[Xen-devel] Re: [ivtv-devel] Problems loading ivtv in Xen - DMA issues?
On 7/11/05, Keir Fraser <Keir.Fraser@cl.cam.ac.uk> wrote:> > One last thing you might want to try is to edit > linux-2.6.11/arch/xen/i386/kernel/pci-dma.c. There you will see that > there are four separate uses of memcpy(), each conditional on a test of > the form ''if (direction != DMA_xxx)''. You *might* have more success if > you remove the direction-conditional tests and always memcpy. > > Apart from that, this needs someone with hardware and expertise, and > who cares about both ivtv and xen, to wade in. :-)I''ll give that a shot tonight, thanks for the suggestion. I really appreciate your efforts and if that doesn''t help, I''ll see if anyone on the ivtv-devel list is into Xen too. I wish I had the kernel programming expertise myself. But at least the drivers work well in dom0. Stupid question, but is there any way to tell Xen to allocate a domU contiguously? Or is that just not the Xen way, so to speak? Dave -- David Muench - davemuench@gmail.com Jabber ID: dave@jabber.wasteland.org _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2005-Jul-11 13:20 UTC
[Xen-devel] Re: [ivtv-devel] Problems loading ivtv in Xen - DMA issues?
On 11 Jul 2005, at 14:08, David Muench wrote:> I''ll give that a shot tonight, thanks for the suggestion. I really > appreciate your efforts and if that doesn''t help, I''ll see if anyone > on the ivtv-devel list is into Xen too. I wish I had the kernel > programming expertise myself. But at least the drivers work well in > dom0. > > Stupid question, but is there any way to tell Xen to allocate a domU > contiguously? Or is that just not the Xen way, so to speak?It would be tricky. Even if we did support it, if the domU is connected to a network interface, or you use the balloon driver, then that will cause memory to start getting fragmented. But maybe it would be good enough -- this fragmenting problem will also occur in domain0 over time. I think you''ll have to stick to dom0 for now and hope someone on the ivtv list has some spare cycles to help you out. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
David Muench
2005-Jul-13 19:51 UTC
[Xen-devel] Re: [ivtv-devel] Problems loading ivtv in Xen - DMA issues?
On 7/11/05, Keir Fraser <Keir.Fraser@cl.cam.ac.uk> wrote:> > One last thing you might want to try is to edit > linux-2.6.11/arch/xen/i386/kernel/pci-dma.c. There you will see that > there are four separate uses of memcpy(), each conditional on a test of > the form ''if (direction != DMA_xxx)''. You *might* have more success if > you remove the direction-conditional tests and always memcpy. > > Apart from that, this needs someone with hardware and expertise, and > who cares about both ivtv and xen, to wade in. :-)Well, life got in the way and I didn''t try this fix until just now.. But it works! I''m going to get my mythtv setup moved into the domU so I can test it completely, but it seems to work perfectly. Thank you very, very much. Is removing the conditionals a permanent solution that can be commited, or is there anything else you''d like me to test? Dave -- David Muench - davemuench@gmail.com Jabber ID: dave@jabber.wasteland.org _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2005-Jul-13 22:06 UTC
[Xen-devel] Re: [ivtv-devel] Problems loading ivtv in Xen - DMA issues?
On 13 Jul 2005, at 20:51, David Muench wrote:> Well, life got in the way and I didn''t try this fix until just now.. > But it works! I''m going to get my mythtv setup moved into the domU so > I can test it completely, but it seems to work perfectly. > > Thank you very, very much. Is removing the conditionals a permanent > solution that can be commited, or is there anything else you''d like me > to test?I believe the code as checked in is strictly correct, but maybe we can relax things a little. What happens if you leave the direction tests in dma_map_single and dma_unmap_single, but remove them from dma_sync_single_for_cpu and dma_sync_single_for_device? If the above works, leave the direction tests in the new modified form; otherwise revert to the previous working state (ie. direction tests removed from all four functions). Either way, another thing that is interesting to try is the following: (1) At the start of dma_sync_single_for_cpu add: if (direction == DMA_TO_DEVICE) dma_sync_single_for_device(dev, dma_handle, size, direction); (2) At the start of dma_sync_single_for_device add: if (direction == DMA_FROM_DEVICE) dma_sync_single_for_cpu(dev, dma_handle, size, direction); This will make the sync functions behave as they do in the ARM architecture, which also uses bounce buffers but has at least been looked over by the core kernel DMA guys. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
David Muench
2005-Jul-15 12:43 UTC
[Xen-devel] Re: [ivtv-devel] Problems loading ivtv in Xen - DMA issues?
On 7/13/05, Keir Fraser <Keir.Fraser@cl.cam.ac.uk> wrote:> > I believe the code as checked in is strictly correct, but maybe we can > relax things a little. What happens if you leave the direction tests in > dma_map_single and dma_unmap_single, but remove them from > dma_sync_single_for_cpu and dma_sync_single_for_device?That works fine.> If the above works, leave the direction tests in the new modified form; > otherwise revert to the previous working state (ie. direction tests > removed from all four functions). Either way, another thing that is > interesting to try is the following: > (1) At the start of dma_sync_single_for_cpu add: > if (direction == DMA_TO_DEVICE) > dma_sync_single_for_device(dev, dma_handle, size, direction); > (2) At the start of dma_sync_single_for_device add: > if (direction == DMA_FROM_DEVICE) > dma_sync_single_for_cpu(dev, dma_handle, size, direction); > > This will make the sync functions behave as they do in the ARM > architecture, which also uses bounce buffers but has at least been > looked over by the core kernel DMA guys.That doesn''t work however: saa7115: decoder disable output saa7115: decoder enable output ivtv: DEC: REG_ENCSG1LEN wait failed ivtv: DMA Registers State: xfer: 0x00000002, state: 0x00000001 dec_addr: 0x00000000 enc_addr: 0x46760060 control: 0x00000003 ivtv: DMA DEC Buffers: 0x00000000:0x00000000:0x00000000 ivtv: DMA ENC Buffers: 0x00000000:0x00000000:0x00000000 0x00000000:0x00000000:0x00000000 0x00000000:0x00000000:0x00000000 0x00000000:0x00000000:0x00000000 0x00000000:0x00000000:0x00000000 0x00000000:0x00000000:0x00000000 0x00000000:0x00000000:0x00000000 0x00000000:0x00000000:0x00000000 ivtv: ENC: REG_DMAXFER wait failed ivtv: DMA Registers State: xfer: 0x00000002, state: 0x00000001 dec_addr: 0x00000000 enc_addr: 0x46760060 control: 0x00000003 ivtv: DMA DEC Buffers: 0x00000000:0x00000000:0x00000000 ivtv: DMA ENC Buffers: 0x00000000:0x00000000:0x00000000 0x00000000:0x00000000:0x00000000 0x00000000:0x00000000:0x00000000 0x00000000:0x00000000:0x00000000 0x00000000:0x00000000:0x00000000 0x00000000:0x00000000:0x00000000 0x00000000:0x00000000:0x00000000 0x00000000:0x00000000:0x00000000 ivtv: ENC: User stopped capture. ivtv: ENC: REG_DMASTATUS2 wait failed ivtv: DMA Registers State: xfer: 0x00000002, state: 0x00000001 dec_addr: 0x00000000 enc_addr: 0x46760060 control: 0x00000003 ivtv: DMA DEC Buffers: 0x00000000:0x00000000:0x00000000 ivtv: DMA ENC Buffers: 0x00000000:0x00000000:0x00000000 0x00000000:0x00000000:0x00000000 0x00000000:0x00000000:0x00000000 0x00000000:0x00000000:0x00000000 0x00000000:0x00000000:0x00000000 0x00000000:0x00000000:0x00000000 0x00000000:0x00000000:0x00000000 0x00000000:0x00000000:0x00000000 I''m going to take that out and run with the conditionals in the map functions only, and see if that holds up under longer testing. Dave -- David Muench - davemuench@gmail.com Jabber ID: dave@jabber.wasteland.org _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2005-Jul-15 12:50 UTC
[Xen-devel] Re: [ivtv-devel] Problems loading ivtv in Xen - DMA issues?
On 15 Jul 2005, at 13:43, David Muench wrote:>> This will make the sync functions behave as they do in the ARM >> architecture, which also uses bounce buffers but has at least been >> looked over by the core kernel DMA guys. > > That doesn''t work however:Well, I think there is probably a grey area over what accesses the kernel can make to memory that it currently has mapped by dma_map_single.... I think I will just remove the direction checks from the sync functions, leave them in the map/unmap functions, and hope that suffices to get most drivers working. Thanks for your testing efforts on this! -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
David Muench
2005-Jul-15 12:57 UTC
[Xen-devel] Re: [ivtv-devel] Problems loading ivtv in Xen - DMA issues?
On 7/15/05, Keir Fraser <Keir.Fraser@cl.cam.ac.uk> wrote:> > Well, I think there is probably a grey area over what accesses the > kernel can make to memory that it currently has mapped by > dma_map_single.... I think I will just remove the direction checks from > the sync functions, leave them in the map/unmap functions, and hope > that suffices to get most drivers working. > > Thanks for your testing efforts on this!No problem, thanks very much for all your help. It still amazes me that a Celeron in my basement can have 2 domains running web servers, 1 doing dns/dhcp/tftp type services, a mysql domain, a domain running Asterisk to control all my VOIP phones, and a mythtv backend domain that can simultaneously record 3 DVD quality mpeg-2 streams off of directv while playing back to (at least) two frontend machines in my house. All thanks to Xen. And I still have another 200 meg of unallocated ram, I wonder what I can do with that.. :) Dave -- David Muench - davemuench@gmail.com Jabber ID: dave@jabber.wasteland.org _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel