Dear Xen developers: I maintain a component in Linux kernel, called usbmon. It provides for snooping of USB traffic, analoguously to tcpdump. Due to pequliarities of USB API in Linux, usbmon has to take a DMA address (coming from dma_alloc_coherent() typically), find the corresponding PFN and call pkmap()/memcpy()/pkunmap(). My question is: how to find PFN under Xen if DMA address is known, on x86_64 and i386? I see some functions which may be useful (e.g. mfn_to_pfn()), but what I need the most is some tribal knowledge about what is the proper way to do it. This includes things like working in both dom0 and domU, not faulting in machine_to_phys_mapping[] while in hot path, and so on. Thank you, -- Pete _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Tue, Jan 09, 2007 at 05:17:10PM -0800, Pete Zaitcev wrote:> Dear Xen developers: > > I maintain a component in Linux kernel, called usbmon. It provides for > snooping of USB traffic, analoguously to tcpdump. Due to pequliarities > of USB API in Linux, usbmon has to take a DMA address (coming from > dma_alloc_coherent() typically), find the corresponding PFN and call > pkmap()/memcpy()/pkunmap().Hi Pete, Regardless of Xen, how is this supposed to work in the presence of IOMMUs? Cheers, Muli _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Wed, 10 Jan 2007 05:25:08 +0200, Muli Ben-Yehuda <muli@il.ibm.com> wrote:> On Tue, Jan 09, 2007 at 05:17:10PM -0800, Pete Zaitcev wrote:> > I maintain a component in Linux kernel, called usbmon. It provides for > > snooping of USB traffic, analoguously to tcpdump. Due to pequliarities > > of USB API in Linux, usbmon has to take a DMA address (coming from > > dma_alloc_coherent() typically), find the corresponding PFN and call > > pkmap()/memcpy()/pkunmap().> Regardless of Xen, how is this supposed to work in the presence of > IOMMUs?It cannot, generically speaking. So, no usbmon on Calgari. But for a specific IOMMU I can place necessary hooks. I am looking at SWIOTLB currently. And don''t forget that it only works on cache-coherent architectures, so no SPARC support ever. -- Pete _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 10/1/07 01:17, "Pete Zaitcev" <zaitcev@redhat.com> wrote:> I maintain a component in Linux kernel, called usbmon. It provides for > snooping of USB traffic, analoguously to tcpdump. Due to pequliarities > of USB API in Linux, usbmon has to take a DMA address (coming from > dma_alloc_coherent() typically), find the corresponding PFN and call > pkmap()/memcpy()/pkunmap(). > > My question is: how to find PFN under Xen if DMA address is known, > on x86_64 and i386?Use mfn_to_local_pfn() which is safe to call from any context. This will return a !pfn_valid() value if the given MFN actually belongs to a foreign domain (e.g., we are issuing a DMA on behalf of another guest via netback or blkback) or if the MFN is not RAM. Otherwise it will return the local PFN. Of course you may then want to do further translation if that happens to be in the swiotlb aperture. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Wed, 10 Jan 2007 11:04:01 +0000, Keir Fraser <keir@xensource.com> wrote:> On 10/1/07 01:17, "Pete Zaitcev" <zaitcev@redhat.com> wrote:> > My question is: how to find PFN under Xen if DMA address is known, > > on x86_64 and i386? > > Use mfn_to_local_pfn() which is safe to call from any context. [...] > Of course you may then want to do further translation if that happens to be > in the swiotlb aperture.Keir, thanks a lot, that worked perfectly. The SWIOTLB is something I can''t understand still. The code sure looks like returning physical addresses to be used as DMA handles, just like any usual routines do: void * swiotlb_alloc_coherent(struct device *hwdev, size_t size, dma_addr_t *dma_handle, gfp_t flags) { .... ret = (void *)__get_free_pages(flags, order); dev_addr = virt_to_phys(ret); *dma_handle = dev_addr; And the code in pci-dma.c does: void * dma_alloc_coherent(struct device *dev, size_t size, dma_addr_t *dma_handle, gfp_t gfp) memory = dma_alloc_pages(dev, gfp, get_order(size)); *dma_handle = virt_to_bus(memory); The virt_to_bus and virt_to_phys are the same on x86_64. So this is a kind of a mystery, but I''ll think about it and maybe ask at linux-kernel. Thanks, -- Pete diff -urp -X dontdiff linux-2.6.17-1.2647.fc6/drivers/usb/mon/mon_dma.c linux-2.6.17-1.2647.fc6.z1/drivers/usb/mon/mon_dma.c --- linux-2.6.17-1.2647.fc6/drivers/usb/mon/mon_dma.c 2006-09-24 19:10:42.000000000 -0700 +++ linux-2.6.17-1.2647.fc6.z1/drivers/usb/mon/mon_dma.c 2007-01-10 17:30:33.000000000 -0800 @@ -19,21 +19,25 @@ #if defined(__i386__) || defined(__x86_64__) /* CONFIG_ARCH_I386 doesn''t exit */ #define MON_HAS_UNMAP 1 -#define phys_to_page(phys) pfn_to_page((phys) >> PAGE_SHIFT) - char mon_dmapeek(unsigned char *dst, dma_addr_t dma_addr, int len) { struct page *pg; unsigned long flags; unsigned char *map; unsigned char *ptr; + unsigned long mfn; /* * On i386, a DMA handle is the "physical" address of a page. * In other words, the bus address is equal to physical address. - * There is no IOMMU. + * But sometimes IOMMUs are present (Calgary, AMD''s x86_64, SWIOTLB). + * Also, Xen introduces "machine" address, which is used for DMA. + * We handle Xen, but not IOMMUs yet. */ - pg = phys_to_page(dma_addr); + mfn = mfn_to_local_pfn(dma_addr >> PAGE_SHIFT); + if (!pfn_valid(mfn)) + return ''X''; + pg = pfn_to_page(mfn); /* * We are called from hardware IRQs in case of callbacks. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 11/1/07 3:21 am, "Pete Zaitcev" <zaitcev@redhat.com> wrote:> The SWIOTLB is something I can''t understand still. The code sure looks > like returning physical addresses to be used as DMA handles, just like > any usual routines do:Yes, but suitably low physical addresses which are kept in sync with the original high addresses by occasional memcpy(). The question is: if you have a ''DMA address'' that belongs to the swiotlb, do you mind if the MFN->PFN translation yields you an address inside the swiotlb aperture rather than the original driver address? Since both addresses are valid local memory, and both should contain a reasonably up-to-date copy of the data you are wanting to sniff, I guess maybe you don''t actually care. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel