Eric Camachat
2011-Sep-08 00:47 UTC
[Xen-devel] Can I specify a physical memory region for a domU
Hi, I am porting our drivers to XEN''s PV domU (with PV PCI passthrouth), I have to allocate a memory block and tell the hardware to access it. But the hardware can address 32-bit only, so I want dedicate a region of memory that below 2GB for the domU only. How do that in XEN? Thanks, /Eric _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Tim Deegan
2011-Sep-08 08:50 UTC
Re: [Xen-devel] Can I specify a physical memory region for a domU
At 17:47 -0700 on 07 Sep (1315417666), Eric Camachat wrote:> Hi, > > I am porting our drivers to XEN''s PV domU (with PV PCI passthrouth), I > have to allocate a memory block and tell the hardware to access it. > But the hardware can address 32-bit only, so I want dedicate a region > of memory that below 2GB for the domU only. > How do that in XEN?Use the XENMEM_echange memory_op hypercall. You can specify a bit-width for the pages you want returned. Of course, on a full Xen host there may be no 32-bit-clean memory left. Xen tries to allocate the higher addresses first to avoid that, but there''s no guarantee. Tim. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Konrad Rzeszutek Wilk
2011-Sep-08 12:59 UTC
Re: [Xen-devel] Can I specify a physical memory region for a domU
On Wed, Sep 07, 2011 at 05:47:46PM -0700, Eric Camachat wrote:> Hi, > > I am porting our drivers to XEN''s PV domU (with PV PCI passthrouth), IUse the DMA API that Linux provides (I presume that is what you meant by PV DomU), and use the dma_alloc_coherent to set your regions. Also pass in ''iommu=soft'' on your Linux command line to enable the Xen SWIOTLB DMA system.> have to allocate a memory block and tell the hardware to access it. > But the hardware can address 32-bit only, so I want dedicate a region > of memory that below 2GB for the domU only.Uh, don''t you mean 4GB? - 32bit is up to 4GB.> How do that in XEN? > > Thanks, > /Eric > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Eric Camachat
2011-Sep-08 17:22 UTC
Re: [Xen-devel] Can I specify a physical memory region for a domU
On Thu, Sep 8, 2011 at 5:59 AM, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:> On Wed, Sep 07, 2011 at 05:47:46PM -0700, Eric Camachat wrote: >> Hi, >> >> I am porting our drivers to XEN''s PV domU (with PV PCI passthrouth), I > > Use the DMA API that Linux provides (I presume that is what you meant > by PV DomU), and use the dma_alloc_coherent to set your regions.That''s what I thought before. We use a shared DMA region for multiple hardware. Maybe I can dma_alloc_coherent for 1st and the others use the same region. I will try it.> > Also pass in ''iommu=soft'' on your Linux command line to enable the > Xen SWIOTLB DMA system. > >> have to allocate a memory block and tell the hardware to access it. >> But the hardware can address 32-bit only, so I want dedicate a region >> of memory that below 2GB for the domU only. > > Uh, don''t you mean 4GB? - 32bit is up to 4GB.The hardware uses 32-bit addressing, but the system will crash if I assigned above 2GB address to it. So, 4GB from hardware spec, 2GB from my test. I am looking into that.>> How do that in XEN? >> >> Thanks, >> /Eric >> >> _______________________________________________ >> Xen-devel mailing list >> Xen-devel@lists.xensource.com >> http://lists.xensource.com/xen-devel >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Konrad Rzeszutek Wilk
2011-Sep-08 18:12 UTC
Re: [Xen-devel] Can I specify a physical memory region for a domU
On Thu, Sep 08, 2011 at 10:22:01AM -0700, Eric Camachat wrote:> On Thu, Sep 8, 2011 at 5:59 AM, Konrad Rzeszutek Wilk > <konrad.wilk@oracle.com> wrote: > > On Wed, Sep 07, 2011 at 05:47:46PM -0700, Eric Camachat wrote: > >> Hi, > >> > >> I am porting our drivers to XEN''s PV domU (with PV PCI passthrouth), I > > > > Use the DMA API that Linux provides (I presume that is what you meant > > by PV DomU), and use the dma_alloc_coherent to set your regions. > > That''s what I thought before. We use a shared DMA region for multiple hardware. > Maybe I can dma_alloc_coherent for 1st and the others use the same region.You can definitly try it. Or use the dmapool API to get a shared pool of coherent memory.> I will try it. > > > > > Also pass in ''iommu=soft'' on your Linux command line to enable the > > Xen SWIOTLB DMA system. > > > >> have to allocate a memory block and tell the hardware to access it. > >> But the hardware can address 32-bit only, so I want dedicate a region > >> of memory that below 2GB for the domU only. > > > > Uh, don''t you mean 4GB? - 32bit is up to 4GB. > > The hardware uses 32-bit addressing, but the system will crash if I > assigned above 2GB address to it.Ha! so buggy hardware.. or you are not using the XEn-SWIOTLB but something else.> So, 4GB from hardware spec, 2GB from my test. I am looking into that.Make sure you set pci_set_consistent_dma_mask(dev, DMA_BIT_MASK(31)); on top of pci_set_dma_mask(dev, DMA_BIT_MASK(31)); in your driver. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Eric Camachat
2011-Sep-08 18:56 UTC
Re: [Xen-devel] Can I specify a physical memory region for a domU
On Thu, Sep 8, 2011 at 11:12 AM, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:> On Thu, Sep 08, 2011 at 10:22:01AM -0700, Eric Camachat wrote: >> On Thu, Sep 8, 2011 at 5:59 AM, Konrad Rzeszutek Wilk >> <konrad.wilk@oracle.com> wrote: >> > On Wed, Sep 07, 2011 at 05:47:46PM -0700, Eric Camachat wrote: >> >> Hi, >> >> >> >> I am porting our drivers to XEN''s PV domU (with PV PCI passthrouth), I >> > >> > Use the DMA API that Linux provides (I presume that is what you meant >> > by PV DomU), and use the dma_alloc_coherent to set your regions. >> >> That''s what I thought before. We use a shared DMA region for multiple hardware. >> Maybe I can dma_alloc_coherent for 1st and the others use the same region. > > You can definitly try it. Or use the dmapool API to get a shared > pool of coherent memory. > >> I will try it. >> >> > >> > Also pass in ''iommu=soft'' on your Linux command line to enable the >> > Xen SWIOTLB DMA system. >> > >> >> have to allocate a memory block and tell the hardware to access it. >> >> But the hardware can address 32-bit only, so I want dedicate a region >> >> of memory that below 2GB for the domU only. >> > >> > Uh, don''t you mean 4GB? - 32bit is up to 4GB. >> >> The hardware uses 32-bit addressing, but the system will crash if I >> assigned above 2GB address to it. > > Ha! so buggy hardware.. or you are not using the XEn-SWIOTLB but something > else. >> So, 4GB from hardware spec, 2GB from my test. I am looking into that. > > Make sure you set > pci_set_consistent_dma_mask(dev, DMA_BIT_MASK(31)); > > on top of pci_set_dma_mask(dev, DMA_BIT_MASK(31)); > in your driver. >A lot of work to port the driver to PV domU, hope it works. Thanks for your help! _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Konrad Rzeszutek Wilk
2011-Sep-08 20:00 UTC
Re: [Xen-devel] Can I specify a physical memory region for a domU
> > Make sure you set > > pci_set_consistent_dma_mask(dev, DMA_BIT_MASK(31)); > > > > on top of pci_set_dma_mask(dev, DMA_BIT_MASK(31)); > > in your driver. > > > A lot of work to port the driver to PV domU, hope it works.Hm? That is the normal way you would write drivers in the Linux kernel. You use the DMA API in it to deal with the PCI devices. Is the PV domU a Linux kernel or something else? _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Eric Camachat
2011-Sep-09 00:15 UTC
Re: [Xen-devel] Can I specify a physical memory region for a domU
On Thu, Sep 8, 2011 at 1:00 PM, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:>> > Make sure you set >> > pci_set_consistent_dma_mask(dev, DMA_BIT_MASK(31)); >> > >> > on top of pci_set_dma_mask(dev, DMA_BIT_MASK(31)); >> > in your driver. >> > >> A lot of work to port the driver to PV domU, hope it works. > > Hm? That is the normal way you would write drivers in the Linux kernel. > You use the DMA API in it to deal with the PCI devices. > > Is the PV domU a Linux kernel or something else? >Yes, the PV domU is linux kernel. I tested pci_alloc_consistent() verified on baremetal it worked with 8GB SDRAM. But while I ran it in XEN, pci_alloc_consistent() cannot allocate memory successfully for 4MB although pci_set_dma_mask(), pci_set_consistent_dma_mask() and dma_set_mask() all succeeded. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Konrad Rzeszutek Wilk
2011-Sep-09 01:05 UTC
Re: [Xen-devel] Can I specify a physical memory region for a domU
On Thu, Sep 08, 2011 at 05:15:06PM -0700, Eric Camachat wrote:> On Thu, Sep 8, 2011 at 1:00 PM, Konrad Rzeszutek Wilk > <konrad.wilk@oracle.com> wrote: > >> > Make sure you set > >> > pci_set_consistent_dma_mask(dev, DMA_BIT_MASK(31)); > >> > > >> > on top of pci_set_dma_mask(dev, DMA_BIT_MASK(31)); > >> > in your driver. > >> > > >> A lot of work to port the driver to PV domU, hope it works. > > > > Hm? That is the normal way you would write drivers in the Linux kernel. > > You use the DMA API in it to deal with the PCI devices. > > > > Is the PV domU a Linux kernel or something else? > > > > Yes, the PV domU is linux kernel. > > I tested pci_alloc_consistent() verified on baremetal it worked with 8GB SDRAM. > But while I ran it in XEN, pci_alloc_consistent() cannot allocate > memory successfully for 4MB although pci_set_dma_mask(),And what is the error?> pci_set_consistent_dma_mask() and dma_set_mask() all succeeded.And what kernel did you use?> > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Eric Camachat
2011-Sep-09 16:10 UTC
Re: [Xen-devel] Can I specify a physical memory region for a domU
On Thu, Sep 8, 2011 at 6:05 PM, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:> On Thu, Sep 08, 2011 at 05:15:06PM -0700, Eric Camachat wrote: >> On Thu, Sep 8, 2011 at 1:00 PM, Konrad Rzeszutek Wilk >> <konrad.wilk@oracle.com> wrote: >> >> > Make sure you set >> >> > pci_set_consistent_dma_mask(dev, DMA_BIT_MASK(31)); >> >> > >> >> > on top of pci_set_dma_mask(dev, DMA_BIT_MASK(31)); >> >> > in your driver. >> >> > >> >> A lot of work to port the driver to PV domU, hope it works. >> > >> > Hm? That is the normal way you would write drivers in the Linux kernel. >> > You use the DMA API in it to deal with the PCI devices. >> > >> > Is the PV domU a Linux kernel or something else? >> > >> >> Yes, the PV domU is linux kernel. >> >> I tested pci_alloc_consistent() verified on baremetal it worked with 8GB SDRAM. >> But while I ran it in XEN, pci_alloc_consistent() cannot allocate >> memory successfully for 4MB although pci_set_dma_mask(), > > And what is the error?I cannot see any error, it just returned NULL.> >> pci_set_consistent_dma_mask() and dma_set_mask() all succeeded. > > And what kernel did you use?I am using linux-2.6.32.24.>> >> _______________________________________________ >> Xen-devel mailing list >> Xen-devel@lists.xensource.com >> http://lists.xensource.com/xen-devel >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Eric Camachat
2011-Sep-09 16:48 UTC
Re: [Xen-devel] Can I specify a physical memory region for a domU
On Fri, Sep 9, 2011 at 9:10 AM, Eric Camachat <eric.camachat@gmail.com> wrote:> On Thu, Sep 8, 2011 at 6:05 PM, Konrad Rzeszutek Wilk > <konrad.wilk@oracle.com> wrote: >> On Thu, Sep 08, 2011 at 05:15:06PM -0700, Eric Camachat wrote: >>> On Thu, Sep 8, 2011 at 1:00 PM, Konrad Rzeszutek Wilk >>> <konrad.wilk@oracle.com> wrote: >>> >> > Make sure you set >>> >> > pci_set_consistent_dma_mask(dev, DMA_BIT_MASK(31)); >>> >> > >>> >> > on top of pci_set_dma_mask(dev, DMA_BIT_MASK(31)); >>> >> > in your driver. >>> >> > >>> >> A lot of work to port the driver to PV domU, hope it works. >>> > >>> > Hm? That is the normal way you would write drivers in the Linux kernel. >>> > You use the DMA API in it to deal with the PCI devices. >>> > >>> > Is the PV domU a Linux kernel or something else? >>> > >>> >>> Yes, the PV domU is linux kernel. >>> >>> I tested pci_alloc_consistent() verified on baremetal it worked with 8GB SDRAM. >>> But while I ran it in XEN, pci_alloc_consistent() cannot allocate >>> memory successfully for 4MB although pci_set_dma_mask(), >> >> And what is the error? > > I cannot see any error, it just returned NULL.BTW, I will succeed if I allocate 2MB only.> >> >>> pci_set_consistent_dma_mask() and dma_set_mask() all succeeded. >> >> And what kernel did you use? > I am using linux-2.6.32.24. > >>> >>> _______________________________________________ >>> Xen-devel mailing list >>> Xen-devel@lists.xensource.com >>> http://lists.xensource.com/xen-devel >> >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Konrad Rzeszutek Wilk
2011-Sep-09 18:06 UTC
Re: [Xen-devel] Can I specify a physical memory region for a domU
On Fri, Sep 09, 2011 at 09:48:03AM -0700, Eric Camachat wrote:> On Fri, Sep 9, 2011 at 9:10 AM, Eric Camachat <eric.camachat@gmail.com> wrote: > > On Thu, Sep 8, 2011 at 6:05 PM, Konrad Rzeszutek Wilk > > <konrad.wilk@oracle.com> wrote: > >> On Thu, Sep 08, 2011 at 05:15:06PM -0700, Eric Camachat wrote: > >>> On Thu, Sep 8, 2011 at 1:00 PM, Konrad Rzeszutek Wilk > >>> <konrad.wilk@oracle.com> wrote: > >>> >> > Make sure you set > >>> >> > pci_set_consistent_dma_mask(dev, DMA_BIT_MASK(31)); > >>> >> > > >>> >> > on top of pci_set_dma_mask(dev, DMA_BIT_MASK(31)); > >>> >> > in your driver. > >>> >> > > >>> >> A lot of work to port the driver to PV domU, hope it works. > >>> > > >>> > Hm? That is the normal way you would write drivers in the Linux kernel. > >>> > You use the DMA API in it to deal with the PCI devices. > >>> > > >>> > Is the PV domU a Linux kernel or something else? > >>> > > >>> > >>> Yes, the PV domU is linux kernel. > >>> > >>> I tested pci_alloc_consistent() verified on baremetal it worked with 8GB SDRAM. > >>> But while I ran it in XEN, pci_alloc_consistent() cannot allocate > >>> memory successfully for 4MB although pci_set_dma_mask(), > >> > >> And what is the error? > > > > I cannot see any error, it just returned NULL. > > BTW, I will succeed if I allocate 2MB only.Ok, that is unsurprising. I don''t know if you can do it any better on baremetal. But I am failing to understand why you need such large swaths of contingous memory. You can''t do scatter gather? Or scatter gather on 2MB chunks? _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Eric Camachat
2011-Sep-09 19:19 UTC
Re: [Xen-devel] Can I specify a physical memory region for a domU
On Fri, Sep 9, 2011 at 11:06 AM, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:> On Fri, Sep 09, 2011 at 09:48:03AM -0700, Eric Camachat wrote: >> On Fri, Sep 9, 2011 at 9:10 AM, Eric Camachat <eric.camachat@gmail.com> wrote: >> > On Thu, Sep 8, 2011 at 6:05 PM, Konrad Rzeszutek Wilk >> > <konrad.wilk@oracle.com> wrote: >> >> On Thu, Sep 08, 2011 at 05:15:06PM -0700, Eric Camachat wrote: >> >>> On Thu, Sep 8, 2011 at 1:00 PM, Konrad Rzeszutek Wilk >> >>> <konrad.wilk@oracle.com> wrote: >> >>> >> > Make sure you set >> >>> >> > pci_set_consistent_dma_mask(dev, DMA_BIT_MASK(31)); >> >>> >> > >> >>> >> > on top of pci_set_dma_mask(dev, DMA_BIT_MASK(31)); >> >>> >> > in your driver. >> >>> >> > >> >>> >> A lot of work to port the driver to PV domU, hope it works. >> >>> > >> >>> > Hm? That is the normal way you would write drivers in the Linux kernel. >> >>> > You use the DMA API in it to deal with the PCI devices. >> >>> > >> >>> > Is the PV domU a Linux kernel or something else? >> >>> > >> >>> >> >>> Yes, the PV domU is linux kernel. >> >>> >> >>> I tested pci_alloc_consistent() verified on baremetal it worked with 8GB SDRAM. >> >>> But while I ran it in XEN, pci_alloc_consistent() cannot allocate >> >>> memory successfully for 4MB although pci_set_dma_mask(), >> >> >> >> And what is the error? >> > >> > I cannot see any error, it just returned NULL. >> >> BTW, I will succeed if I allocate 2MB only. > > Ok, that is unsurprising. I don''t know if you can do it any better on baremetal. > But I am failing to understand why you need such large swaths of contingous > memory. You can''t do scatter gather? Or scatter gather on 2MB chunks? >I changed MAX_CONTIG_ORDER, and MAX_ORDER to 13 to fix this problem, finally. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel