Marek Marczykowski
2011-Sep-06 17:54 UTC
[Xen-devel] Memory fragmentation and PCI passthrough
Hello, I''ve hit known problem with dynamic memory management - memory fragmentation... This dynamic memory management basically does xl mem-set to balance memory. After some time of running system, xen memory is so fragmented that it is impossible to start new VM with PCI device. Sometimes it crashes during boot (no 64MB contiguous memory for SWIOTLB), or later - eg. iwlagn cannot allocate memory for loading firmware (few allocs, each bellow 100k). DomU kernel cmdline: console=hvc0 iommu=soft earlyprintk=xen There is two cases (I think): 1. With IOMMU 2. Without IOMMU I''ve tried only the second one. Is there any known solution for this problem? Some ideas: 1. With IOMMU pass iommu=pv to Xen. AFAIU domU will not need iommu=soft parameter then, right? Will it work then with fragmented memory? 2. Force somehow on xen/libxl to allocate memory (for domU) in chunks of, say 4MB, to not fragment it so badly. Is it doable? In tmem documentation is also described some workaround for this: reserve some memory region for allocations with 0<order<=9. But SWIOTLB tries to allocate 64MB, which much bigger than 2MB... Is it really needed to allocate such big region of contiguous memory in one piece? -- Pozdrawiam / Best Regards, Marek Marczykowski Invisible Things Lab _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jan Beulich
2011-Sep-07 10:01 UTC
Re: [Xen-devel] Memory fragmentation and PCI passthrough
>>> On 06.09.11 at 19:54, Marek Marczykowski <marmarek@invisiblethingslab.com> wrote: > In tmem documentation is also described some workaround for this: > reserve some memory region for allocations with 0<order<=9. But SWIOTLB > tries to allocate 64MB, which much bigger than 2MB... Is it really > needed to allocate such big region of contiguous memory in one piece?For one, you can tweak the size (you shouldn''t normally need 64Mb in a DomU, and even in Dom0 that''s mostly needed for certain SCSI devices iirc). Second, the allocation isn''t done in a single 64M contiguous chunk (from hypervisor perspective) - xen_create_contiguous_region() is being called on much smaller pieces (IO_TLB_SEGSIZE << IO_TLB_SHIFT, i.e. 128 << 11, yielding 256k chunks). Jan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Konrad Rzeszutek Wilk
2011-Sep-07 17:36 UTC
Re: [Xen-devel] Memory fragmentation and PCI passthrough
On Tue, Sep 06, 2011 at 07:54:27PM +0200, Marek Marczykowski wrote:> Hello, > > I''ve hit known problem with dynamic memory management - memory > fragmentation... This dynamic memory management basically does xl > mem-set to balance memory. > > After some time of running system, xen memory is so fragmented that it > is impossible to start new VM with PCI device. Sometimes it crashes > during boot (no 64MB contiguous memory for SWIOTLB), or later - eg. > iwlagn cannot allocate memory for loading firmware (few allocs, each > bellow 100k). > > DomU kernel cmdline: console=hvc0 iommu=soft earlyprintk=xen > > There is two cases (I think): > 1. With IOMMU > 2. Without IOMMU > I''ve tried only the second one. > > Is there any known solution for this problem? > Some ideas: > 1. With IOMMU pass iommu=pv to Xen. AFAIU domU will not need iommu=soft > parameter then, right? Will it work then with fragmented memory?It will still need it. Otherwise the Xen SWIOTLB won''t be used. But you can limit the amount of memory for the SWIOTLB, say by doing ''swiotlb=2048''> > 2. Force somehow on xen/libxl to allocate memory (for domU) in chunks > of, say 4MB, to not fragment it so badly. Is it doable? > > In tmem documentation is also described some workaround for this: > reserve some memory region for allocations with 0<order<=9. But SWIOTLB > tries to allocate 64MB, which much bigger than 2MB... Is it really > needed to allocate such big region of contiguous memory in one piece?Nope. It only uses that pool for 32-bit devices too - so there is not really a big need for it.> > -- > Pozdrawiam / Best Regards, > Marek Marczykowski > Invisible Things Lab >> _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Pasi Kärkkäinen
2011-Sep-07 21:00 UTC
Re: [Xen-devel] Memory fragmentation and PCI passthrough
On Wed, Sep 07, 2011 at 01:36:22PM -0400, Konrad Rzeszutek Wilk wrote:> On Tue, Sep 06, 2011 at 07:54:27PM +0200, Marek Marczykowski wrote: > > Hello, > > > > I''ve hit known problem with dynamic memory management - memory > > fragmentation... This dynamic memory management basically does xl > > mem-set to balance memory. > > > > After some time of running system, xen memory is so fragmented that it > > is impossible to start new VM with PCI device. Sometimes it crashes > > during boot (no 64MB contiguous memory for SWIOTLB), or later - eg. > > iwlagn cannot allocate memory for loading firmware (few allocs, each > > bellow 100k). > > > > DomU kernel cmdline: console=hvc0 iommu=soft earlyprintk=xen > > > > There is two cases (I think): > > 1. With IOMMU > > 2. Without IOMMU > > I''ve tried only the second one. > > > > Is there any known solution for this problem? > > Some ideas: > > 1. With IOMMU pass iommu=pv to Xen. AFAIU domU will not need iommu=soft > > parameter then, right? Will it work then with fragmented memory? > > It will still need it. Otherwise the Xen SWIOTLB won''t be used. >Btw I think Xen hypervisor "iommu=pv" is a deprecated option.. "iommu=soft" for the domU kernel is a valid option though :) -- Pasi> But you can limit the amount of memory for the SWIOTLB, say by > doing ''swiotlb=2048'' > > > > > 2. Force somehow on xen/libxl to allocate memory (for domU) in chunks > > of, say 4MB, to not fragment it so badly. Is it doable? > > > > In tmem documentation is also described some workaround for this: > > reserve some memory region for allocations with 0<order<=9. But SWIOTLB > > tries to allocate 64MB, which much bigger than 2MB... Is it really > > needed to allocate such big region of contiguous memory in one piece? > > Nope. It only uses that pool for 32-bit devices too - so there is not > really a big need for it. > > > > > > -- > > Pozdrawiam / Best Regards, > > Marek Marczykowski > > Invisible Things Lab > > > > > > > _______________________________________________ > > Xen-devel mailing list > > Xen-devel@lists.xensource.com > > http://lists.xensource.com/xen-devel > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel