Charles Arnold
2010-Dec-16 19:23 UTC
[Xen-devel] Xen 4.0.1 "xc_map_foreign_batch: mmap failed: Cannot allocate memory"
Testing has demonstrated a fairly simple way to reproduce the problem
using Xen 4.0.1.
On a large memory server (mine is 16 Gigs), set up the following scenario.
1. Boot the server with dom0_mem=2048M thereby limiting the host to 2 Gig.
Also set (enable-dom0-ballooning no) in xend-config.sxp to disable
ballooning.
2. Create two windows guests. My test case included the following.
Guest 1: win2k3 with 512 Meg and sufficient disk space to hold a 10 Gig file.
Create a shared folder which can be mapped by the win2k8r2 guest.
Create or copy a 10 Gig file to this shared folder.
Guest 2: win2k8r2 with 4 Gig and sufficient disk space to hold a 10 Gig file.
Map the shared folder on the win2k3 guest.
3. Using robocopy on the win2k8r2 guest, copy the 10 Gig file to a local folder.
With the above scenario, the copy gets to about 23.7% when the win2k8r2 guest
is terminated with the mmap error above.
Observations: qemu-dm steadily mmap''s all the amount of memory which
the guest
has been given while doing the robocopy. So if the guest was given 2 Gig, it
eventually mmap''s 2 Gig while the copying continues. When this 2 Gig
limit is
reached, it continues to copy without problem until all the file is copied.
This has been observed with the win2k8r2 guest using 2, 2.5 and 3 Gig. However,
at 3.25 Gig, the mmap call eventually fails for the specific case where the
dom0 memory is set to 2 Gig and ballooning is off. The size of the file being
copied in all tests is 10 Gig.
The bug is that qemu-dm seems to make the assumption that it can mmap from
dom0 all the memory with which the guest has been defined instead of the memory
that is actually available on the host.
- Charles
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Keir Fraser
2010-Dec-16 20:33 UTC
Re: [Xen-devel] Xen 4.0.1 "xc_map_foreign_batch: mmap failed: Cannot allocate memory"
On 16/12/2010 19:23, "Charles Arnold" <carnold@novell.com> wrote:> The bug is that qemu-dm seems to make the assumption that it can mmap from > dom0 all the memory with which the guest has been defined instead of the > memory > that is actually available on the host.32-bit dom0? Hm, I thought the qemu mapcache was supposed to limit the total amount of guest memory mapped at one time, for a 32-bit qemu. For 64-bit qemu I wouldn''t expect to find a limit as low as 3.25G. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Charles Arnold
2010-Dec-16 20:44 UTC
Re: [Xen-devel] Xen 4.0.1 "xc_map_foreign_batch: mmap failed: Cannot allocate memory"
>> On 12/16/2010 at 01:33 PM, in message <C9302813.2966F%keir@xen.org>, KeirFraser <keir@xen.org> wrote:> On 16/12/2010 19:23, "Charles Arnold" <carnold@novell.com> wrote: > >> The bug is that qemu-dm seems to make the assumption that it can mmap from >> dom0 all the memory with which the guest has been defined instead of the >> memory >> that is actually available on the host. > > 32-bit dom0? Hm, I thought the qemu mapcache was supposed to limit the total > amount of guest memory mapped at one time, for a 32-bit qemu. For 64-bit > qemu I wouldn''t expect to find a limit as low as 3.25G.Sorry, I should have specified that it is a 64 bit dom0 / hypervisor. - Charles _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2010-Dec-16 20:54 UTC
Re: [Xen-devel] Xen 4.0.1 "xc_map_foreign_batch: mmap failed: Cannot allocate memory"
On 16/12/2010 20:44, "Charles Arnold" <carnold@novell.com> wrote:>>> On 12/16/2010 at 01:33 PM, in message <C9302813.2966F%keir@xen.org>, Keir > Fraser <keir@xen.org> wrote: >> On 16/12/2010 19:23, "Charles Arnold" <carnold@novell.com> wrote: >> >>> The bug is that qemu-dm seems to make the assumption that it can mmap from >>> dom0 all the memory with which the guest has been defined instead of the >>> memory >>> that is actually available on the host. >> >> 32-bit dom0? Hm, I thought the qemu mapcache was supposed to limit the total >> amount of guest memory mapped at one time, for a 32-bit qemu. For 64-bit >> qemu I wouldn''t expect to find a limit as low as 3.25G. > > Sorry, I should have specified that it is a 64 bit dom0 / hypervisor.Okay, well I''m not sure what limit qemu-dm is hitting then. Mapping 3.25G of guest memory will only require a few megabytes of pagetables for the qemu process in dom0. Perhaps there is a ulimit or something set on the qemu process? If we can work out and detect this limit, perhaps 64-bit qemu-dm could have a mapping cache similar to 32-bit qemu-dm, limited to some fraction of the detected mapping limit. And/or, on mapping failure, we could reclaim resources by simply zapping the existing cached mappings. Seems there''s a few options. I don''t really maintain qemu-dm myself -- you might get some help from Ian Jackson, Stefano, or Anthony Perard if you need more advice. -- Keir> - Charles > > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jan Beulich
2010-Dec-17 09:22 UTC
Re: [Xen-devel] Xen 4.0.1 "xc_map_foreign_batch: mmap failed: Cannot allocate memory"
>>> On 16.12.10 at 21:54, Keir Fraser <keir@xen.org> wrote: > On 16/12/2010 20:44, "Charles Arnold" <carnold@novell.com> wrote: > >>>> On 12/16/2010 at 01:33 PM, in message <C9302813.2966F%keir@xen.org>, Keir >> Fraser <keir@xen.org> wrote: >>> On 16/12/2010 19:23, "Charles Arnold" <carnold@novell.com> wrote: >>> >>>> The bug is that qemu-dm seems to make the assumption that it can mmap from >>>> dom0 all the memory with which the guest has been defined instead of the >>>> memory >>>> that is actually available on the host. >>> >>> 32-bit dom0? Hm, I thought the qemu mapcache was supposed to limit the total >>> amount of guest memory mapped at one time, for a 32-bit qemu. For 64-bit >>> qemu I wouldn''t expect to find a limit as low as 3.25G. >> >> Sorry, I should have specified that it is a 64 bit dom0 / hypervisor. > > Okay, well I''m not sure what limit qemu-dm is hitting then. Mapping 3.25G of > guest memory will only require a few megabytes of pagetables for the qemu > process in dom0. Perhaps there is a ulimit or something set on the qemu > process?I don''t think a ulimit plays in here - if Dom0 is given more memory, qemu-dm won''t fail. The question really is why qemu-dm actually uses unbounded mmap()-s in the first place. Even if the memory went only into page tables (which I doubt), this is a scalability problem. Of course, e.g. an address space ulimit put on qemu-dm should also not cause any failure - clearly there''s a lack of error handling here.> If we can work out and detect this limit, perhaps 64-bit qemu-dm could have > a mapping cache similar to 32-bit qemu-dm, limited to some fraction of the > detected mapping limit. And/or, on mapping failure, we could reclaim > resources by simply zapping the existing cached mappings. Seems there''s a > few options. I don''t really maintain qemu-dm myself -- you might get some > help from Ian Jackson, Stefano, or Anthony Perard if you need more advice.Looking forward to their comments. Jan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2010-Dec-17 10:06 UTC
Re: [Xen-devel] Xen 4.0.1 "xc_map_foreign_batch: mmap failed: Cannot allocate memory"
On 17/12/2010 09:22, "Jan Beulich" <JBeulich@novell.com> wrote:>> If we can work out and detect this limit, perhaps 64-bit qemu-dm could have >> a mapping cache similar to 32-bit qemu-dm, limited to some fraction of the >> detected mapping limit. And/or, on mapping failure, we could reclaim >> resources by simply zapping the existing cached mappings. Seems there''s a >> few options. I don''t really maintain qemu-dm myself -- you might get some >> help from Ian Jackson, Stefano, or Anthony Perard if you need more advice. > > Looking forward to their comments.One issue with a failure path added to just the ''mapping cache'' is that, if we have hit some hard mapping or VM limit for this process, various other syscalls in other qemu subsystems may also start to fail. If we let ourselves get to this point, it seems to be that handling it robustly might be difficult. OTOH, it shouldn''t be hard to do a lot better than we are currently. :-) -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Stefano Stabellini
2011-Jan-05 14:37 UTC
Re: [Xen-devel] Xen 4.0.1 "xc_map_foreign_batch: mmap failed: Cannot allocate memory"
On Thu, 16 Dec 2010, Keir Fraser wrote:> On 16/12/2010 20:44, "Charles Arnold" <carnold@novell.com> wrote: > > >>> On 12/16/2010 at 01:33 PM, in message <C9302813.2966F%keir@xen.org>, Keir > > Fraser <keir@xen.org> wrote: > >> On 16/12/2010 19:23, "Charles Arnold" <carnold@novell.com> wrote: > >> > >>> The bug is that qemu-dm seems to make the assumption that it can mmap from > >>> dom0 all the memory with which the guest has been defined instead of the > >>> memory > >>> that is actually available on the host. > >> > >> 32-bit dom0? Hm, I thought the qemu mapcache was supposed to limit the total > >> amount of guest memory mapped at one time, for a 32-bit qemu. For 64-bit > >> qemu I wouldn''t expect to find a limit as low as 3.25G. > > > > Sorry, I should have specified that it is a 64 bit dom0 / hypervisor. > > Okay, well I''m not sure what limit qemu-dm is hitting then. Mapping 3.25G of > guest memory will only require a few megabytes of pagetables for the qemu > process in dom0. Perhaps there is a ulimit or something set on the qemu > process? > > If we can work out and detect this limit, perhaps 64-bit qemu-dm could have > a mapping cache similar to 32-bit qemu-dm, limited to some fraction of the > detected mapping limit. And/or, on mapping failure, we could reclaim > resources by simply zapping the existing cached mappings. Seems there''s a > few options. I don''t really maintain qemu-dm myself -- you might get some > help from Ian Jackson, Stefano, or Anthony Perard if you need more advice.The mapcache size limit should be 64GB on a 64bit qemu-dm. Any interesting error messages in the qemu logs? _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jan Beulich
2011-Jan-05 15:30 UTC
Re: [Xen-devel] Xen 4.0.1 "xc_map_foreign_batch: mmap failed: Cannot allocate memory"
>>> On 05.01.11 at 15:37, Stefano Stabellini <stefano.stabellini@eu.citrix.com>wrote:> On Thu, 16 Dec 2010, Keir Fraser wrote: >> On 16/12/2010 20:44, "Charles Arnold" <carnold@novell.com> wrote: >> >> >>> On 12/16/2010 at 01:33 PM, in message <C9302813.2966F%keir@xen.org>, Keir >> > Fraser <keir@xen.org> wrote: >> >> On 16/12/2010 19:23, "Charles Arnold" <carnold@novell.com> wrote: >> >> >> >>> The bug is that qemu-dm seems to make the assumption that it can mmap from >> >>> dom0 all the memory with which the guest has been defined instead of the >> >>> memory >> >>> that is actually available on the host. >> >> >> >> 32-bit dom0? Hm, I thought the qemu mapcache was supposed to limit the total >> >> amount of guest memory mapped at one time, for a 32-bit qemu. For 64-bit >> >> qemu I wouldn''t expect to find a limit as low as 3.25G. >> > >> > Sorry, I should have specified that it is a 64 bit dom0 / hypervisor. >> >> Okay, well I''m not sure what limit qemu-dm is hitting then. Mapping 3.25G of >> guest memory will only require a few megabytes of pagetables for the qemu >> process in dom0. Perhaps there is a ulimit or something set on the qemu >> process? >> >> If we can work out and detect this limit, perhaps 64-bit qemu-dm could have >> a mapping cache similar to 32-bit qemu-dm, limited to some fraction of the >> detected mapping limit. And/or, on mapping failure, we could reclaim >> resources by simply zapping the existing cached mappings. Seems there''s a >> few options. I don''t really maintain qemu-dm myself -- you might get some >> help from Ian Jackson, Stefano, or Anthony Perard if you need more advice. > > The mapcache size limit should be 64GB on a 64bit qemu-dm. > Any interesting error messages in the qemu logs?Despite knowing next to nothing about qemu, I''m not certain the mapcache alone matters here: One would expect this to only consume memory for page table construction, but then you wouldn''t need Dom0 to have more memory than the guest for the latter to do heavy I/O. There ought to be something that allocates memory in amounts roughly equivalent to what the guest has under I/O. Jan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Stefano Stabellini
2011-Jan-05 16:22 UTC
Re: [Xen-devel] Xen 4.0.1 "xc_map_foreign_batch: mmap failed: Cannot allocate memory"
On Wed, 5 Jan 2011, Jan Beulich wrote:> >>> On 05.01.11 at 15:37, Stefano Stabellini <stefano.stabellini@eu.citrix.com> > wrote: > > On Thu, 16 Dec 2010, Keir Fraser wrote: > >> On 16/12/2010 20:44, "Charles Arnold" <carnold@novell.com> wrote: > >> > >> >>> On 12/16/2010 at 01:33 PM, in message <C9302813.2966F%keir@xen.org>, Keir > >> > Fraser <keir@xen.org> wrote: > >> >> On 16/12/2010 19:23, "Charles Arnold" <carnold@novell.com> wrote: > >> >> > >> >>> The bug is that qemu-dm seems to make the assumption that it can mmap from > >> >>> dom0 all the memory with which the guest has been defined instead of the > >> >>> memory > >> >>> that is actually available on the host. > >> >> > >> >> 32-bit dom0? Hm, I thought the qemu mapcache was supposed to limit the total > >> >> amount of guest memory mapped at one time, for a 32-bit qemu. For 64-bit > >> >> qemu I wouldn''t expect to find a limit as low as 3.25G. > >> > > >> > Sorry, I should have specified that it is a 64 bit dom0 / hypervisor. > >> > >> Okay, well I''m not sure what limit qemu-dm is hitting then. Mapping 3.25G of > >> guest memory will only require a few megabytes of pagetables for the qemu > >> process in dom0. Perhaps there is a ulimit or something set on the qemu > >> process? > >> > >> If we can work out and detect this limit, perhaps 64-bit qemu-dm could have > >> a mapping cache similar to 32-bit qemu-dm, limited to some fraction of the > >> detected mapping limit. And/or, on mapping failure, we could reclaim > >> resources by simply zapping the existing cached mappings. Seems there''s a > >> few options. I don''t really maintain qemu-dm myself -- you might get some > >> help from Ian Jackson, Stefano, or Anthony Perard if you need more advice. > > > > The mapcache size limit should be 64GB on a 64bit qemu-dm. > > Any interesting error messages in the qemu logs? > > Despite knowing next to nothing about qemu, I''m not certain the > mapcache alone matters here: One would expect this to only > consume memory for page table construction, but then you > wouldn''t need Dom0 to have more memory than the guest for the > latter to do heavy I/O. There ought to be something that > allocates memory in amounts roughly equivalent to what the > guest has under I/O.Qemu-dm allocates a bounce buffer for each in flight dma request, because the aio API used in qemu-dm cannot handle sg lists (it is probably the main reason to switch to the new qemu). However the bounce buffer is going to be free''d as soon as the dma request completes. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jan Beulich
2011-Jan-05 16:33 UTC
Re: [Xen-devel] Xen 4.0.1 "xc_map_foreign_batch: mmap failed: Cannot allocate memory"
>>> On 05.01.11 at 17:22, Stefano Stabellini <stefano.stabellini@eu.citrix.com>wrote:> On Wed, 5 Jan 2011, Jan Beulich wrote: >> >>> On 05.01.11 at 15:37, Stefano Stabellini <stefano.stabellini@eu.citrix.com> >> wrote: >> > On Thu, 16 Dec 2010, Keir Fraser wrote: >> >> On 16/12/2010 20:44, "Charles Arnold" <carnold@novell.com> wrote: >> >> >> >> >>> On 12/16/2010 at 01:33 PM, in message <C9302813.2966F%keir@xen.org>, Keir >> >> > Fraser <keir@xen.org> wrote: >> >> >> On 16/12/2010 19:23, "Charles Arnold" <carnold@novell.com> wrote: >> >> >> >> >> >>> The bug is that qemu-dm seems to make the assumption that it can mmap from >> >> >>> dom0 all the memory with which the guest has been defined instead of the >> >> >>> memory >> >> >>> that is actually available on the host. >> >> >> >> >> >> 32-bit dom0? Hm, I thought the qemu mapcache was supposed to limit the > total >> >> >> amount of guest memory mapped at one time, for a 32-bit qemu. For 64-bit >> >> >> qemu I wouldn''t expect to find a limit as low as 3.25G. >> >> > >> >> > Sorry, I should have specified that it is a 64 bit dom0 / hypervisor. >> >> >> >> Okay, well I''m not sure what limit qemu-dm is hitting then. Mapping 3.25G of >> >> guest memory will only require a few megabytes of pagetables for the qemu >> >> process in dom0. Perhaps there is a ulimit or something set on the qemu >> >> process? >> >> >> >> If we can work out and detect this limit, perhaps 64-bit qemu-dm could have >> >> a mapping cache similar to 32-bit qemu-dm, limited to some fraction of the >> >> detected mapping limit. And/or, on mapping failure, we could reclaim >> >> resources by simply zapping the existing cached mappings. Seems there''s a >> >> few options. I don''t really maintain qemu-dm myself -- you might get some >> >> help from Ian Jackson, Stefano, or Anthony Perard if you need more advice. >> > >> > The mapcache size limit should be 64GB on a 64bit qemu-dm. >> > Any interesting error messages in the qemu logs? >> >> Despite knowing next to nothing about qemu, I''m not certain the >> mapcache alone matters here: One would expect this to only >> consume memory for page table construction, but then you >> wouldn''t need Dom0 to have more memory than the guest for the >> latter to do heavy I/O. There ought to be something that >> allocates memory in amounts roughly equivalent to what the >> guest has under I/O. > > Qemu-dm allocates a bounce buffer for each in flight dma > request, because the aio API used in qemu-dm cannot handle sg lists (it > is probably the main reason to switch to the new qemu). > However the bounce buffer is going to be free''d as soon as the dma > request completes.But this means it can have very close to the total amount of memory the guest has in flight on its own. Clearly this should be throttled based on available memory (just consider you have multiple such I/O hungry guests). Jan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Charles Arnold
2011-Jan-05 17:17 UTC
Re: [Xen-devel] Xen 4.0.1 "xc_map_foreign_batch: mmap failed: Cannot allocate memory"
>>> On 1/5/2011 at 07:37 AM, in message<alpine.DEB.2.00.1101051422360.2390@kaball-desktop>, Stefano Stabellini <stefano.stabellini@eu.citrix.com> wrote:> On Thu, 16 Dec 2010, Keir Fraser wrote: >> On 16/12/2010 20:44, "Charles Arnold" <carnold@novell.com> wrote: >> >> >>> On 12/16/2010 at 01:33 PM, in message <C9302813.2966F%keir@xen.org>, Keir >> > Fraser <keir@xen.org> wrote: >> >> On 16/12/2010 19:23, "Charles Arnold" <carnold@novell.com> wrote: >> >> >> >>> The bug is that qemu-dm seems to make the assumption that it can mmap from >> >>> dom0 all the memory with which the guest has been defined instead of the >> >>> memory >> >>> that is actually available on the host. >> >> >> >> 32-bit dom0? Hm, I thought the qemu mapcache was supposed to limit the total >> >> amount of guest memory mapped at one time, for a 32-bit qemu. For 64-bit >> >> qemu I wouldn''t expect to find a limit as low as 3.25G. >> > >> > Sorry, I should have specified that it is a 64 bit dom0 / hypervisor. >> >> Okay, well I''m not sure what limit qemu-dm is hitting then. Mapping 3.25G of >> guest memory will only require a few megabytes of pagetables for the qemu >> process in dom0. Perhaps there is a ulimit or something set on the qemu >> process? >> >> If we can work out and detect this limit, perhaps 64-bit qemu-dm could have >> a mapping cache similar to 32-bit qemu-dm, limited to some fraction of the >> detected mapping limit. And/or, on mapping failure, we could reclaim >> resources by simply zapping the existing cached mappings. Seems there''s a >> few options. I don''t really maintain qemu-dm myself -- you might get some >> help from Ian Jackson, Stefano, or Anthony Perard if you need more advice. > > The mapcache size limit should be 64GB on a 64bit qemu-dm. > Any interesting error messages in the qemu logs?No, just the usual mmap error. See the attached logs. As a reminder, the following conditions exist to duplicate the problem. Host: x86_64 sles11sp1. Boot with dom0_mem=2048M, set (enable-dom0-ballooning no) in xend-config.sxp Hypervisor: x86_64 Xen 4.0.1 Guests: win2k3 (32bit) and win2k8r2 (64bit) The win2k8r2 guest uses robocopy to copy a 10 Gig file from the win2k3 guest. It copied about 23.2% before failing. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Stefano Stabellini
2011-Jan-05 17:48 UTC
Re: [Xen-devel] Xen 4.0.1 "xc_map_foreign_batch: mmap failed: Cannot allocate memory"
On Wed, 5 Jan 2011, Jan Beulich wrote:> But this means it can have very close to the total amount of > memory the guest has in flight on its own. Clearly this should > be throttled based on available memory (just consider you have > multiple such I/O hungry guests).This is true in theory, but Qemu emulates a PIIX3 chipset that only has two dma engines so I don''t believe is possible to have more than 2 dma requests in flight. However I am not sure what is the maximum size for a single dma request. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jan Beulich
2011-Jan-05 18:09 UTC
Re: [Xen-devel] Xen 4.0.1 "xc_map_foreign_batch: mmap failed: Cannot allocate memory"
>>> On 05.01.11 at 18:48, Stefano Stabellini <stefano.stabellini@eu.citrix.com>wrote:> On Wed, 5 Jan 2011, Jan Beulich wrote: >> But this means it can have very close to the total amount of >> memory the guest has in flight on its own. Clearly this should >> be throttled based on available memory (just consider you have >> multiple such I/O hungry guests). > > This is true in theory, but Qemu emulates a PIIX3 chipset that only has > two dma engines so I don''t believe is possible to have more than 2 dma > requests in flight. > However I am not sure what is the maximum size for a single dma request.Oh, no, we''re talking about guests using pv drivers (I thought that was said in the original mail). Jan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jan Beulich
2011-Jan-05 18:10 UTC
Re: [Xen-devel] Xen 4.0.1 "xc_map_foreign_batch: mmap failed: Cannot allocate memory"
>>> On 05.01.11 at 18:48, Stefano Stabellini <stefano.stabellini@eu.citrix.com>wrote:> On Wed, 5 Jan 2011, Jan Beulich wrote: >> But this means it can have very close to the total amount of >> memory the guest has in flight on its own. Clearly this should >> be throttled based on available memory (just consider you have >> multiple such I/O hungry guests). > > This is true in theory, but Qemu emulates a PIIX3 chipset that only has > two dma engines so I don''t believe is possible to have more than 2 dma > requests in flight. > However I am not sure what is the maximum size for a single dma request.>Oh, no, we''re talking about guests using pv drivers (I thought that >was said in the original mail).Rubbish. Forget that. Mixed up in my brain with something else. Jan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Stefano Stabellini
2011-Jan-06 16:50 UTC
Re: [Xen-devel] Xen 4.0.1 "xc_map_foreign_batch: mmap failed: Cannot allocate memory"
On Wed, 5 Jan 2011, Charles Arnold wrote:> No, just the usual mmap error. See the attached logs. > > As a reminder, the following conditions exist to duplicate the problem. > Host: x86_64 sles11sp1. Boot with dom0_mem=2048M, set (enable-dom0-ballooning no) in xend-config.sxp > Hypervisor: x86_64 Xen 4.0.1 > Guests: win2k3 (32bit) and win2k8r2 (64bit) > The win2k8r2 guest uses robocopy to copy a 10 Gig file from the win2k3 guest. It copied about 23.2% before failing.(I should have read the subject more carefully) xc_map_foreign_batch: mmap failed: Cannot allocate memory That is an mmap error on the xch->fd file descriptor. Considering that the privcmd mmap implementation sets the vma VM_RESERVED | VM_IO | VM_DONTCOPY | VM_PFNMAP and always returns success the ENOMEM must come from do_mmap. It could be that sysctl_max_map_count has been reached but it is 65530 by default while the maximum number of buckets in the mapcache on 64 bits is only 10000. Or the kernel could really be out of memory but in that case isn''t a little strange that we get the error from xc_map_foreign_batch? I think we need to add more tracing in the dom0 kernel to figure out what the problem is. What dom0 kernel are you using? Could you add some printk to the kernel source and try again? A patch like the following should be enough. --- diff --git a/mm/mmap.c b/mm/mmap.c index b179abb..3f65277 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -991,16 +991,20 @@ unsigned long do_mmap_pgoff(struct file *file, unsigned long addr, /* Careful about overflows.. */ len = PAGE_ALIGN(len); - if (!len) + if (!len) { + printk("DEBUG PAGE_ALIGN ENOMEM\n"); return -ENOMEM; + } /* offset overflow? */ if ((pgoff + (len >> PAGE_SHIFT)) < pgoff) return -EOVERFLOW; /* Too many mappings? */ - if (mm->map_count > sysctl_max_map_count) + if (mm->map_count > sysctl_max_map_count) { + printk("DEBUG too many mappings ENOMEM\n"); return -ENOMEM; + } /* Obtain the address to map to. we verify (or select) it and ensure * that it represents a valid section of the address space. @@ -1231,14 +1235,18 @@ unsigned long mmap_region(struct file *file, unsigned long addr, munmap_back: vma = find_vma_prepare(mm, addr, &prev, &rb_link, &rb_parent); if (vma && vma->vm_start < addr + len) { - if (do_munmap(mm, addr, len)) + if (do_munmap(mm, addr, len)) { + printk("DEBUG find_vma_prepare ENOMEM\n"); return -ENOMEM; + } goto munmap_back; } /* Check against address space limit. */ - if (!may_expand_vm(mm, len >> PAGE_SHIFT)) + if (!may_expand_vm(mm, len >> PAGE_SHIFT)) { + printk("DEBUG may_expand_vm ENOMEM\n"); return -ENOMEM; + } /* * Set ''VM_NORESERVE'' if we should not account for the @@ -1259,8 +1267,10 @@ munmap_back: */ if (accountable_mapping(file, vm_flags)) { charged = len >> PAGE_SHIFT; - if (security_vm_enough_memory(charged)) + if (security_vm_enough_memory(charged)) { + printk("DEBUG accountable_mapping ENOMEM\n"); return -ENOMEM; + } vm_flags |= VM_ACCOUNT; } @@ -1278,6 +1288,7 @@ munmap_back: */ vma = kmem_cache_zalloc(vm_area_cachep, GFP_KERNEL); if (!vma) { + printk("DEBUG kmem_cache_zalloc ENOMEM\n"); error = -ENOMEM; goto unacct_error; } _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Charles Arnold
2011-Jan-06 17:14 UTC
Re: [Xen-devel] Xen 4.0.1 "xc_map_foreign_batch: mmap failed: Cannot allocate memory"
>>> On 1/6/2011 at 09:50 AM, in message<alpine.DEB.2.00.1101051748120.2390@kaball-desktop>, Stefano Stabellini <stefano.stabellini@eu.citrix.com> wrote:> On Wed, 5 Jan 2011, Charles Arnold wrote: >> No, just the usual mmap error. See the attached logs. >> >> As a reminder, the following conditions exist to duplicate the problem. >> Host: x86_64 sles11sp1. Boot with dom0_mem=2048M, set (enable-dom0-ballooning > no) in xend-config.sxp >> Hypervisor: x86_64 Xen 4.0.1 >> Guests: win2k3 (32bit) and win2k8r2 (64bit) >> The win2k8r2 guest uses robocopy to copy a 10 Gig file from the win2k3 > guest. It copied about 23.2% before failing. > > (I should have read the subject more carefully) > > xc_map_foreign_batch: mmap failed: Cannot allocate memory > > That is an mmap error on the xch->fd file descriptor. > Considering that the privcmd mmap implementation sets the vma > VM_RESERVED | VM_IO | VM_DONTCOPY | VM_PFNMAP and always returns success > the ENOMEM must come from do_mmap. > It could be that sysctl_max_map_count has been reached but it is 65530 > by default while the maximum number of buckets in the mapcache on 64 > bits is only 10000. > Or the kernel could really be out of memory but in that case isn''t a > little strange that we get the error from xc_map_foreign_batch? > > I think we need to add more tracing in the dom0 kernel to figure out > what the problem is. > What dom0 kernel are you using?uname -a Linux xen86 2.6.32.24-0.2-xen #1 SMP 2010-10-29 16:39:49 +0200 x86_64 x86_64 x86_64 GNU/Linux This is sles11sp1.> Could you add some printk to the kernel source and try again? > A patch like the following should be enough.Yes, I''ll report back after building and testing. Thanks, - Charles> > --- > > > > diff --git a/mm/mmap.c b/mm/mmap.c > index b179abb..3f65277 100644 > --- a/mm/mmap.c > +++ b/mm/mmap.c > @@ -991,16 +991,20 @@ unsigned long do_mmap_pgoff(struct file *file, unsigned > long addr, > > /* Careful about overflows.. */ > len = PAGE_ALIGN(len); > - if (!len) > + if (!len) { > + printk("DEBUG PAGE_ALIGN ENOMEM\n"); > return -ENOMEM; > + } > > /* offset overflow? */ > if ((pgoff + (len >> PAGE_SHIFT)) < pgoff) > return -EOVERFLOW; > > /* Too many mappings? */ > - if (mm->map_count > sysctl_max_map_count) > + if (mm->map_count > sysctl_max_map_count) { > + printk("DEBUG too many mappings ENOMEM\n"); > return -ENOMEM; > + } > > /* Obtain the address to map to. we verify (or select) it and ensure > * that it represents a valid section of the address space. > @@ -1231,14 +1235,18 @@ unsigned long mmap_region(struct file *file, unsigned > long addr, > munmap_back: > vma = find_vma_prepare(mm, addr, &prev, &rb_link, &rb_parent); > if (vma && vma->vm_start < addr + len) { > - if (do_munmap(mm, addr, len)) > + if (do_munmap(mm, addr, len)) { > + printk("DEBUG find_vma_prepare ENOMEM\n"); > return -ENOMEM; > + } > goto munmap_back; > } > > /* Check against address space limit. */ > - if (!may_expand_vm(mm, len >> PAGE_SHIFT)) > + if (!may_expand_vm(mm, len >> PAGE_SHIFT)) { > + printk("DEBUG may_expand_vm ENOMEM\n"); > return -ENOMEM; > + } > > /* > * Set ''VM_NORESERVE'' if we should not account for the > @@ -1259,8 +1267,10 @@ munmap_back: > */ > if (accountable_mapping(file, vm_flags)) { > charged = len >> PAGE_SHIFT; > - if (security_vm_enough_memory(charged)) > + if (security_vm_enough_memory(charged)) { > + printk("DEBUG accountable_mapping ENOMEM\n"); > return -ENOMEM; > + } > vm_flags |= VM_ACCOUNT; > } > > @@ -1278,6 +1288,7 @@ munmap_back: > */ > vma = kmem_cache_zalloc(vm_area_cachep, GFP_KERNEL); > if (!vma) { > + printk("DEBUG kmem_cache_zalloc ENOMEM\n"); > error = -ENOMEM; > goto unacct_error; > }_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Charles Arnold
2011-Jan-06 20:49 UTC
Re: [Xen-devel] Xen 4.0.1 "xc_map_foreign_batch: mmap failed: Cannot allocate memory"
[This email is either empty or too large to be displayed at this time]
Jan Beulich
2011-Jan-07 09:35 UTC
Re: [Xen-devel] Xen 4.0.1 "xc_map_foreign_batch: mmap failed: Cannot allocate memory"
>>> On 06.01.11 at 21:49, Charles Arnold wrote: > >>> On 1/6/2011 at 10:14 AM, in message <4D25C782.5B74.0091.0@novell.com>, Charles Arnold wrote: > Attached is the messages file with the printk output.Hmm, a failure due to may_expand_vm() is really odd. Something must be explicitly setting a non-infinite RLIMIT_AS on qemu-dm (or one of its parents), as the default is "infinite" (as reaching "infinity" - being ~0UL - is simply impossible, and unduly large lengths should be caught by get_unmapped_area() already). /proc/<pid>/limits would at least tell us what the limit is. And certainly qemu-dm needs to be prepared to have a non-infinite address space limit set on it. Jan - Charles> >>> --- >>> >>> >>> >>> diff --git a/mm/mmap.c b/mm/mmap.c >>> index b179abb..3f65277 100644 >>> --- a/mm/mmap.c >>> +++ b/mm/mmap.c >>> @@ -991,16 +991,20 @@ unsigned long do_mmap_pgoff(struct file *file, unsigned > >> >>> long addr, >>> >>> /* Careful about overflows.. */ >>> len = PAGE_ALIGN(len); >>> - if (!len) >>> + if (!len) { >>> + printk("DEBUG PAGE_ALIGN ENOMEM\n"); >>> return -ENOMEM; >>> + } >>> >>> /* offset overflow? */ >>> if ((pgoff + (len >> PAGE_SHIFT)) < pgoff) >>> return -EOVERFLOW; >>> >>> /* Too many mappings? */ >>> - if (mm->map_count > sysctl_max_map_count) >>> + if (mm->map_count > sysctl_max_map_count) { >>> + printk("DEBUG too many mappings ENOMEM\n"); >>> return -ENOMEM; >>> + } >>> >>> /* Obtain the address to map to. we verify (or select) it and ensure >>> * that it represents a valid section of the address space. >>> @@ -1231,14 +1235,18 @@ unsigned long mmap_region(struct file *file, unsigned > >> >>> long addr, >>> munmap_back: >>> vma = find_vma_prepare(mm, addr, &prev, &rb_link, &rb_parent); >>> if (vma && vma->vm_start < addr + len) { >>> - if (do_munmap(mm, addr, len)) >>> + if (do_munmap(mm, addr, len)) { >>> + printk("DEBUG find_vma_prepare ENOMEM\n"); >>> return -ENOMEM; >>> + } >>> goto munmap_back; >>> } >>> >>> /* Check against address space limit. */ >>> - if (!may_expand_vm(mm, len >> PAGE_SHIFT)) >>> + if (!may_expand_vm(mm, len >> PAGE_SHIFT)) { >>> + printk("DEBUG may_expand_vm ENOMEM\n"); >>> return -ENOMEM; >>> + } >>> >>> /* >>> * Set ''VM_NORESERVE'' if we should not account for the >>> @@ -1259,8 +1267,10 @@ munmap_back: >>> */ >>> if (accountable_mapping(file, vm_flags)) { >>> charged = len >> PAGE_SHIFT; >>> - if (security_vm_enough_memory(charged)) >>> + if (security_vm_enough_memory(charged)) { >>> + printk("DEBUG accountable_mapping ENOMEM\n"); >>> return -ENOMEM; >>> + } >>> vm_flags |= VM_ACCOUNT; >>> } >>> >>> @@ -1278,6 +1288,7 @@ munmap_back: >>> */ >>> vma = kmem_cache_zalloc(vm_area_cachep, GFP_KERNEL); >>> if (!vma) { >>> + printk("DEBUG kmem_cache_zalloc ENOMEM\n"); >>> error = -ENOMEM; >>> goto unacct_error; >>> } >> >> >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Stefano Stabellini
2011-Jan-07 11:18 UTC
Re: [Xen-devel] Xen 4.0.1 "xc_map_foreign_batch: mmap failed: Cannot allocate memory"
On Fri, 7 Jan 2011, Jan Beulich wrote:> >>> On 06.01.11 at 21:49, Charles Arnold wrote: > > >>> On 1/6/2011 at 10:14 AM, in message <4D25C782.5B74.0091.0@novell.com>, Charles Arnold wrote: > > Attached is the messages file with the printk output. > > Hmm, a failure due to may_expand_vm() is really odd. Something > must be explicitly setting a non-infinite RLIMIT_AS on qemu-dm (or > one of its parents), as the default is "infinite" (as reaching "infinity" > - being ~0UL - is simply impossible, and unduly large lengths should > be caught by get_unmapped_area() already). > > /proc/<pid>/limits would at least tell us what the limit is. >Knowing this would be very interesting.> And certainly qemu-dm needs to be prepared to have a > non-infinite address space limit set on it. >Currently the number of buckets and the bucket size in the mapcache are statically defined depending on x86_32/x86_64. It shouldn''t be difficult to make them dynamic depending on RLIMIT_AS. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jan Beulich
2011-Jan-07 12:37 UTC
Re: [Xen-devel] Xen 4.0.1 "xc_map_foreign_batch: mmap failed: Cannot allocate memory"
>>> On 07.01.11 at 12:18, Stefano Stabellini <stefano.stabellini@eu.citrix.com>wrote:> On Fri, 7 Jan 2011, Jan Beulich wrote: >> >>> On 06.01.11 at 21:49, Charles Arnold wrote: >> > >>> On 1/6/2011 at 10:14 AM, in message <4D25C782.5B74.0091.0@novell.com>, > Charles Arnold wrote: >> > Attached is the messages file with the printk output. >> >> Hmm, a failure due to may_expand_vm() is really odd. Something >> must be explicitly setting a non-infinite RLIMIT_AS on qemu-dm (or >> one of its parents), as the default is "infinite" (as reaching "infinity" >> - being ~0UL - is simply impossible, and unduly large lengths should >> be caught by get_unmapped_area() already). >> >> /proc/<pid>/limits would at least tell us what the limit is. >> > > Knowing this would be very interesting.I just found that on SLE11 this gets set to 80% (admin controllable) of the sum of physical (not accounting for the balloon) and swap memory. That''s (at least on large systems using a relatively low dom0_mem= value) likely awfully low for qemu-dm serving large guests. However, it''s only rlim_cur that gets set this low by default, and hence it would seem reasonable to me to have qemu-dm bump it to whatever getrlimit() returns in rlimit_max.>> And certainly qemu-dm needs to be prepared to have a >> non-infinite address space limit set on it. >> > > Currently the number of buckets and the bucket size in the mapcache are > statically defined depending on x86_32/x86_64. > It shouldn''t be difficult to make them dynamic depending on RLIMIT_AS.That still wouldn''t help if RLIMIT_AS gets changed when it''s already running. The only proper way to deal with the situation as a whole (including but not limited to rlim_max being relatively low) is to get proper error handling implemented, either causing guest I/O to be throttled when mmap() fails, or no longer used mappings cleared (if that isn''t being done already). Jan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Charles Arnold
2011-Jan-07 17:03 UTC
Re: [Xen-devel] Xen 4.0.1 "xc_map_foreign_batch: mmap failed: Cannot allocate memory"
>>> On 1/7/2011 at 04:18 AM, in message<alpine.DEB.2.00.1101071115280.2390@kaball-desktop>, Stefano Stabellini <stefano.stabellini@eu.citrix.com> wrote:> On Fri, 7 Jan 2011, Jan Beulich wrote: >> >>> On 06.01.11 at 21:49, Charles Arnold wrote: >> > >>> On 1/6/2011 at 10:14 AM, in message <4D25C782.5B74.0091.0@novell.com>, > Charles Arnold wrote: >> > Attached is the messages file with the printk output. >> >> Hmm, a failure due to may_expand_vm() is really odd. Something >> must be explicitly setting a non-infinite RLIMIT_AS on qemu-dm (or >> one of its parents), as the default is "infinite" (as reaching "infinity" >> - being ~0UL - is simply impossible, and unduly large lengths should >> be caught by get_unmapped_area() already). >> >> /proc/<pid>/limits would at least tell us what the limit is. >> > > Knowing this would be very interesting.>From the qemu-dm process handling the win2k8r2 guest.Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size unlimited unlimited bytes Max core file size 1024 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15207 15207 processes Max open files 1024 8192 files Max locked memory unlimited unlimited bytes Max address space 3454238720 unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15207 15207 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us>From the xend parent processLimit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 1024 unlimited bytes Max resident set 1838370816 unlimited bytes Max processes 15207 15207 processes Max open files 1024 8192 files Max locked memory 65536 262144 bytes Max address space 3454238720 unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15207 15207 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us - Charles _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Stefano Stabellini
2011-Jan-10 11:11 UTC
Re: [Xen-devel] Xen 4.0.1 "xc_map_foreign_batch: mmap failed: Cannot allocate memory"
On Fri, 7 Jan 2011, Jan Beulich wrote:> I just found that on SLE11 this gets set to 80% (admin controllable) > of the sum of physical (not accounting for the balloon) and swap > memory. That''s (at least on large systems using a relatively low > dom0_mem= value) likely awfully low for qemu-dm serving large > guests. > > However, it''s only rlim_cur that gets set this low by default, and > hence it would seem reasonable to me to have qemu-dm bump it > to whatever getrlimit() returns in rlimit_max. >It seems reasonable to me too.> >> And certainly qemu-dm needs to be prepared to have a > >> non-infinite address space limit set on it. > >> > > > > Currently the number of buckets and the bucket size in the mapcache are > > statically defined depending on x86_32/x86_64. > > It shouldn''t be difficult to make them dynamic depending on RLIMIT_AS. > > That still wouldn''t help if RLIMIT_AS gets changed when it''s already > running. The only proper way to deal with the situation as a whole > (including but not limited to rlim_max being relatively low) is to get > proper error handling implemented, either causing guest I/O to be > throttled when mmap() fails, or no longer used mappings cleared > (if that isn''t being done already). >I am not sure that handling RLIMIT_AS dynamic changes while qemu is running is worth the effort; but for sure we have to deal with RLIMIT_AS being lower than expect at qemu startup time. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel