I''m interested in the ballooning, so I read the balloon.c I have a questions: How is the ballooned pages be used? After decrease_reservation(), balloon driver tells the vmm the pages that it gets. Then, how xen utilizes this reclaimed pages? When other domains (both dom0 and domU) requests increasing reservation, will xen allocate these pages to them? If yes, from increasing_reservation(), it seems that balloon just releases the pages that it previously occupied. If no, where do these "ballooned" pages go? Thanks a lot in advance! weiming _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Sat, 2007-11-03 at 21:31 -0400, weiming wrote:> I''m interested in the ballooning, so I read the balloon.c > I have a questions: > > How is the ballooned pages be used? After decrease_reservation(), > balloon driver tells the vmm the pages that it gets. Then, how xen > utilizes this reclaimed pages?the pages are unmapped by the domain, i.e. there are no further references to them. the VMM can verify this. after the vmm gets the page list, no new references will be allowed, so the hypervisor knows for sure that the domain won''t mess around with these pages afterwards. it''s ''gone'', and only the balloon knows that it''s not only allocated, but indeed *gone*. alloc/free is maybe best understood like this: to a native guest kernel, there is no such concept of memory ''disappearing''. machine memory does not just disappear (except for such deeply esoteric things like memory hotplug). however, that''s what is happening under the VMM. but still, there''s the possibly of taking memory away from the kernel: by allocating it. as long as the ballon driver allocated that page, no other entity in the system may access it. (note that security were at risk with a buggy guest kernel. that''s why this unmapping thing is so important. the VMM must insist that the guest _cannot_ access the page anymore).> When other domains (both dom0 and domU) requests increasing > reservation, will xen allocate these pages to them?yes. reusing those pages in other domains is the sole purpose of taking them. there''s no point in letting memory idle around. an ideal system has 100% memory utilization. those frames not used by processes are left to buffers and caches. fundamental good-OS principle.> If yes, from increasing_reservation(), it seems that balloon just > releases the pages that it previously occupied. > If no, where do these "ballooned" pages go?it not only releases them, it reinstalls them in the kernel address space, and reassigns them to the kernel physical/machine translation. note that it does not need to be the same machine frame. it may be any other page retrieved from the VMM. that''s what the phys_to_machine, machphys_update and update_va_mapping calls are good for. regards, daniel -- Daniel Stodden LRR - Lehrstuhl für Rechnertechnik und Rechnerorganisation Institut für Informatik der TU München D-85748 Garching http://www.lrr.in.tum.de/~stodden mailto:stodden@cs.tum.edu PGP Fingerprint: F5A4 1575 4C56 E26A 0B33 3D80 457E 82AE B0D8 735B _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
I have been confused about ballooning on dom0 for a while, so this looks like a good time to ask it. I have dom0 with ~1Gb of memory. (About 900Mb as shown in ''xm list'' once it is freshly booted.) I start a guest, and that gets decreased as the memory is allocated to domU: Name ID Mem Domain-0 0 735 etch32-builder.my.flat 24 128 After a while I shutdown that guest and I expect the memory to be available to dom0 again - but it never is. I see dom0 left with the reduced ~700Mb. Is that expected? Steve -- _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Sun, 2007-11-04 at 15:04 +0000, Steve Kemp wrote:> I have been confused about ballooning on dom0 for a while, so > this looks like a good time to ask it. > > I have dom0 with ~1Gb of memory. (About 900Mb as shown in ''xm list'' > once it is freshly booted.) > > I start a guest, and that gets decreased as the memory is allocated > to domU: > > Name ID Mem > Domain-0 0 735 > etch32-builder.my.flat 24 128 > > After a while I shutdown that guest and I expect the memory to > be available to dom0 again - but it never is. I see dom0 left > with the reduced ~700Mb. > > Is that expected?a leak of that size would not go unnoticed :) outline, but no warranties, as i didn''t build it: - in the beginng, .. well, all memory belongs to xen. - dom0, unless otherwise configured (mem=), gets a map of up to the whole amount of memory. so anything beyond the VMM can be in dom0. - lacking any other domain beyond dom0 on boot, the initial reservation of dom0 is filled with all pages beyond the VMM allocation. 900MB. - creating a domU requires dom0 to return part of its reservation (decrease_reservation()). 128MB. - destroying domU is up to the VMM. these pages are not automagically reclaimed by dom0. there is also to ''pushing'' of pages in a domain, as there is no point in doing so. dom0 needs to ask (via increase_reservation()). - dom0, rather idle, is as happy with 735MB as it used to be with 900MB. so the freed domU memory stays on the xen heap for the time being. regards, daniel -- Daniel Stodden LRR - Lehrstuhl für Rechnertechnik und Rechnerorganisation Institut für Informatik der TU München D-85748 Garching http://www.lrr.in.tum.de/~stodden mailto:stodden@cs.tum.edu PGP Fingerprint: F5A4 1575 4C56 E26A 0B33 3D80 457E 82AE B0D8 735B _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hi Daniel, Very appreciate for your so detailed explanation. You clarified some of my confusion. Before I post my question here, I read the paper "Memory resource Management in VMware ESX server", "art of virtualization", read Xen manual, checked the xen Wiki and searched in the maillist archive but can''t get a complete picture about balloon. 1) when a guesting os starts, how does it determine the amount of physical memory? i.e. which value determines the number of entries in mem_map? Is the value specified in configuration file? 2) what''s the exact role that xm mem-max plays? I can set it to be higher than the value in configuration file. I think that it just sets the "new_target" to balloon via xenbus or /proc/xen/balloon, right? 3) Once some pages are "ballooned out", these pages will be utilized by other domains, so if we later try to restore to initial status, how does VMM find available pages? In increase_reservation(), ... rc = HYPERVISOR_ memory_op(XENMEM_populate_physmap, &reservation) if (rc < nr_pages) ... In my understanding, hypervisor *tries* to find some free pages to return to the os. 4) in balloon.c, there are some functions that I can''t find the calling sites. they are dealloc_pte_fn, alloc_empty_pages_and_pagevec, balloon_update_driver_allowance, etc. Are they be called back by hypervisor? Thanks a lot in advance! Weiming On 11/4/07, Daniel Stodden <stodden@cs.tum.edu> wrote:> > On Sat, 2007-11-03 at 21:31 -0400, weiming wrote: > > I''m interested in the ballooning, so I read the balloon.c > > I have a questions: > > > > How is the ballooned pages be used? After decrease_reservation(), > > balloon driver tells the vmm the pages that it gets. Then, how xen > > utilizes this reclaimed pages? > > the pages are unmapped by the domain, i.e. there are no further > references to them. the VMM can verify this. after the vmm gets the page > list, no new references will be allowed, so the hypervisor knows for > sure that the domain won''t mess around with these pages afterwards. it''s > ''gone'', and only the balloon knows that it''s not only allocated, but > indeed *gone*. > > alloc/free is maybe best understood like this: to a native guest kernel, > there is no such concept of memory ''disappearing''. machine memory does > not just disappear (except for such deeply esoteric things like memory > hotplug). however, that''s what is happening under the VMM. but still, > there''s the possibly of taking memory away from the kernel: by > allocating it. as long as the ballon driver allocated that page, no > other entity in the system may access it. > > (note that security were at risk with a buggy guest kernel. that''s why > this unmapping thing is so important. the VMM must insist that the guest > _cannot_ access the page anymore). > > > When other domains (both dom0 and domU) requests increasing > > reservation, will xen allocate these pages to them? > > yes. reusing those pages in other domains is the sole purpose of taking > them. there''s no point in letting memory idle around. an ideal system > has 100% memory utilization. those frames not used by processes are left > to buffers and caches. fundamental good-OS principle. > > > If yes, from increasing_reservation(), it seems that balloon just > > releases the pages that it previously occupied. > > If no, where do these "ballooned" pages go? > > it not only releases them, it reinstalls them in the kernel address > space, and reassigns them to the kernel physical/machine translation. > note that it does not need to be the same machine frame. it may be any > other page retrieved from the VMM. that''s what the phys_to_machine, > machphys_update and update_va_mapping calls are good for. > > regards, > daniel > > -- > Daniel Stodden > LRR - Lehrstuhl für Rechnertechnik und Rechnerorganisation > Institut für Informatik der TU München D-85748 Garching > http://www.lrr.in.tum.de/~stodden mailto:stodden@cs.tum.edu > PGP Fingerprint: F5A4 1575 4C56 E26A 0B33 3D80 457E 82AE B0D8 735B > > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Sun Nov 04, 2007 at 16:30:02 +0100, Daniel Stodden wrote:> a leak of that size would not go unnoticed :) > outline, but no warranties, as i didn''t build it:Thanks, that seems to match what I''m seeing and is a good explaination. ObReference: I use Xen in two ways 1. Xen hosting, where each guest has a lot of memory and dom0 just has a tiny amount of memory, set via dom0_mem 2. My desktop machine which has 0-4 guests running at any given moment. For the desktop case it is really noticeable when dom0 hasn''t had the memory returned to it; firefox gets sluggish and xine doesn''t work so well... Steve -- _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Thanks, Daniel. You answered my first question perfectly that I just post 2 minutes ago. :D On 11/4/07, Daniel Stodden <stodden@cs.tum.edu> wrote:> > > On Sun, 2007-11-04 at 15:04 +0000, Steve Kemp wrote: > > I have been confused about ballooning on dom0 for a while, so > > this looks like a good time to ask it. > > > > I have dom0 with ~1Gb of memory. (About 900Mb as shown in ''xm list'' > > once it is freshly booted.) > > > > I start a guest, and that gets decreased as the memory is allocated > > to domU: > > > > Name ID Mem > > Domain-0 0 735 > > etch32-builder.my.flat 24 128 > > > > After a while I shutdown that guest and I expect the memory to > > be available to dom0 again - but it never is. I see dom0 left > > with the reduced ~700Mb. > > > > Is that expected? > > a leak of that size would not go unnoticed :) > > outline, but no warranties, as i didn''t build it: > > - in the beginng, .. well, all memory belongs to xen. > > - dom0, unless otherwise configured (mem=), gets a map of up to the > whole amount of memory. so anything beyond the VMM can be in dom0. > > - lacking any other domain beyond dom0 on boot, the initial reservation > of dom0 is filled with all pages beyond the VMM allocation. 900MB. > > - creating a domU requires dom0 to return part of its > reservation (decrease_reservation()). 128MB. > > - destroying domU is up to the VMM. these pages are not automagically > reclaimed by dom0. there is also to ''pushing'' of pages in a domain, > as there is no point in doing so. dom0 needs to ask (via > increase_reservation()). > > - dom0, rather idle, is as happy with 735MB as it used to be with > 900MB. > so the freed domU memory stays on the xen heap for the time being. > > regards, > daniel > > > > -- > Daniel Stodden > LRR - Lehrstuhl für Rechnertechnik und Rechnerorganisation > Institut für Informatik der TU München D-85748 Garching > http://www.lrr.in.tum.de/~stodden mailto:stodden@cs.tum.edu > PGP Fingerprint: F5A4 1575 4C56 E26A 0B33 3D80 457E 82AE B0D8 735B > > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Sun, 2007-11-04 at 10:34 -0500, weiming wrote:> Hi Daniel, > > Very appreciate for your so detailed explanation. You clarified some > of my confusion. Before I post my question here, I read the paper > "Memory resource Management in VMware ESX server", "art of > virtualization", read Xen manual, checked the xen Wiki and searched in > the maillist archive but can''t get a complete picture about balloon. > > 1) when a guesting os starts, how does it determine the amount of > physical memory? i.e. which value determines the number of entries in > mem_map? Is the value specified in configuration file?one page of initial domain memory is dedicated to a ''start_info'' structure. you may grep the xen/include sources for the definition. then see e.g. linux-2.6-xen-sparse/arch/i386/kernel/setup-xen.c as i believe you already understood, ther are two important distinctions here: - ''nr_pages'': the size of the physical address range which the domain can use. that''s basically the maximum memory, different from what the domain actually gets. - ''reservation'': the amount in nr_pages actually filled with machine memory. nr_pages is in start info, as is the frame list corresponding to the initial reservation set by the domain builder. the domain builder takes gets nr_pages from the memory= field in the configuration file. not sure about how this bootstraps in the balloon. e.g. i''m not sure whether the whole initial memory is allocated and then returned again only upon demand. or if the initial reservation is full memory and then only grown by the balloon. i believe the former is the case. maybe someone else can comment (please).> 2) what''s the exact role that xm mem-max plays? I can set it to be > higher than the value in configuration file. I think that it just > sets the "new_target" to balloon via xenbus or /proc/xen/balloon, > right?you can tell the domain its physical limit is 1G. that''s e.g. what the guest storage allocator then uses to initialize itself. but you can as well go afterwards and modify a configurable limit below the hard limit. its then up to the balloon to wring the memory out of the guest system. why higher values get accepted i cannot comment on. maybe clipped without further comment? see, the kernel can free a lot of memory even when it is ''in use'' by swapping it to disk. that''s one of the basic ideas of having a balloon driver: do not build your own page replacement, but put put pressure on the existing guest memory management to do it for you. that is what the call to alloc_page() in the balloon driver is essentially doing. otoh, memory in use be the kernel cannot be swapped. that''s why the pages grabbed by the balloon itself remain safe. that memory must be locked. but, again, i should admit that my own understanding gets a bit fuzzy here, regarding which is which in the config file and within xm parameters. you''re right in that the communication is performed via xenbus. i spent more time reading xen and kernel code than the surrounding python source. maybe someone else can comment better or (hopefully) correct me if i''m talking rubbish somewhere above. send me an update once you hit it. :)> 3) Once some pages are "ballooned out", these pages will be utilized > by other domains, so if we later try to restore to initial status, how > does VMM find available pages?the memory gets balanced by asking other domains to decrease their reservation. you can wrench down the domain to the bare kernel when you''re a driver. the kernel gives you anything you ask for -- physically. or, rather, until the linux oom killer kicks in, a rather undesirable side effect, as a recent thread on this list discussed.> In increase_reservation(), > ... > rc = HYPERVISOR_ memory_op(XENMEM_populate_physmap, &reservation) > if (rc < nr_pages) > ... > > In my understanding, hypervisor *tries* to find some free pages to > return to the os.yes, this can fail. there''s no fundamental (i.e. consistency) problem in failing. the kernel will find that all memory is in use, as it will on a native system if something grabbed all the memory. so, there''s a balloon driver saying "admittedly, i''m presently sitting on 80% of all memory, but now it''s mine. you modprobed me, so you trust me, now go look somewhere else". a native driver would not even have been asked.> 4) in balloon.c, there are some functions that I can''t find the > calling sites. they are dealloc_pte_fn, alloc_empty_pages_and_pagevec,no idea.> balloon_update_driver_allowance,this one i can explain. there''s memory apart from balloon entering and leaving the domU. that''s ''I/O memory'' which is moved between frontend and backend drivers to transfer data. for both receiving and sending, domU is required to take memory from its own reservation. so it hands these pages over to the backend driver domain and gets them back not before there backend is finished with the transfer (i.e. map/unmapping similar to ballooning). the balloon driver accounts for this memory, so frontends call this function to tell her about it.> etc. Are they be called back by hypervisor?they are certainly not immediate callback functions. control transfers in/to the guest, if they are initated by the hypervisor, are always done via event channels. the only other path would be an iret. that means there''s no ''call'' between xen and guests. those symbols you see must be in use somewhere. regards, daniel -- Daniel Stodden LRR - Lehrstuhl für Rechnertechnik und Rechnerorganisation Institut für Informatik der TU München D-85748 Garching http://www.lrr.in.tum.de/~stodden mailto:stodden@cs.tum.edu PGP Fingerprint: F5A4 1575 4C56 E26A 0B33 3D80 457E 82AE B0D8 735B _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 4/11/07 17:04, "Daniel Stodden" <stodden@cs.tum.edu> wrote:> not sure about how this bootstraps in the balloon. e.g. i''m not sure > whether the whole initial memory is allocated and then returned again > only upon demand. or if the initial reservation is full memory and then > only grown by the balloon. i believe the former is the case. maybe > someone else can comment (please).The arch-specific code (setup-xen.c) sets up a mem_map big enough for maximum memory ever allocatable to this domain. ''Empty pages'', which are not populated with RAM by the initial reservation, are detected by the balloon driver when it initialises. These empty pages are filled with RAM by the balloon driver on demand, if it detects that the memory target is raised. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Sun, 2007-11-04 at 17:17 +0000, Keir Fraser wrote:> > > On 4/11/07 17:04, "Daniel Stodden" <stodden@cs.tum.edu> wrote: > > > not sure about how this bootstraps in the balloon. e.g. i''m not sure > > whether the whole initial memory is allocated and then returned again > > only upon demand. or if the initial reservation is full memory and then > > only grown by the balloon. i believe the former is the case. maybe > > someone else can comment (please). > > The arch-specific code (setup-xen.c) sets up a mem_map big enough for > maximum memory ever allocatable to this domain. ''Empty pages'', which are not > populated with RAM by the initial reservation, are detected by the balloon > driver when it initialises. These empty pages are filled with RAM by the > balloon driver on demand, if it detects that the memory target is raised.thanks. that helped, even despite me screwing ''full'' and ''initial'' while asking. greetings, daniel -- Daniel Stodden LRR - Lehrstuhl für Rechnertechnik und Rechnerorganisation Institut für Informatik der TU München D-85748 Garching http://www.lrr.in.tum.de/~stodden mailto:stodden@cs.tum.edu PGP Fingerprint: F5A4 1575 4C56 E26A 0B33 3D80 457E 82AE B0D8 735B _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Sun, 2007-11-04 at 18:19 +0100, Daniel Stodden wrote:> On Sun, 2007-11-04 at 17:17 +0000, Keir Fraser wrote: > > > > > > On 4/11/07 17:04, "Daniel Stodden" <stodden@cs.tum.edu> wrote: > > > > > not sure about how this bootstraps in the balloon. e.g. i''m not sure > > > whether the whole initial memory is allocated and then returned again > > > only upon demand. or if the initial reservation is full memory and then > > > only grown by the balloon. i believe the former is the case. maybe > > > someone else can comment (please). > > > > The arch-specific code (setup-xen.c) sets up a mem_map big enough for > > maximum memory ever allocatable to this domain. ''Empty pages'', which are not > > populated with RAM by the initial reservation, are detected by the balloon > > driver when it initialises. These empty pages are filled with RAM by the > > balloon driver on demand, if it detects that the memory target is raised. > > thanks. > > that helped, even despite me screwing ''full'' and ''initial'' while asking.while you are at it, from xmdomain.cfg(5) memory: Xen does not support overcommit of memory, so the total memory of all guests (+ 64 MB needed for Xen) must be less than or equal to the physical RAM in the machine. ''memory'' is mem_map size, right? is this outdated or simply not true, or am i missing something? thanks, daniel -- Daniel Stodden LRR - Lehrstuhl für Rechnertechnik und Rechnerorganisation Institut für Informatik der TU München D-85748 Garching http://www.lrr.in.tum.de/~stodden mailto:stodden@cs.tum.edu PGP Fingerprint: F5A4 1575 4C56 E26A 0B33 3D80 457E 82AE B0D8 735B _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 4/11/07 17:28, "Daniel Stodden" <stodden@cs.tum.edu> wrote:> while you are at it, from xmdomain.cfg(5) > > memory: > Xen does not support overcommit of memory, so the total memory > of all guests (+ 64 MB needed for Xen) must be less than or > equal to the physical RAM in the machine. > > ''memory'' is mem_map size, right? is this outdated or simply not true, or > am i missing something?No, ''memory'' is total RAM currently assigned across all tests. It just means the obvious thing --- you can''t assign more RAM than you have in your entire system! -- KEir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Thanks you, Daniel and Keir! Now I get a more completed picture about ballooning. A quick question: So far, the ballooning is triggered manually, right? On theory, it should be possible to automatically trigger ballooning on domains to get memory resource more balanced. This requires the dom0 to monitor the memory pressure of each domains. These information can be collected by each balloon driver and sent to xen. Is such function available on the xen commercial version? Is there any plans to develop this in xen open source community? Thanks! On Nov 5, 2007 2:35 AM, Keir Fraser <Keir.Fraser@cl.cam.ac.uk> wrote:> On 4/11/07 17:28, "Daniel Stodden" <stodden@cs.tum.edu> wrote: > > > while you are at it, from xmdomain.cfg(5) > > > > memory: > > Xen does not support overcommit of memory, so the total memory > > of all guests (+ 64 MB needed for Xen) must be less than or > > equal to the physical RAM in the machine. > > > > ''memory'' is mem_map size, right? is this outdated or simply not true, or > > am i missing something? > > No, ''memory'' is total RAM currently assigned across all tests. It just means > the obvious thing --- you can''t assign more RAM than you have in your entire > system! > > -- KEir > > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> A quick question: > So far, the ballooning is triggered manually, right? On theory, it > should be possible to automatically trigger ballooning on domains to > get memory resource more balanced. This requires the dom0 to monitor > the memory pressure of each domains. These information can be > collected by each balloon driver and sent to xen. > > Is such function available on the xen commercial version? Is there any > plans to develop this in xen open source community?It''s not currently available in the commercial version as far as I know. But for a definitive reference, you should look at the information on www.citrixxenserver.com. The idea of doing this does get floated about on the list semi-frequently... I think there may have been a masters student looking at improving the balloon driver, so maybe he''ll consider doing it. If he doesn''t, then it could make a good little project for somebody. It wouldn''t have to be very clever to start with. Cheers, Mark> Thanks! > > On Nov 5, 2007 2:35 AM, Keir Fraser <Keir.Fraser@cl.cam.ac.uk> wrote: > > On 4/11/07 17:28, "Daniel Stodden" <stodden@cs.tum.edu> wrote: > > > while you are at it, from xmdomain.cfg(5) > > > > > > memory: > > > Xen does not support overcommit of memory, so the total memory > > > of all guests (+ 64 MB needed for Xen) must be less than or > > > equal to the physical RAM in the machine. > > > > > > ''memory'' is mem_map size, right? is this outdated or simply not true, > > > or am i missing something? > > > > No, ''memory'' is total RAM currently assigned across all tests. It just > > means the obvious thing --- you can''t assign more RAM than you have in > > your entire system! > > > > -- KEir > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel-- Dave: Just a question. What use is a unicyle with no seat? And no pedals! Mark: To answer a question with a question: What use is a skateboard? Dave: Skateboards have wheels. Mark: My wheel has a wheel! _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel