Dan Magenheimer
2010-Aug-23 23:49 UTC
[Xen-devel] Linux mainstream balloon driver fails to grow memory= -> maxmem=
Hi Jeremy -- Vasiliy discovered (and I confirmed) that the mainstream Linux balloon driver won''t balloon above the initial memory with which the Linux guest was launched. E.g. a 2.6.34 guest launched with these in vm.cfg: memory=1024 maxmem=1536 displays properly in xentop, but changing target_kb in /sys/devices/system/xen_memory/xen_memory0 doesn''t "take" for any value greater than 1024*1024. We tested with 2.6.34 but it even fails in RHEL6 Beta2. Is this a known problem/bug (or maybe a feature)? Thanks, Dan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jeremy Fitzhardinge
2010-Aug-24 00:50 UTC
Re: [Xen-devel] Linux mainstream balloon driver fails to grow memory= -> maxmem=
On 08/23/2010 04:49 PM, Dan Magenheimer wrote:> Hi Jeremy -- > > Vasiliy discovered (and I confirmed) that the mainstream > Linux balloon driver won''t balloon above the initial > memory with which the Linux guest was launched. E.g. > a 2.6.34 guest launched with these in vm.cfg: > > memory=1024 > maxmem=1536 > > displays properly in xentop, but changing target_kb > in /sys/devices/system/xen_memory/xen_memory0 > doesn''t "take" for any value greater than 1024*1024. > > We tested with 2.6.34 but it even fails in RHEL6 Beta2. > > Is this a known problem/bug (or maybe a feature)?Known bug. There''s no machinery in there to allocate enough struct pages for maxmem, only for "memory". The immediately workaround is to start the domain with fully populated "memory" and then have it balloon out the unwanted stuff asap, but it does mean you need to have enough memory to start with. There''s a couple of experiments floating around in the tree to try to honour maxmem, but they''re somewhat rotted. Its probably worth trying to resurrect them though. And of course there''s Daniel''s work which is a more flexible andcomprehensive approach, but it does rely on hotplug memory (which has its own overheads). J _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Dan Magenheimer
2010-Aug-24 02:11 UTC
RE: [Xen-devel] Linux mainstream balloon driver fails to grow memory= -> maxmem=
> From: Jeremy Fitzhardinge [mailto:jeremy@goop.org] > Subject: Re: [Xen-devel] Linux mainstream balloon driver fails to grow > memory= -> maxmem> > On 08/23/2010 04:49 PM, Dan Magenheimer wrote: > > Hi Jeremy -- > > > > Vasiliy discovered (and I confirmed) that the mainstream > > Linux balloon driver won''t balloon above the initial > > memory with which the Linux guest was launched. E.g. > > a 2.6.34 guest launched with these in vm.cfg: > > > > memory=1024 > > maxmem=1536 > > > > displays properly in xentop, but changing target_kb > > in /sys/devices/system/xen_memory/xen_memory0 > > doesn''t "take" for any value greater than 1024*1024. > > > > We tested with 2.6.34 but it even fails in RHEL6 Beta2. > > > > Is this a known problem/bug (or maybe a feature)? > > Known bug. There''s no machinery in there to allocate enough struct > pages for maxmem, only for "memory".OK, good to know.> The immediately workaround is to start the domain with fully populated > "memory" and then have it balloon out the unwanted stuff asap, but it > does mean you need to have enough memory to start with. > > There''s a couple of experiments floating around in the tree to try to > honour maxmem, but they''re somewhat rotted. Its probably worth trying > to resurrect them though.Not sure, except for backwards compatibility.> And of course there''s Daniel''s work which is a more flexible > andcomprehensive approach, but it does rely on hotplug memory (which > has > its own overheads).Tmem (the "frontswap" part) also takes advantage of the difference between memory= and maxmem=, allowing pages that are swapped to "hypervisor RAM" to use RAM between the two numbers. This requires that the guest have a swap disk configured that is at least as large as the difference between memory= and maxmem= though. So if Daniel''s stuff works along with tmem, I''m not too bothered about fixing this known difference between 2.6.18-xen and mainstream. And IIRC the maxmem= parameter is required for PoD to work for Windows also. IOW, this should probably be documented (any ideas where?) but the "old way" may best be left to obsolesce with 2.6.18-xen. Dan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Vasiliy G Tolstov
2010-Aug-24 11:29 UTC
RE: [Xen-devel] Linux mainstream balloon driver fails to grow memory= -> maxmem=
В Пнд, 23/08/2010 в 19:11 -0700, Dan Magenheimer пишет:> > From: Jeremy Fitzhardinge [mailto:jeremy@goop.org] > > Subject: Re: [Xen-devel] Linux mainstream balloon driver fails to grow > > memory= -> maxmem> > > > On 08/23/2010 04:49 PM, Dan Magenheimer wrote: > > > Hi Jeremy -- > > > > > > Vasiliy discovered (and I confirmed) that the mainstream > > > Linux balloon driver won''t balloon above the initial > > > memory with which the Linux guest was launched. E.g. > > > a 2.6.34 guest launched with these in vm.cfg: > > > > > > memory=1024 > > > maxmem=1536 > > > > > > displays properly in xentop, but changing target_kb > > > in /sys/devices/system/xen_memory/xen_memory0 > > > doesn''t "take" for any value greater than 1024*1024. > > > > > > We tested with 2.6.34 but it even fails in RHEL6 Beta2. > > > > > > Is this a known problem/bug (or maybe a feature)? > > > > Known bug. There''s no machinery in there to allocate enough struct > > pages for maxmem, only for "memory". > > OK, good to know. > > > The immediately workaround is to start the domain with fully populated > > "memory" and then have it balloon out the unwanted stuff asap, but it > > does mean you need to have enough memory to start with. > > > > There''s a couple of experiments floating around in the tree to try to > > honour maxmem, but they''re somewhat rotted. Its probably worth trying > > to resurrect them though. > > Not sure, except for backwards compatibility. > > > And of course there''s Daniel''s work which is a more flexible > > andcomprehensive approach, but it does rely on hotplug memory (which > > has > > its own overheads). > > Tmem (the "frontswap" part) also takes advantage of the difference > between memory= and maxmem=, allowing pages that are swapped to > "hypervisor RAM" to use RAM between the two numbers. This requires > that the guest have a swap disk configured that is at least as > large as the difference between memory= and maxmem= though. > > So if Daniel''s stuff works along with tmem, I''m not too bothered about > fixing this known difference between 2.6.18-xen and mainstream. > And IIRC the maxmem= parameter is required for PoD to work for Windows > also. IOW, this should probably be documented (any ideas where?) > but the "old way" may best be left to obsolesce with 2.6.18-xen. > > DanStupidly apply patch from Daniel and Dan to linux-2.6.34.5 does not work. Dan, can You check what need to modify with Daniel patch to work together with tmem? P.S. Best, if the domU can selfballooning to memory, after that it can emulate memory-hotplug and add needed memory slot, after that selfballoon to next target.... If that possible. With selfballooning = 0 xm mem-set xxx xxx not work''s domU kernel panic: Failed services in runlevel 3: named [ 491.162333] ------------[ cut here ]------------ [ 491.162333] kernel BUG at drivers/xen/balloon.c:449! [ 491.162333] invalid opcode: 0000 [#1] SMP [ 491.162333] last sysfs file: /sys/devices/system/xen_memory/xen_memory0/selfballooning [ 491.162333] CPU 0 [ 491.162333] Modules linked in: [ 491.162333] [ 491.162333] Pid: 31, comm: events/0 Not tainted 2.6.34.5-cleancache_frontswap_tmem_hotplug #2 / [ 491.162333] RIP: e030:[<ffffffff81172165>] [<ffffffff81172165>] balloon_process+0xc6/0x80d [ 491.162333] RSP: e02b:ffff880176d23db0 EFLAGS: 00010046 [ 491.162333] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000080 [ 491.162333] RDX: ffffffff813d1790 RSI: 6db6db6db6db6db7 RDI: ffffffff815117b0 [ 491.162333] RBP: ffff880176d23e40 R08: dead000000100100 R09: 0000000000000002 [ 492.825231] R10: dead000000100100 R11: 0000000000000000 R12: 0000000000000200 [ 492.825231] R13: 0000000000000200 R14: 00000000000fa080 R15: ffffea00050795a0 [ 492.825231] FS: 00007fadd42eb720(0000) GS:ffff88000251d000(0000) knlGS:0000000000000000 [ 492.825231] CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b [ 492.825231] CR2: 00000000019e00e0 CR3: 00000000d3557000 CR4: 0000000000002660 [ 492.825231] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 492.825231] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 492.825231] Process events/0 (pid: 31, threadinfo ffff880176d22000, task ffff880176d02fb0) [ 492.825231] Stack: [ 492.825231] ffff880176d02fb0 00000000000131c0 00000000000131c0 ffff880176d23df0 [ 492.825231] <0> 0000000000000000 ffff880176d23fd8 ffff880176d23fd8 0000000000000200 [ 492.825231] <0> 0000000000000000 0000000000000000 0000000000000000 0000000000007ff0 [ 492.825231] Call Trace: [ 492.825231] [<ffffffff810429ad>] worker_thread+0x15b/0x1e4 [ 492.825231] [<ffffffff8117209f>] ? balloon_process+0x0/0x80d [ 492.825231] [<ffffffff81046179>] ? autoremove_wake_function+0x0/0x34 [ 492.825231] [<ffffffff81042852>] ? worker_thread+0x0/0x1e4 [ 492.825231] [<ffffffff81045d9b>] kthread+0x7a/0x82 [ 492.825231] [<ffffffff81008a04>] kernel_thread_helper+0x4/0x10 [ 492.825231] [<ffffffff81007e21>] ? int_ret_from_sys_call+0x7/0x1b [ 492.825231] [<ffffffff8125579d>] ? retint_restore_args+0x5/0x6 [ 492.825231] [<ffffffff81008a00>] ? kernel_thread_helper+0x0/0x10 [ 492.825231] Code: 0e 00 48 8b 15 4d f6 25 00 48 8b 4d 90 48 81 fa 90 17 3d 81 48 89 45 a8 48 8d 42 d8 48 0f 44 c1 31 c9 31 db eb 45 48 85 c0 75 02 <0f> 0b 48 ba 00 00 00 00 00 16 00 00 48 be b7 6d db b6 6d db b6 [ 492.825231] RIP [<ffffffff81172165>] balloon_process+0xc6/0x80d [ 492.825231] RSP <ffff880176d23db0> [ 492.825231] ---[ end trace cc4f9bc7a1d45b2e ]--- -- Vasiliy G Tolstov <v.tolstov@selfip.ru> Selfip.Ru _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Daniel Kiper
2010-Aug-24 19:56 UTC
Re: [Xen-devel] Linux mainstream balloon driver fails to grow memory= -> maxmem=
Hello, On Tue, Aug 24, 2010 at 03:29:46PM +0400, Vasiliy G Tolstov wrote:> Stupidly apply patch from Daniel and Dan to linux-2.6.34.5 does not > work. > Dan, can You check what need to modify with Daniel patch to work > together with tmem? > > P.S. Best, if the domU can selfballooning to memory, after that it can > emulate memory-hotplug and add needed memory slot, after that > selfballoon to next target.... If that possible. > > With selfballooning = 0 xm mem-set xxx xxx not work''s domU kernel panic:[...] Thx for report. I will check that also. Daniel _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Vasiliy G Tolstov
2010-Aug-24 20:22 UTC
Re: [Xen-devel] Linux mainstream balloon driver fails to grow memory= -> maxmem=
On Tue, 2010-08-24 at 21:56 +0200, Daniel Kiper wrote:> Hello, > > On Tue, Aug 24, 2010 at 03:29:46PM +0400, Vasiliy G Tolstov wrote: > > Stupidly apply patch from Daniel and Dan to linux-2.6.34.5 does not > > work. > > Dan, can You check what need to modify with Daniel patch to work > > together with tmem? > > > > P.S. Best, if the domU can selfballooning to memory, after that it can > > emulate memory-hotplug and add needed memory slot, after that > > selfballoon to next target.... If that possible. > > > > With selfballooning = 0 xm mem-set xxx xxx not work''s domU kernel panic: > > [...] > > Thx for report. I will check that also. > > Danielrget memory and others.I''m investigate, that memory hotplug work''s fine. Problem with selfballoonin - i think it needs to recalculate some variables, like minimal target memory and others. -- Vasiliy G Tolstov <v.tolstov@selfip.ru> Selfip.Ru _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Dan Magenheimer
2010-Aug-25 22:44 UTC
RE: [Xen-devel] Linux mainstream balloon driver fails to grow memory= -> maxmem=
> So if Daniel''s stuff works along with tmem, I''m not too bothered about > fixing this known difference between 2.6.18-xen and mainstream. > And IIRC the maxmem= parameter is required for PoD to work for Windows > also. IOW, this should probably be documented (any ideas where?) > but the "old way" may best be left to obsolesce with 2.6.18-xen.Hmmm... I spoke too soon. We apparently have a number of customers that use maxmem and would be upset if the functionality goes away.> > There''s a couple of experiments floating around in the tree to try to > > honour maxmem, but they''re somewhat rotted. Its probably worth > > trying to resurrect them though.If you can send more information about the "experiments", I may be able to take a look fairly soon (unless Daniel Kiper is already looking into it). Since the support is currently missing from RHEL6 beta, future customers using RHEL6 as a Xen PV guest may get very confused by what might appear to them as a tools or Xen bug. Thanks, Dan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel