Xen devel - Oct 2011 - Strange (???) xl behavior for save, migrate and migrate-receive

Hi,

During work on memory hotplug for Xen I have received some notices
that it breaks machine migration. I had some time and I done some
tests a few days ago. It looks that source of this problem is
xl command itself. I discovered that generic save/restore mechanism
is used for machine migration. xl save store machine config which
was used at machine startup with current machine state. It means
that it does not take into account any config changes which were made
during machine run. This behavior does not allow migrating domain,
on which memory hotplug was used, to restore on destination host
because current size of memory allocated for machine is larger than
size of memory allocated at startup by memory option. Yes, it is
memory option not maxmem option. However, it is not important here
because I think that generic behavior of xl save, migrate and migrate-receive
should be changed (fix for memory hotplug case is workaround for the
generic problem which will return sooner or later). I think that xl save,
migrate and migrate-receive should use current machine state and __CURRENT__
config (from xenstore ???) to do their tasks. However, I am aware that
this change could have large impact on current users. That is why I decided
to ask you about your opinion and suggested solutions in that case
(in general not memory hotplug only).

Currently, these problems could be workaround by passing
path to config file with current config to xl command.

I have done tests on Xen Ver. 4.1.2-rc3. I have not done tests
on xm command, however, I suppose that it has similar behavior.

Daniel

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> From: Daniel Kiper [mailto:dkiper@net-space.pl]
> Subject: [Xen-devel] Strange (???) xl behavior for save, migrate and
migrate-receive
> 
> During work on memory hotplug for Xen I have received some notices
> that it breaks machine migration. I had some time and I done some
> tests a few days ago. It looks that source of this problem is
> xl command itself. I discovered that generic save/restore mechanism
> is used for machine migration. xl save store machine config which
> was used at machine startup with current machine state. It means
> that it does not take into account any config changes which were made
> during machine run. This behavior does not allow migrating domain,
> on which memory hotplug was used, to restore on destination host
> because current size of memory allocated for machine is larger than
> size of memory allocated at startup by memory option. Yes, it is
> memory option not maxmem option. However, it is not important here
> because I think that generic behavior of xl save, migrate and
migrate-receive
> should be changed (fix for memory hotplug case is workaround for the
> generic problem which will return sooner or later). I think that xl save,
> migrate and migrate-receive should use current machine state and
__CURRENT__
> config (from xenstore ???) to do their tasks. However, I am aware that
> this change could have large impact on current users. That is why I decided
> to ask you about your opinion and suggested solutions in that case
> (in general not memory hotplug only).
> 
> Currently, these problems could be workaround by passing
> path to config file with current config to xl command.
> 
> I have done tests on Xen Ver. 4.1.2-rc3. I have not done tests
> on xm command, however, I suppose that it has similar behavior.
Hi Daniel --

In a recent internal discussion at Oracle, we were thinking about
whether to enable hotplug functionality in a guest kernel and it
raised some concerns about manageability.  I think right now
the system administrator of the guest can arbitrarily increase
memory size beyond maxmem... that is really the whole point
of your implementation, right?  But this may be unacceptable to
the "data center administrator" (the admin who runs the
"cloud"
and determines such things as vcpus and maxmem across all guests)
since multiple guests may try to do this semi-maliciously to grab
as much RAM as they can. And Xen has no way to discourage this,
so will just hand out the RAM first-come-first-serve, right?

I was thinking one way to handle this problem would be
to have a new vm.cfg parameter, e.g. "maxmem_hotplug".
If unspecified (or zero), there are no constraints placed
on the guest.  If specified (in MB), Xen/xl will disallow
hotplug memory requests beyond this maximum.

I suspect, if implemented properly, this might also eliminate
your live migration issue.

Apologies if something like this was previously discussed or
is already working in your implementation.

Dan

P.S. Also FYI, selfballooning is implemented in Oracle''s kernel
so we should work to ensure that selfballooning and hotplug
work properly together.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Mon, Oct 17, Daniel Kiper wrote:
> During work on memory hotplug for Xen I have received some notices
> that it breaks machine migration. I had some time and I done some
> tests a few days ago. It looks that source of this problem is
> xl command itself. I discovered that generic save/restore mechanism
> is used for machine migration. xl save store machine config which
> was used at machine startup with current machine state. It means
> that it does not take into account any config changes which were made
> during machine run. This behavior does not allow migrating domain,
> on which memory hotplug was used, to restore on destination host
> because current size of memory allocated for machine is larger than
> size of memory allocated at startup by memory option. Yes, it is
> memory option not maxmem option. However, it is not important here
> because I think that generic behavior of xl save, migrate and
migrate-receive
> should be changed (fix for memory hotplug case is workaround for the
> generic problem which will return sooner or later). I think that xl save,
> migrate and migrate-receive should use current machine state and
__CURRENT__
> config (from xenstore ???) to do their tasks. However, I am aware that
> this change could have large impact on current users. That is why I decided
> to ask you about your opinion and suggested solutions in that case
> (in general not memory hotplug only).
Its easy to implement in xl by throwing some xenstore watches into the
code.

http://lists.xensource.com/archives/html/xen-devel/2011-10/msg00778.html

Olaf

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Mon, 2011-10-17 at 19:44 +0100, Dan Magenheimer
wrote:
> In a recent internal discussion at Oracle, we were thinking about
> whether to enable hotplug functionality in a guest kernel and it
> raised some concerns about manageability.  I think right now
> the system administrator of the guest can arbitrarily increase
> memory size beyond maxmem...
The memory limit for a guest is ultimately controlled by the host
administrator/toolstack. The in-guest admin cannot exceed that, even
using hotplug. I think that limit is currently always set to the current
balloon target.

AIUI Daniel''s work only allows a guest admin to take advantage of new
memory above maxmem _after_ the host admin has provisioned that RAM to
the guest. IOW it only allows the guest to take advantage of new memory
given to it and does not allow the guest to acquire new memory of its
own accord.

Ian.


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Tue, Oct 18, 2011 at 09:19:01AM +0100, Ian Campbell
wrote:
> On Mon, 2011-10-17 at 19:44 +0100, Dan Magenheimer wrote:
> > In a recent internal discussion at Oracle, we were thinking about
> > whether to enable hotplug functionality in a guest kernel and it
> > raised some concerns about manageability.  I think right now
> > the system administrator of the guest can arbitrarily increase
> > memory size beyond maxmem...
>
> The memory limit for a guest is ultimately controlled by the host
> administrator/toolstack. The in-guest admin cannot exceed that, even
> using hotplug.
Correct.
> I think that limit is currently always set to the current balloon target.
Nope. It is set by maxmem option.
> AIUI Daniel''s work only allows a guest admin to take advantage of
new
> memory above maxmem _after_ the host admin has provisioned that RAM to
> the guest. IOW it only allows the guest to take advantage of new memory
> given to it and does not allow the guest to acquire new memory of its
> own accord.
Guest/host administartor could allocate for given guest no more memory than
maxmem (its value could be changed by xl mem-max <domain>
<new_size>) allows,
regardless of mechanism (ballooning or memory hotplug) used for that allocation.
It means that memory hotplug does not pose any security threat in that area.

Daniel

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Mon, Oct 17, 2011 at 11:44:51AM -0700, Dan Magenheimer
wrote:
> > From: Daniel Kiper [mailto:dkiper@net-space.pl]
> > Subject: [Xen-devel] Strange (???) xl behavior for save, migrate and
migrate-receive
> >
> > During work on memory hotplug for Xen I have received some notices
> > that it breaks machine migration. I had some time and I done some
> > tests a few days ago. It looks that source of this problem is
> > xl command itself. I discovered that generic save/restore mechanism
> > is used for machine migration. xl save store machine config which
> > was used at machine startup with current machine state. It means
> > that it does not take into account any config changes which were made
> > during machine run. This behavior does not allow migrating domain,
> > on which memory hotplug was used, to restore on destination host
> > because current size of memory allocated for machine is larger than
> > size of memory allocated at startup by memory option. Yes, it is
> > memory option not maxmem option. However, it is not important here
> > because I think that generic behavior of xl save, migrate and
migrate-receive
> > should be changed (fix for memory hotplug case is workaround for the
> > generic problem which will return sooner or later). I think that xl
save,
> > migrate and migrate-receive should use current machine state and
__CURRENT__
> > config (from xenstore ???) to do their tasks. However, I am aware that
> > this change could have large impact on current users. That is why I
decided
> > to ask you about your opinion and suggested solutions in that case
> > (in general not memory hotplug only).
> >
> > Currently, these problems could be workaround by passing
> > path to config file with current config to xl command.
> >
> > I have done tests on Xen Ver. 4.1.2-rc3. I have not done tests
> > on xm command, however, I suppose that it has similar behavior.
>
> Hi Daniel --
>
> In a recent internal discussion at Oracle, we were thinking about
> whether to enable hotplug functionality in a guest kernel and it
> raised some concerns about manageability.  I think right now
> the system administrator of the guest can arbitrarily increase
> memory size beyond maxmem... that is really the whole point
> of your implementation, right?  But this may be unacceptable to
> the "data center administrator" (the admin who runs the
"cloud"
> and determines such things as vcpus and maxmem across all guests)
> since multiple guests may try to do this semi-maliciously to grab
> as much RAM as they can. And Xen has no way to discourage this,
> so will just hand out the RAM first-come-first-serve, right?
>
> I was thinking one way to handle this problem would be
> to have a new vm.cfg parameter, e.g. "maxmem_hotplug".
> If unspecified (or zero), there are no constraints placed
> on the guest.  If specified (in MB), Xen/xl will disallow
> hotplug memory requests beyond this maximum.
>
> I suspect, if implemented properly, this might also eliminate
> your live migration issue.
>
> Apologies if something like this was previously discussed or
> is already working in your implementation.
Please look into Ian and my e-mail posted earlier.
> Dan
>
> P.S. Also FYI, selfballooning is implemented in Oracle''s kernel
> so we should work to ensure that selfballooning and hotplug
> work properly together.
I am happy to do that, however, I am very busy now.
Could we postpone this 2-3 months ???

Daniel

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Mon, Oct 17, 2011 at 09:12:05PM +0200, Olaf Hering
wrote:
> On Mon, Oct 17, Daniel Kiper wrote:
>
> > During work on memory hotplug for Xen I have received some notices
> > that it breaks machine migration. I had some time and I done some
> > tests a few days ago. It looks that source of this problem is
> > xl command itself. I discovered that generic save/restore mechanism
> > is used for machine migration. xl save store machine config which
> > was used at machine startup with current machine state. It means
> > that it does not take into account any config changes which were made
> > during machine run. This behavior does not allow migrating domain,
> > on which memory hotplug was used, to restore on destination host
> > because current size of memory allocated for machine is larger than
> > size of memory allocated at startup by memory option. Yes, it is
> > memory option not maxmem option. However, it is not important here
> > because I think that generic behavior of xl save, migrate and
migrate-receive
> > should be changed (fix for memory hotplug case is workaround for the
> > generic problem which will return sooner or later). I think that xl
save,
> > migrate and migrate-receive should use current machine state and
__CURRENT__
> > config (from xenstore ???) to do their tasks. However, I am aware that
> > this change could have large impact on current users. That is why I
decided
> > to ask you about your opinion and suggested solutions in that case
> > (in general not memory hotplug only).
>
> Its easy to implement in xl by throwing some xenstore watches into the
> code.
>
> http://lists.xensource.com/archives/html/xen-devel/2011-10/msg00778.html
Thanks. I saw your e-mail eariler, however, I decided to post my one
separetly because it is a bit different case. As you said implementation
is probably easy. However, I do not want break default behavior of
xl without consent with Xen-devel members.

Daniel

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> From: Daniel Kiper [mailto:dkiper@net-space.pl]
> Subject: Re: [Xen-devel] Strange (???) xl behavior for save, migrate and
migrate-receive
> 
> On Tue, Oct 18, 2011 at 09:19:01AM +0100, Ian Campbell wrote:
> > On Mon, 2011-10-17 at 19:44 +0100, Dan Magenheimer wrote:
> > > In a recent internal discussion at Oracle, we were thinking about
> > > whether to enable hotplug functionality in a guest kernel and it
> > > raised some concerns about manageability.  I think right now
> > > the system administrator of the guest can arbitrarily increase
> > > memory size beyond maxmem...
> >
> > The memory limit for a guest is ultimately controlled by the host
> > administrator/toolstack. The in-guest admin cannot exceed that, even
> > using hotplug.
> 
> Correct.
> 
> > I think that limit is currently always set to the current balloon
target.
> 
> Nope. It is set by maxmem option.
> 
> > AIUI Daniel''s work only allows a guest admin to take
advantage of new
> > memory above maxmem _after_ the host admin has provisioned that RAM to
> > the guest. IOW it only allows the guest to take advantage of new
memory
> > given to it and does not allow the guest to acquire new memory of its
> > own accord.
> 
> Guest/host administartor could allocate for given guest no more memory than
> maxmem (its value could be changed by xl mem-max <domain>
<new_size>) allows,
> regardless of mechanism (ballooning or memory hotplug) used for that
allocation.
> It means that memory hotplug does not pose any security threat in that
area.
OK, thanks for the clarification, Daniel and Ian.
> > P.S. Also FYI, selfballooning is implemented in Oracle''s
kernel
> > so we should work to ensure that selfballooning and hotplug
> > work properly together.
> 
> I am happy to do that, however, I am very busy now.
> Could we postpone this 2-3 months ???
Sure, that''s fine.

Dan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

2011/10/18 Daniel Kiper <dkiper@net-space.pl>
> Guest/host administartor could allocate for given guest no more memory
> than
> maxmem (its value could be changed by xl mem-max <domain>
<new_size>)
> allows,
> regardless of mechanism (ballooning or memory hotplug) used for that
> allocation.
> It means that memory hotplug does not pose any security threat in that
> area.
>
Sorry for bumping thread. Does this mean, that if i use memory=512,
maxmem=4096 grows via balloon to 4096, after that set maxmem to 8192 and
grows guest to 7000mb and do live migration - nothing bad happening and
guest migrate process successed?

-- 
Vasiliy Tolstov,
Clodo.ru
e-mail: v.tolstov@selfip.ru
jabber: vase@selfip.ru


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Wed, Nov 02, 2011 at 06:17:38PM +0400, Vasiliy Tolstov
wrote:
> 2011/10/18 Daniel Kiper <dkiper@net-space.pl>
>
> > Guest/host administartor could allocate for given guest no more memory
than
> > maxmem (its value could be changed by xl mem-max <domain>
<new_size>) allows,
> > regardless of mechanism (ballooning or memory hotplug) used for that
allocation.
> > It means that memory hotplug does not pose any security threat in that
> > area.
>
> Sorry for bumping thread. Does this mean, that if i use memory=512,
> maxmem=4096 grows via balloon to 4096, after that set maxmem to 8192 and
> grows guest to 7000mb and do live migration - nothing bad happening and
> guest migrate process successed?
Yep, however, now you must use workaround for xl (which was described)
earlier in this thread. I have idea how to fix xl but I do not have
time to do it. I am going to do that next year.

Daniel

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel