Xen devel - Mar 2012 - libxl/xl memory paging/sharing/ballooning/etc proposal.

I think we''ve roughly reached consensus on what this could look like.
I''ve combined the various aspects of the proposal into the below.

In hindsight I should probably have written this as a proposed patch to
libxl.h and xl.cfg.pod.5 etc but I''ve written it now so here it is.

There are a few outstanding issues / decisions marked with XXX

Ian J -- I''ve CC''d you since I propose some new uses of the
events
subsystem (as well as being interested in your overall opinion)

high level libxl interface
---------------------------

libxl_domain_set_memory_policy(ctx, domid, const char *actor, const char
*params):
        
        Sets the name of the "actor" which will implement memory
policy
        for this domain by writing the name of the actor to
                /libxl/X/memory-policy/actor
        and the params to
                /libxl/X/memory-policy/actor-params
                        
        actor-params is defined per-actor. normally be a comma separated
        key=value list.

        If libxl_domain_set_memory_policy is not called for a domain
        then the default is considered to be "" i.e. none.

        XXX should the actor config also be part of struct
        libxl_domain_build_info? (I expect so)

libxl_domain_get_memory_policy(ctx, domid, char **actor, char **params)

        Retrieves the current policy and params (if non NULL etc).

libxl_domain_set_memory_target(ctx, domid, target_memkb, bool relative,
bool enforce):

        Replaces existing libxl_domain_memory_target() function.
                
        If memory-policy/actor != "":
                updates /libxl/X/memory-policy/<target,enforce>
        else:
                behaves as libxl_domain_memory_target today, e.g. sets
                balloon target in xenstore and iff enforcing then sets
                maxmem -- probably this is just a call to
                libxl_domain_set_balloon_target() (see below)
                        
        target in xenstore is absolute and may be calculated from the
        existing target and provided target_memkb if relative == true
        
        enforce in xenstore is the one of the strings "true" or
"false".
        (XXX or 0 1? what do we normally do? does it matter?)
        
New libxl_event_type DOMAIN_MEMORY_TARGET_CHANGED

        This event is triggered when any of /libxl/X/memory-policy/*
        change.
        
        Parameters in libxl_event.u are "char *actor", "char
        *actor_params", "bool target" and "bool
enforce" as read from
        XS.
        
        Associated functions libxl_evenable/disable_blah are provided
        and take an optional "char * actor" to match against. if actor
        == NULL then events will be generated for any actor name,
        otherwise only the changes while the actor is the given actor
        will occur (this includes setting the actor to that value)

On domain build libxl shall behave as if libxl_domain_set_memory_target
had been called with:
        target_memkb == libxl_domain_build_info.target_memkb
(XXX I think that''s the right one).

low level libxl interface
-------------------------

It is expected that toolstacks primary user interface will be via the
above high level interface. The lowlevel interface is provided in order
to allow implementation of memory policy actors using libxl and to allow
toolstacks to optionally provide an interface for expert users to shoot
themselves in the foot with ( ;-) )

libxl_domain_(set|get)_balloon_target(ctx, domid, target_memkb, relative,
enforce):
        
        Read/write /local/domain/X/memory/target to control the in guest
        balloon driver. This is basically the existing
        libxl_domain_memory_target with a new name
                
        As before this is a request to the guest balloon driver to try
        and use this much actual RAM. This is an existing guest
        interface which we cannot change.

libxl_domain_(set|get)_paging_target(ctx, domid, target_memkb, relative)

        Read/write whichever XS path controls the xenpaging daemon''s
        target. (Olaf?). I think this path should not be
        under /local/domain since it should not be guest visible
        (probably it already isn''t).

        It is an error to call this for a PV guest (ERROR_??) for a
        guest which does not have paging enabled
        (ERROR_PAGING_DISABLED?).
        
libxl_domain_(enable|disable)_paging(ctx, domid)

        Idempotently starts or stops the xenpaging daemon for a domain.

libxl_domain_(enable|disable)_sharing(ctx, domid)

        Does what it says on the tin.

actor behaviour
---------------

An actor should use the libxl event system to monitor for
DOMAIN_MEMORY_TARGET_CHANGED events. An actor must ignore any event for
which actor != their name.

On receiving such an event the actor should apply its policy and
configuration in order (try and) cause the domain to use "target"
amount
of actual RAM. This may involve calling the libxl low level interfaces,
including enabled paging and/or sharing as it desires.

It is an actor specific policy whether it guarantees to meet the target
(e.g. a balloon only actor would not be able to make that guarantee)

An actor might be either be a single daemon with host wide knowledge or
a daemon per domain (or anything in between). It is expected in the
multiple daemon case that all the daemons will either co-ordinate as
necessary or implement a policy where coordination is not necessary.

Normally the same actor would be used for all domains but we do not rule
out the possibility for non-homogeneous configurations (but it is up to
the user to make sure the actors don''t have conflicting, potentially
catastrophic, interactions...)

XXX I think we need a DOMAIN_INTRODUCE event to allow for a system wide
actor. This seems like a generally useful event anyhow.

xl actor configuration
----------------------

xl will read its default actor from its configuration file. Since it is
usually expected to use the same actor for all domains it would normally
be /etc/xen/xl.conf, but this will be overrideable in a guest config
file, likewise for the policy params. xl will call
libxl_domain_set_memory_policy with the configured value while building
a domain (XXX or use fields in libxl_domain_build_config if we do
that). 

If the configuration key is absent then the default value used is
"xl" (described below). A user can explicitly supply actor =
"" to get
the default libxl behaviour.

XXX Config key names TBD, "memory_policy" and
"memory_policy_args"?

xl high level interface
-----------------------

xl mem-set <target>

        Pretty much same as today. Calls libxl_domain_set_memory_target
        with enforcing == 1. (but note that the behaviour of
        libxl_domain_set_memory_target has changed)

xl mem-policy-set <actor> <param-string>

        Calls libxl_domain_set_memory_policy to set the new actor.

"xl" actor behaviour
--------------------

Each xl daemon instance will listen for DOMAIN_MEMORY_TARGET_CHANGED
events for the domain which it is managing. When the event is triggered
(and actor == "xl") then xl will:

        if actor != "xl": return
        
        call libxl_domain_set_balloon_target(ctx, domid, target,
        enforcing)
        
        if domain == pv: return # cannot do more for a PV guest
        
        XXX if not enforcing: return ??? (not sure about this, we don''t
        actually expose the ability to set enforcing in the xl
        interface)
        
        XXX if actor-params:paging!=true return ??? I think we should do
        this to enable memory_policy_params="paging=true|false" based
        control of paging.
        
        delay for a configurable number of seconds (configurable via
        actor-params:delay=N, default 5s?) XXX does libxl provide a
        guest accessible timeout event? Would be usefully convenient.

        XXX libxl_domain_paging_enable here or on domain build / policy
        set? libxl_domain_paging_enable is idempotent so calling it here
        is always safe?
        
        call libxl_domain_set_paging_target(ctx, domid, target)

The "xl" actor explicitly does not deal with page sharing. It is
assumed
that page sharing requires system wide knowledge and policy which is out
of xl''s scope and hence a different actor would be needed.

xl low level interface
----------------------

All of the following setters functions should error out unless actor
="xl-manual". The getter could return the current value regardless.

xl mem-balloon-(set|get)-target
xl mem-paging-(set|get)-target

        Set or get the relevant targets by calling the equivalent libxl
        function. 
        
        Enables or disables paging as necessary, i.e. target == 0 =>
        disable
        
NB if you call "xl mem-set" when actor == "xl-manual" then
it won''t do
anything since nothing is reacting as that actor.

XXX We could alternatively have xl also implement an "xl-manual" actor
which immediately sets both ballooning and paging targets to the given
value. Probably a good idea?

XXX Maybe "xl" (the default) should be called "xl-auto" ?

If you "xl create -e" (or -F) then you won''t get a monitoring
daemon and therefore you won''t get any actor implementing the memory
policy, just like you won''t get automatic reboot control. I think this
is expected and reasonable.

> I think we''ve roughly reached consensus on what this could look
like.
> I''ve combined the various aspects of the proposal into the below.
Very awesome. Comments inline below.
>
> In hindsight I should probably have written this as a proposed patch to
> libxl.h and xl.cfg.pod.5 etc but I''ve written it now so here it
is.
>
> There are a few outstanding issues / decisions marked with XXX
>
> Ian J -- I''ve CC''d you since I propose some new uses of
the events
> subsystem (as well as being interested in your overall opinion)
>
> high level libxl interface
> ---------------------------
>
> libxl_domain_set_memory_policy(ctx, domid, const char *actor, const char
> *params):
>
>         Sets the name of the "actor" which will implement memory
policy
>         for this domain by writing the name of the actor to
>                 /libxl/X/memory-policy/actor
>         and the params to
>                 /libxl/X/memory-policy/actor-params
>
>         actor-params is defined per-actor. normally be a comma separated
>         key=value list.
>
>         If libxl_domain_set_memory_policy is not called for a domain
>         then the default is considered to be "" i.e. none.
>
>         XXX should the actor config also be part of struct
>         libxl_domain_build_info? (I expect so)
>
> libxl_domain_get_memory_policy(ctx, domid, char **actor, char **params)
>
>         Retrieves the current policy and params (if non NULL etc).
>
> libxl_domain_set_memory_target(ctx, domid, target_memkb, bool relative,
> bool enforce):
>
>         Replaces existing libxl_domain_memory_target() function.
>
>         If memory-policy/actor != "":
>                 updates /libxl/X/memory-policy/<target,enforce>
>         else:
>                 behaves as libxl_domain_memory_target today, e.g. sets
>                 balloon target in xenstore and iff enforcing then sets
>                 maxmem -- probably this is just a call to
>                 libxl_domain_set_balloon_target() (see below)
>
>         target in xenstore is absolute and may be calculated from the
>         existing target and provided target_memkb if relative == true
>
>         enforce in xenstore is the one of the strings "true" or
"false".
>         (XXX or 0 1? what do we normally do? does it matter?)
>
> New libxl_event_type DOMAIN_MEMORY_TARGET_CHANGED
>
>         This event is triggered when any of /libxl/X/memory-policy/*
>         change.
>
>         Parameters in libxl_event.u are "char *actor", "char
>         *actor_params", "bool target" and "bool
enforce" as read from
>         XS.
>
>         Associated functions libxl_evenable/disable_blah are provided
>         and take an optional "char * actor" to match against. if
actor
>         == NULL then events will be generated for any actor name,
>         otherwise only the changes while the actor is the given actor
>         will occur (this includes setting the actor to that value)
>
> On domain build libxl shall behave as if libxl_domain_set_memory_target
> had been called with:
>         target_memkb == libxl_domain_build_info.target_memkb
> (XXX I think that''s the right one).
>
> low level libxl interface
> -------------------------
>
> It is expected that toolstacks primary user interface will be via the
> above high level interface. The lowlevel interface is provided in order
> to allow implementation of memory policy actors using libxl and to allow
> toolstacks to optionally provide an interface for expert users to shoot
> themselves in the foot with ( ;-) )
>
> libxl_domain_(set|get)_balloon_target(ctx, domid, target_memkb, relative,
> enforce):
>
>         Read/write /local/domain/X/memory/target to control the in guest
>         balloon driver. This is basically the existing
>         libxl_domain_memory_target with a new name
>
>         As before this is a request to the guest balloon driver to try
>         and use this much actual RAM. This is an existing guest
>         interface which we cannot change.
>
> libxl_domain_(set|get)_paging_target(ctx, domid, target_memkb, relative)
>
>         Read/write whichever XS path controls the xenpaging
daemon''s
>         target. (Olaf?). I think this path should not be
>         under /local/domain since it should not be guest visible
>         (probably it already isn''t).
It is /local/domain/X/memory/target-tot_pages. I agree it should not be
guest visible, and we''re still in time to change it.

/libxl/X/memory-policy/paging-target? Is there a non-libxl
non-/local/domain path xenpaging could look into, should it be launched
autonomously? Olaf, do you care about xenpaging in non-libxl environments?
>
>         It is an error to call this for a PV guest (ERROR_??) for a
>         guest which does not have paging enabled
>         (ERROR_PAGING_DISABLED?).
>
> libxl_domain_(enable|disable)_paging(ctx, domid)
>
>         Idempotently starts or stops the xenpaging daemon for a domain.
>
Paging disable takes time. It has to swap back in everything it has paged
out. So it falls under the (really) long-lasting asynchronous operation
framework.
> libxl_domain_(enable|disable)_sharing(ctx, domid)
>
>         Does what it says on the tin.
>
> actor behaviour
> ---------------
>
> An actor should use the libxl event system to monitor for
> DOMAIN_MEMORY_TARGET_CHANGED events. An actor must ignore any event for
> which actor != their name.
>
> On receiving such an event the actor should apply its policy and
> configuration in order (try and) cause the domain to use "target"
amount
> of actual RAM. This may involve calling the libxl low level interfaces,
> including enabled paging and/or sharing as it desires.
>
> It is an actor specific policy whether it guarantees to meet the target
> (e.g. a balloon only actor would not be able to make that guarantee)
>
> An actor might be either be a single daemon with host wide knowledge or
> a daemon per domain (or anything in between). It is expected in the
> multiple daemon case that all the daemons will either co-ordinate as
> necessary or implement a policy where coordination is not necessary.
>
> Normally the same actor would be used for all domains but we do not rule
> out the possibility for non-homogeneous configurations (but it is up to
> the user to make sure the actors don''t have conflicting,
potentially
> catastrophic, interactions...)
>
> XXX I think we need a DOMAIN_INTRODUCE event to allow for a system wide
> actor. This seems like a generally useful event anyhow.
System-wide actors (squeezed) and other actors (xenpaging) already place a
watch on xenstore''s @introduceDomain (or whatever it''s
called). Since this
is not their only use of libxenstore, there is little gain to be made of a
libxl-equivalent event. Referring only to this context.
>
> xl actor configuration
> ----------------------
>
> xl will read its default actor from its configuration file. Since it is
> usually expected to use the same actor for all domains it would normally
> be /etc/xen/xl.conf, but this will be overrideable in a guest config
> file, likewise for the policy params. xl will call
> libxl_domain_set_memory_policy with the configured value while building
> a domain (XXX or use fields in libxl_domain_build_config if we do
> that).
>
> If the configuration key is absent then the default value used is
> "xl" (described below). A user can explicitly supply actor =
"" to get
> the default libxl behaviour.
>
> XXX Config key names TBD, "memory_policy" and
"memory_policy_args"?
>
> xl high level interface
> -----------------------
>
> xl mem-set <target>
>
>         Pretty much same as today. Calls libxl_domain_set_memory_target
>         with enforcing == 1. (but note that the behaviour of
>         libxl_domain_set_memory_target has changed)
>
> xl mem-policy-set <actor> <param-string>
>
>         Calls libxl_domain_set_memory_policy to set the new actor.
>
> "xl" actor behaviour
> --------------------
>
> Each xl daemon instance will listen for DOMAIN_MEMORY_TARGET_CHANGED
> events for the domain which it is managing. When the event is triggered
> (and actor == "xl") then xl will:
>
>         if actor != "xl": return
>
>         call libxl_domain_set_balloon_target(ctx, domid, target,
>         enforcing)
>
>         if domain == pv: return # cannot do more for a PV guest
>
>         XXX if not enforcing: return ??? (not sure about this, we
don''t
>         actually expose the ability to set enforcing in the xl
>         interface)
>
>         XXX if actor-params:paging!=true return ??? I think we should do
>         this to enable memory_policy_params="paging=true|false"
based
>         control of paging.
>
>         delay for a configurable number of seconds (configurable via
>         actor-params:delay=N, default 5s?) XXX does libxl provide a
>         guest accessible timeout event? Would be usefully convenient.
Do you want the guest to try to stop ballooning after the timeout? Just
write /local/domain/X/memory/target to the current value of
xcinfo.nr_pages.
>
>         XXX libxl_domain_paging_enable here or on domain build / policy
>         set? libxl_domain_paging_enable is idempotent so calling it here
>         is always safe?
You can enable the pager on build, unless the actor settings preclude it.
Then all you need to do to get the pager to work is write its xenstore
key.

But generally speaking, the pager might have been disabled ex xl-machina,
so you really want to be able to test_and_set the pager here.

Olaf, does xenpaging have standard daemonize behavior of locking a
/var/foo/pid file?
>
>         call libxl_domain_set_paging_target(ctx, domid, target)
>
> The "xl" actor explicitly does not deal with page sharing. It is
assumed
> that page sharing requires system wide knowledge and policy which is out
> of xl''s scope and hence a different actor would be needed.
>
> xl low level interface
> ----------------------
>
> All of the following setters functions should error out unless actor =>
"xl-manual". The getter could return the current value regardless.
>
> xl mem-balloon-(set|get)-target
> xl mem-paging-(set|get)-target
>
>         Set or get the relevant targets by calling the equivalent libxl
>         function.
>
>         Enables or disables paging as necessary, i.e. target == 0 =>
>         disable
>
> NB if you call "xl mem-set" when actor == "xl-manual"
then it won''t do
> anything since nothing is reacting as that actor.
>
> XXX We could alternatively have xl also implement an "xl-manual"
actor
> which immediately sets both ballooning and paging targets to the given
> value. Probably a good idea?
xl-manual makes sense, but setting targets immediately (e.g.
automatically) is an oxymoron :)
>
> XXX Maybe "xl" (the default) should be called "xl-auto"
?
Or xl-manual called "manual"
>
> If you "xl create -e" (or -F) then you won''t get a
monitoring
> daemon and therefore you won''t get any actor implementing the
memory
> policy, just like you won''t get automatic reboot control. I think
this
> is expected and reasonable.
Ok. There actors that conceivably do not depend on libxl events (squeezed,
it can watch on its own memory-policy/target). On one hand they should not
depend on the xl daemon. On the other hand, they would duplicate
functionality that the xl daemon would have to provide anyways, for
monitoring these memory-policy keys.

Thanks Ian,
Andres
>
>
>

On Thu, 2012-03-15 at 14:12 +0000, Andres Lagar-Cavilla wrote:
[...]
> > libxl_domain_(set|get)_paging_target(ctx, domid, target_memkb,
relative)
> >
> >         Read/write whichever XS path controls the xenpaging
daemon''s
> >         target. (Olaf?). I think this path should not be
> >         under /local/domain since it should not be guest visible
> >         (probably it already isn''t).
> It is /local/domain/X/memory/target-tot_pages. I agree it should not be
> guest visible, and we''re still in time to change it.
> 
> /libxl/X/memory-policy/paging-target? Is there a non-libxl
> non-/local/domain path xenpaging could look into, should it be launched
> autonomously? Olaf, do you care about xenpaging in non-libxl environments?
/xenpaging/N? or /tools/xenpaging/N?

For comparison Roger''s hotplug patches seem to be using /hotplug/...
> >         It is an error to call this for a PV guest (ERROR_??) for a
> >         guest which does not have paging enabled
> >         (ERROR_PAGING_DISABLED?).
> >
> > libxl_domain_(enable|disable)_paging(ctx, domid)
> >
> >         Idempotently starts or stops the xenpaging daemon for a
domain.
> >
> 
> Paging disable takes time. It has to swap back in everything it has paged
> out. So it falls under the (really) long-lasting asynchronous operation
> framework.
Good point, thanks.
> 
> > libxl_domain_(enable|disable)_sharing(ctx, domid)
> >
> >         Does what it says on the tin.
Also long running?
> >
> > actor behaviour
> > ---------------
> >
> > An actor should use the libxl event system to monitor for
> > DOMAIN_MEMORY_TARGET_CHANGED events. An actor must ignore any event
for
> > which actor != their name.
> >
> > On receiving such an event the actor should apply its policy and
> > configuration in order (try and) cause the domain to use
"target" amount
> > of actual RAM. This may involve calling the libxl low level
interfaces,
> > including enabled paging and/or sharing as it desires.
> >
> > It is an actor specific policy whether it guarantees to meet the
target
> > (e.g. a balloon only actor would not be able to make that guarantee)
> >
> > An actor might be either be a single daemon with host wide knowledge
or
> > a daemon per domain (or anything in between). It is expected in the
> > multiple daemon case that all the daemons will either co-ordinate as
> > necessary or implement a policy where coordination is not necessary.
> >
> > Normally the same actor would be used for all domains but we do not
rule
> > out the possibility for non-homogeneous configurations (but it is up
to
> > the user to make sure the actors don''t have conflicting,
potentially
> > catastrophic, interactions...)
> >
> > XXX I think we need a DOMAIN_INTRODUCE event to allow for a system
wide
> > actor. This seems like a generally useful event anyhow.
> 
> System-wide actors (squeezed) and other actors (xenpaging) already place a
> watch on xenstore''s @introduceDomain (or whatever it''s
called). Since this
> is not their only use of libxenstore, there is little gain to be made of a
> libxl-equivalent event. Referring only to this context.
What other use of xenstore do they make? Depending what those are it''s
possible that it may fall under the remit of libxl to provide such
interfaces.

It should be possible for an actor to be only implemented using libxl,
which I think requires us to provide such an event. This doesn''t
preclude some actors from using facilities from other places too though.

[...]
> > "xl" actor behaviour
> > --------------------
> >
> > Each xl daemon instance will listen for DOMAIN_MEMORY_TARGET_CHANGED
> > events for the domain which it is managing. When the event is
triggered
> > (and actor == "xl") then xl will:
> >
> >         if actor != "xl": return
> >
> >         call libxl_domain_set_balloon_target(ctx, domid, target,
> >         enforcing)
> >
> >         if domain == pv: return # cannot do more for a PV guest
> >
> >         XXX if not enforcing: return ??? (not sure about this, we
don''t
> >         actually expose the ability to set enforcing in the xl
> >         interface)
> >
> >         XXX if actor-params:paging!=true return ??? I think we should
do
> >         this to enable
memory_policy_params="paging=true|false" based
> >         control of paging.
> >
> >         delay for a configurable number of seconds (configurable via
> >         actor-params:delay=N, default 5s?) XXX does libxl provide a
> >         guest accessible timeout event? Would be usefully convenient.
> 
> Do you want the guest to try to stop ballooning after the timeout? Just
> write /local/domain/X/memory/target to the current value of
> xcinfo.nr_pages.
Could do, I don''t mind.

Leaving it alone means that a guest which becomes co-operative "too
late" will eventually reduce the amount of paging it is subjected to to
0. Doing as you suggest stops the guest waking up every $PERIOD to try
ballooning a bit more.

Could be an actor-param, although it would be better if xl could just Do
The Right Thing.
> >         XXX libxl_domain_paging_enable here or on domain build /
policy
> >         set? libxl_domain_paging_enable is idempotent so calling it
here
> >         is always safe?
> 
> You can enable the pager on build, unless the actor settings preclude it.
> Then all you need to do to get the pager to work is write its xenstore
> key.
My thinking was that if you boot "unpaged" you might want it to be
enabled when you call mem-set. Likewise if you mem-set such that no
paging is required you might like the daemon to go away. I think the
former is more important than the latter. We don''t really want to run
xenpaging for every HVM domain even when it isn''t used, whereas it is
fine if it persists after it has been used.
> 
> But generally speaking, the pager might have been disabled ex xl-machina,
> so you really want to be able to test_and_set the pager here.
> 
> Olaf, does xenpaging have standard daemonize behavior of locking a
> /var/foo/pid file?
If it is running this should be indicated in the same XS path as we
choose above?
> >         call libxl_domain_set_paging_target(ctx, domid, target)
> >
> > The "xl" actor explicitly does not deal with page sharing.
It is assumed
> > that page sharing requires system wide knowledge and policy which is
out
> > of xl''s scope and hence a different actor would be needed.
> >
> > xl low level interface
> > ----------------------
> >
> > All of the following setters functions should error out unless actor
=> > "xl-manual". The getter could return the current value
regardless.
> >
> > xl mem-balloon-(set|get)-target
> > xl mem-paging-(set|get)-target
> >
> >         Set or get the relevant targets by calling the equivalent
libxl
> >         function.
> >
> >         Enables or disables paging as necessary, i.e. target == 0
=>
> >         disable
> >
> > NB if you call "xl mem-set" when actor ==
"xl-manual" then it won''t do
> > anything since nothing is reacting as that actor.
> >
> > XXX We could alternatively have xl also implement an
"xl-manual" actor
> > which immediately sets both ballooning and paging targets to the given
> > value. Probably a good idea?
> 
> xl-manual makes sense, but setting targets immediately (e.g.
> automatically) is an oxymoron :)
> 
> >
> > XXX Maybe "xl" (the default) should be called
"xl-auto" ?
> 
> Or xl-manual called "manual"
I think namespacing the actors makes sense myself, there''s no harm?

Perhaps <actor>-<policy> would be a good guideline for naming the
thing
we''ve been calling the actor?
> > If you "xl create -e" (or -F) then you won''t get a
monitoring
> > daemon and therefore you won''t get any actor implementing the
memory
> > policy, just like you won''t get automatic reboot control. I
think this
> > is expected and reasonable.
> 
> Ok. There actors that conceivably do not depend on libxl events (squeezed,
> it can watch on its own memory-policy/target). On one hand they should not
> depend on the xl daemon.
Squeezed should eventually be implemented in terms of the mechanisms
discussed here IMHO. This would remove the duplication of mechanism,
leaving only the policy. Of course if the author of an actor wants to
use other mechanisms then fine.

Also bear in mind that the toolstack might not be xl, i.e. xapi should
eventually be ported to use the highlevel libxl interface and would
likely choose to set "squeezed" as its default actor.
>  On the other hand, they would duplicate
> functionality that the xl daemon would have to provide anyways, for
> monitoring these memory-policy keys.
If actor="squeezed" then, even if it is the toolstack, xl will not be
providing any mechanism, since it''s own actor will not do anything when
actor=="squeezed".

Ian.

> On Thu, 2012-03-15 at 14:12 +0000, Andres Lagar-Cavilla wrote:
> [...]
>> > libxl_domain_(set|get)_paging_target(ctx, domid, target_memkb,
>> relative)
>> >
>> >         Read/write whichever XS path controls the xenpaging
daemon''s
>> >         target. (Olaf?). I think this path should not be
>> >         under /local/domain since it should not be guest visible
>> >         (probably it already isn''t).
>> It is /local/domain/X/memory/target-tot_pages. I agree it should not be
>> guest visible, and we''re still in time to change it.
>>
>> /libxl/X/memory-policy/paging-target? Is there a non-libxl
>> non-/local/domain path xenpaging could look into, should it be launched
>> autonomously? Olaf, do you care about xenpaging in non-libxl
>> environments?
>
> /xenpaging/N? or /tools/xenpaging/N?
>
> For comparison Roger''s hotplug patches seem to be using
/hotplug/...
>
>> >         It is an error to call this for a PV guest (ERROR_??) for
a
>> >         guest which does not have paging enabled
>> >         (ERROR_PAGING_DISABLED?).
>> >
>> > libxl_domain_(enable|disable)_paging(ctx, domid)
>> >
>> >         Idempotently starts or stops the xenpaging daemon for a
>> domain.
>> >
>>
>> Paging disable takes time. It has to swap back in everything it has
>> paged
>> out. So it falls under the (really) long-lasting asynchronous operation
>> framework.
>
> Good point, thanks.
>
>>
>> > libxl_domain_(enable|disable)_sharing(ctx, domid)
>> >
>> >         Does what it says on the tin.
>
> Also long running?
Enable is simply setting a flag. Disable is not a problem either: the
hypervisor will silently take care of unshares.

However, if there is an ENOMEM in an unshare, someone has to be listening
or the domain will crash. It could be that the xl daemon is tasked with
this for the lifetime of the domain, once sharing has been enabled at
least once. I don''t think it''s something to decide upon right
now.

Andres
>
>> >
>> > actor behaviour
>> > ---------------
>> >
>> > An actor should use the libxl event system to monitor for
>> > DOMAIN_MEMORY_TARGET_CHANGED events. An actor must ignore any
event
>> for
>> > which actor != their name.
>> >
>> > On receiving such an event the actor should apply its policy and
>> > configuration in order (try and) cause the domain to use
"target"
>> amount
>> > of actual RAM. This may involve calling the libxl low level
>> interfaces,
>> > including enabled paging and/or sharing as it desires.
>> >
>> > It is an actor specific policy whether it guarantees to meet the
>> target
>> > (e.g. a balloon only actor would not be able to make that
guarantee)
>> >
>> > An actor might be either be a single daemon with host wide
knowledge
>> or
>> > a daemon per domain (or anything in between). It is expected in
the
>> > multiple daemon case that all the daemons will either co-ordinate
as
>> > necessary or implement a policy where coordination is not
necessary.
>> >
>> > Normally the same actor would be used for all domains but we do
not
>> rule
>> > out the possibility for non-homogeneous configurations (but it is
up
>> to
>> > the user to make sure the actors don''t have conflicting,
potentially
>> > catastrophic, interactions...)
>> >
>> > XXX I think we need a DOMAIN_INTRODUCE event to allow for a system
>> wide
>> > actor. This seems like a generally useful event anyhow.
>>
>> System-wide actors (squeezed) and other actors (xenpaging) already
place
>> a
>> watch on xenstore''s @introduceDomain (or whatever
it''s called). Since
>> this
>> is not their only use of libxenstore, there is little gain to be made
of
>> a
>> libxl-equivalent event. Referring only to this context.
>
> What other use of xenstore do they make? Depending what those are
it''s
> possible that it may fall under the remit of libxl to provide such
> interfaces.
>
> It should be possible for an actor to be only implemented using libxl,
> which I think requires us to provide such an event. This doesn''t
> preclude some actors from using facilities from other places too though.
>
> [...]
>> > "xl" actor behaviour
>> > --------------------
>> >
>> > Each xl daemon instance will listen for
DOMAIN_MEMORY_TARGET_CHANGED
>> > events for the domain which it is managing. When the event is
>> triggered
>> > (and actor == "xl") then xl will:
>> >
>> >         if actor != "xl": return
>> >
>> >         call libxl_domain_set_balloon_target(ctx, domid, target,
>> >         enforcing)
>> >
>> >         if domain == pv: return # cannot do more for a PV guest
>> >
>> >         XXX if not enforcing: return ??? (not sure about this, we
>> don''t
>> >         actually expose the ability to set enforcing in the xl
>> >         interface)
>> >
>> >         XXX if actor-params:paging!=true return ??? I think we
should
>> do
>> >         this to enable
memory_policy_params="paging=true|false" based
>> >         control of paging.
>> >
>> >         delay for a configurable number of seconds (configurable
via
>> >         actor-params:delay=N, default 5s?) XXX does libxl provide
a
>> >         guest accessible timeout event? Would be usefully
convenient.
>>
>> Do you want the guest to try to stop ballooning after the timeout? Just
>> write /local/domain/X/memory/target to the current value of
>> xcinfo.nr_pages.
>
> Could do, I don''t mind.
>
> Leaving it alone means that a guest which becomes co-operative "too
> late" will eventually reduce the amount of paging it is subjected to
to
> 0. Doing as you suggest stops the guest waking up every $PERIOD to try
> ballooning a bit more.
>
> Could be an actor-param, although it would be better if xl could just Do
> The Right Thing.
>
>> >         XXX libxl_domain_paging_enable here or on domain build /
>> policy
>> >         set? libxl_domain_paging_enable is idempotent so calling
it
>> here
>> >         is always safe?
>>
>> You can enable the pager on build, unless the actor settings preclude
>> it.
>> Then all you need to do to get the pager to work is write its xenstore
>> key.
>
> My thinking was that if you boot "unpaged" you might want it to
be
> enabled when you call mem-set. Likewise if you mem-set such that no
> paging is required you might like the daemon to go away. I think the
> former is more important than the latter. We don''t really want to
run
> xenpaging for every HVM domain even when it isn''t used, whereas it
is
> fine if it persists after it has been used.
>
>>
>> But generally speaking, the pager might have been disabled ex
>> xl-machina,
>> so you really want to be able to test_and_set the pager here.
>>
>> Olaf, does xenpaging have standard daemonize behavior of locking a
>> /var/foo/pid file?
>
> If it is running this should be indicated in the same XS path as we
> choose above?
>
>> >         call libxl_domain_set_paging_target(ctx, domid, target)
>> >
>> > The "xl" actor explicitly does not deal with page
sharing. It is
>> assumed
>> > that page sharing requires system wide knowledge and policy which
is
>> out
>> > of xl''s scope and hence a different actor would be
needed.
>> >
>> > xl low level interface
>> > ----------------------
>> >
>> > All of the following setters functions should error out unless
actor
>> =>> > "xl-manual". The getter could return the
current value regardless.
>> >
>> > xl mem-balloon-(set|get)-target
>> > xl mem-paging-(set|get)-target
>> >
>> >         Set or get the relevant targets by calling the equivalent
>> libxl
>> >         function.
>> >
>> >         Enables or disables paging as necessary, i.e. target == 0
=>
>> >         disable
>> >
>> > NB if you call "xl mem-set" when actor ==
"xl-manual" then it won''t do
>> > anything since nothing is reacting as that actor.
>> >
>> > XXX We could alternatively have xl also implement an
"xl-manual" actor
>> > which immediately sets both ballooning and paging targets to the
given
>> > value. Probably a good idea?
>>
>> xl-manual makes sense, but setting targets immediately (e.g.
>> automatically) is an oxymoron :)
>>
>> >
>> > XXX Maybe "xl" (the default) should be called
"xl-auto" ?
>>
>> Or xl-manual called "manual"
>
> I think namespacing the actors makes sense myself, there''s no
harm?
>
> Perhaps <actor>-<policy> would be a good guideline for naming
the thing
> we''ve been calling the actor?
>
>> > If you "xl create -e" (or -F) then you won''t
get a monitoring
>> > daemon and therefore you won''t get any actor implementing
the memory
>> > policy, just like you won''t get automatic reboot control.
I think this
>> > is expected and reasonable.
>>
>> Ok. There actors that conceivably do not depend on libxl events
>> (squeezed,
>> it can watch on its own memory-policy/target). On one hand they should
>> not
>> depend on the xl daemon.
>
> Squeezed should eventually be implemented in terms of the mechanisms
> discussed here IMHO. This would remove the duplication of mechanism,
> leaving only the policy. Of course if the author of an actor wants to
> use other mechanisms then fine.
>
> Also bear in mind that the toolstack might not be xl, i.e. xapi should
> eventually be ported to use the highlevel libxl interface and would
> likely choose to set "squeezed" as its default actor.
>
>>  On the other hand, they would duplicate
>> functionality that the xl daemon would have to provide anyways, for
>> monitoring these memory-policy keys.
>
> If actor="squeezed" then, even if it is the toolstack, xl will
not be
> providing any mechanism, since it''s own actor will not do anything
when
> actor=="squeezed".
>
> Ian.
>
>
>

On Thu, 2012-03-15 at 14:40 +0000, Andres Lagar-Cavilla
wrote:
> > On Thu, 2012-03-15 at 14:12 +0000, Andres Lagar-Cavilla wrote:
> > [...]
> >>
> >> > libxl_domain_(enable|disable)_sharing(ctx, domid)
> >> >
> >> >         Does what it says on the tin.
> >
> > Also long running?
> 
> Enable is simply setting a flag. Disable is not a problem either: the
> hypervisor will silently take care of unshares.
> 
> However, if there is an ENOMEM in an unshare, someone has to be listening
> or the domain will crash. It could be that the xl daemon is tasked with
> this for the lifetime of the domain, once sharing has been enabled at
> least once. I don''t think it''s something to decide upon
right now.
xl has explicitly decided not to care about sharing so this would be the
problem of whichever actor does do sharing...

Given the assumption that a sharing-actor would have to be system wide
perhaps we don''t need these libxl functions to start stop it (since it
will be running already) and the very act (no pun intended) of setting
actor="blah-sharing" should be enough to enable or disable sharing for
a
particular guest?

Ian.

> On Thu, 2012-03-15 at 14:40 +0000, Andres Lagar-Cavilla wrote:
>> > On Thu, 2012-03-15 at 14:12 +0000, Andres Lagar-Cavilla wrote:
>> > [...]
>> >>
>> >> > libxl_domain_(enable|disable)_sharing(ctx, domid)
>> >> >
>> >> >         Does what it says on the tin.
>> >
>> > Also long running?
>>
>> Enable is simply setting a flag. Disable is not a problem either: the
>> hypervisor will silently take care of unshares.
>>
>> However, if there is an ENOMEM in an unshare, someone has to be
>> listening
>> or the domain will crash. It could be that the xl daemon is tasked with
>> this for the lifetime of the domain, once sharing has been enabled at
>> least once. I don''t think it''s something to decide
upon right now.
>
> xl has explicitly decided not to care about sharing so this would be the
> problem of whichever actor does do sharing...
>
> Given the assumption that a sharing-actor would have to be system wide
> perhaps we don''t need these libxl functions to start stop it
(since it
> will be running already) and the very act (no pun intended) of setting
> actor="blah-sharing" should be enough to enable or disable
sharing for a
> particular guest?
Both points are sensible. Ack from me.
Andres
>
> Ian.
>
>
>

On Thu, Mar 15, Ian Campbell wrote:

> low level libxl interface
> -------------------------
> 
> It is expected that toolstacks primary user interface will be via the
> above high level interface. The lowlevel interface is provided in order
> to allow implementation of memory policy actors using libxl and to allow
> toolstacks to optionally provide an interface for expert users to shoot
> themselves in the foot with ( ;-) )
> 
> libxl_domain_(set|get)_balloon_target(ctx, domid, target_memkb, relative,
enforce):
>         
>         Read/write /local/domain/X/memory/target to control the in guest
>         balloon driver. This is basically the existing
>         libxl_domain_memory_target with a new name
>                 
>         As before this is a request to the guest balloon driver to try
>         and use this much actual RAM. This is an existing guest
>         interface which we cannot change.
> 
> libxl_domain_(set|get)_paging_target(ctx, domid, target_memkb, relative)
> 
>         Read/write whichever XS path controls the xenpaging
daemon''s
>         target. (Olaf?). I think this path should not be
>         under /local/domain since it should not be guest visible
>         (probably it already isn''t).
Its currently /local/domain/X/memory/target-tot_pages, but can still be
changed if needed.
>         It is an error to call this for a PV guest (ERROR_??) for a
>         guest which does not have paging enabled
>         (ERROR_PAGING_DISABLED?).
Error is ok.
> libxl_domain_(enable|disable)_paging(ctx, domid)
> 
>         Idempotently starts or stops the xenpaging daemon for a domain.
In what context should this run?
In case of ''xl $enable_paging'' it would be good if this acts
as a
trigger for the xl monitoring process, so that the pager is a child of
the monitoring process.

Olaf

On Thu, Mar 15, Andres Lagar-Cavilla wrote:

> > libxl_domain_(set|get)_paging_target(ctx, domid, target_memkb,
relative)
> >
> >         Read/write whichever XS path controls the xenpaging
daemon''s
> >         target. (Olaf?). I think this path should not be
> >         under /local/domain since it should not be guest visible
> >         (probably it already isn''t).
> It is /local/domain/X/memory/target-tot_pages. I agree it should not be
> guest visible, and we''re still in time to change it.
Is it guest visible? I think the entry point for a guest is /vm/<uuid>/,
I''m almost sure it can not get to /local/domain/X/memory/
> /libxl/X/memory-policy/paging-target? Is there a non-libxl
> non-/local/domain path xenpaging could look into, should it be launched
> autonomously? Olaf, do you care about xenpaging in non-libxl environments?
xenpaging needs to read its target from somewhere, I dont think this has
anything todo with libxl or not.
> >         XXX libxl_domain_paging_enable here or on domain build /
policy
> >         set? libxl_domain_paging_enable is idempotent so calling it
here
> >         is always safe?
> 
> You can enable the pager on build, unless the actor settings preclude it.
> Then all you need to do to get the pager to work is write its xenstore
> key.
> 
> But generally speaking, the pager might have been disabled ex xl-machina,
> so you really want to be able to test_and_set the pager here.
> 
> Olaf, does xenpaging have standard daemonize behavior of locking a
> /var/foo/pid file?
It writes its pid to xenstore and announces its state in
xenpaging/state, like qemu-dm does. Thats in my libxl support patch at
least.

Olaf

On Thu, 2012-03-15 at 15:14 +0000, Olaf Hering wrote:
> > libxl_domain_(enable|disable)_paging(ctx, domid)
> > 
> >         Idempotently starts or stops the xenpaging daemon for a
domain.
> 
> In what context should this run?
> In case of ''xl $enable_paging'' it would be good if this
acts as a
> trigger for the xl monitoring process, so that the pager is a child of
> the monitoring process.
I suggested calling it from the actor callback, but calling it from the
build process would be ok too -- in both cases this would be in the
context of the monitoring process.

Ian.

On Thu, 2012-03-15 at 15:26 +0000, Olaf Hering wrote:
> On Thu, Mar 15, Andres Lagar-Cavilla wrote:
> 
> 
> > > libxl_domain_(set|get)_paging_target(ctx, domid, target_memkb,
relative)
> > >
> > >         Read/write whichever XS path controls the xenpaging
daemon''s
> > >         target. (Olaf?). I think this path should not be
> > >         under /local/domain since it should not be guest visible
> > >         (probably it already isn''t).
> > It is /local/domain/X/memory/target-tot_pages. I agree it should not
be
> > guest visible, and we''re still in time to change it.
> 
> Is it guest visible? I think the entry point for a guest is
/vm/<uuid>/,
> I''m almost sure it can not get to /local/domain/X/memory/
/local/domain/X is generally considered to be part of the "guest
visible" state and there are various guest visible things there,
devices/* and memory/target spring to mind.
> > /libxl/X/memory-policy/paging-target? Is there a non-libxl
> > non-/local/domain path xenpaging could look into, should it be
launched
> > autonomously? Olaf, do you care about xenpaging in non-libxl
environments?
> 
> xenpaging needs to read its target from somewhere, I dont think this has
> anything todo with libxl or not.
Except that libxl needs to know where to write to.
> > >         XXX libxl_domain_paging_enable here or on domain build /
policy
> > >         set? libxl_domain_paging_enable is idempotent so calling
it here
> > >         is always safe?
> > 
> > You can enable the pager on build, unless the actor settings preclude
it.
> > Then all you need to do to get the pager to work is write its xenstore
> > key.
> > 
> > But generally speaking, the pager might have been disabled ex
xl-machina,
> > so you really want to be able to test_and_set the pager here.
> > 
> > Olaf, does xenpaging have standard daemonize behavior of locking a
> > /var/foo/pid file?
> 
> It writes its pid to xenstore and announces its state in
> xenpaging/state, like qemu-dm does. Thats in my libxl support patch at
> least.
This sounds good to me.

On Thu, Mar 15, Ian Campbell wrote:
> On Thu, 2012-03-15 at 15:26 +0000, Olaf Hering wrote:
> > On Thu, Mar 15, Andres Lagar-Cavilla wrote:
> > 
> > 
> > > > libxl_domain_(set|get)_paging_target(ctx, domid,
target_memkb, relative)
> > > >
> > > >         Read/write whichever XS path controls the xenpaging
daemon''s
> > > >         target. (Olaf?). I think this path should not be
> > > >         under /local/domain since it should not be guest
visible
> > > >         (probably it already isn''t).
> > > It is /local/domain/X/memory/target-tot_pages. I agree it should
not be
> > > guest visible, and we''re still in time to change it.
> > 
> > Is it guest visible? I think the entry point for a guest is
/vm/<uuid>/,
> > I''m almost sure it can not get to /local/domain/X/memory/
> 
> /local/domain/X is generally considered to be part of the "guest
> visible" state and there are various guest visible things there,
> devices/* and memory/target spring to mind.
Is memory/target below /local/domain/X or /vm/<uuid> in use by the guet?
I''m sure the latter can only be used to set a watch for example (dont
have a guest at hand to verify).
Also the guest would have to proactivly parse the "links" to find its
domid, to later browse /local/domain/X.
What would it protect if /local/domain/X/memory (and other parts) are not
accesible?
> > > /libxl/X/memory-policy/paging-target? Is there a non-libxl
> > > non-/local/domain path xenpaging could look into, should it be
launched
> > > autonomously? Olaf, do you care about xenpaging in non-libxl
environments?
> > 
> > xenpaging needs to read its target from somewhere, I dont think this
has
> > anything todo with libxl or not.
> 
> Except that libxl needs to know where to write to.
Sure.

Olaf

On Thu, 2012-03-15 at 15:42 +0000, Olaf Hering wrote:
> On Thu, Mar 15, Ian Campbell wrote:
> 
> > On Thu, 2012-03-15 at 15:26 +0000, Olaf Hering wrote:
> > > On Thu, Mar 15, Andres Lagar-Cavilla wrote:
> > > 
> > > 
> > > > > libxl_domain_(set|get)_paging_target(ctx, domid,
target_memkb, relative)
> > > > >
> > > > >         Read/write whichever XS path controls the
xenpaging daemon''s
> > > > >         target. (Olaf?). I think this path should not
be
> > > > >         under /local/domain since it should not be
guest visible
> > > > >         (probably it already isn''t).
> > > > It is /local/domain/X/memory/target-tot_pages. I agree it
should not be
> > > > guest visible, and we''re still in time to change
it.
> > > 
> > > Is it guest visible? I think the entry point for a guest is
/vm/<uuid>/,
> > > I''m almost sure it can not get to
/local/domain/X/memory/
> > 
> > /local/domain/X is generally considered to be part of the "guest
> > visible" state and there are various guest visible things there,
> > devices/* and memory/target spring to mind.
> 
> Is memory/target below /local/domain/X or /vm/<uuid> in use by the
guet?
/local/domain/X. The is no such key under /vm/<uuid>
> I''m sure the latter can only be used to set a watch for example
(dont
> have a guest at hand to verify).
> Also the guest would have to proactivly parse the "links" to find
its
> domid, to later browse /local/domain/X.
No, /local/domain/X is the domains "home directory", it is what
relative
paths are relative to -- so to access /local/domain/X/memory/target a
domain just accesses "memory/target"
> What would it protect if /local/domain/X/memory (and other parts) are not
accesible?
XS has a permissions scheme which allows per domain r/w, r/o or none.

Something like the paging target should not even be r/o for the guest
IMHO. In general we try to keep stuff which the domain should not even
be looking at separate from the stuff which it should.

Ian.

On Thu, Mar 15, Ian Campbell wrote:

> > Is memory/target below /local/domain/X or /vm/<uuid> in use by
the guet?
> 
> /local/domain/X. The is no such key under /vm/<uuid>
Its just "/vm/<uuid>/memory", after ''xm mem-set dom
2G''. This what I had
in mind.
> > I''m sure the latter can only be used to set a watch for
example (dont
> > have a guest at hand to verify).
> > Also the guest would have to proactivly parse the "links" to
find its
> > domid, to later browse /local/domain/X.
> 
> No, /local/domain/X is the domains "home directory", it is what
relative
> paths are relative to -- so to access /local/domain/X/memory/target a
> domain just accesses "memory/target"
Hmm, really?
When I worked on xenbus_reset_state() for kdump the guest started in
/vm/<uuid> for relative paths.
> > What would it protect if /local/domain/X/memory (and other parts) are
not accesible?
> 
> XS has a permissions scheme which allows per domain r/w, r/o or none.
> 
> Something like the paging target should not even be r/o for the guest
> IMHO. In general we try to keep stuff which the domain should not even
> be looking at separate from the stuff which it should.
The target could very well go below /local/domain/X/xenpaging, and this
directory can get the proper permissions to be r/w only for the tools.

Olaf

On Thu, 2012-03-15 at 16:00 +0000, Olaf Hering wrote:
> On Thu, Mar 15, Ian Campbell wrote:
> 
> 
> > > Is memory/target below /local/domain/X or /vm/<uuid> in use
by the guet?
> > 
> > /local/domain/X. The is no such key under /vm/<uuid>
> 
> Its just "/vm/<uuid>/memory", after ''xm mem-set
dom 2G''. This what I had
> in mind.
# xenstore-ls -fp| grep memory
/local/domain/0/memory = ""   (n0)
/local/domain/0/memory/target = "524288"   (n0)
/local/domain/0/memory/static-max = "4294967292"   (n0)
/local/domain/0/memory/freemem-slack = "629049"   (n0)
/local/domain/32/memory = ""   (n0,r32)
/local/domain/32/memory/static-max = "524288"   (n0,r32)
/local/domain/32/memory/target = "516096"   (n0,r32)
/local/domain/32/memory/videoram = "8192"   (n0,r32)
/local/domain/33/memory = ""   (n0,r33)
/local/domain/33/memory/static-max = "32768"   (n0,r33)
/local/domain/33/memory/target = "32769"   (n0,r33)
/local/domain/33/memory/videoram = "-1"   (n0,r33)
/local/domain/47/memory = ""   (n0,r47)
/local/domain/47/memory/static-max = "524288"   (n0,r47)
/local/domain/47/memory/target = "516096"   (n0,r47)
/local/domain/47/memory/videoram = "8192"   (n0,r47)
#
> 
> > > I''m sure the latter can only be used to set a watch for
example (dont
> > > have a guest at hand to verify).
> > > Also the guest would have to proactivly parse the
"links" to find its
> > > domid, to later browse /local/domain/X.
> > 
> > No, /local/domain/X is the domains "home directory", it is
what relative
> > paths are relative to -- so to access /local/domain/X/memory/target a
> > domain just accesses "memory/target"
> 
> Hmm, really?
I am reasonably sure.
# xenstore-ls -fp| grep ijc-foo
# xenstore-write ijc-foo/bar baz
# xenstore-ls -fp| grep ijc-foo
/local/domain/0/ijc-foo = ""   (n0)
/local/domain/0/ijc-foo/bar = "baz"   (n0)
#

See also tools/xenstore/xenstored_domain.c:talloc_domain_path() which
creates the path returned by get_implicit_path():
        static char *talloc_domain_path(void *context, unsigned int
        domid)
        {
                return talloc_asprintf(context, "/local/domain/%u",
        domid);
        }
> When I worked on xenbus_reset_state() for kdump the guest started in
> /vm/<uuid> for relative paths.
> 
> > > What would it protect if /local/domain/X/memory (and other parts)
are not accesible?
> > 
> > XS has a permissions scheme which allows per domain r/w, r/o or none.
> > 
> > Something like the paging target should not even be r/o for the guest
> > IMHO. In general we try to keep stuff which the domain should not even
> > be looking at separate from the stuff which it should.
> 
> The target could very well go below /local/domain/X/xenpaging, and this
> directory can get the proper permissions to be r/w only for the tools.
My point was that but is much better to segregate non-guest visible
stuff outside of this path in order to avoid mistake.

Things have a tendency to leak into the guest visible world and become
an ABI otherwise.

Ian.

Ian Campbell writes ("libxl/xl memory paging/sharing/ballooning/etc
proposal."):
> I think we''ve roughly reached consensus on what this could look
like.
> I''ve combined the various aspects of the proposal into the below.
This looks reasonable to me.
> Ian J -- I''ve CC''d you since I propose some new uses of
the events
> subsystem (as well as being interested in your overall opinion)
That part looks OK.

I have a slight quibble with the structure of the document which is
that it mixes up description of interface with implementation.
> libxl_domain_set_memory_policy(ctx, domid, const char *actor, const char
*params):
>         
>         Sets the name of the "actor" which will implement memory
policy
>         for this domain by writing the name of the actor to
>                 /libxl/X/memory-policy/actor
>         and the params to
>                 /libxl/X/memory-policy/actor-params
>                         
>         actor-params is defined per-actor. normally be a comma separated
>         key=value list.
> 
>         If libxl_domain_set_memory_policy is not called for a domain
>         then the default is considered to be "" i.e. none.
Does this mean that any actor is required to be able to undo the
effects of any other actor ?  Eg, if I switch from paging to
ballooning, is the paging actor required to un-balloon the domain ?
>         XXX should the actor config also be part of struct
>         libxl_domain_build_info? (I expect so)
Certainly.
> libxl_domain_set_memory_target(ctx, domid, target_memkb, bool relative,
> bool enforce):
> 
>         Replaces existing libxl_domain_memory_target() function.
>                 
>         If memory-policy/actor != "":
>                 updates /libxl/X/memory-policy/<target,enforce>
>         else:
>                 behaves as libxl_domain_memory_target today, e.g. sets
>                 balloon target in xenstore and iff enforcing then sets
>                 maxmem -- probably this is just a call to
>                 libxl_domain_set_balloon_target() (see below)
It should be explicitly stated that a toolstack is allowed to
integrate the actor and target-setter functionality, in which case the
actor is allowed to ignore the supposed target.  Ie the use of the
target is optional.
> XXX I think we need a DOMAIN_INTRODUCE event to allow for a system wide
> actor. This seems like a generally useful event anyhow.
Yes.  We need also to make sure that if you ask for domain death
events for an already-nonexistent domain you get either an error or an
event, which I''m not sure is currently the case.
>         XXX if not enforcing: return ??? (not sure about this, we
don''t
>         actually expose the ability to set enforcing in the xl
>         interface)
I think we should abolish the separate "enforce" parameter and make it
a parameter to the xl actor.  It doesn''t make much sense to vary the
setting per-update-event.

(Well really I think it should be abolished and by pretending it''s
always true, but the last N times we had this conversation people
disagreed.)
> xl low level interface
> ----------------------
> 
> All of the following setters functions should error out unless actor =>
"xl-manual". The getter could return the current value regardless.
> 
> xl mem-balloon-(set|get)-target
> xl mem-paging-(set|get)-target
> 
>         Set or get the relevant targets by calling the equivalent libxl
>         function. 
>         
>         Enables or disables paging as necessary, i.e. target == 0 =>
>         disable
>         
> NB if you call "xl mem-set" when actor == "xl-manual"
then it won''t do
> anything since nothing is reacting as that actor.
Right.
> XXX We could alternatively have xl also implement an "xl-manual"
actor
> which immediately sets both ballooning and paging targets to the given
> value. Probably a good idea?
That would be a bad algorithm because it would start thrashing to
page-out-in-the-guest pages which have been paged-out-by-xenpaging.
> XXX Maybe "xl" (the default) should be called "xl-auto"
?
I don''t have an opinion about this.

But "" seems like an odd default value, so perhaps it should be
explict "none" ?

Ian.

> Ian Campbell writes ("libxl/xl memory paging/sharing/ballooning/etc
> proposal."):
>> I think we''ve roughly reached consensus on what this could
look like.
>> I''ve combined the various aspects of the proposal into the
below.
>
> This looks reasonable to me.
>
>> Ian J -- I''ve CC''d you since I propose some new uses
of the events
>> subsystem (as well as being interested in your overall opinion)
>
> That part looks OK.
>
> I have a slight quibble with the structure of the document which is
> that it mixes up description of interface with implementation.
>
>> libxl_domain_set_memory_policy(ctx, domid, const char *actor, const
char
>> *params):
>>
>>         Sets the name of the "actor" which will implement
memory policy
>>         for this domain by writing the name of the actor to
>>                 /libxl/X/memory-policy/actor
>>         and the params to
>>                 /libxl/X/memory-policy/actor-params
>>
>>         actor-params is defined per-actor. normally be a comma
separated
>>         key=value list.
>>
>>         If libxl_domain_set_memory_policy is not called for a domain
>>         then the default is considered to be "" i.e. none.
>
> Does this mean that any actor is required to be able to undo the
> effects of any other actor ?  Eg, if I switch from paging to
> ballooning, is the paging actor required to un-balloon the domain ?
I hope once an actor is set it cannot be undone for the lifetime of the
domain.

If that sounds too rigid, then once an actor is torn down, it is
responsible for bringing things back to where they were. xenpaging does
this. And the xl default policy can easily reset the balloon to the
original target (i.e. no ballooned pages).

The memory-policy/target parameter should also be reset when an actor is
torn down.
>
>>         XXX should the actor config also be part of struct
>>         libxl_domain_build_info? (I expect so)
>
> Certainly.
>
>> libxl_domain_set_memory_target(ctx, domid, target_memkb, bool relative,
>> bool enforce):
>>
>>         Replaces existing libxl_domain_memory_target() function.
>>
>>         If memory-policy/actor != "":
>>                 updates /libxl/X/memory-policy/<target,enforce>
>>         else:
>>                 behaves as libxl_domain_memory_target today, e.g. sets
>>                 balloon target in xenstore and iff enforcing then sets
>>                 maxmem -- probably this is just a call to
>>                 libxl_domain_set_balloon_target() (see below)
>
> It should be explicitly stated that a toolstack is allowed to
> integrate the actor and target-setter functionality, in which case the
> actor is allowed to ignore the supposed target.  Ie the use of the
> target is optional.
>
>> XXX I think we need a DOMAIN_INTRODUCE event to allow for a system wide
>> actor. This seems like a generally useful event anyhow.
>
> Yes.  We need also to make sure that if you ask for domain death
> events for an already-nonexistent domain you get either an error or an
> event, which I''m not sure is currently the case.
>
>>         XXX if not enforcing: return ??? (not sure about this, we
don''t
>>         actually expose the ability to set enforcing in the xl
>>         interface)
>
> I think we should abolish the separate "enforce" parameter and
make it
> a parameter to the xl actor.  It doesn''t make much sense to vary
the
> setting per-update-event.
>
> (Well really I think it should be abolished and by pretending it''s
> always true, but the last N times we had this conversation people
> disagreed.)
>
>> xl low level interface
>> ----------------------
>>
>> All of the following setters functions should error out unless actor
=>> "xl-manual". The getter could return the current value
regardless.
>>
>> xl mem-balloon-(set|get)-target
>> xl mem-paging-(set|get)-target
>>
>>         Set or get the relevant targets by calling the equivalent libxl
>>         function.
>>
>>         Enables or disables paging as necessary, i.e. target == 0 =>
>>         disable
>>
>> NB if you call "xl mem-set" when actor ==
"xl-manual" then it won''t do
>> anything since nothing is reacting as that actor.
>
> Right.
>
>> XXX We could alternatively have xl also implement an
"xl-manual" actor
>> which immediately sets both ballooning and paging targets to the given
>> value. Probably a good idea?
>
> That would be a bad algorithm because it would start thrashing to
> page-out-in-the-guest pages which have been paged-out-by-xenpaging.
Depends on whether xenpaging had been invoked previously. But I agree on
not immediately enforcing any values. If it''s manual, then let the user
manually set all the knobs.

Andres
>
>> XXX Maybe "xl" (the default) should be called
"xl-auto" ?
>
> I don''t have an opinion about this.
>
> But "" seems like an odd default value, so perhaps it should be
> explict "none" ?
>
> Ian.
>

On Mon, 2012-03-19 at 16:10 +0000, Ian Jackson wrote:
> Ian Campbell writes ("libxl/xl memory paging/sharing/ballooning/etc
proposal."):
> > I think we''ve roughly reached consensus on what this could
look like.
> > I''ve combined the various aspects of the proposal into the
below.
> 
> This looks reasonable to me.
> 
> > Ian J -- I''ve CC''d you since I propose some new uses
of the events
> > subsystem (as well as being interested in your overall opinion)
> 
> That part looks OK.
> 
> I have a slight quibble with the structure of the document which is
> that it mixes up description of interface with implementation.
> 
> > libxl_domain_set_memory_policy(ctx, domid, const char *actor, const
char *params):
> >         
> >         Sets the name of the "actor" which will implement
memory policy
> >         for this domain by writing the name of the actor to
> >                 /libxl/X/memory-policy/actor
> >         and the params to
> >                 /libxl/X/memory-policy/actor-params
> >                         
> >         actor-params is defined per-actor. normally be a comma
separated
> >         key=value list.
> > 
> >         If libxl_domain_set_memory_policy is not called for a domain
> >         then the default is considered to be "" i.e. none.
> 
> Does this mean that any actor is required to be able to undo the
> effects of any other actor ?  Eg, if I switch from paging to
> ballooning, is the paging actor required to un-balloon the domain ?
Good question. It seems unreasonable that a balloon actor need to know
about paging.

I think what we really want is some sort of "you are being disabled"
event sent to the previous actor but I''m not at all sure how that would
be represented in the xenstore interface, nor how the interlock with the
new one would work. There''s also the issue of what happens if e.g.
there
isn''t enough memory to undo the work the pager has done.

Big ol'' can-o-worms I think.

We could fudge the issue by removing this function and only exposing
this functionality via build_info. This would require a guest reboot
(or, I suppose, a migration) to change the actor. We''d probably want to
leave the ability to change the actor params in the API though.
> > libxl_domain_set_memory_target(ctx, domid, target_memkb, bool
relative,
> > bool enforce):
> > 
> >         Replaces existing libxl_domain_memory_target() function.
> >                 
> >         If memory-policy/actor != "":
> >                 updates /libxl/X/memory-policy/<target,enforce>
> >         else:
> >                 behaves as libxl_domain_memory_target today, e.g. sets
> >                 balloon target in xenstore and iff enforcing then sets
> >                 maxmem -- probably this is just a call to
> >                 libxl_domain_set_balloon_target() (see below)
> 
> It should be explicitly stated that a toolstack is allowed to
> integrate the actor and target-setter functionality, in which case the
> actor is allowed to ignore the supposed target.  Ie the use of the
> target is optional.
I''m not sure what you mean (there are, confusingly, two targets in the
interface) so I''m not sure what this (presumed) flexibility buys us.

(I have a feeling this ties into the "xl-manual" discussion below)
> > XXX I think we need a DOMAIN_INTRODUCE event to allow for a system
wide
> > actor. This seems like a generally useful event anyhow.
> 
> Yes.  We need also to make sure that if you ask for domain death
> events for an already-nonexistent domain you get either an error or an
> event, which I''m not sure is currently the case.
Right, that would be a bug fix to the existing stuff, right?
> 
> >         XXX if not enforcing: return ??? (not sure about this, we
don''t
> >         actually expose the ability to set enforcing in the xl
> >         interface)
> 
> I think we should abolish the separate "enforce" parameter and
make it
> a parameter to the xl actor.  It doesn''t make much sense to vary
the
> setting per-update-event.
> 
> (Well really I think it should be abolished and by pretending it''s
> always true, but the last N times we had this conversation people
> disagreed.)
I agree with you FWIW.

I think can say that it is up to actor policy if and how it enforces
things (configurable as desired via params) but that xl always forces
the target (like it does today). 

This still leaves the flexibility in place for those who disagree to do
what they need.
> 
> > xl low level interface
> > ----------------------
> > 
> > All of the following setters functions should error out unless actor
=> > "xl-manual". The getter could return the current value
regardless.
> > 
> > xl mem-balloon-(set|get)-target
> > xl mem-paging-(set|get)-target
> > 
> >         Set or get the relevant targets by calling the equivalent
libxl
> >         function. 
> >         
> >         Enables or disables paging as necessary, i.e. target == 0
=>
> >         disable
> >         
> > NB if you call "xl mem-set" when actor ==
"xl-manual" then it won''t do
> > anything since nothing is reacting as that actor.
> 
> Right.
> 
> > XXX We could alternatively have xl also implement an
"xl-manual" actor
> > which immediately sets both ballooning and paging targets to the given
> > value. Probably a good idea?
> 
> That would be a bad algorithm because it would start thrashing to
> page-out-in-the-guest pages which have been paged-out-by-xenpaging.
Yes. It would be expected that you would use the low level
mem-{balloon,paging}-{set,get}-target functions if you set the actor to
manual. Calling mem-set in manual mode is more of a "please shoot me in
the foot thing". mem-set could print an error message if actor =xl-manual.
Or do you have an idea for a better algorithm for the manual
case? Setting only balloon target could be one alternative.
> > XXX Maybe "xl" (the default) should be called
"xl-auto" ?
> 
> I don''t have an opinion about this.
> 
> But "" seems like an odd default value, so perhaps it should be
> explict "none" ?
Makes sense.

Ian.