Xen devel - Jan 2013 - [RFC PATCH 0/3] Improve kexec support in Xen hypervisor

This series of patches improves the kexec hypercall in the Xen
hypervisor.  It is an incomplete prototype but I posting it early for
comments on the proposed ABI/API.

This allows a privileged Xen guest to load kexec images into the
hypervisor from a userspace tool without using the Linux kernel''s
kexec subsystem.  It is the first step to supporting kexec of crash
kernels from a pv-ops dom0 kernel (the required kernel and kexec-tools
patches will be posted later).

The kernel will require a kexec hypercall somewhere in the
crash_kexec() path to actually exec the loaded image.  Any preferences
on how the hook for this should be implemented?  Note that the kernel
won''t be aware that an image as been loaded as it is loaded directly
into the hypervisor and not via the kernel''s kexec_load system call.

Since the kexec hypercall is for use by dom0 I have removed the
implementation of the old load/unload ops and thus guests will require
updated kexec-tools to load images.  Is this acceptable?

David

On 16/01/13 16:29, David Vrabel wrote:
> This series of patches improves the kexec hypercall in the Xen
> hypervisor.  It is an incomplete prototype but I posting it early for
> comments on the proposed ABI/API.
And here is a (very) simple test program and a trivial test image I have
used.  This provides an example of how the libxc API can be used.

David



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

On Wed, 2013-01-16 at 16:29 +0000, David Vrabel wrote:
> Since the kexec hypercall is for use by dom0 I have removed the
> implementation of the old load/unload ops and thus guests will require
> updated kexec-tools to load images.  Is this acceptable? 
How hard would it be to also support the old interface for the benefit
of old kernels and userspaces (e.g. existing distros)?

Rather than a _v2 suffix we have in the past renamed the old ones
_compat and introduced the new ones, with appropriate use of
__XEN_INTERFACE_VERSION__ to paper over things for old users.

See __HYPERVISOR_sched_op for example.

Ian.

On 16/01/13 17:02, Ian Campbell wrote:
> On Wed, 2013-01-16 at 16:29 +0000, David Vrabel wrote:
>> Since the kexec hypercall is for use by dom0 I have removed the
>> implementation of the old load/unload ops and thus guests will require
>> updated kexec-tools to load images.  Is this acceptable? 
> 
> How hard would it be to also support the old interface for the benefit
> of old kernels and userspaces (e.g. existing distros)?
There''s an easy way (see patch 1), but it doesn''t have full
compatibility as it no longer executes the code page supplied by the
guest.  This won''t matter with Linux guests as their code pages can be
replaced by the code in Xen, but may matter with some more obscure
guests that do unusual things in their code pages (are there any like
this -- I doubt it?).

Full compatibility is possible and not that hard.  Is it actually worth
it though?  Will there be people updating Xen to 4.3 and unable to
update their kernel or userspace tools?
> Rather than a _v2 suffix we have in the past renamed the old ones
> _compat and introduced the new ones, with appropriate use of
> __XEN_INTERFACE_VERSION__ to paper over things for old users.
> 
> See __HYPERVISOR_sched_op for example.
Ok. I''ll look at this.

David

On Wed, 2013-01-16 at 17:48 +0000, David Vrabel wrote:
> On 16/01/13 17:02, Ian Campbell wrote:
> > On Wed, 2013-01-16 at 16:29 +0000, David Vrabel wrote:
> >> Since the kexec hypercall is for use by dom0 I have removed the
> >> implementation of the old load/unload ops and thus guests will
require
> >> updated kexec-tools to load images.  Is this acceptable? 
> > 
> > How hard would it be to also support the old interface for the benefit
> > of old kernels and userspaces (e.g. existing distros)?
> 
> There''s an easy way (see patch 1), but it doesn''t have
full
> compatibility as it no longer executes the code page supplied by the
> guest.  This won''t matter with Linux guests as their code pages
can be
> replaced by the code in Xen, but may matter with some more obscure
> guests that do unusual things in their code pages (are there any like
> this -- I doubt it?).
I''m not sure that any other PV dom0 support kexec on Xen at all.
> Full compatibility is possible and not that hard.  Is it actually worth
> it though?  Will there be people updating Xen to 4.3 and unable to
> update their kernel or userspace tools?
We''ve been telling people for a while to try and use their distro
packages where possible. Even if they build Xen from source it would be
nice if they didn''t have to also then rebuild their kernel or the kexec
tools etc and could keep their existing packages.

That said I''m not sure how widespread use of kexec is outside of
distros
anyway, and they can obviously integrate the right bits.
> > Rather than a _v2 suffix we have in the past renamed the old ones
> > _compat and introduced the new ones, with appropriate use of
> > __XEN_INTERFACE_VERSION__ to paper over things for old users.
> > 
> > See __HYPERVISOR_sched_op for example.
> 
> Ok. I''ll look at this.
> 
> David

>>> On 16.01.13 at 18:48, David Vrabel <david.vrabel@citrix.com>
wrote:
> Full compatibility is possible and not that hard.  Is it actually worth
> it though?  Will there be people updating Xen to 4.3 and unable to
> update their kernel or userspace tools?
I think we simply shouldn''t even be considering removal of
functionality that we can''t tell for sure is not being used
anywhere, and that we don''t deliberately want to go away (like
was the case with 32-bit hypervisor support).

Jan

>>> On 17.01.13 at 11:46, "Jan Beulich"
<JBeulich@suse.com> wrote:
>>>> On 16.01.13 at 18:48, David Vrabel
<david.vrabel@citrix.com> wrote:
>> Full compatibility is possible and not that hard.  Is it actually worth
>> it though?  Will there be people updating Xen to 4.3 and unable to
>> update their kernel or userspace tools?
> 
> I think we simply shouldn''t even be considering removal of
> functionality that we can''t tell for sure is not being used
> anywhere, and that we don''t deliberately want to go away (like
> was the case with 32-bit hypervisor support).
Forgot to say - for our distros, the kernel is being updated quite
regularly, whereas I have no idea at what intervals the kexec
user space code gets pulled. Plus I don''t think I''ve seen any
patches to this one yet, so you''re basically disabling all kexec
functionality for the time being when you remove the old code.

Jan

On Wed, Jan 16, 2013 at 04:29:03PM +0000, David Vrabel
wrote:
> This series of patches improves the kexec hypercall in the Xen
> hypervisor.  It is an incomplete prototype but I posting it early for
> comments on the proposed ABI/API.
>
> This allows a privileged Xen guest to load kexec images into the
> hypervisor from a userspace tool without using the Linux kernel''s
> kexec subsystem.  It is the first step to supporting kexec of crash
> kernels from a pv-ops dom0 kernel (the required kernel and kexec-tools
> patches will be posted later).
>
> The kernel will require a kexec hypercall somewhere in the
> crash_kexec() path to actually exec the loaded image.  Any preferences
> on how the hook for this should be implemented?  Note that the kernel
This should be implemented as stub which be called by machine_kexec()
and later it would call relevant hypercall.
> won''t be aware that an image as been loaded as it is loaded
directly
> into the hypervisor and not via the kernel''s kexec_load system
call.
Maybe we should have sepcial kexec hypercall function which allow us
to ask hypervisor that image is loaded or not.
> Since the kexec hypercall is for use by dom0 I have removed the
> implementation of the old load/unload ops and thus guests will require
> updated kexec-tools to load images.  Is this acceptable?
Not yet. I think that old interface should stay as long as Xen Linux Kernel
could run on latest versions of hypervisor.

Daniel

On 17/01/13 11:27, Daniel Kiper wrote:
> On Wed, Jan 16, 2013 at 04:29:03PM +0000, David Vrabel wrote:
>> This series of patches improves the kexec hypercall in the Xen
>> hypervisor.  It is an incomplete prototype but I posting it early for
>> comments on the proposed ABI/API.
>>
>> This allows a privileged Xen guest to load kexec images into the
>> hypervisor from a userspace tool without using the Linux
kernel''s
>> kexec subsystem.  It is the first step to supporting kexec of crash
>> kernels from a pv-ops dom0 kernel (the required kernel and kexec-tools
>> patches will be posted later).
>>
>> The kernel will require a kexec hypercall somewhere in the
>> crash_kexec() path to actually exec the loaded image.  Any preferences
>> on how the hook for this should be implemented?  Note that the kernel
> This should be implemented as stub which be called by machine_kexec()
> and later it would call relevant hypercall.
>
>> won''t be aware that an image as been loaded as it is loaded
directly
>> into the hypervisor and not via the kernel''s kexec_load system
call.
> Maybe we should have sepcial kexec hypercall function which allow us
> to ask hypervisor that image is loaded or not.
But we already have this information.  If the kexec crash hypercall
returns back to dom0 then a crash kernel is not loaded.

One could certainly argue that even if a crash kernel is not loaded, a
kexec crash hypercall means that dom0 is in bad state and Xen should
panic() anyway, which is the case on any other form of dom0 crash.

~Andrew
>
>> Since the kexec hypercall is for use by dom0 I have removed the
>> implementation of the old load/unload ops and thus guests will require
>> updated kexec-tools to load images.  Is this acceptable?
> Not yet. I think that old interface should stay as long as Xen Linux Kernel
> could run on latest versions of hypervisor.
>
> Daniel
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

On Thu, Jan 17, 2013 at 11:37:48AM +0000, Andrew Cooper
wrote:
> On 17/01/13 11:27, Daniel Kiper wrote:
> > On Wed, Jan 16, 2013 at 04:29:03PM +0000, David Vrabel wrote:
> >> This series of patches improves the kexec hypercall in the Xen
> >> hypervisor.  It is an incomplete prototype but I posting it early
for
> >> comments on the proposed ABI/API.
> >>
> >> This allows a privileged Xen guest to load kexec images into the
> >> hypervisor from a userspace tool without using the Linux
kernel''s
> >> kexec subsystem.  It is the first step to supporting kexec of
crash
> >> kernels from a pv-ops dom0 kernel (the required kernel and
kexec-tools
> >> patches will be posted later).
> >>
> >> The kernel will require a kexec hypercall somewhere in the
> >> crash_kexec() path to actually exec the loaded image.  Any
preferences
> >> on how the hook for this should be implemented?  Note that the
kernel
> > This should be implemented as stub which be called by machine_kexec()
> > and later it would call relevant hypercall.
> >
> >> won''t be aware that an image as been loaded as it is
loaded directly
> >> into the hypervisor and not via the kernel''s kexec_load
system call.
> > Maybe we should have sepcial kexec hypercall function which allow us
> > to ask hypervisor that image is loaded or not.
> 
> But we already have this information.  If the kexec crash hypercall
> returns back to dom0 then a crash kernel is not loaded.
> 
> One could certainly argue that even if a crash kernel is not loaded, a
> kexec crash hypercall means that dom0 is in bad state and Xen should
> panic() anyway, which is the case on any other form of dom0 crash.
I thought about somthing which does not do big bang when kexec image is loaded.
Now I think that it should return what type of image is loaded
(crash and/or regular one).

Daniel

On 17/01/13 11:27, Daniel Kiper wrote:
> On Wed, Jan 16, 2013 at 04:29:03PM +0000, David Vrabel wrote:
>> This series of patches improves the kexec hypercall in the Xen
>> hypervisor.  It is an incomplete prototype but I posting it early for
>> comments on the proposed ABI/API.
>>
>> This allows a privileged Xen guest to load kexec images into the
>> hypervisor from a userspace tool without using the Linux
kernel''s
>> kexec subsystem.  It is the first step to supporting kexec of crash
>> kernels from a pv-ops dom0 kernel (the required kernel and kexec-tools
>> patches will be posted later).
>>
>> The kernel will require a kexec hypercall somewhere in the
>> crash_kexec() path to actually exec the loaded image.  Any preferences
>> on how the hook for this should be implemented?  Note that the kernel
> 
> This should be implemented as stub which be called by machine_kexec()
> and later it would call relevant hypercall.
That''s a complicated way of making a simple function call. 
What''s the
justification for doing it this way?

Because Linux doesn''t allow images to be unloaded, it''s not
clear how we
can get sensible behavior if the image is unloaded from Xen.  The stub
will remain loaded in Linux but will not be able to return (easily?).
>> won''t be aware that an image as been loaded as it is loaded
directly
>> into the hypervisor and not via the kernel''s kexec_load system
call.
> 
> Maybe we should have sepcial kexec hypercall function which allow us
> to ask hypervisor that image is loaded or not.
It''s not possible to check for an image atomically. I don''t
think this
is necessary anyway, KEXEC_CMD_exec will return if no image is loaded.

David

David Vrabel <david.vrabel@citrix.com> writes:
> On 17/01/13 11:27, Daniel Kiper wrote:
>> On Wed, Jan 16, 2013 at 04:29:03PM +0000, David Vrabel wrote:
>>> The kernel will require a kexec hypercall somewhere in the
>>> crash_kexec() path to actually exec the loaded image.  Any
preferences
>>> on how the hook for this should be implemented?  Note that the
kernel
>> 
>> This should be implemented as stub which be called by machine_kexec()
>> and later it would call relevant hypercall.
>
> That''s a complicated way of making a simple function call. 
What''s the
> justification for doing it this way?
Because there is no justification for a Xen to have a hack in the
crash_kexec() path.  crash_kexec comes as close as it can to doing
a simple function call.
> Because Linux doesn''t allow images to be unloaded, it''s
not clear how we
> can get sensible behavior if the image is unloaded from Xen.  The stub
> will remain loaded in Linux but will not be able to return (easily?).
Linux allows images to be unloaded by loading an empty image.  Aka an
image with nr_segments == 0.
>>> won''t be aware that an image as been loaded as it is
loaded directly
>>> into the hypervisor and not via the kernel''s kexec_load
system call.
>> 
>> Maybe we should have sepcial kexec hypercall function which allow us
>> to ask hypervisor that image is loaded or not.
>
> It''s not possible to check for an image atomically. I
don''t think this
> is necessary anyway, KEXEC_CMD_exec will return if no image is loaded.
If Xen supported something other than the kexec on panic path this could
be interesting.  The current linux userspace does things conditionally
if you are in the reboot runlevel and kexec is coming up.  Certainly for
the kexec on panic path it is non-interesting.

Eric