Xen devel - Sep 2012 - Xen 4.2 new features and status -- please help me make a list

Hi all,

I''ve been trawling the mercurial logs looking for interesting new
features to update http://wiki.xen.org/wiki/Xen_4.2_Feature_List and
http://wiki.xen.org/wiki/Xen_Release_Features and have come up with a
list of interesting things which I need to know more about before I can
say stuff.

I''ve bcc''d people who I think might know about one or more of
the
following. If you do then please can you tell me:

      * What is the feature, what is the impact for the end user?
      * What is it''s status in 4.2? Is it a proof of concept, an
        experimental feature, tech preview or a finished completed
        feature.
      * Is it new in 4.2? If it isn''t new what was it''s status
in 4.1
        and 4.0 (i.e. from the list in the previous item).

Of course if you know of a feature in 4.2 which isn''t mentioned but
which you think is worthwhile please let me know. I''m sure there are
plenty which I missed.

Also feel free to edit the wiki instead....

TIA for your help.

Ian.

Nested Virtualisation: Allow HVM guests to use virtualisation hardware
(e.g. Windows 7 compat mode)
        Tech Preview, new in 4.2
        
vMSI: ? Emulated (or real?) MSIs for guests?
        State? New in 4.2?
        
vMCE: Forward MCE (Machine Check Exceptions) to guests
        New in 4.2? Preview or complete? I know improvements are pending
        to migration in 4.3.
        
AMD OSVW: What is this?
        Seems to be new in 4.2?
        
xenpaging: Page HVM guest pages to disk
        Was marked as tech preview in 4.1 and earlier, still is?
        
memsharing: Sharing of HVM guest pages.
        Was marked as tech preview in 4.1 and earlier, still is?

Intel HLE: What is this?
        Seems to be new in 4.2?
        
Intel TRM: What is this?
        Seems to be new in 4.2?

OVMF support for HVM guests.
        New in 4.2, but disabled by default => Tech preview?
        
vPMU: Power Management?
        New in 4.2?
        
ASID support:
        4.2 gained an option to control this but I think it was
        pre-existing. When was it first introduced? 4.1 or 4.0 or
        before?
        
Core parking: Offlining CPUs for power reasons?
        New in 4.2?
        
PV netboot: Network boot for PV guests
        New in 4.2

On 09/07/12 16:14, Ian Campbell wrote:
> Hi all,
> 
> I''ve been trawling the mercurial logs looking for interesting new
> features to update http://wiki.xen.org/wiki/Xen_4.2_Feature_List and
> http://wiki.xen.org/wiki/Xen_Release_Features and have come up with a
> list of interesting things which I need to know more about before I can
> say stuff.
> 
> I''ve bcc''d people who I think might know about one or
more of the
> following. If you do then please can you tell me:
> 
>       * What is the feature, what is the impact for the end user?
>       * What is it''s status in 4.2? Is it a proof of concept, an
>         experimental feature, tech preview or a finished completed
>         feature.
>       * Is it new in 4.2? If it isn''t new what was it''s
status in 4.1
>         and 4.0 (i.e. from the list in the previous item).
> 
> Of course if you know of a feature in 4.2 which isn''t mentioned
but
> which you think is worthwhile please let me know. I''m sure there
are
> plenty which I missed.
> 
> Also feel free to edit the wiki instead....
> 
> TIA for your help.
> 
> Ian.
> 
> Nested Virtualisation: Allow HVM guests to use virtualisation hardware
> (e.g. Windows 7 compat mode)
>         Tech Preview, new in 4.2
>         
> vMSI: ? Emulated (or real?) MSIs for guests?
>         State? New in 4.2?
>         
> vMCE: Forward MCE (Machine Check Exceptions) to guests
>         New in 4.2? Preview or complete? I know improvements are pending
>         to migration in 4.3.
>         
> AMD OSVW: What is this?
>         Seems to be new in 4.2?

OSVW (OS Visible Workarounds): Support for guests has been added
to make them disable workarounds for hw bugs not emulated/present
for guests.

Christoph

>         
> xenpaging: Page HVM guest pages to disk
>         Was marked as tech preview in 4.1 and earlier, still is?
>         
> memsharing: Sharing of HVM guest pages.
>         Was marked as tech preview in 4.1 and earlier, still is?
> 
> Intel HLE: What is this?
>         Seems to be new in 4.2?
>         
> Intel TRM: What is this?
>         Seems to be new in 4.2?
> 
> OVMF support for HVM guests.
>         New in 4.2, but disabled by default => Tech preview?
>         
> vPMU: Power Management?
>         New in 4.2?
>         
> ASID support:
>         4.2 gained an option to control this but I think it was
>         pre-existing. When was it first introduced? 4.1 or 4.0 or
>         before?
>         
> Core parking: Offlining CPUs for power reasons?
>         New in 4.2?
>         
> PV netboot: Network boot for PV guests
>         New in 4.2
> 
> 
> 


-- 
---to satisfy European Law for business letters:
Advanced Micro Devices GmbH
Einsteinring 24, 85689 Dornach b. Muenchen
Geschaeftsfuehrer: Alberto Bozzo
Sitz: Dornach, Gemeinde Aschheim, Landkreis Muenchen
Registergericht Muenchen, HRB Nr. 43632

On Sep 7, 2012, at 10:14 AM, Ian Campbell wrote:
> Hi all,
> 
> I''ve been trawling the mercurial logs looking for interesting new
> features to update http://wiki.xen.org/wiki/Xen_4.2_Feature_List and
> http://wiki.xen.org/wiki/Xen_Release_Features and have come up with a
> list of interesting things which I need to know more about before I can
> say stuff.
> 
> I''ve bcc''d people who I think might know about one or
more of the
> following. If you do then please can you tell me:
> 
>      * What is the feature, what is the impact for the end user?
>      * What is it''s status in 4.2? Is it a proof of concept, an
>        experimental feature, tech preview or a finished completed
>        feature.
>      * Is it new in 4.2? If it isn''t new what was it''s
status in 4.1
>        and 4.0 (i.e. from the list in the previous item).
> 
> Of course if you know of a feature in 4.2 which isn''t mentioned
but
> which you think is worthwhile please let me know. I''m sure there
are
> plenty which I missed.
> 
> Also feel free to edit the wiki instead....
> 
> TIA for your help.
> 
> Ian.
> 
> Nested Virtualisation: Allow HVM guests to use virtualisation hardware
> (e.g. Windows 7 compat mode)
>        Tech Preview, new in 4.2
> 
> vMSI: ? Emulated (or real?) MSIs for guests?
>        State? New in 4.2?
> 
> vMCE: Forward MCE (Machine Check Exceptions) to guests
>        New in 4.2? Preview or complete? I know improvements are pending
>        to migration in 4.3.
> 
> AMD OSVW: What is this?
>        Seems to be new in 4.2?
> 
> xenpaging: Page HVM guest pages to disk
>        Was marked as tech preview in 4.1 and earlier, still is?
> 
> memsharing: Sharing of HVM guest pages.
>        Was marked as tech preview in 4.1 and earlier, still is?
Both xenpaging and memsharing are functional as far as I am concerned. Intel EPT
is the platform of choice. There are many reports of success on AMD insofar
xenpaging goes, but I haven''t gotten good traction with either
xen{paging/sharing} on AMD.

Please note that memsharing has been significantly overhauled both at an
interface and internals level. It bears almost no resemblance to the 4.1
release.

Xen{paging/sharing} still have border conditions in which domains are crashed.
They are rare enough that I have not experienced them in practice. This is due
to a need for more mature wait queue code in the hypervisor. The plan is to
address this in 4.3.

Finally, mem-access has seen improvements and extensions. Now you can get a log
of all memory accesses by a guest (if you so wished) using the n2rwx mode.

Cheers,
Andres
> 
> Intel HLE: What is this?
>        Seems to be new in 4.2?
> 
> Intel TRM: What is this?
>        Seems to be new in 4.2?
> 
> OVMF support for HVM guests.
>        New in 4.2, but disabled by default => Tech preview?
> 
> vPMU: Power Management?
>        New in 4.2?
> 
> ASID support:
>        4.2 gained an option to control this but I think it was
>        pre-existing. When was it first introduced? 4.1 or 4.0 or
>        before?
> 
> Core parking: Offlining CPUs for power reasons?
>        New in 4.2?
> 
> PV netboot: Network boot for PV guests
>        New in 4.2
> 
>

>>> On 07.09.12 at 16:14, Ian Campbell <Ian.Campbell@citrix.com>
wrote:
> vMSI: ? Emulated (or real?) MSIs for guests?
>         State? New in 4.2?
Not sure what you''re referring to here. There certainly was a
xen/arch/x86/hvm/vmsi.c in 4.1 already.
> vMCE: Forward MCE (Machine Check Exceptions) to guests
>         New in 4.2? Preview or complete? I know improvements are pending
>         to migration in 4.3.
Should be in reasonable state; certainly not a preview.
> AMD OSVW: What is this?
>         Seems to be new in 4.2?
OS Visible Workaround. A little bit of virtualization of this for
HVM guests got added. Nothing end user visible though.
> Intel HLE: What is this?
>         Seems to be new in 4.2?
Something like "Hardware Lock Elision".

I don''t think there''s any support for this, just white-listing
the
feature for (HVM) guests.
> Intel TRM: What is this?
>         Seems to be new in 4.2?
"Restricted Transactional Memory"

Implementation-wise same as above.
> OVMF support for HVM guests.
>         New in 4.2, but disabled by default => Tech preview?
Wasn''t it that this doesn''t even build?
> vPMU: Power Management?
>         New in 4.2?
Enhanced iirc.

Jan

At 15:14 +0100 on 07 Sep (1347030896), Ian Campbell
wrote:
> ASID support:
>         4.2 gained an option to control this but I think it was
>         pre-existing. When was it first introduced? 4.1 or 4.0 or
>         before?
Long before.  3.0.x of some kind for AMD; 3.3 for Intel (as
''vpid'').

Tim.

>>> On 07.09.12 at 16:14, Ian Campbell <Ian.Campbell@citrix.com>
wrote:
> Of course if you know of a feature in 4.2 which isn''t mentioned
but
> which you think is worthwhile please let me know. I''m sure there
are
> plenty which I missed.
Not sure whether boot time CPU microcode patching is worth
mentioning. Certainly this is nothing user visible.

Jan

On Fri, 2012-09-07 at 16:22 +0100, Jan Beulich wrote:
> >>> On 07.09.12 at 16:14, Ian Campbell
<Ian.Campbell@citrix.com> wrote:
> > vMSI: ? Emulated (or real?) MSIs for guests?
> >         State? New in 4.2?
> 
> Not sure what you''re referring to here. There certainly was a
> xen/arch/x86/hvm/vmsi.c in 4.1 already.
OK, thanks. 4.0 too by the looks of it.

What exactly is this? Is it about injection of MSIs from real
(passthrough) hardware devices to guests (only HVM ones?) or is about
MSIs generated by emulated devices (or both)?
> 
> > vMCE: Forward MCE (Machine Check Exceptions) to guests
> >         New in 4.2? Preview or complete? I know improvements are
pending
> >         to migration in 4.3.
> 
> Should be in reasonable state; certainly not a preview.
Thanks.
> > AMD OSVW: What is this?
> >         Seems to be new in 4.2?
> 
> OS Visible Workaround. A little bit of virtualization of this for
> HVM guests got added. Nothing end user visible though.
> 
> > Intel HLE: What is this?
> >         Seems to be new in 4.2?
> 
> Something like "Hardware Lock Elision".
> 
> I don''t think there''s any support for this, just
white-listing the
> feature for (HVM) guests.
> 
> > Intel TRM: What is this?
> >         Seems to be new in 4.2?
> 
> "Restricted Transactional Memory"
> 
> Implementation-wise same as above.
OK. I don''t think "expose an underlying hardware feature to
guests" is
very interesting from a feature list PoV so I''ll omit both of these.
> 
> > OVMF support for HVM guests.
> >         New in 4.2, but disabled by default => Tech preview?
> 
> Wasn''t it that this doesn''t even build?
Something like that. Or maybe only with certain compilers or something.
Unless Atillio says otherwise I''ll mark it as a preview.
> 
> > vPMU: Power Management?
> >         New in 4.2?
> 
> Enhanced iirc.
But already a full feature in 4.1 and 4.0?

This lets HVM guests thing they have power management hardware, and
tickling it does what?

Thanks!
Ian.

On Fri, 2012-09-07 at 16:36 +0100, Jan Beulich wrote:
> >>> On 07.09.12 at 16:14, Ian Campbell
<Ian.Campbell@citrix.com> wrote:
> > Of course if you know of a feature in 4.2 which isn''t
mentioned but
> > which you think is worthwhile please let me know. I''m sure
there are
> > plenty which I missed.
> 
> Not sure whether boot time CPU microcode patching is worth
> mentioning. Certainly this is nothing user visible.
This is the thing which can load microcode without needing to wait for
the dom0 kernel to pass it up a blob?

I''m not sure about the feature lists (I suppose it is useful to
mention)
but extracting the commit log of 24315:3e5683b6b37f into some doc
somewhere might be useful for users?

(unless you did already and I failed to find it, of course)

Ian.

On Fri, 2012-09-07 at 15:26 +0100, Christoph Egger wrote:
> > AMD OSVW: What is this?
> >         Seems to be new in 4.2?
> 
> 
> OSVW (OS Visible Workarounds): Support for guests has been added
> to make them disable workarounds for hw bugs not emulated/present
> for guests.
Makes sense, thanks!

On Fri, 2012-09-07 at 15:34 +0100, Andres Lagar-Cavilla wrote:
> > xenpaging: Page HVM guest pages to disk
> >        Was marked as tech preview in 4.1 and earlier, still is?
> > 
> > memsharing: Sharing of HVM guest pages.
> >        Was marked as tech preview in 4.1 and earlier, still is?
> 
> Both xenpaging and memsharing are functional as far as I am concerned.
This is on the hypervisor side I guess? Or are the tools as supplied in
the xen tree useful too?

It''s a tricky one in terms of how to describe it on xen.org if not.
I''d
be inclined to go "yes" on the basis of the hypervisor side being
solid,
apart from the two caveats you then mention below (mainly the border
conditions one).
>  Intel EPT is the platform of choice. There are many reports of
> success on AMD insofar xenpaging goes, but I haven''t gotten good
> traction with either xen{paging/sharing} on AMD. 
> 
> Please note that memsharing has been significantly overhauled both at
> an interface and internals level. It bears almost no resemblance to
> the 4.1 release.
I''ll mention that somewhere too.
> Xen{paging/sharing} still have border conditions in which domains are
> crashed. They are rare enough that I have not experienced them in
> practice. This is due to a need for more mature wait queue code in the
> hypervisor. The plan is to address this in 4.3.
Perhaps that''s an argument for leaving it as tech-preview until 4.3
then?
> Finally, mem-access has seen improvements and extensions. Now you can
> get a log of all memory accesses by a guest (if you so wished) using
> the n2rwx mode.
thanks, I''ll be sure to mention that.

Ian.

>>> On 07.09.12 at 17:39, Ian Campbell <Ian.Campbell@citrix.com>
wrote:
> On Fri, 2012-09-07 at 16:22 +0100, Jan Beulich wrote:
>> >>> On 07.09.12 at 16:14, Ian Campbell
<Ian.Campbell@citrix.com> wrote:
>> > vMSI: ? Emulated (or real?) MSIs for guests?
>> >         State? New in 4.2?
>> 
>> Not sure what you''re referring to here. There certainly was a
>> xen/arch/x86/hvm/vmsi.c in 4.1 already.
> 
> OK, thanks. 4.0 too by the looks of it.
> 
> What exactly is this? Is it about injection of MSIs from real
> (passthrough) hardware devices to guests (only HVM ones?) or is about
> MSIs generated by emulated devices (or both)?
I don''t think emulated devices genera MSIs, so to me this is
just for passthrough. But I may be wrong.
>> > vPMU: Power Management?
>> >         New in 4.2?
>> 
>> Enhanced iirc.
> 
> But already a full feature in 4.1 and 4.0?
> 
> This lets HVM guests thing they have power management hardware, and
> tickling it does what?
Oh, I didn''t pay close enough attention - PM stands for
Performance Monitor here. Iirc e.g. Fujitsu have been using
this even in 4.0 already.

Jan

On Fri, 2012-09-07 at 16:49 +0100, Jan Beulich wrote:
> >>> On 07.09.12 at 17:39, Ian Campbell
<Ian.Campbell@citrix.com> wrote:
> > On Fri, 2012-09-07 at 16:22 +0100, Jan Beulich wrote:
> >> >>> On 07.09.12 at 16:14, Ian Campbell
<Ian.Campbell@citrix.com> wrote:
> >> > vMSI: ? Emulated (or real?) MSIs for guests?
> >> >         State? New in 4.2?
> >> 
> >> Not sure what you''re referring to here. There certainly
was a
> >> xen/arch/x86/hvm/vmsi.c in 4.1 already.
> > 
> > OK, thanks. 4.0 too by the looks of it.
> > 
> > What exactly is this? Is it about injection of MSIs from real
> > (passthrough) hardware devices to guests (only HVM ones?) or is about
> > MSIs generated by emulated devices (or both)?
> 
> I don''t think emulated devices genera MSIs, so to me this is
> just for passthrough. But I may be wrong.
It sounds plausible to me ;-)
> 
> >> > vPMU: Power Management?
> >> >         New in 4.2?
> >> 
> >> Enhanced iirc.
> > 
> > But already a full feature in 4.1 and 4.0?
> > 
> > This lets HVM guests thing they have power management hardware, and
> > tickling it does what?
> 
> Oh, I didn''t pay close enough attention - PM stands for
> Performance Monitor here. Iirc e.g. Fujitsu have been using
> this even in 4.0 already.
perf monitoring makes more sense as a guest accessible thing.

Ian.

>>> On 07.09.12 at 17:43, Ian Campbell <Ian.Campbell@citrix.com>
wrote:
> On Fri, 2012-09-07 at 16:36 +0100, Jan Beulich wrote:
>> Not sure whether boot time CPU microcode patching is worth
>> mentioning. Certainly this is nothing user visible.
> 
> This is the thing which can load microcode without needing to wait for
> the dom0 kernel to pass it up a blob?
Yes.
> I''m not sure about the feature lists (I suppose it is useful to
mention)
> but extracting the commit log of 24315:3e5683b6b37f into some doc
> somewhere might be useful for users?
> 
> (unless you did already and I failed to find it, of course)
No, I didn''t. I see that docs/misc/xen-command-line.markdown
only mentions the option, without description. I''ll see to get
that changed early next week.

Jan

On Fri, 2012-09-07 at 16:57 +0100, Jan Beulich wrote:
> >>> On 07.09.12 at 17:43, Ian Campbell
<Ian.Campbell@citrix.com> wrote:
> > On Fri, 2012-09-07 at 16:36 +0100, Jan Beulich wrote:
> >> Not sure whether boot time CPU microcode patching is worth
> >> mentioning. Certainly this is nothing user visible.
> > 
> > This is the thing which can load microcode without needing to wait for
> > the dom0 kernel to pass it up a blob?
> 
> Yes.
> 
> > I''m not sure about the feature lists (I suppose it is useful
to mention)
> > but extracting the commit log of 24315:3e5683b6b37f into some doc
> > somewhere might be useful for users?
> > 
> > (unless you did already and I failed to find it, of course)
> 
> No, I didn''t. I see that docs/misc/xen-command-line.markdown
> only mentions the option, without description. I''ll see to get
> that changed early next week.
Smashing, thanks!

Ian.

On Fri, Sep 07, Ian Campbell wrote:
> On Fri, 2012-09-07 at 15:34 +0100, Andres Lagar-Cavilla wrote:
> 
> > > xenpaging: Page HVM guest pages to disk
> > >        Was marked as tech preview in 4.1 and earlier, still is?
> > > 
> > > memsharing: Sharing of HVM guest pages.
> > >        Was marked as tech preview in 4.1 and earlier, still is?
> > 
> > Both xenpaging and memsharing are functional as far as I am concerned.
> 
> This is on the hypervisor side I guess? Or are the tools as supplied in
> the xen tree useful too?
xenpaging and the hypervisor work ok. The lack of xend/libxl integration
causes small inconvenience because the actual memory footprint has to be
set manually with xenstore-write. And until a target is set in xenstore
the tool does a busyloop. This is fixed with this patch, sent a few days
ago: <b088e473c7fb1a47b957.1346142770@probook.site>

> > Xen{paging/sharing} still have border conditions in which domains are
> > crashed. They are rare enough that I have not experienced them in
> > practice. This is due to a need for more mature wait queue code in the
> > hypervisor. The plan is to address this in 4.3.
> 
> Perhaps that''s an argument for leaving it as tech-preview until
4.3
> then?
I would say yes, at least for paging.

Olaf

On Fri, Sep 07, 2012 at 04:51:15PM +0100, Ian Campbell
wrote:
> On Fri, 2012-09-07 at 16:49 +0100, Jan Beulich wrote:
> > >>> On 07.09.12 at 17:39, Ian Campbell
<Ian.Campbell@citrix.com> wrote:
> > >> > vMSI: ? Emulated (or real?) MSIs for guests?
> > >> >         State? New in 4.2?
> > >> 
> > >> Not sure what you''re referring to here. There
certainly was a
> > >> xen/arch/x86/hvm/vmsi.c in 4.1 already.
> > > 
> > > OK, thanks. 4.0 too by the looks of it.
> > > 
> > > What exactly is this? Is it about injection of MSIs from real
> > > (passthrough) hardware devices to guests (only HVM ones?) or is
about
> > > MSIs generated by emulated devices (or both)?
> > 
> > I don''t think emulated devices genera MSIs, so to me this is
> > just for passthrough. But I may be wrong.
> 
> It sounds plausible to me ;-)
There is in 4.2 an HVM op to generate MSIs from emulated devices, added
at changeset 23413.
The passthrough part was already present before.

Thanks,
Gianluca

On Sat, Sep 8, 2012 at 5:26 AM, Gianluca Guida <glguida@tlbflush.org>
wrote:
>
> On Fri, Sep 07, 2012 at 04:51:15PM +0100, Ian Campbell wrote:
> > On Fri, 2012-09-07 at 16:49 +0100, Jan Beulich wrote:
> > > >>> On 07.09.12 at 17:39, Ian Campbell
<Ian.Campbell@citrix.com>
> > > >>> wrote:
> > > >> > vMSI: ? Emulated (or real?) MSIs for guests?
> > > >> >         State? New in 4.2?
> > > >>
> > > >> Not sure what you''re referring to here. There
certainly was a
> > > >> xen/arch/x86/hvm/vmsi.c in 4.1 already.
> > > >
> > > > OK, thanks. 4.0 too by the looks of it.
> > > >
> > > > What exactly is this? Is it about injection of MSIs from
real
> > > > (passthrough) hardware devices to guests (only HVM ones?) or
is
> > > > about
> > > > MSIs generated by emulated devices (or both)?
> > >
> > > I don''t think emulated devices genera MSIs, so to me
this is
> > > just for passthrough. But I may be wrong.
> >
> > It sounds plausible to me ;-)
>
> There is in 4.2 an HVM op to generate MSIs from emulated devices, added
> at changeset 23413.
> The passthrough part was already present before.
Quick clarification about emulated devices using MSI: the emulation of
vmware''s paravirtualized scsi controller, pvscsi, in qemu (submitted
as patches but not in the official qemu tree yet) uses MSI. Changeset
25588 was necessary in xen to get MSIs to flow from the pvscsi
emulation in qemu to xen vms running the upstream vmware pvscsi driver
in the linux kernel.

Thanks,
Deep