Xen devel - Feb 2008 - [PATCH][RFC] Supporting Enlightened Windows 2008 Server

During the spring Xen Summit in New York, I presented our work on hosting
Enlightened windows 2008 server on  Xen based platforms. Now that Microsoft has
published their Hypervisor specification, we can release our code to the
community. The goal of this effort has been to host Enlightened windows 2008
server as a guest on a Xen based platform. We have addressed this requirement by
:

A) Leveraging the I/O framework in the Xen based platform. Appropriate PV
drivers will be loaded up in the guest to improve the I/O performance.
 
B) Implement an adapter that implements the required Hyper-V functionality.  

We  have implemented only a subset of Hyper-V functionality that is required for
enlightened windows 2008 guest today. However, we have the framework in place to
implement any additional functionality that the windows guests may leverage
going forward. The framework is extensible and one can easily implement OS
specific enlightenments.

I am enclosing three patches that implement our adapter for your consideration:

1) ns_tools.patch: Modifications to xen tools to support the the adapter
2) ns_xen_base.patch: Modifications to the base Xen code to support the adapter
3) ns_xen_extension.patch: New code implementing the adapter 

These patches have been tested on a xen 3.2 based system (SLES10 SP2).

Signed-off by K. Y. Srinivasan (ksrinivasan@novell.com) 

Regards,

K. Y









_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 19/2/08 22:11, "Ky Srinivasan" <ksrinivasan@novell.com>
wrote:
> B) Implement an adapter that implements the required Hyper-V functionality.
> 
> We  have implemented only a subset of Hyper-V functionality that is
required
> for enlightened windows 2008 guest today. However, we have the framework in
> place to implement any additional functionality that the windows guests may
> leverage going forward. The framework is extensible and one can easily
> implement OS specific enlightenments.
The appearance is that you hook on every significant emulation point in the
HVM code and implement a shadow hypervisor. Is that an accurate description?
What performance wins do you get from Hyper-V emulation, ignoring the
obvious wins you will get from installing PV I/O drivers? I think this is a
particularly interesting question since it was our impression that the PV
memory-management interfaces, which you appear to jump through hoops to
support, were not of much benefit with Xen''s more sophisticated
pagetable
algorithm (or with Intel EPT or AMD NPT).

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

>>> On Wed, Feb 20, 2008 at  4:44 AM, in message
<C3E1A703.1CB24%Keir.Fraser@cl.cam.ac.uk>, Keir Fraser
<Keir.Fraser@cl.cam.ac.uk> wrote: 
> On 19/2/08 22:11, "Ky Srinivasan" <ksrinivasan@novell.com>
wrote:
> 
>> B) Implement an adapter that implements the required Hyper- V
functionality.
>> 
>> We  have implemented only a subset of Hyper- V functionality that is
required
>> for enlightened windows 2008 guest today. However, we have the
framework in
>> place to implement any additional functionality that the windows guests
may
>> leverage going forward. The framework is extensible and one can easily
>> implement OS specific enlightenments.
> 
> The appearance is that you hook on every significant emulation point in the
> HVM code and implement a shadow hypervisor. Is that an accurate
description?
My initial goal was to get a framework in place where one could implement any OS
specific enhancements without cluttering up the rest of the Xen code. To that
end, this code does hook into various emulation points of interest in the hvm
code. A secondary goal was to minimize the changes to the base Xen code.
Furthermore, I wanted to reuse as much of the base Xen code as possible. So, I
would not characterize this as a shadow hypervisor, but rather as an adapter
that bridges the semantic gap between the APIs of interest in Hyper-V and Xen.
> What performance wins do you get from Hyper- V emulation, ignoring the
> obvious wins you will get from installing PV I/O drivers? I think this is a
> particularly interesting question since it was our impression that the PV
> memory- management interfaces, which you appear to jump through hoops to
> support, were not of much benefit with Xen''s more sophisticated
pagetable
> algorithm (or with Intel EPT or AMD NPT).
We have done very minimal performance evaluation to date. On NetBench I have
seen a 10% improvement. You are right, TLB flush enlightenments did not make
much of a difference. These results are on a uniprocessor configuration. Of the
enlightenments, MSR based APIC access made the most difference.  Going forward,
we will be doing a more comprehensive performance evaluation with the goal of
having parity with the MS platform.

Regards,

K. Y


> 
>  --  Keir
> 
> 
> 
> _______________________________________________
> Xen- devel mailing list
> Xen- devel@lists.xensource.com
> http://lists.xensource.com/xen- devel

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 19/02/08 15:11 -0700, Ky Srinivasan wrote:
> During the spring Xen Summit in New York, I presented our work on hosting
Enlightened windows 2008 server on  Xen based platforms. Now that Microsoft has
published their Hypervisor specification, we can release our code to the
community. The goal of this effort has been to host Enlightened windows 2008
server as a guest on a Xen based platform. We have addressed this requirement by
:
> 
> A) Leveraging the I/O framework in the Xen based platform. Appropriate PV
drivers will be loaded up in the guest to improve the I/O performance.
>  
> B) Implement an adapter that implements the required Hyper-V functionality.
> 
> We  have implemented only a subset of Hyper-V functionality that is
required for enlightened windows 2008 guest today. However, we have the
framework in place to implement any additional functionality that the windows
guests may leverage going forward. The framework is extensible and one can
easily implement OS specific enlightenments.
> 
> I am enclosing three patches that implement our adapter for your
consideration:
> 
> 1) ns_tools.patch: Modifications to xen tools to support the the adapter
> 2) ns_xen_base.patch: Modifications to the base Xen code to support the
adapter
> 3) ns_xen_extension.patch: New code implementing the adapter 
> 
> These patches have been tested on a xen 3.2 based system (SLES10 SP2).
 
Right now there is not an implementation to support an enlightened
windows guest, just a shim.

First question: Do you plan on implementing the full enlightened guest
support as GPL source (as you have done with the elementary shim), or
are you planning on loading a blob and initializing the hypercall
vector, a la the vmware VMI? Has Microsoft agreed to a source-based
GPL implementation of the hypercall layer?

Second question: what do you believe the performance will be of an
enlightened guest running in the novell shim vs running on hyper-v?
Have you implemented enough to do any profiling? Have you done any
profiling of a standard hvm guest with and without the intercepts in
place? It looks like no measurable impact, just curious.

It doesn''t seem as though you can batch page table updates, or did I
miss something? 

Mike

-- 
Mike D. Day
IBM LTC
Cell: 919 412-3900
Sametime: ncmike@us.ibm.com AIM: ncmikeday  Yahoo: ultra.runner
PGP key: http://www.ncultra.org/ncmike/pubkey.asc

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

I am attaching updated versions of the patches that I posted a couple of weeks
ago. These  have been merged up to the current unstable tip: changeset
17186:854b0704962b

These patches have been tested on the unstable tip.

Kier, what are your thoughts on accepting these patches.

Signed-off by K. Y. Srinivasan (ksrinivasan@novell.com)

Regards,

K. Y 











_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Wed, Mar 05, 2008 at 03:15:19PM -0700, Ky Srinivasan
wrote:
> I am attaching updated versions of the patches that I posted a couple 
> of weeks ago. These  have been merged up to the current unstable tip: 
> changeset 17186:854b0704962b
> 
> These patches have been tested on the unstable tip.
I''m not expert enough to comment on the HV extension implementation
itself,
but in terms of the userspace  side, the user visible configuration file 
option ''extid=1'' is pretty unpleasant.  It is akin to a
''magic constant''
in C code - no understandable meaning at all.

I''d like to see it accept a named extension type - if its possible to
have multiple extensions per guest, then using a list instead of a scalar
would be better. So how about something closer to

  extensions = [ "win2k8" ]
> +gopts.var(''extid'', val=''EXTID'',
> +          fn=set_int, default=0,
> +          use="Specify extention ID for a HVM domain.")
> +
And this help message could list the valid extension names


Regards,
Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496 -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=| 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Wed, Mar 05, 2008 at 10:28:28PM +0000, Daniel P. Berrange
wrote:
> On Wed, Mar 05, 2008 at 03:15:19PM -0700, Ky Srinivasan wrote:
> > I am attaching updated versions of the patches that I posted a couple 
> > of weeks ago. These  have been merged up to the current unstable tip: 
> > changeset 17186:854b0704962b
> > 
> > These patches have been tested on the unstable tip.
> 
> I''m not expert enough to comment on the HV extension
implementation itself,
> but in terms of the userspace  side, the user visible configuration file 
> option ''extid=1'' is pretty unpleasant.  It is akin to a
''magic constant''
> in C code - no understandable meaning at all.
> 
> I''d like to see it accept a named extension type - if its possible
to
> have multiple extensions per guest, then using a list instead of a scalar
> would be better. So how about something closer to
> 
>   extensions = [ "win2k8" ]
Or is there some way you can have some super light weight trap / hook
always loaded, so when Win2k8 makes it first paravirt call, you can
then automatically enable the full extension ? That could let Xen
just ''do the right thing'' without needing a config param, and
without
having to fully enable the full extension for non-Win2k8 guests.
>Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496 -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=| 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Personally I think the approach is ugly, and also you have not yet presented
evidence that supporting the Viridian paravirtualisations improves
performance. If it doesn''t then it''s a waste of time; if it
does then it
raises the question of which hypercalls provide the benefit, and do we get a
smaller neater patch by supporting just those? One final comment is that the
TLB management code that this slaps on top of the core hypervisor looks a
bit scary to me. Tim Deegan may care to comment more on that.

 -- Keir

On 5/3/08 22:15, "Ky Srinivasan" <ksrinivasan@novell.com> wrote:
> I am attaching updated versions of the patches that I posted a couple of
weeks
> ago. These  have been merged up to the current unstable tip: changeset
> 17186:854b0704962b
> 
> These patches have been tested on the unstable tip.
> 
> Kier, what are your thoughts on accepting these patches.
> 
> Signed-off by K. Y. Srinivasan (ksrinivasan@novell.com)
> 
> Regards,
> 
> K. Y 
> 
> 
> 
> 
> 
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

At 07:28 +0000 on 06 Mar (1204788507), Keir Fraser
wrote:
> Personally I think the approach is ugly, and also you have not yet
presented
> evidence that supporting the Viridian paravirtualisations improves
> performance. If it doesn''t then it''s a waste of time; if
it does then it
> raises the question of which hypercalls provide the benefit, and do we get
a
> smaller neater patch by supporting just those? One final comment is that
the
> TLB management code that this slaps on top of the core hypervisor looks a
> bit scary to me. Tim Deegan may care to comment more on that.
Some blame lies with the mismatch between the Viridian interface and
Xen''s; there needs to be a way for the TLB flush hypercall to block
indefinitely.  But I can''t see how that turns into more than an
atomic_t
for TlbFlushInhibit and a block-and-schedule operation.  In the current
patches, there''s quite a lot of locking and ownership going on as well.
I''m confused by the use of wait_on_xen_event_channel(0, xyz); event
channels don''t seem to come into it.

I''ll mention now, since I have the patch in front of me, that I dislike
the addition of an "ext_id" field to the HVM save format header and
associated special treatment in the save/restore code; you should be
able to figure out that this is a w2k8 domain from the presence of your
other records in the save file.



More generally, I agree that the approach is very heavyweight.  I don''t
see the need for a framework here, since there''s no other proposed user
of it that would want the same interface.  It seems to duplicate a lot
of things (does it really need its own spinlock implementation?)

It''s certainly not in Xen coding style, even in the framework
implementation.  (The MS habit of encoding scope and type information in
variable names annoys the heck out of me.  Why does a lock field in an
nsPartition_t need to be called "nsLock"?)

The naming in general could do with a kicking -- calling everything
"Novell Shim" is understandable for historical reasons but not really
descriptive of its function.   But maybe that can wait.

Tim

-- 
Tim Deegan <Tim.Deegan@citrix.com>
Principal Software Engineer, Citrix Systems (R&D) Ltd.
[Company #02300071, SL9 0DZ, UK.]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

>>> On Wed, Mar 5, 2008 at  5:28 PM, in message
<20080305222828.GH19306@redhat.com>, "Daniel P. Berrange"
<berrange@redhat.com>
wrote: 
> On Wed, Mar 05, 2008 at 03:15:19PM -0700, Ky Srinivasan wrote:
>> I am attaching updated versions of the patches that I posted a couple 
>> of weeks ago. These  have been merged up to the current unstable tip: 
>> changeset 17186:854b0704962b
>> 
>> These patches have been tested on the unstable tip.
> 
> I''m not expert enough to comment on the HV extension
implementation itself,
> but in terms of the userspace  side, the user visible configuration file 
> option ''extid=1'' is pretty unpleasant.  It is akin to a
''magic constant''
> in C code - no understandable meaning at all.
Agreed; I could make the extension tag more descriptive.

Regards,

K. Y
> 
> I''d like to see it accept a named extension type - if its possible
to
> have multiple extensions per guest, then using a list instead of a scalar
> would be better. So how about something closer to
> 
>   extensions = [ "win2k8" ]
> 
>> +gopts.var(''extid'', val=''EXTID'',
>> +          fn=set_int, default=0,
>> +          use="Specify extention ID for a HVM domain.")
>> +
> 
> And this help message could list the valid extension names
> 
> 
> Regards,
> Dan.



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

>>> On Wed, Mar 5, 2008 at  5:38 PM, in message
<20080305223827.GI19306@redhat.com>, "Daniel P. Berrange"
<berrange@redhat.com>
wrote: 
> On Wed, Mar 05, 2008 at 10:28:28PM +0000, Daniel P. Berrange wrote:
>> On Wed, Mar 05, 2008 at 03:15:19PM -0700, Ky Srinivasan wrote:
>> > I am attaching updated versions of the patches that I posted a
couple
>> > of weeks ago. These  have been merged up to the current unstable
tip:
>> > changeset 17186:854b0704962b
>> > 
>> > These patches have been tested on the unstable tip.
>> 
>> I''m not expert enough to comment on the HV extension
implementation itself,
>> but in terms of the userspace  side, the user visible configuration
file
>> option ''extid=1'' is pretty unpleasant.  It is akin to
a ''magic constant''
>> in C code - no understandable meaning at all.
>> 
>> I''d like to see it accept a named extension type - if its
possible to
>> have multiple extensions per guest, then using a list instead of a
scalar
>> would be better. So how about something closer to
>> 
>>   extensions = [ "win2k8" ]
> 
> Or is there some way you can have some super light weight trap / hook
> always loaded, so when Win2k8 makes it first paravirt call, you can
> then automatically enable the full extension ? That could let Xen
> just ''do the right thing'' without needing a config param,
and without
> having to fully enable the full extension for non-Win2k8 guests.
I considered this. Unfortunately, we have no control on the mechanisms used by
windows to discover the hypervisor. Furthermore, CPUID leaves used by longhorn
collide with CPUID leaves used by Xen for supporting hypervisor discovery for PV
drivers. Since we have no control on what Microsoft may do here, I felt the best
way might be to tag the guest so that we can interpret the hypercalls, CPUID
calls and MSR (read/write) calls within the guest namespace.

Regards,

K. Y



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

>>> On Thu, Mar 6, 2008 at  2:28 AM, in message
<C3F54D9B.14C64%keir.fraser@eu.citrix.com>, Keir Fraser
<keir.fraser@eu.citrix.com> wrote: 
> Personally I think the approach is ugly, and also you have not yet
presented
> evidence that supporting the Viridian paravirtualisations improves
> performance.
When I first implemented this (about a year ago), it was not clear if Microsoft
would be open sourcing the Veridian specification. Given that, I wanted to have
a narrow set of interfaces both to the adapter as well as from the adapter. I
take it that you don''t care much for this exercise in attempting to
isolate the adapter. Now that Veridian specification has been open sourced, I
agree there is no need to isolate the adapter from the base hypervisor the way
it is currently done. However, given that:
(a) Veridian specification is evolving and Microsoft may define additional
functionality to improve guest performance
(b) CPUID namespace, MSR namespace and hypercall namespace collisions between
Xen and Veridian. This is the case today and it can only get worse over time.

I feel having a framework that allows you to implement these kinds of mapping
layers in complete isolation from the base hypervisor  may in fact be cleaner
than trying to implement the mapping code inline in the base Xen code.

With regards to performance, we have only run NetBench and on NetBench we have
seen a 10% improvement (on a uniprocessor system). We have had some issues with
SMP PV drivers and that is the reason I don''t have SMP numbers (the
adapter has been tested on SMP machines though). We are currently in the process
of running a range of benchmarks and I will keep you posted on what we see. Our
goal here is clearly to be competitive (as far as performance goes) with
Veridian hosting an enlightened windows guest.
 
> If it doesn''t then it''s a waste of time; if it does then
it
> raises the question of which hypercalls provide the benefit, and do we get
a
> smaller neater patch by supporting just those?
I think the only assumption we can make here is  that the enlightenments will
improve the  guest performance. This has been confirmed with the minimal
performance testing we have already done.  I am also going to assume that
Microsoft will continue to evolve Veridian and the set of enlightenments visible
to their guests to improve performance. The question that we need to answer, I
think is how are we going to support these enlightenments and not if we are
going to support Microsoft specific enlightenments. I will be the first one to
admit the patches I submitted need to be cleaned up:

1) Fix coding style
2) Get rid of code that is not being exercised. Based on the Veridian
specification I identified a set of functionality that I thought an enlightened
guest may depend on. It looks like the current shipping windows server 2008 does
not use all the functionality that is currently supported. I am somewhat
hesitant to get rid of   unused functionality since I don''t know what
the next release of windows will use. In fact, the current shipping windows
server 2008 (32 bit version)  is already using an undocumented hypercall!

I do think however that having an environment in which we can implement and
evolve the support for windows enlightenments without constantly churning the
base Xen code  is the right approach.

Regards,

K. Y  




_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

>>> On Thu, Mar 6, 2008 at  5:15 AM, in message
<20080306101542.GA22422@york.uk.xensource.com>, Tim Deegan
<Tim.Deegan@citrix.com> wrote: 
> At 07:28 +0000 on 06 Mar (1204788507), Keir Fraser wrote:
>> Personally I think the approach is ugly, and also you have not yet
presented
>> evidence that supporting the Viridian paravirtualisations improves
>> performance. If it doesn''t then it''s a waste of time;
if it does then it
>> raises the question of which hypercalls provide the benefit, and do we
get a
>> smaller neater patch by supporting just those? One final comment is
that the
>> TLB management code that this slaps on top of the core hypervisor looks
a
>> bit scary to me. Tim Deegan may care to comment more on that.
> 
> Some blame lies with the mismatch between the Viridian interface and
> Xen''s; there needs to be a way for the TLB flush hypercall to
block
> indefinitely.  But I can''t see how that turns into more than an
atomic_t
> for TlbFlushInhibit and a block-and-schedule operation.  In the current
> patches, there''s quite a lot of locking and ownership going on as
well.
> I''m confused by the use of wait_on_xen_event_channel(0, xyz);
event
> channels don''t seem to come into it.
The Veridian API allows the guest to pass in a variable list of arguments to the
TLB flush call ( HvFlushVirtualAddressList). Furthermore, both forms of the
flush APIs (HvFlushVirtualAddressSpace and HvFlushVirtualAdressList) can specify
a list of vcpus that should be involved in the flush process. So, as you have
noted we will need a mechanism to co-ordinate the flush operation amongst the
set of vcpus involved which means we need to be able give up the physical CPU in
the hypervisor waiting for the flush to complete. I have used
wait_on_xen_event_channel() to implement this synchronization. Since we
don''t preserve the stack state when we block in the hypervisor, I have
used a seperate per-vcpu page for dealing with hypercall input parameters for
calls that can potentially block in the hypervisor. From what I have seen,
win2k8 server mostly specifies  all the processors in ProcessorMask. So, I chose
to implement TLB flush operations using a single serialization object that keeps
track of both the set of vcpus involved in the flush operation as well as the
list of pages to be flushed.
> 
> I''ll mention now, since I have the patch in front of me, that I
dislike
> the addition of an "ext_id" field to the HVM save format header
and
> associated special treatment in the save/restore code; you should be
> able to figure out that this is a w2k8 domain from the presence of your
> other records in the save file.
I can fix this.
> 
> 
> 
> More generally, I agree that the approach is very heavyweight.  I
don''t
> see the need for a framework here, since there''s no other proposed
user
> of it that would want the same interface.
I agree that there is no need to isolate the shim''s dependence on the
base Xen code (xen_call_vector_t). I implemented this shim a year ago and at
that point it was not clear what Microsoft might do with the Veridian
specification.  So, clearly some of the design choices that I made a year ago
may not be the right choice today. However, I still think that having an
intercept framework where  one can implement Veridian specific functionality
without cluttering up the base Xen code is still the right approach.
    
> It seems to duplicate a lot
> of things (does it really need its own spinlock implementation?)
Clearly not! As I noted in an earlier email to Kier, I will be the first to
admit that these patches require significant cleanup and I am willing to clean
them up. A lot of what you see has historical baggage and I wanted to get some
feedback before I invested the time to clean things up.
> 
> It''s certainly not in Xen coding style, even in the framework
> implementation.  (The MS habit of encoding scope and type information in
> variable names annoys the heck out of me.  Why does a lock field in an
> nsPartition_t need to be called "nsLock"?)
Agreed. 
> 
> The naming in general could do with a kicking -- calling everything
> "Novell Shim" is understandable for historical reasons but not
really
> descriptive of its function.   But maybe that can wait.
Agreed.

Regards,

K. Y
> 
> Tim


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

At 18:10 -0700 on 06 Mar (1204827047), Ky Srinivasan wrote:
> The Veridian API allows the guest to pass in a variable list of
> arguments to the TLB flush call
> (HvFlushVirtualAddressList). Furthermore, both forms of the flush APIs
> (HvFlushVirtualAddressSpace and HvFlushVirtualAdressList) can specify
> a list of vcpus that should be involved in the flush process.
I expect that the best way to implement the list-of-addresses feature on
Xen is to do a full TLB flush if there''s more than one entry in the
list
(that''s different from the Hyper-V shadow pagetable design, where
explicit lists of addresses to flush make much more sense).

Then all you need are the existing Xen TLB flush operations, and some
means of gating them.  If you use a generation counter with each vcpu''s
inhibit bit, you can probably do that without the need for any locks.
> So, as
> you have noted we will need a mechanism to co-ordinate the flush
> operation amongst the set of vcpus involved which means we need to be
> able give up the physical CPU in the hypervisor waiting for the flush
> to complete. I have used wait_on_xen_event_channel() to implement this
> synchronization.
wait_on_xen_event_channel is more than you need, since you''re not going
to wake on events.
> Since we don''t preserve the stack state when we block
> in the hypervisor, I have used a seperate per-vcpu page for dealing
> with hypercall input parameters for calls that can potentially block
> in the hypervisor.
Xen already has a system of hypercall continuations that might help here.

Cheers,

Tim

-- 
Tim Deegan <Tim.Deegan@citrix.com>
Principal Software Engineer, Citrix Systems (R&D) Ltd.
[Company #02300071, SL9 0DZ, UK.]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 7/3/08 01:10, "Ky Srinivasan" <ksrinivasan@novell.com> wrote:
> The Veridian API allows the guest to pass in a variable list of arguments
to
> the TLB flush call ( HvFlushVirtualAddressList). Furthermore, both forms of
> the flush APIs (HvFlushVirtualAddressSpace and HvFlushVirtualAdressList)
can
> specify a list of vcpus that should be involved in the flush process. So,
as
> you have noted we will need a mechanism to co-ordinate the flush operation
> amongst the set of vcpus involved which means we need to be able give up
the
> physical CPU in the hypervisor waiting for the flush to complete. I have
used
> wait_on_xen_event_channel() to implement this synchronization. Since we
don''t
> preserve the stack state when we block in the hypervisor, I have used a
> seperate per-vcpu page for dealing with hypercall input parameters for
calls
> that can potentially block in the hypervisor. From what I have seen, win2k8
> server mostly specifies  all the processors in ProcessorMask. So, I chose
to
> implement TLB flush operations using a single serialization object that
keeps
> track of both the set of vcpus involved in the flush operation as well as
the
> list of pages to be flushed.
Clearly avoiding emulating IPI-to-all-CPUs is rather likely to be a win. But
is the very selective subset-of-CPUs and subset-of-addresses really that
useful? Do you get any significant win over just calling
hvmop_flush_tlb_all()?

Also we need to weigh up the likely penetration of NPT and EPT capable
processors by the time w2k8 is shipping in any volume. But even ignoring
that, I bet 95% of the benefit of this patch can be got with a much smaller
patch.

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 7/3/08 01:10, "Ky Srinivasan" <ksrinivasan@novell.com> wrote:
> I agree that there is no need to isolate the shim''s dependence on
the base Xen
> code (xen_call_vector_t). I implemented this shim a year ago and at that
point
> it was not clear what Microsoft might do with the Veridian specification. 
So,
> clearly some of the design choices that I made a year ago may not be the
right
> choice today. However, I still think that having an intercept framework
where
> one can implement Veridian specific functionality without cluttering up the
> base Xen code is still the right approach.
Clearly putting the Viridian hypercall shims in a different file/directory
makes sense. But I think the shims would need to go on a diet. The TLB
flushing implementation is a good example -- the useful extra features of
the Viridian flush hypercall (if there are any, when partnered with
Xen''s
shadow code) should be pushed into core Xen HVM TLB-flush handling code.
Otherwise it sits out on the periphery with a correspndingly greater
tendency to rot, and for no benefit (certainly I would strongly argue it is
not cleaner!).

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel