Virtualization - Apr 2007 - Phone meeting about kernel virtualization hooks

Chris,
   Thanks for the reply.  Do you know who is doing the coarse
re-arrangement or the Xen tree?  Do they need help?

Bruce


-----Original Message-----
From: Chris Wright [mailto:chrisw@osdl.org] 
Sent: Wednesday, September 28, 2005 3:23 PM
To: Walker, Bruce J (HP-Labs)
Cc: Chris Wright; zach@vmware.com; Ian Pratt; keir.fraser@cl.cam.ac.uk;
jimix@watson.ibm.com; riel@redhat.com; steven@xensource.com;
cgriffin@novell.com; mbligh@mbligh.org; Magenheimer, Dan (HP Labs Fort
Collins); Byrne, John (HP Labs); ak@suse.de;
virtualization@lists.osdl.org
Subject: Re: Phone meeting about kernel virtualization hooks

* Walker, Bruce J (HP-Labs) (bruce.walker@hp.com) wrote:
> I would like to host a phone meeting to discuss the plans for the xen 
> sub-architecture of x86.  Hopefully Chris can share his vision and we 
> can agree on some next steps to getting this done and into the 
> mainline kernel.  From my understanding, it would be good if the
"To"
list could
> make it.   If anyone believes others should attend, please let them
> know.
> Please let me know if you are going to call in (no doubt some limit on
> the bridge).
I don't think it's necessary right now.  We should be able to start
making progress again once the Xen tree gets the coarse re-arrangement
done.
That's pending, so once it's there we'll have a much better chance
at
making a todo list and splitting up the work.

OTOH, I could make it, just can't commit to being very coherent ;-)

thanks,
-chris

* Walker, Bruce J (HP-Labs) (bruce.walker@hp.com) wrote:
> I would like to host a phone meeting to discuss the plans for the xen
> sub-architecture of x86.  Hopefully Chris can share his vision and we
> can agree on some next steps to getting this done and into the mainline
> kernel.  From my understanding, it would be good if the "To" list
could
> make it.   If anyone believes others should attend, please let them
> know.
> Please let me know if you are going to call in (no doubt some limit on
> the bridge).
I don't think it's necessary right now.  We should be able to start
making
progress again once the Xen tree gets the coarse re-arrangement done.
That's pending, so once it's there we'll have a much better chance
at
making a todo list and splitting up the work.

OTOH, I could make it, just can't commit to being very coherent ;-)

thanks,
-chris

> > Another useful dichotomy would be to separate the changes
> > that are required for Xen into two buckets:  Changes/files
> > that impact existing Linux files, and files that (eventually)
> > end up in a subdirectory that is distinct from any
> > existing Linux subdirectory.  Presumably the latter class
> > will not require (nearly as much) review.
> 
> It depends. Some maintainers like for network device
> drivers have pretty high requirements. Best to submit it to
> them early.
Hmmm... would the network maintainer have (or want to
have) purview over a virtual network driver that is in
drivers/xen/net{front,back}?

Dan

> -----Original Message-----
> From: virtualization-bounces@lists.osdl.org 
> [mailto:virtualization-bounces@lists.osdl.org] On Behalf Of Andi Kleen
> Sent: Wednesday, September 28, 2005 12:59 PM
> To: virtualization@lists.osdl.org; Martin J. Bligh
> Cc: cgriffin@novell.com; Chris Wright; steven@xensource.com; 
> Walker, Bruce J (HP-Labs)
> Subject: Re: Phone meeting about kernel virtualization hooks
> 
> On Wednesday 28 September 2005 20:41, Martin J. Bligh wrote:
> 
> >
> > Would be splendid if we can keep that up-to-date with every 
> major release,
> > plus -rc releases if possible. I know it's a lot of work ...
> 
> It's one of the problems imho that the xen tree is always 
> based on a ancient
> tree (last release). It would be much easier if it tracked 
> mainline more 
> closely. 
Agreed but I'd suggest that this is a symptom of the real problem:
Updating the Xen -sparse tree is still a very manual process.
For someone who's done it a few times (e.g. Keir or Ian), it
"only" takes a couple hours.  As more of the preparatory patches
get into linux, the effort will go down, but it is still
a manual process.

I am intrigued by Rik's proposal of creating a daily mirror that
restructures the directories; if this could also automagically
create patches** and attempt to apply them to several Linux trees
(e.g. last dot release, current -rcX, current -mm), the
process would very quickly uncover potential Xenlinux problem spots
and encourage fixes to them.

** Per my previous message, a three phase patch would be best: First
a patch for existing Linux files, then a patch for new files
in existing Linux directories, then a patch for new files in
new directories.

Another useful dichotomy would be to separate the changes
that are required for Xen into two buckets:  Changes/files
that impact existing Linux files, and files that (eventually)
end up in a subdirectory that is distinct from any
existing Linux subdirectory.  Presumably the latter class
will not require (nearly as much) review.

On a somewhat self-serving note, the first class of
changes will almost certainly be guaranteed to be
arch-neutral (else there will be howls from all of the
linux arch maintainers).  We must also work to ensure that
the second class are also arch-neutral... they currently
are not (even files that are currently targeted for
"linux/drivers/xen/core").

Dan
> -----Original Message-----
> From: virtualization-bounces@lists.osdl.org 
> [mailto:virtualization-bounces@lists.osdl.org] On Behalf Of 
> Martin J. Bligh
> Sent: Wednesday, September 28, 2005 12:41 PM
> To: Walker, Bruce J (HP-Labs); Pratap Subrahmanyam; Rik van Riel
> Cc: cgriffin@novell.com; Chris Wright; steven@xensource.com; 
> virtualization@lists.osdl.org
> Subject: RE: Phone meeting about kernel virtualization hooks
> 
> > I'm very encouraged that "we all know what needs doing"
and
> I'm quite
> > happy not to have the call.  I am concerned, however, that progress
> > appears to have been very slow (Chris is currently on leave and very
> > unavailable).  At the Xen mini-summit at OLS, Ian said it 
> was urgent for
> > Xen's success to get the hooks into the kernel and pleaded for
help.
> > Since then little has visibly happened (Zach has made a little
> > progress).  How is this going to get done?   
> > 
> > So.  I'm cancelling the call but plead for someone to lead 
> the effort
> > (or at least explain what needs doing).  We are willing to 
> help but need
> > some direction.
> 
> If "we all know what needs doing", might be useful to list that
out in
> email just to make sure. Very very roughly, what I was thinking was to
> make (and keep maintained a stack of (in order)):
> 
> mainline kernel + things going upstream + rest of xen
> 
> And keep that stack working. To put it another way, we need 
> to keep slicing
> stuff off the bottom of the Xen patch and merging it.
> 
> I don't think we have a working stack in that form at the 
> moment (at least
> not on top of any recent mainline kernel), which is my main 
> concern right
> now (can't really do much without it). 
> 
> Would be splendid if we can keep that up-to-date with every 
> major release,
> plus -rc releases if possible. I know it's a lot of work ...
> 
> M.
> 
>

On Wednesday 28 September 2005 00:17, Walker, Bruce J (HP-Labs)
wrote:
> I would like to host a phone meeting to discuss the plans for the xen
> sub-architecture of x86.  Hopefully Chris can share his vision and we
> can agree on some next steps to getting this done and into the mainline
> kernel.  From my understanding, it would be good if the "To" list
could
> make it.   If anyone believes others should attend, please let them
> know.
> Please let me know if you are going to call in (no doubt some limit on
> the bridge).
There has been quite a lot of talk about this in the past already
and the situation hasn't changed very much since then. 

It's just a lot of details to be worked out now for which
conference calls are usually not very good.

I would suggest if you have any specific issues you just post
them to the mailing list.

-Andi

> Very very roughly, what 
> I was thinking was to make (and keep maintained a stack of 
> (in order)):
> 
> mainline kernel + things going upstream + rest of xen
> 
> And keep that stack working. To put it another way, we need 
> to keep slicing stuff off the bottom of the Xen patch and merging it.
> 
> I don't think we have a working stack in that form at the 
> moment (at least not on top of any recent mainline kernel), 
> which is my main concern right now (can't really do much without it). 
> 
> Would be splendid if we can keep that up-to-date with every 
> major release, plus -rc releases if possible. I know it's a 
> lot of work ...
Starting with 2.6.14 we're planning on maintaining the Xen subarch in a
Linux hg tree, moving away from the current sparse tree. We're planning
a 'big bang' changeover where we do a course grained re-organisation of
the files (based on Chris Wright's work) and sync up with 2.6.14.

We'll actually have two linux trees, one based on the most recent stable
Linus release that xen subarch fixes/optimiations will go into, and one
that closely tracks the Linux git tree that will be used for subarch
merges and cleanups. We should be able to pull stuff from the former
into the latter easily. We're hoping that this tree will serve as a
focal point to get people collaborating on cleaning up the subarch code.

Using mercurial rather than git isn't a big deal as its easy to export
from one to the other and most people are more comfortable using hg.


Ian

I'm very encouraged that "we all know what needs doing" and
I'm quite
happy not to have the call.  I am concerned, however, that progress
appears to have been very slow (Chris is currently on leave and very
unavailable).  At the Xen mini-summit at OLS, Ian said it was urgent for
Xen's success to get the hooks into the kernel and pleaded for help.
Since then little has visibly happened (Zach has made a little
progress).  How is this going to get done?   

So.  I'm cancelling the call but plead for someone to lead the effort
(or at least explain what needs doing).  We are willing to help but need
some direction.

Bruce walker
Hewlett Packard

P.s. If at any time it does make sense to have a call, I'm willing to
host it.


-----Original Message-----
From: Pratap Subrahmanyam [mailto:pratap@vmware.com] 
Sent: Wednesday, September 28, 2005 9:03 AM
To: Rik van Riel
Cc: Andi Kleen; steven@xensource.com; cgriffin@novell.com; Walker, Bruce
J (HP-Labs); Chris Wright; virtualization@lists.osdl.org;
pratap@vmware.com
Subject: Re: Phone meeting about kernel virtualization hooks

Rik van Riel wrote:
>Seconded.  We all know what needs doing, lets get together using the 
>usual ways (mailing list, #virtualization on OFTC) and get working.
>
>OTOH, if we have a specific agenda for tomorrow's call and it makes 
>sense to discuss those items by phone, I'll try to be there.
>
>  
>
I too feel the same. In addition, I have a conflicting appt, so I may
not be able attend the full hour. Could you please share the agenda, or
elaborate the specific issues that you have in mind? If these are
pressing issues, that cannot be discussed over email, I will try to make
time for the call.

Pratap

I would like to host a phone meeting to discuss the plans for the xen
sub-architecture of x86.  Hopefully Chris can share his vision and we
can agree on some next steps to getting this done and into the mainline
kernel.  From my understanding, it would be good if the "To" list
could
make it.   If anyone believes others should attend, please let them
know.
Please let me know if you are going to call in (no doubt some limit on
the bridge).

Date/time:  Thursday Oct 29  9am pacific, noon eastern, etc.

Call in number:
1-800-826-6074
Or 1-702-387-3409

Pin: 387833427#

Thanks,
   bruce walker
   Hewlett Packard
   310-955-3002