Linux Virtualization - Sep 2007 - [RFC] KVM Source layout Proposal to accommodate new CPU architecture

Zhang, Xiantao wrote:
> Hi Folks,
> 	We are working on enabling KVM support on IA64 platform, and now
> Linux, Windows guests get stable run and achieve reasonable performance
> on KVM with Open GFW. But you know, the current KVM only considers x86
> platform, and is short of cross-architecture framework. Currently, we
> have a proposal for KVM source layout to accommodate new CPU
> architectures. Attached foil describes the detail. With our proposal, we
> can boot x86 guests based on commit
> 2e278972a11eb14f031dea242a9ed118adfa0932, also didn't see regressions.
> For IA64 side, we are rebasing our code to this framework. 
> Main changes to current source:
> 1. Add subdirectories, such as x86 and ia64 to hold arch-specific code.
> 2. Split kvm_main.c to two parts. One is still called kvm_main.c, just
> contains KVM common interfaces with user space, and basic KVM
> infrastructure. The other one is named as kvm_arch.c under sub-directory
> (eg. X86, ia64 etc), which includes arch-specific code to supplement the
> functionality of kvm_main.c
> 3. Add an "include" directory in drivers/kvm. Due to possibly
complex
> code logic in KVM source, maybe many header files need to maintain for
> some architectures. If we put them under top-level include/asm-arch
> directory, it may introduce much more maintain effort. So, we put it
> under "drivers/kvm", and let it be effective when kernel
configuration
> time.
> BTW, Userspace code changes are not involved in this thread. 
> Considering the readability, we didn't attach the diff file in the
mail,
> due to big changes to kvm source structure, and only post the tarball
> including whole directory "drivers/kvm" instead. For comparison,
I
> attached kvm_main.diff as well. 
>
> Any comments are appreciated from you! Hope to see IA64 support on KVM
> earlier!
>   
The whole drivers/kvm/ thing was just a trick to get merged quickly.  I
think the new layout should be something like

  virt/kvm/, include/linux/kvm*.h -> common code
  virt/lguest/ -> the other hypervisor
  virt/virtio/ -> shared I/O infrastructure
  virt/ -> the CONFIG_VIRTIALIZATION menu
  arch/x86/kvm/, include/asm-x86/ -> x86 specific code
  arch/ia64/kvm/, include/asm-ia64/ -> ia64 specific code

etc.

Of course, this depends on the x86 merge which is scheduled for early
2.6.24.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to
panic.

Zhang, Xiantao wrote:
> Hi Avi,
> 	Sound good! But what can we do before the merge? You know, we have to
spend much effort maintaining our patches with sync with upstream tree. Do you
have an interim solution or proposal for merging IA64 code?  Thanks.
> Xiantao 
>   
The merge is due in a few weeks. If that is too far, we can push the x86
parts to arch/i386 and do makefile magic so that x86-64 sees it too.

> -----Original Message-----
> From: Avi Kivity [mailto:avi@qumranet.com] 
> Sent: 2007?9?27? 17:19
> To: Zhang, Xiantao
> Cc: kvm-devel@lists.sourceforge.net; virtualization
> Subject: Re: [RFC] KVM Source layout Proposal to accommodate new CPU
architecture
>
> Zhang, Xiantao wrote:
>   
>> Hi Folks,
>> 	We are working on enabling KVM support on IA64 platform, and now
>> Linux, Windows guests get stable run and achieve reasonable performance
>> on KVM with Open GFW. But you know, the current KVM only considers x86
>> platform, and is short of cross-architecture framework. Currently, we
>> have a proposal for KVM source layout to accommodate new CPU
>> architectures. Attached foil describes the detail. With our proposal,
we
>> can boot x86 guests based on commit
>> 2e278972a11eb14f031dea242a9ed118adfa0932, also didn't see
regressions.
>> For IA64 side, we are rebasing our code to this framework. 
>> Main changes to current source:
>> 1. Add subdirectories, such as x86 and ia64 to hold arch-specific code.
>> 2. Split kvm_main.c to two parts. One is still called kvm_main.c, just
>> contains KVM common interfaces with user space, and basic KVM
>> infrastructure. The other one is named as kvm_arch.c under
sub-directory
>> (eg. X86, ia64 etc), which includes arch-specific code to supplement
the
>> functionality of kvm_main.c
>> 3. Add an "include" directory in drivers/kvm. Due to possibly
complex
>> code logic in KVM source, maybe many header files need to maintain for
>> some architectures. If we put them under top-level include/asm-arch
>> directory, it may introduce much more maintain effort. So, we put it
>> under "drivers/kvm", and let it be effective when kernel
configuration
>> time.
>> BTW, Userspace code changes are not involved in this thread. 
>> Considering the readability, we didn't attach the diff file in the
mail,
>> due to big changes to kvm source structure, and only post the tarball
>> including whole directory "drivers/kvm" instead. For
comparison, I
>> attached kvm_main.diff as well. 
>>
>> Any comments are appreciated from you! Hope to see IA64 support on KVM
>> earlier!
>>   
>>     
>
> The whole drivers/kvm/ thing was just a trick to get merged quickly.  I
> think the new layout should be something like
>
>   virt/kvm/, include/linux/kvm*.h -> common code
>   virt/lguest/ -> the other hypervisor
>   virt/virtio/ -> shared I/O infrastructure
>   virt/ -> the CONFIG_VIRTIALIZATION menu
>   arch/x86/kvm/, include/asm-x86/ -> x86 specific code
>   arch/ia64/kvm/, include/asm-ia64/ -> ia64 specific code
>
> etc.
>
> Of course, this depends on the x86 merge which is scheduled for early
> 2.6.24.
>
>   

-- 
Any sufficiently difficult bug is indistinguishable from a feature.

Hi Avi,
	Sound good! But what can we do before the merge? You know, we have to spend
much effort maintaining our patches with sync with upstream tree. Do you have an
interim solution or proposal for merging IA64 code?  Thanks.
Xiantao 
-----Original Message-----
From: Avi Kivity [mailto:avi@qumranet.com] 
Sent: 2007?9?27? 17:19
To: Zhang, Xiantao
Cc: kvm-devel@lists.sourceforge.net; virtualization
Subject: Re: [RFC] KVM Source layout Proposal to accommodate new CPU
architecture

Zhang, Xiantao wrote:
> Hi Folks,
> 	We are working on enabling KVM support on IA64 platform, and now
> Linux, Windows guests get stable run and achieve reasonable performance
> on KVM with Open GFW. But you know, the current KVM only considers x86
> platform, and is short of cross-architecture framework. Currently, we
> have a proposal for KVM source layout to accommodate new CPU
> architectures. Attached foil describes the detail. With our proposal, we
> can boot x86 guests based on commit
> 2e278972a11eb14f031dea242a9ed118adfa0932, also didn't see regressions.
> For IA64 side, we are rebasing our code to this framework. 
> Main changes to current source:
> 1. Add subdirectories, such as x86 and ia64 to hold arch-specific code.
> 2. Split kvm_main.c to two parts. One is still called kvm_main.c, just
> contains KVM common interfaces with user space, and basic KVM
> infrastructure. The other one is named as kvm_arch.c under sub-directory
> (eg. X86, ia64 etc), which includes arch-specific code to supplement the
> functionality of kvm_main.c
> 3. Add an "include" directory in drivers/kvm. Due to possibly
complex
> code logic in KVM source, maybe many header files need to maintain for
> some architectures. If we put them under top-level include/asm-arch
> directory, it may introduce much more maintain effort. So, we put it
> under "drivers/kvm", and let it be effective when kernel
configuration
> time.
> BTW, Userspace code changes are not involved in this thread. 
> Considering the readability, we didn't attach the diff file in the
mail,
> due to big changes to kvm source structure, and only post the tarball
> including whole directory "drivers/kvm" instead. For comparison,
I
> attached kvm_main.diff as well. 
>
> Any comments are appreciated from you! Hope to see IA64 support on KVM
> earlier!
>   
The whole drivers/kvm/ thing was just a trick to get merged quickly.  I
think the new layout should be something like

  virt/kvm/, include/linux/kvm*.h -> common code
  virt/lguest/ -> the other hypervisor
  virt/virtio/ -> shared I/O infrastructure
  virt/ -> the CONFIG_VIRTIALIZATION menu
  arch/x86/kvm/, include/asm-x86/ -> x86 specific code
  arch/ia64/kvm/, include/asm-ia64/ -> ia64 specific code

etc.

Of course, this depends on the x86 merge which is scheduled for early
2.6.24.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to
panic.

> Zhang, Xiantao wrote:
> 	We are working on enabling KVM support on IA64 platform, and now
> Linux, Windows guests get stable run and achieve reasonable performance
> on KVM with Open GFW. But you know, the current KVM only considers x86
> platform, and is short of cross-architecture framework. Currently, we
> have a proposal for KVM source layout to accommodate new CPU
> architectures.
That's great. I agree that general restructure of current x86 code is 
needed to fit different archs proper. I do strongly appreciate your 
efforts towards this.
> 1. Add subdirectories, such as x86 and ia64 to hold arch-specific code.
I would prefer Avis move to /virt prior to that. But then we'll need 
arch specific subdirectories. I think they should go to 
/arch/<arch>/kvm. That would involve the architecure maintainers, 
which gives us more peer review.
> 2. Split kvm_main.c to two parts. One is still called kvm_main.c, just
> contains KVM common interfaces with user space, and basic KVM
> infrastructure. The other one is named as kvm_arch.c under sub-directory
> (eg. X86, ia64 etc), which includes arch-specific code to supplement the
> functionality of kvm_main.c
I disagree with the split you've made. I think we should try to keep 
as much as possible common, rather then just duplicating the efforts 
for each architecture we have. Thus, I do prefer to refine a clean 
architecutre backend interface based on the current vmx/svm split. We 
just need to move x86 specifics to a "kvm-x86" library, on which 
kvm-intel, kvm-amd and maybe kvm-rusty do depend. Interfacing to there 
needs to go via the same function vector we use for svm/vmx today.
> 3. Add an "include" directory in drivers/kvm. Due to possibly
complex
> code logic in KVM source, maybe many header files need to maintain for
> some architectures. If we put them under top-level include/asm-arch
> directory, it may introduce much more maintain effort. So, we put it
> under "drivers/kvm", and let it be effective when kernel
configuration
> time.
To me, they clearly belong to include/arch.

so long,
Carsten

Zhang, Xiantao wrote:
> Hi Avi,
> 	So you mean IA64 can adopt the similar method as well? 
What method do you mean exactly?
> But even so, I am still thinking we have to come out a solution for
checking IA64 code into existing KVM upstream tree , because KVM infrastructure
in mainline Linux may been a long way to go. Moreover, we also have no method to
avoid the abstraction for existing code to support cross-architecture framework
before what you pointed gets happen. For example, kvm_main.c is also required to
split as common interface for all architectures. So, I think it has no big
conflicts between our proposal and the final KVM infrastructure in mainline
Linux. How about your ideas? :)
>   
The powerpc people had some patches to make kvm_main arch independent.
We should work on that base.

To avoid a dependency on the x86 merge, we can start by working withing
drivers/kvm/, for example creating drivers/kvm/x86.c and
drivers/kvm/ia64.c. Later patches can move these to arch/*/.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to
panic.

On Thu, 2007-09-27 at 11:18 +0200, Avi Kivity wrote:
> The whole drivers/kvm/ thing was just a trick to get merged quickly.  I
> think the new layout should be something like
> 
>   virt/kvm/, include/linux/kvm*.h -> common code
>   virt/lguest/ -> the other hypervisor
>   virt/virtio/ -> shared I/O infrastructure
>   virt/ -> the CONFIG_VIRTIALIZATION menu
>   arch/x86/kvm/, include/asm-x86/ -> x86 specific code
>   arch/ia64/kvm/, include/asm-ia64/ -> ia64 specific code
The problem with this separation is that module source cannot span
directories (at least, not that I could find).

This is why lguest went for "i386_" prefix for arch separation.

Cheers,
Rusty.

Zhang, Xiantao wrote:
> Zhang, Xiantao wrote:
>   
>>> Hi Avi,
>>> 	So you mean IA64 can adopt the similar method as well? 
>>>       
>
>   
>> What method do you mean exactly?
>>     
> Put all arch-specific files into arch/ia64/kvm as you described in
> future KVM infrastructure. 
>   
>> The powerpc people had some patches to make kvm_main arch independent.
>> We should work on that base.
>> To avoid a dependency on the x86 merge, we can start by working withing
>> drivers/kvm/, for example creating drivers/kvm/x86.c and
>> drivers/kvm/ia64.c. Later patches can move these to arch/*/.
>>     
> It may work on x86 side. But for IA64, we have several source files and
> assembly files to implement a VMM module, which contains the
> virtualization logic of CPU, MMU and other platform devices. (In KVM
> forum, Anthony had presented IA64/KVM architecture which is a bit
> different with x86 side due to different approaches for VT.).If we put
> all such these arch-specific files in one directory, it looks very
> strange! 
>   
ia64/ subdirectory is also fine.


-- 
error compiling committee.c: too many arguments to function

Zhang, Xiantao wrote:
> Avi Kivity wrote:
>   
>> Zhang, Xiantao wrote:
>>     
>>> Zhang, Xiantao wrote:
>>>
>>>       
>>>>> Hi Avi,
>>>>> 	So you mean IA64 can adopt the similar method as well?
>>>>>
>>>>>           
>>>       
>>>> What method do you mean exactly?
>>>>
>>>>         
>>> Put all arch-specific files into arch/ia64/kvm as you described in
>>> future KVM infrastructure. 
>>>
>>>       
>>>> The powerpc people had some patches to make kvm_main arch
>>>> independent. We should work on that base. To avoid a dependency
on
>>>> the x86 merge, we can start by working withing drivers/kvm/,
for
>>>> example creating drivers/kvm/x86.c and drivers/kvm/ia64.c.
Later
>>>> patches can move these to arch/*/. 
>>>>
>>>>         
>>> It may work on x86 side. But for IA64, we have several source files
>>> and assembly files to implement a VMM module, which contains the
>>> virtualization logic of CPU, MMU and other platform devices. (In
KVM
>>> forum, Anthony had presented IA64/KVM architecture which is a bit
>>> different with x86 side due to different approaches for VT.).If we
>>> put all such these arch-specific files in one directory, it looks
>>> very strange! 
>>>
>>>       
>> ia64/ subdirectory is also fine.
>>     
>
> But even so , we have to split current code to be arch-independent, and
> to support IA64 and other architectures. 
> So, why not add an more subdirectory x86 in drivers kvm to hold X86-arch
> code?
>   
Sure, that's not an issue.
> And it should also conform with with future infrastructure in Linux.
> Maybe we can borrow the idea from UML code structure.
> Do you think so ?
Eventually I'd like to see the code in arch/*/kvm.  That's probably not 
easily doable right now because modules cannot span directories, but 
once that's solved, we'll do that as this is most consistent with the 
rest of the kernel.


-- 
error compiling committee.c: too many arguments to function

Zhang, Xiantao wrote:
> Hollis Blanchard wrote:
>   
>> On Mon, 2007-10-08 at 10:36 +0800, Zhang, Xiantao wrote:
>>     
>>> Avi Kivity wrote:
>>>       
>>>> Zhang, Xiantao wrote:
>>>>         
>>>>> Avi Kivity wrote:
>>>>>
>>>>>           
>>>>>> Zhang, Xiantao wrote:
>>>>>>
>>>>>>             
>>>>>>> Zhang, Xiantao wrote:
>>>>>>>
>>>>>>>
>>>>>>>               
>>>>>>>>> Hi Avi,
>>>>>>>>> 	So you mean IA64 can adopt the similar
method as well?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                   
>>>>>>>> What method do you mean exactly?
>>>>>>>>
>>>>>>>>
>>>>>>>>                 
>>>>>>> Put all arch-specific files into arch/ia64/kvm as
you described
>>>>>>> in future KVM infrastructure. 
>>>>>>>
>>>>>>>
>>>>>>>               
>>>>>>>> The powerpc people had some patches to make
kvm_main arch
>>>>>>>> independent. We should work on that base. To
avoid a dependency
>>>>>>>> on the x86 merge, we can start by working
withing drivers/kvm/,
>>>>>>>> for example creating drivers/kvm/x86.c and
drivers/kvm/ia64.c.
>>>>>>>> Later patches can move these to arch/*/.
>>>>>>>>
>>>>>>>>
>>>>>>>>                 
>>>>>>> It may work on x86 side. But for IA64, we have
several source
>>>>>>> files and assembly files to implement a VMM module,
which
>>>>>>> contains the virtualization logic of CPU, MMU and
other platform
>>>>>>> devices. (In KVM forum, Anthony had presented
IA64/KVM
>>>>>>> architecture which is a bit different with x86 side
due to
>>>>>>> different approaches for VT.).If we put all such
these
>>>>>>> arch-specific files in one directory, it looks very
strange!
>>>>>>>
>>>>>>>
>>>>>>>               
>>>>>> ia64/ subdirectory is also fine.
>>>>>>
>>>>>>             
>>>>> But even so , we have to split current code to be
arch-independent,
>>>>> and to support IA64 and other architectures.
>>>>> So, why not add an more subdirectory x86 in drivers kvm to
hold
>>>>> X86-arch code? 
>>>>>
>>>>>           
>>>> Sure, that's not an issue.
>>>>         
>>> Could you help to open a branch from master tree for this work? We
>>> are very willing to contribute to it:)
>>>       
>> Do you really need a new branch? Why not just submit patches?
>>     
>
> Due to big changes to current source structure, maybe a new branch would
> help to work, and doesn't
> impact existing quality of KVM.  If it is convenient for you to submit
> patches directly, also we are glad to do in that way.
>   
A branch with such large changes quickly becomes out-of-date, so it's 
best to send patches.

-- 
Any sufficiently difficult bug is indistinguishable from a feature.

On Mon, 2007-10-08 at 10:36 +0800, Zhang, Xiantao wrote:
> Avi Kivity wrote:
> > Zhang, Xiantao wrote:
> >> Avi Kivity wrote:
> >> 
> >>> Zhang, Xiantao wrote:
> >>> 
> >>>> Zhang, Xiantao wrote:
> >>>> 
> >>>> 
> >>>>>> Hi Avi,
> >>>>>> 	So you mean IA64 can adopt the similar method as
well?
> >>>>>> 
> >>>>>> 
> >>>> 
> >>>>> What method do you mean exactly?
> >>>>> 
> >>>>> 
> >>>> Put all arch-specific files into arch/ia64/kvm as you
described in
> >>>> future KVM infrastructure. 
> >>>> 
> >>>> 
> >>>>> The powerpc people had some patches to make kvm_main
arch
> >>>>> independent. We should work on that base. To avoid a
dependency on
> >>>>> the x86 merge, we can start by working withing
drivers/kvm/, for
> >>>>> example creating drivers/kvm/x86.c and
drivers/kvm/ia64.c. Later
> >>>>> patches can move these to arch/*/.
> >>>>> 
> >>>>> 
> >>>> It may work on x86 side. But for IA64, we have several
source files
> >>>> and assembly files to implement a VMM module, which
contains the
> >>>> virtualization logic of CPU, MMU and other platform
devices. (In
> >>>> KVM forum, Anthony had presented IA64/KVM architecture
which is a
> >>>> bit different with x86 side due to different approaches
for
> >>>> VT.).If we put all such these arch-specific files in one
> >>>> directory, it looks very strange! 
> >>>> 
> >>>> 
> >>> ia64/ subdirectory is also fine.
> >>> 
> >> 
> >> But even so , we have to split current code to be
arch-independent,
> >> and to support IA64 and other architectures.
> >> So, why not add an more subdirectory x86 in drivers kvm to hold
> >> X86-arch code? 
> >> 
> > 
> > Sure, that's not an issue.
> 
> Could you help to open a branch from master tree for this work? We are
> very willing to contribute to it:)
Do you really need a new branch? Why not just submit patches?

-- 
Hollis Blanchard
IBM Linux Technology Center