I''m sure everyone has seen the drop of VMI patches for Linux at this point, but just in case, the link is included below. I''ve read this version of the VMI spec and have made my way through most of the patches. While I wasn''t really that impressed with the first spec wrt Xen, the second version seems to be much more palatable. Specifically, the code inlining and afterburner-style padding seems like a really promising approach to native-speed single kernel images. Also, this version seems much more friendly to p2m. There are still a few things missing (like guest DMA support) but I think the basic ideas are pretty sane. So what does everyone else think? Is there anything within VMI that would inhibit some of Xen''s optimizations? Are there any disadvantages to a VMI-style approach to the subarch changes? How close are we to being able to merge our stuff with mainline? Have we gotten feedback yet on how hard this is going to be? Would VMI be an easier approach to inclusion in mainline? Just thought it would be prudent to start a discussion here, at least, about it... http://lkml.org/lkml/2006/3/13/140 Regards, Anthony Liguori _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Just some food for thought, I did a simple analysis of the size of Xen Linux patches verses the VMI patches. To make things more fair, I removed everything from the XenoLinux port accept for the i386 xen subarch (so no drivers and no support for other architectures). VMI: 124 files changed, 4964 insertions(+), 623 deletions(-) Xen: 185 files changed, 31586 insertions(+), 142 deletions(-) So the Xen port adds 6 times more code than VMI. I certainly don''t think VMI for Xen is going to be as fast as native Xen but I don''t know of anything that would cause a substantial change in VMI to add the necessary optimizations that would result in a massive change in the size of the patches. I should also mention that one should also consider the size of the VMI Xen ROM too. For the L4-based ROM that''s going to be ~10k lines but I expect that if the Xen hypercalls were adjusted a little bit, it would drop much less. For instance, Xen already does platform device emulation in the hypervisor for HVM domains so if that were reused it would knock out a fair amount of the ROM code. It seems like there are some merits to the VMI approach. Is there something I''m missing? I admit I don''t understand the XenoLinux changes well enough to know with certainity if there''s something major that justifies the difference in size. I''m hoping someone can hit me with a clue stick though if there is :-) Regards, Anthony Liguori Anthony Liguori wrote:> I''m sure everyone has seen the drop of VMI patches for Linux at this > point, but just in case, the link is included below. > > I''ve read this version of the VMI spec and have made my way through > most of the patches. While I wasn''t really that impressed with the > first spec wrt Xen, the second version seems to be much more > palatable. Specifically, the code inlining and afterburner-style > padding seems like a really promising approach to native-speed single > kernel images. Also, this version seems much more friendly to p2m. > > There are still a few things missing (like guest DMA support) but I > think the basic ideas are pretty sane. So what does everyone else > think? Is there anything within VMI that would inhibit some of Xen''s > optimizations? Are there any disadvantages to a VMI-style approach to > the subarch changes? > > How close are we to being able to merge our stuff with mainline? Have > we gotten feedback yet on how hard this is going to be? Would VMI be > an easier approach to inclusion in mainline? > > Just thought it would be prudent to start a discussion here, at least, > about it... > > http://lkml.org/lkml/2006/3/13/140 > > Regards, > > Anthony Liguori > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> Just some food for thought, I did a simple analysis of the > size of Xen Linux patches verses the VMI patches. To make > things more fair, I removed everything from the XenoLinux > port accept for the i386 xen subarch (so no drivers and no > support for other architectures).Your analysis is still taking all the patches to make dom0 functionality work, which accounts for a lot of changes and is totally outside of the scope of VMI. We''ve been working with a bunch of RH/Novell folks to create a stripped down domU-only Xen patch that would be a fairer comparison. We''re currently looking though the new VMI patchset to see whether there''s anything worth having, in particular, if there''s anything that should be added to our patch to support what we can deduce that their hypervisor probably needs. We''ve also talking to the VirtualIron team to make sure we''re inclusive as possible. The current VMI patchset couldn''t support Xen''s direct-mode (non shadow) MMU virtualization, and hasn''t really been thought through properly for SMP. I''m sure we''ll get something worked out that keeps everyone happy. Ian> VMI: > > 124 files changed, 4964 insertions(+), 623 deletions(-) > > Xen: > > 185 files changed, 31586 insertions(+), 142 deletions(-) > > So the Xen port adds 6 times more code than VMI. I certainly > don''t think VMI for Xen is going to be as fast as native Xen > but I don''t know of anything that would cause a substantial > change in VMI to add the necessary optimizations that would > result in a massive change in the size of the patches. > > I should also mention that one should also consider the size > of the VMI Xen ROM too. For the L4-based ROM that''s going to > be ~10k lines but I expect that if the Xen hypercalls were > adjusted a little bit, it would drop much less. For > instance, Xen already does platform device emulation in the > hypervisor for HVM domains so if that were reused it would > knock out a fair amount of the ROM code. > > It seems like there are some merits to the VMI approach. Is > there something I''m missing? I admit I don''t understand the > XenoLinux changes well enough to know with certainity if > there''s something major that justifies the difference in > size. I''m hoping someone can hit me with a clue stick though > if there is :-) > > Regards, > > Anthony Liguori > > Anthony Liguori wrote: > > I''m sure everyone has seen the drop of VMI patches for > Linux at this > > point, but just in case, the link is included below. > > > > I''ve read this version of the VMI spec and have made my way through > > most of the patches. While I wasn''t really that impressed with the > > first spec wrt Xen, the second version seems to be much more > > palatable. Specifically, the code inlining and afterburner-style > > padding seems like a really promising approach to > native-speed single > > kernel images. Also, this version seems much more friendly to p2m. > > > > There are still a few things missing (like guest DMA support) but I > > think the basic ideas are pretty sane. So what does everyone else > > think? Is there anything within VMI that would inhibit > some of Xen''s > > optimizations? Are there any disadvantages to a VMI-style > approach to > > the subarch changes? > > > > How close are we to being able to merge our stuff with > mainline? Have > > we gotten feedback yet on how hard this is going to be? > Would VMI be > > an easier approach to inclusion in mainline? > > > > Just thought it would be prudent to start a discussion > here, at least, > > about it... > > > > http://lkml.org/lkml/2006/3/13/140 > > > > Regards, > > > > Anthony Liguori > > > > _______________________________________________ > > Xen-devel mailing list > > Xen-devel@lists.xensource.com > > http://lists.xensource.com/xen-devel > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Thanks for the response Ian! Ian Pratt wrote:> Your analysis is still taking all the patches to make dom0 functionality > work, which accounts for a lot of changes and is totally outside of the > scope of VMI. >Could you elaborate a little? Do you mean direct access to hardware or the domain management stuff? I would expect that it''s the former since the later should be well isolated in privcmd.> We''ve been working with a bunch of RH/Novell folks to create a stripped > down domU-only Xen patch that would be a fairer comparison. >Ah, excellent, is there code available anywhere? Has there been progress on microxen?> We''re currently looking though the new VMI patchset to see whether > there''s anything worth having, in particular, if there''s anything that > should be added to our patch to support what we can deduce that their > hypervisor probably needs.Excellent. The thing that stuck out to me the most is binary rewriting for native code. There seems that there has been a bit of performance work done to suggest that that''s the best way to ensure that there''s no different on native.> We''ve also talking to the VirtualIron team to > make sure we''re inclusive as possible. >Sweet.> The current VMI patchset couldn''t support Xen''s direct-mode (non shadow) > MMU virtualization, and hasn''t really been thought through properly for > SMP. I''m sure we''ll get something worked out that keeps everyone happy. >I hope this doesn''t appear too prodding. There seemed to be very little technical objection on LKML and I fear that Xen is not being represented well enough. A note on LKML explaining the problems and how Xen solves them would certainly being really useful. Regards, Anthony Liguori> Ian > > >> VMI: >> >> 124 files changed, 4964 insertions(+), 623 deletions(-) >> >> Xen: >> >> 185 files changed, 31586 insertions(+), 142 deletions(-) >> >> So the Xen port adds 6 times more code than VMI. I certainly >> don''t think VMI for Xen is going to be as fast as native Xen >> but I don''t know of anything that would cause a substantial >> change in VMI to add the necessary optimizations that would >> result in a massive change in the size of the patches. >> >> I should also mention that one should also consider the size >> of the VMI Xen ROM too. For the L4-based ROM that''s going to >> be ~10k lines but I expect that if the Xen hypercalls were >> adjusted a little bit, it would drop much less. For >> instance, Xen already does platform device emulation in the >> hypervisor for HVM domains so if that were reused it would >> knock out a fair amount of the ROM code. >> >> It seems like there are some merits to the VMI approach. Is >> there something I''m missing? I admit I don''t understand the >> XenoLinux changes well enough to know with certainity if >> there''s something major that justifies the difference in >> size. I''m hoping someone can hit me with a clue stick though >> if there is :-) >> >> Regards, >> >> Anthony Liguori >> >> Anthony Liguori wrote: >> >>> I''m sure everyone has seen the drop of VMI patches for >>> >> Linux at this >> >>> point, but just in case, the link is included below. >>> >>> I''ve read this version of the VMI spec and have made my way through >>> most of the patches. While I wasn''t really that impressed with the >>> first spec wrt Xen, the second version seems to be much more >>> palatable. Specifically, the code inlining and afterburner-style >>> padding seems like a really promising approach to >>> >> native-speed single >> >>> kernel images. Also, this version seems much more friendly to p2m. >>> >>> There are still a few things missing (like guest DMA support) but I >>> think the basic ideas are pretty sane. So what does everyone else >>> think? Is there anything within VMI that would inhibit >>> >> some of Xen''s >> >>> optimizations? Are there any disadvantages to a VMI-style >>> >> approach to >> >>> the subarch changes? >>> >>> How close are we to being able to merge our stuff with >>> >> mainline? Have >> >>> we gotten feedback yet on how hard this is going to be? >>> >> Would VMI be >> >>> an easier approach to inclusion in mainline? >>> >>> Just thought it would be prudent to start a discussion >>> >> here, at least, >> >>> about it... >>> >>> http://lkml.org/lkml/2006/3/13/140 >>> >>> Regards, >>> >>> Anthony Liguori >>> >>> _______________________________________________ >>> Xen-devel mailing list >>> Xen-devel@lists.xensource.com >>> http://lists.xensource.com/xen-devel >>> >>_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ian Pratt wrote:>> Just some food for thought, I did a simple analysis of the >> size of Xen Linux patches verses the VMI patches. To make >> things more fair, I removed everything from the XenoLinux >> port accept for the i386 xen subarch (so no drivers and no >> support for other architectures). >> > > Your analysis is still taking all the patches to make dom0 functionality > work, which accounts for a lot of changes and is totally outside of the > scope of VMI. > > We''ve been working with a bunch of RH/Novell folks to create a stripped > down domU-only Xen patch that would be a fairer comparison. > > We''re currently looking though the new VMI patchset to see whether > there''s anything worth having, in particular, if there''s anything that > should be added to our patch to support what we can deduce that their > hypervisor probably needs. We''ve also talking to the VirtualIron team to > make sure we''re inclusive as possible. > > The current VMI patchset couldn''t support Xen''s direct-mode (non shadow) > MMU virtualization, and hasn''t really been thought through properly for > SMP. I''m sure we''ll get something worked out that keeps everyone happy. >Actually, the new spec does support non-shadow paging. All page table updates are done via the VMI ROM. Normally, these are just normal get/set operations but they could be get/sets with p2m looks. The first version of the spec definitely didn''t support non-shadow paging though. We''ll see whether it''s actually implementable when the wedge is released though. I wouldn''t be surprised if they are using shadow paging. The new spec does actually introduce SMP stuff but yeah, you''re probably right, it may not have been thought through enough... The biggest gaps that I can find appear to be DMA support and event channels. I think to properly support event channel you need a bit more pervasive changes. I think DMA is a pretty simply change (you just have to update the DMA bus translation functions to do a p2m lookup). Regards, Anthony Liguori> Ian >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
* Ian Pratt (m+Ian.Pratt@cl.cam.ac.uk) wrote:> > Just some food for thought, I did a simple analysis of the > > size of Xen Linux patches verses the VMI patches. To make > > things more fair, I removed everything from the XenoLinux > > port accept for the i386 xen subarch (so no drivers and no > > support for other architectures). > > Your analysis is still taking all the patches to make dom0 functionality > work, which accounts for a lot of changes and is totally outside of the > scope of VMI. > > We''ve been working with a bunch of RH/Novell folks to create a stripped > down domU-only Xen patch that would be a fairer comparison.Here''s a really imprecise view into that. Keeping with Anthony''s metrics: 88 files changed, 7637 insertions(+), 436 deletions(-) That''s near meaningless, as it''s not something that would compile and run, but gives you a scope of the size of the changes for domU only (shadow mode), no Xen drivers, etc.. thanks, -chris _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ian Pratt wrote:> The current VMI patchset couldn''t support Xen''s direct-mode (non shadow) > MMU virtualization, and hasn''t really been thought through properly for > SMP. I''m sure we''ll get something worked out that keeps everyone happy.I''m not sure what you mean that VMI "hasn''t really been thought through properly for SMP." I''m writing this email from an SMP VMI guest. Using page table shadows makes an SMP implementation really easy. Page table writes can go directly to the virtual page tables. The guest needs to inform the hypervisor which page table entries have been modified, but actually processing these entries and updating the hardware page tables can be deferred until page invalidate, cr3 reassignment, or TLB shootdown time. All of the details here are hidden behind the VMI "set page table entry" call. The guest just uses this call and everything happens on its behalf. We give each VCPU a private queue for notifying the hypervisor of page table updates and other types of CPU state changes that can be deferred. Because we use per-VCPU page table shadows, we''re able to put a private queue in each VCPU at the same linear address space. Updates to the queue thus are just a couple of memory writes, and allows very efficient batching of page table writes, even for SMP. The hypervisor and the ROM contain all the knowledge of this interface (rather than the guest), so it would be relatively easy to change to support shared shadow page tables, or a different mechanism for direct-mode page tables. We''re obviously lacking some interfaces needed for direct-mode paging, but aside from those, are there specific deficiencies for SMP? (Apologies if quoting/replies aren''t set up properly; I wasn''t previously subscribed to xen-devel, and had this forwarded by a colleague.) Dan. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Chris Wright wrote:> * Ian Pratt (m+Ian.Pratt@cl.cam.ac.uk) wrote: > >>> Just some food for thought, I did a simple analysis of the >>> size of Xen Linux patches verses the VMI patches. To make >>> things more fair, I removed everything from the XenoLinux >>> port accept for the i386 xen subarch (so no drivers and no >>> support for other architectures). >>> >> Your analysis is still taking all the patches to make dom0 functionality >> work, which accounts for a lot of changes and is totally outside of the >> scope of VMI. >> >> We''ve been working with a bunch of RH/Novell folks to create a stripped >> down domU-only Xen patch that would be a fairer comparison. >> > > Here''s a really imprecise view into that. Keeping with Anthony''s > metrics: > > 88 files changed, 7637 insertions(+), 436 deletions(-) >Interesting.> That''s near meaningless, as it''s not something that would compile and > run, but gives you a scope of the size of the changes for domU only > (shadow mode), no Xen drivers, etc.. >Okay, but that is on par with the amount of changes present in VMI. That''s a really interesting data point. Thanks, Anthony Liguori> thanks, > -chris >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 17 Mar 2006, at 00:10, Anthony Liguori wrote:>> That''s near meaningless, as it''s not something that would compile and >> run, but gives you a scope of the size of the changes for domU only >> (shadow mode), no Xen drivers, etc.. >> > Okay, but that is on par with the amount of changes present in VMI. > That''s a really interesting data point.Omitting the Xen drivers will account for a fair bit of that reduction. Including those in the VMI comparison is unfair since the VMI interfaces consider core architectural subsystems, not I/O. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 17 Mar 2006, at 08:53, Keir Fraser wrote:>> Okay, but that is on par with the amount of changes present in VMI. >> That''s a really interesting data point. > > Omitting the Xen drivers will account for a fair bit of that > reduction. Including those in the VMI comparison is unfair since the > VMI interfaces consider core architectural subsystems, not I/O.Ah, you omitted them already. Duh. :-) -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 16 Mar 2006, at 23:33, Anthony Liguori wrote:> Actually, the new spec does support non-shadow paging. All page table > updates are done via the VMI ROM. Normally, these are just normal > get/set operations but they could be get/sets with p2m looks. The > first version of the spec definitely didn''t support non-shadow paging > though.I don''t see any hooks for pte reads, which need m2p conversion in a non-shadow environment. And the documentation is out of whack with the code patches now I look more closely at the mmu patch. VMI_xxxPxExxx functions are not defined in the spec, nor is VMI_AllocatePage. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser wrote:> > On 16 Mar 2006, at 23:33, Anthony Liguori wrote: > >> Actually, the new spec does support non-shadow paging. All page >> table updates are done via the VMI ROM. Normally, these are just >> normal get/set operations but they could be get/sets with p2m looks. >> The first version of the spec definitely didn''t support non-shadow >> paging though. > > I don''t see any hooks for pte reads, which need m2p conversion in a > non-shadow environment.VMI_CALL_GetPxE is the hook for pte reads I think. The documentation refers to a SetPte function but that doesn''t actually exist in the code. Instead, I believe the pte access functions are GetPxE and SetPxE and that they can be used for setting ptes or pmds. It''s not really documented though so I may be wrong.> And the documentation is out of whack with the code patches now I look > more closely at the mmu patch. VMI_xxxPxExxx functions are not defined > in the spec, nor is VMI_AllocatePage.Yeah, the documentation is lacking (and wrong) in a couple of key areas. Regards, Anthony Liguori> -- Keir >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Christian pointed out that the current Xen tree copies a number of files and makes changes to them (renaming them to -xen). It''s unclear how to properly count this in my mind since if they went into the tree as-is today, it would be a pure addition. Presumably, they would go in with some conditional logic which would reduce the total insertion count (dramatically) but also increase the diff size. This probably helps explains why Chris is seeing such a dramatic reduction too. It''s becoming clear to me that when reworked, the Xen changes are pretty much identical in code-size to VMI (give or take a couple thousand slocs). Regards, Anthony Liguori Anthony Liguori wrote:> Just some food for thought, I did a simple analysis of the size of > Xen Linux patches verses the VMI patches. To make things more fair, I > removed everything from the XenoLinux port accept for the i386 xen > subarch (so no drivers and no support for other architectures). > > VMI: > > 124 files changed, 4964 insertions(+), 623 deletions(-) > > Xen: > > 185 files changed, 31586 insertions(+), 142 deletions(-) > > So the Xen port adds 6 times more code than VMI. I certainly don''t > think VMI for Xen is going to be as fast as native Xen but I don''t > know of anything that would cause a substantial change in VMI to add > the necessary optimizations that would result in a massive change in > the size of the patches. > > I should also mention that one should also consider the size of the > VMI Xen ROM too. For the L4-based ROM that''s going to be ~10k lines > but I expect that if the Xen hypercalls were adjusted a little bit, it > would drop much less. For instance, Xen already does platform device > emulation in the hypervisor for HVM domains so if that were reused it > would knock out a fair amount of the ROM code. > > It seems like there are some merits to the VMI approach. Is there > something I''m missing? I admit I don''t understand the XenoLinux > changes well enough to know with certainity if there''s something major > that justifies the difference in size. I''m hoping someone can hit me > with a clue stick though if there is :-) > > Regards, > > Anthony Liguori > > Anthony Liguori wrote: >> I''m sure everyone has seen the drop of VMI patches for Linux at this >> point, but just in case, the link is included below. >> >> I''ve read this version of the VMI spec and have made my way through >> most of the patches. While I wasn''t really that impressed with the >> first spec wrt Xen, the second version seems to be much more >> palatable. Specifically, the code inlining and afterburner-style >> padding seems like a really promising approach to native-speed single >> kernel images. Also, this version seems much more friendly to p2m. >> >> There are still a few things missing (like guest DMA support) but I >> think the basic ideas are pretty sane. So what does everyone else >> think? Is there anything within VMI that would inhibit some of Xen''s >> optimizations? Are there any disadvantages to a VMI-style approach >> to the subarch changes? >> >> How close are we to being able to merge our stuff with mainline? >> Have we gotten feedback yet on how hard this is going to be? Would >> VMI be an easier approach to inclusion in mainline? >> >> Just thought it would be prudent to start a discussion here, at >> least, about it... >> >> http://lkml.org/lkml/2006/3/13/140 >> >> Regards, >> >> Anthony Liguori >> >> _______________________________________________ >> Xen-devel mailing list >> Xen-devel@lists.xensource.com >> http://lists.xensource.com/xen-devel > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel