Folks, At the last summit I presented a proposal for rearchitecting the way we do io emulation for fully-virtualized (hvm) guests. I''d really like to try and get the work to implement this underway, as it cleans up a bunch of mess, is a prerequisite for save/restore/relocation of hvm guests, and is a precursor to some significant performance improvements. It involves a fair chunk of work, so we really want to try and get multiple folk working on it. The plan is to move the io emulation code (qemu-dm) from running as a user-space app in domain 0 into a ''mini guest'' that is effectively a small paravirtualized guest in the root hardware context associated with each hvm domain. I guess a very high-level work plan would look something like this: * get minios running well on x86_64; add a few simple infrastructure functions e.g. simple memory allocator. No need for any ''user space'' mmu support * port (simplified)xenbus/netfront/blkfront to minios; test simple net/disk IO * implement enough infrastructure to allow qemu-dm to be compiled into minios, calling into net/blkfront for IO. * plumb the vmexit entry points from MMIO and in/out into minios and hence qemu-dm Once the above is working we''ll be in good shape. We can remove all the skany qemu-dm support from the tools as from their POV paravirt and hvm guests will look identical. It should also be easy to implement save/restore of hvm guests -- just save the miniguest as part of the hvm guests'', memory image. The next stage would then be to improve performance by enhancing the device models, e.g. adding a network card that suports jumbo frames and csum offload, and requires fewer vmexits in operation. How best to move forward on this? Any volunteers? Thanks, Ian _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ian, Gregor and I have been working on getting Mini-OS to run on x86_64. Here is the status: - The hypercall interface has been updated to reflect XenLinux. - x86_64 pagetable builder was added. - I am currently working on getting the scheduler to work in x86_64 mode (I will check in something in a couple of days) - I think Gregor is working on console support. Some of the other things that need to be done are: - Fix traps.c a) To show more useful information. b) Fix pagetable walk. - x86_32 PAE - Look into the stack overrun issue. We will shortly also have a couple of more people from Unisys investing time in this effort. Thanks, Aravindh> -----Original Message----- > From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel- > bounces@lists.xensource.com] On Behalf Of Ian Pratt > Sent: Sunday, March 12, 2006 4:26 PM > To: xen-devel > Subject: [Xen-devel] mini-guest io emulation > > > Folks, > > At the last summit I presented a proposal for rearchitecting the waywe> do io emulation for fully-virtualized (hvm) guests. I''d really like to > try and get the work to implement this underway, as it cleans up abunch> of mess, is a prerequisite for save/restore/relocation of hvm guests, > and is a precursor to some significant performance improvements. It > involves a fair chunk of work, so we really want to try and getmultiple> folk working on it. > > The plan is to move the io emulation code (qemu-dm) from running as a > user-space app in domain 0 into a ''mini guest'' that is effectively a > small paravirtualized guest in the root hardware context associatedwith> each hvm domain. > > I guess a very high-level work plan would look something like this: > > * get minios running well on x86_64; add a few simple infrastructure > functions e.g. simple memory allocator. No need for any ''user space''mmu> support > * port (simplified)xenbus/netfront/blkfront to minios; test simple > net/disk IO > * implement enough infrastructure to allow qemu-dm to be compiled into > minios, calling into net/blkfront for IO. > * plumb the vmexit entry points from MMIO and in/out into minios and > hence qemu-dm > > Once the above is working we''ll be in good shape. We can remove allthe> skany qemu-dm support from the tools as from their POV paravirt andhvm> guests will look identical. It should also be easy to implement > save/restore of hvm guests -- just save the miniguest as part of thehvm> guests'', memory image. The next stage would then be to improve > performance by enhancing the device models, e.g. adding a network card > that suports jumbo frames and csum offload, and requires fewer vmexits > in operation. > > How best to move forward on this? Any volunteers? > > Thanks, > Ian > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ian Pratt wrote:> Folks, > > At the last summit I presented a proposal for rearchitecting the way > we do io emulation for fully-virtualized (hvm) guests. I''d really > like to try and get the work to implement this underway, as it cleans > up a bunch of mess, is a prerequisite for save/restore/relocation of > hvm guests, and is a precursor to some significant performance > improvements. It involves a fair chunk of work, so we really want to > try and get multiple folk working on it. > > The plan is to move the io emulation code (qemu-dm) from running as a > user-space app in domain 0 into a ''mini guest'' that is effectively a > small paravirtualized guest in the root hardware context associated > with each hvm domain. > > I guess a very high-level work plan would look something like this: > > * get minios running well on x86_64; add a few simple infrastructure > functions e.g. simple memory allocator. No need for any ''user space'' > mmu support > * port (simplified)xenbus/netfront/blkfront to minios; test simple > net/disk IO > * implement enough infrastructure to allow qemu-dm to be compiled into > minios, calling into net/blkfront for IO. > * plumb the vmexit entry points from MMIO and in/out into minios and > hence qemu-dmFor the "mini guest", I think it could be much easier if we substantially strip down xenlinux rather than adding (eventually) a lot of stuff to the current mini-os, mainly because we need probably a multi-threaded run-time environment, scheduler, memory allocator, event handling, drivers such as xenbus/netfront/blkfront, etc. At least, I think we can use xenlinux as the development platform. For example, implement the qemu-dm as a driver adding the infrastructure required (e.g. small in-kernel glibc).> > Once the above is working we''ll be in good shape. We can remove all > the skany qemu-dm support from the tools as from their POV paravirt > and hvm guests will look identical. It should also be easy to > implement save/restore of hvm guests -- just save the miniguest as > part of the hvm guests'', memory image. The next stage would then be > to improve performance by enhancing the device models, e.g. adding a > network card that suports jumbo frames and csum offload, and requires > fewer vmexits in operation. > > How best to move forward on this? Any volunteers? > > Thanks, > Ian >Jun --- Intel Open Source Technology Center _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 13 Mar 2006, at 05:04, Nakajima, Jun wrote:> For the "mini guest", I think it could be much easier if we > substantially strip down xenlinux rather than adding (eventually) a lot > of stuff to the current mini-os, mainly because we need probably a > multi-threaded run-time environment, scheduler, memory allocator, event > handling, drivers such as xenbus/netfront/blkfront, etc. At least, I > think we can use xenlinux as the development platform. For example, > implement the qemu-dm as a driver adding the infrastructure required > (e.g. small in-kernel glibc).This sounds sensible. At least it allows ioemu development in parallel with mini-os development. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hello all, In this occasion, may I ask : what is the puprpse of the mini-os (under extras in the Xen tree) ? What is its usages ? Can soembody say 3-4 sentences about xen mini-os? Mark On 3/13/06, Keir Fraser <Keir.Fraser@cl.cam.ac.uk> wrote:> > On 13 Mar 2006, at 05:04, Nakajima, Jun wrote: > > > For the "mini guest", I think it could be much easier if we > > substantially strip down xenlinux rather than adding (eventually) a lot > > of stuff to the current mini-os, mainly because we need probably a > > multi-threaded run-time environment, scheduler, memory allocator, event > > handling, drivers such as xenbus/netfront/blkfront, etc. At least, I > > think we can use xenlinux as the development platform. For example, > > implement the qemu-dm as a driver adding the infrastructure required > > (e.g. small in-kernel glibc). > > This sounds sensible. At least it allows ioemu development in parallel > with mini-os development. > > -- Keir > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 3/13/06, Nakajima, Jun <jun.nakajima@intel.com> wrote:> For the "mini guest", I think it could be much easier if we > substantially strip down xenlinux rather than adding (eventually) a lot > of stuff to the current mini-os, mainly because we need probably a > multi-threaded run-time environment, scheduler, memory allocator, event > handling, drivers such as xenbus/netfront/blkfront, etc. At least, I > think we can use xenlinux as the development platform. For example, > implement the qemu-dm as a driver adding the infrastructure required > (e.g. small in-kernel glibc).It seems to me the main reason we needs threads and scheduling is to interact with Xenstore. A page allocator can''t be that hard to implement. I wonder if the Xenstore API could be simplified in a way that does not require threading, thus making the job of implementing drivers in a min-os a bit simpler? My feeling is that even a stripped down Linux would take some work to maintain, at least if we wish to remove the need for hotplug scripts for driver backends and the like from the miniLinux. I have little interest in hvm guests, but having a functional mini-os with an extensible, perhaps oskit- or TinyOS-like, structure would be a huge win in a number of other situations as well. If I can I would like to help. If the mini-os ever gets functional, I suppose it would help to include it in the regression tests, to prevent the bit-rot it is currently suffering from. Jacob _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 3/13/06, Mark Ryden <markryde@gmail.com> wrote:> Hello all, > > In this occasion, may I ask : what is the puprpse of the mini-os (under extras > in the Xen tree) ? What is its usages ? > Can soembody say 3-4 sentences about xen mini-os?The mini-os (or similar small Xen guests, like my own cstrap), are useful for stuff like: 1) testing the Xen APIs. 2) as a base on which to implement or port a new Xen guest OS. 3) as a boot loader which can do ''dangerous'' stuff such as ELF parsing from guest rather than dom0 content. 4) for writing small and quickly booting Xen applications. For instance, I once implemented a slide-show viewer in a Xen VM. The problem with mini-os-like projects is that they need constant maintenance to keep up with changes in the Xen API, and that the code of such projects often becomes a bit unwieldy because a lot of bloat is often ''faulted in'' from Linux in order to make copy-pasted code such as frontend drivers compile out of their native environment. Jacob _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
As Ian mentioned some work needs to be done to make mini-os work on 64 bit platform but the current 32 bit version of mini-os already has: - simple scheduler (RR like) - memory allocator (buddy allocator, and ported xmalloc build on top of buddy) - xenbus implementation I''ve implemented all of it last year. Obviously if we start using it for I/O emulation it will have to be tested more extensively, but it seemed to work just when I used it for my trusted domain builder. As Aravindh mentioned it his email, we''ve recently added 64-bit page table builder, which means that the memory allocator should now work on 64bit platform as well. There are still a few things on TODO list we''ve created with Aravindh but I think it should be much easier to resolve those then strip Linux down. And I do agree with Jacob, mini-os could be useful for many other things then just I/O emulation. Cheers Gregor>> For the "mini guest", I think it could be much easier if we >> substantially strip down xenlinux rather than adding (eventually) a lot >> of stuff to the current mini-os, mainly because we need probably a >> multi-threaded run-time environment, scheduler, memory allocator, event >> handling, drivers such as xenbus/netfront/blkfront, etc. At least, I >> think we can use xenlinux as the development platform. For example, >> implement the qemu-dm as a driver adding the infrastructure required >> (e.g. small in-kernel glibc). > > It seems to me the main reason we needs threads and scheduling is to > interact with Xenstore. A page allocator can''t be that hard to > implement. I wonder if the Xenstore API could be simplified in a way > that does not require threading, thus making the job of implementing > drivers in a min-os a bit simpler? > > My feeling is that even a stripped down Linux would take some work to > maintain, at least if we wish to remove the need for hotplug scripts > for driver backends and the like from the miniLinux. > > I have little interest in hvm guests, but having a functional mini-os > with an extensible, perhaps oskit- or TinyOS-like, structure would be > a huge win in a number of other situations as well. If I can I would > like to help. If the mini-os ever gets functional, I suppose it would > help to include it in the regression tests, to prevent the bit-rot it > is currently suffering from. > > Jacob > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Nakajima, Jun wrote:> For the "mini guest", I think it could be much easier if we > substantially strip down xenlinux rather than adding (eventually) a lot > of stuff to the current mini-os, mainly because we need probably a > multi-threaded run-time environment, scheduler, memory allocator, event > handling, drivers such as xenbus/netfront/blkfront, etc. At least, I > think we can use xenlinux as the development platform. For example, > implement the qemu-dm as a driver adding the infrastructure required > (e.g. small in-kernel glibc). >Once you get past vl.c, qemu-dm has very little reliance on glibc functions. Since we''re only trying to do hardware emulation here, I''d expect that vl.c would not be included. I suspect stripping down Linux is going to prove harder in the long run. As Jacob mentioned, you only really need a simple page allocator. The only reasons I can think of to use threads is XenBus (threads shouldn''t be required to implement it) and asynchronous IO. I think an interesting alternative for AIO would actually be to create another VCPU specifically for the mini-os code to run in. The physically analogy is sane and if you truly do need more parallelism you can always just use two VCPUs. Regards, Anthony Liguori>> Once the above is working we''ll be in good shape. We can remove all >> the skany qemu-dm support from the tools as from their POV paravirt >> and hvm guests will look identical. It should also be easy to >> implement save/restore of hvm guests -- just save the miniguest as >> part of the hvm guests'', memory image. The next stage would then be >> to improve performance by enhancing the device models, e.g. adding a >> network card that suports jumbo frames and csum offload, and requires >> fewer vmexits in operation. >> >> How best to move forward on this? Any volunteers? >> >> Thanks, >> Ian >> >> > > Jun > --- > Intel Open Source Technology Center > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Anthony Liguori wrote:> Nakajima, Jun wrote: >> For the "mini guest", I think it could be much easier if we >> substantially strip down xenlinux rather than adding (eventually) a >> lot of stuff to the current mini-os, mainly because we need probably >> a multi-threaded run-time environment, scheduler, memory allocator, >> event handling, drivers such as xenbus/netfront/blkfront, etc. At >> least, I think we can use xenlinux as the development platform. For >> example, implement the qemu-dm as a driver adding the infrastructure >> required (e.g. small in-kernel glibc). >> > Once you get past vl.c, qemu-dm has very little reliance on glibc > functions. Since we''re only trying to do hardware emulation here, I''d > expect that vl.c would not be included.I think it depends on how much code we borrow from Linux to get the netfront/blkfront working. We don''t need the plumbing code for user-level apps, but we need to identify which networking/block layer code we need for hardware emulation (and virutal device drivers, such as VBD/VIF for HVM guests). The dependencies in the code often cause the domino effect, ending up with a lot of codes.> > I suspect stripping down Linux is going to prove harder in the long > run. As Jacob mentioned, you only really need a simple page > allocator. The only reasons I can think of to use threads is XenBus > (threads shouldn''t be required to implement it) and asynchronous IO.I agree that in the long run Linux-based mini-OS will be harder, and it depends on how soon we need it. If it''s a couple of months of timeframe (which was in my mind), I would choose xenlinux to start with -- cut the code as much as possible so that we only use the kernel threads and drivers. You can easily tell if you broke it because you start from something working fine. The reason we need threads and scheduling is to handle simultaneous I/O requests from SMP HVM guests. Getting reliable SMP environment is not a trivial thing. Jun --- Intel Open Source Technology Center _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Le Dimanche 12 Mars 2006 22:26, Ian Pratt a écrit :> Folks, > > At the last summit I presented a proposal for rearchitecting the way we > do io emulation for fully-virtualized (hvm) guests.[...]> > How best to move forward on this? Any volunteers?Just a comment from an xen/ia64 member. xen/ia64 has a vmx-like feature named VTi. The implementation of this feature relies heavily on the current vmx/hvm implementation: it uses qemu-dm. If VMX switches to mini-os, this also means a lot of works for ia64. And the Xen/ia64 team is very small compared to Xen/x86. So either we should have two ways: current qemu-dm and mini-os, or we will have to write our own mini-os. [Joke: As a consequence AMD may like mini-os while Intel may prefer current qemu :-)] Tristan. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ian, have some questions for mini-os io emulation 1. seems dom0 still need do the user interaction( screen painting, kbd/mouse detection), so how much performance gain with extra dom switch(dom0<->mini os<->VMX guest) ? 2. even with mini-os io emulation, pit/pic/iopaic/lapic still sit in Hypervisior, right? 3. we are currently doing the save/restore for VMX guest with the expection that mini-os emulation only affect device model save/resotore. we can continue current work (just save the qemu-dm state) and sync with mini os in future. On Sun, Mar 12, 2006 at 09:26:26PM -0000, Ian Pratt wrote:> > Folks, > > At the last summit I presented a proposal for rearchitecting the way we > do io emulation for fully-virtualized (hvm) guests. I''d really like to > try and get the work to implement this underway, as it cleans up a bunch > of mess, is a prerequisite for save/restore/relocation of hvm guests, > and is a precursor to some significant performance improvements. It > involves a fair chunk of work, so we really want to try and get multiple > folk working on it. > > The plan is to move the io emulation code (qemu-dm) from running as a > user-space app in domain 0 into a ''mini guest'' that is effectively a > small paravirtualized guest in the root hardware context associated with > each hvm domain. > > I guess a very high-level work plan would look something like this: > > * get minios running well on x86_64; add a few simple infrastructure > functions e.g. simple memory allocator. No need for any ''user space'' mmu > support > * port (simplified)xenbus/netfront/blkfront to minios; test simple > net/disk IO > * implement enough infrastructure to allow qemu-dm to be compiled into > minios, calling into net/blkfront for IO. > * plumb the vmexit entry points from MMIO and in/out into minios and > hence qemu-dm > > Once the above is working we''ll be in good shape. We can remove all the > skany qemu-dm support from the tools as from their POV paravirt and hvm > guests will look identical. It should also be easy to implement > save/restore of hvm guests -- just save the miniguest as part of the hvm > guests'', memory image. The next stage would then be to improve > performance by enhancing the device models, e.g. adding a network card > that suports jumbo frames and csum offload, and requires fewer vmexits > in operation. > > How best to move forward on this? Any volunteers? > > Thanks, > Ian > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel-- thanks, edwin _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 15 Mar 2006, at 03:03, Edwin Zhai wrote:> 1. seems dom0 still need do the user interaction( screen painting, > kbd/mouse > detection), so how much performance gain with extra dom > switch(dom0<->mini > os<->VMX guest) ?Some of those will not be critical path operations, and are moving to mini-os only to clean up the control interfaces in dom0 (consistent device interfaces for pv and hvm guests). In other cases we will still get better performance because we get to define the minios<->dom0 interface and make it amenable to e.g., batching.> 2. even with mini-os io emulation, pit/pic/iopaic/lapic still sit in > Hypervisior, > right?Yes. The only reason not to put something in Xen is because the backend cannot be implemented in Xen. Xen doesn''t know about displays, so display emulation does not go in Xen, for example.> 3. we are currently doing the save/restore for VMX guest with the > expection that > mini-os emulation only affect device model save/resotore. we can > continue > current work (just save the qemu-dm state) and sync with mini os in > future.yes. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel