In alt.bk.changesets.1.1508 Keir wrote:> Some more ioemu config hardcoding. Our ''target arch'' is always i386. > Do we really need linker scripts?? > Signed-off-by: Keir Fraser <keir@xensource.com>Yes, we do need linker scripts to support VMX guests with large memory (> 256MB). Without the linker scripts, we found instabilities with a 512MB guest. The reason for this is, qemu is mapping all of guest physical in it''s address space. The linker script moves all of the qemu text/data/stack to the upper part of the user virtual to leave a large enough hole for mapping guest physical using xc_map_foreign. Our intention is to use qemu for i386, x86-64 and ia64. The "target-i386-dm" is a misnomer and inherited from upstream. I should probably kill the concept of a "target architecture" since ioemu is cpu architecture independent right now. -Arun _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 23 May 2005, at 07:00, Arun Sharma wrote:> Yes, we do need linker scripts to support VMX guests with large memory > (> 256MB). Without the linker scripts, we found instabilities with a > 512MB guest. > > The reason for this is, qemu is mapping all of guest physical in it''s > address space. The linker script moves all of the qemu text/data/stack > to the upper part of the user virtual to leave a large enough hole for > mapping guest physical using xc_map_foreign.I suppose it improves performance and simplifies the data path, but it should probably dynamically map the vmx guest''s pages for I/O on demand. After all, this approach isn''t going to work for bigmem domains on i386 (particularly once PAE is working). So it seems a bad idea to come to rely on it any more than as a bit of a hack to get you up and running. Best of all (imo) would be for it to act as a ''driver in the middle'': connecting the fully virt guest through to an ordinary backend driver, and not itself touching I/O buffers at all. Perhaps ultimately we can push the qemu drivers into the vmx container itself, perhaps in ''virtual smm mode'', so that from outside the container it looks just like any other paravirt I/O channel.> Our intention is to use qemu for i386, x86-64 and ia64. The > "target-i386-dm" is a misnomer and inherited from upstream. I should > probably kill the concept of a "target architecture" since ioemu is > cpu architecture independent right now.Sounds good. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 23 May 2005, at 09:02, Keir Fraser wrote:> >> Our intention is to use qemu for i386, x86-64 and ia64. The >> "target-i386-dm" is a misnomer and inherited from upstream. I should >> probably kill the concept of a "target architecture" since ioemu is >> cpu architecture independent right now. > > Sounds good.In fact, I think it would be good to kill the configure script and have a clean up of the Makefiles -- it''s really hard to see what''s going on in them right now, and we aren''t going to use all that flexibility/genericity anyway. I reckon the Makefiles should end up a good deal shorter and clearer. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
(Just some rough idea, since I don''t know much detail of DM part. ;-)>I suppose it improves performance and simplifies the data path, but it >should probably dynamically map the vmx guest''s pages for I/O on >demand. After all, this approach isn''t going to work for bigmem domains >on i386 (particularly once PAE is working). So it seems a bad idea to >come to rely on it any more than as a bit of a hack to get you up and >running.Another requirement for dynamic map on demand is that guest pfn -> machine pfn mapping may be changed due to balloon driver. It''s not that perfect for a DM to assume that mapping info never changed, and thus maps all the guest memory in one request at the start...> >Best of all (imo) would be for it to act as a ''driver in the middle'': >connecting the fully virt guest through to an ordinary backend driver, >and not itself touching I/O buffers at all. Perhaps ultimately we can >push the qemu drivers into the vmx container itself, perhaps in >''virtual smm mode'', so that from outside the container it looks just >like any other paravirt I/O channel. >Frist, to clarify a term: Do you mean vmx container as "vmx domain" ? :) Major benefit of your approach, from my thought, is that backend driver in service OS can service both para-virtualization and full-firtualization domain then, with a unified channel interface and logic, right? I''m trying to understand how ''virtual smm mode'' you mentioned can be achieved, and thinking at least following modifications may be required: - DM can not only be user application. A bunch of kernel module like a collection of existing frontend drivers have to be created for each component of DM in vmx container. - DM itself has to be bound with a faked device (Maybe PCI device), to be able to receive emulation message coming from HV, after HV captures unmodified domain''s access to MMIO, etc. - More backend-frontend drivers need time to be cultivated gradually, like SCSI support, etc, to behave like a complete DM.s However the major defect somehow may lie with the maintain effort. Also take USB port as the example, there''ll be around 10 or more maintainers of different backend-frontend components, who have to keep up with kernel evolvement both on service OS side and domainN side, since kernel module is tightly coupled with kernel internal interfaces. However current DM model has no such problem. With more deployment of xen, there''ll be more such effort. Of course, also with more deployment of xen, there''ll be more people who''d like to contribute to xen then. ;-) Thanks, Kevin _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 23 May 2005, at 11:22, Tian, Kevin wrote:> Frist, to clarify a term: Do you mean vmx container as "vmx domain" ? > :)I mean the ''guest mode'' of VMX, as opposed to code that runs in VMX domain context, but within the hypervisor in ''host mode''.> Major benefit of your approach, from my thought, is that backend driver > in service OS can service both para-virtualization and > full-firtualization domain then, with a unified channel interface and > logic, right? I''m trying to understand how ''virtual smm mode'' you > mentioned can be achieved, and thinking at least following > modifications > may be required:I thought that VMX provided a virtual equivalent of SMM, where management and emulation code can run under the OS''s feet without it realising? If this is not provided then I do not think the trick can work, as you would need to steal some virtual address space in which to execute the qemu code. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Mon, May 23, 2005 at 01:54:29PM +0100, Keir Fraser wrote:> > I thought that VMX provided a virtual equivalent of SMM, where > management and emulation code can run under the OS''s feet without it > realising?Yes, this is how vmxassist implements real mode support.> If this is not provided then I do not think the trick can > work, as you would need to steal some virtual address space in which to > execute the qemu code.The "world switch" can switch to a different address space (CR3 on x86/RID on ia64), without having to steal guest virtual. -Arun _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> > Major benefit of your approach, from my thought, is that backend > > driver in service OS can service both para-virtualization and > > full-firtualization domain then, with a unified channel > interface and > > logic, right? I''m trying to understand how ''virtual smm mode'' you > > mentioned can be achieved, and thinking at least following > > modifications may be required: > > I thought that VMX provided a virtual equivalent of SMM, > where management and emulation code can run under the OS''s > feet without it realising? If this is not provided then I do > not think the trick can work, as you would need to steal some > virtual address space in which to execute the qemu code.I''d be inclined to move to a model where we execute the device emulation in the root (monitor) VMCS, using the same protection mechanism we use for para-virtualized guests e.g. segmentation for x86, paging for x86_64. The device emulation should should work like a normal front-end driver, connecting via a device channel to a normal backend. Infact, I really like this approach. It gives good performance, safety, code reuse, and unifies the control interface. It does require a bit of hacking of qemu, to give it the execution environment it needs and make it connect onto the existing back ends. Arguments against? Ian _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
>> I thought that VMX provided a virtual equivalent of SMM, >> where management and emulation code can run under the OS''s >> feet without it realising? If this is not provided then I do >> not think the trick can work, as you would need to steal some >> virtual address space in which to execute the qemu code. > >I''d be inclined to move to a model where we execute the deviceemulation>in the root (monitor) VMCS, using the same protection mechanism we use >for para-virtualized guests e.g. segmentation for x86, paging for >x86_64. The device emulation should should work like a normal front-end >driver, connecting via a device channel to a normal backend.I like the idea. :) but where is the resource of device emulation component getting allocation from, like physical frame allocated to that module? If it''s managed by vmx domain''s kernel, I''m not sure it''s easy for such type of DM to acquire system resource and also access the user eco system across multiple space, like: - page fault handle - Access special files like /proc/privcmd - Where are configuration files of qemu saved? In service OS file system, or in vmx domain''s FS? - How to sync between DM and corresponding vmx domain? ... Also that style of DM will be another type of domain, which has to been considered by HV specially with resource and schedule... Just some immature concerns, and like to learn more in this direction. :) Thanks, Kevin _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Then, after some look at vmxassist mechanism and with some discussion with Susie Li, I''m now aware that my previous understanding completely bias from your intent. :) Following the ''hack'' you mentioned, that DM may finally be standalone component, without any dependency with upper vmx domain. Control panel can load it to appropriate position, which has its own trap table, own page table (maybe fixed), device emulation logic, simplified event channel interface, similar assist hook in HV for context save/restore, etc. No need to have memory management, since DMA buffer will only be touched by backend. En... now I''m clearer about the feasibility, which is always the first thing for me to care when learning new things. :) Regarding performance, it saves many context switches between kernel and user space, compared to current DM model which runs as application in service''s OS. But the maximum granularity of this model is still instruction level when vmx domain tries to do mmio access. Instead, for some specific device, frontend driver module may be more effective to utilize frontend-backend model, since it''s based on transaction/session. The example is the recent VBD/VNIF driver by Xiaofeng Lin, which can be installed into vmx domain dynamically and then talk to backend effectively. Thanks, Kevin>-----Original Message----- >From: xen-devel-bounces@lists.xensource.com >[mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Tian, Kevin >Sent: Tuesday, May 24, 2005 11:41 AM >To: Ian Pratt; Keir Fraser >Cc: Sharma, Arun; xen-devel@lists.xensource.com >Subject: RE: [Xen-devel] Re: Are linker scripts needed? > >>> I thought that VMX provided a virtual equivalent of SMM, >>> where management and emulation code can run under the OS''s >>> feet without it realising? If this is not provided then I do >>> not think the trick can work, as you would need to steal some >>> virtual address space in which to execute the qemu code. >> >>I''d be inclined to move to a model where we execute the device >emulation >>in the root (monitor) VMCS, using the same protection mechanism we use >>for para-virtualized guests e.g. segmentation for x86, paging for >>x86_64. The device emulation should should work like a normalfront-end>>driver, connecting via a device channel to a normal backend. > >I like the idea. :) but where is the resource of device emulation >component getting allocation from, like physical frame allocated tothat>module? If it''s managed by vmx domain''s kernel, I''m not sure it''s easy >for such type of DM to acquire system resource and also access the user >eco system across multiple space, like: > - page fault handle > - Access special files like /proc/privcmd > - Where are configuration files of qemu saved? In service OS >file system, or in vmx domain''s FS? > - How to sync between DM and corresponding vmx domain? > ... > >Also that style of DM will be another type of domain, which has to been >considered by HV specially with resource and schedule... > >Just some immature concerns, and like to learn more in this direction. >:) > >Thanks, >Kevin > >_______________________________________________ >Xen-devel mailing list >Xen-devel@lists.xensource.com >http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> >I''d be inclined to move to a model where we execute the device > emulation > >in the root (monitor) VMCS, using the same protection > mechanism we use > >for para-virtualized guests e.g. segmentation for x86, paging for > >x86_64. The device emulation should should work like a > normal front-end > >driver, connecting via a device channel to a normal backend. > > I like the idea. :) but where is the resource of device > emulation component getting allocation from, like physical > frame allocated to that module?Page frames in use by the device emulation will be on the domain''s page list, just like the ''real guest'' running under VT-x.> If it''s managed by vmx > Also that style of DM will be another type of domain, which > has to been considered by HV specially with resource and schedule...As far as I can see, the device emulation will be entirely event driven, driven by VM exits and events coming back from the backend drivers. The normal scheduling should work just fine. The device emulation modules could basically be linked against a revamped mini-os to provide the services they need (e.g printk). The configuration for qemu could be passed in on the ''command line'' of the mini-os, supplied by the domain builder. The device emulation shouldn''t assume it has all of the ''real'' guest''s memory mapped into its address space (it can''t for PAE), but can have a very large map cache. Ian _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> Then, after some look at vmxassist mechanism and with some > discussion with Susie Li, I''m now aware that my previous > understanding completely bias from your intent. :) Following > the ''hack'' you mentioned, that DM may finally be standalone > component, without any dependency with upper vmx domain. > Control panel can load it to appropriate position, which has > its own trap table, own page table (maybe fixed), device > emulation logic, simplified event channel interface, similar > assist hook in HV for context save/restore, etc. No need to > have memory management, since DMA buffer will only be touched > by backend. En... now I''m clearer about the feasibility, > which is always the first thing for me to care when learning > new things. :)I don''t quite follow your terminiology, but I think we''re on the same wavelength. I guess the first step would be getting mini-os working on the unstable tree again -- shouldn''t be hard.> Regarding performance, it saves many context switches between > kernel and user space, compared to current DM model which > runs as application in service''s OS. But the maximum > granularity of this model is still instruction level when vmx > domain tries to do mmio access. Instead, for some specific > device, frontend driver module may be more effective to > utilize frontend-backend model, since it''s based on > transaction/session. > The example is the recent VBD/VNIF driver by Xiaofeng Lin, > which can be installed into vmx domain dynamically and then > talk to backend effectively.Yep, performance won''t be as good as the real para-virtualized drivers, but if we pick the device to emulate carefully it should be possible to get half decent performance. [We''ll probably use the qemu models in the first instance, but ideally we want to emulate a network device that supports DMA and cheksum offload. Using the L4/Afterburner team''s DP83820 emulation code might actually be a better starting point than qemu. ] Only question is, who''s going to do the work? :-) Best, Ian _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser wrote:> > In fact, I think it would be good to kill the configure script and have > a clean up of the Makefiles -- it''s really hard to see what''s going on > in them right now, and we aren''t going to use all that > flexibility/genericity anyway. I reckon the Makefiles should end up a > good deal shorter and clearer.Agree that configure isn''t adding much value today. The only useful thing it''s doing is to detect if the build env has libvncserver and disabling VNC support if it doesn''t have it. Will send a makefile patch as soon as I have one... -Arun _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel