Hi, I found vmware workstation 5 in xen will write page table in user mode. I don''t know the rational behind this... XEN doesn''t emulate writing page table in user mode. If I get rid of this constraint, vmware will go a little bit further. Is it safe to remove this constraint? Best Regards, Disheng, Su _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
It should be safe, but I don''t know how much it would affect performance (it depends on how much we are actually relying on it as an unshadowing heuristic). -- Keir On 1/2/08 18:28, "Su, Disheng" <disheng.su@intel.com> wrote:> Hi, > I found vmware workstation 5 in xen will write page table in > user mode. I don''t know the rational behind this... > XEN doesn''t emulate writing page table in user mode. If I get > rid of this constraint, vmware will go a little bit further. > Is it safe to remove this constraint? > > Best Regards, > Disheng, Su > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Just now I made a conclusion that no such unshadow heuristic is hit after checking my archived perfc counter logs, since there''s no item like "unshadow by user-level access". But immediately I found this is mixed into "emulator fails", and there does be dozens of count for Linux KB and several for one Windows benchmark (not sure the detail now). Whatever the count may be, even one user page slipped out of unshadow may hurt performance a lot if access to that page is frequent... Maybe we can disable such heuristics only when observing guest clear CR0.wp actively? Or later out-of-sync can also help here. BTW, it''s worthy of adding a seperate perfc counter for user-level access caused unshadow. :-) Thanks, kevin>From: Keir Fraser >Sent: 2008年2月2日 18:31 > >It should be safe, but I don''t know how much it would affect >performance (it >depends on how much we are actually relying on it as an unshadowing >heuristic). > > -- Keir > >On 1/2/08 18:28, "Su, Disheng" <disheng.su@intel.com> wrote: > >> Hi, >> I found vmware workstation 5 in xen will write page table in >> user mode. I don''t know the rational behind this... >> XEN doesn''t emulate writing page table in user mode. If I get >> rid of this constraint, vmware will go a little bit further. >> Is it safe to remove this constraint? >> >> Best Regards, >> Disheng, Su >> >> _______________________________________________ >> Xen-devel mailing list >> Xen-devel@lists.xensource.com >> http://lists.xensource.com/xen-devel > > > >_______________________________________________ >Xen-devel mailing list >Xen-devel@lists.xensource.com >http://lists.xensource.com/xen-devel >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 2/2/08 14:08, "Tian, Kevin" <kevin.tian@intel.com> wrote:> Maybe we can disable such heuristics only when observing guest > clear CR0.wp actively? Or later out-of-sync can also help here. > > BTW, it''s worthy of adding a seperate perfc counter for user-level > access caused unshadow. :-)Yes, and if we can do better at working out when pages should be unshadowed, we can then perhaps get rid of the unshadow-on-user-access heuristic. Presumably things still work though? The page will simply get re-shadowed on next use. I suppose the guest will crash if the shadow code is unable to unshadow the page though (e.g., because it is current base page directory). Perhaps we should emulate the access anyway if the page cannot be unshadowed? -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
>From: Keir Fraser [mailto:Keir.Fraser@cl.cam.ac.uk] >Sent: 2008年2月2日 23:16 > >On 2/2/08 14:08, "Tian, Kevin" <kevin.tian@intel.com> wrote: > >> Maybe we can disable such heuristics only when observing guest >> clear CR0.wp actively? Or later out-of-sync can also help here. >> >> BTW, it''s worthy of adding a seperate perfc counter for user-level >> access caused unshadow. :-) > >Yes, and if we can do better at working out when pages should >be unshadowed, >we can then perhaps get rid of the unshadow-on-user-access heuristic. > >Presumably things still work though? The page will simply get >re-shadowed on >next use. I suppose the guest will crash if the shadow code is >unable to >unshadow the page though (e.g., because it is current base >page directory). >Perhaps we should emulate the access anyway if the page cannot be >unshadowed? >It''s said to be a forward progress issue, that instruction page of faulting IP falls into mapped virtual range by same L1 as the target frame it tries to update. So the implication is that the unshadow unfortunately succeeds. Then re-execute faulting IP causes another shadow fault and unshadowed L1 is reshadowed again. Finally unshadow and reshadow conducts a dead trap. (Disheng is in Chinese New Year vacation now and hope I didn''t make thing wrong :-) For this special case, I guess we may add some postponed track info, like if previously unshadowed page triggers a reshadow immediately without any other faults in between, we then disable that heuristics on this special page. Possibly we can do page-based heuristics, like adding one flag with bit set indicating disallowed heuristics? As long as the right write doesn''t affect instruction itself, forward progress can be anyway assured though performance is bad at thrash between unshadow and reshadow... Thanks, Kevin _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 2/2/08 15:29, "Tian, Kevin" <kevin.tian@intel.com> wrote:> It''s said to be a forward progress issue, that instruction page of faulting > IP falls into mapped virtual range by same L1 as the target frame it tries > to update. So the implication is that the unshadow unfortunately > succeeds. Then re-execute faulting IP causes another shadow fault and > unshadowed L1 is reshadowed again. Finally unshadow and reshadow > conducts a dead trap. (Disheng is in Chinese New Year vacation now > and hope I didn''t make thing wrong :-) > > For this special case, I guess we may add some postponed track info, > like if previously unshadowed page triggers a reshadow immediately > without any other faults in between, we then disable that heuristics on > this special page. Possibly we can do page-based heuristics, like adding > one flag with bit set indicating disallowed heuristics?Ugh, it would be much nicer to remove the unshadow-on-user-access heuristic entirely. We just need to do enough testing to ensure it doesn''t harm performance in other cases we care about. I really don''t know how much we rely on that heuristic these days. If we can show good performance without it then I''m happy to remove it (if Tim also agrees). -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
At 15:58 +0000 on 02 Feb (1201967880), Keir Fraser wrote:> > It''s said to be a forward progress issue, that instruction page of faulting > > IP falls into mapped virtual range by same L1 as the target frame it tries > > to update. So the implication is that the unshadow unfortunately > > succeeds.Do you mean that the guest instruction stream is changing the VM mapping of the page under %eip, from user-space? Wow.> Ugh, it would be much nicer to remove the unshadow-on-user-access heuristic > entirely. We just need to do enough testing to ensure it doesn''t harm > performance in other cases we care about. I really don''t know how much we > rely on that heuristic these days.Nor do I (since as Kevin points out we don''t count this separately) but with the increasing completeness of the emulator, we''ve already lost a lot of heuristics for unshadowing pages -- this is what the unshadow-after-three-writes heuristic is trying to fix, for example. It seems odd to be making the unshadowing more aggressive with one hand and less aggressive with the other.> If we can show good performance without > it then I''m happy to remove it (if Tim also agrees).There''ll need to be an audit of the rest of the emulate callbacks, which were written with the assumption that they know the''yre not in ring 3 -- for example the pagefault error codes used in PT walks will be wrong. Cheers, Tim. -- Tim Deegan <Tim.Deegan@citrix.com> Principal Software Engineer, Citrix Systems (R&D) Ltd. [Company #02300071, SL9 0DZ, UK.] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 4/2/08 11:00, "Tim Deegan" <Tim.Deegan@citrix.com> wrote:>> Ugh, it would be much nicer to remove the unshadow-on-user-access heuristic >> entirely. We just need to do enough testing to ensure it doesn''t harm >> performance in other cases we care about. I really don''t know how much we >> rely on that heuristic these days. > > Nor do I (since as Kevin points out we don''t count this separately) but > with the increasing completeness of the emulator, we''ve already lost a > lot of heuristics for unshadowing pages -- this is what the > unshadow-after-three-writes heuristic is trying to fix, for example. It > seems odd to be making the unshadowing more aggressive with one hand and > less aggressive with the other.Well, this isn''t just a matter of degree of aggressiveness. We currently don''t allow updates from user space at all, ever. The motivation for changing that is not to do with aggressiveness of unshadowing, but with increased compatibility. OTOH it''s arguable whether we care about running VMware Workstation inside an HVM guest. :-) -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
At 11:06 +0000 on 04 Feb (1202123216), Keir Fraser wrote:> Well, this isn''t just a matter of degree of aggressiveness. We currently > don''t allow updates from user space at all, ever. The motivation for > changing that is not to do with aggressiveness of unshadowing, but with > increased compatibility.If the problem is really one of unshadowing and reshadowing, then I''m not sure how to fix this without significantly blunting the teeth of this heuristic -- even replacing this with an attempted "fast" unshadow could fail to solve the problem for VMware.> OTOH it''s arguable whether we care about running > VMware Workstation inside an HVM guest. :-)We could always fix the VPC performance instead and run VMware inside VPC inside Xen. Tim. -- Tim Deegan <Tim.Deegan@citrix.com> Principal Software Engineer, Citrix Systems (R&D) Ltd. [Company #02300071, SL9 0DZ, UK.] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> > > It''s said to be a forward progress issue, that instruction page of faulting > > > IP falls into mapped virtual range by same L1 as the target frame it tries > > > to update. So the implication is that the unshadow unfortunately > > > succeeds. > > Do you mean that the guest instruction stream is changing the VM mapping > of the page under %eip, from user-space? Wow.When I first read Kevin''s description, I took it to mean that the same L1 *page* (not entry) was mapping the current EIP and the target of the write. If this is case, then the current code''s prefetching of L1 entries on the same page is probably interacting with the user-space write heuristic, causing the shadow->unshadow->shadow livelock, making for a much easier fix. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
>From: Michael A Fetterman >Sent: 2008年2月4日 20:02 > >> > > It''s said to be a forward progress issue, that >instruction page of faulting >> > > IP falls into mapped virtual range by same L1 as the >target frame it tries >> > > to update. So the implication is that the unshadow unfortunately >> > > succeeds. >> >> Do you mean that the guest instruction stream is changing >the VM mapping >> of the page under %eip, from user-space? Wow. > >When I first read Kevin''s description, I took it to mean that >the same L1 *page* >(not entry) was mapping the current EIP and the target of the >write. If this is >case, then the current code''s prefetching of L1 entries on the >same page is >probably interacting with the user-space write heuristic, causing the >shadow->unshadow->shadow livelock, making for a much easier fix. >Yes, same L1 page. Sorry for confusion. It may be some map-cache- like usage within same 4M virtual range including code itself. But I''m not clear about what you mean as a ''much easier'' fix. Could you elaborate? Thanks, Kevin _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
>From: Tim Deegan [mailto:Tim.Deegan@citrix.com] >Sent: 2008年2月4日 19:01 > >> If we can show good performance without >> it then I''m happy to remove it (if Tim also agrees). > >There''ll need to be an audit of the rest of the emulate >callbacks, which >were written with the assumption that they know the''yre not in >ring 3 -- >for example the pagefault error codes used in PT walks will be wrong. >Yes, we need full audit. A quick update is that, at least for multi.c, Disheng made things working simply by disabling that heuristics and remove ring 3 check from emulate_map_dest. So such assumptions shouldn''t be many. :-) Thanks, Kevin _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
>From: Tim Deegan [mailto:Tim.Deegan@citrix.com] >Sent: 2008年2月4日 19:38 > >At 11:06 +0000 on 04 Feb (1202123216), Keir Fraser wrote: >> Well, this isn''t just a matter of degree of aggressiveness. >We currently >> don''t allow updates from user space at all, ever. The motivation for >> changing that is not to do with aggressiveness of >unshadowing, but with >> increased compatibility. > >If the problem is really one of unshadowing and reshadowing, then I''m >not sure how to fix this without significantly blunting the teeth of >this heuristic -- even replacing this with an attempted "fast" unshadow >could fail to solve the problem for VMware.IMO, to avoid shadow-unshadow-reshadow loop, we can simply add a check whether L1e for faulting IP falls into same page it tries to update. This shouldn''t interfere this heuristic much, since it only resides in emulation logic and above check is easy to make.> >> OTOH it''s arguable whether we care about running >> VMware Workstation inside an HVM guest. :-) > >We could always fix the VPC performance instead and run VMware inside >VPC inside Xen. >Wow, double nest virtualization. :-) One point I can come up is to keep user experience if he is used to use VMware or VirtualPC. Whatever the underlying ''box'' is physical or not, that guy may just want to create multiple nested VMs for his own usage, like test infrastrucure, etc. It would be annoying for him to ask lower layer administrator for help, if this VM is loaned. I bet usage on nested virtualization is unlikely to be performance centric though... Thanks, Kevin _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
>From: Tim Deegan [mailto:Tim.Deegan@citrix.com] >Sent: 2008年2月4日 19:01 > >Nor do I (since as Kevin points out we don''t count this separately) but >with the increasing completeness of the emulator, we''ve already lost a >lot of heuristics for unshadowing pages -- this is what the >unshadow-after-three-writes heuristic is trying to fix, for >example. It >seems odd to be making the unshadowing more aggressive with >one hand and >less aggressive with the other. >BTW, as a heads up we''re trying another heuristic to take DMA page into unshadow candidate, which requires Qemu''s help indicating whether data in ioreq is a DMA target. This should help capture one vague area when released page tables are used by kernel drivers. DMA is normally one big factor for frequent page allocation/free. It''s also not affected by more complete emulator. Now well, patch is just in progress and we need many experiments to see how it behaves like compared to 3-write-unshadow, etc. But We may be only able to get something after Chinese New Year vacation. :-) Thanks, Kevin _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel