Hi, everyone, I have a question, in the shadow_page_fault or ept mechanism, xen will use the x86_emulation for some instructions, I''m wondering why it must use it, if after we fix the SPT or EPT table, just VMEntry to HVM to re-excute this instruction but not emulate in xen, is there some problems? can sb. tell me why? thanks:) -- - Luit @ Parallel Processing Institute, Fudan University _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Tim Deegan
2011-Oct-11 08:18 UTC
Re: [Xen-devel] why xen use x86_emulation() in page_fault
At 09:39 +0800 on 11 Oct (1318325957), cc Luit wrote:> Hi, everyone, I have a question, > in the shadow_page_fault or ept mechanism, xen will use the x86_emulation > for some instructions, I''m wondering why it must use it, if after we fix the > SPT or EPT table, just VMEntry to HVM to re-excute this instruction but not > emulate in xen, is there some problems?In the shadow pagetable code, we keep the shadows up-to-date by: 1 - making all shadowed pagetables read-only; 2 - intercepting the page faults when the guest writes to them; and 3 - updating the guest pagetable and the shadow at the same time, with whatever change the guest was making. For step 3 we need to emulate the instruction that caused the pagefault so that we can tell what was being written. There are other reasons for the emulator to be called (emulating MMIO instructions, emulating real-mode &c) but that''s why the shadow pagetable code uses it. Cheers, Tim. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Tue, Oct 11, 2011 at 4:18 PM, Tim Deegan <tim@xen.org> wrote:> At 09:39 +0800 on 11 Oct (1318325957), cc Luit wrote: > > Hi, everyone, I have a question, > > in the shadow_page_fault or ept mechanism, xen will use the x86_emulation > > for some instructions, I''m wondering why it must use it, if after we fix > the > > SPT or EPT table, just VMEntry to HVM to re-excute this instruction but > not > > emulate in xen, is there some problems? > > > In the shadow pagetable code, we keep the shadows up-to-date by: > > 1 - making all shadowed pagetables read-only; > > 2 - intercepting the page faults when the guest writes to them; and > > 3 - updating the guest pagetable and the shadow at the same time, > > with whatever change the guest was making. > > > For step 3 we need to emulate the instruction that caused the pagefault > > so that we can tell what was being written. > > > There are other reasons for the emulator to be called (emulating MMIO > > instructions, emulating real-mode &c) but that''s why the shadow > > pagetable code uses it. > > Thanks first of all, I know now it is the Eager mode that SPT is sync upwith GPT when guest want to change the page table using the instructions emulator, but if for some reason I don''t want xen to emulate such an instruction, but just VMEntry to HVM to retry, is there any feasibility problems? in other words, in the old time, the shadow page is the Lazy mode, that xen will not emulate, and the GPT and SPT is out of sync for some time, besides the lose in performance, is there other problems?> > Cheers, > > Tim. >-- - Luit @ Parallel Processing Institute, Fudan University _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Tim Deegan
2011-Oct-11 11:29 UTC
Re: [Xen-devel] why xen use x86_emulation() in page_fault
At 18:14 +0800 on 11 Oct (1318356859), cc Luit wrote:> On Tue, Oct 11, 2011 at 4:18 PM, Tim Deegan <tim@xen.org> wrote: > > > At 09:39 +0800 on 11 Oct (1318325957), cc Luit wrote: > > > Hi, everyone, I have a question, > > > in the shadow_page_fault or ept mechanism, xen will use the x86_emulation > > > for some instructions, I''m wondering why it must use it, if after we fix > > the > > > SPT or EPT table, just VMEntry to HVM to re-excute this instruction but > > not > > > emulate in xen, is there some problems? > > > > > In the shadow pagetable code, we keep the shadows up-to-date by: > > > 1 - making all shadowed pagetables read-only; > > > 2 - intercepting the page faults when the guest writes to them; and > > > 3 - updating the guest pagetable and the shadow at the same time, > > > with whatever change the guest was making. > > > > > For step 3 we need to emulate the instruction that caused the pagefault > > > so that we can tell what was being written. > > > > > There are other reasons for the emulator to be called (emulating MMIO > > > instructions, emulating real-mode &c) but that''s why the shadow > > > pagetable code uses it. > > > > Thanks first of all, I know now it is the Eager mode that SPT is sync up > with GPT when guest want to change the page table using the instructions > emulator, but if for some reason I don''t want xen to emulate such an > instruction, but just VMEntry to HVM to retry, is there any feasibility > problems?The emulation can be avoided - in fact the current shadow pagetable sometimes lets guests write to shadowed pages and fixes up the shadows afterwards (this is called out-of-sync or oos in the code). But if you just return to the guest and retry, the guest will take the same fault again unless you have done something to change that. If you _have_ done something to make it OK, then just returning EXCRET_fault_fixed from sh_page_fault will return to the guest and retry the instruction. Why do you want to avoid calling the emulator? What is your overall goal? It might be that tinkering in the shadow pagetables isn''t the best way to acheive it.> in other words, in the old time, the shadow page is the Lazy mode, > that xen will not emulate, and the GPT and SPT is out of sync for some time, > besides the lose in performance, is there other problems?No, it was really about the performance cost of syncing up all the shadows on a TLB flush. In retrospect, having fixed some nasty bugs in the OOS code, I suspect the old shadow code was also incorrect in some ways but that was an implementation detail, noit architectural. Tim. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Tue, Oct 11, 2011 at 7:29 PM, Tim Deegan <tim@xen.org> wrote:> At 18:14 +0800 on 11 Oct (1318356859), cc Luit wrote: > > On Tue, Oct 11, 2011 at 4:18 PM, Tim Deegan <tim@xen.org> wrote: > > > > > At 09:39 +0800 on 11 Oct (1318325957), cc Luit wrote: > > > > Hi, everyone, I have a question, > > > > in the shadow_page_fault or ept mechanism, xen will use the > x86_emulation > > > > for some instructions, I''m wondering why it must use it, if after we > fix > > > the > > > > SPT or EPT table, just VMEntry to HVM to re-excute this instruction > but > > > not > > > > emulate in xen, is there some problems? > > > > > > > In the shadow pagetable code, we keep the shadows up-to-date by: > > > > 1 - making all shadowed pagetables read-only; > > > > 2 - intercepting the page faults when the guest writes to them; and > > > > 3 - updating the guest pagetable and the shadow at the same time, > > > > with whatever change the guest was making. > > > > > > > For step 3 we need to emulate the instruction that caused the > pagefault > > > > so that we can tell what was being written. > > > > > > > There are other reasons for the emulator to be called (emulating MMIO > > > > instructions, emulating real-mode &c) but that''s why the shadow > > > > pagetable code uses it. > > > > > > Thanks first of all, I know now it is the Eager mode that SPT is sync > up > > with GPT when guest want to change the page table using the instructions > > emulator, but if for some reason I don''t want xen to emulate such an > > instruction, but just VMEntry to HVM to retry, is there any feasibility > > problems? > > > The emulation can be avoided - in fact the current shadow pagetable > > sometimes lets guests write to shadowed pages and fixes up the shadows > > afterwards (this is called out-of-sync or oos in the code). > > > But if you just return to the guest and retry, the guest will take the > > same fault again unless you have done something to change that. If you > > _have_ done something to make it OK, then just returning > > EXCRET_fault_fixed from sh_page_fault will return to the guest and > > retry the instruction. >I''ve read a slide said that in Lazy mode: when guest os modify the GPT, do not emulate (there is no write-protected PTE, so guest can directly modify it) 1) when the access right ascension, the guest OS will INVLPG to shootdown TLB, so hypervisor can catch the INPLPG inst to sync up the SPT/GPT 2) when access right down, when guest OS access this page it will trap to xen, xen will catch #PF to sync up SPT/GPT, I''m not sure if this can work?> Why do you want to avoid calling the emulator? What is your overall goal? > It might be that tinkering in the shadow pagetables isn''t the best way > to acheive it.because we''re doing some research of security aspect about xen, what''s our goal is avoid xen to access the HVM''s memory in the page fault situation, it''s hard to say it out in short words, we have thought a lot of ways but there is no a simpler one than avoiding the emulation in page_fault. Thanks> > > in other words, in the old time, the shadow page is the Lazy mode, > > that xen will not emulate, and the GPT and SPT is out of sync for some > time, > > besides the lose in performance, is there other problems? > > No, it was really about the performance cost of syncing up all the > shadows on a TLB flush. In retrospect, having fixed some nasty bugs in > the OOS code, I suspect the old shadow code was also incorrect in some > ways but that was an implementation detail, noit architectural. > > Tim. >-- - Luit @ Parallel Processing Institute, Fudan University _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Zhengwang Ruan (Royo)
2011-Oct-11 12:04 UTC
Re: [Xen-devel] why xen use x86_emulation() in page_fault
Hi Tim, I am a freshman to this member-list and interested in some contents in your reply, looking forward to your answer. :) At 2011-10-11 16:18, Tim Deegan wrote:> At 09:39 +0800 on 11 Oct (1318325957), cc Luit wrote: >> Hi, everyone, I have a question, >> in the shadow_page_fault or ept mechanism, xen will use the x86_emulation >> for some instructions, I''m wondering why it must use it, if after we fix the >> SPT or EPT table, just VMEntry to HVM to re-excute this instruction but not >> emulate in xen, is there some problems? > In the shadow pagetable code, we keep the shadows up-to-date by: > 1 - making all shadowed pagetables read-only; > 2 - intercepting the page faults when the guest writes to them; and > 3 - updating the guest pagetable and the shadow at the same time, > with whatever change the guest was making. > > For step 3 we need to emulate the instruction that caused the pagefault > so that we can tell what was being written. > > There are other reasons for the emulator to be called (emulating MMIO > instructions, emulating real-mode&c) but that''s why the shadow > pagetable code uses it.1, Does this emulator set up a emulation environment to execute instructions for a VM(guest OS)? 2, Real-mode code you talked above is used to execute the first 16-bit initial code of a guest OS? Does it have other usages in current Xen implementation? Thanks, Royo (Zhengwang)> Cheers, > > Tim. > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Tim Deegan
2011-Oct-11 12:21 UTC
Re: [Xen-devel] why xen use x86_emulation() in page_fault
At 20:03 +0800 on 11 Oct (1318363429), cc Luit wrote:> I''ve read a slide said that in Lazy mode: > when guest os modify the GPT, do not emulate (there is no > write-protected PTE, so guest can directly modify it) > 1) when the access right ascension, the guest OS will INVLPG to shootdown > TLB, so hypervisor can catch the INPLPG inst to sync up the SPT/GPT > 2) when access right down, when guest OS access this page it will trap to > xen, xen will catch #PF to sync up SPT/GPT, > > I''m not sure if this can work?I don''t quite understand your English, but yes, you can let the guest write to page tables and sync up the shadows afterwards. You need to sync the shadows whenever real hardware would re-read pagetables. That is, at least: - When the guest executes INVLPG - When the guest writes to CR3 - When the shadowing code adds a new l2e, l3e or l4e that points to an existing shadow. (and possibly some other cases I don;t remember right now).> > Why do you want to avoid calling the emulator? What is your overall goal? > > It might be that tinkering in the shadow pagetables isn''t the best way > > to acheive it. > > because we''re doing some research of security aspect about xen, what''s our > goal is avoid xen to access the HVM''s memory in the page fault situation, > it''s hard to say it out in short words, we have thought a lot of ways but > there is no a simpler one than avoiding the emulation in page_fault.Ah, I see - this is following on from the CloudVisor work, is it? Why don''t you just use EPT/NPT instead of shadow pagetables? That code is a lot simpler and the end result is easier to validate than shadow pagetables are. Is it particularly important to support shadow pagetables? In either case, there are some instructions that Xen will always emulate - for example: - MMIO accesses to emulated hardware - Some real-mode instructions on older Intel hardware so even if you rewrote the shadow pagetable code not to emulate PT writes you still have to handle those. Cheers, Tim. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Tue, Oct 11, 2011 at 8:21 PM, Tim Deegan <tim@xen.org> wrote:> At 20:03 +0800 on 11 Oct (1318363429), cc Luit wrote: > > I''ve read a slide said that in Lazy mode: > > when guest os modify the GPT, do not emulate (there is no > > write-protected PTE, so guest can directly modify it) > > 1) when the access right ascension, the guest OS will INVLPG to > shootdown > > TLB, so hypervisor can catch the INPLPG inst to sync up the SPT/GPT > > 2) when access right down, when guest OS access this page it will trap > to > > xen, xen will catch #PF to sync up SPT/GPT, > > > > I''m not sure if this can work? > > > I don''t quite understand your English, but yes, you can let the guest > > write to page tables and sync up the shadows afterwards. You need to > > sync the shadows whenever real hardware would re-read pagetables. That > > is, at least: > > - When the guest executes INVLPG > > - When the guest writes to CR3 > > - When the shadowing code adds a new l2e, l3e or l4e that points to an > > existing shadow. > > (and possibly some other cases I don;t remember right now). >really sorry about my poor english:-) I will consider your suggestions carefully~ thanks!> > > > Why do you want to avoid calling the emulator? What is your overall > goal? > > > It might be that tinkering in the shadow pagetables isn''t the best way > > > to acheive it. > > > > because we''re doing some research of security aspect about xen, what''s > our > > goal is avoid xen to access the HVM''s memory in the page fault situation, > > it''s hard to say it out in short words, we have thought a lot of ways but > > there is no a simpler one than avoiding the emulation in page_fault. > > > Ah, I see - this is following on from the CloudVisor work, is it? >yeah~ CloudVisor was just our lab''s work, but ours is a little different:-)> > > Why don''t you just use EPT/NPT instead of shadow pagetables? That code > > is a lot simpler and the end result is easier to validate than shadow > > pagetables are. Is it particularly important to support shadow > pagetables? >yeah, we have consider that, and of course the EPT is much more convenient, BTW, do you know is there any simulator that can well support EPT of xen? the shadow page''s problem is a little relevent to the selection of simulator~> > > In either case, there are some instructions that Xen will always > > emulate - for example: > > - MMIO accesses to emulated hardware > > - Some real-mode instructions on older Intel hardware >yeah, that''s true, and we are also considering it:-)> > > so even if you rewrote the shadow pagetable code not to emulate PT writes > > you still have to handle those. > > Cheers, > > Tim. >-- - Luit @ Parallel Processing Institute, Fudan University _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Zhengwang Ruan (Royo)
2011-Oct-11 13:21 UTC
Re: [Xen-devel] why xen use x86_emulation() in page_fault
Hi Tim, I am a freshman to this member-list and interested in some contents in your reply, looking forward to your answer. :) At 2011-10-11 16:18, Tim Deegan wrote:> At 09:39 +0800 on 11 Oct (1318325957), cc Luit wrote: >> Hi, everyone, I have a question, >> in the shadow_page_fault or ept mechanism, xen will use the x86_emulation >> for some instructions, I''m wondering why it must use it, if after we fix the >> SPT or EPT table, just VMEntry to HVM to re-excute this instruction but not >> emulate in xen, is there some problems? > In the shadow pagetable code, we keep the shadows up-to-date by: > 1 - making all shadowed pagetables read-only; > 2 - intercepting the page faults when the guest writes to them; and > 3 - updating the guest pagetable and the shadow at the same time, > with whatever change the guest was making. > > For step 3 we need to emulate the instruction that caused the pagefault > so that we can tell what was being written. > > There are other reasons for the emulator to be called (emulating MMIO > instructions, emulating real-mode&c) but that''s why the shadow > pagetable code uses it.1, Does this emulator set up a emulation environment to execute instructions for a VM(guest OS)? 2, Real-mode code you talked above is used to execute the first 16-bit initial code of a guest OS? Does it have other usages in current Xen implementation? Thanks, Royo (Zhengwang)> Cheers, > > Tim. > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Tue, Oct 11, 2011 at 8:04 PM, Zhengwang Ruan (Royo) < zhengwang.ruan@gmail.com> wrote:> Hi Tim, > > I am a freshman to this member-list and interested in some contents in your > reply, looking forward to your answer. :) > > > > At 2011-10-11 16:18, Tim Deegan wrote: > >> At 09:39 +0800 on 11 Oct (1318325957), cc Luit wrote: >> >>> Hi, everyone, I have a question, >>> in the shadow_page_fault or ept mechanism, xen will use the x86_emulation >>> for some instructions, I''m wondering why it must use it, if after we fix >>> the >>> SPT or EPT table, just VMEntry to HVM to re-excute this instruction but >>> not >>> emulate in xen, is there some problems? >>> >> In the shadow pagetable code, we keep the shadows up-to-date by: >> 1 - making all shadowed pagetables read-only; >> 2 - intercepting the page faults when the guest writes to them; and >> 3 - updating the guest pagetable and the shadow at the same time, >> with whatever change the guest was making. >> >> For step 3 we need to emulate the instruction that caused the pagefault >> so that we can tell what was being written. >> >> There are other reasons for the emulator to be called (emulating MMIO >> instructions, emulating real-mode&c) but that''s why the shadow >> pagetable code uses it. >> > > 1, Does this emulator set up a emulation environment to execute > instructions for a VM(guest OS)? >there is a x86_emulate() in xen/arch/x86/x86_emulate/x86_emulate.c(my version is xen-4.0.1), and it will emulate some instructions (I think both guest OS and Domain-0) in some conditions(like what Tim said)> 2, Real-mode code you talked above is used to execute the first 16-bit > initial code of a guest OS? Does it have other usages in current Xen > implementation? >I''m also curious about the real-mode, looking forward to the replay~> > Thanks, > Royo (Zhengwang) > > Cheers, >> >> Tim. >> >> >> ______________________________**_________________ >> Xen-devel mailing list >> Xen-devel@lists.xensource.com >> http://lists.xensource.com/**xen-devel<http://lists.xensource.com/xen-devel> >> > > ______________________________**_________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/**xen-devel<http://lists.xensource.com/xen-devel> >-- - Luit @ Parallel Processing Institute, Fudan University _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Tim Deegan
2011-Oct-13 11:47 UTC
Re: [Xen-devel] why xen use x86_emulation() in page_fault
At 21:07 +0800 on 11 Oct (1318367233), cc Luit wrote:> > > Why don''t you just use EPT/NPT instead of shadow pagetables? That code > > > is a lot simpler and the end result is easier to validate than shadow > > > pagetables are. Is it particularly important to support shadow > > pagetables? > > > yeah, we have consider that, and of course the EPT is much more convenient, > BTW, do you know is there any simulator that can well support EPT of xen? > the shadow page''s problem is a little relevent to the selection of > simulator~There''s experimental support in Xen itself for running a hypervisor inside a HVM guest. On AMD, that includes (and indeed requires) NPT in the guest hypervisor. Unfortunately, on Intel it does not include EPT. There may be some other simulators that have EPT/NPT - I don''t know. Cheers, Tim. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Tim Deegan
2011-Oct-13 11:53 UTC
Re: [Xen-devel] why xen use x86_emulation() in page_fault
Hi, At 21:46 +0800 on 11 Oct (1318369564), cc Luit wrote:> On Tue, Oct 11, 2011 at 8:04 PM, Zhengwang Ruan (Royo) <zhengwang.ruan@gmail.com> wrote: > > 2, Real-mode code you talked above is used to execute the first 16-bit > > initial code of a guest OS? Does it have other usages in current Xen > > implementation? > > > I''m also curious about the real-mode, looking forward to the replay~It is used whenever the guest OS runs in real mode. That''s usually just for a short while in the boot loader and the start of the kernel, but that would depend on what OS you run in the guest. DOS, for example, would use it all the time. :) Linux (like most modern OSes) avoids real mode once it''s booted, but if it ever needs to reset a CPU (e.g. if you support CPU hotplug) then that CPU will run in real mode briefly after it''s reset. Tim. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Christoph Egger
2011-Oct-13 12:21 UTC
Re: [Xen-devel] why xen use x86_emulation() in page_fault
On 10/13/11 13:47, Tim Deegan wrote:> At 21:07 +0800 on 11 Oct (1318367233), cc Luit wrote: >>>> Why don''t you just use EPT/NPT instead of shadow pagetables? That code >>>> is a lot simpler and the end result is easier to validate than shadow >>>> pagetables are. Is it particularly important to support shadow >>> pagetables? >>> >> yeah, we have consider that, and of course the EPT is much more convenient, >> BTW, do you know is there any simulator that can well support EPT of xen? >> the shadow page''s problem is a little relevent to the selection of >> simulator~ > > There''s experimental support in Xen itself for running a hypervisor > inside a HVM guest. On AMD, that includes (and indeed requires) NPT in > the guest hypervisor.Not quite right: guest hypervisor can use NPT but this is not required. It may use shadow paging. But an AMD cpu with NPT capability is required since shadow-on-shadow is not supported. > Unfortunately, on Intel it does not include EPT.> > There may be some other simulators that have EPT/NPT - I don''t know. > > Cheers, > > Tim.-- ---to satisfy European Law for business letters: Advanced Micro Devices GmbH Einsteinring 24, 85689 Dornach b. Muenchen Geschaeftsfuehrer: Alberto Bozzo, Andrew Bowd Sitz: Dornach, Gemeinde Aschheim, Landkreis Muenchen Registergericht Muenchen, HRB Nr. 43632 _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel