weiming
2008-Mar-09 01:50 UTC
[Xen-devel] how page faults are handled in paravirtualized xen guests?
Hi, I got contradicted information about the path of page fault handling under para-virtualized linux: 1) From some web resources: page fault interruption is trapped by xen hypervisor and it then dispatches to corresponding guest OS 2) From the book "a definitive guide to xen hypervisor": when a guest is scheduled to run by hypervisor, hypervisor installes the guest''s IDT (except for special interrupts like timer) to CPU. So almost all interrupts are handled by guest OS directly, bypassing the xen hypervisor. Which one is correct? (My guess is second one because it''s more efficient) Thanks Weiming _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
房海峰
2008-Mar-09 02:27 UTC
Re: [Xen-devel] how page faults are handled in paravirtualized xenguests?
hello, my friend: According to my research with source code of xen, i think the process is that: first, when Guest OS is created by Xen/Domian0, it install it's IDT through "set_trap_table" hypercall. second, during Xen/GuestOS running, all interrupt/trap will result a trap into Ring0, that is the layer of Xen. for some trap, such as system-call, they will be dealt with directly through interrupt hardware. For the most of other traps, they will be dealt by xen, and then dispatched to corresponding Guest OS. in addition, there are other details to study in-depth. 2008-03-09 房海峰 发件人： weiming 发送时间： 2008-03-09 09:33:41 收件人： Xen Developers 抄送： 主题： [Xen-devel] how page faults are handled in paravirtualized xenguests? Hi, I got contradicted information about the path of page fault handling under para-virtualized linux: 1) From some web resources: page fault interruption is trapped by xen hypervisor and it then dispatches to corresponding guest OS 2) From the book "a definitive guide to xen hypervisor": when a guest is scheduled to run by hypervisor, hypervisor installes the guest's IDT (except for special interrupts like timer) to CPU. So almost all interrupts are handled by guest OS directly, bypassing the xen hypervisor. Which one is correct? (My guess is second one because it's more efficient) Thanks Weiming _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Daniel Stodden
2008-Mar-09 14:33 UTC
Re: [Xen-devel] how page faults are handled in paravirtualized xenguests?
On Sun, 2008-03-09 at 10:27 +0800, 房海峰 wrote:> hello, my friend: > > According to my research with source code of xen, i think the process > is that: > > first, when Guest OS is created by Xen/Domian0, it install it''s IDT > through "set_trap_table" hypercall. > second, during Xen/GuestOS running, all interrupt/trap will result a > trap into Ring0, that is the layer of Xen. > for some trap, such as system-call, they will be dealt with directly > through interrupt hardware. > For the most of other traps, they will be dealt by xen, and then > dispatched to corresponding Guest OS. > > in addition, there are other details to study in-depth.Generally, the IDT entry may either point to xen or directly to the trap handler installed by the guest. For system calls, it presently depends on the architecture: x86_32 has separate trap instructions for hypercalls and system calls, which can be forwarded directly. But is not the case for 64-bit systems. In fault handling, it depends on the type of fault. It may be either due to the virtualization layer, then transparently fixed by Xen (e.g. instruction emulation). Or the guest may be responsible (e.g. a process page fault). In that case the fault will be forwarded. Some faults (e.g. division by zero) are indeed never Xen''s business. Still, you''ll find the idt entry to point to xen, instead of directly to the ring1 kernel. See the use of DO_ERROR_*() in x86/traps.c. The reason is simply that bugs in Xen (of course, that''s impossible :), but you never know) would not be caught otherwise. So, the bottom line is that all goes through Xen, except for 32-on-32 bit system calls. hth, daniel -- Daniel Stodden LRR - Lehrstuhl für Rechnertechnik und Rechnerorganisation Institut für Informatik der TU München D-85748 Garching http://www.lrr.in.tum.de/~stodden mailto:stodden@cs.tum.edu PGP Fingerprint: F5A4 1575 4C56 E26A 0B33 3D80 457E 82AE B0D8 735B _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
weiming
2008-Mar-09 16:37 UTC
Re: [Xen-devel] how page faults are handled in paravirtualized xenguests?
Hi Daniel & Haifeng, Thanks for your elaboration. Now I'm almost clear on this problem. Daniel, Could you please explain why "not the case for 64-bit systems." For system calls, a regular OS will issure INT 0x80 to do system calls. In Xen, this interrupt will be handled by the guest directly. In guest OS can use 0x82 interrupt to call hypercalls. So how is x86_64 different? I'm more interested in 64-bit system because I'll work on it. Thanks a lot! Weiming On Sun, Mar 9, 2008 at 10:33 AM, Daniel Stodden <stodden@cs.tum.edu> wrote:> On Sun, 2008-03-09 at 10:27 +0800, 房海峰 wrote: > > hello, my friend: > > > > According to my research with source code of xen, i think the process > > is that: > > > > first, when Guest OS is created by Xen/Domian0, it install it's IDT > > through "set_trap_table" hypercall. > > second, during Xen/GuestOS running, all interrupt/trap will result a > > trap into Ring0, that is the layer of Xen. > > for some trap, such as system-call, they will be dealt with directly > > through interrupt hardware. > > For the most of other traps, they will be dealt by xen, and then > > dispatched to corresponding Guest OS. > > > > in addition, there are other details to study in-depth. > > Generally, the IDT entry may either point to xen or directly to the > trap handler installed by the guest. > > For system calls, it presently depends on the architecture: x86_32 has > separate trap instructions for hypercalls and system calls, which can be > forwarded directly. But is not the case for 64-bit systems. > > In fault handling, it depends on the type of fault. It may be either due > to the virtualization layer, then transparently fixed by Xen (e.g. > instruction emulation). Or the guest may be responsible (e.g. a process > page fault). In that case the fault will be forwarded. > > Some faults (e.g. division by zero) are indeed never Xen's business. > Still, you'll find the idt entry to point to xen, instead of directly to > the ring1 kernel. See the use of DO_ERROR_*() in x86/traps.c. The > reason is simply that bugs in Xen (of course, that's impossible :), but > you never know) would not be caught otherwise. > > So, the bottom line is that all goes through Xen, except for 32-on-32 > bit system calls. > > hth, > daniel > > -- > Daniel Stodden > LRR - Lehrstuhl für Rechnertechnik und Rechnerorganisation > Institut für Informatik der TU München D-85748 Garching > http://www.lrr.in.tum.de/~stodden <http://www.lrr.in.tum.de/%7Estodden> > mailto:stodden@cs.tum.edu > PGP Fingerprint: F5A4 1575 4C56 E26A 0B33 3D80 457E 82AE B0D8 735B > > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2008-Mar-09 17:00 UTC
Re: [Xen-devel] how page faults are handled in paravirtualized xenguests?
Hypercalls and syscalls are both performed using the SYSCALL instruction for 64-bit guests. SYSCALL from 64-bit guest kernel triggers the hypercall path; SYSCALL from 64-bit guest application causes entry to 64-bit guest kernel. -- Keir On 9/3/08 16:37, "weiming" <zephyr.zhao@gmail.com> wrote:> Hi Daniel & Haifeng, > > Thanks for your elaboration. > Now I''m almost clear on this problem. > > Daniel, Could you please explain why "not the case for 64-bit systems." > For system calls, a regular OS will issure INT 0x80 to do system calls. In > Xen, this interrupt will be handled by the guest directly. In guest OS can use > 0x82 interrupt to call hypercalls. > > So how is x86_64 different? I''m more interested in 64-bit system because I''ll > work on it. > > Thanks a lot! > Weiming > > > > On Sun, Mar 9, 2008 at 10:33 AM, Daniel Stodden <stodden@cs.tum.edu> wrote: >> On Sun, 2008-03-09 at 10:27 +0800, 房海峰 wrote: >>> > hello, my friend: >>> > >>> > According to my research with source code of xen, i think the process >>> > is that: >>> > >>> > first, when Guest OS is created by Xen/Domian0, it install it''s IDT >>> > through "set_trap_table" hypercall. >>> > second, during Xen/GuestOS running, all interrupt/trap will result a >>> > trap into Ring0, that is the layer of Xen. >>> > for some trap, such as system-call, they will be dealt with directly >>> > through interrupt hardware. >>> > For the most of other traps, they will be dealt by xen, and then >>> > dispatched to corresponding Guest OS. >>> > >>> > in addition, there are other details to study in-depth. >> >> Generally, the IDT entry may either point to xen or directly to the >> trap handler installed by the guest. >> >> For system calls, it presently depends on the architecture: x86_32 has >> separate trap instructions for hypercalls and system calls, which can be >> forwarded directly. But is not the case for 64-bit systems. >> >> In fault handling, it depends on the type of fault. It may be either due >> to the virtualization layer, then transparently fixed by Xen (e.g. >> instruction emulation). Or the guest may be responsible (e.g. a process >> page fault). In that case the fault will be forwarded. >> >> Some faults (e.g. division by zero) are indeed never Xen''s business. >> Still, you''ll find the idt entry to point to xen, instead of directly to >> the ring1 kernel. See the use of DO_ERROR_*() in x86/traps.c. The >> reason is simply that bugs in Xen (of course, that''s impossible :), but >> you never know) would not be caught otherwise. >> >> So, the bottom line is that all goes through Xen, except for 32-on-32 >> bit system calls. >> >> hth, >> daniel >> >> -- >> Daniel Stodden >> LRR - Lehrstuhl für Rechnertechnik und Rechnerorganisation >> Institut für Informatik der TU München D-85748 Garching >> http://www.lrr.in.tum.de/~stodden <http://www.lrr.in.tum.de/%7Estodden> >> mailto:stodden@cs.tum.edu >> PGP Fingerprint: F5A4 1575 4C56 E26A 0B33 3D80 457E 82AE B0D8 735B >> >> > > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
tgh
2008-Mar-11 01:38 UTC
Re: [Xen-devel] how page faults are handled in paravirtualized xenguests?
hi and that is ,in 64-bit,guest application entry into guestos kernel directly through SYSCALL,or not? Thnaks Keir Fraser 写道:> Hypercalls and syscalls are both performed using the SYSCALL > instruction for 64-bit guests. SYSCALL from 64-bit guest kernel > triggers the hypercall path; SYSCALL from 64-bit guest application > causes entry to 64-bit guest kernel. > > -- Keir > > On 9/3/08 16:37, "weiming" <zephyr.zhao@gmail.com> wrote: > > Hi Daniel & Haifeng, > > Thanks for your elaboration. > Now I''m almost clear on this problem. > > Daniel, Could you please explain why "not the case for 64-bit > systems." > For system calls, a regular OS will issure INT 0x80 to do system > calls. In Xen, this interrupt will be handled by the guest > directly. In guest OS can use 0x82 interrupt to call hypercalls. > > So how is x86_64 different? I''m more interested in 64-bit system > because I''ll work on it. > > Thanks a lot! > Weiming > > > > On Sun, Mar 9, 2008 at 10:33 AM, Daniel Stodden > <stodden@cs.tum.edu> wrote: > > On Sun, 2008-03-09 at 10:27 +0800, 房海峰 wrote: > > hello, my friend: > > > > According to my research with source code of xen, i think the > process > > is that: > > > > first, when Guest OS is created by Xen/Domian0, it install > it''s IDT > > through "set_trap_table" hypercall. > > second, during Xen/GuestOS running, all interrupt/trap will > result a > > trap into Ring0, that is the layer of Xen. > > for some trap, such as system-call, they will be dealt with > directly > > through interrupt hardware. > > For the most of other traps, they will be dealt by xen, and then > > dispatched to corresponding Guest OS. > > > > in addition, there are other details to study in-depth. > > Generally, the IDT entry may either point to xen or directly > to the > trap handler installed by the guest. > > For system calls, it presently depends on the architecture: > x86_32 has > separate trap instructions for hypercalls and system calls, > which can be > forwarded directly. But is not the case for 64-bit systems. > > In fault handling, it depends on the type of fault. It may be > either due > to the virtualization layer, then transparently fixed by Xen (e.g. > instruction emulation). Or the guest may be responsible (e.g. > a process > page fault). In that case the fault will be forwarded. > > Some faults (e.g. division by zero) are indeed never Xen''s > business. > Still, you''ll find the idt entry to point to xen, instead of > directly to > the ring1 kernel. See the use of DO_ERROR_*() in x86/traps.c. The > reason is simply that bugs in Xen (of course, that''s > impossible :), but > you never know) would not be caught otherwise. > > So, the bottom line is that all goes through Xen, except for > 32-on-32 > bit system calls. > > hth, > daniel > > -- > Daniel Stodden > LRR - Lehrstuhl für Rechnertechnik und > Rechnerorganisation > Institut für Informatik der TU München D-85748 > Garching > http://www.lrr.in.tum.de/~stodden > <http://www.lrr.in.tum.de/%7Estodden> > <http://www.lrr.in.tum.de/%7Estodden> > mailto:stodden@cs.tum.edu > PGP Fingerprint: F5A4 1575 4C56 E26A 0B33 3D80 457E 82AE B0D8 > 735B > > > > > ------------------------------------------------------------------------ > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel > > > ------------------------------------------------------------------------ > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Daniel Stodden
2008-Mar-11 06:21 UTC
Re: [Xen-devel] how page faults are handled in paravirtualized xenguests?
On Tue, 2008-03-11 at 09:38 +0800, tgh wrote:> hi > and that is ,in 64-bit,guest application entry into guestos kernel > directly through SYSCALL,or not?no, you can''t. programs use syscall to enter the kernel, and the kernel is using syscall to enter xen. since you can only configure one entry point at a time (and both xen and applications run in ring 3 anyway), xen is handling all syscalls, then checks the context it was issued from, and forwards the call to to the guest kernel if it was user code. regards, daniel -- Daniel Stodden LRR - Lehrstuhl für Rechnertechnik und Rechnerorganisation Institut für Informatik der TU München D-85748 Garching http://www.lrr.in.tum.de/~stodden mailto:stodden@cs.tum.edu PGP Fingerprint: F5A4 1575 4C56 E26A 0B33 3D80 457E 82AE B0D8 735B _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
weiming
2008-Mar-11 14:43 UTC
Re: [Xen-devel] how page faults are handled in paravirtualized xenguests?
some material says that guest kernel runs in ring 1 and applications in ring 3. Is this for x86 32 ? On Tue, Mar 11, 2008 at 2:21 AM, Daniel Stodden <stodden@cs.tum.edu> wrote:> On Tue, 2008-03-11 at 09:38 +0800, tgh wrote: > > hi > > and that is ,in 64-bit,guest application entry into guestos kernel > > directly through SYSCALL,or not? > > no, you can''t. programs use syscall to enter the kernel, and the kernel > is using syscall to enter xen. since you can only configure one entry > point at a time (and both xen and applications run in ring 3 anyway), > xen is handling all syscalls, then checks the context it was issued > from, and forwards the call to to the guest kernel if it was user code. > > regards, > daniel > > -- > Daniel Stodden > LRR - Lehrstuhl für Rechnertechnik und Rechnerorganisation > Institut für Informatik der TU München D-85748 Garching > http://www.lrr.in.tum.de/~stodden <http://www.lrr.in.tum.de/%7Estodden> > mailto:stodden@cs.tum.edu > PGP Fingerprint: F5A4 1575 4C56 E26A 0B33 3D80 457E 82AE B0D8 735B > > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
weiming
2008-Mar-11 15:37 UTC
Re: [Xen-devel] how page faults are handled in paravirtualized xenguests?
and how can xen tell if the hypercall is from kernel space or user space in x86_64? On Tue, Mar 11, 2008 at 9:43 AM, weiming <zephyr.zhao@gmail.com> wrote:> some material says that guest kernel runs in ring 1 and applications in > ring 3. Is this for x86 32 ? > > On Tue, Mar 11, 2008 at 2:21 AM, Daniel Stodden <stodden@cs.tum.edu> > wrote: > > > On Tue, 2008-03-11 at 09:38 +0800, tgh wrote: > > > hi > > > and that is ,in 64-bit,guest application entry into guestos kernel > > > directly through SYSCALL,or not? > > > > no, you can''t. programs use syscall to enter the kernel, and the kernel > > is using syscall to enter xen. since you can only configure one entry > > point at a time (and both xen and applications run in ring 3 anyway), > > xen is handling all syscalls, then checks the context it was issued > > from, and forwards the call to to the guest kernel if it was user code. > > > > regards, > > daniel > > > > -- > > Daniel Stodden > > LRR - Lehrstuhl für Rechnertechnik und Rechnerorganisation > > Institut für Informatik der TU München D-85748 Garching > > http://www.lrr.in.tum.de/~stodden <http://www.lrr.in.tum.de/%7Estodden> mailto: > > stodden@cs.tum.edu > > PGP Fingerprint: F5A4 1575 4C56 E26A 0B33 3D80 457E 82AE B0D8 735B > > > > > > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Daniel Stodden
2008-Mar-11 16:26 UTC
Re: [Xen-devel] how page faults are handled in paravirtualized xenguests?
On Tue, 2008-03-11 at 10:43 -0400, weiming wrote:> some material says that guest kernel runs in ring 1 and applications > in ring 3. Is this for x86 32 ?yes. The ring1 kernel only works if you can play some tricks on virtual memory management in order to protect ring1 kernel memory from ring3 user space (paging alone only distinguishes 0 from 1-3). On x86_32, these tricks involve segmentation. x86_64 promotes ''flat'' addressing instead, with only a few segment register left for convenience. So x86_32 runs in a ''0/1/3'' model (xen/kernel/user), while x86_64 has to fall back to a more general ''0/3/3''. user versus supervisor mode is basically virtual. Xen will switch page tables when transferring control between user and kernel code. regards, daniel -- Daniel Stodden LRR - Lehrstuhl für Rechnertechnik und Rechnerorganisation Institut für Informatik der TU München D-85748 Garching http://www.lrr.in.tum.de/~stodden mailto:stodden@cs.tum.edu PGP Fingerprint: F5A4 1575 4C56 E26A 0B33 3D80 457E 82AE B0D8 735B _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Daniel Stodden
2008-Mar-11 16:27 UTC
Re: [Xen-devel] how page faults are handled in paravirtualized xenguests?
On Tue, 2008-03-11 at 10:37 -0500, weiming wrote:> and how can xen tell if the hypercall is from kernel space or user > space in x86_64?The VMM simply keeps track of it in software. Starting from kernel mode, an IRET will return to virtual user mode, a subsequent ''SYSCALL'' will move control back to the kernel. hth, daniel -- Daniel Stodden LRR - Lehrstuhl für Rechnertechnik und Rechnerorganisation Institut für Informatik der TU München D-85748 Garching http://www.lrr.in.tum.de/~stodden mailto:stodden@cs.tum.edu PGP Fingerprint: F5A4 1575 4C56 E26A 0B33 3D80 457E 82AE B0D8 735B _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Daniel Stodden
2008-Mar-12 09:37 UTC
Re: [Xen-devel] how page faults are handled in paravirtualized xenguests?
On Tue, 2008-03-11 at 17:28 +0100, Daniel Stodden wrote:> yes. The ring1 kernel only works if you can play some tricks on virtual > memory management in order to protect ring1 kernel memory from ring3 > user space (paging alone only distinguishes 0 from 1-3). On x86_32,i should probably correct myself. wrong story, sorry. of course, paging distingushes ring 0-2 from 3. so 32-bit in 0/1/3 with segmentation protects xen from the kernel, and 32-bit paging then protects xen and the kernel from user mode. 64-bit paging in 0/3/3 protects xen from both kernel and user mode, but as i mentioned, you need a PT switch to keep the kernel safe. maybe more than you asked for :} daniel -- Daniel Stodden LRR - Lehrstuhl für Rechnertechnik und Rechnerorganisation Institut für Informatik der TU München D-85748 Garching http://www.lrr.in.tum.de/~stodden mailto:stodden@cs.tum.edu PGP Fingerprint: F5A4 1575 4C56 E26A 0B33 3D80 457E 82AE B0D8 735B _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
weiming
2008-Mar-23 02:29 UTC
Re: [Xen-devel] how page faults are handled in paravirtualized xenguests?
Hi Daniel, Now, I'm asking for more. :} 1) As you said, in 64-bit mode, PT switch is needed. But on your other post, you said: "So instead one lets a process, its guest kernel and the VMM share one virtual address space. As with regular OSes: one address space per guest process. Control transfers between process, kernel and VMM does not switch between virtual memories, but only privileges, stacks, and EIP. -> much faster." The statements above implies that PT switch is not needed. Is it true? 2) in 0/3/3 mode, how guest kernel protects it from application? Simply use different page tables? Will it be too costly (need TLB flush)? Why not use FS, GS to implement 0/1/3 mode? Many thanks! Weiming On Wed, Mar 12, 2008 at 5:37 AM, Daniel Stodden <stodden@cs.tum.edu> wrote:> On Tue, 2008-03-11 at 17:28 +0100, Daniel Stodden wrote: > > yes. The ring1 kernel only works if you can play some tricks on virtual > > memory management in order to protect ring1 kernel memory from ring3 > > user space (paging alone only distinguishes 0 from 1-3). On x86_32, > > i should probably correct myself. wrong story, sorry. > > of course, paging distingushes ring 0-2 from 3. > > so 32-bit in 0/1/3 with segmentation protects xen from the kernel, and > 32-bit paging then protects xen and the kernel from user mode. > > 64-bit paging in 0/3/3 protects xen from both kernel and user mode, but > as i mentioned, you need a PT switch to keep the kernel safe. > > maybe more than you asked for :} > > daniel > > -- > Daniel Stodden > LRR - Lehrstuhl für Rechnertechnik und Rechnerorganisation > Institut für Informatik der TU München D-85748 Garching > http://www.lrr.in.tum.de/~stodden <http://www.lrr.in.tum.de/%7Estodden> > mailto:stodden@cs.tum.edu > PGP Fingerprint: F5A4 1575 4C56 E26A 0B33 3D80 457E 82AE B0D8 735B > > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel