Good morning,
I am new to Xen, and trying to understand how Xen works. I have some
questions about virtualization of interrupts on Xen. Thanks in advance
for all helps.
1) From papers about Xen architecture found at Xen homepage, Xen
seemingly intercepts interrupts of Dom1 (paravirtualization). So
interrupts occur in Dom1''s context are sent to Xen, and Xen handles
them. Is that correct?
But then how can Dom1 gets the interrupts? For example if int3 occurs
inside Dom1, Xen intercepts int3, but then how debugger (like gdb) can
still work in Dom1? Maybe Xen sends that interrupt back to Dom1 after
processing it in VMM layer?
2) I have read
linux-xen-sparse/arch/i386/kernel/{traps-xen.c,traps.c,vmlinux.lds.S},
and looks like Dom1 still handles interrupts itself? Why, because I
suppose that the VMM already intercepts those interrupts?
3) How is linux-xen-sparse/arch/i386/kernel/traps-xen.c compiled,
while it is not refereed to in
linux-xen-sparse/arch/i386/kernel/Makefile? I see traps.c is refereed
instead of traps-xen.c, but I still saw traps-xen.o there after
compiling? What traps.c does, and what traps-xen.c does, since both
are very similar to me.
Thanks.
Jun
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
On Fri, 2006-08-11 at 19:03 +0900, Jun Koi wrote:> Good morning, > > I am new to Xen, and trying to understand how Xen works. I have some > questions about virtualization of interrupts on Xen. Thanks in advance > for all helps. > > 1) From papers about Xen architecture found at Xen homepage, Xen > seemingly intercepts interrupts of Dom1 (paravirtualization). So > interrupts occur in Dom1''s context are sent to Xen, and Xen handles > them. Is that correct?similar to interrupts to a native operating systems, interrupts could occur in just any context (as long as they''re not blocked). the domU never ''receives'' the interrupt in a way that it won''t even be able to install a handler for it. the idt belongs to xen, and xen is where it is actually handled, interrupting whatever used to be running when it occurred.> But then how can Dom1 gets the interrupts? For example if int3 occurs > inside Dom1, Xen intercepts int3, but then how debugger (like gdb) can > still work in Dom1? Maybe Xen sends that interrupt back to Dom1 after > processing it in VMM layer?hardware interrupts in turn drive ''virtual interrupts'', which are in the paravirtualized case a pure software abstraction. in xen, you''ll find a lot of references to a concept called ''event channels'', and that''s just that. so it''s xen making then decision on whether a domain interrupt handler is called, which one''s called, when it''s called, and whether anything called at all. don''t worry, in practice, xen is doing the right thing, but that''s the way it needs to work. :).> 2) I have read > linux-xen-sparse/arch/i386/kernel/{traps-xen.c,traps.c,vmlinux.lds.S}, > and looks like Dom1 still handles interrupts itself? Why, because I > suppose that the VMM already intercepts those interrupts?an guest system, whether it''s virtualized or not, is not living in a vacuum. it needs some kind of signalling from the hardware environment, whether it''s virtual or physical, in order to get notified of events. that''s ''asynchronous input'' of some kind, not necessarily purelay hardware interrupt driven, though. e.g. it needs a regular timer interrupt as well as as notification of events regarding virtual device interfaces. so it still has interrupt handlers. it is, however, not modifying the real IDT. see e.g. trap_init() in traps-xen.c and read about HYPERVISOR_set_trap_table in the interface documentations.> 3) How is linux-xen-sparse/arch/i386/kernel/traps-xen.c compiled, > while it is not refereed to in > linux-xen-sparse/arch/i386/kernel/Makefile? I see traps.c is refereed > instead of traps-xen.c, but I still saw traps-xen.o there after > compiling? What traps.c does, and what traps-xen.c does, since both > are very similar to me.uhhm. now you got me. i noticed the file names changed at some point, i didn''t bother why yet, or how the makefiles changed. maybe someone from xensource can tell. regards, daneil -- Daniel Stodden LRR - Lehrstuhl für Rechnertechnik und Rechnerorganisation Institut für Informatik der TU München D-85748 Garching http://www.lrr.in.tum.de/~stodden mailto:stodden@cs.tum.edu PGP Fingerprint: F5A4 1575 4C56 E26A 0B33 3D80 457E 82AE B0D8 735B _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Fri, 2006-08-11 at 12:39 +0200, Daniel Stodden wrote:> > > 3) How is linux-xen-sparse/arch/i386/kernel/traps-xen.c compiled, > > while it is not refereed to in > > linux-xen-sparse/arch/i386/kernel/Makefile? I see traps.c is > refereed > > instead of traps-xen.c, but I still saw traps-xen.o there after > > compiling? What traps.c does, and what traps-xen.c does, since both > > are very similar to me. > > uhhm. now you got me. i noticed the file names changed at some point, > i didn''t bother why yet, or how the makefiles changed. maybe someone > from xensource can tell.obj-y := $(call cherrypickxen, $(obj-y)) at the end of the Makefile will rewrite foo.o as foo-xen.o if there is a foo-xen.c or foo-xen.S available. The function is defined in scripts/Makefile.xen Ian. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hi Daniel, On 8/11/06, Daniel Stodden <stodden@cs.tum.edu> wrote:> On Fri, 2006-08-11 at 19:03 +0900, Jun Koi wrote: > > Good morning, > > > > I am new to Xen, and trying to understand how Xen works. I have some > > questions about virtualization of interrupts on Xen. Thanks in advance > > for all helps. > > > > 1) From papers about Xen architecture found at Xen homepage, Xen > > seemingly intercepts interrupts of Dom1 (paravirtualization). So > > interrupts occur in Dom1''s context are sent to Xen, and Xen handles > > them. Is that correct? > > similar to interrupts to a native operating systems, interrupts could > occur in just any context (as long as they''re not blocked). > > the domU never ''receives'' the interrupt in a way that it won''t even be > able to install a handler for it. the idt belongs to xen, and xen is > where it is actually handled, interrupting whatever used to be running > when it occurred. > > > But then how can Dom1 gets the interrupts? For example if int3 occurs > > inside Dom1, Xen intercepts int3, but then how debugger (like gdb) can > > still work in Dom1? Maybe Xen sends that interrupt back to Dom1 after > > processing it in VMM layer? > > hardware interrupts in turn drive ''virtual interrupts'', which are in the > paravirtualized case a pure software abstraction. in xen, you''ll find a > lot of references to a concept called ''event channels'', and that''s just > that. > > so it''s xen making then decision on whether a domain interrupt handler > is called, which one''s called, when it''s called, and whether anything > called at all. don''t worry, in practice, xen is doing the right thing, > but that''s the way it needs to work. :). > > > 2) I have read > > linux-xen-sparse/arch/i386/kernel/{traps-xen.c,traps.c,vmlinux.lds.S}, > > and looks like Dom1 still handles interrupts itself? Why, because I > > suppose that the VMM already intercepts those interrupts? > > an guest system, whether it''s virtualized or not, is not living in a > vacuum. it needs some kind of signalling from the hardware environment, > whether it''s virtual or physical, in order to get notified of events. > that''s ''asynchronous input'' of some kind, not necessarily purelay > hardware interrupt driven, though. > > e.g. it needs a regular timer interrupt as well as as notification of > events regarding virtual device interfaces. > so it still has interrupt handlers. > > it is, however, not modifying the real IDT. see e.g. trap_init() in > traps-xen.c and read about HYPERVISOR_set_trap_table in the interface > documentations. >Very interesting. So here is what I imagine about things in Xen: - When Dom1 boots, it asks VMM (via set_trap_table hypercall) to setup the IDT for it. Meanwhile Dom1 still setups its IDT but all the interrupts are from event-channels instead of from physical interrupts like in native case. An event sent to Dom1 via event-channel will fire a virtual interrupt, and Dom1''s interrupt handler will handle it like in native case. - When an interrupt occurs in Dom1''s context, VMM handles it first, then fires an event corresponding to the interrupt via event-channel to Dom1, thus creates an virtual interrupt. Dom1 processes this virtual interrupt. Is that all correct? I follow the above assumption, and as an example, trying to track down how int3 is processed if we use gdb to debug an application in Dom1. But looks like the int3 handler in VMM never send event to Dom1 as I supposed. Or I missed something? Any pointer on which code of VMM does this? Thanks. Jun _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 8/11/06, Ian Campbell <Ian.Campbell@xensource.com> wrote:> On Fri, 2006-08-11 at 12:39 +0200, Daniel Stodden wrote: > > > > > 3) How is linux-xen-sparse/arch/i386/kernel/traps-xen.c compiled, > > > while it is not refereed to in > > > linux-xen-sparse/arch/i386/kernel/Makefile? I see traps.c is > > refereed > > > instead of traps-xen.c, but I still saw traps-xen.o there after > > > compiling? What traps.c does, and what traps-xen.c does, since both > > > are very similar to me. > > > > uhhm. now you got me. i noticed the file names changed at some point, > > i didn''t bother why yet, or how the makefiles changed. maybe someone > > from xensource can tell. > > obj-y := $(call cherrypickxen, $(obj-y)) > > at the end of the Makefile will rewrite foo.o as foo-xen.o if there is a > foo-xen.c or foo-xen.S available. The function is defined in > scripts/Makefile.xenInteresting! Would you tell me why we need that trick? In that case, we can empty file traps-xen.c, no? Because according to your mention, only the filename (and its existence) matters, but not its content. Thanks. Jun _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
[snip]> > obj-y := $(call cherrypickxen, $(obj-y)) > > > > at the end of the Makefile will rewrite foo.o as foo-xen.o > if there is a > > foo-xen.c or foo-xen.S available. The function is defined in > > scripts/Makefile.xen > > Interesting! Would you tell me why we need that trick? > > In that case, we can empty file traps-xen.c, no? Because according to > your mention, only the filename (and its existence) matters, but not > its content.I may be wrong here, but I think what is meant is not that it replaces foo.o with foo-xen.o by renaming foo.o in the filesystem, but rather that the file listed in the variable obj is renamed, and thus it wants to build foo-xen.o instead of foo.o, and use it for linking. To build foo-xen.o, you''d obviously need a foo-xen.[cS] file as a source-file... I gues Christian (or someone else who actually knows how it works) can either confirm this, or tell us how it works. -- Mats> > Thanks. > Jun > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel > > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Fri, 2006-08-11 at 16:20 +0200, Petersson, Mats wrote:> > > obj-y := $(call cherrypickxen, $(obj-y)) > > > > > > at the end of the Makefile will rewrite foo.o as foo-xen.o > > if there is a > > > foo-xen.c or foo-xen.S available. The function is defined in > > > scripts/Makefile.xen > > > > Interesting! Would you tell me why we need that trick? > > > > In that case, we can empty file traps-xen.c, no? Because according > to > > your mention, only the filename (and its existence) matters, but not > > its content. > > I may be wrong here, but I think what is meant is not that it replaces > foo.o with foo-xen.o by renaming foo.o in the filesystem, but rather > that the file listed in the variable obj is renamed, and thus it wants > to build foo-xen.o instead of foo.o, and use it for linking. To build > foo-xen.o, you''d obviously need a foo-xen.[cS] file as a > source-file...That''s exactly right. Ian. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Fri, 2006-08-11 at 23:13 +0900, Jun Koi wrote:> Hi Daniel, > > On 8/11/06, Daniel Stodden <stodden@cs.tum.edu> wrote: > > On Fri, 2006-08-11 at 19:03 +0900, Jun Koi wrote: > > > Good morning, > > > > > > I am new to Xen, and trying to understand how Xen works. I have some > > > questions about virtualization of interrupts on Xen. Thanks in advance > > > for all helps. > > > > > > 1) From papers about Xen architecture found at Xen homepage, Xen > > > seemingly intercepts interrupts of Dom1 (paravirtualization). So > > > interrupts occur in Dom1''s context are sent to Xen, and Xen handles > > > them. Is that correct? > > > > similar to interrupts to a native operating systems, interrupts could > > occur in just any context (as long as they''re not blocked). > > > > the domU never ''receives'' the interrupt in a way that it won''t even be > > able to install a handler for it. the idt belongs to xen, and xen is > > where it is actually handled, interrupting whatever used to be running > > when it occurred. > > > > > But then how can Dom1 gets the interrupts? For example if int3 occurs > > > inside Dom1, Xen intercepts int3, but then how debugger (like gdb) can > > > still work in Dom1? Maybe Xen sends that interrupt back to Dom1 after > > > processing it in VMM layer? > > > > hardware interrupts in turn drive ''virtual interrupts'', which are in the > > paravirtualized case a pure software abstraction. in xen, you''ll find a > > lot of references to a concept called ''event channels'', and that''s just > > that. > > > > so it''s xen making then decision on whether a domain interrupt handler > > is called, which one''s called, when it''s called, and whether anything > > called at all. don''t worry, in practice, xen is doing the right thing, > > but that''s the way it needs to work. :). > > > > > 2) I have read > > > linux-xen-sparse/arch/i386/kernel/{traps-xen.c,traps.c,vmlinux.lds.S}, > > > and looks like Dom1 still handles interrupts itself? Why, because I > > > suppose that the VMM already intercepts those interrupts? > > > > an guest system, whether it''s virtualized or not, is not living in a > > vacuum. it needs some kind of signalling from the hardware environment, > > whether it''s virtual or physical, in order to get notified of events. > > that''s ''asynchronous input'' of some kind, not necessarily purelay > > hardware interrupt driven, though. > > > > e.g. it needs a regular timer interrupt as well as as notification of > > events regarding virtual device interfaces. > > so it still has interrupt handlers. > > > > it is, however, not modifying the real IDT. see e.g. trap_init() in > > traps-xen.c and read about HYPERVISOR_set_trap_table in the interface > > documentations. > > > > Very interesting. So here is what I imagine about things in Xen: > > - When Dom1 boots, it asks VMM (via set_trap_table hypercall) to setup > the IDT for it.should correct myself here, there''s a few more calls to know about. may i point you to the xen interface documentation? the hypercall listings are comparatively complete regarding that matter.> Meanwhile Dom1 still setups its IDTin the paravirtualized case, the domains are well aware that there is no true IDT for them and fully resort to the hypercall interface. which means they''re basically registering a callback. kernel stack setup and context saving would be similar to a real idt. i must admit that i didn''t look to close at the stack setup. so you''ll need to dig through entry.S yourself :)> but all the > interrupts are from event-channels instead of from physical interrupts > like in native case. An event sent to Dom1 via event-channel will fire > a virtual interrupt, and Dom1''s interrupt handler will handle it like > in native case.correct.> - When an interrupt occurs in Dom1''s context, VMM handles it first, > then fires an event corresponding to the interrupt via event-channel > to Dom1, thus creates an virtual interrupt. Dom1 processes this > virtual interrupt. > > Is that all correct? > > > I follow the above assumption, and as an example, trying to track down > how int3 is processed if we use gdb to debug an application in Dom1. > But looks like the int3 handler in VMM never send event to Dom1 as I > supposed. Or I missed something?do_int3(), which i presume at which you''ve been looking, is not signalling the event immediately. what it does is, it sets a flag in the virtual processor control structure for that guest. this again is processed upon return from the exception handler. you''ll need to dig trough entry.S again. watch the path follwoing ''jmp error_code'' in the handler. there look for e.g. create_bounce_frame and labels which call it. this is due to the fact that a hypervisor can never just ''call back'' into the guest in a synchronous fashion (like a function call). apart from a whole bunch of reasons why this would not work, it would recurse with any later interrupt, right? instead, one needs to modify the return path from the hypervisor to make return to the guest stack look like a regular exception/interrupt occurring on the real machine. get the idea? it''s indeed a little difficult, if you really want to understand every bit, you need to know how interrupt and exception handling is implemented on the processor. means: keep your x86 processor manuals near.> Any pointer on which code of VMM does > this?for the case of synchronous interrupts (exceptions, int3, etc.) it''s all about entry.S. for the asynchronous case (hardware interrupts), see the event channel code, which in turn will lead you to the xen scheduler. regards, daniel -- Daniel Stodden LRR - Lehrstuhl für Rechnertechnik und Rechnerorganisation Institut für Informatik der TU München D-85748 Garching http://www.lrr.in.tum.de/~stodden mailto:stodden@cs.tum.edu PGP Fingerprint: F5A4 1575 4C56 E26A 0B33 3D80 457E 82AE B0D8 735B _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel