Xen devel - Apr 2007 - Communication between HVM and dom0 through the Hypervisor

Is there currently a method for passing information
between a HVM domain and domain 0 directly through the
hypervisor?
I understand there could be at least three ways to do
so:

-VMCALL from HVM + params in registers.
-using "PV-on-HVM", but I think the frontend drivers
need to be ported in the mainline Linux kernel and
this is currently being done, right?
-using XenStore/XenBus (is that possible?)

As an example, suppose I have a HVM running Linux (on
Intel IVT-x), and I want to call a callback function
in dom0 each time the HVM contacts the hypervisor,
what would you suggest me to do? Imagine I want to
notify dom0 each time a process issues a system call,
by adding some code in the prologue of the syscall
handler to notify the hypervisor.

Thank you very much :-)


      ___________________________________ 
L''email della prossima generazione? Puoi averla con la nuova Yahoo!
Mail: http://it.docs.yahoo.com/nowyoucan.html

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> -----Original Message-----
> From: xen-devel-bounces@lists.xensource.com 
> [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of 
> Daniele Sgandurra
> Sent: 18 April 2007 16:22
> To: xen-devel@lists.xensource.com
> Subject: [Xen-devel] Communication between HVM and dom0 
> through the Hypervisor
> 
> Is there currently a method for passing information
> between a HVM domain and domain 0 directly through the
> hypervisor?
> I understand there could be at least three ways to do
> so:
> 
> -VMCALL from HVM + params in registers.
> -using "PV-on-HVM", but I think the frontend drivers
> need to be ported in the mainline Linux kernel and
> this is currently being done, right?
What do you mean. There is the "unmodified_drivers" directory in Xen,
which contains, as far as I know, complete drivers for block and network
on HVM Linux. Of course, you''d need a differnet source-code if you want
Linux 2.4 drivers or Windows drivers, neither of which is supplied with
Xen''s sources. 
> -using XenStore/XenBus (is that possible?)
I believe so, but I''m not entirely sure about how that works in a HVM
guest.
> 
> As an example, suppose I have a HVM running Linux (on
> Intel IVT-x), and I want to call a callback function
> in dom0 each time the HVM contacts the hypervisor,
> what would you suggest me to do? Imagine I want to
> notify dom0 each time a process issues a system call,
> by adding some code in the prologue of the syscall
> handler to notify the hypervisor.
You can use VM[M]CALL, although this is ALREADY being used for PV-on-HVM
drivers, so you''d have to filter your calls from the ones used by
PV-on-HVM drivers (using a high bit to indicate that those are your
functions rather than PV-calls would be the easiest method). 

Another possibility is to find a unused IO-port or memory address and
cause a IOIO-exit or Page-fault on a MMIO address that isn''t otherwise
used. IO-ports such as 0xBCDE, 0xAAA0 are most likely not used, so you
could to a 8, 16 or 32-bit IO to one of those pretty easily. 

The NEXT problem is of course getting this information out of the
hypervisor into Dom0. 

--
Mats
> 
> Thank you very much :-)
> 
> 
>       ___________________________________ 
> L''email della prossima generazione? Puoi averla con la nuova 
> Yahoo! Mail: http://it.docs.yahoo.com/nowyoucan.html
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
> 
> 
> 


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> > Is there currently a method for passing information
> > between a HVM domain and domain 0 directly through the
> > hypervisor?
> > I understand there could be at least three ways to do
> > so:
> >
> > -VMCALL from HVM + params in registers.
> > -using "PV-on-HVM", but I think the frontend drivers
> > need to be ported in the mainline Linux kernel and
> > this is currently being done, right?
>
> What do you mean. There is the "unmodified_drivers" directory in
Xen,
> which contains, as far as I know, complete drivers for block and network
> on HVM Linux. Of course, you''d need a differnet source-code if you
want
> Linux 2.4 drivers or Windows drivers, neither of which is supplied with
> Xen''s sources.
I think Daniele has read confusing accounts of the PV-on-HVM drivers vs the 
mainline kernel merge.  These are (semi) independent issues: for Linux 2.6 
PV-on-HVM drivers can simply be built by the administrator and installed into 
their VM if they want them.

The code that''s being proposed for the Linux merge is to support full
PV, not
PV-on-HVM (as far as I know).
> > -using XenStore/XenBus (is that possible?)
>
> I believe so, but I''m not entirely sure about how that works in a
HVM
> guest.
Yes, but it requires the PV-on-HVM driver infrastructure.  I''m not
clear on
the exact implementation details, but it involves the guest talking to a 
fake "Xen platform" PCI device, which enables it to then set up a
Xenstore
connection and share configuration data on it.
> > As an example, suppose I have a HVM running Linux (on
> > Intel IVT-x), and I want to call a callback function
> > in dom0 each time the HVM contacts the hypervisor,
> > what would you suggest me to do? Imagine I want to
> > notify dom0 each time a process issues a system call,
> > by adding some code in the prologue of the syscall
> > handler to notify the hypervisor.
>
> You can use VM[M]CALL, although this is ALREADY being used for PV-on-HVM
> drivers, so you''d have to filter your calls from the ones used by
> PV-on-HVM drivers (using a high bit to indicate that those are your
> functions rather than PV-calls would be the easiest method).
>
> Another possibility is to find a unused IO-port or memory address and
> cause a IOIO-exit or Page-fault on a MMIO address that isn''t
otherwise
> used. IO-ports such as 0xBCDE, 0xAAA0 are most likely not used, so you
> could to a 8, 16 or 32-bit IO to one of those pretty easily.
>
> The NEXT problem is of course getting this information out of the
> hypervisor into Dom0.
It it possible to arrange for a VMEXIT to occur on int 0x80 (which IIRC is the 
Linux system call...  I guess VT-x would also be able to intercept sysenter)?  
That way the guest wouldn''t have to be modified to notify Xen.  Xen
would
then have to e.g. share a ring buffer with dom0 regarding events of this 
nature for each domain.  You could hack something up so that the guest 
blocked on each syscall until dom0 acknowledged it...

There''s also the Xen gdb server - maybe (if it supports HVM guests?)
you could
use this to watch the syscall paths.

HTH,
cheers,
Mark

-- 
Dave: Just a question. What use is a unicyle with no seat?  And no pedals!
Mark: To answer a question with a question: What use is a skateboard?
Dave: Skateboards have wheels.
Mark: My wheel has a wheel!

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> 
> There''s also the Xen gdb server - maybe (if it
> supports HVM guests?) you could 
> use this to watch the syscall paths.
> HTH,
> cheers,
> Mark
>
Yes, gdb-server works on HVM too (I was able to make
it work with a HVM domain running Fedora and
installing the kernel-debuginfo on dom0). There is
also a useful paper on kernel debugging with gbd on
Xen:
https://ols2006.108.redhat.com/reprints/kamble-reprint.pdf

Thank you very much for your hints :-)





      ___________________________________ 
L''email della prossima generazione? Puoi averla con la nuova Yahoo!
Mail: http://it.docs.yahoo.com/nowyoucan.html

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> -----Original Message-----
> From: M.A. Williamson [mailto:maw48@hermes.cam.ac.uk] On 
> Behalf Of Mark Williamson
> Sent: 19 April 2007 03:22
> To: xen-devel@lists.xensource.com
> Cc: Petersson, Mats; Daniele Sgandurra
> Subject: Re: [Xen-devel] Communication between HVM and dom0 
> through the Hypervisor
> 
> > > Is there currently a method for passing information
> > > between a HVM domain and domain 0 directly through the
> > > hypervisor?
> > > I understand there could be at least three ways to do
> > > so:
> > >
> > > -VMCALL from HVM + params in registers.
> > > -using "PV-on-HVM", but I think the frontend drivers
> > > need to be ported in the mainline Linux kernel and
> > > this is currently being done, right?
> >
> > What do you mean. There is the "unmodified_drivers" 
> directory in Xen,
> > which contains, as far as I know, complete drivers for 
> block and network
> > on HVM Linux. Of course, you''d need a differnet source-code 
> if you want
> > Linux 2.4 drivers or Windows drivers, neither of which is 
> supplied with
> > Xen''s sources.
> 
> I think Daniele has read confusing accounts of the PV-on-HVM 
> drivers vs the 
> mainline kernel merge.  These are (semi) independent issues: 
> for Linux 2.6 
> PV-on-HVM drivers can simply be built by the administrator 
> and installed into 
> their VM if they want them.
> 
> The code that''s being proposed for the Linux merge is to 
> support full PV, not 
> PV-on-HVM (as far as I know).
> 
> > > -using XenStore/XenBus (is that possible?)
> >
> > I believe so, but I''m not entirely sure about how that 
> works in a HVM
> > guest.
> 
> Yes, but it requires the PV-on-HVM driver infrastructure.  
> I''m not clear on 
> the exact implementation details, but it involves the guest 
> talking to a 
> fake "Xen platform" PCI device, which enables it to then set 
> up a Xenstore 
> connection and share configuration data on it.
Thanks for filling that in. 
> 
> > > As an example, suppose I have a HVM running Linux (on
> > > Intel IVT-x), and I want to call a callback function
> > > in dom0 each time the HVM contacts the hypervisor,
> > > what would you suggest me to do? Imagine I want to
> > > notify dom0 each time a process issues a system call,
> > > by adding some code in the prologue of the syscall
> > > handler to notify the hypervisor.
> >
> > You can use VM[M]CALL, although this is ALREADY being used 
> for PV-on-HVM
> > drivers, so you''d have to filter your calls from the ones
used by
> > PV-on-HVM drivers (using a high bit to indicate that those are your
> > functions rather than PV-calls would be the easiest method).
> >
> > Another possibility is to find a unused IO-port or memory 
> address and
> > cause a IOIO-exit or Page-fault on a MMIO address that 
> isn''t otherwise
> > used. IO-ports such as 0xBCDE, 0xAAA0 are most likely not 
> used, so you
> > could to a 8, 16 or 32-bit IO to one of those pretty easily.
One thing I forgot to add: At the point of the intercept, you will have
access to all CPU registers, so it''s not really important which method
you use for the intercept from that perspective.
> >
> > The NEXT problem is of course getting this information out of the
> > hypervisor into Dom0.
> 
> It it possible to arrange for a VMEXIT to occur on int 0x80 
> (which IIRC is the 
> Linux system call...  I guess VT-x would also be able to 
> intercept sysenter)?  
> That way the guest wouldn''t have to be modified to notify 
> Xen.  Xen would 
> then have to e.g. share a ring buffer with dom0 regarding 
> events of this 
> nature for each domain.  You could hack something up so that 
> the guest 
> blocked on each syscall until dom0 acknowledged it...
Now,that''s a brilliant idea. Why didn''t I think of that. 

Only one minor problem: It will only work for INT n instructions (but
since it''s possible to hide SYSENTER/SYSCALL instrutions from the guest
via CPUID intercept, it can at least for 32-bit be forced to only do INT
80h calls), and only on AMD processors (this is more of a problem, as
the original post talks about Intel processors). Intel processors don''t
allow intercepts of INT n instructions. Neither processor allow
intercepts of SYSENTER/SYSCALL instructions. [It''s still a very good
idea, and I wish I had at least THOUGHT of it!]

--
Mats
> 
> There''s also the Xen gdb server - maybe (if it supports HVM 
> guests?) you could 
> use this to watch the syscall paths.
> 
> HTH,
> cheers,
> Mark
> 
> -- 
> Dave: Just a question. What use is a unicyle with no seat?  
> And no pedals!
> Mark: To answer a question with a question: What use is a skateboard?
> Dave: Skateboards have wheels.
> Mark: My wheel has a wheel!
> 
> 
> 


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> > It it possible to arrange for a VMEXIT to occur on int 0x80
> > (which IIRC is the
> > Linux system call...  I guess VT-x would also be able to
> > intercept sysenter)?
> > That way the guest wouldn''t have to be modified to notify
> > Xen.  Xen would
> > then have to e.g. share a ring buffer with dom0 regarding
> > events of this
> > nature for each domain.  You could hack something up so that
> > the guest
> > blocked on each syscall until dom0 acknowledged it...
>
> Now,that''s a brilliant idea. Why didn''t I think of that.
>
> Only one minor problem: It will only work for INT n instructions (but
> since it''s possible to hide SYSENTER/SYSCALL instrutions from the
guest
> via CPUID intercept, it can at least for 32-bit be forced to only do INT
> 80h calls), and only on AMD processors (this is more of a problem, as
> the original post talks about Intel processors). Intel processors
don''t
> allow intercepts of INT n instructions. Neither processor allow
> intercepts of SYSENTER/SYSCALL instructions. [It''s still a very
good
> idea, and I wish I had at least THOUGHT of it!]
Perhaps on intel, something could be bodged together?  E.g. intercept loading 
the IDT and replace the handler for the int n you''re interested in with
something that''ll cause a trap?  I''m not really clear out
sysenter etc
actually works, so not so sure about that.

I did read through the VT-x / VT-i manuals at some stage, but it''s a
long time
ago and things are a bit hazy now :-)

Cheers,
Mark

-- 
Dave: Just a question. What use is a unicyle with no seat?  And no pedals!
Mark: To answer a question with a question: What use is a skateboard?
Dave: Skateboards have wheels.
Mark: My wheel has a wheel!

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> -----Original Message-----
> From: M.A. Williamson [mailto:maw48@hermes.cam.ac.uk] On 
> Behalf Of Mark Williamson
> Sent: 19 April 2007 15:07
> To: Petersson, Mats
> Cc: xen-devel@lists.xensource.com; Daniele Sgandurra
> Subject: Re: [Xen-devel] Communication between HVM and dom0 
> through the Hypervisor
> 
> > > It it possible to arrange for a VMEXIT to occur on int 0x80
> > > (which IIRC is the
> > > Linux system call...  I guess VT-x would also be able to
> > > intercept sysenter)?
> > > That way the guest wouldn''t have to be modified to
notify
> > > Xen.  Xen would
> > > then have to e.g. share a ring buffer with dom0 regarding
> > > events of this
> > > nature for each domain.  You could hack something up so that
> > > the guest
> > > blocked on each syscall until dom0 acknowledged it...
> >
> > Now,that''s a brilliant idea. Why didn''t I think of
that.
> >
> > Only one minor problem: It will only work for INT n 
> instructions (but
> > since it''s possible to hide SYSENTER/SYSCALL instrutions 
> from the guest
> > via CPUID intercept, it can at least for 32-bit be forced 
> to only do INT
> > 80h calls), and only on AMD processors (this is more of a 
> problem, as
> > the original post talks about Intel processors). Intel 
> processors don''t
> > allow intercepts of INT n instructions. Neither processor allow
> > intercepts of SYSENTER/SYSCALL instructions. [It''s still a
very good
> > idea, and I wish I had at least THOUGHT of it!]
> 
> Perhaps on intel, something could be bodged together?  E.g. 
> intercept loading 
> the IDT and replace the handler for the int n you''re 
> interested in with 
> something that''ll cause a trap?  I''m not really clear out
> sysenter etc 
> actually works, so not so sure about that.
Possibly. The obvious solution of just stuffing an invalid value in CS
would perhaps work, and then fix up CS and "continue". Of course, the
difficulty may well be that IDT isn''t actually hard-coded, so the IDT
entry for INT 80h isn''t in the IDT when it''s written to - of
course, we
could change the IDT to an "invalid" address so that it generates a
page-fault, identify that it''s the IDT and then set the entries
manually
in the page-fault handler. Ideally, for this, you''d want to set the IDT
read-only, and then trap only on writes - that way there''s no penalty
on
interrupt traffic itself (which only reads the IDT). 

SYSENTER/SYSCALL uses MSR''s to determine the RIP, RSP, CS etc. Not sure
if it''s possible to set those up with a bad register (CS) or some such
-
possibly it will work - it''s possible to intercept the MSR writes to
these registers (in fact Intel does for some of them already, if not
all). 

An alternative to an invalid CS is an invalid RIP that causes a
page-fault, and replace the RIP with the correct value (as per the
MSR/IDT write). If the RIP is wrong, I expect the trap to come after the
IDT has been read and the old values stored on the stack - not quite
sure if that''s the case for invalid CS for example, so using an invalid
RIP value may be better]. 
> 
> I did read through the VT-x / VT-i manuals at some stage, but 
> it''s a long time 
> ago and things are a bit hazy now :-)
I had to look it up, so I don''t expect someone who isn''t
actively
working on HVM-stuff to know these type of things. 

--
Mats
> 
> Cheers,
> Mark
> 
> -- 
> Dave: Just a question. What use is a unicyle with no seat?  
> And no pedals!
> Mark: To answer a question with a question: What use is a skateboard?
> Dave: Skateboards have wheels.
> Mark: My wheel has a wheel!
> 
> 
> 


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel