Hi, My name is Jose and I work at Google on the Ganeti project. I am using Xen 4.0. I would like to ask your advice on what you think is the best way to send data between the guest and the host, for example, the guest OS tells Ganeti that it wants to be reinstalled at reboot time. Even a simple integer suffices for our purposes. We have considered adding a new device, such as, a char/block device. For example, we could add an SSD device that is backed by a file. We have also considered a separate console. What do you think is the preferred approach to this problem? Thanks, Jose -- Jose Antonio Lopes Ganeti Engineering Google Germany GmbH Dienerstr. 12, 80331, München Registergericht und -nummer: Hamburg, HRB 86891 Sitz der Gesellschaft: Hamburg Geschäftsführer: Graham Law, Christine Elizabeth Flores Steuernummer: 48/725/00206 Umsatzsteueridentifikationsnummer: DE813741370
> -----Original Message----- > From: xen-devel-bounces@lists.xen.org [mailto:xen-devel- > bounces@lists.xen.org] On Behalf Of Jose A. Lopes > Sent: 22 October 2013 09:00 > To: xen-devel@lists.xenproject.org > Subject: [Xen-devel] Guest to Host communication > > Hi, > > My name is Jose and I work at Google on the Ganeti project. > I am using Xen 4.0. > > I would like to ask your advice on what you think is the best way to > send data between the guest and the host, for example, the guest OS > tells Ganeti that it wants to be reinstalled at reboot time. Even a > simple integer suffices for our purposes. > > We have considered adding a new device, such as, a char/block device. > For example, we could add an SSD device that is backed by a file. We > have also considered a separate console. > > What do you think is the preferred approach to this problem? >If you''re talking about a small amount of data and only need a low bandwidth channel then xenstore is probably what you want. Paul
Hi, If I understood correctly, Xenstore requires writing a driver and loading it inside the VM. Is this correct? If so, then it seems this approach would require writing different drivers for different OSes, such as, Linux and Windows. Currently, we are just exploring different design options and we would like to aim at a uniform approach across different OSes. Is there an option like that ? Thank you, Jose On Tue, Oct 22, 2013 at 08:48:24AM +0000, Paul Durrant wrote:> > -----Original Message----- > > From: xen-devel-bounces@lists.xen.org [mailto:xen-devel- > > bounces@lists.xen.org] On Behalf Of Jose A. Lopes > > Sent: 22 October 2013 09:00 > > To: xen-devel@lists.xenproject.org > > Subject: [Xen-devel] Guest to Host communication > > > > Hi, > > > > My name is Jose and I work at Google on the Ganeti project. > > I am using Xen 4.0. > > > > I would like to ask your advice on what you think is the best way to > > send data between the guest and the host, for example, the guest OS > > tells Ganeti that it wants to be reinstalled at reboot time. Even a > > simple integer suffices for our purposes. > > > > We have considered adding a new device, such as, a char/block device. > > For example, we could add an SSD device that is backed by a file. We > > have also considered a separate console. > > > > What do you think is the preferred approach to this problem? > > > > If you''re talking about a small amount of data and only need a low bandwidth channel then xenstore is probably what you want. > > Paul-- Jose Antonio Lopes Ganeti Engineering Google Germany GmbH Dienerstr. 12, 80331, München Registergericht und -nummer: Hamburg, HRB 86891 Sitz der Gesellschaft: Hamburg Geschäftsführer: Graham Law, Christine Elizabeth Flores Steuernummer: 48/725/00206 Umsatzsteueridentifikationsnummer: DE813741370
> -----Original Message----- > From: Jose A. Lopes [mailto:jabolopes@google.com] > Sent: 22 October 2013 13:49 > To: Paul Durrant > Cc: xen-devel@lists.xenproject.org > Subject: Re: [Xen-devel] Guest to Host communication > > Hi, > > If I understood correctly, Xenstore requires writing a driver and > loading it inside the VM. Is this correct? >Well, yes you''ll need some code in the guest, but drivers already exist for linux and windows so you could just use them.> If so, then it seems this approach would require writing different > drivers for different OSes, such as, Linux and Windows. Currently, we > are just exploring different design options and we would like to aim > at a uniform approach across different OSes. Is there an option like that ? >What option do you expect that doesn''t involve writing at least some code for each OS you want to use? Paul
On Tue, Oct 22, 2013 at 01:18:56PM +0000, Paul Durrant wrote:> > -----Original Message----- > > From: Jose A. Lopes [mailto:jabolopes@google.com] > > Sent: 22 October 2013 13:49 > > To: Paul Durrant > > Cc: xen-devel@lists.xenproject.org > > Subject: Re: [Xen-devel] Guest to Host communication > > > > Hi, > > > > If I understood correctly, Xenstore requires writing a driver and > > loading it inside the VM. Is this correct? > > > > Well, yes you''ll need some code in the guest, but drivers already exist for linux and windows so you could just use them. > > > If so, then it seems this approach would require writing different > > drivers for different OSes, such as, Linux and Windows. Currently, we > > are just exploring different design options and we would like to aim > > at a uniform approach across different OSes. Is there an option like that ? > > > > What option do you expect that doesn''t involve writing at least some code for each OS you want to use?We were thinking of simply attaching a device. For example, add a SSD disk in DomU which is backed in Dom0 by a file. Is this possible?> > Paul-- Jose Antonio Lopes Ganeti Engineering Google Germany GmbH Dienerstr. 12, 80331, München Registergericht und -nummer: Hamburg, HRB 86891 Sitz der Gesellschaft: Hamburg Geschäftsführer: Graham Law, Christine Elizabeth Flores Steuernummer: 48/725/00206 Umsatzsteueridentifikationsnummer: DE813741370
On Tue, Oct 22, 2013 at 01:18:56PM +0000, Paul Durrant wrote:> > -----Original Message----- > > From: Jose A. Lopes [mailto:jabolopes@google.com] > > Sent: 22 October 2013 13:49 > > To: Paul Durrant > > Cc: xen-devel@lists.xenproject.org > > Subject: Re: [Xen-devel] Guest to Host communication > > > > Hi, > > > > If I understood correctly, Xenstore requires writing a driver and > > loading it inside the VM. Is this correct? > > > > Well, yes you''ll need some code in the guest, but drivers already exist for linux and windows so you could just use them.So, Microsoft came up with this kvp (key value) driver that can communicate with the host. The guest only needs to use a socket interface to talk - and best of all it is in the upstream kernel. The driver (user-space) is this one: http://lxr.free-electrons.com/source/tools/hv/hv_kvp_daemon.c and it listens to the socket. The kernel driver gets the key/values from the host and it ends up pushing them to the daemon. The daemon does whatever it needs to do with them (say, alter the IP of the guest, return the OS info back and so on). It should be possible to make the kernel driver use a XenBus instead of a VMBus and piggyback on this.> > > If so, then it seems this approach would require writing different > > drivers for different OSes, such as, Linux and Windows. Currently, weThe neat thing is that it would also work out of the box on Windows (I hope).> > are just exploring different design options and we would like to aim > > at a uniform approach across different OSes. Is there an option like that ? > > > > What option do you expect that doesn''t involve writing at least some code for each OS you want to use? > > Paul > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel
On Tue, Oct 22, 2013 at 2:42 PM, Jose A. Lopes <jabolopes@google.com> wrote:> On Tue, Oct 22, 2013 at 01:18:56PM +0000, Paul Durrant wrote: >> > -----Original Message----- >> > From: Jose A. Lopes [mailto:jabolopes@google.com] >> > Sent: 22 October 2013 13:49 >> > To: Paul Durrant >> > Cc: xen-devel@lists.xenproject.org >> > Subject: Re: [Xen-devel] Guest to Host communication >> > >> > Hi, >> > >> > If I understood correctly, Xenstore requires writing a driver and >> > loading it inside the VM. Is this correct? >> > >> >> Well, yes you''ll need some code in the guest, but drivers already exist for linux and windows so you could just use them. >> >> > If so, then it seems this approach would require writing different >> > drivers for different OSes, such as, Linux and Windows. Currently, we >> > are just exploring different design options and we would like to aim >> > at a uniform approach across different OSes. Is there an option like that ? >> > >> >> What option do you expect that doesn''t involve writing at least some code for each OS you want to use? > > We were thinking of simply attaching a device. > For example, add a SSD disk in DomU which is backed in Dom0 by a file. > Is this possible?Of course you can attach as many disks as you want; or you could just have dom0 look for a file in the primary filesystem. But whether you use the primary filesystem or a secondary one, you still have the basic problem of introspection: if this is going to be a filesystem, then there may be a difference at any given time between what domU thinks the state of the filesystem is, and what dom0 can see on the disk (because there may be things in domU''s buffer cache which haven''t "hit the disk" yet). I suppose one option would be to make a block device with "magic" -- i.e., "write to block 0 for this bit of info, block 2 for this bit". But the way that you write specific blocks is different for each OS anyway -- you''re essentially writing a driver for this "magic block device" anyway; *and* you have to write new code in dom0 to look at this device and notice changes to it, which will be non-trivial. Fortunately, there already is a well-defined way of doing this which already has drivers and libraries for all major operating systems (Linux, Windows, NetBSD, FreeBSD, probably even Gnu Hurd): xenstore. It already has "watch" functionality, so that a program in dom0 can say, "Tell me when this key {....} is created or changed" and then contentedly go to sleep, secure in the knowledge that the well-tested data path will wake it up when the guest writes to it. It''s almost certainly your best solution. -George
On Thu, Oct 31, 2013 at 02:21:38PM +0000, George Dunlap wrote:> On Tue, Oct 22, 2013 at 2:42 PM, Jose A. Lopes <jabolopes@google.com> wrote: > > On Tue, Oct 22, 2013 at 01:18:56PM +0000, Paul Durrant wrote: > >> > -----Original Message----- > >> > From: Jose A. Lopes [mailto:jabolopes@google.com] > >> > Sent: 22 October 2013 13:49 > >> > To: Paul Durrant > >> > Cc: xen-devel@lists.xenproject.org > >> > Subject: Re: [Xen-devel] Guest to Host communication > >> > > >> > Hi, > >> > > >> > If I understood correctly, Xenstore requires writing a driver and > >> > loading it inside the VM. Is this correct? > >> > > >> > >> Well, yes you''ll need some code in the guest, but drivers already exist for linux and windows so you could just use them. > >> > >> > If so, then it seems this approach would require writing different > >> > drivers for different OSes, such as, Linux and Windows. Currently, we > >> > are just exploring different design options and we would like to aim > >> > at a uniform approach across different OSes. Is there an option like that ? > >> > > >> > >> What option do you expect that doesn''t involve writing at least some code for each OS you want to use? > > > > We were thinking of simply attaching a device. > > For example, add a SSD disk in DomU which is backed in Dom0 by a file. > > Is this possible? > > Of course you can attach as many disks as you want; or you could just > have dom0 look for a file in the primary filesystem. But whether you > use the primary filesystem or a secondary one, you still have the > basic problem of introspection: if this is going to be a filesystem, > then there may be a difference at any given time between what domU > thinks the state of the filesystem is, and what dom0 can see on the > disk (because there may be things in domU''s buffer cache which haven''t > "hit the disk" yet).We have changed our use case to the situation where the instance writes a few bytes to this device. However, Ganeti only needs this information once the instance has stopped. Therefore, while the instance is running it can write and overwrite the information because it is not important. I wonder if it is possible to change the caching policy of the device in order to immediately flush the writes. This is not so important because I imagine that by the time the instance has stopped (if it didn''t crash), the cache should have been flushed. We were however thinking about a usb serial device to make it easier to do successive writes within the instance. Jose> > I suppose one option would be to make a block device with "magic" -- > i.e., "write to block 0 for this bit of info, block 2 for this bit". > But the way that you write specific blocks is different for each OS > anyway -- you''re essentially writing a driver for this "magic block > device" anyway; *and* you have to write new code in dom0 to look at > this device and notice changes to it, which will be non-trivial. > > Fortunately, there already is a well-defined way of doing this which > already has drivers and libraries for all major operating systems > (Linux, Windows, NetBSD, FreeBSD, probably even Gnu Hurd): xenstore. > It already has "watch" functionality, so that a program in dom0 can > say, "Tell me when this key {....} is created or changed" and then > contentedly go to sleep, secure in the knowledge that the well-tested > data path will wake it up when the guest writes to it. It''s almost > certainly your best solution. > > -George-- Jose Antonio Lopes Ganeti Engineering Google Germany GmbH Dienerstr. 12, 80331, München Registergericht und -nummer: Hamburg, HRB 86891 Sitz der Gesellschaft: Hamburg Geschäftsführer: Graham Law, Christine Elizabeth Flores Steuernummer: 48/725/00206 Umsatzsteueridentifikationsnummer: DE813741370
On 31/10/13 16:08, Jose A. Lopes wrote:> On Thu, Oct 31, 2013 at 02:21:38PM +0000, George Dunlap wrote: >> On Tue, Oct 22, 2013 at 2:42 PM, Jose A. Lopes <jabolopes@google.com> wrote: >>> On Tue, Oct 22, 2013 at 01:18:56PM +0000, Paul Durrant wrote: >>>>> -----Original Message----- >>>>> From: Jose A. Lopes [mailto:jabolopes@google.com] >>>>> Sent: 22 October 2013 13:49 >>>>> To: Paul Durrant >>>>> Cc: xen-devel@lists.xenproject.org >>>>> Subject: Re: [Xen-devel] Guest to Host communication >>>>> >>>>> Hi, >>>>> >>>>> If I understood correctly, Xenstore requires writing a driver and >>>>> loading it inside the VM. Is this correct? >>>>> >>>> Well, yes you''ll need some code in the guest, but drivers already exist for linux and windows so you could just use them. >>>> >>>>> If so, then it seems this approach would require writing different >>>>> drivers for different OSes, such as, Linux and Windows. Currently, we >>>>> are just exploring different design options and we would like to aim >>>>> at a uniform approach across different OSes. Is there an option like that ? >>>>> >>>> What option do you expect that doesn''t involve writing at least some code for each OS you want to use? >>> We were thinking of simply attaching a device. >>> For example, add a SSD disk in DomU which is backed in Dom0 by a file. >>> Is this possible? >> Of course you can attach as many disks as you want; or you could just >> have dom0 look for a file in the primary filesystem. But whether you >> use the primary filesystem or a secondary one, you still have the >> basic problem of introspection: if this is going to be a filesystem, >> then there may be a difference at any given time between what domU >> thinks the state of the filesystem is, and what dom0 can see on the >> disk (because there may be things in domU''s buffer cache which haven''t >> "hit the disk" yet). > We have changed our use case to the situation where the instance > writes a few bytes to this device. However, Ganeti only needs this > information once the instance has stopped. Therefore, while the > instance is running it can write and overwrite the information because > it is not important. > > I wonder if it is possible to change the caching policy of the device > in order to immediately flush the writes. This is not so important > because I imagine that by the time the instance has stopped (if it > didn''t crash), the cache should have been flushed. > > We were however thinking about a usb serial device to make it easier > to do successive writes within the instance.Oh, right -- in that case, probably the easiest thing to do would be to just create a file in the guest filesystem and have your tools look inside to see if it''s there. You could have a secondary disk just for that purpose, if that makes it easier to make it the same across instances. -George
On Thu, Oct 31, 2013 at 5:13 PM, George Dunlap <george.dunlap@eu.citrix.com>wrote:> On 31/10/13 16:08, Jose A. Lopes wrote: > >> On Thu, Oct 31, 2013 at 02:21:38PM +0000, George Dunlap wrote: >> >>> On Tue, Oct 22, 2013 at 2:42 PM, Jose A. Lopes <jabolopes@google.com> >>> wrote: >>> >>>> On Tue, Oct 22, 2013 at 01:18:56PM +0000, Paul Durrant wrote: >>>> >>>>> -----Original Message----- >>>>>> From: Jose A. Lopes [mailto:jabolopes@google.com] >>>>>> Sent: 22 October 2013 13:49 >>>>>> To: Paul Durrant >>>>>> Cc: xen-devel@lists.xenproject.org >>>>>> Subject: Re: [Xen-devel] Guest to Host communication >>>>>> >>>>>> Hi, >>>>>> >>>>>> If I understood correctly, Xenstore requires writing a driver and >>>>>> loading it inside the VM. Is this correct? >>>>>> >>>>>> Well, yes you''ll need some code in the guest, but drivers already >>>>> exist for linux and windows so you could just use them. >>>>> >>>>> If so, then it seems this approach would require writing different >>>>>> drivers for different OSes, such as, Linux and Windows. Currently, we >>>>>> are just exploring different design options and we would like to aim >>>>>> at a uniform approach across different OSes. Is there an option like >>>>>> that ? >>>>>> >>>>>> What option do you expect that doesn''t involve writing at least some >>>>> code for each OS you want to use? >>>>> >>>> We were thinking of simply attaching a device. >>>> For example, add a SSD disk in DomU which is backed in Dom0 by a file. >>>> Is this possible? >>>> >>> Of course you can attach as many disks as you want; or you could just >>> have dom0 look for a file in the primary filesystem. But whether you >>> use the primary filesystem or a secondary one, you still have the >>> basic problem of introspection: if this is going to be a filesystem, >>> then there may be a difference at any given time between what domU >>> thinks the state of the filesystem is, and what dom0 can see on the >>> disk (because there may be things in domU''s buffer cache which haven''t >>> "hit the disk" yet). >>> >> We have changed our use case to the situation where the instance >> writes a few bytes to this device. However, Ganeti only needs this >> information once the instance has stopped. Therefore, while the >> instance is running it can write and overwrite the information because >> it is not important. >> >> I wonder if it is possible to change the caching policy of the device >> in order to immediately flush the writes. This is not so important >> because I imagine that by the time the instance has stopped (if it >> didn''t crash), the cache should have been flushed. >> >> We were however thinking about a usb serial device to make it easier >> to do successive writes within the instance. >> > > Oh, right -- in that case, probably the easiest thing to do would be to > just create a file in the guest filesystem and have your tools look inside > to see if it''s there. You could have a secondary disk just for that > purpose, if that makes it easier to make it the same across instances. > >When you say "look inside" the filesystem do you mean to mount that filesystem in the host OS? If so, it seems that it is very dangerous to mount guest filesystems due to a number of exploits.> -George >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
On Thu, 2013-10-31 at 19:42 +0100, Jose A. Lopes wrote:> When you say "look inside" the filesystem do you mean to mount that > filesystem > in the host OS? If so, it seems that it is very dangerous to mount > guest filesystems > due to a number of exploits.If you only need r/o access from the host then you could use tools/libfsimage (which has python binding, used by pygrub) which at least constrains things to a userspace process and not a kernel mode exploit. With suitable privilege dropping this can be made reasonably safe... Ian.
Hi, Thanks, we''ll have a look at that. Regards, Jose On Thu, Oct 31, 2013 at 08:02:29PM +0000, Ian Campbell wrote:> On Thu, 2013-10-31 at 19:42 +0100, Jose A. Lopes wrote: > > > When you say "look inside" the filesystem do you mean to mount that > > filesystem > > in the host OS? If so, it seems that it is very dangerous to mount > > guest filesystems > > due to a number of exploits. > > If you only need r/o access from the host then you could use > tools/libfsimage (which has python binding, used by pygrub) which at > least constrains things to a userspace process and not a kernel mode > exploit. With suitable privilege dropping this can be made reasonably > safe... > > Ian. > >-- Jose Antonio Lopes Ganeti Engineering Google Germany GmbH Dienerstr. 12, 80331, München Registergericht und -nummer: Hamburg, HRB 86891 Sitz der Gesellschaft: Hamburg Geschäftsführer: Graham Law, Christine Elizabeth Flores Steuernummer: 48/725/00206 Umsatzsteueridentifikationsnummer: DE813741370
Possibly Parallel Threads
- Change on_poweroff while domain is running
- [PATCH 0 of 2] Support for VM generation ID save/restore and migrate
- [PATCH 0 of 3] Support for VM generation ID save/restore and migrate
- [PATCH 0 of 6] Add support for a VM generation ID virtual device (v2)
- [PATCH 0 of 4] Support for VM generation ID save/restore and migrate