Hi, When combining clustering with cluster storage, the situation arises where virtual machines are not only managed by the cluster software, but also need to participate in the cluster software themselves... In order for this to work better, it would be good if there was a way for software in a domU to communicate with software on the local dom0 - so after being migrated elsewhere, a guest would start talking to the new dom0 automatically. Now what I''m not sure about is, what would be the best way to do this? Should it be some virtual device analogous to the virtual network driver, or should we have some other kind of socket for node local communications ? What kind of approach would the Xen community (that''s you, if you read this far) prefer ? -- The Theory of Escalating Commitment: "The cost of continuing mistakes is borne by others, while the cost of admitting mistakes is borne by yourself." -- Joseph Stiglitz, Nobel Laureate in Economics _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Rik van Riel wrote:>Hi, > >When combining clustering with cluster storage, the situation >arises where virtual machines are not only managed by the cluster >software, but also need to participate in the cluster software >themselves... > >In order for this to work better, it would be good if there >was a way for software in a domU to communicate with software >on the local dom0 - so after being migrated elsewhere, a guest >would start talking to the new dom0 automatically. > >Now what I''m not sure about is, what would be the best way to >do this? > >Should it be some virtual device analogous to the virtual >network driver, or should we have some other kind of socket >for node local communications ? > >What kind of approach would the Xen community (that''s you, if >you read this far) prefer ? > >I can only speak for myself, but I''d personally like to see a small userspace library that established a datagram connection between two domains (using an event channel and a shared page) that could be reused for many applications. If it could be used with normal read/write calls that would be even better (although first thought is that that would require a daemon). Regards, Anthony Liguori _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Anthony Liguori wrote:> Rik van Riel wrote: > >> Hi, >> >> When combining clustering with cluster storage, the situation >> arises where virtual machines are not only managed by the cluster >> software, but also need to participate in the cluster software >> themselves... >> >> In order for this to work better, it would be good if there >> was a way for software in a domU to communicate with software >> on the local dom0 - so after being migrated elsewhere, a guest >> would start talking to the new dom0 automatically.Rik, is this a heartbeat kind of protocol or some such thing? Is there value to making it be a generic solution that would work when communicating in a non-xen environment?>> Should it be some virtual device analogous to the virtual >> network driver, or should we have some other kind of socket >> for node local communications ?Using protocols like TCP for node-local comms is undesirable as TCP is engineered for the heavy machinery of the Internet. UDP or raw sockets will be somewhat better, although AF_UNIX would be best of all (albeit with its own constraints).>> What kind of approach would the Xen community (that''s you, if >> you read this far) prefer ? >> >> > I can only speak for myself, but I''d personally like to see a small > userspace library that established a datagram connection between two > domains (using an event channel and a shared page) that could be reused > for many applications.Which datagram service, Anthony? Having a small lib like this would be a good idea, although the benefit would be in the details, and of course, depending on how many apps we need to support that we don''t want to modify...> If it could be used with normal read/write calls that would be even > better (although first thought is that that would require a daemon).Probably don''t want to do that... thanks, Nivedita _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> In order for this to work better, it would be good if there > was a way for software in a domU to communicate with software > on the local dom0 - so after being migrated elsewhere, a > guest would start talking to the new dom0 automatically. > > Now what I''m not sure about is, what would be the best way to do this? > > Should it be some virtual device analogous to the virtual > network driver, or should we have some other kind of socket > for node local communications ? > > What kind of approach would the Xen community (that''s you, if > you read this far) prefer ?For relatively static stuff (e.g. textual configuration data), it would be reasonable to use xenbus/xenstore. We could give each domain a subtree in the store that it can read/write via a /proc/sys interface. For other low bandwidth stuff we could have a simple protocol for multiplexing datagrams over the virtual console channel. For high bandwidth stuff we''d want to set up separate shared memory rings. Rik: Does this mean you could be pursuaded to look at splitting the console handling out of xend? :-) Ian _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Nivedita Singhvi wrote:> UDP or raw sockets will be somewhat better, although AF_UNIX > would be best of all (albeit with its own constraints).I was thinking of a raw interface although a UDP-like protocol would be kind of neat (so that you could do multiplexing).> Which datagram service, Anthony? Having a small lib like this > would be a good idea, although the benefit would be in the details, > and of course, depending on how many apps we need to support that > we don''t want to modify...I must admit I''ve not put much thought into it. What are your thoughts? Regards, Anthony Liguori _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Anthony Liguori wrote:> Nivedita Singhvi wrote: > >> UDP or raw sockets will be somewhat better, although AF_UNIX >> would be best of all (albeit with its own constraints). > > > I was thinking of a raw interface although a UDP-like protocol would be > kind of neat (so that you could do multiplexing).It would be less work and easier on the apps to use UDP than raw, but that''s just my experience (and, potentially more bugs). This could be solved by having a user space library wrap the details and call a simple low level protocol over raw sockets.>> Which datagram service, Anthony? Having a small lib like this >> would be a good idea, although the benefit would be in the details, >> and of course, depending on how many apps we need to support that >> we don''t want to modify... > > > I must admit I''ve not put much thought into it. What are your thoughts?Me neither, much, yet, :). But, it would be nice to make things easy for the large number of apps which need intra-node communication. At the moment, they fall into the two categories of strictly intra-node versus common-over-intra_off_node. Much easier designing things for the former. thanks, Nivedita _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ian Pratt wrote:> For relatively static stuff (e.g. textual configuration data), it would > be reasonable to use xenbus/xenstore. We could give each domain a > subtree in the store that it can read/write via a /proc/sys interface.Ian, how flexible is the format/layout of the xenstore going to be? Does it allow for more of a general purpose use? That would be neat.. thanks, Nivedita _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> -----Original Message----- > From: Nivedita Singhvi [mailto:niv@us.ibm.com] > Sent: 15 June 2005 23:11 > To: Ian Pratt > Cc: Rik van Riel; xen-devel@lists.xensource.com; Lon Hohberger > Subject: Re: [Xen-devel] RFC: "local" communications > > Ian Pratt wrote: > > > For relatively static stuff (e.g. textual configuration data), it > > would be reasonable to use xenbus/xenstore. We could give > each domain > > a subtree in the store that it can read/write via a > /proc/sys interface. > > Ian, how flexible is the format/layout of the xenstore going > to be? Does it allow for more of a general purpose use? That > would be neat..It enables something akin to /proc. i.e. you can have an arbitrary hierarchy of files. To retain the same spirit as everything else in xenstore it would be preferable that both file names and contents were text (rather than binary). I think I quite like the idea of having a /proc/sys interface onto xenstore. We''ll still need the console multiplexing approach for stream/datagram data. Ian _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Wed, 15 Jun 2005, Ian Pratt wrote:> > What kind of approach would the Xen community (that''s you, if > > you read this far) prefer ? > > For relatively static stuff (e.g. textual configuration data), it would > be reasonable to use xenbus/xenstore. We could give each domain a > subtree in the store that it can read/write via a /proc/sys interface.I guess xenbus should work fine... The main uses are for doing things like querying "which physical system am I running on", fencing off a virtual cluster member, etc. -- The Theory of Escalating Commitment: "The cost of continuing mistakes is borne by others, while the cost of admitting mistakes is borne by yourself." -- Joseph Stiglitz, Nobel Laureate in Economics _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Wed, 2005-06-15 at 15:24 -0500, Anthony Liguori wrote:> Rik van Riel wrote: > >In order for this to work better, it would be good if there > >was a way for software in a domU to communicate with software > >on the local dom0 - so after being migrated elsewhere, a guest > >would start talking to the new dom0 automatically. > > I can only speak for myself, but I''d personally like to see a small > userspace library that established a datagram connection between two > domains (using an event channel and a shared page) that could be reused > for many applications.We have such a thing in the xenstore code. It could be extracted fairly easily. Naturally, you need a setup mechanism, but the store could do that. Cheers, Rusty. -- A bad analogy is like a leaky screwdriver -- Richard Braakman _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Wed, 2005-06-15 at 23:32 +0100, Ian Pratt wrote:> I think I quite like the idea of having a /proc/sys interface onto > xenstore.It''s one of those tempting things that doesn''t *quite* work. Creation of new directories and nodes is kinda important if userspace is going to be doing anything non-trivial. 8( Rusty. -- A bad analogy is like a leaky screwdriver -- Richard Braakman _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Wed, 2005-06-15 at 14:31 -0700, Nivedita Singhvi wrote:> >> Should it be some virtual device analogous to the virtual > >> network driver, or should we have some other kind of socket > >> for node local communications ? > > Using protocols like TCP for node-local comms is undesirable > as TCP is engineered for the heavy machinery of the Internet. > UDP or raw sockets will be somewhat better, although AF_UNIX > would be best of all (albeit with its own constraints).For domU->dom0 within a node, I agree. For domU->domU communications, I kind of disagree: If we migrate a domU to another node, we shouldn''t have to change how we''re talking to that domain. Then again, in this case, we might not care for the type of information we''re sending/receiving. Also, I think there''s only a very limited set of information which should be available over whatever communications model is employed (at least, initially). If a domU is compromised, we certainly don''t want the intruder to be able to find out anything significant about other domains. (Sorry for sounding like Captain Obvious, I''m sure this has been considered) -- Lon _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Wed, 2005-06-15 at 18:43 -0400, Rik van Riel wrote:> > For relatively static stuff (e.g. textual configuration data), it would > > be reasonable to use xenbus/xenstore. We could give each domain a > > subtree in the store that it can read/write via a /proc/sys interface. > > I guess xenbus should work fine... > > The main uses are for doing things like querying "which physical > system am I running on", fencing off a virtual cluster member, etc.Agreed, that''s the primary thing we need right now. There are other things, but one thing at a time. Xenbus sounds like a good start, but we''ll need something for more dynamic information too. -- Lon _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel