Xen devel - Jan 2005 - [PATCH] xenctld - a control channel multiplexing daemon

As I mentioned in earlier posts, we implemented a small multiplexing
daemon for the control channels in Xen.  It was designed to help
understand how Xen worked and not initially as a management framework. 
I''m posting it because I think some people might find it useful.

It''s very much like xcs.

I also included two example tools for xenctld.  One thing that I see in
common between these two apps is that they both implemented better
interfaces to Xen.  Here''s what I propose:

1) Change xcs to use unix domain sockets.
2) Add support to xcs to export ptys (storing info in the filesystem
much the same way xenctld does)
3) Change xenctld tools to use xcs.
4) Factor out most of xen interaction in xcs to standard libraries.

I see a three level architecture, the first level being highly portable
libraries that simplify interacting with Xen.  This would target every
platform Xen runs on.

The second level would be a daemon that is not as portable but still
very portable (for instance, you may have a posix daemon, a win32
daemon, etc.).

The third level would be simple applications that present a
platform-level interface (here it might make sense to have a set of
Linux tools, a set of BSD tools, etc.).

Thoughts?  I''m willing to code these things up.  Just want to make sure
it''s agreeable first.

You can get a copy of xenctld at:

http://www.cs.utexas.edu/users/aliguori/xenctld-0.0.1.tar.bz2

Regards,

-- 
Anthony Liguori
Linux Technology Center (LTC) - IBM Austin
E-mail: aliguori@us.ibm.com
Phone: (512) 838-1208





-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

> 1) Change xcs to use unix domain sockets.
The original thought behind using ip sockets behind xcs was in
planning for cluster management.  Obviously there are other ways to
map this in, I''m not terribly fussy on this point.
> 2) Add support to xcs to export ptys (storing info in the filesystem
> much the same way xenctld does)
I would much rather see xcs only handle control messages, and see the
console stuff broken off onto a seperate split driver.  ptys/ip
sockets/whatever can then be mapped in at the backend independent of
how the control plane works.  I just chatted with keir about this, and
he agrees that pulling console messages off onto their own rings and
providing a separate backend for them would be a good thing.
> 3) Change xenctld tools to use xcs.
sure... although i think there is a fair bit more involved in
create/destroy than what those tools are providing.
> 4) Factor out most of xen interaction in xcs to standard libraries.
Much of the xen interaction in xcs _is_ already in shared libraries
(libxc and libxutil).  The control channels can only be safely bound
by a single consumer in dom0 -- xcs just serves to multiplex this
access.  The interfaces in xcs could probably do with some cleaning
up, as they are reflective of my pulling a lot of the structure out of
the python code in December.  I''m not sure what bits of it
you''d like
to see generalized out of xcs though...  can you be more specific?
 
> I see a three level architecture, the first level being highly portable
> libraries that simplify interacting with Xen.  This would target every
> platform Xen runs on.
>  ...
This is what we have been shooting for with the new control tools: 1.
libxc/xutil, 2. xcs, 3. higher-level tools.
> Thoughts?  I''m willing to code these things up.  Just want to make
sure
> it''s agreeable first.
Our current plan with the control tools is in fixing up a couple of
things (1) how vm state is represented in dom0, and (2) how easy it is
to add and maintain new tools, drivers, etc.

xcs is a first step, in that it allows new tools be run along side xend.

The next step, coming soon, will be a ''persistent store'' which
will be
a repository for vm state.  This will hold things like active domain
configurations, device details, etc.

In addition to this, we have been discussing the option of adding
endpoint addressing to the control messages.  So driver setup, for
instance, would move toward a control tool pushing details into the
store prior to starting the domain.  the frontend driver would then
query the store for the backend details, and then address the backend
directly.  This should make extending the tools considerably easier.

This is all very high on the to do list, and should start to emerge
over the next while.  It would be great to discuss design points in
more detail on the list.

Things are a little busy here right now, so i hope this isn''t too
brief. ;)

best,
a.


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

On Fri, 21 Jan 2005, Andrew Warfield wrote:
> > 1) Change xcs to use unix domain sockets.
> 
> The original thought behind using ip sockets behind xcs was in
> planning for cluster management.  Obviously there are other ways to
> map this in, I''m not terribly fussy on this point.
how will this work if we run a 256-node xen cluster (which we will be
doing here for testing)? Wouldn''t we want the IP socket?
> I would much rather see xcs only handle control messages, and see the
> console stuff broken off onto a seperate split driver.  ptys/ip
> sockets/whatever can then be mapped in at the backend independent of how
> the control plane works.  I just chatted with keir about this, and he
> agrees that pulling console messages off onto their own rings and
> providing a separate backend for them would be a good thing.
make sense to me, again I sure hope we can keep things so that we can put 
lightweight xend''s on bproc slave nodes (i.e. in the end we''d
like to be
python-free) and control from a single front-end node via tcp.

I think python-free is beyond the scope of this discussion, just thought 
I''d mention it -- it''s a real problem to run python tools in
bproc
clusters.

ron


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

On Fri, 2005-01-21 at 10:39, Andrew Warfield wrote:
> The original thought behind using ip sockets behind xcs was in
> planning for cluster management.  Obviously there are other ways to
> map this in, I''m not terribly fussy on this point.
Yeah, I figured as much.  My only reason for domain sockets is that in
the minimal configuration, I figured that domain-0 should not expose any
network sockets.  I''m fine with exporting as both a domain and an ip
socket (based on a command line option).
> > 2) Add support to xcs to export ptys (storing info in the filesystem
> > much the same way xenctld does)
> 
> I would much rather see xcs only handle control messages, and see the
> console stuff broken off onto a seperate split driver.  ptys/ip
> sockets/whatever can then be mapped in at the backend independent of
> how the control plane works.  I just chatted with keir about this, and
> he agrees that pulling console messages off onto their own rings and
> providing a separate backend for them would be a good thing.
Is this just what it sounds like?
> > 3) Change xenctld tools to use xcs.
> 
> sure... although i think there is a fair bit more involved in
> create/destroy than what those tools are providing.
Oh yes.  The vm-create tool does not work.  I hacked it up last night
hoping that it would work for at least ramdisk-based kernels.  I did not
get it to work.
> > 4) Factor out most of xen interaction in xcs to standard libraries.
> 
> Much of the xen interaction in xcs _is_ already in shared libraries
> (libxc and libxutil).  The control channels can only be safely bound
> by a single consumer in dom0 -- xcs just serves to multiplex this
> access.  The interfaces in xcs could probably do with some cleaning
> up, as they are reflective of my pulling a lot of the structure out of
> the python code in December.  I''m not sure what bits of it
you''d like
> to see generalized out of xcs though...  can you be more specific?
The thought was that xcs might be something that has to be rewritten. 
If ptys are not in it than I guess it doesn''t really matter.
> > I see a three level architecture, the first level being highly
portable
> > libraries that simplify interacting with Xen.  This would target every
> > platform Xen runs on.
> >  ...
> 
> This is what we have been shooting for with the new control tools: 1.
> libxc/xutil, 2. xcs, 3. higher-level tools.
> 
> > Thoughts?  I''m willing to code these things up.  Just want to
make sure
> > it''s agreeable first.
> 
> Our current plan with the control tools is in fixing up a couple of
> things (1) how vm state is represented in dom0, and (2) how easy it is
> to add and maintain new tools, drivers, etc.
> 
> xcs is a first step, in that it allows new tools be run along side xend.
Great.
> The next step, coming soon, will be a ''persistent store''
which will be
> a repository for vm state.  This will hold things like active domain
> configurations, device details, etc.
What are the thoughts on how this will be exposed?  Is this targeted for
user space or kernel space?

My thoughts along these lines was to use a directory and simulate a
/proc-like structure in user space.  This has problems though.
> In addition to this, we have been discussing the option of adding
> endpoint addressing to the control messages.  So driver setup, for
> instance, would move toward a control tool pushing details into the
> store prior to starting the domain.  the frontend driver would then
> query the store for the backend details, and then address the backend
> directly.  This should make extending the tools considerably easier.
Indeed.
> This is all very high on the to do list, and should start to emerge
> over the next while.  It would be great to discuss design points in
> more detail on the list.
> 
> Things are a little busy here right now, so i hope this isn''t too
brief. ;)
If there''s things we can do to help out, let us know.  I''m
going to
working on xcs.  If you''ve got feature requests or anything just say
them on the list, I''m sure someone will pick it up :-)

Thanks,
> best,
> a.
> 
> 
> -------------------------------------------------------
> This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
> Tool for open source databases. Create drag-&-drop reports. Save time
> by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
> Download a FREE copy at http://www.intelliview.com/go/osdn_nl
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xen-devel
-- 
Anthony Liguori
Linux Technology Center (LTC) - IBM Austin
E-mail: aliguori@us.ibm.com
Phone: (512) 838-1208




-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

On Fri, 2005-01-21 at 11:19, Ronald G. Minnich wrote:
> On Fri, 21 Jan 2005, Andrew Warfield wrote:
> 
> > > 1) Change xcs to use unix domain sockets.
> > 
> > The original thought behind using ip sockets behind xcs was in
> > planning for cluster management.  Obviously there are other ways to
> > map this in, I''m not terribly fussy on this point.
> 
> how will this work if we run a 256-node xen cluster (which we will be
> doing here for testing)? Wouldn''t we want the IP socket?
It''s not so much about whether there would be an IP interface at all,
more about what''s the lowest possible interface.  A daemon that uses
the
domain socket interface could then export an IP interface.

Just running a single Xen server though, you probably don''t want a port
open if it doesn''t have to be.

Regards,
-- 
Anthony Liguori
Linux Technology Center (LTC) - IBM Austin
E-mail: aliguori@us.ibm.com
Phone: (512) 838-1208




-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

On Fri, 21 Jan 2005, Anthony Liguori wrote:
> It''s not so much about whether there would be an IP interface at
all,
> more about what''s the lowest possible interface.  A daemon that
uses the
> domain socket interface could then export an IP interface.
another daemon would be a concern. On these slave nodes, the fewer daemons 
the better.

ron


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

On Fri, 21 Jan 2005, Anthony Liguori wrote:
> Out of curiosity, why is that?  Is it the storage overhead?  Increasing
> the number of processes?  Or simply just the additional administrative
> headache?
more daemons means more things to get run when you least want them to run. 
It''s just a good idea to keep the daemons to a minimum. Requiring a
daemon
to translate a unix domain socket to ip just strikes me as very 
unnecessary.

But, I''m not writing it, so past a point I think you should ignore my 
requests if you don''t like them :-)

ron


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

On Fri, 2005-01-21 at 14:44, Ronald G. Minnich wrote:
> On Fri, 21 Jan 2005, Anthony Liguori wrote:
> 
> > It''s not so much about whether there would be an IP interface
at all,
> > more about what''s the lowest possible interface.  A daemon
that uses the
> > domain socket interface could then export an IP interface.
> 
> another daemon would be a concern. On these slave nodes, the fewer daemons 
> the better.
Out of curiosity, why is that?  Is it the storage overhead?  Increasing
the number of processes?  Or simply just the additional administrative
headache?
> ron
-- 
Anthony Liguori
Linux Technology Center (LTC) - IBM Austin
E-mail: aliguori@us.ibm.com
Phone: (512) 838-1208




-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

On Fri, 2005-01-21 at 14:54, Ronald G. Minnich wrote:
> On Fri, 21 Jan 2005, Anthony Liguori wrote:
> 
> > Out of curiosity, why is that?  Is it the storage overhead? 
Increasing
> > the number of processes?  Or simply just the additional administrative
> > headache?
> 
> more daemons means more things to get run when you least want them to run. 
> It''s just a good idea to keep the daemons to a minimum. Requiring
a daemon
> to translate a unix domain socket to ip just strikes me as very 
> unnecessary.
Sure, it''s not quite that simple though.  The unix domain interface
would be very very low-level.  The messages would pretty much get passed
directly to the domain''s ring queue.

The ip interface might support SNMP or some other standardized network
protocol.  In the very least, the IP interface would have to be
endian-neutral which means that it''s going to be a more complicated
protocol.
> But, I''m not writing it, so past a point I think you should ignore
my
> requests if you don''t like them :-)
Feedback is always a good thing.  I''m very curious to see what people
would want the control software to do.

Thanks,
> ron
-- 
Anthony Liguori
Linux Technology Center (LTC) - IBM Austin
E-mail: aliguori@us.ibm.com
Phone: (512) 838-1208




-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

On Fri, 2005-01-21 at 13:54 -0700, Ronald G. Minnich
wrote:
> more daemons means more things to get run when you least want them to run. 
> It''s just a good idea to keep the daemons to a minimum.
I prefer to instead keep the _function_ of daemons to a minimum, and run
as many purpose-built, lightweight daemons as needed to build the
functionality I want.
> Requiring a daemon to translate a unix domain socket to ip just
> strikes me as very unnecessary.
Reasonable people can differ, as it strikes me as perfectly reasonable
to build TCP translation as a separate layer on top of base UDP
communications.  This let''s those who want to avoid TCP from being
forced to deal with it, and makes any TCP policy encoded into the daemon
plugable via alternative implementations.

-- 
Jared Rhine <jared@wordzoo.com>



-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

On Fri, 21 Jan 2005, Jared Rhine wrote:
> I prefer to instead keep the _function_ of daemons to a minimum, and run
> as many purpose-built, lightweight daemons as needed to build the
> functionality I want.
I understand the preference from an esthetics point of view. From a
quantitative performance point of view, in an environment where you want
to minimize the activity that is running on a lightweight cluster node, it
has been shown that 1 is better than 2.

So, all other things being equal, if we can have 1 daemon with TCP sockets 
instead of a daemon with unix domain sockets and a daemon that turns that 
into TCP sockets, on a reasonably large cluster (1024 or more nodes) 
you''re much better off with 1 daemon. 1024 node clusters are not
uncommon
these days.

Again, this is not an issue of esthetics, it''s an issue of measured 
performance. 

ron


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

On Mon, 2005-01-24 at 09:33, Ronald G. Minnich wrote:
> On Fri, 21 Jan 2005, Jared Rhine wrote:
> Again, this is not an issue of esthetics, it''s an issue of
measured
> performance. 
Where''s the performance issue?  For instance, if you had one daemon
that
forked off two children, would that be better than having two individual
processes?  Is it simply a matter of having that extra file on disk?

I''m not sure I understand how this could create a performance problem. 
I simply don''t know that much about cluster-performance and am very
curious :-)

It seems like the only thing that would make sense is that it''s
multiple
processes but unfortunately this is the exact opposite performance
behavior of a n-way system...
> ron
> 
> 
> -------------------------------------------------------
> This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
> Tool for open source databases. Create drag-&-drop reports. Save time
> by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
> Download a FREE copy at http://www.intelliview.com/go/osdn_nl
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xen-devel
-- 
Anthony Liguori
Linux Technology Center (LTC) - IBM Austin
E-mail: aliguori@us.ibm.com
Phone: (512) 838-1208




-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

> > Again, this is not an issue of esthetics, it''s an issue of
measured
> > performance. 
> 
> Where''s the performance issue?
I think Ron was suggesting that the dual handoff of the data between the
UDP listener and the TCP listener would be foolish and avoidable and
inarguably is a measurable throughput degredation over a direct TCP
processing loop.

Which is a fair argument, though not raised in Ron''s earlier
objections.

It''s certainly not unreasonable to have a direct TCP listener as well
as
a UDP listener in the same daemon.  Though single-purpose daemons are
better for maintainability and robustness, the minor extra code to have
both Unix domain and TCP branches is straightforward.

I do think that the "no TCP is my lightweight dom0" people should be
supported, though I''m not personally one of them.  So a Unix domain
listener seems like it should be available through xcs.  Making those
people build and maintainer and _alternative_ multiplexing switch isn''t
good for anyone.

I now favor xcs supporting both listening approaches, though I''m not
volunteering code.  I''d rather let the experts sort out the right
layering and then contribute on the tools that use the final mechanism.
> I''m not sure I understand how this could create a performance
problem.
> I simply don''t know that much about cluster-performance and am
very
> curious :-)
Ron, is your performance concern because large clusters need to pass a
very high volume of control messages?  For any low-volume situation,
this unix domain/TCP argument is a non-issue, mostly?

-- 
Jared Rhine <jared@wordzoo.com>



-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

[Ronald == rminnich@lanl.gov on Mon, 24 Jan 2005 10:04:29 -0700 (MST)]

  Jared> Ron, is your performance concern because large clusters need
  Jared> to pass a very high volume of control messages?

  Ronald> re-reading this, however, I think my concerns about the
  Ronald> extra daemon shrink into insignificance when I think about
  Ronald> Python-based Xend ...

I think most people are agreed that a non-Python control
infrastructure will be developed in the not-so-distant future.  So
to the extent your concerns about multiple daemons apply in a
non-Python environment, they should be addressed.

  Ronald> The issue is that, in any large scale cluster, if you have
  Ronald> enough processes running besides your application on the
  Ronald> nodes, then the act of scheduling those processes -- even
  Ronald> for select calls that return instantly
  Ronald> -- can derange the application performance.

It may be that clustering people need a whole different toolset.  If
you were trying to minimize process count, then you probably wouldn''t
even want xcs, a multiplexing switch anyway.  Indeed, why multiplex at
all?  You might use a single control mechanism, preferably combining
whatever subset of xcs, xensv, xm, etc you needed.

I''ll leave it others to figure out what all this means for
xcs''s
future.

Thanks for your cluster scheduling overview; it was helpful.

-- jared@wordzoo.com

http://www.geekcode.com/geek.html
-----BEGIN GEEK CODE BLOCK-----
Version: 3.12
GCM/CS/B
d s:++>+ a-
C++(++++)$ ULBSH++++ P+++ L+++ E++(+++) W++++ N- !o !K w !O M V
PS+++(-) PE++(--) Y+ PGP>++
t@ 5 X+ R>+ tv++>-- b>++ DI+ D- G
e++ h- r++>+++ y+++
------END GEEK CODE BLOCK------


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

On Mon, 2005-01-24 at 10:55, Jared Rhine wrote:
> > > Again, this is not an issue of esthetics, it''s an issue
of measured
> > > performance. 
> > 
> > Where''s the performance issue?
> 
> I think Ron was suggesting that the dual handoff of the data between the
> UDP listener and the TCP listener would be foolish and avoidable and
> inarguably is a measurable throughput degredation over a direct TCP
> processing loop.
You''re not going to have a direct listener though on TCP.  I think you
mean unix domain sockets too instead of UDP.  The current message
passing architecture assumes the messages are originating from the
native architecture.

They do not take into account things like endianness into
consideration.  They certainly don''t security into account at all.  To
export a proper TCP interface requires a higher level protocol.  The TCP
interface will have to do a certain amount of work to take these things
into account.

I think we all agree that we need a TCP interface.

The only real question is whether you export a lower level protocol
interface via something like domain sockets.  Domain sockets are
extremely fast on most platforms (usually reducing to something like a
memcpy).  Considering the message size here I do not think you can make
a performance argument for having a TCP daemon go through a domain
socket.

If we agree that we need a higher level protocol for TCP, then it
becomes pretty apparently that there''s going to be desire for multiple
protocols.  Maybe a less security conscious protocol for isolated
clusters (like xcs implements right now) verses a more hardened protocol
for enterprise usage.

Using multiple daemons here becomes a no-brainer.  You just start the
appropriate daemon for whatever protocol you wish to export.  Properly
architected, multi daemons could even be run simultaneously to support
multiple sets of control tools.
> Ron, is your performance concern because large clusters need to pass a
> very high volume of control messages?  For any low-volume situation,
> this unix domain/TCP argument is a non-issue, mostly?
I have a feeling that the concern is that there''s a general fear that
more daemons == more resource usage == more administrative overhead.  I
don''t think this is true though.  I think it simply depends on how
things are architected.

-- 
Anthony Liguori
Linux Technology Center (LTC) - IBM Austin
E-mail: aliguori@us.ibm.com
Phone: (512) 838-1208




-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

On Mon, 2005-01-24 at 11:52, Jared Rhine wrote:
>   Ronald> The issue is that, in any large scale cluster, if you have
>   Ronald> enough processes running besides your application on the
>   Ronald> nodes, then the act of scheduling those processes -- even
>   Ronald> for select calls that return instantly
>   Ronald> -- can derange the application performance.
This is why schedulers have priority.  Just bump up the priority of your
desired app.  Also, with the Linux O(1) scheduler, a process blocked on
I/O (at least, by my understanding) should have no impact on other
processes scheduling.

-- 
Anthony Liguori
Linux Technology Center (LTC) - IBM Austin
E-mail: aliguori@us.ibm.com
Phone: (512) 838-1208




-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

As I see it, we have two (or more) camps/uses for xen.  One (micro) that
is dealing primarily with a relatively small number of machines -
typically one and is concerned with speed and security of the local
machine.  The other (macro) is HPC or large enterprise/server farm type
setups that requires out of the physical box remote connectivity.  The
third camp just wants something reliable and easy to use (probably -
though they have been quiet up to this point so I''m just guessing
here).  The real question is can one tool be made to fit all, or is the
gap so large that two different tools are required.

B.



On Mon, 2005-01-24 at 15:07, Anthony Liguori wrote:
> On Mon, 2005-01-24 at 10:55, Jared Rhine wrote:
> > > > Again, this is not an issue of esthetics, it''s an
issue of measured
> > > > performance. 
> > > 
> > > Where''s the performance issue?
> > 
> > I think Ron was suggesting that the dual handoff of the data between
the
> > UDP listener and the TCP listener would be foolish and avoidable and
> > inarguably is a measurable throughput degredation over a direct TCP
> > processing loop.
> 
> You''re not going to have a direct listener though on TCP.  I think
you
> mean unix domain sockets too instead of UDP.  The current message
> passing architecture assumes the messages are originating from the
> native architecture.
> 
> They do not take into account things like endianness into
> consideration.  They certainly don''t security into account at all.
To
> export a proper TCP interface requires a higher level protocol.  The TCP
> interface will have to do a certain amount of work to take these things
> into account.
> 
> I think we all agree that we need a TCP interface.
> 
> The only real question is whether you export a lower level protocol
> interface via something like domain sockets.  Domain sockets are
> extremely fast on most platforms (usually reducing to something like a
> memcpy).  Considering the message size here I do not think you can make
> a performance argument for having a TCP daemon go through a domain
> socket.
> 
> If we agree that we need a higher level protocol for TCP, then it
> becomes pretty apparently that there''s going to be desire for
multiple
> protocols.  Maybe a less security conscious protocol for isolated
> clusters (like xcs implements right now) verses a more hardened protocol
> for enterprise usage.
> 
> Using multiple daemons here becomes a no-brainer.  You just start the
> appropriate daemon for whatever protocol you wish to export.  Properly
> architected, multi daemons could even be run simultaneously to support
> multiple sets of control tools.
> 
> > Ron, is your performance concern because large clusters need to pass a
> > very high volume of control messages?  For any low-volume situation,
> > this unix domain/TCP argument is a non-issue, mostly?
> 
> I have a feeling that the concern is that there''s a general fear
that
> more daemons == more resource usage == more administrative overhead.  I
> don''t think this is true though.  I think it simply depends on how
> things are architected.

-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

On Fri, 2005-01-21 at 16:39 +0000, Andrew Warfield wrote:

Thanks for the info.  

> > 4) Factor out most of xen interaction in xcs to standard libraries.
> 
> Much of the xen interaction in xcs _is_ already in shared libraries
> (libxc and libxutil).  The control channels can only be safely bound
> by a single consumer in dom0 -- xcs just serves to multiplex this
> access.  The interfaces in xcs could probably do with some cleaning
> up, as they are reflective of my pulling a lot of the structure out of
> the python code in December.  I''m not sure what bits of it
you''d like
> to see generalized out of xcs though...  can you be more specific?
>  
> > I see a three level architecture, the first level being highly
portable
> > libraries that simplify interacting with Xen.  This would target every
> > platform Xen runs on.
> >  ...
> 
> This is what we have been shooting for with the new control tools: 1.
> libxc/xutil, 2. xcs, 3. higher-level tools.
Is it reasonable to expect that a port to another machine architecture,
assuming the libxc/xutil are provided, then xcs and higher-level tools
should just follow?
> 
> > Thoughts?  I''m willing to code these things up.  Just want to
make sure
> > it''s agreeable first.
> 
> Our current plan with the control tools is in fixing up a couple of
> things (1) how vm state is represented in dom0, and (2) how easy it is
> to add and maintain new tools, drivers, etc.
> 
> xcs is a first step, in that it allows new tools be run along side xend.
> 
> The next step, coming soon, will be a ''persistent store''
which will be
> a repository for vm state.  This will hold things like active domain
> configurations, device details, etc.
What form do you see this persistent store taking?  Is it just for 
currently present VM''s, or can it be used also to pre-configure
VM''s and/or have pre-defined VM''s?  I assume some sort of
filesystem
backed entity is to be used for the persistent store, correct?
Format, content, etc.?
> 
> In addition to this, we have been discussing the option of adding
> endpoint addressing to the control messages.  So driver setup, for
> instance, would move toward a control tool pushing details into the
> store prior to starting the domain.  the frontend driver would then
> query the store for the backend details, and then address the backend
> directly.  This should make extending the tools considerably easier.
> 
> This is all very high on the to do list, and should start to emerge
> over the next while.  It would be great to discuss design points in
> more detail on the list.
Yes.  This is a good overview.  More details especially in regards to
the persistent store, and the set of tools being built would add to
this discussion.  Also, it is likely that I (and others) can provide
development and testing assistance if we understand what is being
developed and where our contributions can be used.

Thanks.

              Michael
> 
> Things are a little busy here right now, so i hope this isn''t too
brief. ;)
> 
> best,
> a.
-- 
Michael Hohnbaum                             503 578 5486
hohnbaum@us.ibm.com                          t/l 775 5486



-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

Anthony Liguori wrote:
> They do not take into account things like endianness into
> consideration.  They certainly don''t security into account at all.
To
> export a proper TCP interface requires a higher level protocol.  The TCP
> interface will have to do a certain amount of work to take these things
> into account.
> 
> I think we all agree that we need a TCP interface.
No. In my implementation I think I have shown that you can run your 
cluster with a much more minimal network-interface.

My current implementation is less than 100 lines of network-facing code 
(using ICMP payloads, but the same could be achieved with UDP), though 
that increases to about 300 if you count in the SHA-1 implementation 
that I am using for verifying customer credentials. So 300 lines is for 
a complete solution including crypto and accounting/payment.

My nodes answer ARPs and pings, but that is all.

# nmap -sS blackbox07

Starting nmap 3.55 ( http://www.insecure.org/nmap/ ) at 2005-01-25 18:26 PST
All 1660 scanned ports on blackbox07 are: filtered

Nmap run completed -- 1 IP address (1 host up) scanned in 71.969 seconds

Jacob


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

Jacob Gorm Hansen wrote:
> My current implementation is less than 100 lines of network-facing 
> code (using ICMP payloads, but the same could be achieved with UDP), 
> though that increases to about 300 if you count in the SHA-1 
> implementation that I am using for verifying customer credentials. So 
> 300 lines is for a complete solution including crypto and 
> accounting/payment.
That''s certainly interesting.  Are you using ICMP simply for high-level
management or are you passing event channel messages via ICMP? Are you 
just punting issues of reliability?

I agree with you, TCP is not the only network solution.  However, it 
seems logically that a large number of people will want a TCP solution :-)

Regards,

-- 

Anthony Liguori
anthony@codemonkey.ws



-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

Anthony Liguori wrote:
> Jacob Gorm Hansen wrote:
> 
>> My current implementation is less than 100 lines of network-facing 
>> code (using ICMP payloads, but the same could be achieved with UDP), 
>> though that increases to about 300 if you count in the SHA-1 
>> implementation that I am using for verifying customer credentials. So 
>> 300 lines is for a complete solution including crypto and 
>> accounting/payment.
> 
> 
> That''s certainly interesting.  Are you using ICMP simply for
high-level
> management or are you passing event channel messages via ICMP? Are you 
> just punting issues of reliability?
> 
> I agree with you, TCP is not the only network solution.  However, it 
> seems logically that a large number of people will want a TCP solution :-)
> 
I am using TCP, but I am doing both bootstrap+shutdown of new domains, 
migration, and accounting mostly within domUs. I send an ICMP payload 
with an initial ticket/capability describing the max memory, disk 
extents, external IP etc., and the ICMP server in dom0 decodes/verifies 
that, an fires up a domU with a minimal TPC/IP stack and server inside. 
The domain builder is then uploaded on that connection, takes over, and 
handles the bootstrap from there. Same is true for migration.

Jacob


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

Hi Michael,
> Is it reasonable to expect that a port to another machine architecture,
> assuming the libxc/xutil are provided, then xcs and higher-level tools
> should just follow?
This seems pretty reasonable to me.  Most of what happens in xend
right now is managing the state of the vms, much of the real mechanism
(domain building, suspension, migration, etc.) is really inside the
underlying libraries (xc, xutil).  The only vaguely dependent thing i
can think of that isn''t masked by these libraries is the shared memory
interface ot the control rings accessed through xcs.  This isn''t a
very big deal though.
 
> What form do you see this persistent store taking?  Is it just for
> currently present VM''s, or can it be used also to pre-configure
> VM''s and/or have pre-defined VM''s?  I assume some sort of
filesystem
> backed entity is to be used for the persistent store, correct?
> Format, content, etc.?
We have had some discussion about this, and Mike Wray has written some
design documentation on the new store plan.  I think the principle
goal of the store is to encapsulate all of the state relating to the
currently running set of VMs in a single place, that has clear API for
access.  Our basic view of this at the moment is a hierarchical set of
key-value pairs that looks remarkably like a file system. ;)

Having the store worked in properly should 1. make rebooting dom0 with
active VMs unaffected more possible, 2. make adding new functionality
(e.g. new split device types) more straightforward, and 3. make
writing tools that need state information, and 4. scaling control up
to a cluster all a lot more tractable.

Including more long-lived stuff like config files seems like a
potentially reasonable thing to do as well.
> Yes.  This is a good overview.  More details especially in regards to
> the persistent store, and the set of tools being built would add to
> this discussion.  Also, it is likely that I (and others) can provide
> development and testing assistance if we understand what is being
> developed and where our contributions can be used.
Fantastic!  We would be happy to have the help.  We are still in the
process of hashing out exactly what the new interfaces should look
like and how to go about moving to a new set of tools, potentially for
the 3.0 release.  Probably the best way forward is for us to post
design sketches to the list as they become available, and iteratively
move towards bits of work that people can bite off.  Any specific
ideas that you guys want to throw in now would be more than welcome as
well.

sound okay?

a.


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

> In reality, what''s the different between a domain that''s
suspended to
> disk and a clone domain that just hasn''t been started yet.
> 
> Does the current persistent store architecture attempt to solve this
> problem?  Does it attempt to solve both problems?
No.  There has been no discussion of storing suspended domains.  For
the moment, i think it''s probably best to design the store to contain
runtime state for the currently executing set of VMs on the physical
host.  So records in the store are persistent across daemon restarts,
and dom0 restarts.

I don''t think any of use really imagine the store as being a very
complicated bit of code at all.  The bit of design that needs a little
more thought is how things like device driver setup should look with
the new control tools.  This isn''t necessarily terrible -- but
changing e.g. device channel negotiation at driver setup is going to
be a lot less work this round (with three split drivers) than it is
going to be if we need yet another set of control tools for Xen 4. ;)

a.


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

On Wed, 2005-01-26 at 08:33, Andrew Warfield wrote:
> Having the store worked in properly should 1. make rebooting dom0 with
> active VMs unaffected more possible, 2. make adding new functionality
> (e.g. new split device types) more straightforward, and 3. make
> writing tools that need state information, and 4. scaling control up
> to a cluster all a lot more tractable.
It seems like there''s two distinct problems.

1) Managing domain specific state for running domains.

This can be done easily enough since a domid is a unique identifier. 
Any associative database would work just fine.

2) Managing domain specific state for non-running domains.

Since non-running domains have to have some sort of disk representation
(either a config file of some sort or a snapshot image) this also is
pretty easy.


I''m not convinced there''s a great way to merge the two though.
It gets
really hairy once you start doing things like cloning or migration. 
There''s a lot of subtle things like reassigning MAC addresses that gets
nasty.

In reality, what''s the different between a domain that''s
suspended to
disk and a clone domain that just hasn''t been started yet.

Does the current persistent store architecture attempt to solve this
problem?  Does it attempt to solve both problems?

I''d like to see a persistent store that just addressed problem #1.  I
think once you get into trying to solve #2, you''ll create a solution
that won''t work for everyone.

I''d like to see Xen with a tool that can create domains from a
configuration file but at the same time, it would be nice if the
persistent store worked well with domain''s that are created by hand.

Regards,

-- 
Anthony Liguori
Linux Technology Center (LTC) - IBM Austin
E-mail: aliguori@us.ibm.com
Phone: (512) 838-1208




-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

On Wed, 2005-01-26 at 13:11, Andrew Warfield wrote:
> > In reality, what''s the different between a domain
that''s suspended to
> > disk and a clone domain that just hasn''t been started yet.
> > 
> > Does the current persistent store architecture attempt to solve this
> > problem?  Does it attempt to solve both problems?
> 
> No.  There has been no discussion of storing suspended domains.  For
> the moment, i think it''s probably best to design the store to
contain
> runtime state for the currently executing set of VMs on the physical
> host.  So records in the store are persistent across daemon restarts,
> and dom0 restarts.
Excellent!  Then I have no worries at all about it.

You mentioned the current design looks a lot like a file system, any
thoughts on actually using the file system?  Either within the normal
file system (within something like /var/xen/<domid>) or as a pid-like
/proc entry (like /proc/xen/<domid>).

Regards,
-- 
Anthony Liguori
Linux Technology Center (LTC) - IBM Austin
E-mail: aliguori@us.ibm.com
Phone: (512) 838-1208




-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

On Wed, 2005-01-26 at 12:01, Anthony Liguori wrote:
> On Wed, 2005-01-26 at 13:11, Andrew Warfield wrote:
> > > In reality, what''s the different between a domain
that''s suspended to
> > > disk and a clone domain that just hasn''t been started
yet.
> > > 
> > > Does the current persistent store architecture attempt to solve
this
> > > problem?  Does it attempt to solve both problems?
> > 
> > No.  There has been no discussion of storing suspended domains.  For
> > the moment, i think it''s probably best to design the store to
contain
> > runtime state for the currently executing set of VMs on the physical
> > host.  So records in the store are persistent across daemon restarts,
> > and dom0 restarts.
> 
> Excellent!  Then I have no worries at all about it.
> 
> You mentioned the current design looks a lot like a file system, any
> thoughts on actually using the file system?  Either within the normal
> file system (within something like /var/xen/<domid>) or as a pid-like
> /proc entry (like /proc/xen/<domid>).

Do you really want the store in Kernel Space? Wouldn''t it be better if
it were stored in User Space? Could you restart Dom0 if it''s in Dom0?
Wouldn''t it be better to store out on disk that can be recovered? You
could use the store for more than the above, domain configuration could
be stored and accessed there. 

Thanks,
Dan



-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

On Wed, 2005-01-26 at 14:01 -0600, Anthony Liguori
wrote:
> On Wed, 2005-01-26 at 13:11, Andrew Warfield wrote:
> > > In reality, what''s the different between a domain
that''s suspended to
> > > disk and a clone domain that just hasn''t been started
yet.
> > > 
> > > Does the current persistent store architecture attempt to solve
this
> > > problem?  Does it attempt to solve both problems?
> > 
> > No.  There has been no discussion of storing suspended domains.  For
> > the moment, i think it''s probably best to design the store to
contain
> > runtime state for the currently executing set of VMs on the physical
> > host.  So records in the store are persistent across daemon restarts,
> > and dom0 restarts.
> 
> Excellent!  Then I have no worries at all about it.
> 
> You mentioned the current design looks a lot like a file system, any
> thoughts on actually using the file system?  Either within the normal
> file system (within something like /var/xen/<domid>) or as a pid-like
> /proc entry (like /proc/xen/<domid>).
Wouldn''t want to do this in the kernel (i.e., /proc or /sysfs) as it
would not be persistent across dom0 restarts.  In a regular filesystem
as {directories, subdirectories, files} would be reasonable.  In either
case, persistent store would need to determine whether the entire
system (Xen) has restarted, dom0 has restarted, or just the xend has
restarted and clean up residual info appropriately.

While beyond the current focus, persistent store could feasibly
be used to hold domain definitions for non-existent domains and 
suspended domains.  One could envision adding a state field into
the domain configuration/definition.  Valid states for current 
capabilities would be {active, suspended, migrated, inactive}.  On
the other hand, keeping persistent store info on only active domains
in this mechanism and moving info to higher level tools about predefined
domains, suspended domains, etc. would work.  It''s a tradeoff between
similar info being maintained in multiple places, or added complexity
in a single place.

                      Michael
> 
> Regards,
-- 
Michael Hohnbaum                             503 578 5486
hohnbaum@us.ibm.com                          t/l 775 5486



-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

On Wed, 2005-01-26 at 16:28 -0600, Anthony Liguori
wrote:
> On Wed, 2005-01-26 at 15:21, Daniel Stekloff wrote:
> > On Wed, 2005-01-26 at 12:01, Anthony Liguori wrote:
> > > You mentioned the current design looks a lot like a file system,
any
> > > thoughts on actually using the file system?  Either within the
normal
> > > file system (within something like /var/xen/<domid>) or as
a pid-like
> > > /proc entry (like /proc/xen/<domid>).
> > 
> > 
> > Do you really want the store in Kernel Space? Wouldn''t it be
better if
> > it were stored in User Space? Could you restart Dom0 if it''s
in Dom0?
> > Wouldn''t it be better to store out on disk that can be
recovered? You
> > could use the store for more than the above, domain configuration
could
> > be stored and accessed there. 
> 
> Yup.  Automatic relaunching of domains wouldn''t work if it was in
kernel
> space.
> 
> A downside of using disk also would be that you couldn''t enforce
any
> sort of schema.

Sorry, I think I misunderstood what you were stating. I like the idea of
storing it in a file system-like architecture that''s easily traversed
and you can leverage existing file system security, calls, etc. I
thought for a minute that you wanted to implement it as a virtual file
system in Kernel Space. 

My bad... sorry...

Dan



-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

On Wed, 2005-01-26 at 15:21, Daniel Stekloff wrote:
> On Wed, 2005-01-26 at 12:01, Anthony Liguori wrote:
> > You mentioned the current design looks a lot like a file system, any
> > thoughts on actually using the file system?  Either within the normal
> > file system (within something like /var/xen/<domid>) or as a
pid-like
> > /proc entry (like /proc/xen/<domid>).
> 
> 
> Do you really want the store in Kernel Space? Wouldn''t it be
better if
> it were stored in User Space? Could you restart Dom0 if it''s in
Dom0?
> Wouldn''t it be better to store out on disk that can be recovered?
You
> could use the store for more than the above, domain configuration could
> be stored and accessed there. 
Yup.  Automatic relaunching of domains wouldn''t work if it was in
kernel
space.

A downside of using disk also would be that you couldn''t enforce any
sort of schema.
> Thanks,
> Dan
-- 
Anthony Liguori
Linux Technology Center (LTC) - IBM Austin
E-mail: aliguori@us.ibm.com
Phone: (512) 838-1208




-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

On Wed, 2005-01-26 at 15:49, Michael Hohnbaum wrote:
> While beyond the current focus, persistent store could feasibly
> be used to hold domain definitions for non-existent domains and 
> suspended domains.  One could envision adding a state field into
> the domain configuration/definition.  Valid states for current 
> capabilities would be {active, suspended, migrated, inactive}.  On
Yes, but the problem that occurs is uniquely identifying a domain.  In
other words, what''s the key used within the persistent store?

If it''s domain id (which is what I assume it''s going to be),
you cannot
tag it as having an "inactive" state because there''s nothing
that
prevents a domain from being created with the same domain id.

Also, if you try to assign domains UUIDs or something, what do you do
for cloning/checkpointing?  Do you assign a new UUID on a clone but not
on a checkpoint?  Does assigning new UUIDs propagate to things like MAC
addresses or other things that are supposed to be unique?

There''s a lot to be thought about.  I think punting the problem (as
Andy
suggests) is the right approach for now.

Regards,

-- 
Anthony Liguori
Linux Technology Center (LTC) - IBM Austin
E-mail: aliguori@us.ibm.com
Phone: (512) 838-1208




-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

On Wed, 2005-01-26 at 06:33, Andrew Warfield wrote:
> Hi Michael,
> 
> > Is it reasonable to expect that a port to another machine
architecture,
> > assuming the libxc/xutil are provided, then xcs and higher-level tools
> > should just follow?
> 
> This seems pretty reasonable to me.  Most of what happens in xend
> right now is managing the state of the vms, much of the real mechanism
> (domain building, suspension, migration, etc.) is really inside the
> underlying libraries (xc, xutil).  The only vaguely dependent thing i
> can think of that isn''t masked by these libraries is the shared
memory
> interface ot the control rings accessed through xcs.  This isn''t a
> very big deal though.
>  
> > What form do you see this persistent store taking?  Is it just for
> > currently present VM''s, or can it be used also to
pre-configure
> > VM''s and/or have pre-defined VM''s?  I assume some
sort of filesystem
> > backed entity is to be used for the persistent store, correct?
> > Format, content, etc.?
> 
> We have had some discussion about this, and Mike Wray has written some
> design documentation on the new store plan.  I think the principle
> goal of the store is to encapsulate all of the state relating to the
> currently running set of VMs in a single place, that has clear API for
> access.  Our basic view of this at the moment is a hierarchical set of
> key-value pairs that looks remarkably like a file system. ;)

Could we see Mike Wray''s design documentation for the new store? 

> Having the store worked in properly should 1. make rebooting dom0 with
> active VMs unaffected more possible, 2. make adding new functionality
> (e.g. new split device types) more straightforward, and 3. make
> writing tools that need state information, and 4. scaling control up
> to a cluster all a lot more tractable.
> 
> Including more long-lived stuff like config files seems like a
> potentially reasonable thing to do as well.
> 
> > Yes.  This is a good overview.  More details especially in regards to
> > the persistent store, and the set of tools being built would add to
> > this discussion.  Also, it is likely that I (and others) can provide
> > development and testing assistance if we understand what is being
> > developed and where our contributions can be used.
> 
> Fantastic!  We would be happy to have the help.  We are still in the
> process of hashing out exactly what the new interfaces should look
> like and how to go about moving to a new set of tools, potentially for
> the 3.0 release.  Probably the best way forward is for us to post
> design sketches to the list as they become available, and iteratively
> move towards bits of work that people can bite off.  Any specific
> ideas that you guys want to throw in now would be more than welcome as
> well.

Do you have an outline of what the new set of tools will be? It''s been
mentioned that xcs will make way for multiple management apps rather
than just xm/xend. I''m curious to know what you''ve outlined so
far for
management apps. 

Thanks,

Dan



-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

On Wed, 2005-01-26 at 16:57 -0600, Anthony Liguori
wrote:
> On Wed, 2005-01-26 at 15:49, Michael Hohnbaum wrote:
> 
> > While beyond the current focus, persistent store could feasibly
> > be used to hold domain definitions for non-existent domains and 
> > suspended domains.  One could envision adding a state field into
> > the domain configuration/definition.  Valid states for current 
> > capabilities would be {active, suspended, migrated, inactive}.  On
> 
> Yes, but the problem that occurs is uniquely identifying a domain.  In
> other words, what''s the key used within the persistent store?
> 
> If it''s domain id (which is what I assume it''s going to
be), you cannot
> tag it as having an "inactive" state because there''s
nothing that
> prevents a domain from being created with the same domain id.
Active domains would have a domain ID; inactive domains could be
labeled.
> 
> Also, if you try to assign domains UUIDs or something, what do you do
> for cloning/checkpointing?  Do you assign a new UUID on a clone but not
> on a checkpoint?  Does assigning new UUIDs propagate to things like MAC
> addresses or other things that are supposed to be unique?
Now you are making my head hurt.
> 
> There''s a lot to be thought about.  I think punting the problem
(as Andy
> suggests) is the right approach for now.
Punting is reasonable.  The immediate need is persistent store for 
active domains.  Other needs can be pushed up the management tool 
stack to reside with the consumer of the information.  Thus, the 
specific tool used to start a domain can define its own configuration
file (but please not a python script).  Whatever suspends/restarts
a domain can provide its own file for tracking suspended domains, etc.

Different usage scenarios could result in different tools providing
similar functions each which could choose to use a common config file
or define their own, based on needs.

While it is attractive to not have multiple similar config files,
trying to define a common one to contain all anticipated needed
information could easily become unwieldly.

                Michael
> 
> Regards,
> 
-- 
Michael Hohnbaum                             503 578 5486
hohnbaum@us.ibm.com                          t/l 775 5486



-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

On Wed, 2005-01-26 at 14:57, Anthony Liguori wrote:
> On Wed, 2005-01-26 at 15:49, Michael Hohnbaum wrote:
> 
> > While beyond the current focus, persistent store could feasibly
> > be used to hold domain definitions for non-existent domains and 
> > suspended domains.  One could envision adding a state field into
> > the domain configuration/definition.  Valid states for current 
> > capabilities would be {active, suspended, migrated, inactive}.  On
> 
> Yes, but the problem that occurs is uniquely identifying a domain.  In
> other words, what''s the key used within the persistent store?
> 
> If it''s domain id (which is what I assume it''s going to
be), you cannot
> tag it as having an "inactive" state because there''s
nothing that
> prevents a domain from being created with the same domain id.

Can''t we add functionality in Xen to make domain id''s unique?
When
creating a new domain, the management tools can query the running (or
even suspended) domains and find a unique domain id to use. I think you
can tag a domains with their state. 

> Also, if you try to assign domains UUIDs or something, what do you do
> for cloning/checkpointing?  Do you assign a new UUID on a clone but not
> on a checkpoint?  Does assigning new UUIDs propagate to things like MAC
> addresses or other things that are supposed to be unique?

If you clone a domain, it isn''t the same domain anymore and would get a
new id. As for checkpointing - you could use the time/date of the
checkpoint and add that to the domain''s id. 

MAC addresses for specific domains that are in the configuration file or
generated could be stored with the domain configuration and available
for queries.  


> There''s a lot to be thought about.  I think punting the problem
(as Andy
> suggests) is the right approach for now.

I disagree. If we''re to consider the larger management world, we need
to
lay the groundwork for managing domains now. I think the questions
aren''t easily answered, but I believe they should be. If we
don''t
implement everything to start, we should at least have an idea where
we''re going. 

Thanks,

Dan



-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

On Wed, 2005-01-26 at 14:57, Anthony Liguori wrote:
> On Wed, 2005-01-26 at 15:49, Michael Hohnbaum wrote:
> 
> > While beyond the current focus, persistent store could feasibly
> > be used to hold domain definitions for non-existent domains and 
> > suspended domains.  One could envision adding a state field into
> > the domain configuration/definition.  Valid states for current 
> > capabilities would be {active, suspended, migrated, inactive}.  On
> 
> Yes, but the problem that occurs is uniquely identifying a domain.  In
> other words, what''s the key used within the persistent store?
> 
> If it''s domain id (which is what I assume it''s going to
be), you cannot
> tag it as having an "inactive" state because there''s
nothing that
> prevents a domain from being created with the same domain id.
> 
> Also, if you try to assign domains UUIDs or something, what do you do
> for cloning/checkpointing?  Do you assign a new UUID on a clone but not
> on a checkpoint?  Does assigning new UUIDs propagate to things like MAC
> addresses or other things that are supposed to be unique?
> 
> There''s a lot to be thought about.  I think punting the problem
(as Andy
> suggests) is the right approach for now.

I believe we should create a single store and API. We can start with the
current domain configuration files. We create a single store and a set
of functions to access it. Then atop that, we can create multiple single
function management apps if you want. You could have an app that just
manages the active domains. You could have an app that does suspend,
etc. Yet you have one store with a common set of commands to access the
information. 

Having this architecture, starting it small even, will help with
answering the harder questions down the road. 

Thanks,

Dan



-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

On Wed, 2005-01-26 at 14:57, Anthony Liguori wrote:
> On Wed, 2005-01-26 at 15:49, Michael Hohnbaum wrote:
> 
> > While beyond the current focus, persistent store could feasibly
> > be used to hold domain definitions for non-existent domains and 
> > suspended domains.  One could envision adding a state field into
> > the domain configuration/definition.  Valid states for current 
> > capabilities would be {active, suspended, migrated, inactive}.  On
> 
> Yes, but the problem that occurs is uniquely identifying a domain.  In
> other words, what''s the key used within the persistent store?
> 
> If it''s domain id (which is what I assume it''s going to
be), you cannot
> tag it as having an "inactive" state because there''s
nothing that
> prevents a domain from being created with the same domain id.
> 
> Also, if you try to assign domains UUIDs or something, what do you do
> for cloning/checkpointing?  Do you assign a new UUID on a clone but not
> on a checkpoint?  Does assigning new UUIDs propagate to things like MAC
> addresses or other things that are supposed to be unique?

I think checkpoints should be stored by the domain id/key for the domain
they are from, using date as an identifier. If you''re going to run that
checkpoint image, it then gets a new id to run under and a new entry as
a domain. 

Sorry about blasting out messages, just thinking about the questions
everyone has raised.

Thanks,

Dan 


> There''s a lot to be thought about.  I think punting the problem
(as Andy
> suggests) is the right approach for now.
> 
> Regards,


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

On Thursday 27 January 2005 00:59, Daniel Stekloff
wrote:
> Can''t we add functionality in Xen to make domain id''s
unique? When
> creating a new domain, the management tools can query the running (or
> even suspended) domains and find a unique domain id to use. I think you
> can tag a domains with their state.
Hey, that is exactly what I was thinking when reading that mail:-) Some new 
states like ''unconfigured'' etc. might need to get added.
> If you clone a domain, it isn''t the same domain anymore and would
get a
> new id.
... so the ID needs to be unique on a per machine basis only: If you clone and 
then migrate a domain to another machine the copy will get a new ID there.
> As for checkpointing - you could use the time/date of the 
> checkpoint and add that to the domain''s id.
Or a a simple counter that gets incremented, that is easier to work with. 
Creation date (and maybe creator) could get stored as some attribute along 
with the configuration, independent of whether the set of attributes is a 
checkpoint or a "normal" domain.
> MAC addresses for specific domains that are in the configuration file or
> generated could be stored with the domain configuration and available
> for queries.
Yeap. Management tools can then query the state and migrate those settings as 
necessary. This approach reduces the amount of policy coded into the core of 
xen. I like it.
> I disagree. If we''re to consider the larger management world, we
need to
> lay the groundwork for managing domains now. I think the questions
> aren''t easily answered, but I believe they should be. If we
don''t
> implement everything to start, we should at least have an idea where
> we''re going.
I have to agree with Dan again:-) Xen can proof that its concepts hold and can 
be implemented. But all the flexibility xen adds is only worth as much as the 
tools provided to manage it.

In my opinion unique DomIds are a good idea anyway: They make configuration 
easier since you can just hardcode DomIds into the config files. Currently 
you have to use domain names in some places and those have to be parsed at 
setup time to figure out which domains are actually referred to. I even have 
to edit some of my config files all the time since the name-hack is not 
supported in all cases (namely backend in the vif-setup). As an additional 
complication there is no requirement on the DomNames to be unique.

PS: This proposed filesystem-based config storage sounds like you could 
leverage the elektra project for data storage. That is a simple lib for 
registry-like storage of key/value pairs. It features few dependencies and 
python bindings:-) The URL is http://elektra.sf.net/.

-- 
Gruss,
Tobias

------------------------------------------------------------
Tobias Hunger           The box said: ''Windows 95 or better''
tobias@aquazul.com                     So I installed Linux.
------------------------------------------------------------

Daniel Stekloff wrote:
>Can''t we add functionality in Xen to make domain id''s
unique? When
>creating a new domain, the management tools can query the running (or
>even suspended) domains and find a unique domain id to use. I think you
>can tag a domains with their state. 
>
>  
>
Yes, I think eventually domains have to have some sort of UUID.  I''m
not
convinced that those can''t be mapped onto the current domain ids by 
configuration tools though.

Here''s a few scenarios to consider:

1) Someone suspends domain A to disk.  Domain A continues to run.  
Without stopping the original instance of Domain A, they start up the 
suspended image.

2) Someone takes the suspended form of Domain A and transfers it to 
another machine.  They start it up while Domain A is still running on 
another machine (creating a resource conflict).  If you''re using DHCP, 
VMware can handle this scenario gracefully btw.

Are these errors?  I think it can be argued either way.
>I disagree. If we''re to consider the larger management world, we
need to
>lay the groundwork for managing domains now. I think the questions
>aren''t easily answered, but I believe they should be. If we
don''t
>implement everything to start, we should at least have an idea where
>we''re going. 
>  
>
I completely agree that we need to lay a groundwork.  I think 
large-scale domain management tools our outside the scope of the current 
focus.  What I''d like to see is an architecture that''s good
enough that
when these large-scale domain management tools are finally worked out 
they can just be plugged in without rewriting the Xen management 
infrastructure.

I think the key to this is to keep things as simple as possible.  Don''t
require configuration files, don''t rely on any sort of internal
database
for persistent state.

BTW, this discussion is great.  I''m very interested to see what others 
think of the broader management picture.

Regards,

-- 
Anthony Liguori
anthony@codemonkey.ws



-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

On Thu, 2005-01-27 at 08:19 -0600, Anthony Liguori
wrote:
> Daniel Stekloff wrote:
> 
> >Can''t we add functionality in Xen to make domain id''s
unique? When
> >creating a new domain, the management tools can query the running (or
> >even suspended) domains and find a unique domain id to use. I think you
> >can tag a domains with their state. 
> >
> >  
> >
> Yes, I think eventually domains have to have some sort of UUID. 
I''m not
> convinced that those can''t be mapped onto the current domain ids
by
> configuration tools though.
> 
> Here''s a few scenarios to consider:
> 
> 1) Someone suspends domain A to disk.  Domain A continues to run.  
> Without stopping the original instance of Domain A, they start up the 
> suspended image.

At some point you need to make rules for operating, deciding based on
how you plan to use it what is the right way and the wrong way. A domain
needs to be unique. It can move through phases like being active or
being suspended. In the case you mention, I don''t think Domain A can be
suspended and active at the same time. It''s one or the other. If you
need to restart from a checkpoint, you clone the Domain A to Domain B.
We could think of it like process management.

> 2) Someone takes the suspended form of Domain A and transfers it to 
> another machine.  They start it up while Domain A is still running on 
> another machine (creating a resource conflict).  If you''re using
DHCP,
> VMware can handle this scenario gracefully btw.
> 
> Are these errors?  I think it can be argued either way.

You''ve raised a good question: are Domain ID''s unique only to
the system
they are running on? Do we let the cluster manager or virtual farm
manager handle Domains per virtual host? How is the process going to be
managed? Can you have multiple Xen systems in the same network without
having a cluster manager? What are the policies for migrating from one
system to another? What if I wish to start a checkpoint of a particular
domain on another system, which could mean resource conflicts? 

I''d like to make sure we can easily reference resources to domains to
hosts, to enable better management - especially on error. 

These are the questions we need to be answering now, even if we don''t
implement it right away. 

> >I disagree. If we''re to consider the larger management world,
we need to
> >lay the groundwork for managing domains now. I think the questions
> >aren''t easily answered, but I believe they should be. If we
don''t
> >implement everything to start, we should at least have an idea where
> >we''re going. 
> >  
> >
> I completely agree that we need to lay a groundwork.  I think 
> large-scale domain management tools our outside the scope of the current 
> focus.  What I''d like to see is an architecture that''s
good enough that
> when these large-scale domain management tools are finally worked out 
> they can just be plugged in without rewriting the Xen management 
> infrastructure.
> 
> I think the key to this is to keep things as simple as possible. 
Don''t
> require configuration files, don''t rely on any sort of internal
database
> for persistent state.

I agree that we can''t implement the world now, we must do it in small
steps. But we should start working on an overall architecture with at
least a stab at some of these questions. We don''t want to be surprised
a
year down the road with a situation that our current architecture cannot
handle. 

We can start small with incremental releases. The first release (from a
management standpoint only) could include the xcs layer with xend using
it. Another incremental release could put the domain configuration file
into a small store that xend uses through a limited API. This second
release could include the logic for unique domain ids. The third release
targets replacing the current Python tools xm/xend with other management
applications that build upon the initial store. As we add functionality
and management routines, we add to our store and what it can do.
Eventually, we enter the cluster management world. 

Overall, we have a strategy for where we want to go and the uses we want
it to have. We just split it up into manageable chunks.

> BTW, this discussion is great.  I''m very interested to see what
others
> think of the broader management picture.

Me too.

Thanks,

Dan



-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

> On Thu, 2005-01-27 at 08:19 -0600, Anthony Liguori wrote:
> > Daniel Stekloff wrote:
> > 
> > >Can''t we add functionality in Xen to make domain
id''s unique? When
> > >creating a new domain, the management tools can query the running
(or
> > >even suspended) domains and find a unique domain id to use. I
think you
> > >can tag a domains with their state. 
> > >
> > >  
> > >
> > Yes, I think eventually domains have to have some sort of UUID. 
I''m not
> > convinced that those can''t be mapped onto the current domain
ids by
> > configuration tools though.
> > 
> > Here''s a few scenarios to consider:
> > 
> > 1) Someone suspends domain A to disk.  Domain A continues to run.  
> > Without stopping the original instance of Domain A, they start up the 
> > suspended image.
> 
> At some point you need to make rules for operating, deciding based on
> how you plan to use it what is the right way and the wrong way. A domain
> needs to be unique. It can move through phases like being active or
> being suspended. In the case you mention, I don''t think Domain A
can be
> suspended and active at the same time. It''s one or the other. If
you
> need to restart from a checkpoint, you clone the Domain A to Domain B.
> We could think of it like process management.
So it may be useful to revisit one of the original distinctions that
we made, that between "domain" and "virtual machine": 

  - A domain is a Xen concept that has an associated domain struct,
    allocation of physical memory, CPU time, event channels, etc. A 
    domain has an ID which is unique to all the domains currently
    existing on a given physical machine. In general, think of
"domain"
    as the analog of "process".

  - A virtual machine is the thing that inhabits a domain. It may 
    inhabit different domains (and hence have different domain IDs) 
    at various points in its life-cycle (e.g. due to suspend/resume, 
    live migration or even reboot). 

Most management tools will want to manage virtual machines as first
class entities, and worry less about domains (all that is required 
is a dynamic mapping from VM to (machine id, domain id)). In the 
current tools (xend etc), VMs are named by system-wide ASCII names. 
Xen knows (and cares) nothing about these. The tools make a basic 
effort to ensure that VM names are unique but currently this is 
only guaranteed per physical machine rather than per cluster; furhermore 
there are some potential race conditions in the current management, 
and some inconveniences (e.g. VM name is part of the saved state 
rather than being outside it) which makes e.g. VM fork a bit tricky. 


> > 2) Someone takes the suspended form of Domain A and transfers it to 
> > another machine.  They start it up while Domain A is still running on 
> > another machine (creating a resource conflict).  If you''re
using DHCP,
> > VMware can handle this scenario gracefully btw.
> > 
> > Are these errors?  I think it can be argued either way.
> 
> 
> You''ve raised a good question: are Domain ID''s unique
only to the system
> they are running on? Do we let the cluster manager or virtual farm
> manager handle Domains per virtual host? How is the process going to be
> managed? Can you have multiple Xen systems in the same network without
> having a cluster manager? What are the policies for migrating from one
> system to another? What if I wish to start a checkpoint of a particular
> domain on another system, which could mean resource conflicts? 
> 
> I''d like to make sure we can easily reference resources to domains
to
> hosts, to enable better management - especially on error. 
> 
> These are the questions we need to be answering now, even if we
don''t
> implement it right away. 
See above: domain IDs are per machine (per Xen instance) things. They 
are low level handles which Xen uses. They are not site wide unique, 
nor are they VM instance unique. 

A sensible management tool strategy would be to allow a set of physical 
nodes to be managed in a coordinated fashion (particularly due to 
live migration, but also due to other forms of location independence, 
e.g. "creating" a VM from a stored (booted) template image). This 
also implies that VM names should be (a) opaque and (b) site-wide unique. 

> > >I disagree. If we''re to consider the larger management
world, we need to
> > >lay the groundwork for managing domains now. I think the questions
> > >aren''t easily answered, but I believe they should be. If
we don''t
> > >implement everything to start, we should at least have an idea
where
> > >we''re going. 
> > >  
> > >
> > I completely agree that we need to lay a groundwork.  I think 
> > large-scale domain management tools our outside the scope of the
current
> > focus.  What I''d like to see is an architecture
that''s good enough that
> > when these large-scale domain management tools are finally worked out 
> > they can just be plugged in without rewriting the Xen management 
> > infrastructure.
> > 
> > I think the key to this is to keep things as simple as possible. 
Don''t
> > require configuration files, don''t rely on any sort of
internal database
> > for persistent state.
> 
> 
> I agree that we can''t implement the world now, we must do it in
small
> steps. But we should start working on an overall architecture with at
> least a stab at some of these questions. We don''t want to be
surprised a
> year down the road with a situation that our current architecture cannot
> handle. 
> 
> We can start small with incremental releases. The first release (from a
> management standpoint only) could include the xcs layer with xend using
> it. Another incremental release could put the domain configuration file
> into a small store that xend uses through a limited API. This second
> release could include the logic for unique domain ids. The third release
> targets replacing the current Python tools xm/xend with other management
> applications that build upon the initial store. As we add functionality
> and management routines, we add to our store and what it can do.
> Eventually, we enter the cluster management world. 
> 
> Overall, we have a strategy for where we want to go and the uses we want
> it to have. We just split it up into manageable chunks.
Right - and one part of that split is separating out "per Xen
instance"
functionality and "per site" functionality. Most of the xcs and libxc
and other low level bits and pieces are in the former camp; the new 
(in unstable) xend is logically in the latter. 



cheers,

S.


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

On Friday 28 January 2005 01:17, Steven Hand wrote:
> > On Thu, 2005-01-27 at 08:19 -0600, Anthony Liguori wrote:
> > > Daniel Stekloff wrote:
> > > >Can''t we add functionality in Xen to make domain
id''s unique? When
> > > >creating a new domain, the management tools can query the
running (or
> > > >even suspended) domains and find a unique domain id to use. I
think
> > > > you can tag a domains with their state.
> > >
> > > Yes, I think eventually domains have to have some sort of UUID. 
I''m
> > > not convinced that those can''t be mapped onto the
current domain ids by
> > > configuration tools though.
> > >
> > > Here''s a few scenarios to consider:
> > >
> > > 1) Someone suspends domain A to disk.  Domain A continues to run.
> > > Without stopping the original instance of Domain A, they start up
the
> > > suspended image.
> >
> > At some point you need to make rules for operating, deciding based on
> > how you plan to use it what is the right way and the wrong way. A
domain
> > needs to be unique. It can move through phases like being active or
> > being suspended. In the case you mention, I don''t think
Domain A can be
> > suspended and active at the same time. It''s one or the other.
If you
> > need to restart from a checkpoint, you clone the Domain A to Domain B.
> > We could think of it like process management.
>
> So it may be useful to revisit one of the original distinctions that
> we made, that between "domain" and "virtual machine":
>
>   - A domain is a Xen concept that has an associated domain struct,
>     allocation of physical memory, CPU time, event channels, etc. A
>     domain has an ID which is unique to all the domains currently
>     existing on a given physical machine. In general, think of
"domain"
>     as the analog of "process".
>
>   - A virtual machine is the thing that inhabits a domain. It may
>     inhabit different domains (and hence have different domain IDs)
>     at various points in its life-cycle (e.g. due to suspend/resume,
>     live migration or even reboot).

Ok, I misunderstood. I thought you''d tie one to the other.

Is there a need for and how easy would it be to  reference a VM to domains 
over time or virtual resources to physical resources? What happens when an 
error occurs with a bit of hardware? Does Xen and Xen tools report it? What 
about the VM that''s running on the system? What if a VM reports an
error
while running in one Domain and then the VM is moved to another Domain with 
different resources? Would we need to store history of VM to machine id and 
domain id? 

Or am I missing something?

Thanks,

Dan


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel