Xen devel - Oct 2006 - xend / xenstored performance & scalability issues

I''ve been trying to track down just why talking to XenD is resulting in
so
much CPU time being comsumed by both xend & xenstored. As a test case,
I''m
running ''virsh dominfo demo''  which results in a single HTTP
request to
Xend to fetch domain info, eg ''GET /xend/domains/demo''

Run this in a tight loop & I''ll see xenstored taking > 50% CPU,
and
XenD taking another 11%

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 2647 root      16   0  6188  840  464 R   52  0.0   0:55.04 xenstored
11600 root      18   0  259m 7568 1516 S   11  0.2   0:04.53 python

Its not surprising that xend is consuming time since we are making many 
requests per second, but for an operation which is only doing reads it
having so much time attributed to xenstored seems very excessive. So I
ran oprofile & collected some data about xenstored:

CPU: AMD64 processors, speed 2211.33 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Cycles outside of halt state) with a unit mask
of 0x00 (No unit mask) count 100000
samples  %        image name               symbol name
347226   45.9445  ext3                     (no symbols)
264664   35.0200  jbd                      (no symbols)
31778     4.2048  libc-2.5.so              memset
10763     1.4241  xenstored                main
8884      1.1755  libc-2.5.so              _int_malloc
7053      0.9332  libc-2.5.so              vfprintf
4264      0.5642  xenstored                initialize_set


So almost 80% of xenstored''s CPU time is attributed to ext3 &
journalling
modules, suggesting xenstored is doing alot of disk I/O. strace()''ing 
the xenstored process shows the only file it is opening is:

# strace -p 2647 -e trace=open,rename,unlink
Process 2647 attached - interrupt to quit
open("/var/lib/xenstored/tdb.0x62aa80", O_WRONLY|O_CREAT|O_TRUNC,
0640) = 13
open("/var/lib/xenstored/tdb.0x62aa80", O_RDWR) = 15
rename("/var/lib/xenstored/tdb.0x62aa80",
"/var/lib/xenstored/tdb") = 0
unlink("/var/lib/xenstored/tdb.0x62aa80") = -1 ENOENT (No such file or
directory)
open("/var/lib/xenstored/tdb.0x62b2b0", O_WRONLY|O_CREAT|O_TRUNC,
0640) = 13
open("/var/lib/xenstored/tdb.0x62b2b0", O_RDWR) = 14
rename("/var/lib/xenstored/tdb.0x62b2b0",
"/var/lib/xenstored/tdb") = 0
unlink("/var/lib/xenstored/tdb.0x62b2b0") = -1 ENOENT (No such file or
directory)
...

So basically it is repeatedly copying its persistent TBD database over and
over again. The TDB on this system is 128 KB in size and each individual HTTP
GET on /xend/domain/demo  is resulting in 16 copies being made. 

Do the maths - 128 * 16 == 2 MB of reads, and 2 MB of writes - for a single
*read* in XenD. Now if I monitor the status of 20 domains, once per second
that''s causing 40 MB of writes & 40 MB of reads every second which
is utterly
ridiculous & completely non scalable for enterprise deployment :-(

There''s two problems I see here:

  1. Why the need for xenstored to be doing any of this I/O in the first place?
     If the DB needs to be kept on disk at all, it really needs to have a much
     saner update/transactional model to only update bits which actually change,
     rather than re-creating the entire DB on every transaction.
     But it strikes me that the DB could potentially be kept entirely in memory
     removing the disk I/O completely. Sure yyou wouldn''t be able to
restart
     the daemon then, but even today you can''t restart xenstored &
expect things
     to still be working.

  2. Why does XenD create sooo many transactions in XenStored for a read op ?
     Having instrumented Xend it sems that the root cause of the problem is the
     xen.xend.xenstore.xstransact  class. This alllows one to start a
transaction,
     do a bunch of reads/writes & then commit the transaction. At the same
time
     though it has a bunch of static ''convenience'' methods for
read & write which
     will implicitly start & commit a transaction. Well 90% of the code in
XenD
     seems to be using these ''convenience'' methods instead of
explicitly starting
     a transaction to cover a piece of work - the result is a simple GET causes
     16 transactions ....and an ''xm create'' results in 80
transactions. These
     convenience methods are utterly destroying performance.

Clearly we can''t address these for 3.0.3, but I think both of these
areas need
serious work in 3.0.4 if we want a scalable control plane in Dom0. Fixing
the XenD bit looks particularly hard because any single method using the
convenience xenstored read functions can be called under many different
contexts, so of which needs transactions, others which don''t. It ought
to
be possible to trace back all the calls & make it possible to pass explicit
xstransct objects into all calls & then kill off the convenience methods.

Regards,
Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496 -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=| 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 12/10/06 15:16, "Daniel P. Berrange" <berrange@redhat.com>
wrote:
> Clearly we can''t address these for 3.0.3, but I think both of
these areas need
> serious work in 3.0.4 if we want a scalable control plane in Dom0. Fixing
> the XenD bit looks particularly hard because any single method using the
> convenience xenstored read functions can be called under many different
> contexts, so of which needs transactions, others which don''t. It
ought to
> be possible to trace back all the calls & make it possible to pass
explicit
> xstransct objects into all calls & then kill off the convenience
methods.
Yes, xenstored is very simple minded in many respects. We will certainly be
improving this during 3.0.4 development -- I think we can get the costs down
very significantly for commonplace operations without enormous effort.

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 12/10/06 15:16, "Daniel P. Berrange" <berrange@redhat.com>
wrote:
>   1. Why the need for xenstored to be doing any of this I/O in the first
> place?
>      If the DB needs to be kept on disk at all, it really needs to have a
much
>      saner update/transactional model to only update bits which actually
> change,
>      rather than re-creating the entire DB on every transaction.
>      But it strikes me that the DB could potentially be kept entirely in
> memory
>      removing the disk I/O completely. Sure yyou wouldn''t be able
to restart
>      the daemon then, but even today you can''t restart xenstored
& expect
> things
>      to still be working.
We plan to keep transactional state in memory, and commit to tdb only on
transaction completion. In which case you''ll be able to achieve what
you
describe above by keeping the tdb file in a ramdisk. I suspect that''ll
turn
out to be unnecessary and once we keep a persistent handle on a single tdb
file and only update what has changed, we''ll find the buffer cache will
work
quite nicely.

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Thu, Oct 12, 2006 at 03:33:28PM +0100, Keir Fraser
wrote:
> On 12/10/06 15:16, "Daniel P. Berrange"
<berrange@redhat.com> wrote:
> 
> >   1. Why the need for xenstored to be doing any of this I/O in the
first
> > place?
> >      If the DB needs to be kept on disk at all, it really needs to
have a much
> >      saner update/transactional model to only update bits which
actually
> > change,
> >      rather than re-creating the entire DB on every transaction.
> >      But it strikes me that the DB could potentially be kept entirely
in
> > memory
> >      removing the disk I/O completely. Sure yyou wouldn''t be
able to restart
> >      the daemon then, but even today you can''t restart
xenstored & expect
> > things
> >      to still be working.
> 
> We plan to keep transactional state in memory, and commit to tdb only on
> transaction completion. In which case you''ll be able to achieve
what you
> describe above by keeping the tdb file in a ramdisk. I suspect
that''ll turn
> out to be unnecessary and once we keep a persistent handle on a single tdb
> file and only update what has changed, we''ll find the buffer cache
will work
> quite nicely.
Agreed,that sounds like it ought to be able to significantly reduce overhead
without neccessarily needing to go to a purely memory backed storage.

Regards,
Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496 -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=| 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Thu, Oct 12, 2006 at 03:16:38PM +0100, Daniel P. Berrange wrote:
> I''ve been trying to track down just why talking to XenD is
resulting in so
> much CPU time being comsumed by both xend & xenstored. As a test case,
I''m
> running ''virsh dominfo demo''  which results in a single
HTTP request to
> Xend to fetch domain info, eg ''GET /xend/domains/demo''
>
> [Snip]
>
>   2. Why does XenD create sooo many transactions in XenStored for a read op
?
>      Having instrumented Xend it sems that the root cause of the problem is
the
>      xen.xend.xenstore.xstransact  class. This alllows one to start a
transaction,
>      do a bunch of reads/writes & then commit the transaction. At the
same time
>      though it has a bunch of static ''convenience''
methods for read & write which
>      will implicitly start & commit a transaction. Well 90% of the code
in XenD
>      seems to be using these ''convenience'' methods
instead of explicitly starting
>      a transaction to cover a piece of work - the result is a simple GET
causes
>      16 transactions ....and an ''xm create'' results in 80
transactions. These
>      convenience methods are utterly destroying performance.
> 
> Clearly we can''t address these for 3.0.3, but I think both of
these areas need
> serious work in 3.0.4 if we want a scalable control plane in Dom0. Fixing
> the XenD bit looks particularly hard because any single method using the
> convenience xenstored read functions can be called under many different
> contexts, so of which needs transactions, others which don''t. It
ought to
> be possible to trace back all the calls & make it possible to pass
explicit
> xstransct objects into all calls & then kill off the convenience
methods.
Yes, we''ve already committed to fixing this in Xen 3.0.4.  The Xen-API
preview
tree already has this work underway.

Ewan.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel