I''m waiting about 2 seconds from typing ''xm list'' and for something to appear. Is this normal? The machine is fairly old, it''s a Compaq Proliant 1600 with a single CPU running at 350Mhz. It''s hardware RAID but that is pretty slow too. James ------------------------------------------------------- This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170 Project Admins to receive an Apple iPod Mini FREE for your judgement on who ports your project to Linux PPC the best. Sponsored by IBM. Deadline: Sept. 13. Go here: http://sf.net/ppc_contest.php _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
> I''m waiting about 2 seconds from typing ''xm list'' and for something to > appear. Is this normal? The machine is fairly old, it''s a Compaq > Proliant 1600 with a single CPU running at 350Mhz. It''s hardware RAID > but that is pretty slow too.If you look at the output of ''xm list'' does the CPU time consumed by dom0 go up by 2s every time? Is dom0 competing for CPU0 with a bunch of other busy domains? On our 2.4GHz Xeon machines, I''ve noticed that running ''xm list'' causes the amount of CPU consumed by dom0 to go up by around 100ms, which is quite a staggering amount given what it''s actually doing. Python isn''t the most efficient of languages, and we are doing a HTTP RPC between xm and xend, but this level of resource usage has always rather surprised me. I guess we could use one of the python profiling tools to find out where the overhead is. Since this is all ''control plane'' we haven''t made any effort to make it efficient and have concentrated on flexibility. It it starts becoming a bottleneck it may be necessary to selectively recode bits in C. Ian ------------------------------------------------------- This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170 Project Admins to receive an Apple iPod Mini FREE for your judgement on who ports your project to Linux PPC the best. Sponsored by IBM. Deadline: Sept. 13. Go here: http://sf.net/ppc_contest.php _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
To do a very simple test I created a little script that does ''time xm list'' 6 times. Here''s the output: Name Id Mem(MB) CPU State Time(s) Console Domain-0 0 123 0 r---- 198.7 gaia2 1 0 0 -b--- 59.8 9601 real 0m9.403s user 0m1.030s sys 0m0.080s Name Id Mem(MB) CPU State Time(s) Console Domain-0 0 123 0 r---- 200.0 gaia2 1 0 0 -b--- 59.8 9601 real 0m9.997s user 0m0.980s sys 0m0.130s Name Id Mem(MB) CPU State Time(s) Console Domain-0 0 123 0 r---- 201.4 gaia2 1 0 0 -b--- 59.8 9601 real 0m9.999s user 0m0.990s sys 0m0.140s Name Id Mem(MB) CPU State Time(s) Console Domain-0 0 123 0 r---- 202.7 gaia2 1 0 0 -b--- 59.8 9601 real 0m9.999s user 0m1.040s sys 0m0.090s Name Id Mem(MB) CPU State Time(s) Console Domain-0 0 123 0 r---- 204.0 gaia2 1 0 0 -b--- 59.8 9601 real 0m9.998s user 0m0.990s sys 0m0.120s Name Id Mem(MB) CPU State Time(s) Console Domain-0 0 123 0 r---- 205.3 gaia2 1 0 0 -b--- 59.8 9601 real 0m9.998s user 0m1.040s sys 0m0.100s It''s closer to 10 seconds as you can see. After I type ''xm list'', it takes about 5 seconds for the heading to appear, and then roughly 2 seconds per domain. Load under Domain-0 is almost 0. Another Proliant 1600 I have that is running xen has 2x550Mhz CPU''s and runs much better... roughly 3 seconds from typing ''xm list'' to getting output, and all the output rolls past pretty quick (4 domains on that one). The domain gaia2 is actually destroyed in the output above, it just doesn''t know it. Testing iscsi and it went into a spin logging bulk stuff to the console. I find killing a domain in that state doesn''t work properly and I have to reboot Dom0 to get xend functioning properly again. James> -----Original Message----- > From: Ian Pratt [mailto:Ian.Pratt@cl.cam.ac.uk] > Sent: Friday, 10 September 2004 22:55 > To: James Harper > Cc: xen-devel@lists.sourceforge.net; Ian.Pratt@cl.cam.ac.uk > Subject: Re: [Xen-devel] xm speed > > > I''m waiting about 2 seconds from typing ''xm list'' and for somethingto> > appear. Is this normal? The machine is fairly old, it''s a Compaq > > Proliant 1600 with a single CPU running at 350Mhz. It''s hardwareRAID> > but that is pretty slow too. > > If you look at the output of ''xm list'' does the CPU time consumed > by dom0 go up by 2s every time? > > Is dom0 competing for CPU0 with a bunch of other busy domains? > > On our 2.4GHz Xeon machines, I''ve noticed that running ''xm list'' > causes the amount of CPU consumed by dom0 to go up by around > 100ms, which is quite a staggering amount given what it''s > actually doing. Python isn''t the most efficient of languages, and > we are doing a HTTP RPC between xm and xend, but this level of > resource usage has always rather surprised me. > > I guess we could use one of the python profiling tools to find > out where the overhead is. Since this is all ''control plane'' we > haven''t made any effort to make it efficient and have > concentrated on flexibility. It it starts becoming a bottleneck > it may be necessary to selectively recode bits in C. > > Ian > >------------------------------------------------------- This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170 Project Admins to receive an Apple iPod Mini FREE for your judgement on who ports your project to Linux PPC the best. Sponsored by IBM. Deadline: Sept. 13. Go here: http://sf.net/ppc_contest.php _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
May I make a suggestion? I''ve worked with daemons of my own in the past and found it very usefull and faster for third party development to use a light XML parser library to handle the control interface parsing. I''d think that going with XML structures instead of flat text/html/binary would make it easy to self document the interface and command/response data structures (standard XML argument). I wouldn''t even bother with HTTP at all. Just accept an XML structure that describes the task, or list of tasks that need to be done. then return a simple xml structure with results and error codes so we know what commands suceeded and which one it failed at. Could also tag commands as continue on fail for batches such as gathering stats on each domain. This would make it far easier for poeple to write their own control interfaces and just interact with xend (meaning, I''d have an easier time writing a controller process for the cluster of xen boxes vs rewriting xend ;) ). If you like this idea I''d be VERY interested in helping to work on it. Won''t matter to me if it''s interpreted language like perl/python/ruby/etc or if it was in C/C++. Either works for me (just not too experienced in python.) This is the approach that I was taking with the replacement xend I started for xen 1.2 (which I''ve abandoned until 2.0 is released, and it will be restarted from scratch). A master and slave master node daemons that talked to the xen node daemons via the network to track each xen servers vhosts/loads/etc, sort of like a cluster manager. I do believe that we should attempt to make it as easy as possible to hook into the xen system either at the libxc or xend levels. Thanks for listening! Brian On Fri, 2004-09-10 at 07:55, Ian Pratt wrote:> > I''m waiting about 2 seconds from typing ''xm list'' and for something to > > appear. Is this normal? The machine is fairly old, it''s a Compaq > > Proliant 1600 with a single CPU running at 350Mhz. It''s hardware RAID > > but that is pretty slow too. > > If you look at the output of ''xm list'' does the CPU time consumed > by dom0 go up by 2s every time? > > Is dom0 competing for CPU0 with a bunch of other busy domains? > > On our 2.4GHz Xeon machines, I''ve noticed that running ''xm list'' > causes the amount of CPU consumed by dom0 to go up by around > 100ms, which is quite a staggering amount given what it''s > actually doing. Python isn''t the most efficient of languages, and > we are doing a HTTP RPC between xm and xend, but this level of > resource usage has always rather surprised me. > > I guess we could use one of the python profiling tools to find > out where the overhead is. Since this is all ''control plane'' we > haven''t made any effort to make it efficient and have > concentrated on flexibility. It it starts becoming a bottleneck > it may be necessary to selectively recode bits in C. > > Ian > > > > > ------------------------------------------------------- > This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170 > Project Admins to receive an Apple iPod Mini FREE for your judgement on > who ports your project to Linux PPC the best. Sponsored by IBM. > Deadline: Sept. 13. Go here: http://sf.net/ppc_contest.php > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/xen-devel------------------------------------------------------- This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170 Project Admins to receive an Apple iPod Mini FREE for your judgement on who ports your project to Linux PPC the best. Sponsored by IBM. Deadline: Sept. 13. Go here: http://sf.net/ppc_contest.php _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
> To do a very simple test I created a little script that does ''time xm > list'' 6 times. Here''s the output:So, real time is ~10s, total CPU consumed by dom 0 is ~1.4s. (That''s terrible!) Is it like this even when you haven''t got a domain in a bad state? Of the 1.4s of CPU, consumed, 1.2s seems to being accounted to ''xm'', so we can assume that ''xend'' is consuming the other 0.2. What''s odd about this is that _xend_ is the guy that''s actually doing most of the work -- xm just does an RPC to it. Anyhow, it might be more informative to figure out where the difference between real time and CPU time is coming from. Presumably xm/xend is blocking on IO. I wander if running an strace on xm and/or xend might be informative. BTW: how are you accessing your root file system (local disk, NFS) etc? Ian ------------------------------------------------------- This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170 Project Admins to receive an Apple iPod Mini FREE for your judgement on who ports your project to Linux PPC the best. Sponsored by IBM. Deadline: Sept. 13. Go here: http://sf.net/ppc_contest.php _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
> Is it like this even when you haven''t got a domain in a bad > state?I think so, although I''ve never actually timed it before.> Of the 1.4s of CPU, consumed, 1.2s seems to being accounted to > ''xm'', so we can assume that ''xend'' is consuming the other > 0.2. What''s odd about this is that _xend_ is the guy that''s > actually doing most of the work -- xm just does an RPC to it. > > Anyhow, it might be more informative to figure out where the > difference between real time and CPU time is coming > from. Presumably xm/xend is blocking on IO. I wander if running > an strace on xm and/or xend might be informative. > > BTW: how are you accessing your root file system (local disk, > NFS) etc?Root filesystem in dom0 is a local raid array, albeit a very slow one. The disk appears to do a fair bit of chugging when I run it, although I''m not normally near the box. After a checkout and rebuild just now, it appears to be much faster! Name Id Mem(MB) CPU State Time(s) Console Domain-0 0 123 0 r---- 81.8 gaia2 1 126 0 -b--- 22.0 9601 real 0m2.254s user 0m1.100s sys 0m0.090s Name Id Mem(MB) CPU State Time(s) Console Domain-0 0 123 0 r---- 83.9 gaia2 1 126 0 -b--- 22.1 9601 real 0m2.544s user 0m1.110s sys 0m0.100s Name Id Mem(MB) CPU State Time(s) Console Domain-0 0 123 0 r---- 86.5 gaia2 1 126 0 -b--- 22.2 9601 real 0m2.465s user 0m1.060s sys 0m0.140s Name Id Mem(MB) CPU State Time(s) Console Domain-0 0 123 0 r---- 88.6 gaia2 1 126 0 -b--- 22.3 9601 real 0m2.529s user 0m1.060s sys 0m0.150s Name Id Mem(MB) CPU State Time(s) Console Domain-0 0 123 0 r---- 91.1 gaia2 1 126 0 -b--- 22.4 9601 real 0m2.371s user 0m1.080s sys 0m0.130s Name Id Mem(MB) CPU State Time(s) Console Domain-0 0 123 0 r---- 93.3 gaia2 1 126 0 -b--- 22.5 9601 real 0m2.326s user 0m1.130s sys 0m0.130s 2 seconds is still a bit slow considering what it is actually doing, but better than 10. James ------------------------------------------------------- This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170 Project Admins to receive an Apple iPod Mini FREE for your judgement on who ports your project to Linux PPC the best. Sponsored by IBM. Deadline: Sept. 13. Go here: http://sf.net/ppc_contest.php _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
James Harper wrote:>>Is it like this even when you haven''t got a domain in a bad >>state? > > > I think so, although I''ve never actually timed it before. > > >>Of the 1.4s of CPU, consumed, 1.2s seems to being accounted to >>''xm'', so we can assume that ''xend'' is consuming the other >>0.2. What''s odd about this is that _xend_ is the guy that''s >>actually doing most of the work -- xm just does an RPC to it. >> >>Anyhow, it might be more informative to figure out where the >>difference between real time and CPU time is coming >>from. Presumably xm/xend is blocking on IO. I wander if running >>an strace on xm and/or xend might be informative. >> >>BTW: how are you accessing your root file system (local disk, >>NFS) etc? > > > Root filesystem in dom0 is a local raid array, albeit a very slow one. > The disk appears to do a fair bit of chugging when I run it, although > I''m not normally near the box.This could be your problem - xm is going to load quite a few python classes to make the call.> > After a checkout and rebuild just now, it appears to be much faster! > > Name Id Mem(MB) CPU State Time(s) Console > Domain-0 0 123 0 r---- 81.8 > gaia2 1 126 0 -b--- 22.0 9601 > > real 0m2.254s > user 0m1.100s > sys 0m0.090s > Name Id Mem(MB) CPU State Time(s) Console > Domain-0 0 123 0 r---- 83.9 > gaia2 1 126 0 -b--- 22.1 9601 > > real 0m2.544s > user 0m1.110s > sys 0m0.100s > Name Id Mem(MB) CPU State Time(s) Console > Domain-0 0 123 0 r---- 86.5 > gaia2 1 126 0 -b--- 22.2 9601 > > real 0m2.465s > user 0m1.060s > sys 0m0.140s > Name Id Mem(MB) CPU State Time(s) Console > Domain-0 0 123 0 r---- 88.6 > gaia2 1 126 0 -b--- 22.3 9601 > > real 0m2.529s > user 0m1.060s > sys 0m0.150s > Name Id Mem(MB) CPU State Time(s) Console > Domain-0 0 123 0 r---- 91.1 > gaia2 1 126 0 -b--- 22.4 9601 > > real 0m2.371s > user 0m1.080s > sys 0m0.130s > Name Id Mem(MB) CPU State Time(s) Console > Domain-0 0 123 0 r---- 93.3 > gaia2 1 126 0 -b--- 22.5 9601 > > real 0m2.326s > user 0m1.130s > sys 0m0.130s > > 2 seconds is still a bit slow considering what it is actually doing, but > better than 10. > > James > >Anyway, here are some more data points. I ran xm list and compared it with wget to see what the difference was. I only show the times once, but they were consistently repoducible. Timing of ''xm list'': mjw@wray-m-4 1028> time xm list Name Id Mem(MB) CPU State Time(s) Console Domain-0 0 123 0 r---- 13.1 real 0m0.178s user 0m0.160s sys 0m0.000s Timing of getting python to print hello: mjw@wray-m-4 1034> time python -c ''print "Hello"'' Hello real 0m0.016s user 0m0.010s sys 0m0.000s So xm''s real work takes about 0.16 s (160ms). For a comparison let''s try getting the domain list with wget: mjw@wray-m-4 1009> echo $http_proxy http://15.144.59.5:8088 mjw@wray-m-4 1010> unset http_proxy mjw@wray-m-4 1014> time wget -q http://localhost:8000/xend/domain/ real 0m0.013s user 0m0.010s sys 0m0.000s This takes 0.013s (13ms) so the time is spent in xm not xend. mjw@wray-m-4 1042> uname -a Linux wray-m-4.hpl.hp.com 2.4.27-xen0 #1 Tue Sep 14 12:00:35 BST 2004 i686 i686 i386 GNU/Linux mjw@wray-m-4 1043> cat /proc/cpuinfo processor : 0 vendor_id : GenuineIntel cpu family : 15 model : 2 model name : Intel(R) Xeon(TM) CPU 2.80GHz stepping : 9 cpu MHz : 2791.039 cache size : 512 KB fdiv_bug : no hlt_bug : no f00f_bug : no coma_bug : no fpu : yes fpu_exception : yes cpuid level : 2 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe cid bogomips : 5570.56 Mike Wray ------------------------------------------------------- This SF.Net email is sponsored by: YOU BE THE JUDGE. Be one of 170 Project Admins to receive an Apple iPod Mini FREE for your judgement on who ports your project to Linux PPC the best. Sponsored by IBM. Deadline: Sept. 24. Go here: http://sf.net/ppc_contest.php _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel