Hi Folks,
I know it is kind of vague problem, but I am banking of somebody seeing the
exact thing.
I have two machines with everything identical, except on one machine I compile
xen and on other I just use the binaries compiled on the previous machine.
Now on first machine, where I compile everything, xm invocation is veeeeeeeeeery
slow. Although it doesn''t seem to be "system" problem. Look
at a
time xm list
invocation:
real 32s
user .2s
sys .02s
On another machine the real time is very close to user+sys.
I just can''t comprehend why this is going on. Any clues will be deeply
appreciated.
-Himanshu
--
-------------------------------------------------------------------------
Himanshu Raj
PhD Student, GaTech (www.cc.gatech.edu/~rhim)
I prefer to receive attachments in an open, non-proprietary format.
-------------------------------------------------------------------------
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
On Wed, Jan 25, 2006 at 09:19:12AM -0500, Himanshu Raj wrote:> Hi Folks, > > I know it is kind of vague problem, but I am banking of somebody seeing the > exact thing. > > I have two machines with everything identical, except on one machine I compile > xen and on other I just use the binaries compiled on the previous machine. > > Now on first machine, where I compile everything, xm invocation is veeeeeeeeeery > slow. Although it doesn''t seem to be "system" problem. Look at a > time xm list > invocation: > real 32s > user .2s > sys .02s > > On another machine the real time is very close to user+sys. > > I just can''t comprehend why this is going on. Any clues will be deeply appreciated.xm is a client to Xend, talking over HTTP -- xm is simply blocking for 30 seconds waiting for the server -- you need to figure out why the server is spinning or stalling. /var/log/xend.log may help. top may help also. Ewan. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Thanks Ewan. I also saw one more discrepency which might be the root of this problem. When I do xend start on first machine, 8 processes start up (i.e. ps aux | grep xend shows actually 8 proceeses). On my other machine, I see only 2 processes. Do you think it is the problem? -Himanshu> > I know it is kind of vague problem, but I am banking of somebody seeing the > > exact thing. > > > > I have two machines with everything identical, except on one machine I compile > > xen and on other I just use the binaries compiled on the previous machine. > > > > Now on first machine, where I compile everything, xm invocation is veeeeeeeeeery > > slow. Although it doesn''t seem to be "system" problem. Look at a > > time xm list > > invocation: > > real 32s > > user .2s > > sys .02s > > > > On another machine the real time is very close to user+sys. > > > > I just can''t comprehend why this is going on. Any clues will be deeply appreciated. > > xm is a client to Xend, talking over HTTP -- xm is simply blocking for 30 > seconds waiting for the server -- you need to figure out why the server is > spinning or stalling. > > /var/log/xend.log may help. top may help also. > > Ewan.-- ------------------------------------------------------------------------- Himanshu Raj PhD Student, GaTech (www.cc.gatech.edu/~rhim) I prefer to receive attachments in an open, non-proprietary format. ------------------------------------------------------------------------- _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
top shows heavy activity by xenstored for a long time (like 30-40% CPU
utilization).
Also there is a huge gap between xenstore write and devcontroller to kick in.
Excerpt from xend.log follows:
[2006-01-25 19:22:48 xend.XendDomainInfo] DEBUG (XendDomainInfo:688) Storing dom
ain details: {''console/ring-ref'': ''124478'',
''console/port'': ''2'',
''name'': ''vm1'',
''console/limit'': ''1048576'',
''vm'':
''/vm/c1b17d02-ed25-f67e-2bef-0bbed31d765f'', ''d
omid'': ''1'', ''cpu/0/availability'':
''online'', ''memory/target'':
''65536'', ''store/rin
g-ref'': ''124479'', ''store/port'':
''1''}
[2006-01-25 19:23:00 xend] DEBUG (DevController:132) Waiting for devices vif.
[2006-01-25 19:23:00 xend] DEBUG (DevController:132) Waiting for devices usb.
[2006-01-25 19:23:01 xend] DEBUG (DevController:132) Waiting for devices vbd.
Does it ring any bells?
Best regards,
Himanshu
On Wed, Jan 25, 2006 at 05:06:36PM -0500, Himanshu Raj
wrote:> Thanks Ewan.
>
> I also saw one more discrepency which might be the root of this problem.
> When I do xend start on first machine, 8 processes start up (i.e. ps aux |
grep
> xend shows actually 8 proceeses).
> On my other machine, I see only 2 processes. Do you think it is the
problem?
>
> -Himanshu
>
> > > I know it is kind of vague problem, but I am banking of somebody
seeing the
> > > exact thing.
> > >
> > > I have two machines with everything identical, except on one
machine I compile
> > > xen and on other I just use the binaries compiled on the previous
machine.
> > >
> > > Now on first machine, where I compile everything, xm invocation
is veeeeeeeeeery
> > > slow. Although it doesn''t seem to be "system"
problem. Look at a
> > > time xm list
> > > invocation:
> > > real 32s
> > > user .2s
> > > sys .02s
> > >
> > > On another machine the real time is very close to user+sys.
> > >
> > > I just can''t comprehend why this is going on. Any clues
will be deeply appreciated.
> >
> > xm is a client to Xend, talking over HTTP -- xm is simply blocking for
30
> > seconds waiting for the server -- you need to figure out why the
server is
> > spinning or stalling.
> >
> > /var/log/xend.log may help. top may help also.
> >
> > Ewan.
>
> --
> -------------------------------------------------------------------------
> Himanshu Raj
> PhD Student, GaTech (www.cc.gatech.edu/~rhim)
> I prefer to receive attachments in an open, non-proprietary format.
> -------------------------------------------------------------------------
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
--
-------------------------------------------------------------------------
Himanshu Raj
PhD Student, GaTech (www.cc.gatech.edu/~rhim)
I prefer to receive attachments in an open, non-proprietary format.
-------------------------------------------------------------------------
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
I finally figured it out. The tdb database used by xenstored got out of control - it had become ~300MB, and that''s why xenstored was taking a lot of time accessing it. After cleaning it up, things are back to normal. Thanks and Best Regards, Himanshu On Wed, Jan 25, 2006 at 07:37:04PM -0500, Himanshu Raj wrote:> top shows heavy activity by xenstored for a long time (like 30-40% CPU utilization). > Also there is a huge gap between xenstore write and devcontroller to kick in. > Excerpt from xend.log follows: > > [2006-01-25 19:22:48 xend.XendDomainInfo] DEBUG (XendDomainInfo:688) Storing dom > ain details: {''console/ring-ref'': ''124478'', ''console/port'': ''2'', ''name'': ''vm1'', > ''console/limit'': ''1048576'', ''vm'': ''/vm/c1b17d02-ed25-f67e-2bef-0bbed31d765f'', ''d > omid'': ''1'', ''cpu/0/availability'': ''online'', ''memory/target'': ''65536'', ''store/rin > g-ref'': ''124479'', ''store/port'': ''1''} > [2006-01-25 19:23:00 xend] DEBUG (DevController:132) Waiting for devices vif. > [2006-01-25 19:23:00 xend] DEBUG (DevController:132) Waiting for devices usb. > [2006-01-25 19:23:01 xend] DEBUG (DevController:132) Waiting for devices vbd. > > Does it ring any bells? > > Best regards, > Himanshu > > On Wed, Jan 25, 2006 at 05:06:36PM -0500, Himanshu Raj wrote: > > Thanks Ewan. > > > > I also saw one more discrepency which might be the root of this problem. > > When I do xend start on first machine, 8 processes start up (i.e. ps aux | grep > > xend shows actually 8 proceeses). > > On my other machine, I see only 2 processes. Do you think it is the problem? > > > > -Himanshu > > > > > > I know it is kind of vague problem, but I am banking of somebody seeing the > > > > exact thing. > > > > > > > > I have two machines with everything identical, except on one machine I compile > > > > xen and on other I just use the binaries compiled on the previous machine. > > > > > > > > Now on first machine, where I compile everything, xm invocation is veeeeeeeeeery > > > > slow. Although it doesn''t seem to be "system" problem. Look at a > > > > time xm list > > > > invocation: > > > > real 32s > > > > user .2s > > > > sys .02s > > > > > > > > On another machine the real time is very close to user+sys. > > > > > > > > I just can''t comprehend why this is going on. Any clues will be deeply appreciated. > > > > > > xm is a client to Xend, talking over HTTP -- xm is simply blocking for 30 > > > seconds waiting for the server -- you need to figure out why the server is > > > spinning or stalling. > > > > > > /var/log/xend.log may help. top may help also. > > > > > > Ewan. > > > > -- > > ------------------------------------------------------------------------- > > Himanshu Raj > > PhD Student, GaTech (www.cc.gatech.edu/~rhim) > > I prefer to receive attachments in an open, non-proprietary format. > > ------------------------------------------------------------------------- > > > > _______________________________________________ > > Xen-devel mailing list > > Xen-devel@lists.xensource.com > > http://lists.xensource.com/xen-devel > > -- > ------------------------------------------------------------------------- > Himanshu Raj > PhD Student, GaTech (www.cc.gatech.edu/~rhim) > I prefer to receive attachments in an open, non-proprietary format. > ------------------------------------------------------------------------- > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel-- ------------------------------------------------------------------------- Himanshu Raj PhD Student, GaTech (www.cc.gatech.edu/~rhim) I prefer to receive attachments in an open, non-proprietary format. ------------------------------------------------------------------------- _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel