Xen users - Jul 2006 - when dom0 loadavg above 1, domUs not available

I have a xen 3.0.2 running on a 733 MHz server, 1 GB RAM.
It has two domU domains.
dom0 has 512 MB, domUs have 256 MB each.

Whenever load average on dom0 is above 1-2 (for example, compressing 200 
MB file), I can''t reach any domU domain.

Ping replies to domUs take very long time (it''s on LAN), and there are 
packet losses:

64 bytes from 192.168.11.61: icmp_seq=7 ttl=64 time=4140 ms
64 bytes from 192.168.11.61: icmp_seq=8 ttl=64 time=39548 ms


When I ping dom0, it replies immediately, there are no packet losses:

64 bytes from 192.168.111.173: icmp_seq=1 ttl=64 time=4.07 ms
64 bytes from 192.168.111.173: icmp_seq=2 ttl=64 time=0.406 ms


It''s problematic to get to the domU with "xm console domain":
it takes
really long to get the login prompt, and even if I get it, it times out 
when waiting for the password.

There are no problems logging to dom0.


I found a similar problem here, but it didn''t have any conclusions:

http://lists.xensource.com/archives/html/xen-users/2005-12/msg00728.html


Is it a known problem, and if so, is there a remedy for this?

It makes things a bit unreliable when one system makes some job, and 
other can''t be accessed because of it.


-- 
Tomasz Chmielewski
http://syneticon.net

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Mon, 2006-07-03 at 12:45 +0200, Tomasz Chmielewski
wrote:
> I have a xen 3.0.2 running on a 733 MHz server, 1 GB RAM.
That''s part of the problem, but should work much better than it does. 
> It has two domU domains.
> dom0 has 512 MB, domUs have 256 MB each.
> 
Have you considered shrinking dom-0 to 128 MB and not using it for
anything but managing the dom-u''s? I''m guessing
you''re also :

1 - Using file backed VBD''s
2 - Using legacy IDE/EIDE drives

Your problem is I/O congestion due to a very slow CPU and slow drives. I
wouldn''t use dom-0 to do anything, make yourself a dom-u to use. 
> Whenever load average on dom0 is above 1-2 (for example, compressing 200 
> MB file), I can''t reach any domU domain.
> 
See above related to I/O. You''re choking the dom-u''s from
accessing
their VBD''s by doing that on dom-0. You could try setting scheduling
appropriately and see if that makes a difference, however even if you
used a dom-u to do that kind of work, I/O is going to be a problem on
that kind of legacy system.
> Ping replies to domUs take very long time (it''s on LAN), and there
are
> packet losses:
> 
> 64 bytes from 192.168.11.61: icmp_seq=7 ttl=64 time=4140 ms
> 64 bytes from 192.168.11.61: icmp_seq=8 ttl=64 time=39548 ms
> 
Yes, i''d imagine so. :) The dom-u''s can''t access
their own file system
because the disk is backed up tarring up your 200 mb file on dom-0 - and
dom-0 isn''t scheduling cpu access because its very busy waiting for
your
drives to write data.
> 
> When I ping dom0, it replies immediately, there are no packet losses:
> 
> 64 bytes from 192.168.111.173: icmp_seq=1 ttl=64 time=4.07 ms
> 64 bytes from 192.168.111.173: icmp_seq=2 ttl=64 time=0.406 ms
> 
> 
> It''s problematic to get to the domU with "xm console
domain": it takes
> really long to get the login prompt, and even if I get it, it times out 
> when waiting for the password.
> 
See above about choking the dom-u from access to their FS.
> There are no problems logging to dom0.
> 
> 
> I found a similar problem here, but it didn''t have any
conclusions:
> 
> http://lists.xensource.com/archives/html/xen-users/2005-12/msg00728.html
> 
> 
> Is it a known problem, and if so, is there a remedy for this?
> 
Use slightly better hardware and stop doing intense stuff on dom-0.
> It makes things a bit unreliable when one system makes some job, and 
> other can''t be accessed because of it.
> 
> 
However, your CPU cooks eggs beautifully! That''s not a bug its a
feature! <kidding>

Try re-arranging things a bit, using partitions instead of file backed
VBD''s and then try fine tuning cpu scheduling and priority based on
what
you want each dom-u doing. Only run services that are absolutely
necessary on dom-0, and try to avoid i/o intense workouts on dom-0
itself.

HTH
Tim



_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

I also forgot to mention, on older hardware like that (and newer too)
there is some fine tuning you can do on the dom-u''s themselves. 

If you''re running out-of-the-box Fedora, RH, CentOS / etc
you''ll find
most of your daemons are running with a priority of 0 (see man nice for
more). 

Since everything is running at the same priority, its going to cause
some congestion. Try making use of a tool like spri.
http://www.rfxnetworks.com/spri.php , and you may be able to squeeze a
little more out of the dom-u''s themselves. 

I wouldn''t recommend spri on dom-0. 

Hope this helps :)
Tim

On Tue, 2006-07-04 at 05:44 +0800, Tim Post wrote:
> On Mon, 2006-07-03 at 12:45 +0200, Tomasz Chmielewski wrote:
> > I have a xen 3.0.2 running on a 733 MHz server, 1 GB RAM.
> 
> That''s part of the problem, but should work much better than it
does.
> 
> > It has two domU domains.
> > dom0 has 512 MB, domUs have 256 MB each.
> > 
> 
> Have you considered shrinking dom-0 to 128 MB and not using it for
> anything but managing the dom-u''s? I''m guessing
you''re also :
> 
> 1 - Using file backed VBD''s
> 2 - Using legacy IDE/EIDE drives
> 
> Your problem is I/O congestion due to a very slow CPU and slow drives. I
> wouldn''t use dom-0 to do anything, make yourself a dom-u to use. 
> 
> > Whenever load average on dom0 is above 1-2 (for example, compressing
200
> > MB file), I can''t reach any domU domain.
> > 
> 
> See above related to I/O. You''re choking the dom-u''s from
accessing
> their VBD''s by doing that on dom-0. You could try setting
scheduling
> appropriately and see if that makes a difference, however even if you
> used a dom-u to do that kind of work, I/O is going to be a problem on
> that kind of legacy system.
> 
> > Ping replies to domUs take very long time (it''s on LAN), and
there are
> > packet losses:
> > 
> > 64 bytes from 192.168.11.61: icmp_seq=7 ttl=64 time=4140 ms
> > 64 bytes from 192.168.11.61: icmp_seq=8 ttl=64 time=39548 ms
> > 
> 
> Yes, i''d imagine so. :) The dom-u''s can''t access
their own file system
> because the disk is backed up tarring up your 200 mb file on dom-0 - and
> dom-0 isn''t scheduling cpu access because its very busy waiting
for your
> drives to write data.
> 
> > 
> > When I ping dom0, it replies immediately, there are no packet losses:
> > 
> > 64 bytes from 192.168.111.173: icmp_seq=1 ttl=64 time=4.07 ms
> > 64 bytes from 192.168.111.173: icmp_seq=2 ttl=64 time=0.406 ms
> > 
> > 
> > It''s problematic to get to the domU with "xm console
domain": it takes
> > really long to get the login prompt, and even if I get it, it times
out
> > when waiting for the password.
> > 
> 
> See above about choking the dom-u from access to their FS.
> 
> > There are no problems logging to dom0.
> > 
> > 
> > I found a similar problem here, but it didn''t have any
conclusions:
> > 
> >
http://lists.xensource.com/archives/html/xen-users/2005-12/msg00728.html
> > 
> > 
> > Is it a known problem, and if so, is there a remedy for this?
> > 
> 
> Use slightly better hardware and stop doing intense stuff on dom-0.
> 
> > It makes things a bit unreliable when one system makes some job, and 
> > other can''t be accessed because of it.
> > 
> > 
> 
> However, your CPU cooks eggs beautifully! That''s not a bug its a
> feature! <kidding>
> 
> Try re-arranging things a bit, using partitions instead of file backed
> VBD''s and then try fine tuning cpu scheduling and priority based
on what
> you want each dom-u doing. Only run services that are absolutely
> necessary on dom-0, and try to avoid i/o intense workouts on dom-0
> itself.
> 
> HTH
> Tim
> 
> 
> 
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users
> 
> 

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Tim Post wrote:

(...)
> However, your CPU cooks eggs beautifully! That''s not a bug its a
> feature! <kidding>
> 
> Try re-arranging things a bit, using partitions instead of file backed
> VBD''s and then try fine tuning cpu scheduling and priority based
on what
> you want each dom-u doing. Only run services that are absolutely
> necessary on dom-0, and try to avoid i/o intense workouts on dom-0
> itself.
Thanks.
I was just evaluating Xen on an old machine, and was worried about the 
performance on the real system.

What you said seem to explain everything.


-- 
Tomasz Chmielewski
http://syneticon.net




_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users