Xen users - Oct 2010 - High iowait every day at the same time.

Hi,

I have a strange and the same problem on two servers. I''m running Xen
on two
different servers (2.6.18.8-xen Debian 4.0 and 2.6.18-xen Debian 4.0). About
120 guest domains.
Every day exactly at 08:25 AM, high iowait ~99% load starts and lasts for
about 50 minutes.

When guests domains are shut down, the problem is not appearing. There is no
tasks in crontab on Domain0 and on guest domains.

I tried to figure out the process that is responsible for this iowat but the
only thing I found is that kjournald appears in a "D" state from time
to
time (during that ~1 hour problematic period).

Any help will be appreciated.

Thanks,
Michael.


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Michael Borokhovich wrote:
>I have a strange and the same problem on two servers. I''m running 
>Xen on two different servers (2.6.18.8-xen Debian 4.0 and 2.6.18-xen 
>Debian 4.0). About 120 guest domains.
>Every day exactly at 08:25 AM, high iowait ~99% load starts and 
>lasts for about 50 minutes.
>
>When guests domains are shut down, the problem is not appearing. 
>There is no tasks in crontab on Domain0 and on guest domains.
By default, Debian runs a bunch of tasks at 6:25, have a look in 
/etc/cron.* and /etc/crontab. I see you are in a +2 timezone, so that 
would make 6:25 into 8:25 by my reckoning.

If you have a lot of guests all firing up at the same time, then that 
could well saturate your I/O and make it all slow down.

-- 
Simon Hobson

Visit http://www.magpiesnestpublishing.co.uk/ for books by acclaimed
author Gladys Hobson. Novels - poetry - short stories - ideal as
Christmas stocking fillers. Some available as e-books.

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> -----Original Message-----
> From: xen-users-bounces@lists.xensource.com [mailto:xen-users-
> bounces@lists.xensource.com] On Behalf Of Simon Hobson
> Sent: Friday, October 01, 2010 3:18 AM
> To: xen-users@lists.xensource.com
> Subject: Re: [Xen-users] High iowait every day at the same time.
> 
> Michael Borokhovich wrote:
> 
> >I have a strange and the same problem on two servers. I''m
running Xen
> >on two different servers (2.6.18.8-xen Debian 4.0 and 2.6.18-xen
Debian
> >4.0). About 120 guest domains.
> >Every day exactly at 08:25 AM, high iowait ~99% load starts and lasts
> >for about 50 minutes.
> >
> >When guests domains are shut down, the problem is not appearing.
> >There is no tasks in crontab on Domain0 and on guest domains.
> 
> By default, Debian runs a bunch of tasks at 6:25, have a look in
> /etc/cron.* and /etc/crontab. I see you are in a +2 timezone, so that
would make 6:25
> into 8:25 by my reckoning.
We don''t run Debian but have had to make similar adjustments.  On
CentOS
for instance, remove the "mlocate" package.

Log rotation still runs daily on our domU hosts--I''ve considered
staggering the times so they don''t all run at 4 AM.

(Seriously, is there a Linux OS designed for virtualization.  No matter
which I''ve started with, I end up stripping them down to the
essentials.
I''ve been meaning to take a look at Amazon''s Linux AMI for EC2
to see
how it measures up.)

-Jeff



_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Simon, thank you very much.

Since all my VMs are running Debian, there was the cron.daily schedule at
8:25 in /etc/crontab file. The problematic process was
''updatedb'' which was
initiated by the ''find'' script in the cron.daily directory.

I will comment this line in /etc/crontab and it hopefully will solve the
problem.

Thanks again,
Michael.

On Fri, Oct 1, 2010 at 09:18, Simon Hobson <linux@thehobsons.co.uk> wrote:
> Michael Borokhovich wrote:
>
>  I have a strange and the same problem on two servers. I''m running
Xen on
>> two different servers (2.6.18.8-xen Debian 4.0 and 2.6.18-xen Debian
4.0).
>> About 120 guest domains.
>> Every day exactly at 08:25 AM, high iowait ~99% load starts and lasts
for
>> about 50 minutes.
>>
>> When guests domains are shut down, the problem is not appearing. There
is
>> no tasks in crontab on Domain0 and on guest domains.
>>
>
> By default, Debian runs a bunch of tasks at 6:25, have a look in
> /etc/cron.* and /etc/crontab. I see you are in a +2 timezone, so that would
> make 6:25 into 8:25 by my reckoning.
>
> If you have a lot of guests all firing up at the same time, then that could
> well saturate your I/O and make it all slow down.
>
> --
> Simon Hobson
>
> Visit http://www.magpiesnestpublishing.co.uk/ for books by acclaimed
> author Gladys Hobson. Novels - poetry - short stories - ideal as
> Christmas stocking fillers. Some available as e-books.
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users
>

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users