CentOS - Dec 2007 - weird load values

Hi List,

I'm stumped by this:

	load average: 10.65, 594.71, 526.58

We're monitoring load every ~3 minutes. It'll be fine (i.e. something  
like load average: 2.14, 1.27, 1.03), and then in a single sample,  
jump to something like the above. This seems to happen once a week or  
so on a few different servers (all running in a similar application).  
I've never seen the 1 minute sample spike as high as the 5 or 15  
minute samples.

Seeing as that last value is a 15 minute period, well, it doesn't seem  
possible that one can have a 500+ 15 minute sample without having  
observed a spike in the 5 minute sample at least 5 minutes before.

Also, there aren't 500+ processes on these systems -- it's typically  
around 100 total processes (ps auxw | wc -l). (Is there a way to see  
the total count of kernel-level threads?)

Thoughts?

best,
Jeff

Linux someHostName 2.6.18-8.1.8.el5 #1 SMP Tue Jul 10 06:39:17 EDT  
2007 x86_64 x86_64 x86_64 GNU/Linux

CentOS release 5 (Final)

  09:31:15 up 65 days, 17:45,  2 users,  load average: 0.92, 200.91,  
371.30

IO load takes IO wait into it's calculation, so it may be a writer process
hammering the disks which are under powered for the IO load.

Run an iostat under these situations to find the disks causing the high waits.

-Ross


-----Original Message-----
From: centos-bounces at centos.org <centos-bounces at centos.org>
To: CentOS mailing list <centos at centos.org>
Sent: Wed Dec 05 09:39:41 2007
Subject: [CentOS] weird load values


Hi List,

I'm stumped by this:

	load average: 10.65, 594.71, 526.58

We're monitoring load every ~3 minutes. It'll be fine (i.e. something  
like load average: 2.14, 1.27, 1.03), and then in a single sample,  
jump to something like the above. This seems to happen once a week or  
so on a few different servers (all running in a similar application).  
I've never seen the 1 minute sample spike as high as the 5 or 15  
minute samples.

Seeing as that last value is a 15 minute period, well, it doesn't seem  
possible that one can have a 500+ 15 minute sample without having  
observed a spike in the 5 minute sample at least 5 minutes before.

Also, there aren't 500+ processes on these systems -- it's typically  
around 100 total processes (ps auxw | wc -l). (Is there a way to see  
the total count of kernel-level threads?)

Thoughts?

best,
Jeff

Linux someHostName 2.6.18-8.1.8.el5 #1 SMP Tue Jul 10 06:39:17 EDT  
2007 x86_64 x86_64 x86_64 GNU/Linux

CentOS release 5 (Final)

  09:31:15 up 65 days, 17:45,  2 users,  load average: 0.92, 200.91,  
371.30

_______________________________________________
CentOS mailing list
CentOS at centos.org
http://lists.centos.org/mailman/listinfo/centos

______________________________________________________________________
This e-mail, and any attachments thereto, is intended only for use by
the addressee(s) named herein and may contain legally privileged
and/or confidential information. If you are not the intended recipient
of this e-mail, you are hereby notified that any dissemination,
distribution or copying of this e-mail, and any attachments thereto,
is strictly prohibited. If you have received this e-mail in error,
please immediately notify the sender and permanently delete the
original and any copy or printout thereof.

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.centos.org/pipermail/centos/attachments/20071205/355fe92d/attachment-0003.html>

Wednesday 05 December 2007 15:39:41 J. Potter
napisa?(a):
> Hi List,
>
> I'm stumped by this:
>
> 	load average: 10.65, 594.71, 526.58
>
> We're monitoring load every ~3 minutes. It'll be fine (i.e.
something
> like load average: 2.14, 1.27, 1.03), and then in a single sample,
> jump to something like the above. This seems to happen once a week or
> so on a few different servers (all running in a similar application).
> I've never seen the 1 minute sample spike as high as the 5 or 15
> minute samples.
>
> Seeing as that last value is a 15 minute period, well, it doesn't seem
> possible that one can have a 500+ 15 minute sample without having
> observed a spike in the 5 minute sample at least 5 minutes before.
>
> Also, there aren't 500+ processes on these systems -- it's
typically
> around 100 total processes (ps auxw | wc -l). (Is there a way to see
> the total count of kernel-level threads?)
>
> Thoughts?
As mentioned before, IO could give such strange results. I suggest launching 
dstat with logging to a file, and analyzing the file afterwards.

-- 
Tomasz Napierala
System Administrator
QXL Poland - Allegro.pl Team
http://www.allegro.pl/