Ext3 users - May 2002 - High load on Squid server after change from reiserfs to ext3

We are running Zope behind Squid 2.4Stable6 with squid in acceleration mode.
The squid box (dual Pentium III 1 GHz, RH 7.2, Linux 2.4.9-21smp, 2GB Ram)
has during busy hours a normal load of 0.2-0.3 . From time to time
we see spikes over some hours where the load average of the machine
is higher than 1.5 although there are no spikes in the CPU
utilization. Also there is no increase in the number of incoming requests.

The only change we made was the switch from reiserfs to ext3 for the cache
partition since reiserfs corrupted its partition some time ago.

After migrating a second box (same hardware) to ext3 this box show the same
odd behaviour.

Both boxes sit behind a loadbalancer and receive the same number of requests
each. There is connection between the spikes on both machine. One machine
idles
almost while the other has a load >2.0 Looking at Squid internal statistics
show that the requests are equally distributed between both servers.
There are also no other services running runing the boxes.

Any insights on this issue?

Thanks,
Andreas



    ---------------------------------------------------------------------
   -    Andreas Jung                            Zope Corporation       -
  -   EMail: andreas@zope.com                http://www.zope.com      -
 -  "Python Powered"                       http://www.python.org     -
  -   "Makers of Zope"                       http://www.zope.org     
-
   -            "Life is too short to (re)write parsers"              
-
    ---------------------------------------------------------------------

On May 24, 2002  13:01 -0400, Andreas Jung wrote:
> We are running Zope behind Squid 2.4Stable6 with squid in acceleration
mode.
> The squid box (dual Pentium III 1 GHz, RH 7.2, Linux 2.4.9-21smp, 2GB Ram)
> has during busy hours a normal load of 0.2-0.3 . From time to time
> we see spikes over some hours where the load average of the machine
> is higher than 1.5 although there are no spikes in the CPU
> utilization. Also there is no increase in the number of incoming requests.
> 
> The only change we made was the switch from reiserfs to ext3 for the cache
> partition since reiserfs corrupted its partition some time ago.
One simple question - does the increase in the load value actually
affect squid performance?  You say there is no increase in CPU
utilization.  That may be good or it may be bad, depending if the
"idle"
CPU is actually waiting on I/O to complete.

What does the output of vmstat on both systems show?  Please send two
traces, one during low load and one during high load.

Cheers, Andreas
--
Andreas Dilger
http://www-mddsp.enel.ucalgary.ca/People/adilger/
http://sourceforge.net/projects/ext2resize/

----- Original Message -----
From: "Andreas Dilger" <adilger@clusterfs.com>
To: "Andreas Jung" <andreas@zope.com>
Cc: <ext3-users@redhat.com>
Sent: Friday, May 24, 2002 13:35
Subject: Re: High load on Squid server after change from reiserfs to ext3

> On May 24, 2002  13:01 -0400, Andreas Jung wrote:
> > We are running Zope behind Squid 2.4Stable6 with squid in acceleration
mode.
> > The squid box (dual Pentium III 1 GHz, RH 7.2, Linux 2.4.9-21smp, 2GB
Ram)
> > has during busy hours a normal load of 0.2-0.3 . From time to time
> > we see spikes over some hours where the load average of the machine
> > is higher than 1.5 although there are no spikes in the CPU
> > utilization. Also there is no increase in the number of incoming
requests.
> >
> > The only change we made was the switch from reiserfs to ext3 for the
cache
> > partition since reiserfs corrupted its partition some time ago.
>
> One simple question - does the increase in the load value actually
> affect squid performance?  You say there is no increase in CPU
> utilization.  That may be good or it may be bad, depending if the
"idle"
> CPU is actually waiting on I/O to complete.
>
> What does the output of vmstat on both systems show?  Please send two
> traces, one during low load and one during high load.
Our big brother diagrams show that the CPU utilization remains *low* all
the time. I have been running vmstat on both machines when one of the
machine
had a spike. But both machines showed more ore less identical vmstat output:
CPU was about 85% idle, no swapping, not much interrupts and context
switches.
In general, vmstat showed a perfect behaviour except that the load average
went up on one box.


Andreas

On May 24, 2002  13:40 -0400, Andreas Jung wrote:
> From: "Andreas Dilger" <adilger@clusterfs.com>
> > One simple question - does the increase in the load value actually
> > affect squid performance?  You say there is no increase in CPU
> > utilization.  That may be good or it may be bad, depending if the
"idle"
> > CPU is actually waiting on I/O to complete.
> >
> > What does the output of vmstat on both systems show?  Please send two
> > traces, one during low load and one during high load.
> 
> Our big brother diagrams show that the CPU utilization remains *low* all
> the time. I have been running vmstat on both machines when one of the
> machine had a spike. But both machines showed more ore less identical
> vmstat output:  CPU was about 85% idle, no swapping, not much interrupts
> and context switches.
> In general, vmstat showed a perfect behaviour except that the load average
> went up on one box.
Then it is nothing to worry about.  The "load average" does not
indicate
anything other than the number of processes runnable or waiting for
disk I/O.  It is likely that when you are doing a large journal flush
the kjournald process is waiting for the disk I/O to complete, so it is
counted 1 towards the load average.  This in itself does not affect the
performance of the system.

Cheers, Andreas
--
Andreas Dilger
http://www-mddsp.enel.ucalgary.ca/People/adilger/
http://sourceforge.net/projects/ext2resize/