freebsd stable - Feb 2006 - Odd performance problems after upgrade from 4.11 to 6.0-Stable

> Kevin Oberman wrote:
> I recently upgraded my last desktop system from 4.11-Stable to
> 6.0-Stable. I did an update to 5.3 then to RELENG_5,
> RELENG_6_0_0_RELEASE and on to RELENG6_0.
>
> This system has been updated regularly from the days of at least 4.1.
> The hardware is a 1GHz PIII with an ICH2 chipset. 256 MB of memory. 
>
> After the upgrade, the system performs poorly. Specifically, it seems to
> be blocking on I/O. If I have a dump or bsdtar running, a CPU intensive
> job produced minimal CPU utilization and little progress. The system is
> 70% or more idle all of the time. X is particularly sluggish. Many times
> the system will simply appear to be locked up for seconds at a time like
> all of the I/O is synchronous and the entire system is blocking on GIANT
> until the disk write finishes. A level 0 dump of the system drive piped
> through gzip and over the network took about 10 hours. Now it is 17
> hours. My network link only ran at 3.4 Mbps (yes, that's bits, not
> Bytes) over a 100 Mbps connection to the storage server. The rate was
> much higher under 4.11.
>
> My attempts to track down the source of the problem simply failed. I
> have examined my kernel configuration and found nothing out of the
> ordinary. In fact, the the configuration varies from GENERIC only in
> devices (both added and deleted), dropping 486 and 586 CPU code and a
> larger SC_HISTORY. I have nothing in sysctl.conf and only enable DMA on
> my ATAPI devices in loader.conf. I have confirmed that the disks are
> running ATA100.
>
> Any suggestions on where to look? At this point I am baffled.
Well, I have tracked the problem down a bit. It happens when the system
runs out of memory and starts to use the swap file. The glimpse indexing
operation is very memory intensive and under V6 it simply crawls when it
starts paging. On my old 256 MB system, this operation was taking over 6
hours. On my new system (1 GB), it takes about 7 minutes. 

This is not really a surprise. I know how running out of memory slows
things down. What does bother me is why it went from 6 hours in V4 14
hours on V6.

Unfortunately, my old box is going away in two weeks and I doubt that
I'll have much time to look at things before then. All I can say is
that either the wired memory for V6 is up so much that I page and swap a
lot mote or that the operations have slowed down way more than they
should have.
-- 
R. Kevin Oberman, Network Engineer
Energy Sciences Network (ESnet)
Ernest O. Lawrence Berkeley National Laboratory (Berkeley Lab)
E-mail: oberman@es.net			Phone: +1 510 486-8634

On Sun, 2006-Feb-12 18:27:47 -0800, Kevin Oberman wrote:
>This is not really a surprise. I know how running out of memory slows
>things down. What does bother me is why it went from 6 hours in V4 14
>hours on V6.
It could be that glimpse's memory access pattern makes it very
sensitive to the available RAM and a slight reduction is causing a
massive slowdown.
>Unfortunately, my old box is going away in two weeks and I doubt that
>I'll have much time to look at things before then. All I can say is
>that either the wired memory for V6 is up so much that I page and swap a
>lot mote or that the operations have slowed down way more than they
>should have.
Whilst you have access to the box, it might be useful to capture some
of the VM subsystem information for both 4.x and 6.x (eg a couple of
'sysctl vm' at known intervals) under stress.  This might help
identify where the problem is.

-- 
Peter Jeremy

On Sun, Feb 12, 2006 at 06:27:47PM -0800, Kevin Oberman wrote:
> Well, I have tracked the problem down a bit. It happens when the system
> runs out of memory and starts to use the swap file. The glimpse indexing
> operation is very memory intensive and under V6 it simply crawls when it
> starts paging. On my old 256 MB system, this operation was taking over 6
> hours. On my new system (1 GB), it takes about 7 minutes. 
Can you try increasing the vm.swap_async_max sysctl to 32 and see if
that helps with the speed of paging?

Kris
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 187 bytes
Desc: not available
Url :
http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20060216/8f0b5246/attachment.bin