I have a heavily loaded Apache 2 server which is experiencing what
appear to be "write storms". The server is a dual Xeon box with
hyperthreading, and appears to linux as a four-cpu box. It has 4GB of
physical RAM and is never hitting swap.
It's serving a large number of static image files off of four 135GB SCSI
drives with external journals on a fifth volume. The journal volume is
an identical disk to the other four, split into four equal journal
volumes. The data volumes are mounted noatime,data=journal. This
configuration was chosen because users are uploading images to the
server and I thought that putting writes on an external volume would
improve read efficiency.
I'm running kernel 2.4.21 with Andrea Arcangeli's rc8aa1 patch set
(mostly for the elevator-lowlatency patches which fix a problem with
huge amounts of cpu being used by kswapd). This patch set also includes
the akpm jbd_refile_buffer patch to transaction.c.
The symptoms are that the system chugs along for hours until the load
suddenly shoots through the ceiling (500+) as the buffer utilization
drops to nearly zero, rendering the box mostly unresponsive for several
minutes. What I suspect is happening is that my huge journals are
filling up without being flushed to disk, and then ext3 is flushing the
entire journal.
My current bdflush settings are
40 500 0 0 60 300 60 0 0
as recommended by others on this list and elsewhere. It doesn't seem to
help.
I'm a little stumped at the moment. Any hints would be greatly appreciated.
--
Kevin Murphy
kevin@eorbit.net
