Ext3 users - Jan 2003 - ext3 instability in RH8.0?

We've been noticing some problems with ext3 and, in our
case, postfix, that appear to come to light under heavy
disk i/o loading.  A previous thread found here
(http://www.redhat.com/mailing-lists/ext3-users/msg04189.html)
documents an identical set of symptons (and similar setup),
but with RH 7.2

The specifics are this:

   1. Stock install of RH8.0 with postfix.
   2. Linux mx 2.4.18-14 #1 date i686 i686 i386 GNU/Linux
   3. Email list delivery to ~40,000 recipients, each with
      their own customized msg. (In other words, these are
      40K unique messages going out, not one msg to
      40K unique recipients).
   4. Postfix parameters (processes) and system resources
      such that no file swapping occurs. (1 Gig processor,
      80 Gig 7200 RPM drives, 1 Gig RAM)

7 distinct processes on another system start firing off as
fast as they can email to this system which then deilivers
it as necessary to other mx servers.

The results are interesting:

1. Very quickly while running "vmstat", one will see decent
    "bo" activity corresponding to the acceptance of the
    various messages, and context switches in the neighbourhood
    of 30-40,000 per second.

2. After about 10-15 seconds, "bo" activity drops to ZERO, and
    context switching ramps up to about 1,000,000 per second.
    After a while, the context switches drop back down, and we're
    back to seeing heavy "bo" activity.

3. After cycling through the behaviour in step 2 several times,
    (after 3-4K msgs are delivered), the context switching remains
    high, and "bo" activity remains at zero.

4. Examining the system with "top" shows a huge run queue - all
    smtp processes are running full tilt.

This above set of conditions were repeated numerous times after
reboots.

Finally, taking the heavily used disk partition (/var) and
setting it to ext2 and rebooting and resuming mail delivery,
absolutely NO problems occur.

To summarize: the mail configuration works in an old RH6.2
setup, in RH8.0 with ext2, but makes the system unusable
with RH8.0 with ext3.

On numerous other non-heavily loaded systems, we've never
had problems with ext3 (all RH8.0 base).

Cheers, Thomas

* Thomas Reinke <reinke@e-softinc.com>:
> We've been noticing some problems with ext3 and, in our
> case, postfix, that appear to come to light under heavy
> disk i/o loading.  A previous thread found here
> (http://www.redhat.com/mailing-lists/ext3-users/msg04189.html)
> documents an identical set of symptons (and similar setup),
> but with RH 7.2
Have you read:
http://www.stahl.bau.tu-bs.de/~hildeb/postfix/ext3.shtml
?

-- 
Ralf Hildebrandt (Im Auftrag des Referat V a)   Ralf.Hildebrandt@charite.de
Charite Campus Mitte                            Tel.  +49 (0)30-450 570-155
Referat V a - Kommunikationsnetze -             Fax.  +49 (0)30-450 570-916
"C++: Hard to learn and built to stay that way."

Hi,

On Thu, 2003-01-02 at 01:00, Thomas Reinke wrote:
> 2. After about 10-15 seconds, "bo" activity drops to ZERO, and
>     context switching ramps up to about 1,000,000 per second.
>     After a while, the context switches drop back down, and we're
>     back to seeing heavy "bo" activity.
That's definitely "wrong".  I'm wondering if there's an
O(1) scheduler
interaction at play here, as postfix's synchronous IO patterns trigger
yielding behaviour which changed significantly in O(1).

Could you possibly either capture a kernel profile of this situation, or
use alt-sysrq-r to capture a few samples of what the CPU is up to when
the problem occurs?

Thanks,
 Stephen