CentOS - Dec 2006 - Good value for /proc/sys/vm/min_free_kbytes

Hi,

 what would be a good value for

/proc/sys/vm/min_free_kbytes

 on a Dual-CPU EM64T System with 8 GB of memory. Kernel is
2.6.9-42.0.3.ELsmp. I know that the default might be a bit low.
Unfortunatelly the documentation is a bit weak in this area.

 We are experiencing responsiveness problems (and higher than expected
load) when the system is under combined memory+network+disk-IO stress.

Thanks
Martin
Please CC me on replies, as I am only getting the digest.

------------------------------------------------------
Martin Knoblauch
email: k n o b i AT knobisoft DOT de
www:   http://www.knobisoft.de

Hi,

On Thu, 7 Dec 2006 08:03:54 -0800 (PST)
Martin Knoblauch <spamtrap at knobisoft.de> wrote:
>  what would be a good value for
> 
> /proc/sys/vm/min_free_kbytes
> 
>  on a Dual-CPU EM64T System with 8 GB of memory. Kernel is
> 2.6.9-42.0.3.ELsmp. I know that the default might be a bit low.
> Unfortunatelly the documentation is a bit weak in this area.
> 
>  We are experiencing responsiveness problems (and higher than expected
> load) when the system is under combined memory+network+disk-IO stress.
Before changing this tunable, it is important to realize what it is
for. The free area is primarily useful to give the kernel enough memory
to act in situations where the swapper suddenly has to do a lot of work.
In other situations, you can make things worse, since you are
effectively giving less memory for applications. The following thread
may be interesting in this respect:

http://lists.centos.org/pipermail/centos/2006-August/thread.html#68306

Use the usual suspects to nail down the bottleneck. It may well be the
case that the system has too little disk bandwidth, too little CPU
power, or too little memory to handle its load. But there's no way to
tell without more information. Touch vm_min_free bytes, if analysis
proves that e.g. kswapd has trouble to page out memory pages in a
timely fashion. A good value to start with is 4096KB if it is higher
than your current vm_min_free bytes, though I have seen others setting
it much higher.

-- Daniel

Martin Knoblauch wrote:
>  We are experiencing responsiveness problems (and higher than expected
> load) when the system is under combined memory+network+disk-IO stress.
>   
                                                         ^^^^^^^^^^^^^^^

First, I'd check the paging with `vmstat 5` ... if you see excessive SI 
(swap in/second), you need more physical memory, no amount of dinking 
with vm parameters can change this.

If you're not seeing excessive paging, I'd be inclined to monitor the 
disk IO with `iostat -x 5`...   if the avgqu-sz and/or await on any 
device is high, you need to balance your disk IO across more physical 
devices and/or more channels.   await = 500 means disk physical IO 
requests are taking an average of 500mS (0.5 seconds) to satisfy.    If 
many processes are waiting for disk IO, you'll see high load factors 
even though CPU usage is fairly low.

iostat is in yum package systat (not installed by default in most 
configs), vmstat is in procps (generally installed by default).   on 
both of these commands, ignore the first output, thats the system 
average since reboot, generally meaningless.  the 2nd and successive 
outputs are at the intervals specified (5 seconds in my above examples).

On our database servers, which experience very high disk IO loads, we 
often use 4 separate RAIDs...  / and the other normal system volumes are 
partitions on a raid1 (typically 2 x 36GB 15k scsi or sas), then the 
database itself will be spread across 3 volumes /u10 /u11 /u12, which 
are each RAID 1+0 built from 4 x 72GB 15k scsi/sas or FC SAN volumes.   
We'll always use RAID controllers with hardware battery-protected raid 
write-back cache for the database volumes, as this hugely accelerates 
'commits'.    Note, we don't use mysql, I have no idea if its
capable of
taking advantage of configurations like this, but postgresql and oracle 
certainly are.    The database adminstrators will spend hours pouring 
over IO logs and database statistics in order to better optimize the 
distribution of tables and indicies across the available tablespaces.

Under these sorts of heavy concurrent random access patterns, SATA and 
software RAID just don't cut it, regardless of how good its sequential 
benchmarks may be.
> Please CC me on replies, as I am only getting the digest.
   spamtrap at knobisoft.de  ??!?   no thanks.

Quoting Martin Knoblauch <spamtrap at knobisoft.de>:
> Hi,
>
>  what would be a good value for
>
> /proc/sys/vm/min_free_kbytes
>
>  on a Dual-CPU EM64T System with 8 GB of memory. Kernel is
> 2.6.9-42.0.3.ELsmp. I know that the default might be a bit low.
> Unfortunatelly the documentation is a bit weak in this area.
>
>  We are experiencing responsiveness problems (and higher than expected
> load) when the system is under combined memory+network+disk-IO stress.
Actually, under similar load, I experinced ext3 file system corruption  
(RHEL4/CentOS4 kernels).  In my case it was memory+cpu+network.  All I  
was doing was running a simple Perl script.  For details check out:

https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=202955

Appernetly, Red Hat considers increasing default value for  
min_free_kbytes in update 5.

Increasing it to 8192 seems to do good job in preventing Linux kernel  
to deny memory to itself (and destroying my valuable data).  Well, at  
least in my case.