CentOS - Jun 2011 - Possible to use multiple disk to bypass I/O wait?

I'm trying to resolve an I/O problem on a CentOS 5.6 server. The
process basically scans through Maildirs, checking for space usage and
quota. Because there are hundred odd user folders and several 10s of
thousands of small files, this sends the I/O wait % way high. The
server hits a very high load level and stops responding to other
requests until the crawl is done.

I am wondering if I add another disk and symlink the sub-directories
to that, would that free up the server to respond to other requests
despite the wait on that disk?

Alternatively, if I mdraid mirror the existing disk, would md be smart
enough to read using the other disk while the first's tied up with the
first process?

On 06/09/2011 02:24 AM, Emmanuel Noobadmin wrote:
> I'm trying to resolve an I/O problem on a CentOS 5.6 server. The
> process basically scans through Maildirs, checking for space usage and
> quota. Because there are hundred odd user folders and several 10s of
> thousands of small files, this sends the I/O wait % way high. The
> server hits a very high load level and stops responding to other
> requests until the crawl is done.
>
> I am wondering if I add another disk and symlink the sub-directories
> to that, would that free up the server to respond to other requests
> despite the wait on that disk?
>
> Alternatively, if I mdraid mirror the existing disk, would md be smart
> enough to read using the other disk while the first's tied up with the
> first process?
You should look at running your process using 'ionice -c3 program'. That
way it won't starve everything else for I/O cycles. Also, you may want 
to experiment with using the 'deadline' elevator instead of the default 
'cfq' (see http://www.redhat.com/magazine/008jun05/features/schedulers/ 
and http://www.wlug.org.nz/LinuxIoScheduler). Neither of those would 
require you to change your hardware out. Also, setting 'noatime' for the
mount options for partition holding the files will reduce the number of 
required I/Os quite a lot.

But yes, in general, distributing your load across more disks should 
improve your I/O profile.

-- 
Benjamin Franz

On 6/9/11, Mathias Bur?n <mathias.buren at gmail.com>
wrote:
> The first thing that comes to my mind: Have you tried another IO scheduler?
and the first thing that came to this noob's mind was: Wait, you mean
there's actually more than one? AND I get to choose?

I'll probably be experimenting with deadline and anticipatory since
the i/o wait seems to be due to the disk running back and fro trying
to serve the file scan as well as legit read request so having that
small wait for reads in the same area sounds like it would help.

On 06/09/11 2:24 AM, Emmanuel Noobadmin wrote:
> Alternatively, if I mdraid mirror the existing disk, would md be smart
> enough to read using the other disk while the first's tied up with the
> first process?
that woudl be my first choice, and yes, queued read IO could be 
satisfied by either mirror, hence they'd have double the read performance.

next step would be a raid 1+0 with yet more disks.

-- 
john r pierce                            N 37, W 122
santa cruz ca                         mid-left coast

On Thu, 9 Jun 2011, Emmanuel Noobadmin wrote:
> I'm trying to resolve an I/O problem on a CentOS 5.6 server. The
> process basically scans through Maildirs, checking for space usage and
> quota. Because there are hundred odd user folders and several 10s of
> thousands of small files, this sends the I/O wait % way high. The
> server hits a very high load level and stops responding to other
> requests until the crawl is done.
If the server is reduced to a crawl, it's possible that you are hitting 
the dirty_ratio limit due to writes and the server has entered synchronous 
I/O mode. As others have mentioned, setting noatime could have a 
significant effect, especially if there are many files and the server 
doesn't have much memory. You can try increasing dirty_ratio to see if it 
has an effect, eg:

 	# sysctl vm.dirty_ratio
 	# sysctl -w vm.dirty_ratio=50

Steve