For those who have wondered why NFS on EL5 is slower than on EL4 I
provide these links for your edification.
http://kbase.redhat.com/faq/docs/DOC-15355
http://bugzilla.redhat.com/show_bug.cgi?id=448130
Problem is kernel threads cannot create or assign an io context as
there is no api in the kernel available to do so, so each is given a
different context and there is an 8ms latency between switching
contexts. The knowledge base article recommends disabling this latency
to get around it, but in doing that you might as well just use the
deadline scheduler.
Some other interesting tidbits for those using NFS.
NFS with many nfsd threads will create a lot of file system fragments
when writing large files.
XFS does have the advantage of a defragger (I hope some smart person
will develop one for ext2/3/4), but I am still looking for a better
way to protect against fragmentation then running a defragger, so if
anyone has an alternative I'm all ears.
Another hint for those using NFS with ESX, make sure your vmdk
partitions are 4k aligned or there is a serious performance penalty as
2 reads for each write will happen if the io request stradles a page
boundary. This is more serious for random io than sequential, but both
are effected.
To give an example, my NFS storage with write-back cache was only able
to do 1MB/s (4k direct io 4 outstanding) on a non-aligned partition,
on an aligned partition I was able to get 13MB/s with the same
workload. That's 13x improvement.
For Linux you can use fdisk/sfdisk to start your first partition on
sector 64 instead of the default 63. On Windows use diskpart to create
a partition that is aligned on a given offset.
-Ross