Jim Perrin
2010-Jan-25 13:27 UTC
[CentOS] Too much cpu wait on nfs server when we need to read data on it
On Mon, Jan 25, 2010 at 8:39 AM, fabien faye <fabien at faye.eu> wrote:> Hi, > > I have a big server with 24 Disk on 2 3ware card.Define 'big'. You're not giving much in terms of memory or cpu specs.> When i write data on my nfs server everything is fine but when i want to read data i have a lot of cpu wait.Okay. Have you looked at any performance tuning or elevator/scheduler adjustments? File system tuning? What have you tried on your own so far to debug this?> about the file system, i use ext4 on LVM partition.Why? Is there a performance reason for you to use ext4 vs ext3?> Do you have any idea about that.Not based on the little information you've provided. It would help for you to supply more detail, and the things you've already looked at. -- During times of universal deceit, telling the truth becomes a revolutionary act. George Orwell
fabien faye
2010-Jan-25 13:39 UTC
[CentOS] Too much cpu wait on nfs server when we need to read data on it
Hi, I have a big server with 24 Disk on 2 3ware card. When i write data on my nfs server everything is fine but when i want to read data i have a lot of cpu wait. [root at NFS /]# vmstat 1 procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------ r b swpd free buff cache si so bi bo in cs us sy id wa st 0 1 112 9592 7140 1879496 0 0 621 1180 6 2 0 2 86 11 0 0 9 112 10668 7148 1877760 0 0 84 21600 1926 454 0 3 49 49 0 1 3 112 9308 7156 1879092 0 0 1400 1580 2123 2009 0 2 50 48 0 0 0 112 9432 7156 1879476 0 0 3664 0 2275 2862 0 3 81 17 0 0 1 112 9212 7172 1879588 0 0 2196 32 2132 2293 0 2 66 32 0 1 0 112 9848 7180 1878864 0 0 1268 4168 1852 1934 0 2 84 14 0 0 7 112 11208 7180 1876936 0 0 92 13124 1339 355 0 2 50 48 0 0 0 112 10708 7188 1877512 0 0 628 16 1552 1014 0 1 84 16 0 0 1 112 10344 7188 1878044 0 0 472 0 1334 721 0 1 89 10 0 0 1 112 10372 7196 1878136 0 0 1936 32 2268 2850 0 2 50 48 0 0 0 112 10336 7192 1878312 0 0 2220 0 2308 2966 0 2 51 47 0 0 0 112 9716 7200 1879012 0 0 696 28 1484 885 0 1 89 11 0 0 2 112 9400 7200 1879280 0 0 236 0 2112 5819 0 8 75 18 0 0 0 112 9980 7196 1878816 0 0 1360 20 1840 1743 0 1 83 15 0 0 15 112 9168 7196 1879580 0 0 768 0 1423 728 0 1 91 9 0 0 0 112 10756 7200 1877752 0 0 4068 32 2788 4225 0 4 70 26 0 I have tuned nfs server with this option : RPCNFSDARGS="-N 2" RPCNFSDARGS="-N 4" MOUNTD_NFS_V3="yes" RPCNFSDCOUNT=160 about the file system, i use ext4 on LVM partition. Do you have any idea about that. Fabien FAYE RHCE www.generationip.com Free network tools & HOWTO for centos and Redhat
fabien faye
2010-Jan-25 15:03 UTC
[CentOS] Too much cpu wait on nfs server when we need to read data on it
Hi,
You can find below the spec of my server:
cat /proc/cpuinfo
processor : 0
vendor_id : AuthenticAMD
cpu family : 15
model : 37
model name : AMD Opteron(tm) Processor 248
stepping : 1
cpu MHz : 2200.034
cache size : 1024 KB
fpu : yes
fpu_exception : yes
cpuid level : 1
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36
clflush mmx fxsr sse sse2 syscall nx mmxext fxsr_opt lm 3dnowext 3dnow pni
lahf_lm
bogomips : 4400.06
TLB size : 1024 4K pages
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management: ts fid vid ttp
processor : 1
vendor_id : AuthenticAMD
cpu family : 15
model : 37
model name : AMD Opteron(tm) Processor 248
stepping : 1
cpu MHz : 2200.034
cache size : 1024 KB
fpu : yes
fpu_exception : yes
cpuid level : 1
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36
clflush mmx fxsr sse sse2 syscall nx mmxext fxsr_opt lm 3dnowext 3dnow pni
lahf_lm
bogomips : 4399.42
TLB size : 1024 4K pages
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management: ts fid vid ttp
free
total used free shared buffers cached
Mem: 2059236 2049896 9340 0 9800 1874828
-/+ buffers/cache: 165268 1893968
Swap: 4095992 112 4095880
Linux 1-NFS.domain.com 2.6.18-164.9.1.el5 #1 SMP Tue Dec 15 20:57:57 EST 2009
x86_64 x86_64 x86_64 GNU/Linux
--- Physical volume ---
PV Name /dev/sdc1
VG Name vgspace
PV Size 4.55 TB / not usable 1.97 MB
Allocatable yes
PE Size (KByte) 4096
Total PE 1192067
Free PE 795267
Allocated PE 396800
--- Physical volume ---
PV Name /dev/sdd1
VG Name vgspace
PV Size 4.55 TB / not usable 1.97 MB
Allocatable yes
PE Size (KByte) 4096
Total PE 1192067
Free PE 808067
Allocated PE 384000
I need ext4 for file bigger than 2TB.
/dev/mapper/vgspace-vol1 on /vol/vol1 type ext4 (rw,noatime)
/dev/mapper/vgspace-vol2 on /vol/vol2 type ext4 (rw,noatime)
# tune2fs -l /dev/mapper/vgspace-vol1
tune2fs 1.39 (29-May-2006)
tune2fs: Filesystem has unsupported feature(s) while trying to open
/dev/mapper/vgspace-vol1
Couldn't find valid filesystem superblock.
# tune2fs -l /dev/mapper/vgspace-vol2
tune2fs 1.39 (29-May-2006)
tune2fs: Filesystem has unsupported feature(s) while trying to open
/dev/mapper/vgspace-vol2
Couldn't find valid filesystem superblock.
export nfs :
/vol/vol1 192.168.*.0/255.255.255.0(async,no_subtree_check,rw,no_root_squash)
/vol/vol2 192.168.*.0/255.255.255.0(async,no_subtree_check,rw,no_root_squash)
Best regards
Fabien FAYE
----- Original Message -----
From: "Jim Perrin" <jperrin at gmail.com>
To: "CentOS mailing list" <centos at centos.org>
Sent: Monday, January 25, 2010 2:27:41 PM
Subject: Re: [CentOS] Too much cpu wait on nfs server when we need to read data
on it
On Mon, Jan 25, 2010 at 8:39 AM, fabien faye <fabien at faye.eu>
wrote:> Hi,
>
> I have a big server with 24 Disk on 2 3ware card.
Define 'big'. You're not giving much in terms of memory or cpu
specs.
> When i write data on my nfs server everything is fine but when i want to
read data i have a lot of cpu wait.
Okay. Have you looked at any performance tuning or elevator/scheduler
adjustments? File system tuning? What have you tried on your own so
far to debug this?
> about the file system, i use ext4 on LVM partition.
Why? Is there a performance reason for you to use ext4 vs ext3?
> Do you have any idea about that.
Not based on the little information you've provided. It would help for
you to supply more detail, and the things you've already looked at.
--
During times of universal deceit, telling the truth becomes a revolutionary act.
George Orwell
_______________________________________________
CentOS mailing list
CentOS at centos.org
http://lists.centos.org/mailman/listinfo/centos