Arnold Maderthaner
2007-Aug-22 00:44 UTC
[Ocfs2-users] Slow concurrent actions on the same LVM logical volume
Hi 2 all ! I have problems with concurrent filesystem actions on a ocfs2 filesystem which is mounted by 2 nodes. OS=RH5ES and OCFS2=1.2.6 F.e.: If I have a LV called testlv which is mounted on /mnt on both servers and I do a "dd if=/dev/zero of=/mnt/test.a bs=1024 count=1000000" on server 1 and do at the same time a du -hs /mnt/test.a it takes about 5 seconds for du -hs to execute: 270M test.a real 0m3.627s user 0m0.000s sys 0m0.002s or even longer. If i do a du -hs ond a file which is not written by another server its fast: 977M test.b real 0m0.001s user 0m0.001s sys 0m0.000s If I write 2 different files from both servers to the same LV I get only 2.8mb/sec. If I write 2 different files from both servers to a different LV (testlv,test2lv) then I get about 30mb/sec on each write (so 60mb/sec combined which is very good). does anyone has an idea what could be wrong. Are there any mount options for ocfs2 which could help me? is this a lvm2 or ocfs2 problem (locking,blocking,.... issue) ? Any help will be appriciated. yours Arnold -- Arnold Maderthaner J4Care Inc.
Sunil Mushran
2007-Aug-22 10:37 UTC
[Ocfs2-users] Slow concurrent actions on the same LVM logical volume
Repeat the first test. This time run top on the first server. Which process is eating the cpu? From the time it appears that the second server is waiting for the AST that the first server is slow to send. And it could be slow because it may be busy flushing the data to disk. How much memory/cpu do these servers have? As far as the second test goes, what is the concurrent write performance directly to the LV (minus fs). On Wed, Aug 22, 2007 at 01:14:02PM +0530, Arnold Maderthaner wrote:> Hi 2 all ! > > I have problems with concurrent filesystem actions on a ocfs2 > filesystem which is mounted by 2 nodes. OS=RH5ES and OCFS2=1.2.6 > F.e.: If I have a LV called testlv which is mounted on /mnt on both > servers and I do a "dd if=/dev/zero of=/mnt/test.a bs=1024 > count=1000000" on server 1 and do at the same time a du -hs > /mnt/test.a it takes about 5 seconds for du -hs to execute: > 270M test.a > > real 0m3.627s > user 0m0.000s > sys 0m0.002s > > or even longer. If i do a du -hs ond a file which is not written by > another server its fast: > 977M test.b > > real 0m0.001s > user 0m0.001s > sys 0m0.000s > > If I write 2 different files from both servers to the same LV I get > only 2.8mb/sec. > If I write 2 different files from both servers to a different LV > (testlv,test2lv) then I get about 30mb/sec on each write (so 60mb/sec > combined which is very good). > does anyone has an idea what could be wrong. Are there any mount > options for ocfs2 which could help me? is this a lvm2 or ocfs2 problem > (locking,blocking,.... issue) ? > > Any help will be appriciated. > > yours > > Arnold > -- > Arnold Maderthaner > J4Care Inc. > > _______________________________________________ > Ocfs2-users mailing list > Ocfs2-users@oss.oracle.com > http://oss.oracle.com/mailman/listinfo/ocfs2-users
Arnold Maderthaner
2007-Aug-23 04:12 UTC
[Ocfs2-users] Slow concurrent actions on the same LVM logical volume
Hi ! I watched the servers today and noticed that the server load is very high but not much is working on the server: server1: 16:37:11 up 32 min, 1 user, load average: 6.45, 7.90, 7.45 server2: 16:37:19 up 32 min, 4 users, load average: 9.79, 9.76, 8.26 i also tried to do some selects on the oracle database today. to do a "select count(*) from xyz" where xyz has 120 000 rows it took me about 10 sec. always the process who does the filesystem access is the first one. and every command (ls,du,vgdisplay,lvdisplay,dd,cp,....) which needs access to a ocfs2 filesystem seams like blocking or it seams to me that there is a lock. Can anyone please help me ? yours Arnold On 8/22/07, Arnold Maderthaner <arnold.maderthaner@j4care.com> wrote:> > There is not really a high server load when writing to the disks but will > try it asap. what do you mean with AST ? > Can I enable some debuging on ocfs2 ? We have 2 Dell servers with about > 4gb of ram each and 2 Dual Core CPUs with 2GHZ each. > Can you please describe which tests todo again ? > > Thx for your help > > Arnold > > On 8/22/07, Sunil Mushran < Sunil.Mushran@oracle.com> wrote: > > > > Repeat the first test. This time run top on the first server. Which > > process is eating the cpu? From the time it appears that the second > > server is waiting for the AST that the first server is slow to send. And > > it could be slow because it may be busy flushing the data to disk. > > How much memory/cpu do these servers have? > > > > As far as the second test goes, what is the concurrent write performance > > directly to the LV (minus fs). > > > > On Wed, Aug 22, 2007 at 01:14:02PM +0530, Arnold Maderthaner wrote: > > > Hi 2 all ! > > > > > > I have problems with concurrent filesystem actions on a ocfs2 > > > filesystem which is mounted by 2 nodes. OS=RH5ES and OCFS2=1.2.6 > > > F.e.: If I have a LV called testlv which is mounted on /mnt on both > > > servers and I do a "dd if=/dev/zero of=/mnt/test.a bs=1024 > > > count=1000000" on server 1 and do at the same time a du -hs > > > /mnt/test.a it takes about 5 seconds for du -hs to execute: > > > 270M test.a > > > > > > real 0m3.627s > > > user 0m0.000s > > > sys 0m0.002s > > > > > > or even longer. If i do a du -hs ond a file which is not written by > > > another server its fast: > > > 977M test.b > > > > > > real 0m0.001s > > > user 0m0.001s > > > sys 0m0.000s > > > > > > If I write 2 different files from both servers to the same LV I get > > > only 2.8mb/sec. > > > If I write 2 different files from both servers to a different LV > > > (testlv,test2lv) then I get about 30mb/sec on each write (so 60mb/sec > > > combined which is very good). > > > does anyone has an idea what could be wrong. Are there any mount > > > options for ocfs2 which could help me? is this a lvm2 or ocfs2 problem > > > > > (locking,blocking,.... issue) ? > > > > > > Any help will be appriciated. > > > > > > yours > > > > > > Arnold > > > -- > > > Arnold Maderthaner > > > J4Care Inc. > > > > > > _______________________________________________ > > > Ocfs2-users mailing list > > > Ocfs2-users@oss.oracle.com > > > http://oss.oracle.com/mailman/listinfo/ocfs2-users > > > > > > -- > Arnold Maderthaner > J4Care Inc.-- Arnold Maderthaner J4Care Inc. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://oss.oracle.com/pipermail/ocfs2-users/attachments/20070823/c0530303/attachment.html
Alexei_Roudnev
2007-Aug-23 11:43 UTC
[Ocfs2-users] Slow concurrent actions on the same LVM logicalvolume
Run and send here, please:
vmstat 5
(5 lines)
iostat -x 5
(2 - 3 outputs)
top
(1 screen)
slabtop (or equivalent, I dont remember)
----- Original Message -----
From: Arnold Maderthaner
To: Sunil Mushran ; ocfs2-users@oss.oracle.com
Sent: Thursday, August 23, 2007 4:12 AM
Subject: Re: [Ocfs2-users] Slow concurrent actions on the same LVM
logicalvolume
Hi !
I watched the servers today and noticed that the server load is very high but
not much is working on the server:
server1:
16:37:11 up 32 min, 1 user, load average: 6.45, 7.90, 7.45
server2:
16:37:19 up 32 min, 4 users, load average: 9.79, 9.76, 8.26
i also tried to do some selects on the oracle database today.
to do a "select count(*) from xyz" where xyz has 120 000 rows it
took me about 10 sec.
always the process who does the filesystem access is the first one.
and every command (ls,du,vgdisplay,lvdisplay,dd,cp,....) which needs access to
a ocfs2 filesystem seams like blocking
or it seams to me that there is a lock.
Can anyone please help me ?
yours
Arnold
On 8/22/07, Arnold Maderthaner <arnold.maderthaner@j4care.com> wrote:
There is not really a high server load when writing to the disks but will
try it asap. what do you mean with AST ?
Can I enable some debuging on ocfs2 ? We have 2 Dell servers with about 4gb
of ram each and 2 Dual Core CPUs with
2GHZ each.
Can you please describe which tests todo again ?
Thx for your help
Arnold
On 8/22/07, Sunil Mushran < Sunil.Mushran@oracle.com> wrote:
Repeat the first test. This time run top on the first server. Which
process is eating the cpu? From the time it appears that the second
server is waiting for the AST that the first server is slow to send. And
it could be slow because it may be busy flushing the data to disk.
How much memory/cpu do these servers have?
As far as the second test goes, what is the concurrent write performance
directly to the LV (minus fs).
On Wed, Aug 22, 2007 at 01:14:02PM +0530, Arnold Maderthaner wrote:
> Hi 2 all !
>
> I have problems with concurrent filesystem actions on a ocfs2
> filesystem which is mounted by 2 nodes. OS=RH5ES and OCFS2=1.2.6
> F.e.: If I have a LV called testlv which is mounted on /mnt on both
> servers and I do a "dd if=/dev/zero of=/mnt/test.a bs=1024
> count=1000000" on server 1 and do at the same time a du -hs
> /mnt/test.a it takes about 5 seconds for du -hs to execute:
> 270M test.a
>
> real 0m3.627s
> user 0m0.000s
> sys 0m0.002s
>
> or even longer. If i do a du -hs ond a file which is not written by
> another server its fast:
> 977M test.b
>
> real 0m0.001s
> user 0m0.001s
> sys 0m0.000s
>
> If I write 2 different files from both servers to the same LV I get
> only 2.8mb/sec.
> If I write 2 different files from both servers to a different LV
> (testlv,test2lv) then I get about 30mb/sec on each write (so 60mb/sec
> combined which is very good).
> does anyone has an idea what could be wrong. Are there any mount
> options for ocfs2 which could help me? is this a lvm2 or ocfs2
problem
> (locking,blocking,.... issue) ?
>
> Any help will be appriciated.
>
> yours
>
> Arnold
> --
> Arnold Maderthaner
> J4Care Inc.
>
> _______________________________________________
> Ocfs2-users mailing list
> Ocfs2-users@oss.oracle.com
> http://oss.oracle.com/mailman/listinfo/ocfs2-users
--
Arnold Maderthaner
J4Care Inc.
--
Arnold Maderthaner
J4Care Inc.
------------------------------------------------------------------------------
_______________________________________________
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://oss.oracle.com/pipermail/ocfs2-users/attachments/20070823/636b7453/attachment.html
Arnold Maderthaner
2007-Aug-23 21:53 UTC
[Ocfs2-users] Slow concurrent actions on the same LVM logicalvolume
vmstat server1:
[root@impax ~]# vmstat 5
procs -----------memory---------- ---swap-- -----io---- --system--
-----cpu------
r b swpd free buff cache si so bi bo in cs us sy id
wa st
0 0 0 2760468 119924 872240 0 0 4 4 107 63 0 0 99
0 0
0 2 0 2760468 119928 872240 0 0 4 30 1095 277 1 0 99
0 0
0 0 0 2760468 119940 872248 0 0 7 11 1076 232 0 0
100 0 0
0 0 0 2760468 119948 872272 0 0 4 18 1084 244 0 0
100 0 0
0 0 0 2760468 119948 872272 0 0 7 34 1063 220 0 0
100 0 0
0 0 0 2760468 119948 872272 0 0 0 1 1086 243 0 0
100 0 0
0 0 0 2760716 119956 872272 0 0 7 10 1065 234 2 0 98
0 0
vmstart server2:
[root@impaxdb ~]# vmstat 5
procs -----------memory---------- ---swap-- -----io---- --system--
-----cpu------
r b swpd free buff cache si so bi bo in cs us sy id
wa st
0 0 24676 143784 46428 3652848 0 0 11 20 107 122 0 0 84
16 0
0 1 24676 143784 46452 3652832 0 0 4 14 1080 237 0 0 89
11 0
0 1 24672 143660 46460 3652812 6 0 14 34 1074 246 0 0 90
10 0
0 0 24672 143660 46460 3652852 0 0 7 22 1076 246 0 0 80
20 0
0 1 24672 143660 46476 3652844 0 0 7 10 1068 227 0 0 85
15 0
iostat server1:
[root@impax ~]# iostat -x 5
Linux 2.6.18-8.el5PAE (impax) 08/24/2007
avg-cpu: %user %nice %system %iowait %steal %idle
0.07 0.00 0.04 0.45 0.00 99.44
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz
avgqu-sz await svctm %util
sda 0.24 0.90 0.23 0.76 15.72 13.33 29.22
0.01 12.68 4.77 0.47
sdb 0.03 1.23 0.02 0.58 1.34 6.81 13.59
0.00 0.79 0.30 0.02
sdc 0.58 0.87 3.88 3.78 16.69 10.87 3.60
7.98 1042.08 89.69 68.68
dm-0 0.00 0.00 0.31 1.14 0.96 6.99 5.46
1.55 1061.28 459.00 66.92
dm-1 0.00 0.00 0.31 0.31 0.95 0.31 2.01
0.66 1054.16 1054.17 66.15
dm-2 0.00 0.00 0.31 0.31 0.95 0.31 2.01
0.66 1057.72 1057.84 66.24
dm-3 0.00 0.00 0.96 0.34 6.15 0.55 5.12
0.77 592.01 504.72 66.04
dm-4 0.00 0.00 0.32 0.33 0.96 0.41 2.14
0.66 1031.13 1028.75 66.00
dm-5 0.00 0.00 0.31 0.32 0.95 0.40 2.12
0.66 1037.27 1036.55 66.09
dm-6 0.00 0.00 0.31 0.31 0.94 0.31 2.01
0.66 1064.34 1064.45 66.37
dm-7 0.00 0.00 0.32 0.31 0.95 0.32 2.01
0.66 1043.05 1042.99 65.77
dm-8 0.00 0.00 0.31 0.31 0.95 0.32 2.03
0.66 1050.73 1048.23 65.97
dm-9 0.00 0.00 0.34 0.31 0.99 0.31 2.01
0.67 1027.41 1006.68 65.71
dm-10 0.00 0.00 0.32 0.32 0.95 0.32 2.01
0.68 1078.71 1050.22 66.41
dm-12 0.00 0.00 0.31 0.31 0.96 0.31 2.03
0.66 1065.33 1065.10 66.43
dm-13 0.00 0.00 0.04 1.27 1.27 2.54 2.91
0.00 2.43 0.10 0.01
dm-18 0.00 0.00 0.01 0.35 0.04 2.80 8.00
0.01 25.42 0.03 0.00
dm-19 0.00 0.00 0.00 0.18 0.01 1.47 8.00
0.00 0.42 0.21 0.00
dm-21 0.00 0.00 0.02 0.01 2.20 0.11 65.49
0.00 9.84 3.01 0.01
dm-26 0.00 0.00 0.01 0.05 0.12 0.43 9.21
0.00 11.66 7.05 0.04
dm-27 0.00 0.00 0.00 0.21 0.02 1.70 8.02
0.00 12.09 4.57 0.10
dm-28 0.00 0.00 0.26 0.11 8.90 0.91 26.40
0.01 14.94 1.85 0.07
dm-29 0.00 0.00 0.05 1.03 1.04 8.22 8.56
0.01 13.76 2.48 0.27
avg-cpu: %user %nice %system %iowait %steal %idle
0.00 0.00 0.00 0.40 0.00 99.60
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz
avgqu-sz await svctm %util
sda 0.00 2.80 0.00 2.20 0.00 40.00 18.18
0.03 12.27 4.73 1.04
sdb 0.00 0.40 0.00 0.40 0.00 1.60 4.00
0.00 5.50 5.50 0.22
sdc 0.00 0.00 4.80 4.80 14.40 4.80 2.00
7.21 1560.12 63.67 61.12
dm-0 0.00 0.00 0.40 0.40 1.20 0.40 2.00
0.60 1559.50 750.25 60.02
dm-1 0.00 0.00 0.40 0.40 1.20 0.40 2.00
0.60 1558.50 750.50 60.04
dm-2 0.00 0.00 0.40 0.40 1.20 0.40 2.00
0.60 1559.50 752.75 60.22
dm-3 0.00 0.00 0.40 0.40 1.20 0.40 2.00
0.60 1562.75 752.25 60.18
dm-4 0.00 0.00 0.40 0.40 1.20 0.40 2.00
0.60 1559.50 752.50 60.20
dm-5 0.00 0.00 0.40 0.40 1.20 0.40 2.00
0.60 1563.00 750.25 60.02
dm-6 0.00 0.00 0.40 0.40 1.20 0.40 2.00
0.60 1558.50 750.25 60.02
dm-7 0.00 0.00 0.40 0.40 1.20 0.40 2.00
0.60 1559.50 752.50 60.20
dm-8 0.00 0.00 0.40 0.40 1.20 0.40 2.00
0.60 1559.75 750.25 60.02
dm-9 0.00 0.00 0.40 0.40 1.20 0.40 2.00
0.60 1554.00 750.25 60.02
dm-10 0.00 0.00 0.40 0.40 1.20 0.40 2.00
0.60 1555.75 752.25 60.18
dm-12 0.00 0.00 0.40 0.40 1.20 0.40 2.00
0.60 1571.25 750.25 60.02
dm-13 0.00 0.00 0.00 0.80 0.00 1.60 2.00
0.01 8.25 2.75 0.22
dm-18 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
dm-19 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
dm-21 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
dm-26 0.00 0.00 0.00 0.60 0.00 4.80 8.00
0.01 12.67 8.33 0.50
dm-27 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
dm-28 0.00 0.00 0.00 1.80 0.00 14.40 8.00
0.02 10.22 2.44 0.44
dm-29 0.00 0.00 0.00 0.80 0.00 6.40 8.00
0.01 10.25 3.75 0.30
avg-cpu: %user %nice %system %iowait %steal %idle
0.05 0.00 0.00 0.00 0.00 99.95
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz
avgqu-sz await svctm %util
sda 0.00 0.20 0.00 0.60 0.00 6.40 10.67
0.01 14.67 6.00 0.36
sdb 0.00 0.40 0.00 0.40 0.00 1.60 4.00
0.00 0.00 0.00 0.00
sdc 0.00 0.00 4.80 4.80 14.40 4.80 2.00
8.81 1292.92 77.06 73.98
dm-0 0.00 0.00 0.20 0.40 0.60 0.40 1.67
0.74 1723.33 1233.00 73.98
dm-1 0.00 0.00 0.20 0.40 0.60 0.40 1.67
0.73 1722.00 1223.33 73.40
dm-2 0.00 0.00 0.20 0.40 0.60 0.40 1.67
0.73 1726.33 1222.00 73.32
dm-3 0.00 0.00 0.20 0.40 0.60 0.40 1.67
0.73 1725.67 1222.00 73.32
dm-4 0.00 0.00 0.20 0.40 0.60 0.40 1.67
0.73 1726.00 1222.00 73.32
dm-5 0.00 0.00 0.20 0.40 0.60 0.40 1.67
0.73 1722.33 1222.00 73.32
dm-6 0.00 0.00 0.20 0.40 0.60 0.40 1.67
0.73 1722.00 1223.67 73.42
dm-7 0.00 0.00 0.20 0.40 0.60 0.40 1.67
0.73 1726.00 1222.00 73.32
dm-8 0.00 0.00 0.20 0.40 0.60 0.40 1.67
0.73 1722.33 1222.00 73.32
dm-9 0.00 0.00 0.20 0.40 0.60 0.40 1.67
0.73 1723.00 1214.33 72.86
dm-10 0.00 0.00 0.20 0.40 0.60 0.40 1.67
0.73 1725.67 1222.00 73.32
dm-12 0.00 0.00 0.20 0.40 0.60 0.40 1.67
0.74 1722.00 1231.67 73.90
dm-13 0.00 0.00 0.00 0.80 0.00 1.60 2.00
0.00 0.00 0.00 0.00
dm-18 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
dm-19 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
dm-21 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
dm-26 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
dm-27 0.00 0.00 0.00 0.60 0.00 4.80 8.00
0.01 13.00 4.33 0.26
dm-28 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
dm-29 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
iostat server2:
[root@impaxdb ~]# iostat -x 5
Linux 2.6.18-8.el5PAE (impaxdb) 08/24/2007
avg-cpu: %user %nice %system %iowait %steal %idle
0.48 0.00 0.10 15.70 0.00 83.72
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz
avgqu-sz await svctm %util
sda 0.28 1.38 0.32 0.82 18.52 17.59 31.55
0.00 1.98 0.83 0.10
sdb 0.01 1.73 0.02 0.04 3.20 14.15 297.21
0.00 9.18 0.87 0.01
sdc 6.44 14.59 3.94 4.26 65.36 126.12 23.36
9.52 1160.63 91.33 74.88
dm-0 0.00 0.00 0.29 0.29 0.89 0.29 2.01
0.71 1214.34 1214.01 71.18
dm-1 0.00 0.00 0.29 0.29 0.89 0.29 2.01
0.71 1206.47 1206.38 70.95
dm-2 0.00 0.00 0.29 0.29 0.89 0.29 2.01
0.71 1207.35 1207.27 70.96
dm-3 0.00 0.00 7.10 9.84 55.33 76.63 7.79
25.51 1506.27 43.35 73.42
dm-4 0.00 0.00 0.32 5.48 1.10 41.82 7.39
9.21 1586.36 122.41 71.06
dm-5 0.00 0.00 0.30 0.88 0.90 5.02 5.02
0.72 606.80 600.72 70.89
dm-6 0.00 0.00 0.29 0.29 0.89 0.29 2.01
0.71 1206.90 1206.94 70.91
dm-7 0.00 0.00 0.29 0.29 0.89 0.29 2.01
0.71 1202.65 1202.69 70.82
dm-8 0.00 0.00 0.29 0.29 0.89 0.29 2.01
0.71 1204.47 1204.43 70.84
dm-9 0.00 0.00 0.29 0.29 0.89 0.29 2.01
0.71 1202.38 1202.42 70.77
dm-10 0.00 0.00 0.29 0.29 0.88 0.29 2.01
0.71 1208.53 1208.48 70.93
dm-12 0.00 0.00 0.30 0.29 0.91 0.29 2.04
0.71 1202.82 1202.49 70.92
dm-13 0.00 0.00 0.00 0.00 0.01 0.00 6.99
0.00 0.47 0.06 0.00
dm-18 0.00 0.00 0.02 1.77 3.16 14.15 9.67
0.03 16.69 0.03 0.01
dm-19 0.00 0.00 0.00 0.00 0.01 0.00 8.02
0.00 0.69 0.34 0.00
dm-20 0.00 0.00 0.05 0.28 3.48 2.26 17.18
0.00 3.37 0.47 0.02
dm-25 0.00 0.00 0.01 0.05 0.16 0.44 9.68
0.00 1.20 0.65 0.00
dm-26 0.00 0.00 0.00 0.14 0.02 1.11 8.03
0.00 3.28 0.05 0.00
dm-27 0.00 0.00 0.31 0.15 9.69 1.24 23.61
0.00 9.39 0.88 0.04
dm-28 0.00 0.00 0.06 1.08 1.24 8.61 8.65
0.00 1.04 0.21 0.02
avg-cpu: %user %nice %system %iowait %steal %idle
0.05 0.00 0.05 23.62 0.00 76.27
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz
avgqu-sz await svctm %util
sda 0.00 4.85 0.00 2.22 0.00 56.57 25.45
0.00 0.91 0.36 0.08
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
sdc 0.00 1.21 4.85 5.45 14.55 24.24 3.76
12.51 1556.98 94.24 97.09
dm-0 0.00 0.00 0.20 0.40 0.61 0.40 1.67
0.97 2055.67 1602.00 97.09
dm-1 0.00 0.00 0.20 0.40 0.61 0.40 1.67
0.96 2064.67 1589.00 96.30
dm-2 0.00 0.00 0.20 0.40 0.61 0.40 1.67
0.96 2063.67 1582.67 95.92
dm-3 0.00 0.00 0.20 0.40 0.61 0.40 1.67
4.79 8988.33 1582.00 95.88
dm-4 0.00 0.00 0.20 1.21 0.61 6.87 5.29
0.96 886.57 679.14 96.04
dm-5 0.00 0.00 0.20 1.21 0.61 6.87 5.29
0.96 878.00 680.00 96.16
dm-6 0.00 0.00 0.20 0.40 0.61 0.40 1.67
0.97 2055.67 1593.00 96.55
dm-7 0.00 0.00 0.20 0.40 0.61 0.40 1.67
0.96 2064.67 1584.33 96.02
dm-8 0.00 0.00 0.20 0.40 0.61 0.40 1.67
0.96 2064.67 1586.33 96.14
dm-9 0.00 0.00 0.20 0.40 0.61 0.40 1.67
0.96 2064.67 1583.33 95.96
dm-10 0.00 0.00 0.20 0.40 0.61 0.40 1.67
0.97 2055.67 1597.67 96.83
dm-12 0.00 0.00 0.20 0.40 0.61 0.40 1.67
0.97 2071.67 1595.67 96.71
dm-13 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
dm-18 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
dm-19 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
dm-20 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
dm-25 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
dm-26 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
dm-27 0.00 0.00 0.00 0.81 0.00 6.46 8.00
0.00 0.00 0.00 0.00
dm-28 0.00 0.00 0.00 4.85 0.00 38.79 8.00
0.01 2.17 0.17 0.08
avg-cpu: %user %nice %system %iowait %steal %idle
0.00 0.00 0.00 19.85 0.00 80.15
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz
avgqu-sz await svctm %util
sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
sdc 0.00 0.61 0.00 0.00 0.00 0.00 0.00
9.12 0.00 0.00 80.26
dm-0 0.00 0.00 0.20 0.00 0.61 0.00 3.00
0.65 0.00 3196.00 64.70
dm-1 0.00 0.00 0.20 0.00 0.61 0.00 3.00
0.65 0.00 3235.00 65.49
dm-2 0.00 0.00 0.20 0.00 0.61 0.00 3.00
0.66 0.00 3254.00 65.87
dm-3 0.00 0.00 0.40 0.81 2.23 6.48 7.17
3.29 0.00 660.83 80.26
dm-4 0.00 0.00 0.20 0.00 0.61 0.00 3.00
0.66 0.00 3248.00 65.75
dm-5 0.00 0.00 0.20 0.00 0.61 0.00 3.00
0.66 0.00 3242.00 65.63
dm-6 0.00 0.00 0.20 0.00 0.61 0.00 3.00
0.65 0.00 3223.00 65.24
dm-7 0.00 0.00 0.20 0.00 0.61 0.00 3.00
0.66 0.00 3249.00 65.77
dm-8 0.00 0.00 0.20 0.00 0.61 0.00 3.00
0.66 0.00 3243.00 65.65
dm-9 0.00 0.00 0.20 0.00 0.61 0.00 3.00
0.66 0.00 3252.00 65.83
dm-10 0.00 0.00 0.20 0.00 0.61 0.00 3.00
0.65 0.00 3209.00 64.96
dm-12 0.00 0.00 0.20 0.00 0.61 0.00 3.00
0.65 0.00 3215.00 65.08
dm-13 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
dm-18 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
dm-19 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
dm-20 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
dm-25 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
dm-26 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
dm-27 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
dm-28 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
top server1:
Tasks: 217 total, 1 running, 214 sleeping, 0 stopped, 2 zombie
Cpu(s): 0.1%us, 0.0%sy, 0.0%ni, 99.4%id, 0.4%wa, 0.0%hi, 0.0%si,
0.0%st
Mem: 4084444k total, 1323728k used, 2760716k free, 120072k buffers
Swap: 8193020k total, 0k used, 8193020k free, 872284k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
14245 root 16 0 2288 988 704 R 2 0.0 0:00.01 top
1 root 15 0 2036 652 564 S 0 0.0 0:01.13 init
2 root RT 0 0 0 0 S 0 0.0 0:00.00 migration/0
3 root 34 19 0 0 0 S 0 0.0 0:00.46 ksoftirqd/0
4 root RT 0 0 0 0 S 0 0.0 0:00.00 watchdog/0
5 root RT 0 0 0 0 S 0 0.0 0:00.00 migration/1
6 root 34 19 0 0 0 S 0 0.0 0:00.07 ksoftirqd/1
7 root RT 0 0 0 0 S 0 0.0 0:00.00 watchdog/1
8 root RT 0 0 0 0 S 0 0.0 0:00.00 migration/2
9 root 34 19 0 0 0 S 0 0.0 0:00.19 ksoftirqd/2
10 root RT 0 0 0 0 S 0 0.0 0:00.00 watchdog/2
11 root RT 0 0 0 0 S 0 0.0 0:00.00 migration/3
12 root 34 19 0 0 0 S 0 0.0 0:00.23 ksoftirqd/3
13 root RT 0 0 0 0 S 0 0.0 0:00.00 watchdog/3
14 root 10 -5 0 0 0 S 0 0.0 0:00.00 events/0
15 root 10 -5 0 0 0 S 0 0.0 0:00.00 events/1
16 root 10 -5 0 0 0 S 0 0.0 0:00.00 events/2
17 root 10 -5 0 0 0 S 0 0.0 0:00.00 events/3
18 root 19 -5 0 0 0 S 0 0.0 0:00.00 khelper
19 root 10 -5 0 0 0 S 0 0.0 0:00.00 kthread
25 root 10 -5 0 0 0 S 0 0.0 0:00.00 kblockd/0
26 root 10 -5 0 0 0 S 0 0.0 0:00.04 kblockd/1
27 root 10 -5 0 0 0 S 0 0.0 0:00.00 kblockd/2
28 root 10 -5 0 0 0 S 0 0.0 0:00.06 kblockd/3
29 root 14 -5 0 0 0 S 0 0.0 0:00.00 kacpid
147 root 14 -5 0 0 0 S 0 0.0 0:00.00 cqueue/0
148 root 14 -5 0 0 0 S 0 0.0 0:00.00 cqueue/1
149 root 14 -5 0 0 0 S 0 0.0 0:00.00 cqueue/2
150 root 14 -5 0 0 0 S 0 0.0 0:00.00 cqueue/3
153 root 10 -5 0 0 0 S 0 0.0 0:00.00 khubd
155 root 10 -5 0 0 0 S 0 0.0 0:00.00 kseriod
231 root 19 0 0 0 0 S 0 0.0 0:00.00 pdflush
232 root 15 0 0 0 0 S 0 0.0 0:00.15 pdflush
233 root 14 -5 0 0 0 S 0 0.0 0:00.00 kswapd0
234 root 14 -5 0 0 0 S 0 0.0 0:00.00 aio/0
235 root 14 -5 0 0 0 S 0 0.0 0:00.00 aio/1
236 root 14 -5 0 0 0 S 0 0.0 0:00.00 aio/2
top server2:
Tasks: 264 total, 1 running, 260 sleeping, 0 stopped, 3 zombie
Cpu(s): 0.5%us, 0.1%sy, 0.0%ni, 83.7%id, 15.7%wa, 0.0%hi, 0.0%si,
0.0%st
Mem: 4084444k total, 3941216k used, 143228k free, 46608k buffers
Swap: 8193020k total, 24660k used, 8168360k free, 3652864k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
9507 root 15 0 2292 1020 704 R 2 0.0 0:00.01 top
1 root 15 0 2032 652 564 S 0 0.0 0:01.20 init
2 root RT 0 0 0 0 S 0 0.0 0:00.00 migration/0
3 root 34 19 0 0 0 S 0 0.0 0:00.25 ksoftirqd/0
4 root RT 0 0 0 0 S 0 0.0 0:00.00 watchdog/0
5 root RT 0 0 0 0 S 0 0.0 0:00.00 migration/1
6 root 34 19 0 0 0 S 0 0.0 0:00.42 ksoftirqd/1
7 root RT 0 0 0 0 S 0 0.0 0:00.00 watchdog/1
8 root RT 0 0 0 0 S 0 0.0 0:00.00 migration/2
9 root 34 19 0 0 0 S 0 0.0 0:00.56 ksoftirqd/2
10 root RT 0 0 0 0 S 0 0.0 0:00.00 watchdog/2
11 root RT 0 0 0 0 S 0 0.0 0:00.00 migration/3
12 root 39 19 0 0 0 S 0 0.0 0:00.08 ksoftirqd/3
13 root RT 0 0 0 0 S 0 0.0 0:00.00 watchdog/3
14 root 10 -5 0 0 0 S 0 0.0 0:00.03 events/0
15 root 10 -5 0 0 0 S 0 0.0 0:00.00 events/1
16 root 10 -5 0 0 0 S 0 0.0 0:00.02 events/2
17 root 10 -5 0 0 0 S 0 0.0 0:00.02 events/3
18 root 10 -5 0 0 0 S 0 0.0 0:00.00 khelper
19 root 11 -5 0 0 0 S 0 0.0 0:00.00 kthread
25 root 10 -5 0 0 0 S 0 0.0 0:00.08 kblockd/0
26 root 10 -5 0 0 0 S 0 0.0 0:00.00 kblockd/1
27 root 10 -5 0 0 0 S 0 0.0 0:00.00 kblockd/2
28 root 10 -5 0 0 0 S 0 0.0 0:00.08 kblockd/3
29 root 18 -5 0 0 0 S 0 0.0 0:00.00 kacpid
147 root 18 -5 0 0 0 S 0 0.0 0:00.00 cqueue/0
148 root 20 -5 0 0 0 S 0 0.0 0:00.00 cqueue/1
149 root 10 -5 0 0 0 S 0 0.0 0:00.00 cqueue/2
150 root 10 -5 0 0 0 S 0 0.0 0:00.00 cqueue/3
153 root 10 -5 0 0 0 S 0 0.0 0:00.00 khubd
155 root 10 -5 0 0 0 S 0 0.0 0:00.00 kseriod
233 root 15 -5 0 0 0 S 0 0.0 0:00.84 kswapd0
234 root 20 -5 0 0 0 S 0 0.0 0:00.00 aio/0
235 root 20 -5 0 0 0 S 0 0.0 0:00.00 aio/1
236 root 20 -5 0 0 0 S 0 0.0 0:00.00 aio/2
237 root 20 -5 0 0 0 S 0 0.0 0:00.00 aio/3
409 root 11 -5 0 0 0 S 0 0.0 0:00.00 kpsmoused
slabtop server1:
Active / Total Objects (% used) : 301060 / 312027 (96.5%)
Active / Total Slabs (% used) : 12055 / 12055 (100.0%)
Active / Total Caches (% used) : 106 / 146 (72.6%)
Active / Total Size (% used) : 45714.46K / 46814.27K (97.7%)
Minimum / Average / Maximum Object : 0.01K / 0.15K / 128.00K
OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
148176 148095 99% 0.05K 2058 72 8232K buffer_head
44573 44497 99% 0.13K 1537 29 6148K dentry_cache
32888 32878 99% 0.48K 4111 8 16444K ext3_inode_cache
10878 10878 100% 0.27K 777 14 3108K radix_tree_node
7943 7766 97% 0.02K 47 169 188K dm_io
7917 7766 98% 0.02K 39 203 156K dm_tio
6424 5414 84% 0.09K 146 44 584K vm_area_struct
6384 6223 97% 0.04K 76 84 304K sysfs_dir_cache
5989 5823 97% 0.03K 53 113 212K size-32
5074 4833 95% 0.06K 86 59 344K size-64
4488 4488 100% 0.33K 408 11 1632K inode_cache
2610 2561 98% 0.12K 87 30 348K size-128
2540 1213 47% 0.01K 10 254 40K anon_vma
2380 1475 61% 0.19K 119 20 476K filp
1430 1334 93% 0.35K 130 11 520K proc_inode_cache
1326 1031 77% 0.05K 17 78 68K
selinux_inode_security
1320 1304 98% 0.25K 88 15 352K size-256
1015 629 61% 0.02K 5 203 20K biovec-1
1008 125 12% 0.05K 14 72 56K journal_head
930 841 90% 0.12K 31 30 124K bio
920 666 72% 0.19K 46 20 184K skbuff_head_cache
798 757 94% 2.00K 399 2 1596K size-2048
791 690 87% 0.03K 7 113 28K ocfs2_em_ent
784 748 95% 0.50K 98 8 392K size-512
590 503 85% 0.06K 10 59 40K biovec-4
564 562 99% 0.88K 141 4 564K ocfs2_inode_cache
546 324 59% 0.05K 7 78 28K delayacct_cache
531 497 93% 0.43K 59 9 236K shmem_inode_cache
520 487 93% 0.19K 26 20 104K biovec-16
505 327 64% 0.04K 5 101 20K pid
500 487 97% 0.75K 100 5 400K biovec-64
460 376 81% 0.04K 5 92 20K Acpi-Operand
452 104 23% 0.03K 4 113 16K pgd
347 347 100% 4.00K 347 1 1388K size-4096
338 303 89% 0.02K 2 169 8K Acpi-Namespace
336 175 52% 0.04K 4 84 16K crq_pool
slabtop server2:
Active / Total Objects (% used) : 886188 / 907066 (97.7%)
Active / Total Slabs (% used) : 20722 / 20723 (100.0%)
Active / Total Caches (% used) : 104 / 147 (70.7%)
Active / Total Size (% used) : 76455.01K / 78620.30K (97.2%)
Minimum / Average / Maximum Object : 0.01K / 0.09K / 128.00K
OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
736344 729536 99% 0.05K 10227 72 40908K buffer_head
37671 34929 92% 0.13K 1299 29 5196K dentry_cache
32872 32666 99% 0.48K 4109 8 16436K ext3_inode_cache
20118 20097 99% 0.27K 1437 14 5748K radix_tree_node
10912 9437 86% 0.09K 248 44 992K vm_area_struct
7714 7452 96% 0.02K 38 203 152K dm_tio
7605 7452 97% 0.02K 45 169 180K dm_io
6552 6475 98% 0.04K 78 84 312K sysfs_dir_cache
5763 5461 94% 0.03K 51 113 204K size-32
4661 4276 91% 0.06K 79 59 316K size-64
4572 3569 78% 0.01K 18 254 72K anon_vma
4440 3770 84% 0.19K 222 20 888K filp
2288 1791 78% 0.35K 208 11 832K proc_inode_cache
1980 1884 95% 0.12K 66 30 264K size-128
1804 1471 81% 0.33K 164 11 656K inode_cache
1404 1046 74% 0.05K 18 78 72K
selinux_inode_security
1060 670 63% 0.19K 53 20 212K skbuff_head_cache
904 767 84% 0.03K 8 113 32K ocfs2_em_ent
870 870 100% 0.12K 29 30 116K bio
840 816 97% 0.50K 105 8 420K size-512
812 578 71% 0.02K 4 203 16K biovec-1
810 800 98% 0.25K 54 15 216K size-256
800 761 95% 2.00K 400 2 1600K size-2048
549 514 93% 0.43K 61 9 244K shmem_inode_cache
546 289 52% 0.05K 7 78 28K delayacct_cache
531 478 90% 0.06K 9 59 36K biovec-4
520 471 90% 0.19K 26 20 104K biovec-16
505 292 57% 0.04K 5 101 20K pid
490 479 97% 0.75K 98 5 392K biovec-64
460 376 81% 0.04K 5 92 20K Acpi-Operand
452 175 38% 0.03K 4 113 16K pgd
440 357 81% 0.38K 44 10 176K sock_inode_cache
369 303 82% 0.44K 41 9 164K UNIX
360 320 88% 1.00K 90 4 360K size-1024
354 124 35% 0.06K 6 59 24K fs_cache
351 351 100% 4.00K 351 1 1404K pmd
uptime server1:
[root@impax ~]# uptime
10:21:09 up 18:16, 1 user, load average: 7.22, 7.72, 7.83
uptime server2:
[root@impaxdb ~]# uptime
10:21:17 up 18:16, 1 user, load average: 8.79, 9.02, 8.98
yours
arnold
On 8/24/07, Alexei_Roudnev <Alexei_Roudnev@exigengroup.com>
wrote:>
> Run and send here, please:
>
> vmstat 5
> (5 lines)
>
> iostat -x 5
> (2 - 3 outputs)
>
> top
> (1 screen)
>
> slabtop (or equivalent, I dont remember)
>
>
> ----- Original Message -----
>
> *From:* Arnold Maderthaner <arnold.maderthaner@j4care.com>
> *To:* Sunil Mushran <Sunil.Mushran@oracle.com> ;
> ocfs2-users@oss.oracle.com
> *Sent:* Thursday, August 23, 2007 4:12 AM
> *Subject:* Re: [Ocfs2-users] Slow concurrent actions on the same LVM
> logicalvolume
>
>
> Hi !
>
> I watched the servers today and noticed that the server load is very high
> but not much is working on the server:
> server1:
> 16:37:11 up 32 min, 1 user, load average: 6.45, 7.90, 7.45
> server2:
> 16:37:19 up 32 min, 4 users, load average: 9.79, 9.76, 8.26
> i also tried to do some selects on the oracle database today.
> to do a "select count(*) from xyz" where xyz has 120 000 rows it
took me
> about 10 sec.
> always the process who does the filesystem access is the first one.
> and every command (ls,du,vgdisplay,lvdisplay,dd,cp,....) which needs
> access to a ocfs2 filesystem seams like blocking or it seams to me that
> there is a lock.
> Can anyone please help me ?
>
> yours
>
> Arnold
>
> On 8/22/07, Arnold Maderthaner <arnold.maderthaner@j4care.com> wrote:
> >
> > There is not really a high server load when writing to the disks but
> > will try it asap. what do you mean with AST ?
> > Can I enable some debuging on ocfs2 ? We have 2 Dell servers with
about
> > 4gb of ram each and 2 Dual Core CPUs with 2GHZ each.
> > Can you please describe which tests todo again ?
> >
> > Thx for your help
> >
> > Arnold
> >
> > On 8/22/07, Sunil Mushran < Sunil.Mushran@oracle.com> wrote:
> > >
> > > Repeat the first test. This time run top on the first server.
Which
> > > process is eating the cpu? From the time it appears that the
second
> > > server is waiting for the AST that the first server is slow to
send.
> > > And
> > > it could be slow because it may be busy flushing the data to
disk.
> > > How much memory/cpu do these servers have?
> > >
> > > As far as the second test goes, what is the concurrent write
> > > performance
> > > directly to the LV (minus fs).
> > >
> > > On Wed, Aug 22, 2007 at 01:14:02PM +0530, Arnold Maderthaner
wrote:
> > > > Hi 2 all !
> > > >
> > > > I have problems with concurrent filesystem actions on a
ocfs2
> > > > filesystem which is mounted by 2 nodes. OS=RH5ES and
OCFS2=1.2.6
> > > > F.e.: If I have a LV called testlv which is mounted on /mnt
on both
> > > > servers and I do a "dd if=/dev/zero of=/mnt/test.a
bs=1024
> > > > count=1000000" on server 1 and do at the same time a du
-hs
> > > > /mnt/test.a it takes about 5 seconds for du -hs to execute:
> > > > 270M test.a
> > > >
> > > > real 0m3.627s
> > > > user 0m0.000s
> > > > sys 0m0.002s
> > > >
> > > > or even longer. If i do a du -hs ond a file which is not
written by
> > > > another server its fast:
> > > > 977M test.b
> > > >
> > > > real 0m0.001s
> > > > user 0m0.001s
> > > > sys 0m0.000s
> > > >
> > > > If I write 2 different files from both servers to the same
LV I get
> > > > only 2.8mb/sec.
> > > > If I write 2 different files from both servers to a
different LV
> > > > (testlv,test2lv) then I get about 30mb/sec on each write (so
> > > 60mb/sec
> > > > combined which is very good).
> > > > does anyone has an idea what could be wrong. Are there any
mount
> > > > options for ocfs2 which could help me? is this a lvm2 or
ocfs2
> > > problem
> > > > (locking,blocking,.... issue) ?
> > > >
> > > > Any help will be appriciated.
> > > >
> > > > yours
> > > >
> > > > Arnold
> > > > --
> > > > Arnold Maderthaner
> > > > J4Care Inc.
> > > >
> > > > _______________________________________________
> > > > Ocfs2-users mailing list
> > > > Ocfs2-users@oss.oracle.com
> > > > http://oss.oracle.com/mailman/listinfo/ocfs2-users
> > >
> >
> >
> >
> > --
> > Arnold Maderthaner
> > J4Care Inc.
>
>
>
>
> --
> Arnold Maderthaner
> J4Care Inc.
>
> ------------------------------
>
> _______________________________________________
> Ocfs2-users mailing list
> Ocfs2-users@oss.oracle.com
> http://oss.oracle.com/mailman/listinfo/ocfs2-users
>
>
--
Arnold Maderthaner
J4Care Inc.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://oss.oracle.com/pipermail/ocfs2-users/attachments/20070824/24d485e4/attachment-0001.html
Alexei_Roudnev
2007-Aug-24 10:32 UTC
[Ocfs2-users] Slow concurrent actions on the same LVM logicalvolume
You better do the same under more heavy load (you had 0 - 1 active tasks while
running this reports).
But look here:
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz
await svctm %util
sdc 0.00 0.00 4.80 4.80 14.40 4.80 2.00
7.21 1560.12 63.67 61.12
This device is waiting for IO to be completed 60% of the time, doing almost
nothing (very littel writing there). Can you check it first of all?
It may be that you have a bad driver or bad device, which experience long delays
during writes. While such thing can be unnoticed on normal file system (because
of bufferring), it can cause a sugnificant delays on clustered file system
(which wait for IO to be completed in many cases).
----- Original Message -----
From: Arnold Maderthaner
To: Alexei_Roudnev
Cc: Sunil Mushran ; ocfs2-users@oss.oracle.com
Sent: Thursday, August 23, 2007 9:53 PM
Subject: Re: [Ocfs2-users] Slow concurrent actions on the same LVM
logicalvolume
vmstat server1:
[root@impax ~]# vmstat 5
procs -----------memory---------- ---swap-- -----io---- --system--
-----cpu------
r b swpd free buff cache si so bi bo in cs us sy id wa
st
0 0 0 2760468 119924 872240 0 0 4 4 107 63 0 0 99 0
0
0 2 0 2760468 119928 872240 0 0 4 30 1095 277 1 0 99 0
0
0 0 0 2760468 119940 872248 0 0 7 11 1076 232 0 0 100
0 0
0 0 0 2760468 119948 872272 0 0 4 18 1084 244 0 0 100
0 0
0 0 0 2760468 119948 872272 0 0 7 34 1063 220 0 0 100
0 0
0 0 0 2760468 119948 872272 0 0 0 1 1086 243 0 0 100
0 0
0 0 0 2760716 119956 872272 0 0 7 10 1065 234 2 0 98 0
0
vmstart server2:
[root@impaxdb ~]# vmstat 5
procs -----------memory---------- ---swap-- -----io---- --system--
-----cpu------
r b swpd free buff cache si so bi bo in cs us sy id wa
st
0 0 24676 143784 46428 3652848 0 0 11 20 107 122 0 0 84 16
0
0 1 24676 143784 46452 3652832 0 0 4 14 1080 237 0 0 89 11
0
0 1 24672 143660 46460 3652812 6 0 14 34 1074 246 0 0 90 10
0
0 0 24672 143660 46460 3652852 0 0 7 22 1076 246 0 0 80 20
0
0 1 24672 143660 46476 3652844 0 0 7 10 1068 227 0 0 85 15
0
iostat server1:
[root@impax ~]# iostat -x 5
Linux 2.6.18-8.el5PAE (impax) 08/24/2007
avg-cpu: %user %nice %system %iowait %steal %idle
0.07 0.00 0.04 0.45 0.00 99.44
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz
avgqu-sz await svctm %util
sda 0.24 0.90 0.23 0.76 15.72 13.33 29.22
0.01 12.68 4.77 0.47
sdb 0.03 1.23 0.02 0.58 1.34 6.81 13.59
0.00 0.79 0.30 0.02
sdc 0.58 0.87 3.88 3.78 16.69 10.87 3.60
7.98 1042.08 89.69 68.68
dm-0 0.00 0.00 0.31 1.14 0.96 6.99 5.46
1.55 1061.28 459.00 66.92
dm-1 0.00 0.00 0.31 0.31 0.95 0.31 2.01
0.66 1054.16 1054.17 66.15
dm-2 0.00 0.00 0.31 0.31 0.95 0.31 2.01
0.66 1057.72 1057.84 66.24
dm-3 0.00 0.00 0.96 0.34 6.15 0.55 5.12
0.77 592.01 504.72 66.04
dm-4 0.00 0.00 0.32 0.33 0.96 0.41 2.14
0.66 1031.13 1028.75 66.00
dm-5 0.00 0.00 0.31 0.32 0.95 0.40 2.12
0.66 1037.27 1036.55 66.09
dm-6 0.00 0.00 0.31 0.31 0.94 0.31 2.01
0.66 1064.34 1064.45 66.37
dm-7 0.00 0.00 0.32 0.31 0.95 0.32 2.01
0.66 1043.05 1042.99 65.77
dm-8 0.00 0.00 0.31 0.31 0.95 0.32 2.03
0.66 1050.73 1048.23 65.97
dm-9 0.00 0.00 0.34 0.31 0.99 0.31 2.01
0.67 1027.41 1006.68 65.71
dm-10 0.00 0.00 0.32 0.32 0.95 0.32 2.01
0.68 1078.71 1050.22 66.41
dm-12 0.00 0.00 0.31 0.31 0.96 0.31 2.03
0.66 1065.33 1065.10 66.43
dm-13 0.00 0.00 0.04 1.27 1.27 2.54 2.91
0.00 2.43 0.10 0.01
dm-18 0.00 0.00 0.01 0.35 0.04 2.80 8.00
0.01 25.42 0.03 0.00
dm-19 0.00 0.00 0.00 0.18 0.01 1.47 8.00
0.00 0.42 0.21 0.00
dm-21 0.00 0.00 0.02 0.01 2.20 0.11 65.49
0.00 9.84 3.01 0.01
dm-26 0.00 0.00 0.01 0.05 0.12 0.43 9.21
0.00 11.66 7.05 0.04
dm-27 0.00 0.00 0.00 0.21 0.02 1.70 8.02
0.00 12.09 4.57 0.10
dm-28 0.00 0.00 0.26 0.11 8.90 0.91 26.40
0.01 14.94 1.85 0.07
dm-29 0.00 0.00 0.05 1.03 1.04 8.22 8.56
0.01 13.76 2.48 0.27
avg-cpu: %user %nice %system %iowait %steal %idle
0.00 0.00 0.00 0.40 0.00 99.60
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz
avgqu-sz await svctm %util
sda 0.00 2.80 0.00 2.20 0.00 40.00 18.18
0.03 12.27 4.73 1.04
sdb 0.00 0.40 0.00 0.40 0.00 1.60 4.00
0.00 5.50 5.50 0.22
sdc 0.00 0.00 4.80 4.80 14.40 4.80 2.00
7.21 1560.12 63.67 61.12
dm-0 0.00 0.00 0.40 0.40 1.20 0.40 2.00
0.60 1559.50 750.25 60.02
dm-1 0.00 0.00 0.40 0.40 1.20 0.40 2.00
0.60 1558.50 750.50 60.04
dm-2 0.00 0.00 0.40 0.40 1.20 0.40 2.00
0.60 1559.50 752.75 60.22
dm-3 0.00 0.00 0.40 0.40 1.20 0.40 2.00
0.60 1562.75 752.25 60.18
dm-4 0.00 0.00 0.40 0.40 1.20 0.40 2.00
0.60 1559.50 752.50 60.20
dm-5 0.00 0.00 0.40 0.40 1.20 0.40 2.00
0.60 1563.00 750.25 60.02
dm-6 0.00 0.00 0.40 0.40 1.20 0.40 2.00
0.60 1558.50 750.25 60.02
dm-7 0.00 0.00 0.40 0.40 1.20 0.40 2.00
0.60 1559.50 752.50 60.20
dm-8 0.00 0.00 0.40 0.40 1.20 0.40 2.00
0.60 1559.75 750.25 60.02
dm-9 0.00 0.00 0.40 0.40 1.20 0.40 2.00
0.60 1554.00 750.25 60.02
dm-10 0.00 0.00 0.40 0.40 1.20 0.40 2.00
0.60 1555.75 752.25 60.18
dm-12 0.00 0.00 0.40 0.40 1.20 0.40 2.00
0.60 1571.25 750.25 60.02
dm-13 0.00 0.00 0.00 0.80 0.00 1.60 2.00
0.01 8.25 2.75 0.22
dm-18 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
dm-19 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
dm-21 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
dm-26 0.00 0.00 0.00 0.60 0.00 4.80 8.00
0.01 12.67 8.33 0.50
dm-27 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
dm-28 0.00 0.00 0.00 1.80 0.00 14.40 8.00
0.02 10.22 2.44 0.44
dm-29 0.00 0.00 0.00 0.80 0.00 6.40 8.00
0.01 10.25 3.75 0.30
avg-cpu: %user %nice %system %iowait %steal %idle
0.05 0.00 0.00 0.00 0.00 99.95
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz
avgqu-sz await svctm %util
sda 0.00 0.20 0.00 0.60 0.00 6.40 10.67
0.01 14.67 6.00 0.36
sdb 0.00 0.40 0.00 0.40 0.00 1.60 4.00
0.00 0.00 0.00 0.00
sdc 0.00 0.00 4.80 4.80 14.40 4.80 2.00
8.81 1292.92 77.06 73.98
dm-0 0.00 0.00 0.20 0.40 0.60 0.40 1.67
0.74 1723.33 1233.00 73.98
dm-1 0.00 0.00 0.20 0.40 0.60 0.40 1.67
0.73 1722.00 1223.33 73.40
dm-2 0.00 0.00 0.20 0.40 0.60 0.40 1.67
0.73 1726.33 1222.00 73.32
dm-3 0.00 0.00 0.20 0.40 0.60 0.40 1.67
0.73 1725.67 1222.00 73.32
dm-4 0.00 0.00 0.20 0.40 0.60 0.40 1.67
0.73 1726.00 1222.00 73.32
dm-5 0.00 0.00 0.20 0.40 0.60 0.40 1.67
0.73 1722.33 1222.00 73.32
dm-6 0.00 0.00 0.20 0.40 0.60 0.40 1.67
0.73 1722.00 1223.67 73.42
dm-7 0.00 0.00 0.20 0.40 0.60 0.40 1.67
0.73 1726.00 1222.00 73.32
dm-8 0.00 0.00 0.20 0.40 0.60 0.40 1.67
0.73 1722.33 1222.00 73.32
dm-9 0.00 0.00 0.20 0.40 0.60 0.40 1.67
0.73 1723.00 1214.33 72.86
dm-10 0.00 0.00 0.20 0.40 0.60 0.40 1.67
0.73 1725.67 1222.00 73.32
dm-12 0.00 0.00 0.20 0.40 0.60 0.40 1.67
0.74 1722.00 1231.67 73.90
dm-13 0.00 0.00 0.00 0.80 0.00 1.60 2.00
0.00 0.00 0.00 0.00
dm-18 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
dm-19 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
dm-21 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
dm-26 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
dm-27 0.00 0.00 0.00 0.60 0.00 4.80 8.00
0.01 13.00 4.33 0.26
dm-28 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
dm-29 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
iostat server2:
[root@impaxdb ~]# iostat -x 5
Linux 2.6.18-8.el5PAE (impaxdb) 08/24/2007
avg-cpu: %user %nice %system %iowait %steal %idle
0.48 0.00 0.10 15.70 0.00 83.72
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz
avgqu-sz await svctm %util
sda 0.28 1.38 0.32 0.82 18.52 17.59 31.55
0.00 1.98 0.83 0.10
sdb 0.01 1.73 0.02 0.04 3.20 14.15 297.21
0.00 9.18 0.87 0.01
sdc 6.44 14.59 3.94 4.26 65.36 126.12 23.36
9.52 1160.63 91.33 74.88
dm-0 0.00 0.00 0.29 0.29 0.89 0.29 2.01
0.71 1214.34 1214.01 71.18
dm-1 0.00 0.00 0.29 0.29 0.89 0.29 2.01
0.71 1206.47 1206.38 70.95
dm-2 0.00 0.00 0.29 0.29 0.89 0.29 2.01
0.71 1207.35 1207.27 70.96
dm-3 0.00 0.00 7.10 9.84 55.33 76.63 7.79
25.51 1506.27 43.35 73.42
dm-4 0.00 0.00 0.32 5.48 1.10 41.82 7.39
9.21 1586.36 122.41 71.06
dm-5 0.00 0.00 0.30 0.88 0.90 5.02 5.02
0.72 606.80 600.72 70.89
dm-6 0.00 0.00 0.29 0.29 0.89 0.29 2.01
0.71 1206.90 1206.94 70.91
dm-7 0.00 0.00 0.29 0.29 0.89 0.29 2.01
0.71 1202.65 1202.69 70.82
dm-8 0.00 0.00 0.29 0.29 0.89 0.29 2.01
0.71 1204.47 1204.43 70.84
dm-9 0.00 0.00 0.29 0.29 0.89 0.29 2.01
0.71 1202.38 1202.42 70.77
dm-10 0.00 0.00 0.29 0.29 0.88 0.29 2.01
0.71 1208.53 1208.48 70.93
dm-12 0.00 0.00 0.30 0.29 0.91 0.29 2.04
0.71 1202.82 1202.49 70.92
dm-13 0.00 0.00 0.00 0.00 0.01 0.00 6.99
0.00 0.47 0.06 0.00
dm-18 0.00 0.00 0.02 1.77 3.16 14.15 9.67
0.03 16.69 0.03 0.01
dm-19 0.00 0.00 0.00 0.00 0.01 0.00 8.02
0.00 0.69 0.34 0.00
dm-20 0.00 0.00 0.05 0.28 3.48 2.26 17.18
0.00 3.37 0.47 0.02
dm-25 0.00 0.00 0.01 0.05 0.16 0.44 9.68
0.00 1.20 0.65 0.00
dm-26 0.00 0.00 0.00 0.14 0.02 1.11 8.03
0.00 3.28 0.05 0.00
dm-27 0.00 0.00 0.31 0.15 9.69 1.24 23.61
0.00 9.39 0.88 0.04
dm-28 0.00 0.00 0.06 1.08 1.24 8.61 8.65
0.00 1.04 0.21 0.02
avg-cpu: %user %nice %system %iowait %steal %idle
0.05 0.00 0.05 23.62 0.00 76.27
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz
avgqu-sz await svctm %util
sda 0.00 4.85 0.00 2.22 0.00 56.57 25.45
0.00 0.91 0.36 0.08
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
sdc 0.00 1.21 4.85 5.45 14.55 24.24 3.76
12.51 1556.98 94.24 97.09
dm-0 0.00 0.00 0.20 0.40 0.61 0.40 1.67
0.97 2055.67 1602.00 97.09
dm-1 0.00 0.00 0.20 0.40 0.61 0.40 1.67
0.96 2064.67 1589.00 96.30
dm-2 0.00 0.00 0.20 0.40 0.61 0.40 1.67
0.96 2063.67 1582.67 95.92
dm-3 0.00 0.00 0.20 0.40 0.61 0.40 1.67
4.79 8988.33 1582.00 95.88
dm-4 0.00 0.00 0.20 1.21 0.61 6.87 5.29
0.96 886.57 679.14 96.04
dm-5 0.00 0.00 0.20 1.21 0.61 6.87 5.29
0.96 878.00 680.00 96.16
dm-6 0.00 0.00 0.20 0.40 0.61 0.40 1.67
0.97 2055.67 1593.00 96.55
dm-7 0.00 0.00 0.20 0.40 0.61 0.40 1.67
0.96 2064.67 1584.33 96.02
dm-8 0.00 0.00 0.20 0.40 0.61 0.40 1.67
0.96 2064.67 1586.33 96.14
dm-9 0.00 0.00 0.20 0.40 0.61 0.40 1.67
0.96 2064.67 1583.33 95.96
dm-10 0.00 0.00 0.20 0.40 0.61 0.40 1.67
0.97 2055.67 1597.67 96.83
dm-12 0.00 0.00 0.20 0.40 0.61 0.40 1.67
0.97 2071.67 1595.67 96.71
dm-13 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
dm-18 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
dm-19 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
dm-20 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
dm-25 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
dm-26 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
dm-27 0.00 0.00 0.00 0.81 0.00 6.46 8.00
0.00 0.00 0.00 0.00
dm-28 0.00 0.00 0.00 4.85 0.00 38.79 8.00
0.01 2.17 0.17 0.08
avg-cpu: %user %nice %system %iowait %steal %idle
0.00 0.00 0.00 19.85 0.00 80.15
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz
avgqu-sz await svctm %util
sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
sdc 0.00 0.61 0.00 0.00 0.00 0.00 0.00
9.12 0.00 0.00 80.26
dm-0 0.00 0.00 0.20 0.00 0.61 0.00 3.00
0.65 0.00 3196.00 64.70
dm-1 0.00 0.00 0.20 0.00 0.61 0.00 3.00
0.65 0.00 3235.00 65.49
dm-2 0.00 0.00 0.20 0.00 0.61 0.00 3.00
0.66 0.00 3254.00 65.87
dm-3 0.00 0.00 0.40 0.81 2.23 6.48 7.17
3.29 0.00 660.83 80.26
dm-4 0.00 0.00 0.20 0.00 0.61 0.00 3.00
0.66 0.00 3248.00 65.75
dm-5 0.00 0.00 0.20 0.00 0.61 0.00 3.00
0.66 0.00 3242.00 65.63
dm-6 0.00 0.00 0.20 0.00 0.61 0.00 3.00
0.65 0.00 3223.00 65.24
dm-7 0.00 0.00 0.20 0.00 0.61 0.00 3.00
0.66 0.00 3249.00 65.77
dm-8 0.00 0.00 0.20 0.00 0.61 0.00 3.00
0.66 0.00 3243.00 65.65
dm-9 0.00 0.00 0.20 0.00 0.61 0.00 3.00
0.66 0.00 3252.00 65.83
dm-10 0.00 0.00 0.20 0.00 0.61 0.00 3.00
0.65 0.00 3209.00 64.96
dm-12 0.00 0.00 0.20 0.00 0.61 0.00 3.00
0.65 0.00 3215.00 65.08
dm-13 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
dm-18 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
dm-19 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
dm-20 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
dm-25 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
dm-26 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
dm-27 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
dm-28 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
top server1:
Tasks: 217 total, 1 running, 214 sleeping, 0 stopped, 2 zombie
Cpu(s): 0.1%us, 0.0%sy, 0.0%ni, 99.4%id, 0.4%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 4084444k total, 1323728k used, 2760716k free, 120072k buffers
Swap: 8193020k total, 0k used, 8193020k free, 872284k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
14245 root 16 0 2288 988 704 R 2 0.0 0:00.01 top
1 root 15 0 2036 652 564 S 0 0.0 0:01.13 init
2 root RT 0 0 0 0 S 0 0.0 0:00.00 migration/0
3 root 34 19 0 0 0 S 0 0.0 0:00.46 ksoftirqd/0
4 root RT 0 0 0 0 S 0 0.0 0:00.00 watchdog/0
5 root RT 0 0 0 0 S 0 0.0 0:00.00 migration/1
6 root 34 19 0 0 0 S 0 0.0 0:00.07 ksoftirqd/1
7 root RT 0 0 0 0 S 0 0.0 0:00.00 watchdog/1
8 root RT 0 0 0 0 S 0 0.0 0:00.00 migration/2
9 root 34 19 0 0 0 S 0 0.0 0:00.19 ksoftirqd/2
10 root RT 0 0 0 0 S 0 0.0 0:00.00 watchdog/2
11 root RT 0 0 0 0 S 0 0.0 0:00.00 migration/3
12 root 34 19 0 0 0 S 0 0.0 0:00.23 ksoftirqd/3
13 root RT 0 0 0 0 S 0 0.0 0:00.00 watchdog/3
14 root 10 -5 0 0 0 S 0 0.0 0:00.00 events/0
15 root 10 -5 0 0 0 S 0 0.0 0:00.00 events/1
16 root 10 -5 0 0 0 S 0 0.0 0:00.00 events/2
17 root 10 -5 0 0 0 S 0 0.0 0:00.00 events/3
18 root 19 -5 0 0 0 S 0 0.0 0:00.00 khelper
19 root 10 -5 0 0 0 S 0 0.0 0:00.00 kthread
25 root 10 -5 0 0 0 S 0 0.0 0: 00.00 kblockd/0
26 root 10 -5 0 0 0 S 0 0.0 0:00.04 kblockd/1
27 root 10 -5 0 0 0 S 0 0.0 0:00.00 kblockd/2
28 root 10 -5 0 0 0 S 0 0.0 0: 00.06 kblockd/3
29 root 14 -5 0 0 0 S 0 0.0 0:00.00 kacpid
147 root 14 -5 0 0 0 S 0 0.0 0:00.00 cqueue/0
148 root 14 -5 0 0 0 S 0 0.0 0:00.00 cqueue/1
149 root 14 -5 0 0 0 S 0 0.0 0:00.00 cqueue/2
150 root 14 -5 0 0 0 S 0 0.0 0:00.00 cqueue/3
153 root 10 -5 0 0 0 S 0 0.0 0:00.00 khubd
155 root 10 -5 0 0 0 S 0 0.0 0:00.00 kseriod
231 root 19 0 0 0 0 S 0 0.0 0:00.00 pdflush
232 root 15 0 0 0 0 S 0 0.0 0:00.15 pdflush
233 root 14 -5 0 0 0 S 0 0.0 0:00.00 kswapd0
234 root 14 -5 0 0 0 S 0 0.0 0:00.00 aio/0
235 root 14 -5 0 0 0 S 0 0.0 0:00.00 aio/1
236 root 14 -5 0 0 0 S 0 0.0 0:00.00 aio/2
top server2:
Tasks: 264 total, 1 running, 260 sleeping, 0 stopped, 3 zombie
Cpu(s): 0.5%us, 0.1%sy, 0.0%ni, 83.7%id, 15.7%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 4084444k total, 3941216k used, 143228k free, 46608k buffers
Swap: 8193020k total, 24660k used, 8168360k free, 3652864k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
9507 root 15 0 2292 1020 704 R 2 0.0 0:00.01 top
1 root 15 0 2032 652 564 S 0 0.0 0:01.20 init
2 root RT 0 0 0 0 S 0 0.0 0:00.00 migration/0
3 root 34 19 0 0 0 S 0 0.0 0:00.25 ksoftirqd/0
4 root RT 0 0 0 0 S 0 0.0 0:00.00 watchdog/0
5 root RT 0 0 0 0 S 0 0.0 0:00.00 migration/1
6 root 34 19 0 0 0 S 0 0.0 0:00.42 ksoftirqd/1
7 root RT 0 0 0 0 S 0 0.0 0:00.00 watchdog/1
8 root RT 0 0 0 0 S 0 0.0 0:00.00 migration/2
9 root 34 19 0 0 0 S 0 0.0 0:00.56 ksoftirqd/2
10 root RT 0 0 0 0 S 0 0.0 0:00.00 watchdog/2
11 root RT 0 0 0 0 S 0 0.0 0:00.00 migration/3
12 root 39 19 0 0 0 S 0 0.0 0:00.08 ksoftirqd/3
13 root RT 0 0 0 0 S 0 0.0 0:00.00 watchdog/3
14 root 10 -5 0 0 0 S 0 0.0 0:00.03 events/0
15 root 10 -5 0 0 0 S 0 0.0 0:00.00 events/1
16 root 10 -5 0 0 0 S 0 0.0 0:00.02 events/2
17 root 10 -5 0 0 0 S 0 0.0 0:00.02 events/3
18 root 10 -5 0 0 0 S 0 0.0 0:00.00 khelper
19 root 11 -5 0 0 0 S 0 0.0 0:00.00 kthread
25 root 10 -5 0 0 0 S 0 0.0 0: 00.08 kblockd/0
26 root 10 -5 0 0 0 S 0 0.0 0:00.00 kblockd/1
27 root 10 -5 0 0 0 S 0 0.0 0:00.00 kblockd/2
28 root 10 -5 0 0 0 S 0 0.0 0: 00.08 kblockd/3
29 root 18 -5 0 0 0 S 0 0.0 0:00.00 kacpid
147 root 18 -5 0 0 0 S 0 0.0 0:00.00 cqueue/0
148 root 20 -5 0 0 0 S 0 0.0 0:00.00 cqueue/1
149 root 10 -5 0 0 0 S 0 0.0 0:00.00 cqueue/2
150 root 10 -5 0 0 0 S 0 0.0 0:00.00 cqueue/3
153 root 10 -5 0 0 0 S 0 0.0 0:00.00 khubd
155 root 10 -5 0 0 0 S 0 0.0 0:00.00 kseriod
233 root 15 -5 0 0 0 S 0 0.0 0:00.84 kswapd0
234 root 20 -5 0 0 0 S 0 0.0 0:00.00 aio/0
235 root 20 -5 0 0 0 S 0 0.0 0:00.00 aio/1
236 root 20 -5 0 0 0 S 0 0.0 0:00.00 aio/2
237 root 20 -5 0 0 0 S 0 0.0 0:00.00 aio/3
409 root 11 -5 0 0 0 S 0 0.0 0:00.00 kpsmoused
slabtop server1:
Active / Total Objects (% used) : 301060 / 312027 (96.5%)
Active / Total Slabs (% used) : 12055 / 12055 (100.0%)
Active / Total Caches (% used) : 106 / 146 (72.6%)
Active / Total Size (% used) : 45714.46K / 46814.27K (97.7%)
Minimum / Average / Maximum Object : 0.01K / 0.15K / 128.00K
OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
148176 148095 99% 0.05K 2058 72 8232K buffer_head
44573 44497 99% 0.13K 1537 29 6148K dentry_cache
32888 32878 99% 0.48K 4111 8 16444K ext3_inode_cache
10878 10878 100% 0.27K 777 14 3108K radix_tree_node
7943 7766 97% 0.02K 47 169 188K dm_io
7917 7766 98% 0.02K 39 203 156K dm_tio
6424 5414 84% 0.09K 146 44 584K vm_area_struct
6384 6223 97% 0.04K 76 84 304K sysfs_dir_cache
5989 5823 97% 0.03K 53 113 212K size-32
5074 4833 95% 0.06K 86 59 344K size-64
4488 4488 100% 0.33K 408 11 1632K inode_cache
2610 2561 98% 0.12K 87 30 348K size-128
2540 1213 47% 0.01K 10 254 40K anon_vma
2380 1475 61% 0.19K 119 20 476K filp
1430 1334 93% 0.35K 130 11 520K proc_inode_cache
1326 1031 77% 0.05K 17 78 68K selinux_inode_security
1320 1304 98% 0.25K 88 15 352K size-256
1015 629 61% 0.02K 5 203 20K biovec-1
1008 125 12% 0.05K 14 72 56K journal_head
930 841 90% 0.12K 31 30 124K bio
920 666 72% 0.19K 46 20 184K skbuff_head_cache
798 757 94% 2.00K 399 2 1596K size-2048
791 690 87% 0.03K 7 113 28K ocfs2_em_ent
784 748 95% 0.50K 98 8 392K size-512
590 503 85% 0.06K 10 59 40K biovec-4
564 562 99% 0.88K 141 4 564K ocfs2_inode_cache
546 324 59% 0.05K 7 78 28K delayacct_cache
531 497 93% 0.43K 59 9 236K shmem_inode_cache
520 487 93% 0.19K 26 20 104K biovec-16
505 327 64% 0.04K 5 101 20K pid
500 487 97% 0.75K 100 5 400K biovec-64
460 376 81% 0.04K 5 92 20K Acpi-Operand
452 104 23% 0.03K 4 113 16K pgd
347 347 100% 4.00K 347 1 1388K size-4096
338 303 89% 0.02K 2 169 8K Acpi-Namespace
336 175 52% 0.04K 4 84 16K crq_pool
slabtop server2:
Active / Total Objects (% used) : 886188 / 907066 (97.7%)
Active / Total Slabs (% used) : 20722 / 20723 (100.0%)
Active / Total Caches (% used) : 104 / 147 (70.7%)
Active / Total Size (% used) : 76455.01K / 78620.30K (97.2%)
Minimum / Average / Maximum Object : 0.01K / 0.09K / 128.00K
OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
736344 729536 99% 0.05K 10227 72 40908K buffer_head
37671 34929 92% 0.13K 1299 29 5196K dentry_cache
32872 32666 99% 0.48K 4109 8 16436K ext3_inode_cache
20118 20097 99% 0.27K 1437 14 5748K radix_tree_node
10912 9437 86% 0.09K 248 44 992K vm_area_struct
7714 7452 96% 0.02K 38 203 152K dm_tio
7605 7452 97% 0.02K 45 169 180K dm_io
6552 6475 98% 0.04K 78 84 312K sysfs_dir_cache
5763 5461 94% 0.03K 51 113 204K size-32
4661 4276 91% 0.06K 79 59 316K size-64
4572 3569 78% 0.01K 18 254 72K anon_vma
4440 3770 84% 0.19K 222 20 888K filp
2288 1791 78% 0.35K 208 11 832K proc_inode_cache
1980 1884 95% 0.12K 66 30 264K size-128
1804 1471 81% 0.33K 164 11 656K inode_cache
1404 1046 74% 0.05K 18 78 72K selinux_inode_security
1060 670 63% 0.19K 53 20 212K skbuff_head_cache
904 767 84% 0.03K 8 113 32K ocfs2_em_ent
870 870 100% 0.12K 29 30 116K bio
840 816 97% 0.50K 105 8 420K size-512
812 578 71% 0.02K 4 203 16K biovec-1
810 800 98% 0.25K 54 15 216K size-256
800 761 95% 2.00K 400 2 1600K size-2048
549 514 93% 0.43K 61 9 244K shmem_inode_cache
546 289 52% 0.05K 7 78 28K delayacct_cache
531 478 90% 0.06K 9 59 36K biovec-4
520 471 90% 0.19K 26 20 104K biovec-16
505 292 57% 0.04K 5 101 20K pid
490 479 97% 0.75K 98 5 392K biovec-64
460 376 81% 0.04K 5 92 20K Acpi-Operand
452 175 38% 0.03K 4 113 16K pgd
440 357 81% 0.38K 44 10 176K sock_inode_cache
369 303 82% 0.44K 41 9 164K UNIX
360 320 88% 1.00K 90 4 360K size-1024
354 124 35% 0.06K 6 59 24K fs_cache
351 351 100% 4.00K 351 1 1404K pmd
uptime server1:
[root@impax ~]# uptime
10:21:09 up 18:16, 1 user, load average: 7.22, 7.72, 7.83
uptime server2:
[root@impaxdb ~]# uptime
10:21:17 up 18:16, 1 user, load average: 8.79, 9.02, 8.98
yours
arnold
On 8/24/07, Alexei_Roudnev <Alexei_Roudnev@exigengroup.com> wrote:
Run and send here, please:
vmstat 5
(5 lines)
iostat -x 5
(2 - 3 outputs)
top
(1 screen)
slabtop (or equivalent, I dont remember)
----- Original Message -----
From: Arnold Maderthaner
To: Sunil Mushran ; ocfs2-users@oss.oracle.com
Sent: Thursday, August 23, 2007 4:12 AM
Subject: Re: [Ocfs2-users] Slow concurrent actions on the same LVM
logicalvolume
Hi !
I watched the servers today and noticed that the server load is very high
but not much is working on the server:
server1:
16:37:11 up 32 min, 1 user, load average: 6.45, 7.90, 7.45
server2:
16:37:19 up 32 min, 4 users, load average: 9.79, 9.76, 8.26
i also tried to do some selects on the oracle database today.
to do a "select count(*) from xyz" where xyz has 120 000 rows it
took me about 10 sec.
always the process who does the filesystem access is the first one.
and every command (ls,du,vgdisplay,lvdisplay,dd,cp,....) which needs
access to a ocfs2 filesystem seams like blocking or it seams to me that there is
a lock.
Can anyone please help me ?
yours
Arnold
On 8/22/07, Arnold Maderthaner <arnold.maderthaner@j4care.com >
wrote:
There is not really a high server load when writing to the disks but
will try it asap. what do you mean with AST ?
Can I enable some debuging on ocfs2 ? We have 2 Dell servers with about
4gb of ram each and 2 Dual Core CPUs with 2GHZ each.
Can you please describe which tests todo again ?
Thx for your help
Arnold
On 8/22/07, Sunil Mushran < Sunil.Mushran@oracle.com > wrote:
Repeat the first test. This time run top on the first server. Which
process is eating the cpu? From the time it appears that the second
server is waiting for the AST that the first server is slow to send.
And
it could be slow because it may be busy flushing the data to disk.
How much memory/cpu do these servers have?
As far as the second test goes, what is the concurrent write
performance
directly to the LV (minus fs).
On Wed, Aug 22, 2007 at 01:14:02PM +0530, Arnold Maderthaner wrote:
> Hi 2 all !
>
> I have problems with concurrent filesystem actions on a ocfs2
> filesystem which is mounted by 2 nodes. OS=RH5ES and OCFS2= 1.2.6
> F.e.: If I have a LV called testlv which is mounted on /mnt on
both
> servers and I do a "dd if=/dev/zero of=/mnt/test.a bs=1024
> count=1000000" on server 1 and do at the same time a du -hs
> /mnt/test.a it takes about 5 seconds for du -hs to execute:
> 270M test.a
>
> real 0m3.627s
> user 0m0.000s
> sys 0m0.002s
>
> or even longer. If i do a du -hs ond a file which is not written
by
> another server its fast:
> 977M test.b
>
> real 0m0.001s
> user 0m0.001s
> sys 0m0.000s
>
> If I write 2 different files from both servers to the same LV I
get
> only 2.8mb/sec.
> If I write 2 different files from both servers to a different LV
> (testlv,test2lv) then I get about 30mb/sec on each write (so
60mb/sec
> combined which is very good).
> does anyone has an idea what could be wrong. Are there any mount
> options for ocfs2 which could help me? is this a lvm2 or ocfs2
problem
> (locking,blocking,.... issue) ?
>
> Any help will be appriciated.
>
> yours
>
> Arnold
> --
> Arnold Maderthaner
> J4Care Inc.
>
> _______________________________________________
> Ocfs2-users mailing list
> Ocfs2-users@oss.oracle.com
> http://oss.oracle.com/mailman/listinfo/ocfs2-users
--
Arnold Maderthaner
J4Care Inc.
--
Arnold Maderthaner
J4Care Inc.
--------------------------------------------------------------------------
_______________________________________________
Ocfs2-users mailing list
Ocfs2-users@oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users
--
Arnold Maderthaner
J4Care Inc.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://oss.oracle.com/pipermail/ocfs2-users/attachments/20070824/d9354b6c/attachment-0001.html
Seemingly Similar Threads
- Filesystem usage after mkfs.ocfs2
- Configuration question for syslinux 3.11 menu and ontimeout option
- Development Question for version 2.11 of isolinux
- no idle CPU ... system hogging it all ...
- How do I 'merge' a altered subset of a data.frame back into the same data.frame