I know this has been discussed before, but I am seeing high iowait times and the occasional deadlock between my two node OCFS2 cluster. I will be turning on "noatime" for the mount on both nodes in a maintenance window, which should help. What I am not sure of, whether that will be of much help. For example, sar is reporting the following from one of the nodes over the last little bit: 10:00:01 AM CPU %user %nice %system %iowait %steal %idle 10:10:01 AM all 2.45 0.00 5.11 23.94 0.00 68.51 10:20:01 AM all 2.61 0.00 5.56 23.50 0.00 68.33 10:30:01 AM all 2.22 0.00 5.31 23.27 0.00 69.20 10:40:01 AM all 2.48 0.00 5.27 23.83 0.00 68.42 10:50:01 AM all 2.53 0.00 5.29 22.02 0.00 70.16 11:00:01 AM all 2.39 0.00 5.66 24.42 0.00 67.53 11:10:01 AM all 2.78 0.00 5.75 30.11 0.00 61.37 11:20:01 AM all 2.90 0.00 5.74 25.83 0.00 65.53 11:30:01 AM all 2.78 0.00 5.61 24.49 0.00 67.12 11:40:01 AM all 2.78 0.00 5.37 24.43 0.00 67.41 11:50:01 AM all 2.64 0.00 5.00 21.90 0.00 70.47 12:00:01 PM all 2.60 0.00 5.41 24.22 0.00 67.78 12:10:01 PM all 2.51 0.00 5.20 23.89 0.00 68.39 12:20:01 PM all 2.50 0.00 5.02 25.70 0.00 66.78 12:30:01 PM all 2.49 0.00 5.24 24.61 0.00 67.67 12:40:01 PM all 2.53 0.00 4.90 24.36 0.00 68.21 12:50:01 PM all 2.32 0.00 4.81 23.18 0.00 69.69 01:00:01 PM all 2.38 0.00 4.92 23.07 0.00 69.64 01:10:01 PM all 2.09 0.00 5.17 23.45 0.00 69.29 01:20:01 PM all 2.63 0.00 5.35 23.24 0.00 68.77 01:30:01 PM all 2.28 0.00 4.63 25.15 0.00 67.93 01:40:01 PM all 2.39 0.00 4.76 20.93 0.00 71.92 01:50:01 PM all 2.24 0.00 4.95 24.81 0.00 68.01 02:00:01 PM all 2.36 0.00 4.99 27.45 0.00 65.19 02:10:01 PM all 3.31 0.00 6.65 43.20 0.00 46.85 02:20:01 PM all 2.88 0.00 6.47 42.47 0.00 48.18 02:30:01 PM all 2.66 0.00 6.61 44.67 0.00 46.06 02:40:01 PM all 2.67 0.00 5.97 43.46 0.00 47.90 The iowait percentage is evening itself back down, but a 44.67% iowait? WOW! Any suggestions? Specs: OEL 5.5 - Kernel 2.6.18-194.8.1.0.1.el5 ocfs2-tools-1.4.4-1.el5 ocfs2-2.6.18-194.8.1.0.1.el5-1.4.7-1.el5 ocfs2console-1.4.4-1.el5 Help or suggestions appreciated. Thanks! -dan ______________________ Dan Lark, RHCE Systems Engineer, Hargray Communications dan.lark at htc.hargray.com Office:?843-706-1819 | Cell: 843-227-2263 7 Arley Way, Suite 200 (2nd Floor) P.O. Box 3380 Bluffton, SC 29910
Sunil Mushran
2010-Sep-07 18:59 UTC
[Ocfs2-users] Extremely high memory usage and iowait times
Can you describe your usage and your setup a bit? On 09/07/2010 11:55 AM, Dan Lark wrote:> I know this has been discussed before, but I am seeing high iowait times and the occasional deadlock between my two node OCFS2 cluster. I will be turning on "noatime" for the mount on both nodes in a maintenance window, which should help. What I am not sure of, whether that will be of much help. > > For example, sar is reporting the following from one of the nodes over the last little bit: > > 10:00:01 AM CPU %user %nice %system %iowait %steal %idle > 10:10:01 AM all 2.45 0.00 5.11 23.94 0.00 68.51 > 10:20:01 AM all 2.61 0.00 5.56 23.50 0.00 68.33 > 10:30:01 AM all 2.22 0.00 5.31 23.27 0.00 69.20 > 10:40:01 AM all 2.48 0.00 5.27 23.83 0.00 68.42 > 10:50:01 AM all 2.53 0.00 5.29 22.02 0.00 70.16 > 11:00:01 AM all 2.39 0.00 5.66 24.42 0.00 67.53 > 11:10:01 AM all 2.78 0.00 5.75 30.11 0.00 61.37 > 11:20:01 AM all 2.90 0.00 5.74 25.83 0.00 65.53 > 11:30:01 AM all 2.78 0.00 5.61 24.49 0.00 67.12 > 11:40:01 AM all 2.78 0.00 5.37 24.43 0.00 67.41 > 11:50:01 AM all 2.64 0.00 5.00 21.90 0.00 70.47 > 12:00:01 PM all 2.60 0.00 5.41 24.22 0.00 67.78 > 12:10:01 PM all 2.51 0.00 5.20 23.89 0.00 68.39 > 12:20:01 PM all 2.50 0.00 5.02 25.70 0.00 66.78 > 12:30:01 PM all 2.49 0.00 5.24 24.61 0.00 67.67 > 12:40:01 PM all 2.53 0.00 4.90 24.36 0.00 68.21 > 12:50:01 PM all 2.32 0.00 4.81 23.18 0.00 69.69 > 01:00:01 PM all 2.38 0.00 4.92 23.07 0.00 69.64 > 01:10:01 PM all 2.09 0.00 5.17 23.45 0.00 69.29 > 01:20:01 PM all 2.63 0.00 5.35 23.24 0.00 68.77 > 01:30:01 PM all 2.28 0.00 4.63 25.15 0.00 67.93 > 01:40:01 PM all 2.39 0.00 4.76 20.93 0.00 71.92 > 01:50:01 PM all 2.24 0.00 4.95 24.81 0.00 68.01 > 02:00:01 PM all 2.36 0.00 4.99 27.45 0.00 65.19 > 02:10:01 PM all 3.31 0.00 6.65 43.20 0.00 46.85 > 02:20:01 PM all 2.88 0.00 6.47 42.47 0.00 48.18 > 02:30:01 PM all 2.66 0.00 6.61 44.67 0.00 46.06 > 02:40:01 PM all 2.67 0.00 5.97 43.46 0.00 47.90 > > The iowait percentage is evening itself back down, but a 44.67% iowait? WOW! Any suggestions? > > Specs: > OEL 5.5 - Kernel 2.6.18-194.8.1.0.1.el5 > ocfs2-tools-1.4.4-1.el5 > ocfs2-2.6.18-194.8.1.0.1.el5-1.4.7-1.el5 > ocfs2console-1.4.4-1.el5 > > Help or suggestions appreciated. > > Thanks! > > -dan > ______________________ > Dan Lark, RHCE > Systems Engineer, Hargray Communications > dan.lark at htc.hargray.com > Office: 843-706-1819 | Cell: 843-227-2263 > 7 Arley Way, Suite 200 (2nd Floor) > P.O. Box 3380 > Bluffton, SC 29910 > > > _______________________________________________ > Ocfs2-users mailing list > Ocfs2-users at oss.oracle.com > http://oss.oracle.com/mailman/listinfo/ocfs2-users >