Roger Heflin
2007-Jun-07 13:47 UTC
[Lustre-discuss] Odd performance behavior on write test
I am running RHEL4U4.08 equivalent kernel with 1.6.0.1 of Lustre.
I have 2 OST''s setup with 3 partitions/raid5 units on each,
setup to stripe across all 6, I tested both with 4M and 8M stripe
sizes across all 6. The metadata server is on one of the 2
OST''s.
I am running bonnie on 6 separate clients (all remote-across
DDR infiniband).
According to "vmstat 10" on both OST''s I see the 10 second IO
rate change from 100-140MB/second up to 210-240MB/second at about
1/2 of the way through the first write test. I tried with both
noop and deadline, and the shape of the graph is similar, though
slightly higher in both regions with noop. I have ran the
tests several different ways, and the results appear to be
repeatable, and appear to speedup on both OST''s at the same
time.
Any ideas of what exactly is happening at this point? This point
is about 1/2 of the way into the bonnie intelligent write test, so
nothing in the test has changed (the stupid write test is disabled),
the event appears to happen 200-300 seconds into the test every
time.
The cpu speed is not changing, and nothing else that I can find
is changing to explain the speedup.
Any ideas?
Roger
0 0 0 6642336 769268 160904 0 0 0 0 1108 2027 0 0
100 0
0 0 0 6642336 769272 160900 0 0 0 22 1109 2016 0
0 100 0
Start of write test:
0 3 0 6633672 770776 160824 0 0 0 118952 3255 11444 0
13 68 20
1 3 0 6632392 771068 160872 0 0 0 150574 1741 4490 0
4 65 31
1 3 0 6631376 771384 160828 0 0 0 132292 1621 4021 0
4 69 27
10 2 0 6630800 771876 160880 0 0 0 145800 1742 4576 0
4 71 26
0 3 0 6630352 772304 160860 0 0 0 117165 1615 4061 0
3 72 25
0 3 0 6629840 772776 160864 0 0 0 128718 1670 4248 0
4 71 26
0 3 0 6629392 773288 160896 0 0 0 130684 1675 4251 0
4 65 32
0 3 0 6628792 773724 160868 0 0 0 126655 1663 4248 0
3 68 29
1 3 0 6627960 774176 160892 0 0 0 104198 1569 3880 0
3 73 24
procs -----------memory---------- ---swap-- -----io---- --system--
----cpu----
r b swpd free buff cache si so bi bo in cs us
sy id wa
2 3 0 6627368 774864 160816 0 0 0 157648 1792 4752 0
4 65 31
1 3 0 6627120 775320 160768 0 0 0 108000 1574 3903 0
3 70 27
0 3 0 6626752 775748 160884 0 0 0 127970 1658 4250 0
3 72 24
0 3 0 6626464 776172 160868 0 0 0 103917 1564 3878 0
3 73 24
1 3 0 6625816 776688 160828 0 0 0 144541 1741 4544 0
4 72 25
1 3 0 6625096 777104 160888 0 0 0 100322 1548 3636 0
5 69 26
1 3 0 6624920 777532 160800 0 0 0 106761 1567 3881 0
3 72 26
1 3 0 6624296 778052 160824 0 0 0 131802 1669 4320 0
3 72 25
0 3 0 6623912 778516 160836 0 0 0 108140 1587 3948 0
3 66 32
1 3 0 6623288 779056 160840 0 0 0 119576 1643 4204 0
3 73 25
1 3 0 6622864 779472 160832 0 0 0 126676 1662 4233 0
3 70 27
2 3 0 6622080 779876 160836 0 0 0 115972 1598 4005 0
3 66 30
1 3 0 6621736 780216 160904 0 0 0 99236 1558 3771 0
3 62 35
1 3 0 6621392 780812 160852 0 0 0 132058 1685 4378 0
3 63 34
1 3 0 6620888 781404 160804 0 0 0 135839 1703 4415 0
3 67 29
1 3 0 6620576 781672 160808 0 0 0 68690 1408 3220 0
2 73 25
1 3 0 6619968 782192 160764 0 0 0 120592 1633 4161 0
3 72 25
1 3 0 6619512 782696 160872 0 0 0 113932 1618 4095 0
3 73 24
1 3 0 6619032 783156 160888 0 0 0 133218 1665 4284 0
3 72 25
1 3 0 6618616 783540 160844 0 0 0 94863 1532 3746 0
2 72 25
1 3 0 6618256 783980 160880 0 0 0 122657 1662 4265 0
3 69 28
Jump in IO rate, context switches, and interrupts-stays until test complete.
0 3 0 6617304 784820 160856 0 0 0 221593 2056 5833 0
5 70 25
1 3 0 6616304 785868 160828 0 0 0 234390 2146 6302 0
6 70 24
1 3 0 6614600 786912 160804 0 0 0 226641 2105 5908 0
8 68 24
1 3 0 6613680 787912 160892 0 0 0 239477 2131 5998 0
6 67 27
0 3 0 6612824 788868 160888 0 0 0 215342 2013 5608 0
5 71 24
0 4 0 6611800 789916 160792 0 0 0 235022 2125 6113 0
6 67 27
1 3 0 6610800 790860 160868 0 0 0 226365 2105 6059 0
6 69 25
0 3 0 6610032 791708 160836 0 0 0 215070 2045 5786 0
5 70 24
1 3 0 6609008 792736 160828 0 0 0 222035 2072 5939 0
6 70 24
1 3 0 6607728 793848 160804 0 0 0 241741 2169 6315 0
6 68 26
1 3 0 6606912 794752 160784 0 0 0 221420 2068 5933 0
6 69 25
0 3 0 6606240 795536 160884 0 0 0 222700 2086 5975 0
5 69 26