Help solving a performance mystery would be welcome.
I have two test programs:
* one totally CPU bound (it just infinitely loops)
* the other I/O bound (it loops, each time doing
* a write,
* an fsync or fdatasync (I get similar behavior with either),
* an lseek to back up to the same position again
Under ext2, if I run these two together, they each run nearly as fast
as they would alone, since when the disk completes a write, the I/O
bound process has been sleeping long enough to have gained priority
over the CPU bound process, preempts for long enough to submit the next
write request, and then goes back to sleep, letting the CPU bound one
run while the disk does its thing.
Under ext3, I get a totally different, and very mysterious, behavior.
Sometimes the I/O bound process will make very low rates of progress
for tens of seconds. For example, this is evident in the following
"vmstat 1" output. I start with just the I/O bound process running.
Then at some point I start the CPU bound process, and the "bo" column
drops precipitously. After tens of seconds, the "bo" column goes back
up again, for no apparent reason. How long this takes to happens
varies considerably. Sometimes the I/O rate never dips in the first
place, or at most only very briefly.
Can anyone explain this phenomenon? I'm wondering if it has something
to do with the need to schedule two different threads -- my I/O-bound
benchmark and the kjournald. But I can't think why either of them
wouldn't gain priority.
max 30% vmstat 1
procs memory swap io system cpu
r b w swpd free buff cache si so bi bo in cs us sy id
2 0 0 11064 2316 27888 30260 0 3 22 816 788 1441 34 5 61
3 1 1 11196 2332 27936 30224 0 132 0 1156 480 948 93 7 0
1 1 1 11196 2324 28008 30152 0 0 0 1100 346 945 97 3 0
2 0 1 11196 2320 27956 30208 0 0 0 1120 348 772 97 3 0
2 1 1 11196 2304 28032 30144 0 0 0 1160 348 716 99 1 0
1 1 1 11196 2304 28048 30128 0 0 0 1116 366 728 99 1 0
1 1 1 11196 2304 28080 30096 0 0 0 1116 345 691 96 4 0
1 1 1 11196 2304 28000 30176 0 0 0 1148 350 717 98 2 0
1 1 1 11196 2304 28028 30148 0 0 0 1132 342 707 98 2 0
0 1 1 11196 2356 27904 30272 0 0 0 1136 351 758 39 3 58
1 0 0 11196 2356 27904 30272 0 0 0 1152 356 758 1 3 96
2 0 0 11196 2336 27840 30304 0 0 0 820 287 585 34 2 64
2 0 0 11196 2332 27840 30304 0 0 0 92 146 192 99 1 0
2 0 0 11196 2332 27804 30340 0 0 0 76 154 198 100 0 0
2 0 0 11196 2332 27804 30340 0 0 0 64 142 176 99 1 0
2 0 0 11196 2332 27704 30440 0 0 0 64 147 180 100 0 0
2 0 0 11196 2332 27704 30440 0 0 0 80 147 179 100 0 0
4 0 0 11196 2332 27676 30468 0 0 0 64 145 176 99 1 0
3 0 0 11196 2336 27676 30468 0 0 0 64 145 187 99 1 0
2 0 0 11196 2336 27648 30496 0 0 0 72 141 180 100 0 0
2 0 0 11196 2336 27648 30496 0 0 0 68 143 186 99 1 0
4 0 0 11196 2332 27464 30680 0 0 0 64 142 168 100 0 0
3 0 0 11196 2332 27464 30680 0 0 0 72 144 181 100 0 0
3 0 0 11196 2332 27420 30724 0 0 0 64 141 181 99 1 0
2 0 0 11196 2324 27428 30724 0 0 0 64 141 175 99 1 0
2 0 0 11196 2324 27432 30720 0 0 0 72 146 189 99 1 0
3 0 0 11196 2324 27440 30712 0 0 0 64 140 169 99 1 0
3 0 0 11196 2324 27204 30944 0 0 0 64 143 177 99 1 0
3 0 0 11196 2324 27204 30940 0 0 0 72 141 190 99 1 0
2 0 0 11196 2324 27068 31076 0 0 0 64 146 177 99 1 0
3 0 0 11196 2324 27068 31076 0 0 0 64 146 177 99 1 0
2 0 0 11196 2324 26972 31172 0 0 0 72 150 182 98 2 0
2 0 0 11196 2324 26972 31172 0 0 0 64 146 181 98 2 0
2 0 0 11196 2324 26852 31292 0 0 0 64 148 174 100 0 0
2 0 0 11196 2324 26700 31444 0 0 0 76 157 195 99 1 0
2 0 0 11196 2324 26700 31440 0 0 0 64 148 174 99 1 0
2 0 0 11196 2324 26276 31864 0 0 0 64 142 172 99 1 0
4 0 0 11196 2324 26284 31856 0 0 0 72 145 189 99 1 0
2 0 0 11196 2324 26256 31884 0 0 0 64 144 171 99 1 0
3 0 0 11196 2324 26256 31884 0 0 0 64 155 190 99 1 0
2 0 0 11196 2324 26256 31884 0 0 0 72 142 179 100 0 0
2 0 0 11196 2324 26256 31884 0 0 0 64 141 177 98 2 0
5 0 0 11196 2324 26276 31864 0 0 0 64 149 189 100 0 0
2 0 0 11196 2324 26292 31844 0 0 0 72 143 180 100 0 0
2 0 0 11196 2324 26308 31828 0 0 0 68 144 187 99 1 0
2 0 0 11196 2324 26320 31816 0 0 0 64 140 173 99 1 0
3 0 0 11196 2324 26332 31804 0 0 0 72 145 180 100 0 0
2 0 0 11196 2324 26336 31800 0 0 0 60 142 181 99 1 0
2 0 0 11196 2324 26340 31796 0 0 0 60 159 174 99 1 0
2 0 0 11196 2324 26352 31780 0 0 0 72 154 188 99 1 0
2 0 0 11196 2324 26364 31768 0 0 0 60 141 177 99 1 0
3 0 0 11196 2324 26372 31760 0 0 0 64 140 173 99 1 0
3 0 0 11196 2324 26376 31756 0 0 0 72 144 183 99 1 0
3 0 0 11196 2324 26392 31740 0 0 0 60 140 175 99 1 0
4 0 0 11196 2324 26400 31732 0 0 0 64 143 178 99 1 0
3 0 0 11196 2324 26404 31728 0 0 0 72 142 181 99 1 0
3 0 0 11196 2324 26424 31708 0 0 0 64 142 178 99 1 0
4 0 0 11196 2324 26432 31696 0 0 0 64 142 187 99 1 0
3 0 0 11196 2324 26460 31668 0 0 0 72 143 180 99 1 0
3 0 0 11196 2324 26480 31648 0 0 0 68 145 189 99 1 0
4 0 0 11196 2324 26496 31632 0 0 0 64 143 175 100 0 0
3 0 0 11196 2324 26512 31616 0 0 0 72 143 180 99 1 0
3 0 0 11196 2324 25848 32280 0 0 0 64 142 181 99 1 0
3 0 0 11196 2324 25852 32276 0 0 0 72 170 193 100 0 0
3 0 0 11196 2324 25852 32276 0 0 0 72 123 178 99 1 0
4 0 0 11196 2324 25852 32276 0 0 0 64 137 166 99 1 0
4 0 0 11196 2324 25868 32260 0 0 0 68 144 172 99 1 0
6 0 0 11196 2324 25876 32252 0 0 0 72 141 182 99 1 0
5 0 0 11196 2324 25900 32232 0 0 0 64 142 180 99 1 0
4 0 0 11196 2324 25920 32212 0 0 0 68 158 184 99 1 0
3 0 0 11196 2324 25924 32208 0 0 0 72 144 188 99 1 0
5 0 0 11196 2324 25932 32196 0 0 0 64 142 175 99 1 0
5 0 0 11196 2324 25952 32176 0 0 0 68 151 191 99 1 0
5 0 0 11196 2324 25972 32156 0 0 0 72 142 185 99 1 0
4 0 0 11196 2324 26000 32128 0 0 0 72 153 184 98 2 0
3 0 0 11196 2324 26012 32116 0 0 0 68 142 181 100 0 0
4 0 0 11196 2324 26028 32100 0 0 0 72 143 183 99 1 0
5 0 0 11196 2324 26048 32080 0 0 0 64 151 185 100 0 0
5 0 0 11196 2324 26064 32060 0 0 0 72 171 205 98 2 0
3 0 0 11196 2324 26096 32024 0 0 0 72 122 172 99 1 0
3 0 0 11196 2324 26116 32004 0 0 0 64 139 172 99 1 0
3 0 0 11196 2324 26128 31992 0 0 0 68 147 182 99 1 0
3 0 0 11196 2324 26136 31984 0 0 0 72 143 184 98 2 0
3 0 0 11196 2324 26148 31972 0 0 0 64 142 176 98 2 0
4 0 0 11196 2324 26156 31964 0 0 0 68 145 185 99 1 0
3 0 0 11196 2324 26156 31960 0 0 0 72 147 174 99 1 0
3 0 0 11196 2324 26156 31960 0 0 0 64 142 183 99 1 0
4 0 0 11196 2324 26156 31960 0 0 0 68 142 177 99 1 0
3 0 0 11196 2324 26172 31944 0 0 0 72 150 178 98 2 0
4 0 0 11196 2324 26180 31936 0 0 0 64 145 179 97 3 0
4 0 0 11196 2324 26196 31852 0 0 324 68 177 216 99 1 0
2 1 1 11196 2388 25732 31980 24 0 212 652 269 490 99 1 0
2 1 1 11196 2360 25960 31780 0 0 0 1060 332 653 100 0 0
3 0 1 11192 2660 26252 31460 0 0 0 1164 350 739 97 3 0
1 1 1 11192 2568 26488 31316 0 0 0 1172 350 725 99 1 0
1 1 1 11192 2424 26020 31928 0 0 0 1176 359 724 98 2 0
2 1 1 11192 2364 26324 31684 0 0 0 1116 346 704 97 3 0
1 1 1 11192 2364 26584 31424 0 0 0 1116 340 692 98 2 0
2 1 1 11192 2364 26924 31084 0 0 0 1116 541 1280 94 6 0
2 1 1 11192 2364 27192 30816 0 0 0 1148 459 1119 97 3 0
Just a "me too" post... I've noticed somewhat similar behavior, but with a different test. I ran an application that forked off two threads on dual processor workstation. One thread tried to write data once every 2 seconds , and the other tried to read data constantly (although it was writing a tiny bit as well because of file atime updates). Both apps were contending for the same physical device. When using ext3, the writes would be delayed for a fairly substantial arbitrary amount of time, and then they many of them would go through all at once. This pattern would then repeat. When using reiserfs, each write went through about every two seconds. I chalked it up to an overall bandwidth vs. latency optimization tradeoff. Sometimes though, in real world conditions when CPU usage is very high, I've noticed odd latency problems. Sometimes after quitting vi I'll be hanging for 10 seconds waiting to close out the file and fsync, while I can work normally in other terminals. On Tue, 2002-09-10 at 16:06, Max Hailperin wrote:> Help solving a performance mystery would be welcome. > > I have two test programs: > > * one totally CPU bound (it just infinitely loops) > * the other I/O bound (it loops, each time doing > * a write, > * an fsync or fdatasync (I get similar behavior with either), > * an lseek to back up to the same position again > > Under ext2, if I run these two together, they each run nearly as fast > as they would alone, since when the disk completes a write, the I/O > bound process has been sleeping long enough to have gained priority > over the CPU bound process, preempts for long enough to submit the next > write request, and then goes back to sleep, letting the CPU bound one > run while the disk does its thing. > > Under ext3, I get a totally different, and very mysterious, behavior. > Sometimes the I/O bound process will make very low rates of progress > for tens of seconds. For example, this is evident in the following > "vmstat 1" output. I start with just the I/O bound process running. > Then at some point I start the CPU bound process, and the "bo" column > drops precipitously. After tens of seconds, the "bo" column goes back > up again, for no apparent reason. How long this takes to happens > varies considerably. Sometimes the I/O rate never dips in the first > place, or at most only very briefly. > > Can anyone explain this phenomenon? I'm wondering if it has something > to do with the need to schedule two different threads -- my I/O-bound > benchmark and the kjournald. But I can't think why either of them > wouldn't gain priority. > > max 30% vmstat 1 > procs memory swap io system cpu > r b w swpd free buff cache si so bi bo in cs us sy id > 2 0 0 11064 2316 27888 30260 0 3 22 816 788 1441 34 5 61 > 3 1 1 11196 2332 27936 30224 0 132 0 1156 480 948 93 7 0 > 1 1 1 11196 2324 28008 30152 0 0 0 1100 346 945 97 3 0 > 2 0 1 11196 2320 27956 30208 0 0 0 1120 348 772 97 3 0 > 2 1 1 11196 2304 28032 30144 0 0 0 1160 348 716 99 1 0 > 1 1 1 11196 2304 28048 30128 0 0 0 1116 366 728 99 1 0 > 1 1 1 11196 2304 28080 30096 0 0 0 1116 345 691 96 4 0 > 1 1 1 11196 2304 28000 30176 0 0 0 1148 350 717 98 2 0 > 1 1 1 11196 2304 28028 30148 0 0 0 1132 342 707 98 2 0 > 0 1 1 11196 2356 27904 30272 0 0 0 1136 351 758 39 3 58 > 1 0 0 11196 2356 27904 30272 0 0 0 1152 356 758 1 3 96 > 2 0 0 11196 2336 27840 30304 0 0 0 820 287 585 34 2 64 > 2 0 0 11196 2332 27840 30304 0 0 0 92 146 192 99 1 0 > 2 0 0 11196 2332 27804 30340 0 0 0 76 154 198 100 0 0 > 2 0 0 11196 2332 27804 30340 0 0 0 64 142 176 99 1 0 > 2 0 0 11196 2332 27704 30440 0 0 0 64 147 180 100 0 0 > 2 0 0 11196 2332 27704 30440 0 0 0 80 147 179 100 0 0 > 4 0 0 11196 2332 27676 30468 0 0 0 64 145 176 99 1 0 > 3 0 0 11196 2336 27676 30468 0 0 0 64 145 187 99 1 0 > 2 0 0 11196 2336 27648 30496 0 0 0 72 141 180 100 0 0 > 2 0 0 11196 2336 27648 30496 0 0 0 68 143 186 99 1 0 > 4 0 0 11196 2332 27464 30680 0 0 0 64 142 168 100 0 0 > 3 0 0 11196 2332 27464 30680 0 0 0 72 144 181 100 0 0 > 3 0 0 11196 2332 27420 30724 0 0 0 64 141 181 99 1 0 > 2 0 0 11196 2324 27428 30724 0 0 0 64 141 175 99 1 0 > 2 0 0 11196 2324 27432 30720 0 0 0 72 146 189 99 1 0 > 3 0 0 11196 2324 27440 30712 0 0 0 64 140 169 99 1 0 > 3 0 0 11196 2324 27204 30944 0 0 0 64 143 177 99 1 0 > 3 0 0 11196 2324 27204 30940 0 0 0 72 141 190 99 1 0 > 2 0 0 11196 2324 27068 31076 0 0 0 64 146 177 99 1 0 > 3 0 0 11196 2324 27068 31076 0 0 0 64 146 177 99 1 0 > 2 0 0 11196 2324 26972 31172 0 0 0 72 150 182 98 2 0 > 2 0 0 11196 2324 26972 31172 0 0 0 64 146 181 98 2 0 > 2 0 0 11196 2324 26852 31292 0 0 0 64 148 174 100 0 0 > 2 0 0 11196 2324 26700 31444 0 0 0 76 157 195 99 1 0 > 2 0 0 11196 2324 26700 31440 0 0 0 64 148 174 99 1 0 > 2 0 0 11196 2324 26276 31864 0 0 0 64 142 172 99 1 0 > 4 0 0 11196 2324 26284 31856 0 0 0 72 145 189 99 1 0 > 2 0 0 11196 2324 26256 31884 0 0 0 64 144 171 99 1 0 > 3 0 0 11196 2324 26256 31884 0 0 0 64 155 190 99 1 0 > 2 0 0 11196 2324 26256 31884 0 0 0 72 142 179 100 0 0 > 2 0 0 11196 2324 26256 31884 0 0 0 64 141 177 98 2 0 > 5 0 0 11196 2324 26276 31864 0 0 0 64 149 189 100 0 0 > 2 0 0 11196 2324 26292 31844 0 0 0 72 143 180 100 0 0 > 2 0 0 11196 2324 26308 31828 0 0 0 68 144 187 99 1 0 > 2 0 0 11196 2324 26320 31816 0 0 0 64 140 173 99 1 0 > 3 0 0 11196 2324 26332 31804 0 0 0 72 145 180 100 0 0 > 2 0 0 11196 2324 26336 31800 0 0 0 60 142 181 99 1 0 > 2 0 0 11196 2324 26340 31796 0 0 0 60 159 174 99 1 0 > 2 0 0 11196 2324 26352 31780 0 0 0 72 154 188 99 1 0 > 2 0 0 11196 2324 26364 31768 0 0 0 60 141 177 99 1 0 > 3 0 0 11196 2324 26372 31760 0 0 0 64 140 173 99 1 0 > 3 0 0 11196 2324 26376 31756 0 0 0 72 144 183 99 1 0 > 3 0 0 11196 2324 26392 31740 0 0 0 60 140 175 99 1 0 > 4 0 0 11196 2324 26400 31732 0 0 0 64 143 178 99 1 0 > 3 0 0 11196 2324 26404 31728 0 0 0 72 142 181 99 1 0 > 3 0 0 11196 2324 26424 31708 0 0 0 64 142 178 99 1 0 > 4 0 0 11196 2324 26432 31696 0 0 0 64 142 187 99 1 0 > 3 0 0 11196 2324 26460 31668 0 0 0 72 143 180 99 1 0 > 3 0 0 11196 2324 26480 31648 0 0 0 68 145 189 99 1 0 > 4 0 0 11196 2324 26496 31632 0 0 0 64 143 175 100 0 0 > 3 0 0 11196 2324 26512 31616 0 0 0 72 143 180 99 1 0 > 3 0 0 11196 2324 25848 32280 0 0 0 64 142 181 99 1 0 > 3 0 0 11196 2324 25852 32276 0 0 0 72 170 193 100 0 0 > 3 0 0 11196 2324 25852 32276 0 0 0 72 123 178 99 1 0 > 4 0 0 11196 2324 25852 32276 0 0 0 64 137 166 99 1 0 > 4 0 0 11196 2324 25868 32260 0 0 0 68 144 172 99 1 0 > 6 0 0 11196 2324 25876 32252 0 0 0 72 141 182 99 1 0 > 5 0 0 11196 2324 25900 32232 0 0 0 64 142 180 99 1 0 > 4 0 0 11196 2324 25920 32212 0 0 0 68 158 184 99 1 0 > 3 0 0 11196 2324 25924 32208 0 0 0 72 144 188 99 1 0 > 5 0 0 11196 2324 25932 32196 0 0 0 64 142 175 99 1 0 > 5 0 0 11196 2324 25952 32176 0 0 0 68 151 191 99 1 0 > 5 0 0 11196 2324 25972 32156 0 0 0 72 142 185 99 1 0 > 4 0 0 11196 2324 26000 32128 0 0 0 72 153 184 98 2 0 > 3 0 0 11196 2324 26012 32116 0 0 0 68 142 181 100 0 0 > 4 0 0 11196 2324 26028 32100 0 0 0 72 143 183 99 1 0 > 5 0 0 11196 2324 26048 32080 0 0 0 64 151 185 100 0 0 > 5 0 0 11196 2324 26064 32060 0 0 0 72 171 205 98 2 0 > 3 0 0 11196 2324 26096 32024 0 0 0 72 122 172 99 1 0 > 3 0 0 11196 2324 26116 32004 0 0 0 64 139 172 99 1 0 > 3 0 0 11196 2324 26128 31992 0 0 0 68 147 182 99 1 0 > 3 0 0 11196 2324 26136 31984 0 0 0 72 143 184 98 2 0 > 3 0 0 11196 2324 26148 31972 0 0 0 64 142 176 98 2 0 > 4 0 0 11196 2324 26156 31964 0 0 0 68 145 185 99 1 0 > 3 0 0 11196 2324 26156 31960 0 0 0 72 147 174 99 1 0 > 3 0 0 11196 2324 26156 31960 0 0 0 64 142 183 99 1 0 > 4 0 0 11196 2324 26156 31960 0 0 0 68 142 177 99 1 0 > 3 0 0 11196 2324 26172 31944 0 0 0 72 150 178 98 2 0 > 4 0 0 11196 2324 26180 31936 0 0 0 64 145 179 97 3 0 > 4 0 0 11196 2324 26196 31852 0 0 324 68 177 216 99 1 0 > 2 1 1 11196 2388 25732 31980 24 0 212 652 269 490 99 1 0 > 2 1 1 11196 2360 25960 31780 0 0 0 1060 332 653 100 0 0 > 3 0 1 11192 2660 26252 31460 0 0 0 1164 350 739 97 3 0 > 1 1 1 11192 2568 26488 31316 0 0 0 1172 350 725 99 1 0 > 1 1 1 11192 2424 26020 31928 0 0 0 1176 359 724 98 2 0 > 2 1 1 11192 2364 26324 31684 0 0 0 1116 346 704 97 3 0 > 1 1 1 11192 2364 26584 31424 0 0 0 1116 340 692 98 2 0 > 2 1 1 11192 2364 26924 31084 0 0 0 1116 541 1280 94 6 0 > 2 1 1 11192 2364 27192 30816 0 0 0 1148 459 1119 97 3 0 > > > > _______________________________________________ > Ext3-users mailing list > Ext3-users@redhat.com > https://listman.redhat.com/mailman/listinfo/ext3-users
Hi, On Tue, Sep 10, 2002 at 03:06:57PM -0500, Max Hailperin wrote:> Under ext3, I get a totally different, and very mysterious, behavior. > Sometimes the I/O bound process will make very low rates of progress > for tens of seconds.Which kernel is this with? --Stephen
Date: Wed, 11 Sep 2002 00:02:29 +0100 From: "Stephen C. Tweedie" <sct@redhat.com> Hi, On Tue, Sep 10, 2002 at 03:06:57PM -0500, Max Hailperin wrote: > Under ext3, I get a totally different, and very mysterious, behavior. > Sometimes the I/O bound process will make very low rates of progress > for tens of seconds. Which kernel is this with? --Stephen Whoops, I should have included that. My apologies for an uncharacteristically incomplete bug report. Here is my /proc/version: Linux version 2.4.18-3 (bhcompile@daffy.perf.redhat.com) (gcc version 2.96 20000731 (Red Hat Linux 7.3 2.96-110)) #1 Thu Apr 18 07:37:53 EDT 2002 This happens to be on a uniprocessor Pentium II, but I have replicated similar results on other uniprocessor x86 machines. Thanks for any light you can shed. -max
I can't find such a beast and was wondering what a sensible configuration would be for an NFSv3 / SMB file server using quotas on hardware RAID volumes. I reckon this is a pretty typical setup for an academic dept. (I think our poor student fileserver's going to have its work cut out when our lovely students return!) I'm currently using default options on 2.4.19-pre10-ac2, which seems stable but perhaps as not as fast as it can be. I'm thinking of upgrading to 2.4.19-ac4, maybe increasing the journal size (or trying an external journal--is that stable?), maybe a judicious elvtune.. Has anyone any experience they can share?
Reasonably Related Threads
- Fwd: Migrating server
- [Bug 2324] New: remote port forward w/ empty bind_address via multiplexed connection: doc violation
- Asterisk freezes with Fixup failed on channel SIP/...<MASQ>
- [Bug 2332] New: Show more secure fingerprints than MD5 (e.g. SHA256) in ssh and ssh-keygen
- Assertion failure with Dovecot 1.0rc10