TL;DR scrub''s ioprio argument isn''t really helpful - a scrub murders system performance til it''s done. My system: 3.11 kernel (from Ubuntu Saucy) btrfs-tools from 2013-07 (from Debian Sid) Opteron 8-core CPU 32GB RAM 4 WD 1TB Black drives in a btrfs RAID10 (data and metadata). iotop shows that the idle priority really is set on the processes: >> me@box:~$ sudo iotop -bn 1 | grep idle >> 28326 idle root 82.06 M/s 0.00 B/s 0.00 % 0.00 % btrfs scrub start -c3 /data >> 28327 idle root 89.30 M/s 0.00 B/s 0.00 % 0.00 % btrfs scrub start -c3 /data >> 28329 idle root 84.47 M/s 0.00 B/s 0.00 % 0.00 % btrfs scrub start -c3 /data >> 28331 idle root 79.64 M/s 0.00 B/s 0.00 % 0.00 % btrfs scrub start -c3 /data But unfortunately, despite being on "idle" priority, the scrub is KILLING system performance. This is one of my eight CPU cores. Any core with a significant amount of non-idle time has three to four times as much wait time as it does user/system time. Brutal: >> Cpu4 : 4.4%us, 16.1%sy, 0.0%ni, 12.4%id, 67.2%wa, 0.0%hi, 0.0%si, 0.0%st Anybody got any ideas that might make the scrub go a little less... enthusiastically... and leave the system more usable? Right now, with a scrub running, the system is so burdened that it may take as much as two to three seconds to so much as display a manpage. God help you if you want to, say, log into a VM running on the system. It''ll GET there, but it''ll look like it''s iSCSI on TCP over carrier pigeon. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On 2013-11-24 22:45, Jim Salter wrote:> TL;DR scrub''s ioprio argument isn''t really helpful - a scrub murders > system performance til it''s done. > > My system: > > 3.11 kernel (from Ubuntu Saucy) > btrfs-tools from 2013-07 (from Debian Sid) > Opteron 8-core CPU > 32GB RAM > 4 WD 1TB Black drives in a btrfs RAID10 (data and metadata). > > iotop shows that the idle priority really is set on the processes: > >>> me@box:~$ sudo iotop -bn 1 | grep idle >>> 28326 idle root 82.06 M/s 0.00 B/s 0.00 % 0.00 % btrfs > scrub start -c3 /data >>> 28327 idle root 89.30 M/s 0.00 B/s 0.00 % 0.00 % btrfs > scrub start -c3 /data >>> 28329 idle root 84.47 M/s 0.00 B/s 0.00 % 0.00 % btrfs > scrub start -c3 /data >>> 28331 idle root 79.64 M/s 0.00 B/s 0.00 % 0.00 % btrfs > scrub start -c3 /data > > But unfortunately, despite being on "idle" priority, the scrub is > KILLING system performance. This is one of my eight CPU cores. Any core > with a significant amount of non-idle time has three to four times as > much wait time as it does user/system time. Brutal: > >>> Cpu4 : 4.4%us, 16.1%sy, 0.0%ni, 12.4%id, 67.2%wa, 0.0%hi, > 0.0%si, 0.0%st > > Anybody got any ideas that might make the scrub go a little less... > enthusiastically... and leave the system more usable? Right now, with a > scrub running, the system is so burdened that it may take as much as two > to three seconds to so much as display a manpage. God help you if you > want to, say, log into a VM running on the system. It''ll GET there, but > it''ll look like it''s iSCSI on TCP over carrier pigeon.A large part of this might be that the idle I/O scheduling class does not put a limit on bandwidth utilization, so even though it is using ''idle'' bandwidth, it is actually using all of the bandwidth that is ''unused'' by other tasks, which kills your disk latency. As i see it, the best option is probably either: 1. set up something to suspend the scrub if there are other pending I/O requests (which would probably need kernel support) 2. allow the user to set a reasonable I/O bandwidth limit on the scrub processes (you could already do this with the BIO cgroup, but it would be nice to not need that to be compiled into the kernel to have this happen) -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Can you elaborate on this please? I''m not directly familiar with cgroups, I''d greatly appreciate a quick-and-dirty example of using BIO cgroup to limit I/O bandwidth. Limiting bandwidth definitely would ameliorate the problem for me; I already use pv''s bw-limiting feature to make btrfs send | btrfs receive tolerable. On 11/25/2013 07:25 AM, Austin S Hemmelgarn wrote:> 2. allow the user to set a reasonable I/O bandwidth limit on the scrub > processes (you could already do this with the BIO cgroup, but it would > be nice to not need that to be compiled into the kernel to have this > happen)-- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On 2013-11-25 07:55, Jim Salter wrote:> Can you elaborate on this please? I''m not directly familiar with > cgroups, I''d greatly appreciate a quick-and-dirty example of using BIO > cgroup to limit I/O bandwidth. > > Limiting bandwidth definitely would ameliorate the problem for me; I > already use pv''s bw-limiting feature to make btrfs send | btrfs receive > tolerable. > > On 11/25/2013 07:25 AM, Austin S Hemmelgarn wrote: >> 2. allow the user to set a reasonable I/O bandwidth limit on the scrub >> processes (you could already do this with the BIO cgroup, but it would >> be nice to not need that to be compiled into the kernel to have this >> happen) >I''m not very clear on the specifics, (the only cgroup I personally use with any regularity is freezer), but I would suggest looking at Documentation/cgroups/blkio-controller.txt in the kernel source tree (cgroups are one of the better subsystems WRT documentation). I''m pretty certain that most distributions have the required kernel options compiled in by default, and automatically mount the needed filesystems. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Sun, 24 Nov 2013 22:45:59 -0500, Jim Salter wrote:> TL;DR scrub''s ioprio argument isn''t really helpful - a scrub murders > system performance til it''s done. > > My system: > > 3.11 kernel (from Ubuntu Saucy)I don''t run Ubuntu, but *maybe* they use the deadline IO scheduler by default, which does not handle IO priorities (by design). So maybe make sure your devices don''t say "deadline" in /sys/block/sdX/queue/scheduler, and if they do, set them to cfq. -h -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html