I know this topic has been discussed many times... but what the hell makes zpool resilvering so slow? I''m running OpenSolaris 2009.06. I have had a large number of problematic disks due to a bad production batch, leading me to resilver quite a few times, progressively replacing each disk as it dies (and now preemptively removing disks.) My complaint is that resilvering ends up taking... days! The average write rate to the disk being resilvered is 1 to 3 MB/sec. You can see zpool status and iostat -v output here: http://pastebin.com/mcbb8dfd When I read files off the zpool, I get quite a few MB/sec even in a degraded state, although the zpool is idle while resilvering in this case - no snapshots or anything happening on it. The system has 3 GB of RAM and a 2.8 GHz dual core CPU which is always >90% idle while resilvering. The number of I/O operations per second is nowhere near the disk''s limits. Scrubbing takes 3-4 hours at the most, so it''s clearly not a read bottleneck. Even if I have a configuration where only one disk is being replaced (and all others are OK), I never pass the 1-3 MB/sec limit. What is going on? I have had to resilver 4 times so far, and I have to resilver at least once more. Each resilvering takes a day or two, and I cant see why... it''s not CPU, it''s not sustained read throughput, it''s not IOPS, so what is it?? Galen
i might be wrong because i''m kind of new but i THINK you need to disable automatic snapshots when resilvering, at least on the older version you did. if not it would restart every time a new snapshot was made....but then again, i may be wrong. On Fri, Jul 10, 2009 at 6:18 PM, Galen <galenz at zinkconsulting.com> wrote:> I know this topic has been discussed many times... but what the hell makes > zpool resilvering so slow? I''m running OpenSolaris 2009.06. > > I have had a large number of problematic disks due to a bad production > batch, leading me to resilver quite a few times, progressively replacing > each disk as it dies (and now preemptively removing disks.) My complaint is > that resilvering ends up taking... days! The average write rate to the disk > being resilvered is 1 to 3 MB/sec. > > You can see zpool status and iostat -v output here: > http://pastebin.com/mcbb8dfd > > When I read files off the zpool, I get quite a few MB/sec even in a > degraded state, although the zpool is idle while resilvering in this case - > no snapshots or anything happening on it. The system has 3 GB of RAM and a > 2.8 GHz dual core CPU which is always >90% idle while resilvering. The > number of I/O operations per second is nowhere near the disk''s limits. > Scrubbing takes 3-4 hours at the most, so it''s clearly not a read > bottleneck. Even if I have a configuration where only one disk is being > replaced (and all others are OK), I never pass the 1-3 MB/sec limit. > > What is going on? I have had to resilver 4 times so far, and I have to > resilver at least once more. Each resilvering takes a day or two, and I cant > see why... it''s not CPU, it''s not sustained read throughput, it''s not IOPS, > so what is it?? > > Galen > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20090711/8bed525b/attachment.html>
On Jul 10, 2009, at 3:32 PM, Miles Nordin wrote:>>>>>> "g" == Galen <galenz at zinkconsulting.com> writes: > > g> the disk being resilvered is 1 to 3 MB/sec. > > see: > > 6592835 resilvering is at least 10x too slow > 6602697 pool has small write throughput during resilver > http://bugs.opensolaris.org/view_bug.do?bug_id=6722540 -- some > mention of the Dirty Time Log > http://blogs.sun.com/ahrens/entry/new_scrub_code > 6698575 -- zfs scrub causes multi-second lag over NFS > > In particular 6333409 in snv_102 is supposed to speed up > scrub/resilver.I''m running OpenSolaris 2009.06, and while those are interesting to read, none of them really explain or solve my issues. -Galenh
On Jul 11, 2009, at 6:10 PM, Miles Nordin wrote:>>>>>> "g" == Galen <galenz at zinkconsulting.com> writes: > > g> I''m running OpenSolaris 2009.06, and while those are > g> interesting to read, none of them really explain or solve my > g> issues. > > read this again: > >>> In particular 6333409 in snv_102 is supposed to speed up >>> scrub/resilver.It is horribly slow and I am running a newer build than that - this is why I say none of them solve or explain. galen at solaribyte:~# uname -a SunOS solaribyte 5.11 snv_111b i86pc i386 i86pc -Galen
I''m still struggling with slow resilvering performance. There doesn''t seem to be any clear bottleneck at this point.. and it''s going glacially slow. scrub: resilver in progress for 11h2m, 27.86% done, 28h35m to go Load averages are like 0.13-0.15 range, CPU usage is <10%, the machine is doing nothing else. zpool iostat -v shows that read/write operations per second on each disk are in the single digit range. As these are decent 7200 RPM 3.5" disks, I know they can do more IOPS than than that. I''ve seen it before. Is there any way to give this process a kick in the pants and speed things up? Because once this resilvering is done, I need to shuffle disks again and resilver at least once more, if not twice... and at this rate, we''re measuring resilvering time in days! Here''s the zpool iostat -v output: galen at solaribyte:~# zpool iostat -v 1 capacity operations bandwidth pool used avail read write read write ---------- ----- ----- ----- ----- ----- ----- olympic 2.15T 2.38T 88 6 10.6M 23.5K raidz2 1.42T 2.21T 67 4 8.16M 14.9K replacing - - 62 8 1.26M 107K c14d0 - - 41 6 1.29M 107K 13143843205485599815 - - 0 0 13 2 c10d0 - - 14 1 1.39M 2.33K replacing - - 67 3 1.36M 2.84K c13d0 - - 44 2 1.39M 2.56K c10d1 - - 0 16 61 1.37M replacing - - 8 61 176K 1.20M 2673037112181665188 - - 0 0 0 0 c11d0 - - 7 57 178K 1.20M c8t0d0 - - 42 1 1.39M 2.49K c12d0 - - 3 0 117K 489 c7t0d0 - - 42 1 1.39M 2.45K c8t1d0 - - 43 1 1.39M 2.36K raidz2 750G 178G 20 2 2.43M 8.67K replacing - - 20 2 1.22M 4.44K c15d1 - - 16 1 1.24M 3.27K 8862062963069576548 - - 0 0 0 0 replacing - - 20 2 1.22M 4.36K c16d1 - - 16 1 1.24M 3.20K 2970292359499355257 - - 0 0 10 1 replacing - - 0 0 0 1.88K 3106783608214265238 - - 0 0 0 0 348116080896813745 - - 0 0 10 1 replacing - - 0 20 0 1.21M 217004158856088173 - - 0 0 10 1 2711430129275390205 - - 0 0 10 1 -- This message posted from opensolaris.org
I''m still struggling with slow resilvering performance. There doesn''t seem to be any clear bottleneck at this point.. and it''s going glacially slow. scrub: resilver in progress for 11h2m, 27.86% done, 28h35m to go Load averages are like 0.13-0.15 range, CPU usage is <10%, the machine is doing nothing else. zpool iostat -v shows that read/write operations per second on each disk are in the single digit range. As these are decent 7200 RPM 3.5" disks, I know they can do more IOPS than than that. I''ve seen it before. Is there any way to give this process a kick in the pants and speed things up? Because once this resilvering is done, I need to shuffle disks again and resilver at least once more, if not twice... and at this rate, we''re measuring resilvering time in days! Here''s the zpool iostat -v output: galen at solaribyte:~# zpool iostat -v 1 capacity operations bandwidth pool used avail read write read write ---------- ----- ----- ----- ----- ----- ----- olympic 2.15T 2.38T 88 6 10.6M 23.5K raidz2 1.42T 2.21T 67 4 8.16M 14.9K replacing - - 62 8 1.26M 107K c14d0 - - 41 6 1.29M 107K 13143843205485599815 - - 0 0 13 2 c10d0 - - 14 1 1.39M 2.33K replacing - - 67 3 1.36M 2.84K c13d0 - - 44 2 1.39M 2.56K c10d1 - - 0 16 61 1.37M replacing - - 8 61 176K 1.20M 2673037112181665188 - - 0 0 0 0 c11d0 - - 7 57 178K 1.20M c8t0d0 - - 42 1 1.39M 2.49K c12d0 - - 3 0 117K 489 c7t0d0 - - 42 1 1.39M 2.45K c8t1d0 - - 43 1 1.39M 2.36K raidz2 750G 178G 20 2 2.43M 8.67K replacing - - 20 2 1.22M 4.44K c15d1 - - 16 1 1.24M 3.27K 8862062963069576548 - - 0 0 0 0 replacing - - 20 2 1.22M 4.36K c16d1 - - 16 1 1.24M 3.20K 2970292359499355257 - - 0 0 10 1 replacing - - 0 0 0 1.88K 3106783608214265238 - - 0 0 0 0 348116080896813745 - - 0 0 10 1 replacing - - 0 20 0 1.21M 217004158856088173 - - 0 0 10 1 2711430129275390205 - - 0 0 10 1 -- This message posted from opensolaris.org