thr3ads.net - zfs discuss - [zfs-discuss] Increase resilver priority [Jul 2010]

If this information is useful, please help other people find it:
Share via:

Giovanni Tirloni

2010-Jul-23 09:31 UTC

[zfs-discuss] Increase resilver priority

Hello,

 We''ve seen some resilvers on idle servers that are taking ages. Is it
possible to speed up resilver operations somehow?

 Eg. iostat shows <5MB/s writes on the replaced disks.

 I''m thinking a small performance degradation would be sometimes
better than the increased risk window (where a vdev is degraded).

Thank you,

-- 
Giovanni Tirloni
gtirloni at sysdroid.com

Richard Elling

2010-Jul-23 14:59 UTC

head link

[zfs-discuss] Increase resilver priority

On Jul 23, 2010, at 2:31 AM, Giovanni Tirloni wrote:
> Hello,
> 
> We''ve seen some resilvers on idle servers that are taking ages. Is
it
> possible to speed up resilver operations somehow?
> 
> Eg. iostat shows <5MB/s writes on the replaced disks.
This is lower than I expect, but It may be IOPS bound. What does
iostat say about the IOPS and asvc_t ?
 -- richard

-- 
Richard Elling
richard at nexenta.com   +1-760-896-4422
Enterprise class storage for everyone
www.nexenta.com

Bill Sommerfeld

2010-Jul-23 15:50 UTC

head link

[zfs-discuss] Increase resilver priority

On 07/23/10 02:31, Giovanni Tirloni wrote:>   We''ve seen some resilvers on idle servers that are taking ages.
Is it
> possible to speed up resilver operations somehow?
>
>   Eg. iostat shows<5MB/s writes on the replaced disks.
What build of opensolaris are you running?  There were some recent 
improvements (notably the addition of prefetch to the pool traverse used 
by scrub and resilver) which sped this up significantly for my systems.

Also: if there are large numbers of snapshots, pools seem to take longer 
to resilver, particularly when there''s a lot of metadata divergence 
between snapshots.  Turning off atime updates (if you and your 
applications can cope with this) may also help going forward.

				- Bill

Giovanni Tirloni

2010-Jul-23 16:08 UTC

head link

[zfs-discuss] Increase resilver priority

On Fri, Jul 23, 2010 at 11:59 AM, Richard Elling <richard at nexenta.com>
wrote:> On Jul 23, 2010, at 2:31 AM, Giovanni Tirloni wrote:
>
>> Hello,
>>
>> We''ve seen some resilvers on idle servers that are taking
ages. Is it
>> possible to speed up resilver operations somehow?
>>
>> Eg. iostat shows <5MB/s writes on the replaced disks.
>
> This is lower than I expect, but It may be IOPS bound. What does
> iostat say about the IOPS and asvc_t ?
> ?-- richard
It seems to have improved a bit.

 scrub: resilver in progress for 7h19m, 75.37% done, 2h23m to go
config:

        NAME              STATE     READ WRITE CKSUM
        storage           DEGRADED     0     1     0
          mirror          DEGRADED     0     0     0
            c3t2d0        ONLINE       0     0     0
            replacing     DEGRADED 1.29M     0     0
              c3t3d0s0/o  FAULTED      0     0     0  corrupted data
              c3t3d0      DEGRADED     0     0 1.29M  too many errors
          mirror          ONLINE       0     0     0
            c3t4d0        ONLINE       0     0     0
            c3t5d0        ONLINE       0     0     0
          mirror          DEGRADED     0     0     0
            c3t6d0        ONLINE       0     0     0
            c3t7d0        REMOVED      0     0     0
          mirror          ONLINE       0     0     0
            c3t8d0        ONLINE       0     0     0
            c3t9d0        ONLINE       0     0     0
          mirror          ONLINE       0     0     0
            c3t10d0       ONLINE       0     0     0
            c3t11d0       ONLINE       0     0     0
          mirror          ONLINE       0     0     0
            c3t12d0       ONLINE       0     0     0
            c3t13d0       ONLINE       0     0     0
          mirror          ONLINE       0     0     0
            c3t14d0       ONLINE       0     0     0
            c3t15d0       ONLINE       0     0     0
          mirror          ONLINE       0     0     0
            c3t16d0       ONLINE       0     0     0
            c3t17d0       ONLINE       0     0     0
          mirror          ONLINE       0     0     0
            c3t18d0       ONLINE       0     0     0
            c3t19d0       ONLINE       0     0     0
          mirror          ONLINE       0     0     0
            c3t20d0       ONLINE       0     0     0
            c3t21d0       ONLINE       0     0     0
        logs              DEGRADED     0     1     0
          mirror          ONLINE       0     0     0
            c3t1d0        ONLINE       0     0     0
            c3t22d0       ONLINE       0     0     0


                    extended device statistics
    r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
  582.2  864.3 68925.2 37511.2  0.0 52.7    0.0   36.4   0 610 c3
    0.0  201.7    0.0  806.7  0.0  0.0    0.0    0.1   0   2 c3t0d0
    0.0  268.2    0.0 10531.2  0.0  0.1    0.0    0.4   0  10 c3t1d0
  144.1    0.0 18375.9    0.0  0.0  9.5    0.0   65.7   0 100 c3t2d0
   79.5  125.2 10109.9 15634.3  0.0 35.0    0.0  171.0   0 100 c3t3d0
   10.9    0.0 1181.3    0.0  0.0  0.1    0.0   13.3   0  10 c3t4d0
   19.9    0.0 2120.6    0.0  0.0  0.3    0.0   15.6   0  19 c3t5d0
   35.8    0.0 3819.5    0.0  0.0  0.6    0.0   18.1   0  28 c3t6d0
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 c3t7d0
   22.9    0.0 2506.6    0.0  0.0  0.5    0.0   22.0   0  22 c3t8d0
   15.9    0.0 1639.8    0.0  0.0  0.3    0.0   20.5   0  15 c3t9d0
   23.8    0.0 2889.6    0.0  0.0  0.5    0.0   19.8   0  27 c3t10d0
   21.9    0.0 2558.3    0.0  0.0  0.6    0.0   28.9   0  19 c3t11d0
   32.8    0.0 3151.9    0.0  0.0  1.2    0.0   37.4   0  25 c3t12d0
   25.8    0.0 2707.8    0.0  0.0  0.5    0.0   18.8   0  26 c3t13d0
   19.9    0.0 2281.1    0.0  0.0  0.3    0.0   17.5   0  24 c3t14d0
   23.8    0.0 2782.3    0.0  0.0  0.3    0.0   14.6   0  20 c3t15d0
   18.9    0.0 2249.8    0.0  0.0  0.4    0.0   19.7   0  23 c3t16d0
   21.9    0.0 2519.5    0.0  0.0  0.5    0.0   22.6   0  27 c3t17d0
   12.9    0.0 1653.2    0.0  0.0  0.2    0.0   16.8   0  18 c3t18d0
   26.8    0.0 3262.7    0.0  0.0  0.8    0.0   28.4   0  29 c3t19d0
    9.9    0.0 1271.7    0.0  0.0  0.1    0.0   14.3   0  13 c3t20d0
   14.9    0.0 1843.9    0.0  0.0  0.3    0.0   20.5   0  19 c3t21d0
    0.0  269.2    0.0 10539.0  0.0  0.4    0.0    1.3   0  33 c3t22d0
                    extended device statistics
    r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
  405.1  745.2 51057.2 29893.8  0.0 53.3    0.0   46.4   0 457 c3
    0.0  252.1    0.0 1008.3  0.0  0.0    0.0    0.1   0   2 c3t0d0
    0.0  177.1    0.0 5485.8  0.0  0.0    0.0    0.3   0   4 c3t1d0
  145.0    0.0 18438.0    0.0  0.0 15.1    0.0  104.0   0 100 c3t2d0
   80.0  140.0 10147.8 17925.9  0.0 35.0    0.0  159.0   0 100 c3t3d0
    8.0    0.0 1024.3    0.0  0.0  0.2    0.0   19.9   0  12 c3t4d0
    7.0    0.0  768.3    0.0  0.0  0.1    0.0   15.5   0  11 c3t5d0
   24.0    0.0 3009.0    0.0  0.0  0.5    0.0   19.0   0  25 c3t6d0
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 c3t7d0
    4.0    0.0  512.2    0.0  0.0  0.1    0.0   13.6   0   4 c3t8d0
    7.0    0.0  896.3    0.0  0.0  0.1    0.0   11.8   0   8 c3t9d0
   12.0    0.0 1536.5    0.0  0.0  0.2    0.0   15.7   0  16 c3t10d0
   11.0    0.0 1408.5    0.0  0.0  0.2    0.0   20.3   0  15 c3t11d0
   14.0    0.0 1728.6    0.0  0.0  0.2    0.0   17.5   0  16 c3t12d0
    9.0    0.0 1152.4    0.0  0.0  0.1    0.0   11.3   0  10 c3t13d0
   14.0    0.0 1792.6    0.0  0.0  0.2    0.0   17.4   0  20 c3t14d0
    9.0    0.0 1152.4    0.0  0.0  0.1    0.0   12.8   0  12 c3t15d0
    9.0    0.0 1088.4    0.0  0.0  0.1    0.0   12.0   0  11 c3t16d0
    6.0    0.0  768.3    0.0  0.0  0.1    0.0   11.6   0   6 c3t17d0
   11.0    0.0 1280.4    0.0  0.0  0.2    0.0   15.9   0  17 c3t18d0
   12.0    0.0 1472.5    0.0  0.0  0.2    0.0   16.7   0  16 c3t19d0
    6.0    0.0  768.3    0.0  0.0  0.1    0.0   17.9   0   9 c3t20d0
   17.0    0.0 2112.7    0.0  0.0  0.3    0.0   18.1   0  21 c3t21d0
    0.0  176.1    0.0 5473.8  0.0  0.2    0.0    1.3   0  22 c3t22d0



-- 
Giovanni Tirloni
gtirloni at sysdroid.com

Giovanni Tirloni

2010-Jul-23 16:09 UTC

head link

[zfs-discuss] Increase resilver priority

On Fri, Jul 23, 2010 at 12:50 PM, Bill Sommerfeld
<bill.sommerfeld at oracle.com> wrote:> On 07/23/10 02:31, Giovanni Tirloni wrote:
>>
>> ?We''ve seen some resilvers on idle servers that are taking
ages. Is it
>> possible to speed up resilver operations somehow?
>>
>> ?Eg. iostat shows<5MB/s writes on the replaced disks.
>
> What build of opensolaris are you running? ?There were some recent
> improvements (notably the addition of prefetch to the pool traverse used by
> scrub and resilver) which sped this up significantly for my systems.
b111. Thanks for the heads up regarding these improvements, I''ll try
that in b134.
> Also: if there are large numbers of snapshots, pools seem to take longer to
> resilver, particularly when there''s a lot of metadata divergence
between
> snapshots. ?Turning off atime updates (if you and your applications can
cope
> with this) may also help going forward.
There are 7 snapshots and atime is disabled.

-- 
Giovanni Tirloni
gtirloni at sysdroid.com

zfs discuss - Jul 2010 - Increase resilver priority

[zfs-discuss] Increase resilver priority

[zfs-discuss] Increase resilver priority

[zfs-discuss] Increase resilver priority

[zfs-discuss] Increase resilver priority

[zfs-discuss] Increase resilver priority