Robert Milkowski
2007-Jan-24 23:39 UTC
[zfs-discuss] Why replacing a drive generates writes to other disks?
Hello zfs-discuss,
Subject says it all.
I first checked - no IO activity at all to the pool named thumper-2.
So I started replacing one drive with ''zpool replace thumper-2 c7t7d0
c4t1d0''.
Now the question is why am I seeing writes to other disks than c7t7d0?
Also why in case of replacing a disk we do not just copy disk-to-disk?
It would be MUCH faster here. Probably ''coz we''re traversing
meta-data? But perhaps it could be done in a clever way so we endup
just copying from one disk to another. Checking parity or checksum
here it''s not necessary - scrub is for it. What we want in most cases
is to replace drive as fast as possible.
On another thumper I have a failing drive (port resets, etc.) so I
issued over a week ago drive replacement. Well it still hasn''t
completed even 4% in a week! The pool config is the same. It''s just
waaaay to slow and in a long term risky.
bash-3.00# zpool status
pool: thumper-2
state: ONLINE
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scrub: resilver in progress, 0,01% done, 350h29m to go
config:
NAME STATE READ WRITE CKSUM
thumper-2 ONLINE 0 0 0
raidz2 ONLINE 0 0 0
c0t0d0 ONLINE 0 0 0
c1t0d0 ONLINE 0 0 0
c4t0d0 ONLINE 0 0 0
c6t0d0 ONLINE 0 0 0
c7t0d0 ONLINE 0 0 0
c0t1d0 ONLINE 0 0 0
c1t1d0 ONLINE 0 0 0
c5t1d0 ONLINE 0 0 0
c6t1d0 ONLINE 0 0 0
c7t1d0 ONLINE 0 0 0
c0t2d0 ONLINE 0 0 0
raidz2 ONLINE 0 0 0
c1t2d0 ONLINE 0 0 0
c5t2d0 ONLINE 0 0 0
c6t2d0 ONLINE 0 0 0
c7t2d0 ONLINE 0 0 0
c0t4d0 ONLINE 0 0 0
c1t4d0 ONLINE 0 0 0
c4t4d0 ONLINE 0 0 0
c6t4d0 ONLINE 0 0 0
c7t4d0 ONLINE 0 0 0
c0t3d0 ONLINE 0 0 0
c1t3d0 ONLINE 0 0 0
raidz2 ONLINE 0 0 0
c4t3d0 ONLINE 0 0 0
c5t3d0 ONLINE 0 0 0
c6t3d0 ONLINE 0 0 0
c7t3d0 ONLINE 0 0 0
c0t5d0 ONLINE 0 0 0
c1t5d0 ONLINE 0 0 0
c4t5d0 ONLINE 0 0 0
c5t5d0 ONLINE 0 0 0
c6t5d0 ONLINE 0 0 0
c7t5d0 ONLINE 0 0 0
c0t6d0 ONLINE 0 0 0
raidz2 ONLINE 0 0 0
c1t6d0 ONLINE 0 0 0
c4t6d0 ONLINE 0 0 0
c5t6d0 ONLINE 0 0 0
c6t6d0 ONLINE 0 0 0
c7t6d0 ONLINE 0 0 0
c0t7d0 ONLINE 0 0 0
c1t7d0 ONLINE 0 0 0
c4t7d0 ONLINE 0 0 0
c5t7d0 ONLINE 0 0 0
c6t7d0 ONLINE 0 0 0
spare ONLINE 0 0 0
c7t7d0 ONLINE 0 0 0
c4t1d0 ONLINE 0 0 0
spares
c4t1d0 INUSE currently in use
c4t2d0 AVAIL
errors: No known data errors
pool: zones
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
zones ONLINE 0 0 0
mirror ONLINE 0 0 0
c5t0d0s4 ONLINE 0 0 0
c5t4d0s4 ONLINE 0 0 0
errors: No known data errors
bash-3.00#
# iostat -xnz 1
[...]
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
114.0 0.0 7232.3 0.0 5.9 0.8 51.9 6.7 74 76 c0t0d0
132.0 0.0 8320.6 0.0 9.0 1.0 68.4 7.5 95 98 c6t0d0
123.0 0.0 7807.7 0.0 7.3 0.8 59.3 6.3 76 77 c7t0d0
115.0 0.0 7296.3 0.0 7.9 0.8 68.7 7.1 80 81 c4t0d0
100.0 0.0 6336.4 0.0 3.6 0.6 36.3 6.0 56 60 c6t1d0
0.0 297.0 0.0 151.0 0.0 0.0 0.0 0.2 0 5 c4t1d0
106.0 0.0 6720.3 0.0 5.3 0.6 50.0 6.1 63 65 c7t1d0
122.0 0.0 7743.7 0.0 6.9 0.7 56.8 6.0 72 73 c0t1d0
120.0 0.0 7679.2 0.0 5.6 0.7 46.9 5.7 66 68 c1t1d0
4.0 0.0 129.5 0.0 0.0 0.0 0.0 4.8 0 2 c1t2d0
3.0 0.0 65.5 0.0 0.0 0.0 3.0 6.4 1 2 c7t2d0
109.0 0.0 6912.3 0.0 6.6 0.7 60.7 6.1 65 66 c0t2d0
74.0 0.0 4672.5 0.0 2.4 0.5 31.9 6.7 45 49 c4t3d0
5.0 0.0 130.0 0.0 0.0 0.0 0.8 7.4 0 4 c0t3d0
82.0 0.0 5121.5 0.0 2.6 0.5 31.9 6.3 49 52 c7t3d0
3.0 0.0 65.5 0.0 0.0 0.0 1.0 5.4 0 2 c0t4d0
4.0 0.0 130.0 0.0 0.0 0.0 0.0 5.4 0 2 c4t4d0
5.0 0.0 130.5 0.0 0.0 0.0 1.2 6.1 1 3 c7t4d0
66.0 0.0 4097.6 0.0 1.7 0.4 25.5 6.2 37 41 c0t5d0
60.0 0.0 3650.6 0.0 1.4 0.4 24.0 6.1 34 37 c4t5d0
54.0 0.0 3266.2 0.0 1.3 0.3 24.0 6.4 31 35 c7t5d0
69.0 0.0 4289.1 0.0 2.2 0.4 32.6 6.3 40 43 c0t6d0
0.0 296.0 0.0 150.0 0.0 0.0 0.0 0.1 0 4 c4t6d0
0.0 296.0 0.0 150.0 0.0 0.0 0.0 0.1 0 4 c7t6d0
5.0 0.0 130.0 0.0 0.0 0.0 1.2 3.7 1 2 c6t2d0
0.0 289.0 0.0 147.0 0.0 0.0 0.0 0.1 0 4 c0t7d0
0.0 298.0 0.0 152.0 0.0 0.0 0.0 0.1 0 4 c4t7d0
0.0 297.0 0.0 151.0 0.0 0.0 0.0 0.1 0 4 c7t7d0
111.0 0.0 7103.3 0.0 5.7 0.7 51.1 6.3 66 70 c5t1d0
78.0 0.0 4802.0 0.0 2.3 0.5 28.9 6.2 46 48 c6t3d0
5.0 0.0 130.5 0.0 0.0 0.0 0.0 6.4 0 3 c5t2d0
5.0 0.0 131.0 0.0 0.0 0.0 1.0 6.4 0 3 c6t4d0
4.0 0.0 129.5 0.0 0.0 0.0 0.7 3.6 0 1 c1t3d0
70.0 0.0 4353.6 0.0 2.0 0.5 28.0 6.5 42 45 c5t3d0
57.0 0.0 3458.6 0.0 1.3 0.4 23.6 6.4 33 36 c6t5d0
4.0 0.0 130.0 0.0 0.0 0.0 0.0 4.2 0 2 c1t4d0
0.0 290.0 0.0 147.5 0.0 0.0 0.0 0.1 0 4 c6t6d0
64.0 0.0 3906.6 0.0 1.4 0.4 22.6 6.2 36 40 c1t5d0
0.0 292.0 0.0 148.5 0.0 0.0 0.0 0.2 0 4 c1t6d0
62.0 0.0 3778.6 0.0 1.7 0.4 26.9 6.1 35 38 c5t5d0
0.0 285.0 0.0 144.5 0.0 0.0 0.0 0.1 0 4 c1t7d0
0.0 292.0 0.0 147.5 0.0 0.0 0.0 0.1 0 4 c5t6d0
0.0 300.0 0.0 152.5 0.0 0.0 0.0 0.2 0 5 c5t7d0
0.0 294.0 0.0 150.0 0.0 0.0 0.0 0.1 0 4 c6t7d0
112.0 0.0 7104.3 0.0 5.6 0.7 50.3 6.1 66 68 c1t0d0
bash-3.00# zpool iostat 1
capacity operations bandwidth
pool used avail read write read write
---------- ----- ----- ----- ----- ----- -----
thumper-2 15,5T 4,54T 226 64 1,34M 401K
zones 16,6G 423G 0 0 8,13K 22,6K
---------- ----- ----- ----- ----- ----- -----
thumper-2 15,5T 4,54T 512 418 1,57M 1,24M
zones 16,6G 423G 0 0 0 0
---------- ----- ----- ----- ----- ----- -----
thumper-2 15,5T 4,54T 607 402 1,83M 1,21M
zones 16,6G 423G 0 0 0 0
---------- ----- ----- ----- ----- ----- -----
thumper-2 15,5T 4,54T 399 446 1,23M 1,32M
zones 16,6G 423G 0 0 0 0
---------- ----- ----- ----- ----- ----- -----
thumper-2 15,5T 4,54T 175 430 547K 1,30M
zones 16,6G 423G 0 0 0 0
---------- ----- ----- ----- ----- ----- -----
^C
bash-3.00#
--
Best regards,
Robert mailto:rmilkowski at task.gda.pl
http://milek.blogspot.com
Brian Hechinger
2007-Jan-25 02:02 UTC
[zfs-discuss] Why replacing a drive generates writes to other disks?
On Thu, Jan 25, 2007 at 12:39:25AM +0100, Robert Milkowski wrote:> Hello zfs-discuss, > > On another thumper I have a failing drive (port resets, etc.) so I > issued over a week ago drive replacement. Well it still hasn''t > completed even 4% in a week! The pool config is the same. It''s just > waaaay to slow and in a long term risky.The last time I saw something like this was on a D1000 that had serious parity issues. Overall it spent so much time retrying and backing down the transfer rate that the data path to the disks was so slow as to be unusable. Got new cables and the problem went away. Don''t know if that applies to you or not. -brian
Matthew Ahrens
2007-Mar-14 08:00 UTC
[zfs-discuss] Why replacing a drive generates writes to other disks?
Robert Milkowski wrote:> Hello zfs-discuss, > > Subject says it all. > > > I first checked - no IO activity at all to the pool named thumper-2. > So I started replacing one drive with ''zpool replace thumper-2 c7t7d0 > c4t1d0''. > > Now the question is why am I seeing writes to other disks than c7t7d0?Are you *sure* that nothing else is going on? Not even atime updates? Do ''zfs umount -a'' and see if there''s still writes to other disks. There may be some small amount of writes to update some metadata with the resilvering status. I just did this yesterday on a raidz2 pool and didn''t see writes to the other disks. Maybe the code has changed since you tried?> Also why in case of replacing a disk we do not just copy disk-to-disk? > It would be MUCH faster here. Probably ''coz we''re traversing > meta-data? But perhaps it could be done in a clever way so we endup > just copying from one disk to another. Checking parity or checksum > here it''s not necessary - scrub is for it. What we want in most cases > is to replace drive as fast as possible.In some cases it would be faster but others not. For example, if the pool is not very full, it would be slower. Also if the disk you''re replacing is not available, a straight disk-to-disk copy would not be possible.> On another thumper I have a failing drive (port resets, etc.) so I > issued over a week ago drive replacement. Well it still hasn''t > completed even 4% in a week! The pool config is the same. It''s just > waaaay to slow and in a long term risky.Are you taking snapshots? They cause scrubbing / resilvering to restart (this is bug 6343667). --matt
Robert Milkowski
2007-Mar-22 16:50 UTC
[zfs-discuss] Why replacing a drive generates writes to other disks?
Hello Matthew, Wednesday, March 14, 2007, 9:00:28 AM, you wrote: MA> Robert Milkowski wrote:>> Hello zfs-discuss, >> >> Subject says it all. >> >> >> I first checked - no IO activity at all to the pool named thumper-2. >> So I started replacing one drive with ''zpool replace thumper-2 c7t7d0 >> c4t1d0''. >> >> Now the question is why am I seeing writes to other disks than c7t7d0?MA> Are you *sure* that nothing else is going on? Not even atime updates? MA> Do ''zfs umount -a'' and see if there''s still writes to other disks. MA> There may be some small amount of writes to update some metadata with MA> the resilvering status. MA> I just did this yesterday on a raidz2 pool and didn''t see writes to the MA> other disks. Maybe the code has changed since you tried? There''re data in thumper-8 but no activity at all is happening to local disk on a server. I did check it with iostat and zpool iostat for dozen of seconds - no IOs at all. bash-3.00# zpool replace thumper-8 c7t7d0 c4t1d0 bash-3.00# zpool status pool: misc state: ONLINE scrub: none requested config: NAME STATE READ WRITE CKSUM misc ONLINE 0 0 0 mirror ONLINE 0 0 0 c5t0d0s4 ONLINE 0 0 0 c5t4d0s4 ONLINE 0 0 0 errors: No known data errors pool: thumper-8 state: ONLINE status: One or more devices is currently being resilvered. The pool will continue to function, possibly in a degraded state. action: Wait for the resilver to complete. scrub: resilver in progress, 0.12% done, 9h1m to go config: NAME STATE READ WRITE CKSUM thumper-8 ONLINE 0 0 0 raidz2 ONLINE 0 0 0 c0t0d0 ONLINE 0 0 0 c1t0d0 ONLINE 0 0 0 c4t0d0 ONLINE 0 0 0 c6t0d0 ONLINE 0 0 0 c7t0d0 ONLINE 0 0 0 c0t1d0 ONLINE 0 0 0 c1t1d0 ONLINE 0 0 0 c5t1d0 ONLINE 0 0 0 c6t1d0 ONLINE 0 0 0 c7t1d0 ONLINE 0 0 0 c0t2d0 ONLINE 0 0 0 raidz2 ONLINE 0 0 0 c1t2d0 ONLINE 0 0 0 c5t2d0 ONLINE 0 0 0 c6t2d0 ONLINE 0 0 0 c7t2d0 ONLINE 0 0 0 c0t4d0 ONLINE 0 0 0 c1t4d0 ONLINE 0 0 0 c4t4d0 ONLINE 0 0 0 c6t4d0 ONLINE 0 0 0 c7t4d0 ONLINE 0 0 0 c0t3d0 ONLINE 0 0 0 c1t3d0 ONLINE 0 0 0 raidz2 ONLINE 0 0 0 c4t3d0 ONLINE 0 0 0 c5t3d0 ONLINE 0 0 0 c6t3d0 ONLINE 0 0 0 c7t3d0 ONLINE 0 0 0 c0t5d0 ONLINE 0 0 0 c1t5d0 ONLINE 0 0 0 c4t5d0 ONLINE 0 0 0 c5t5d0 ONLINE 0 0 0 c6t5d0 ONLINE 0 0 0 c7t5d0 ONLINE 0 0 0 c0t6d0 ONLINE 0 0 0 raidz2 ONLINE 0 0 0 c1t6d0 ONLINE 0 0 0 c4t6d0 ONLINE 0 0 0 c5t6d0 ONLINE 0 0 0 c6t6d0 ONLINE 0 0 0 c7t6d0 ONLINE 0 0 0 c0t7d0 ONLINE 0 0 0 c1t7d0 ONLINE 0 0 0 c4t7d0 ONLINE 0 0 0 c5t7d0 ONLINE 0 0 0 c6t7d0 ONLINE 0 0 0 spare ONLINE 0 0 0 c7t7d0 ONLINE 0 0 0 c4t1d0 ONLINE 0 0 0 spares c4t1d0 INUSE currently in use c4t2d0 AVAIL errors: No known data errors bash-3.00# bash-3.00# iostat -xnz 1 [stripped out first output] extended device statistics r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device 58.2 0.0 195.2 0.0 0.0 0.1 0.3 1.6 2 10 c4t0d0 60.2 0.0 218.7 0.0 0.0 0.1 0.6 1.7 4 10 c6t0d0 72.2 0.0 239.8 0.0 0.0 0.1 0.2 1.4 1 10 c0t0d0 0.0 335.1 0.0 1114.8 0.0 0.1 0.1 0.2 2 5 c4t1d0 62.2 0.0 221.2 0.0 0.0 0.1 0.0 1.5 0 9 c6t1d0 66.2 0.0 207.2 0.0 0.0 0.1 0.0 1.4 0 9 c0t1d0 66.2 0.0 214.2 0.0 0.0 0.1 0.1 1.5 0 10 c5t1d0 16.1 0.0 45.7 0.0 0.0 0.0 0.0 1.3 0 2 c6t2d0 45.2 0.0 233.3 0.0 0.0 0.1 0.2 2.2 1 10 c4t3d0 70.2 0.0 231.3 0.0 0.0 0.1 0.4 2.0 2 14 c0t2d0 18.1 0.0 46.7 0.0 0.0 0.0 0.0 2.5 0 5 c5t2d0 34.1 0.0 182.6 0.0 0.0 0.1 0.3 2.7 1 9 c6t3d0 39.1 0.0 222.8 0.0 0.0 0.1 0.7 2.1 3 8 c5t3d0 21.1 0.0 48.7 0.0 0.0 0.0 0.0 1.9 0 4 c0t3d0 24.1 0.0 72.2 0.0 0.0 0.1 0.0 2.6 0 6 c4t4d0 15.1 0.0 52.7 0.0 0.0 0.0 0.0 2.9 0 4 c6t4d0 23.1 0.0 64.2 0.0 0.0 0.1 0.3 2.2 1 5 c0t4d0 47.2 0.0 249.3 0.0 0.0 0.1 0.0 1.5 0 7 c4t5d0 39.1 0.0 207.7 0.0 0.0 0.1 0.4 1.4 1 5 c6t5d0 66.2 0.0 206.7 0.0 0.0 0.1 0.4 1.5 2 10 c1t0d0 38.1 0.0 199.7 0.0 0.0 0.1 0.2 1.6 1 6 c0t5d0 49.2 223.8 1126.8 123.4 1.6 0.3 5.9 1.3 30 35 c4t6d0 39.1 0.0 222.8 0.0 0.0 0.1 0.5 1.9 1 7 c5t5d0 65.2 0.0 221.2 0.0 0.0 0.1 0.4 1.8 2 12 c1t1d0 45.2 202.7 1127.8 108.9 1.5 0.4 5.9 1.5 35 37 c6t6d0 42.1 0.0 216.7 0.0 0.0 0.1 0.2 1.8 1 8 c0t6d0 44.1 210.7 1141.9 114.9 1.5 0.4 6.0 1.4 35 37 c4t7d0 39.1 217.7 1126.8 124.9 1.8 0.4 7.0 1.5 35 38 c5t6d0 15.1 0.0 45.2 0.0 0.0 0.0 0.7 3.0 1 4 c1t2d0 40.1 182.6 947.7 97.8 1.5 0.3 6.7 1.5 32 34 c6t7d0 58.2 204.7 1136.3 115.9 1.3 0.3 5.0 1.2 28 32 c0t7d0 37.1 178.6 1067.1 97.8 2.0 0.4 9.3 1.8 37 38 c5t7d0 17.1 0.0 46.2 0.0 0.0 0.0 0.0 1.4 0 2 c1t3d0 25.1 0.0 57.7 0.0 0.0 0.0 0.0 1.3 0 3 c1t4d0 44.1 0.0 217.7 0.0 0.0 0.0 0.0 1.0 0 4 c1t5d0 43.1 217.7 1024.5 118.4 2.7 0.4 10.2 1.7 43 44 c1t6d0 59.2 209.7 1145.4 114.4 1.2 0.3 4.3 1.0 26 28 c1t7d0 65.2 0.0 215.2 0.0 0.0 0.1 0.2 2.0 2 13 c7t0d0 61.2 0.0 219.2 0.0 0.0 0.1 0.4 1.3 2 8 c7t1d0 15.1 0.0 52.7 0.0 0.0 0.0 0.0 1.4 0 2 c7t2d0 37.1 0.0 191.6 0.0 0.0 0.1 0.3 2.1 1 8 c7t3d0 13.0 0.0 29.1 0.0 0.0 0.0 0.0 1.3 0 2 c7t4d0 42.1 0.0 224.3 0.0 0.0 0.1 0.1 1.3 0 5 c7t5d0 41.1 191.6 1079.7 102.8 1.7 0.3 7.1 1.5 30 34 c7t6d0 0.0 335.1 0.0 1114.8 0.0 0.1 0.1 0.1 2 5 c7t7d0 extended device statistics r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device 52.0 0.0 109.0 0.0 0.0 0.1 0.2 1.2 1 6 c4t0d0 38.0 0.0 80.0 0.0 0.0 0.0 0.1 1.0 1 4 c6t0d0 41.0 0.0 111.5 0.0 0.0 0.1 0.4 1.4 1 6 c0t0d0 0.0 336.0 0.0 1579.0 0.0 0.1 0.1 0.2 2 5 c4t1d0 41.0 0.0 111.0 0.0 0.0 0.0 0.1 1.1 0 4 c6t1d0 40.0 0.0 103.5 0.0 0.0 0.1 0.7 1.4 2 6 c0t1d0 43.0 0.0 112.0 0.0 0.0 0.1 0.5 2.1 2 9 c5t1d0 32.0 0.0 53.5 0.0 0.0 0.0 0.0 1.0 0 3 c6t2d0 44.0 0.0 277.5 0.0 0.0 0.0 0.3 0.9 1 4 c4t3d0 41.0 0.0 111.5 0.0 0.0 0.0 0.5 1.2 1 5 c0t2d0 33.0 0.0 54.0 0.0 0.0 0.0 0.0 1.4 0 5 c5t2d0 50.0 0.0 287.5 0.0 0.0 0.0 0.5 1.0 2 5 c6t3d0 43.0 0.0 262.0 0.0 0.0 0.1 0.4 1.8 1 8 c5t3d0 34.0 0.0 47.0 0.0 0.0 0.0 0.0 1.0 0 3 c0t3d0 36.0 0.0 63.0 0.0 0.0 0.1 0.0 1.6 0 6 c4t4d0 35.0 0.0 63.0 0.0 0.0 0.0 0.0 1.1 0 4 c6t4d0 39.0 0.0 49.5 0.0 0.0 0.1 0.0 1.7 0 6 c0t4d0 46.0 0.0 255.5 0.0 0.0 0.1 0.4 1.6 1 7 c4t5d0 44.0 0.0 269.5 0.0 0.0 0.1 0.7 2.4 1 11 c6t5d0 49.0 0.0 116.0 0.0 0.0 0.1 0.2 1.4 1 7 c1t0d0 48.0 0.0 271.5 0.0 0.0 0.1 0.5 2.5 2 12 c0t5d0 44.0 212.0 1509.0 114.0 1.5 0.3 5.7 1.0 25 27 c4t6d0 43.0 0.0 246.5 0.0 0.0 0.1 0.7 1.6 1 7 c5t5d0 43.0 0.0 112.5 0.0 0.0 0.1 0.2 1.5 1 6 c1t1d0 42.0 209.0 1516.5 113.0 1.6 0.3 6.4 1.0 23 26 c6t6d0 42.0 0.0 276.0 0.0 0.1 0.1 1.3 1.8 3 7 c0t6d0 40.0 210.0 1501.0 112.0 2.3 0.3 9.0 1.2 28 30 c4t7d0 47.0 214.0 1516.5 113.5 1.7 0.3 6.6 1.1 27 29 c5t6d0 33.0 0.0 54.0 0.0 0.0 0.0 0.0 1.1 0 4 c1t2d0 51.0 207.0 1407.0 117.0 1.5 0.3 5.7 1.1 28 30 c6t7d0 43.0 214.0 1515.0 116.5 2.6 0.3 10.3 1.3 32 34 c0t7d0 42.0 224.0 1525.0 125.0 2.2 0.3 8.4 1.2 29 31 c5t7d0 37.0 0.0 71.0 0.0 0.0 0.0 0.0 1.2 0 4 c1t3d0 33.0 0.0 69.0 0.0 0.0 0.0 0.0 1.2 0 4 c1t4d0 49.0 0.0 272.0 0.0 0.0 0.1 0.3 1.4 1 7 c1t5d0 46.0 205.0 1356.5 114.5 2.6 0.3 10.5 1.3 31 33 c1t6d0 40.0 210.0 1501.0 112.0 2.5 0.3 9.8 1.3 30 31 c1t7d0 34.0 0.0 84.5 0.0 0.0 0.0 0.1 1.0 0 3 c7t0d0 40.0 0.0 103.5 0.0 0.0 0.1 0.5 1.7 2 7 c7t1d0 37.0 0.0 48.5 0.0 0.0 0.0 0.0 1.2 0 5 c7t2d0 50.0 0.0 272.5 0.0 0.1 0.1 1.3 1.7 5 8 c7t3d0 31.0 0.0 53.5 0.0 0.0 0.1 0.0 2.0 0 6 c7t4d0 38.0 0.0 244.5 0.0 0.0 0.1 0.7 1.6 1 6 c7t5d0 48.0 212.0 1571.0 118.0 2.9 0.3 11.1 1.2 30 32 c7t6d0 0.0 339.0 0.0 1579.0 0.0 0.1 0.1 0.2 2 5 c7t7d0 extended device statistics r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device 82.0 0.0 289.0 0.0 0.0 0.1 0.4 1.7 3 14 c4t0d0 85.0 0.0 313.0 0.0 0.0 0.1 0.2 1.3 2 11 c6t0d0 80.0 0.0 273.0 0.0 0.0 0.1 0.1 1.6 1 13 c0t0d0 0.0 438.0 0.0 1692.5 0.0 0.1 0.1 0.1 3 6 c4t1d0 83.0 0.0 290.0 0.0 0.0 0.1 0.1 1.5 1 13 c6t1d0 77.0 0.0 280.0 0.0 0.0 0.1 0.2 1.4 1 11 c0t1d0 80.0 0.0 265.5 0.0 0.0 0.1 0.1 1.3 1 10 c5t1d0 18.0 0.0 62.0 0.0 0.0 0.1 0.0 3.0 0 5 c6t2d0 28.0 0.0 171.5 0.0 0.0 0.1 1.1 2.6 2 7 c4t3d0 83.0 0.0 290.0 0.0 0.0 0.1 0.0 1.1 0 9 c0t2d0 17.0 0.0 68.5 0.0 0.0 0.0 0.0 2.8 0 5 c5t2d0 27.0 0.0 178.5 0.0 0.0 0.1 1.2 3.0 3 8 c6t3d0 28.0 0.0 179.0 0.0 0.0 0.1 0.3 2.8 1 8 c5t3d0 16.0 0.0 45.5 0.0 0.0 0.0 0.5 2.4 1 4 c0t3d0 19.0 0.0 77.0 0.0 0.0 0.0 0.0 1.6 0 3 c4t4d0 21.0 0.0 78.0 0.0 0.0 0.0 0.6 2.3 1 5 c6t4d0 22.0 0.0 63.5 0.0 0.0 0.0 0.0 1.9 0 4 c0t4d0 30.0 0.0 157.5 0.0 0.0 0.1 0.0 3.0 0 9 c4t5d0 31.0 0.0 165.5 0.0 0.0 0.1 1.2 2.3 2 7 c6t5d0 83.0 0.0 275.5 0.0 0.0 0.1 0.3 1.5 2 12 c1t0d0 31.0 0.0 158.0 0.0 0.0 0.1 0.6 2.2 2 7 c0t5d0 70.0 231.0 1712.5 131.0 2.4 0.5 8.0 1.7 48 52 c4t6d0 25.0 0.0 147.5 0.0 0.0 0.0 0.2 1.9 1 5 c5t5d0 81.0 0.0 282.0 0.0 0.0 0.1 0.0 1.2 0 10 c1t1d0 67.0 230.0 1735.5 127.0 1.8 0.5 6.2 1.8 46 52 c6t6d0 25.0 0.0 147.5 0.0 0.0 0.1 0.4 2.4 0 6 c0t6d0 74.0 237.0 1718.5 133.0 1.8 0.5 5.7 1.5 41 47 c4t7d0 70.0 241.0 1713.0 131.5 2.2 0.5 7.0 1.6 42 48 c5t6d0 15.0 0.0 52.5 0.0 0.0 0.1 0.0 3.5 0 5 c1t2d0 68.0 225.0 1523.0 128.5 2.6 0.5 9.0 1.7 43 49 c6t7d0 66.0 230.0 1724.0 132.0 2.4 0.6 8.0 2.0 52 58 c0t7d0 78.0 230.0 1717.5 130.5 1.5 0.5 4.8 1.5 39 46 c5t7d0 16.0 0.0 45.5 0.0 0.0 0.1 0.0 4.2 0 7 c1t3d0 24.0 0.0 79.5 0.0 0.0 0.1 0.0 3.2 0 8 c1t4d0 34.0 0.0 167.0 0.0 0.0 0.1 0.1 2.4 0 8 c1t5d0 69.0 233.0 1566.0 134.5 1.9 0.5 6.4 1.7 46 50 c1t6d0 70.0 236.0 1717.5 131.0 1.7 0.4 5.5 1.5 39 45 c1t7d0 84.0 0.0 283.0 0.0 0.0 0.1 0.0 1.5 0 12 c7t0d0 86.0 0.0 291.5 0.0 0.0 0.1 0.2 1.3 1 11 c7t1d0 16.0 0.0 45.5 0.0 0.0 0.0 0.0 1.8 0 3 c7t2d0 27.0 0.0 171.0 0.0 0.0 0.1 0.8 2.2 2 6 c7t3d0 17.0 0.0 53.5 0.0 0.0 0.0 0.0 2.3 0 4 c7t4d0 29.0 0.0 164.5 0.0 0.0 0.1 0.8 3.0 2 9 c7t5d0 68.0 225.0 1718.0 130.5 2.5 0.5 8.6 1.8 46 53 c7t6d0 0.0 434.0 0.0 1692.5 0.1 0.1 0.1 0.1 3 6 c7t7d0 ^C bash-3.00# As you can see there are writes to other disks. bash-3.00# zfs get -r atime thumper-8 NAME PROPERTY VALUE SOURCE thumper-8 atime off local thumper-8/poczta atime off inherited from thumper-8 thumper-8/poczta/d6601 atime off inherited from thumper-8 thumper-8/poczta/d6602 atime off inherited from thumper-8 thumper-8/poczta/d6603 atime off inherited from thumper-8 thumper-8/poczta/d6604 atime off inherited from thumper-8 thumper-8/poczta/d6605 atime off inherited from thumper-8 thumper-8/poczta/d6605 at snapA atime - - thumper-8/poczta/d6605 at snapB atime - - thumper-8/poczta/d6606 atime off inherited from thumper-8 thumper-8/poczta/d6607 atime off inherited from thumper-8 thumper-8/poczta/d6608 atime off inherited from thumper-8 thumper-8/poczta/d6609 atime off inherited from thumper-8 thumper-8/poczta/d6610 atime off inherited from thumper-8 bash-3.00# So atime is off for all file systems in the pool. I did ''zfs umount -a'' and checked with ''df -h'' that all zfs file systems are unmounted but still I can see many writes to other disks. System is: bash-3.00# uname -a SunOS thumper-8.srv 5.10 Generic_125101-02 i86pc i386 i86pc Also ''zpool iostat 1'' looks strange: bash-3.00# zpool iostat 1 capacity operations bandwidth pool used avail read write read write ---------- ----- ----- ----- ----- ----- ----- misc 78.5K 440G 0 0 0 0 thumper-8 1.40T 18.6T 109 62 656K 1.20M ---------- ----- ----- ----- ----- ----- ----- misc 78.5K 440G 0 0 0 0 thumper-8 1.40T 18.6T 1.06K 489 19.0M 705K ---------- ----- ----- ----- ----- ----- ----- misc 78.5K 440G 0 0 0 0 thumper-8 1.40T 18.6T 944 431 29.3M 616K ---------- ----- ----- ----- ----- ----- ----- misc 78.5K 440G 0 0 0 0 thumper-8 1.40T 18.6T 754 305 22.5M 410K ---------- ----- ----- ----- ----- ----- ----- misc 78.5K 440G 0 0 0 0 thumper-8 1.40T 18.6T 904 401 13.8M 614K ---------- ----- ----- ----- ----- ----- ----- misc 78.5K 440G 0 0 0 0 thumper-8 1.40T 18.6T 911 525 10.3M 759K ---------- ----- ----- ----- ----- ----- ----- misc 78.5K 440G 0 0 0 0 thumper-8 1.40T 18.6T 898 348 30.6M 511K ---------- ----- ----- ----- ----- ----- ----- misc 78.5K 440G 0 0 0 0 thumper-8 1.40T 18.6T 993 385 24.2M 563K ---------- ----- ----- ----- ----- ----- ----- misc 78.5K 440G 0 0 0 0 thumper-8 1.40T 18.6T 1.24K 622 23.1M 837K ---------- ----- ----- ----- ----- ----- ----- ^C bash-3.00# And also: bash-3.00# iostat -xnz 1|egrep "c7t7d0|c4t1d0" 0.0 0.3 0.0 1.3 0.0 0.0 2.9 0.2 0 0 c4t1d0 5.4 8.9 45.3 41.0 0.0 0.0 0.9 1.4 0 2 c7t7d0 0.0 784.2 0.0 1000.6 0.1 0.1 0.1 0.1 4 11 c4t1d0 0.0 780.2 0.0 1000.1 0.1 0.1 0.1 0.2 5 12 c7t7d0 0.0 994.7 0.0 1733.0 0.1 0.1 0.1 0.1 6 14 c4t1d0 0.0 994.7 0.0 1733.0 0.1 0.1 0.1 0.1 7 15 c7t7d0 0.0 811.1 0.0 1175.2 0.1 0.1 0.1 0.1 4 12 c4t1d0 0.0 815.1 0.0 1175.1 0.1 0.1 0.1 0.1 5 12 c7t7d0 0.0 849.3 0.0 3577.6 0.3 0.2 0.4 0.2 9 15 c4t1d0 0.0 846.3 0.0 3577.6 0.2 0.1 0.3 0.2 8 15 c7t7d0 0.0 1029.0 0.0 1630.5 0.8 0.3 0.8 0.2 19 26 c4t1d0 0.0 1027.0 0.0 1630.5 0.9 0.2 0.9 0.2 16 23 c7t7d0 0.0 1009.1 0.0 532.5 0.2 0.2 0.2 0.2 8 17 c4t1d0 0.0 1024.0 0.0 532.5 0.1 0.2 0.1 0.2 7 17 c7t7d0 0.0 1132.7 0.0 2599.6 0.2 0.2 0.1 0.2 8 18 c4t1d0 0.0 1137.6 0.0 2599.8 0.2 0.2 0.1 0.2 9 17 c7t7d0 0.0 1266.3 0.0 1730.5 0.4 0.2 0.3 0.2 10 19 c4t1d0 0.0 1285.5 0.0 1730.3 0.2 0.2 0.2 0.2 9 20 c7t7d0 0.0 963.8 0.0 5626.8 1.3 0.2 1.3 0.2 14 21 c4t1d0 0.0 963.8 0.0 5626.9 0.7 0.2 0.8 0.2 13 21 c7t7d0 0.0 1023.0 0.0 2565.5 0.1 0.2 0.1 0.2 8 17 c4t1d0 0.0 1025.0 0.0 2565.5 0.1 0.2 0.1 0.2 8 17 c7t7d0 ^C bash-3.00# Data should be copied onto c4t1d0 disk - as you can see data are being written also to c7t7d0 which is being replaced. Lets see how it looks on all controllers of x4500: bash-3.00# iostat -xnzC 1|egrep "device| c[0-9]$" extended device statistics r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device 43.4 69.9 364.5 317.4 0.1 0.2 0.9 1.4 0 16 c0 43.5 70.0 366.4 317.5 0.1 0.2 1.0 1.4 0 16 c1 32.6 52.8 275.9 240.2 0.1 0.1 1.0 1.4 0 12 c4 32.6 54.1 276.1 241.1 0.1 0.1 1.0 1.4 0 12 c5 43.4 70.0 366.4 317.5 0.1 0.2 1.0 1.4 0 16 c6 43.4 70.1 364.5 319.4 0.1 0.2 1.0 1.4 0 16 c7 extended device statistics r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device 1012.2 1023.3 9568.7 898.6 28.8 3.7 14.2 1.8 5 371 c0 621.7 2167.6 5802.9 1812.1 30.8 3.0 11.0 1.1 4 301 c1 611.7 3059.6 5863.9 2700.2 58.3 4.4 15.9 1.2 7 438 c4 610.7 2091.4 5851.8 1879.7 36.4 3.3 13.5 1.2 5 331 c5 788.9 1998.3 7714.5 1799.6 50.1 4.2 18.0 1.5 6 423 c6 804.9 2076.4 7675.4 1832.1 36.6 4.0 12.7 1.4 6 401 c7 extended device statistics r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device 869.0 1347.0 7780.3 1227.0 21.3 3.7 9.6 1.7 5 366 c0 522.0 2496.2 4626.9 2421.7 32.7 3.0 10.8 1.0 4 299 c1 511.0 3692.4 4513.4 3642.9 45.3 3.7 10.8 0.9 5 371 c4 512.1 2381.2 4516.5 2381.7 33.8 2.9 11.7 1.0 4 294 c5 704.1 2200.2 6146.2 2436.3 44.5 3.9 15.3 1.3 6 388 c6 706.1 2180.3 6184.7 2413.8 41.5 3.9 14.4 1.3 5 388 c7 extended device statistics r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device 220.0 113.0 1356.5 56.5 0.5 0.9 1.4 2.6 0 85 c0 182.0 240.0 1570.0 121.0 0.4 0.7 1.0 1.7 0 72 c1 121.0 383.0 1521.5 677.5 0.3 0.5 0.6 0.9 0 48 c4 149.0 238.0 1549.5 119.5 0.3 0.5 0.8 1.4 0 54 c5 183.0 224.0 1585.0 112.5 0.3 0.6 0.7 1.5 0 63 c6 171.0 259.0 1166.5 614.0 0.4 0.7 0.9 1.5 0 66 c7 ^C bash-3.00# So there''re reads and writes basically to all disks. Again, I''m 100% sure no other activity is on the server.>> Also why in case of replacing a disk we do not just copy disk-to-disk? >> It would be MUCH faster here. Probably ''coz we''re traversing >> meta-data? But perhaps it could be done in a clever way so we endup >> just copying from one disk to another. Checking parity or checksum >> here it''s not necessary - scrub is for it. What we want in most cases >> is to replace drive as fast as possible.MA> In some cases it would be faster but others not. For example, if the MA> pool is not very full, it would be slower. Also if the disk you''re MA> replacing is not available, a straight disk-to-disk copy would not be MA> possible. bash-3.00# iostat -xnz 1|egrep "c7t7d0|c4t1d0" 0.0 0.3 0.0 1.3 0.0 0.0 2.9 0.2 0 0 c4t1d0 5.4 8.9 45.3 41.0 0.0 0.0 0.9 1.4 0 2 c7t7d0 0.0 784.2 0.0 1000.6 0.1 0.1 0.1 0.1 4 11 c4t1d0 0.0 780.2 0.0 1000.1 0.1 0.1 0.1 0.2 5 12 c7t7d0 0.0 994.7 0.0 1733.0 0.1 0.1 0.1 0.1 6 14 c4t1d0 0.0 994.7 0.0 1733.0 0.1 0.1 0.1 0.1 7 15 c7t7d0 0.0 811.1 0.0 1175.2 0.1 0.1 0.1 0.1 4 12 c4t1d0 0.0 815.1 0.0 1175.1 0.1 0.1 0.1 0.1 5 12 c7t7d0 0.0 849.3 0.0 3577.6 0.3 0.2 0.4 0.2 9 15 c4t1d0 0.0 846.3 0.0 3577.6 0.2 0.1 0.3 0.2 8 15 c7t7d0 0.0 1029.0 0.0 1630.5 0.8 0.3 0.8 0.2 19 26 c4t1d0 0.0 1027.0 0.0 1630.5 0.9 0.2 0.9 0.2 16 23 c7t7d0 0.0 1009.1 0.0 532.5 0.2 0.2 0.2 0.2 8 17 c4t1d0 0.0 1024.0 0.0 532.5 0.1 0.2 0.1 0.2 7 17 c7t7d0 0.0 1132.7 0.0 2599.6 0.2 0.2 0.1 0.2 8 18 c4t1d0 0.0 1137.6 0.0 2599.8 0.2 0.2 0.1 0.2 9 17 c7t7d0 0.0 1266.3 0.0 1730.5 0.4 0.2 0.3 0.2 10 19 c4t1d0 0.0 1285.5 0.0 1730.3 0.2 0.2 0.2 0.2 9 20 c7t7d0 0.0 963.8 0.0 5626.8 1.3 0.2 1.3 0.2 14 21 c4t1d0 0.0 963.8 0.0 5626.9 0.7 0.2 0.8 0.2 13 21 c7t7d0 0.0 1023.0 0.0 2565.5 0.1 0.2 0.1 0.2 8 17 c4t1d0 0.0 1025.0 0.0 2565.5 0.1 0.2 0.1 0.2 8 17 c7t7d0 ^C bash-3.00# Looks like data are being written with about 1-5MB/s on c4t1d0 - which is very slow. Now the system is not doing anything else other than replacing disk. When I put actual production workload I get 2-3 to resynchronize single disk with such a config.>> On another thumper I have a failing drive (port resets, etc.) so I >> issued over a week ago drive replacement. Well it still hasn''t >> completed even 4% in a week! The pool config is the same. It''s just >> waaaay to slow and in a long term risky.MA> Are you taking snapshots? They cause scrubbing / resilvering to restart MA> (this is bug 6343667). At first I did and I haven''t noticed. All above data are without taking/deleteing any snapshots or doing anything else. -- Best regards, Robert mailto:rmilkowski at task.gda.pl http://milek.blogspot.com