Rasmus Fauske
2011-Jun-14 11:15 UTC
[zfs-discuss] Disk replacement need to scan full pool ?
Hi,
I want to replace some slow consumer drives with new edc re4 ones but
when I do a replace it needs to scan the full pool and not only that
disk set (or just the old drive)
Is this normal ? (the speed is always slow in the start so thats not
what I am wondering about, but that it needs to scan all of my 18.7T to
replace one drive)
The disk config:
pool: tank
state: ONLINE
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Tue Jun 14 10:03:37 2011
16.8G scanned out of 18.7T at 1.52M/s, (scan is slow, no estimated
time)
388M resilvered, 0.09% done
config:
NAME STATE READ WRITE CKSUM
tank ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
c10t14d0 ONLINE 0 0 0
c10t15d0 ONLINE 0 0 0
c10t16d0 ONLINE 0 0 0
c10t17d0 ONLINE 0 0 0
c10t18d0 ONLINE 0 0 0
c10t19d0 ONLINE 0 0 0
c10t20d0 ONLINE 0 0 0
raidz2-1 ONLINE 0 0 0
c10t21d0 ONLINE 0 0 0
replacing-1 ONLINE 0 0 0
c10t35d0 ONLINE 0 0 0
c10t22d0 ONLINE 0 0 0 (resilvering)
c10t23d0 ONLINE 0 0 0
c10t24d0 ONLINE 0 0 0
c10t25d0 ONLINE 0 0 0
c10t26d0 ONLINE 0 0 0
c10t27d0 ONLINE 0 0 0
raidz2-2 ONLINE 0 0 0
c10t28d0 ONLINE 0 0 0
c10t29d0 ONLINE 0 0 0
c10t30d0 ONLINE 0 0 0
c10t31d0 ONLINE 0 0 0
c10t32d0 ONLINE 0 0 0
c10t33d0 ONLINE 0 0 0
c10t34d0 ONLINE 0 0 0
raidz2-3 ONLINE 0 0 0
c10t0d0 ONLINE 0 0 0
c10t1d0 ONLINE 0 0 0
c10t2d0 ONLINE 0 0 0
c10t3d0 ONLINE 0 0 0
c10t4d0 ONLINE 0 0 0
c10t5d0 ONLINE 0 0 0
c10t6d0 ONLINE 0 0 0
raidz2-4 ONLINE 0 0 0
c10t7d0 ONLINE 0 0 0
c10t8d0 ONLINE 0 0 0
c10t9d0 ONLINE 0 0 0
c10t10d0 ONLINE 0 0 0
c10t11d0 ONLINE 0 0 0
c10t12d0 ONLINE 0 0 0
c10t13d0 ONLINE 0 0 0
errors: No known data errors
Edward Ned Harvey
2011-Jun-14 12:06 UTC
[zfs-discuss] Disk replacement need to scan full pool ?
> From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss- > bounces at opensolaris.org] On Behalf Of Rasmus Fauske > > I want to replace some slow consumer drives with new edc re4 ones but > when I do a replace it needs to scan the full pool and not only that > disk set (or just the old drive) > > Is this normal ? (the speed is always slow in the start so thats not > what I am wondering about, but that it needs to scan all of my 18.7T to > replace one drive) > > The disk config: > pool: tank > state: ONLINE > status: One or more devices is currently being resilvered. The pool will > continue to function, possibly in a degraded state. > action: Wait for the resilver to complete. > scan: resilver in progress since Tue Jun 14 10:03:37 2011 > 16.8G scanned out of 18.7T at 1.52M/s, (scan is slow, no estimated > time) > 388M resilvered, 0.09% done > config: > > NAME STATE READ WRITE CKSUM > tank ONLINE 0 0 0 > raidz2-0 ONLINE 0 0 0 > c10t14d0 ONLINE 0 0 0 > c10t15d0 ONLINE 0 0 0 > c10t16d0 ONLINE 0 0 0 > c10t17d0 ONLINE 0 0 0 > c10t18d0 ONLINE 0 0 0 > c10t19d0 ONLINE 0 0 0 > c10t20d0 ONLINE 0 0 0 > raidz2-1 ONLINE 0 0 0 > c10t21d0 ONLINE 0 0 0 > replacing-1 ONLINE 0 0 0 > c10t35d0 ONLINE 0 0 0 > c10t22d0 ONLINE 0 0 0 (resilvering) > c10t23d0 ONLINE 0 0 0 > c10t24d0 ONLINE 0 0 0 > c10t25d0 ONLINE 0 0 0 > c10t26d0 ONLINE 0 0 0 > c10t27d0 ONLINE 0 0 0Only this raidz2 vdev is resilvering. The other raidz2-vdev''s are idle. You can verify this with the command: zpool iostat 30
Rasmus Fauske
2011-Jun-14 13:12 UTC
[zfs-discuss] Disk replacement need to scan full pool ?
Den 14.06.2011 14:06, skrev Edward Ned Harvey:>> From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss- >> bounces at opensolaris.org] On Behalf Of Rasmus Fauske >> >> I want to replace some slow consumer drives with new edc re4 ones but >> when I do a replace it needs to scan the full pool and not only that >> disk set (or just the old drive) >> >> Is this normal ? (the speed is always slow in the start so thats not >> what I am wondering about, but that it needs to scan all of my 18.7T to >> replace one drive) >> >> The disk config: >> pool: tank >> state: ONLINE >> status: One or more devices is currently being resilvered. The pool will >> continue to function, possibly in a degraded state. >> action: Wait for the resilver to complete. >> scan: resilver in progress since Tue Jun 14 10:03:37 2011 >> 16.8G scanned out of 18.7T at 1.52M/s, (scan is slow, no estimated >> time) >> 388M resilvered, 0.09% done >> config: >> >> NAME STATE READ WRITE CKSUM >> tank ONLINE 0 0 0 >> raidz2-1 ONLINE 0 0 0 >> c10t21d0 ONLINE 0 0 0 >> replacing-1 ONLINE 0 0 0 >> c10t35d0 ONLINE 0 0 0 >> c10t22d0 ONLINE 0 0 0 (resilvering) >> c10t23d0 ONLINE 0 0 0 >> c10t24d0 ONLINE 0 0 0 >> c10t25d0 ONLINE 0 0 0 >> c10t26d0 ONLINE 0 0 0 >> c10t27d0 ONLINE 0 0 0 > Only this raidz2 vdev is resilvering. The other raidz2-vdev''s are idle. > You can verify this with the command: > zpool iostat 30 >Yes but still it wants to scan all of the data in the pool: action: Wait for the resilver to complete. scan: resilver in progress since Tue Jun 14 10:03:37 2011 16.8G scanned out of 18.7T at 1.52M/s, (scan is slow, no estimated time) Each raidz is 7 x 1TB disks, around 50% filled. So should it not only scan that data and not the full pool that holds around 18T ? -- Rasmus F
Stephan Budach
2011-Jun-14 18:08 UTC
[zfs-discuss] Disk replacement need to scan full pool ?
Am 14.06.11 15:12, schrieb Rasmus Fauske:> Den 14.06.2011 14:06, skrev Edward Ned Harvey: >>> From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss- >>> bounces at opensolaris.org] On Behalf Of Rasmus Fauske >>> >>> I want to replace some slow consumer drives with new edc re4 ones but >>> when I do a replace it needs to scan the full pool and not only that >>> disk set (or just the old drive) >>> >>> Is this normal ? (the speed is always slow in the start so thats not >>> what I am wondering about, but that it needs to scan all of my 18.7T to >>> replace one drive) >>> >>> The disk config: >>> pool: tank >>> state: ONLINE >>> status: One or more devices is currently being resilvered. The pool >>> will >>> continue to function, possibly in a degraded state. >>> action: Wait for the resilver to complete. >>> scan: resilver in progress since Tue Jun 14 10:03:37 2011 >>> 16.8G scanned out of 18.7T at 1.52M/s, (scan is slow, no >>> estimated >>> time) >>> 388M resilvered, 0.09% done >>> config: >>> >>> NAME STATE READ WRITE CKSUM >>> tank ONLINE 0 0 0 >>> raidz2-1 ONLINE 0 0 0 >>> c10t21d0 ONLINE 0 0 0 >>> replacing-1 ONLINE 0 0 0 >>> c10t35d0 ONLINE 0 0 0 >>> c10t22d0 ONLINE 0 0 0 (resilvering) >>> c10t23d0 ONLINE 0 0 0 >>> c10t24d0 ONLINE 0 0 0 >>> c10t25d0 ONLINE 0 0 0 >>> c10t26d0 ONLINE 0 0 0 >>> c10t27d0 ONLINE 0 0 0 >> Only this raidz2 vdev is resilvering. The other raidz2-vdev''s are idle. >> You can verify this with the command: >> zpool iostat 30 >> > Yes but still it wants to scan all of the data in the pool: > > action: Wait for the resilver to complete. > scan: resilver in progress since Tue Jun 14 10:03:37 2011 > 16.8G scanned out of 18.7T at 1.52M/s, (scan is slow, no > estimated time) > > Each raidz is 7 x 1TB disks, around 50% filled. So should it not only > scan that data and not the full pool that holds around 18T ?This is what I experienced as well. I had a zpool made out of 16 mirrors and in that zpool 12 mirrors were made out of 1 TB drives, while 4 mirrors were made out of 2 TB drives. When I started to swap out the 1 TB drives with 2 TB ones, zfs just didn''t read from the corresponding drive of the mirrored vdev, but from all drives in the zpool. iostat showed that all disks were nearly equally busy. The scan started very slow and picked up speed after some time. Interestingly, I was able to swap a number 1 TB drives with 2 TB ones - I think I had up to 8 drives at once in progress, so I was able to complete the whole task in less than 48 hrs.
Roy Sigurd Karlsbakk
2011-Jun-14 18:40 UTC
[zfs-discuss] Disk replacement need to scan full pool ?
> Hi, > > I want to replace some slow consumer drives with new edc re4 ones but > when I do a replace it needs to scan the full pool and not only that > disk set (or just the old drive) > > Is this normal ? (the speed is always slow in the start so thats not > what I am wondering about, but that it needs to scan all of my 18.7T > to replace one drive)I had a pool where some three VDEVs got full and replaced the 2TB (green) members with new and faster 3TB drives. To do this, I replaced one drive in each VDEV at a time. This worked well for me. Just make sure you have a backup in case the shit hits the fan.... Vennlige hilsener / Best regards roy -- Roy Sigurd Karlsbakk (+47) 97542685 roy at karlsbakk.net http://blogg.karlsbakk.net/ -- I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det er et element?rt imperativ for alle pedagoger ? unng? eksessiv anvendelse av idiomer med fremmed opprinnelse. I de fleste tilfeller eksisterer adekvate og relevante synonymer p? norsk.
Bill Sommerfeld
2011-Jun-14 18:49 UTC
[zfs-discuss] Disk replacement need to scan full pool ?
On 06/14/11 04:15, Rasmus Fauske wrote:> I want to replace some slow consumer drives with new edc re4 ones but > when I do a replace it needs to scan the full pool and not only that > disk set (or just the old drive) > > Is this normal ? (the speed is always slow in the start so thats not > what I am wondering about, but that it needs to scan all of my 18.7T to > replace one drive)This is normal. The resilver is not reading all data blocks; it''s reading all of the metadata blocks which contain one or more block pointers, which is the only way to find all the allocated data (and in the case of raidz, know precisely how it''s spread and encoded across the members of the vdev). And it''s reading all the data blocks needed to reconstruct the disk to be replaced. - Bill
Pawel Jakub Dawidek
2011-Jun-15 11:22 UTC
[zfs-discuss] Disk replacement need to scan full pool ?
On Tue, Jun 14, 2011 at 11:49:56AM -0700, Bill Sommerfeld wrote:> On 06/14/11 04:15, Rasmus Fauske wrote: > > I want to replace some slow consumer drives with new edc re4 ones but > > when I do a replace it needs to scan the full pool and not only that > > disk set (or just the old drive) > > > > Is this normal ? (the speed is always slow in the start so thats not > > what I am wondering about, but that it needs to scan all of my 18.7T to > > replace one drive) > > This is normal. The resilver is not reading all data blocks; it''s > reading all of the metadata blocks which contain one or more block > pointers, which is the only way to find all the allocated data (and in > the case of raidz, know precisely how it''s spread and encoded across the > members of the vdev). And it''s reading all the data blocks needed to > reconstruct the disk to be replaced.Maybe it would be faster to just offline this one disk, use dd(1) to copy entire disk content, disconnect old disk on online the new one. Not sure how well this will work on Solaris as the new disk serial number won''t match the one in metadata, but it will surely work on FreeBSD. -- Pawel Jakub Dawidek http://www.wheelsystems.com FreeBSD committer http://www.FreeBSD.org Am I Evil? Yes, I Am! http://yomoli.com -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 196 bytes Desc: not available URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20110615/87b4f5ac/attachment.bin>
Richard Elling
2011-Jun-15 17:35 UTC
[zfs-discuss] Disk replacement need to scan full pool ?
On Jun 15, 2011, at 4:22 AM, Pawel Jakub Dawidek wrote:> On Tue, Jun 14, 2011 at 11:49:56AM -0700, Bill Sommerfeld wrote: >> On 06/14/11 04:15, Rasmus Fauske wrote: >>> I want to replace some slow consumer drives with new edc re4 ones but >>> when I do a replace it needs to scan the full pool and not only that >>> disk set (or just the old drive) >>> >>> Is this normal ? (the speed is always slow in the start so thats not >>> what I am wondering about, but that it needs to scan all of my 18.7T to >>> replace one drive) >> >> This is normal. The resilver is not reading all data blocks; it''s >> reading all of the metadata blocks which contain one or more block >> pointers, which is the only way to find all the allocated data (and in >> the case of raidz, know precisely how it''s spread and encoded across the >> members of the vdev). And it''s reading all the data blocks needed to >> reconstruct the disk to be replaced. > > Maybe it would be faster to just offline this one disk, use dd(1) to > copy entire disk content, disconnect old disk on online the new one. > Not sure how well this will work on Solaris as the new disk serial > number won''t match the one in metadata, but it will surely work on > FreeBSD.This works in Solaris too. -- richard
Jordan Schwartz
2011-Jun-15 22:49 UTC
[zfs-discuss] Disk replacement need to scan full pool ?
>> Maybe it would be faster to just offline this one disk, use dd(1) to >> copy entire disk content, disconnect old disk on online the new one. >> Not sure how well this will work on Solaris as the new disk serial >> number won''t match the one in metadata, but it will surely work on >> FreeBSD. > > This works in Solaris too.Any details/blog entries on this procedure? I haven''t had the need to swap disks like this yet, but would like to become familiar with the procedure and caveats. Thanks, Jordan On Wed, Jun 15, 2011 at 10:35 AM, Richard Elling <richard.elling at gmail.com> wrote:> On Jun 15, 2011, at 4:22 AM, Pawel Jakub Dawidek wrote: > >> On Tue, Jun 14, 2011 at 11:49:56AM -0700, Bill Sommerfeld wrote: >>> On 06/14/11 04:15, Rasmus Fauske wrote: >>>> I want to replace some slow consumer drives with new edc re4 ones but >>>> when I do a replace it needs to scan the full pool and not only that >>>> disk set (or just the old drive) >>>> >>>> Is this normal ? (the speed is always slow in the start so thats not >>>> what I am wondering about, but that it needs to scan all of my 18.7T to >>>> replace one drive) >>> >>> This is normal. ?The resilver is not reading all data blocks; it''s >>> reading all of the metadata blocks which contain one or more block >>> pointers, which is the only way to find all the allocated data (and in >>> the case of raidz, know precisely how it''s spread and encoded across the >>> members of the vdev). ?And it''s reading all the data blocks needed to >>> reconstruct the disk to be replaced. >> >> Maybe it would be faster to just offline this one disk, use dd(1) to >> copy entire disk content, disconnect old disk on online the new one. >> Not sure how well this will work on Solaris as the new disk serial >> number won''t match the one in metadata, but it will surely work on >> FreeBSD. > > This works in Solaris too. > ?-- richard > > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >