I have an 8-drive ZFS array consisting of WD15EADS drives. One of my disks
has started to fail, so I got a replacement disk. I have replaced a disk
before by:
zpool offline tank /dev/da5
shutting down, swapping from old disk to new disk
booting
zpool replace tank /dev/da5
This worked fine.
This time the failing disk was da3, and I tried the same thing:
zpool offline tank /dev/da3
zpool status showed da3 offline.
shut down, swapped old disk to new disk.
When I booted again, I got:
Code:
NAME STATE READ WRITE CKSUM
tank UNAVAIL 0 0 0 insufficient replicas
raidz1 UNAVAIL 0 0 0 corrupted data
da0 ONLINE 0 0 0
da1 ONLINE 0 0 0
da2 ONLINE 0 0 0
da3 ONLINE 0 0 0
da4 ONLINE 0 0 0
da5 ONLINE 0 0 0
da6 ONLINE 0 0 0
da7 ONLINE 0 0 0
I switched back to the old disk and booted again and then I could access my
data again, and da3 still showed as offline. I tried 'zpool online tank
/dev/da3' and after a few seconds resilvering completed and all 8 drives are
back online again, but with the 'dying' disk as da3 still.
I tried shutting down WITHOUT first offlining /dev/da3, and swapping the
disks, and when I booted I again got 'insufficient replicas'.
Why am I getting this error, and how come it worked ok the last time I
replaced a disk? And more importantly, how do I switch to my new replacement
disk without losing data?
Thanks.
On Mon, Jul 19, 2010 at 11:21:38AM -0400, Garrett Moore wrote:> I have an 8-drive ZFS array consisting of WD15EADS drives. One of my disks > has started to fail, so I got a replacement disk. I have replaced a disk > before by: > > zpool offline tank /dev/da5 > shutting down, swapping from old disk to new disk > booting > zpool replace tank /dev/da5 > > This worked fine. > > This time the failing disk was da3, and I tried the same thing: > zpool offline tank /dev/da3 > zpool status showed da3 offline. > shut down, swapped old disk to new disk. > > When I booted again, I got: > Code: > > NAME STATE READ WRITE CKSUM > tank UNAVAIL 0 0 0 insufficient replicas > raidz1 UNAVAIL 0 0 0 corrupted data > da0 ONLINE 0 0 0 > da1 ONLINE 0 0 0 > da2 ONLINE 0 0 0 > da3 ONLINE 0 0 0 > da4 ONLINE 0 0 0 > da5 ONLINE 0 0 0 > da6 ONLINE 0 0 0 > da7 ONLINE 0 0 0 > > I switched back to the old disk and booted again and then I could access my > data again, and da3 still showed as offline. I tried 'zpool online tank > /dev/da3' and after a few seconds resilvering completed and all 8 drives are > back online again, but with the 'dying' disk as da3 still. > > I tried shutting down WITHOUT first offlining /dev/da3, and swapping the > disks, and when I booted I again got 'insufficient replicas'. > > Why am I getting this error, and how come it worked ok the last time I > replaced a disk? And more importantly, how do I switch to my new replacement > disk without losing data?Can you please provide uname -a output? Thanks. -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB |
On Mon, Jul 19, 2010 at 8:21 AM, Garrett Moore <garrettmoore@gmail.com> wrote:> I have an 8-drive ZFS array consisting of WD15EADS drives. One of my disks > has started to fail, so I got a replacement disk. I have replaced a disk > before by: > > ?zpool offline tank /dev/da5 > shutting down, swapping from old disk to new disk > booting > ?zpool replace tank /dev/da5 > > This worked fine. > > This time the failing disk was da3, and I tried the same thing: > ?zpool offline tank /dev/da3 > zpool status showed da3 offline. > shut down, swapped old disk to new disk.For some reason, ZFS is getting confused by the device names, possibly due to the controller renumbering device nodes? Try the following: zpool offline tank /dev/da3 zpool status tank to make sure it offlined the correct drive zpool export tank might have to do this from single-user mode reboot zpool import tank this forces ZFS to re-taste each drive to read the metadata zpool replace tank /dev/da3 this should force it to use the correct drive Note: if you have / on ZFS, the above may not be doable. -- Freddie Cash fjwcash@gmail.com