Hi there,
I run in production a FreeBSD-8 stable server with a ZFS file system.
It used raidz2, so that ought to be very safe, I thought.
Recently a disk failed, and it was replaced today. However, the file
pool doesn't want to come back.
Below is the status. da4 was bad and was replaced (by a somewhat bigger
disk) in-place. The raidz2 shows to be DEGRADED but somehow the pool as
a whole is FAULTED? And whatever I do, the new disk isn't recognised. I
cut and pasted some commands below. In all cases it complains that pool
"tank" is unavailable...
What to do?
No I/O errors show up in /var/log/messages.
Previously, I had some weird directory where there were CRC errors in
the directory entries, and as long as you didn't try to do anything with
the files, there was no problem. Removing them was impossible though. I
hope it isn't that what is holding my whole pool hostage... I don't mind
losing that one directory.
fourquid.1:/lib$ sudo zpool status
Password:
pool: tank
state: FAULTED
status: One or more devices could not be opened. There are insufficient
replicas for the pool to continue functioning.
action: Attach the missing device and online it using 'zpool online'.
see: http://www.sun.com/msg/ZFS-8000-3C
scan: scrub repaired 0 in 49h3m with 2 errors on Fri Jan 20 15:10:35 2012
config:
NAME STATE READ WRITE CKSUM
tank FAULTED 0 0 2
raidz2-0 DEGRADED 0 0 8
da0 ONLINE 0 0 0
da1 ONLINE 0 0 0
da2 ONLINE 0 0 0
da3 ONLINE 0 0 0
3758301462980058947 UNAVAIL 0 0 0 was /dev/da4
da5 ONLINE 0 0 0
fourquid.1:/lib$ sudo zpool replace tank da4
cannot open 'tank': pool is unavailable
fourquid.1:/lib$ sudo zpool replace raidz2-0 da4
cannot open 'raidz2-0': no such pool
fourquid.1:/lib$ sudo zpool replace tank da4 da4
cannot open 'tank': pool is unavailable
fourquid.1:/lib$ sudo zpool online tank da4
cannot open 'tank': pool is unavailable
fourquid.1:/lib$ sudo zpool clear tank da4
cannot clear errors for da4: I/O error
fourquid.1:/lib$ sudo zpool clear tank
cannot clear errors for tank: I/O error
fourquid.1:/lib$ sudo zpool replace tank 3758301462980058947 da4
cannot open 'tank': pool is unavailable
fourquid.1:/lib$ sudo zpool scrub tank
cannot scrub 'tank': pool is currently unavailable
-Olaf.
--
Pipe rene = new PipePicture(); assert(Not rene.GetType().Equals(Pipe));
I'm still (or again) in more or less the same situation as before.
I have tried things like exporting the pool and re-importing it. However
the import didn't even want to work. Only once I found an undocumented
"zpool import -V" option by UTSLing, the pool got imported again, but
still without an attempt at resilvering.
There also exist (undocumented) -F (rewind) and -X (extreme rewind)
options for zpool import and zpool clear. However, the effect of -X
seems to be that some extremely lengthy operation is attempted, that
never finishes. In some cases, a reboot was necessary to terminate the
process.
I have also tried to remove an extra disk, on the theory that with
raidz2 you can miss 2 disks, and that the problem might be restricted to
a single disk. I haven't done each disk yet, but so far there was no
success. (I "removed" each disk by unconfiguring the pass-through on
the
Areca web interface).
$ zpool status
pool: tank
state: FAULTED
status: One or more devices could not be opened. There are insufficient
replicas for the pool to continue functioning.
action: Attach the missing device and online it using 'zpool online'.
see: http://www.sun.com/msg/ZFS-8000-3C
scan: scrub repaired 0 in 49h3m with 2 errors on Fri Jan 20 15:10:35 2012
config:
NAME STATE READ WRITE CKSUM
tank FAULTED 0 0 2
raidz2-0 DEGRADED 0 0 8
da0 ONLINE 0 0 0
da1 ONLINE 0 0 0
da2 ONLINE 0 0 0
da3 ONLINE 0 0 0
3758301462980058947 UNAVAIL 0 0 0 was /dev/da4
da5 ONLINE 0 0 0
fourquid.1:~$ sudo zpool online tank da4
cannot open 'tank': pool is unavailable
fourquid.1:~$ sudo zpool clear -nF tank
fourquid.1:~$ sudo zpool clear -F tank
cannot clear errors for tank: I/O error
fourquid.1:~$ sudo zpool clear -nFX tank
(no output, uses some cpu, some I/O)
zdb -v ok
zdb -v -c tank zdb: can't open 'tank': input/output
error
zdb -v -l /dev/da[01235] ok
zdb -v -u tank zdb: can't open 'tank': Input/output
error
zdb -v -l -u /dev/da[01235] ok
zdb -v -m tank zdb: can't open 'tank': Input/output
error
zdb -v -m -X tank no output, uses cpu and I/O$
zdb -v -i tank zdb: can't open 'tank': Input/output
error
zdb -v -i -F tank zdb: can't open 'tank': Input/output
error
zdb -v -i -X tank no output, uses cpu and I/O
Any ideas?
-Olaf.
--
Pipe rene = new PipePicture(); assert(Not rene.GetType().Equals(Pipe));