I'm trying to destroy a zfs array which I recently created. It contains
nothing of value.
# zpool status
pool: storage
state: ONLINE
status: One or more devices could not be used because the label is
missing or
invalid. Sufficient replicas exist for the pool to continue
functioning in a degraded state.
action: Replace the device using 'zpool replace'.
see: http://www.sun.com/msg/ZFS-8000-4J
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
storage ONLINE 0 0 0
raidz2 ONLINE 0 0 0
gpt/disk01 ONLINE 0 0 0
gpt/disk02 ONLINE 0 0 0
gpt/disk03 ONLINE 0 0 0
gpt/disk04 ONLINE 0 0 0
gpt/disk05 ONLINE 0 0 0
/tmp/sparsefile1.img UNAVAIL 0 0 0 corrupted
data
/tmp/sparsefile2.img UNAVAIL 0 0 0 corrupted
data
errors: No known data errors
Why sparse files? See this post:
http://docs.freebsd.org/cgi/getmsg.cgi?fetch=1007077+0+archive/2010/freebsd-stable/20100725.freebsd-stable
The two tmp files were created via:
dd if=/dev/zero of=/tmp/sparsefile1.img bs=1 count=0 oseek=1862g
dd if=/dev/zero of=/tmp/sparsefile2.img bs=1 count=0 oseek=1862g
And the array created with:
zpool create -f storage raidz2 gpt/disk01 gpt/disk02 gpt/disk03 \
gpt/disk04 gpt/disk05 /tmp/sparsefile1.img /tmp/sparsefile2.img
The -f flag was required to avoid this message:
invalid vdev specification
use '-f' to override the following errors:
mismatched replication level: raidz contains both files and devices
I tried to offline one of the sparse files:
zpool offline storage /tmp/sparsefile2.img
That caused a panic: http://www.langille.org/tmp/zpool-offline-panic.jpg
After rebooting, I rm'd both /tmp/sparsefile1.img and
/tmp/sparsefile2.img without thinking they were still in the zpool. Now
I am unable to destroy the pool. The system panics. I disabled ZFS via
/etc/rc.conf, rebooted, recreated the two sparse files, then did a
forcestart of zfs. Then I saw:
# zpool status
pool: storage
state: ONLINE
status: One or more devices could not be used because the label is
missing or
invalid. Sufficient replicas exist for the pool to continue
functioning in a degraded state.
action: Replace the device using 'zpool replace'.
see: http://www.sun.com/msg/ZFS-8000-4J
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
storage ONLINE 0 0 0
raidz2 ONLINE 0 0 0
gpt/disk01 ONLINE 0 0 0
gpt/disk02 ONLINE 0 0 0
gpt/disk03 ONLINE 0 0 0
gpt/disk04 ONLINE 0 0 0
gpt/disk05 ONLINE 0 0 0
/tmp/sparsefile1.img UNAVAIL 0 0 0 corrupted
data
/tmp/sparsefile2.img UNAVAIL 0 0 0 corrupted
data
errors: No known data errors
Another attempt to destroy the array created a panic.
Suggestions as to how to remove this array and get started again?
--
Dan Langille - http://langille.org/
On 7/25/2010 1:58 PM, Dan Langille wrote:> I'm trying to destroy a zfs array which I recently created. It contains > nothing of value.Oh... I left this out: FreeBSD kraken.unixathome.org 8.0-STABLE FreeBSD 8.0-STABLE #0: Fri Mar 5 00:46:11 EST 2010 dan@kraken.example.org:/usr/obj/usr/src/sys/KRAKEN amd64> > # zpool status > pool: storage > state: ONLINE > status: One or more devices could not be used because the label is > missing or > invalid. Sufficient replicas exist for the pool to continue > functioning in a degraded state. > action: Replace the device using 'zpool replace'. > see: http://www.sun.com/msg/ZFS-8000-4J > scrub: none requested > config: > > NAME STATE READ WRITE CKSUM > storage ONLINE 0 0 0 > raidz2 ONLINE 0 0 0 > gpt/disk01 ONLINE 0 0 0 > gpt/disk02 ONLINE 0 0 0 > gpt/disk03 ONLINE 0 0 0 > gpt/disk04 ONLINE 0 0 0 > gpt/disk05 ONLINE 0 0 0 > /tmp/sparsefile1.img UNAVAIL 0 0 0 corrupted data > /tmp/sparsefile2.img UNAVAIL 0 0 0 corrupted data > > errors: No known data errors > > Why sparse files? See this post: > > http://docs.freebsd.org/cgi/getmsg.cgi?fetch=1007077+0+archive/2010/freebsd-stable/20100725.freebsd-stable > > > The two tmp files were created via: > > dd if=/dev/zero of=/tmp/sparsefile1.img bs=1 count=0 oseek=1862g > dd if=/dev/zero of=/tmp/sparsefile2.img bs=1 count=0 oseek=1862g > > And the array created with: > > zpool create -f storage raidz2 gpt/disk01 gpt/disk02 gpt/disk03 \ > gpt/disk04 gpt/disk05 /tmp/sparsefile1.img /tmp/sparsefile2.img > > The -f flag was required to avoid this message: > > invalid vdev specification > use '-f' to override the following errors: > mismatched replication level: raidz contains both files and devices > > > I tried to offline one of the sparse files: > > zpool offline storage /tmp/sparsefile2.img > > That caused a panic: http://www.langille.org/tmp/zpool-offline-panic.jpg > > After rebooting, I rm'd both /tmp/sparsefile1.img and > /tmp/sparsefile2.img without thinking they were still in the zpool. Now > I am unable to destroy the pool. The system panics. I disabled ZFS via > /etc/rc.conf, rebooted, recreated the two sparse files, then did a > forcestart of zfs. Then I saw: > > # zpool status > pool: storage > state: ONLINE > status: One or more devices could not be used because the label is > missing or > invalid. Sufficient replicas exist for the pool to continue > functioning in a degraded state. > action: Replace the device using 'zpool replace'. > see: http://www.sun.com/msg/ZFS-8000-4J > scrub: none requested > config: > > NAME STATE READ WRITE CKSUM > storage ONLINE 0 0 0 > raidz2 ONLINE 0 0 0 > gpt/disk01 ONLINE 0 0 0 > gpt/disk02 ONLINE 0 0 0 > gpt/disk03 ONLINE 0 0 0 > gpt/disk04 ONLINE 0 0 0 > gpt/disk05 ONLINE 0 0 0 > /tmp/sparsefile1.img UNAVAIL 0 0 0 corrupted data > /tmp/sparsefile2.img UNAVAIL 0 0 0 corrupted data > > errors: No known data errors > > > Another attempt to destroy the array created a panic. > > Suggestions as to how to remove this array and get started again? >-- Dan Langille - http://langille.org/
On Sun, Jul 25, 2010 at 01:58:34PM -0400, Dan Langille wrote:> [...] > NAME STATE READ WRITE CKSUM > storage ONLINE 0 0 0 > raidz2 ONLINE 0 0 0 > gpt/disk01 ONLINE 0 0 0 > gpt/disk02 ONLINE 0 0 0 > gpt/disk03 ONLINE 0 0 0 > gpt/disk04 ONLINE 0 0 0 > gpt/disk05 ONLINE 0 0 0 > /tmp/sparsefile1.img UNAVAIL 0 0 0 corrupted data > /tmp/sparsefile2.img UNAVAIL 0 0 0 corrupted data > > [...] > > Another attempt to destroy the array created a panic. > Suggestions as to how to remove this array and get started again? > > [...] > > FreeBSD kraken.unixathome.org 8.0-STABLE FreeBSD 8.0-STABLE #0: Fri Mar 5 00:46:11 EST 2010 dan@kraken.example.org:/usr/obj/usr/src/sys/KRAKEN amd641) Try upgrading the system (to 8.1-STABLE). There have been numerous changes to ZFS on RELENG_8 since March 5th. I don't know if any of them would address your problem. http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c#rev1.8.2.4 http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c#rev1.8.2.3 http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c#rev1.8.2.2 http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c#rev1.8.2.1 2) Try bringing the system down into single-user mode and zeroing out the first and last 64kbytes of each gpt/diskXX (you'll have to figure this out on your own, I'm not familiar with GPT) so that the ZFS metadata goes away. Footnote: can someone explain to me how ZFS would, upon reboot, know that /tmp/sparsefile[12].img are part of the pool? How would ZFS taste metadata in this situation? -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB |
25.07.2010 20:58, Dan Langille wrote:> NAME STATE READ WRITE CKSUM > storage ONLINE 0 0 0 > raidz2 ONLINE 0 0 0 > gpt/disk01 ONLINE 0 0 0 > gpt/disk02 ONLINE 0 0 0 > gpt/disk03 ONLINE 0 0 0 > gpt/disk04 ONLINE 0 0 0 > gpt/disk05 ONLINE 0 0 0 > /tmp/sparsefile1.img UNAVAIL 0 0 0 corrupted data > /tmp/sparsefile2.img UNAVAIL 0 0 0 corrupted data0k, i'll try it from here. UNAVAIL means ZFS can't locate correct vdev for this pool member. Even if this file exists it's not used by ZFS because it lacks ZFS headers/footers. You can (I think so) reinsert empty file to the pool with: # zpool replace storage /tmp/sparsefile1.img /tmp/sparsefile1.img ^- pool ^- ZFS old vdev name ^- current file If you replace both files you can theoretically bring pool to fully consistent state. Also you can use md to convert files to devices: # mdconfig -a -t vnode -f /tmp/sparsefile1.img md0 And you can use md0 with your pool. -- Sphinx of black quartz judge my vow.
On 7/25/2010 1:58 PM, Dan Langille wrote:> I'm trying to destroy a zfs array which I recently created. It contains > nothing of value. > > # zpool status > pool: storage > state: ONLINE > status: One or more devices could not be used because the label is > missing or > invalid. Sufficient replicas exist for the pool to continue > functioning in a degraded state. > action: Replace the device using 'zpool replace'. > see: http://www.sun.com/msg/ZFS-8000-4J > scrub: none requested > config: > > NAME STATE READ WRITE CKSUM > storage ONLINE 0 0 0 > raidz2 ONLINE 0 0 0 > gpt/disk01 ONLINE 0 0 0 > gpt/disk02 ONLINE 0 0 0 > gpt/disk03 ONLINE 0 0 0 > gpt/disk04 ONLINE 0 0 0 > gpt/disk05 ONLINE 0 0 0 > /tmp/sparsefile1.img UNAVAIL 0 0 0 corrupted data > /tmp/sparsefile2.img UNAVAIL 0 0 0 corrupted data > > errors: No known data errors > > Why sparse files? See this post: > > http://docs.freebsd.org/cgi/getmsg.cgi?fetch=1007077+0+archive/2010/freebsd-stable/20100725.freebsd-stable > > > The two tmp files were created via: > > dd if=/dev/zero of=/tmp/sparsefile1.img bs=1 count=0 oseek=1862g > dd if=/dev/zero of=/tmp/sparsefile2.img bs=1 count=0 oseek=1862g > > And the array created with: > > zpool create -f storage raidz2 gpt/disk01 gpt/disk02 gpt/disk03 \ > gpt/disk04 gpt/disk05 /tmp/sparsefile1.img /tmp/sparsefile2.img > > The -f flag was required to avoid this message: > > invalid vdev specification > use '-f' to override the following errors: > mismatched replication level: raidz contains both files and devices > > > I tried to offline one of the sparse files: > > zpool offline storage /tmp/sparsefile2.img > > That caused a panic: http://www.langille.org/tmp/zpool-offline-panic.jpg > > After rebooting, I rm'd both /tmp/sparsefile1.img and > /tmp/sparsefile2.img without thinking they were still in the zpool. Now > I am unable to destroy the pool. The system panics. I disabled ZFS via > /etc/rc.conf, rebooted, recreated the two sparse files, then did a > forcestart of zfs. Then I saw: > > # zpool status > pool: storage > state: ONLINE > status: One or more devices could not be used because the label is > missing or > invalid. Sufficient replicas exist for the pool to continue > functioning in a degraded state. > action: Replace the device using 'zpool replace'. > see: http://www.sun.com/msg/ZFS-8000-4J > scrub: none requested > config: > > NAME STATE READ WRITE CKSUM > storage ONLINE 0 0 0 > raidz2 ONLINE 0 0 0 > gpt/disk01 ONLINE 0 0 0 > gpt/disk02 ONLINE 0 0 0 > gpt/disk03 ONLINE 0 0 0 > gpt/disk04 ONLINE 0 0 0 > gpt/disk05 ONLINE 0 0 0 > /tmp/sparsefile1.img UNAVAIL 0 0 0 corrupted data > /tmp/sparsefile2.img UNAVAIL 0 0 0 corrupted data > > errors: No known data errors > > > Another attempt to destroy the array created a panic. > > Suggestions as to how to remove this array and get started again?I fixed this by: * reboot zfs_enable="NO" in /etc/rc.conf * rm /boot/zfs/zpool.cache * wiping the first and last 16KB of each partition involved in the array Now I'm trying mdconfig instead of sparse files. Making progress, but not all the way there yet. :) -- Dan Langille - http://langille.org/