zfs discuss - Mar 2009 - invalid vdev configuration after power failure

What does it mean for a vdev to have an invalid configuration and how
can it be fixed or reset? As you can see, the following pool can no
longer be imported: (Note that the "last accessed by another system"
warning is because I moved these drives to my test workstation.)

~$ zpool import -f pool0
cannot import ''pool0'': invalid vdev configuration

~$ zpool import
  pool: pool0
    id: 5915552147942272438
 state: UNAVAIL
status: The pool was last accessed by another system.
action: The pool cannot be imported due to damaged devices or data.
   see: http://www.sun.com/msg/ZFS-8000-EY
config:

	pool0       UNAVAIL  insufficient replicas
	  raidz1    UNAVAIL  corrupted data
	    c5d1p0  ONLINE
	    c4d0p0  ONLINE
	    c4d1p0  ONLINE
	    c6d0p0  ONLINE
	    c5d0p0  ONLINE

that link suggests that this is a problem with a dirty export:

http://www.sun.com/msg/ZFS-8000-EY

maybe try importing on system A again, doing a ''zpool export'',
waiting
for completion, then moving to system B to import?

On Sun, Mar 1, 2009 at 2:29 PM, Kyle Kakligian <smallart at gmail.com>
wrote:
> What does it mean for a vdev to have an invalid configuration and how
> can it be fixed or reset? As you can see, the following pool can no
> longer be imported: (Note that the "last accessed by another
system"
> warning is because I moved these drives to my test workstation.)
>
> ~$ zpool import -f pool0
> cannot import ''pool0'': invalid vdev configuration
>
> ~$ zpool import
> ?pool: pool0
> ? ?id: 5915552147942272438
> ?state: UNAVAIL
> status: The pool was last accessed by another system.
> action: The pool cannot be imported due to damaged devices or data.
> ? see: http://www.sun.com/msg/ZFS-8000-EY
> config:
>
> ? ? ? ?pool0 ? ? ? UNAVAIL ?insufficient replicas
> ? ? ? ? ?raidz1 ? ?UNAVAIL ?corrupted data
> ? ? ? ? ? ?c5d1p0 ?ONLINE
> ? ? ? ? ? ?c4d0p0 ?ONLINE
> ? ? ? ? ? ?c4d1p0 ?ONLINE
> ? ? ? ? ? ?c6d0p0 ?ONLINE
> ? ? ? ? ? ?c5d0p0 ?ONLINE
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
>

On Mon, Mar 2, 2009 at 8:30 AM, Blake <blake.irvin at gmail.com>
wrote:
> that link suggests that this is a problem with a dirty export:
Yes, a loss of power should mean there was no clean export.

On Mon, Mar 2, 2009 at 8:30 AM, Blake <blake.irvin at gmail.com>
wrote:
> maybe try importing on system A again, doing a ''zpool
export'', waiting
> for completion, then moving to system B to import?
I tried that first, but system A will have none of it. It fails with
the "cannot import ''pool0'': invalid vdev
configuration" error just
like system B above.

On Mon, Mar 2, 2009 at 3:57 AM, Victor Latushkin
<Victor.Latushkin at sun.com> wrote:
> What OpenSolaris build are you running?
snv_101b

On Mon, Mar 2, 2009 at 3:57 AM, Victor Latushkin
<Victor.Latushkin at sun.com> wrote:
> Could you please provide output of the following commands:
> zdb -u pool0
> zdb -bcsv pool0
>
> Add ''-e'' if you are running it on your test system.
In both cases, the output is "zdb: can''t open pool0: Invalid
argument"

On Mon, Mar 2, 2009 at 8:30 AM, Blake <blake.irvin at gmail.com>
wrote:
> that link suggests that this is a problem with a dirty export:
Yes, a loss of power should mean there was no clean export.

On Mon, Mar 2, 2009 at 8:30 AM, Blake <blake.irvin at gmail.com>
wrote:
> maybe try importing on system A again, doing a ''zpool
export'', waiting
> for completion, then moving to system B to import?
I tried that first, but system A will have none of it. It fails with
the "cannot import ''pool0'': invalid vdev
configuration" error just
like system B above.

On Mon, Mar 2, 2009 at 3:57 AM, Victor Latushkin
<Victor.Latushkin at sun.com> wrote:
> What OpenSolaris build are you running?
snv_101b

On Mon, Mar 2, 2009 at 3:57 AM, Victor Latushkin
<Victor.Latushkin at sun.com> wrote:
> Could you please provide output of the following commands:
> zdb -u pool0
> zdb -bcsv pool0
>
> Add ''-e'' if you are running it on your test system.
In both cases, the output is "zdb: can''t open pool0: Invalid
argument"

additional comment below...

Kyle Kakligian wrote:
> On Mon, Mar 2, 2009 at 8:30 AM, Blake <blake.irvin at gmail.com>
wrote:
>   
>> that link suggests that this is a problem with a dirty export:
>>     
> Yes, a loss of power should mean there was no clean export.
>
> On Mon, Mar 2, 2009 at 8:30 AM, Blake <blake.irvin at gmail.com>
wrote:
>   
>> maybe try importing on system A again, doing a ''zpool
export'', waiting
>> for completion, then moving to system B to import?
>>     
> I tried that first, but system A will have none of it. It fails with
> the "cannot import ''pool0'': invalid vdev
configuration" error just
> like system B above.
>
> On Mon, Mar 2, 2009 at 3:57 AM, Victor Latushkin
> <Victor.Latushkin at sun.com> wrote:
>   
>> What OpenSolaris build are you running?
>>     
> snv_101b
>
> On Mon, Mar 2, 2009 at 3:57 AM, Victor Latushkin
> <Victor.Latushkin at sun.com> wrote:
>   
>> Could you please provide output of the following commands:
>> zdb -u pool0
>> zdb -bcsv pool0
>>
>> Add ''-e'' if you are running it on your test system.
>>     
> In both cases, the output is "zdb: can''t open pool0: Invalid
argument"
>   
I''ve been meaning to do add some basic "can''t import the
pool"
troubleshooting
tips to the ZFS Troubleshooting Guide, but am a little bit behind right now.
For reference, the guide is at:
http://www.solarisinternals.com/wiki/index.php/ZFS_Troubleshooting_Guide

If the pool cannot be imported, then trying to run "zdb [options]
poolname"
may (will?) not work.  A good first step to troubleshoot this to ensure that
all of the labels can be read from each vdev.  As a reminder, there are 4
labels per vdev.  To read them, try
    zdb -l /dev/rdsk/<vdev>
where <vdev> is the physical device name, usually something like c0t0d0s0.
If you cannot read all 4 labels from all of the vdevs, then you should 
try to
solve that problem first, before moving onto further troubleshooting.
 -- richard

On Wed, Mar 4, 2009 at 7:59 PM, Richard Elling <richard.elling at
gmail.com> wrote:
> additional comment below...
>
> Kyle Kakligian wrote:
>>
>> On Mon, Mar 2, 2009 at 8:30 AM, Blake <blake.irvin at gmail.com>
wrote:
>>
>>>
>>> that link suggests that this is a problem with a dirty export:
>>>
>>
>> Yes, a loss of power should mean there was no clean export.
>>
>> On Mon, Mar 2, 2009 at 8:30 AM, Blake <blake.irvin at gmail.com>
wrote:
>>
>>>
>>> maybe try importing on system A again, doing a ''zpool
export'', waiting
>>> for completion, then moving to system B to import?
>>>
>>
>> I tried that first, but system A will have none of it. It fails with
>> the "cannot import ''pool0'': invalid vdev
configuration" error just
>> like system B above.
>>
>> On Mon, Mar 2, 2009 at 3:57 AM, Victor Latushkin
>> <Victor.Latushkin at sun.com> wrote:
>>
>>>
>>> What OpenSolaris build are you running?
>>>
>>
>> snv_101b
>>
>> On Mon, Mar 2, 2009 at 3:57 AM, Victor Latushkin
>> <Victor.Latushkin at sun.com> wrote:
>>
>>>
>>> Could you please provide output of the following commands:
>>> zdb -u pool0
>>> zdb -bcsv pool0
>>>
>>> Add ''-e'' if you are running it on your test
system.
>>>
>>
>> In both cases, the output is "zdb: can''t open pool0:
Invalid argument"
>>
>
> I''ve been meaning to do add some basic "can''t import
the pool"
> troubleshooting
> tips to the ZFS Troubleshooting Guide, but am a little bit behind right
now.
> For reference, the guide is at:
> http://www.solarisinternals.com/wiki/index.php/ZFS_Troubleshooting_Guide
>
> If the pool cannot be imported, then trying to run "zdb [options]
poolname"
> may (will?) not work.  A good first step to troubleshoot this to ensure
that
> all of the labels can be read from each vdev.  As a reminder, there are 4
> labels per vdev.  To read them, try
>   zdb -l /dev/rdsk/<vdev>
> where <vdev> is the physical device name, usually something like
c0t0d0s0.
> If you cannot read all 4 labels from all of the vdevs, then you should try
> to
> solve that problem first, before moving onto further troubleshooting.
> -- richard
>
>
I will look into that doc, as it appears that my label2 and label3
guys are not parse-able.

On Thu, Mar 5, 2009 at 1:09 PM, Kyle Kakligian <kakligian at google.com>
wrote:
> On Wed, Mar 4, 2009 at 7:59 PM, Richard Elling <richard.elling at
gmail.com> wrote:
>> additional comment below...
>>
>> Kyle Kakligian wrote:
>>>
>>> On Mon, Mar 2, 2009 at 8:30 AM, Blake <blake.irvin at
gmail.com> wrote:
>>>
>>>>
>>>> that link suggests that this is a problem with a dirty export:
>>>>
>>>
>>> Yes, a loss of power should mean there was no clean export.
>>>
Hmmm. I was under the impression that a power loss wouldn''t pose a
threat to my ZFS filesystems..

SOLVED

According to `zdb -l /dev/rdsk/<vdev>`, one of my drives was missing
two of its four redundant labels. (#2 and #3) These two are next to
each other at the end of the device so it makes some sense that they
could both be garbled.

I''m not sure why `zfs import` choked on this [typical?] error case,
but its easy to fix with a very careful dd. I took a different and
very roundabout approach to recover my data, however, since I''m not
confident in my ''careful'' skills. (after all, where''s
my backup?)
Instead, on a linux workstation where I am more cozy, I compiled
zfs-fuse from the source with a slight modification to ignore labels 2
and 3. fusermount worked great and I recovered my data without issue.

Thanks to everyone for all the help. I''m impressed by the amount of
community support here on the list!

On Fri, Mar 6, 2009 at 12:17 AM, Joe S <js.lists at gmail.com>
wrote:
> On Thu, Mar 5, 2009 at 1:09 PM, Kyle Kakligian <kakligian at
google.com> wrote:
>> On Wed, Mar 4, 2009 at 7:59 PM, Richard Elling <richard.elling at
gmail.com> wrote:
>>> additional comment below...
>>>
>>> Kyle Kakligian wrote:
>>>>
>>>> On Mon, Mar 2, 2009 at 8:30 AM, Blake <blake.irvin at
gmail.com> wrote:
>>>>
>>>>>
>>>>> that link suggests that this is a problem with a dirty
export:
>>>>>
>>>>
>>>> Yes, a loss of power should mean there was no clean export.
>>>>
>
> Hmmm. I was under the impression that a power loss wouldn''t pose a
> threat to my ZFS filesystems..
>

Kyle Kakligian <smallart <at> gmail.com> writes:
> I''m not sure why `zfs import` choked on this [typical?] error
case,
> but its easy to fix with a very careful dd. I took a different and
> very roundabout approach to recover my data, however, since I''m
not
> confident in my ''careful'' skills. (after all,
where''s my backup?)
> Instead, on a linux workstation where I am more cozy, I compiled
> zfs-fuse from the source with a slight modification to ignore labels 2
> and 3. fusermount worked great and I recovered my data without issue.
Hi,

waking up the old thread,

would you mind sharing the information how to
edit the zfs-fuse to ignoring labels?

thanks,

regards,

diyanc