zfs discuss - Jul 2009 - zpool import is trying to tell me something...

I recently had an X86 system (running Nexenta Elatte, if that matters -- b101
kernel, I think) suffer hardware failure and refuse to boot.  I''ve
migrated the disks into a SPARC system (b115) in an attempt to bring the data
back online while I see about repairing the former system.  However,
I''m having some trouble with the import process:

hydra# zpool import
  pool: tank
    id: 12156936533394213216
 state: UNAVAIL
status: The pool is formatted using an older on-disk version.
action: The pool cannot be imported due to damaged devices or data.
config:

        tank           UNAVAIL  insufficient replicas
          raidz1       UNAVAIL  corrupted data
            c3t11d0s0  ONLINE
            c3t10d0s0  ONLINE
            c3t8d0s0   ONLINE
            c3t9d0s0   ONLINE

hydra# /usr/lib/zfs/availdevs -p
...
    pool name="tank" id="12156936533394213216"
poolversion="14" used="0" size="0"
replacementsize="0" readbytes="0" writebytes="0"
readoperations="0" writeoperations="0"
readerrors="0" writeerrors="0" checksumerrors="0"
devicestate="VDEV_STATE_CANT_OPEN"
devicestatus="VDEV_AUX_NO_REPLICAS"
poolstate="POOL_STATE_ACTIVE"
poolstatus="ZPOOL_STATUS_VERSION_OLDER"
...

Somebody is trying to tell me something, but I''m not receiving.  All
four disks (there were only ever four in the array, since creation until the
present) appear present, are readable (via dd if=...), have sane looking
uberblock arrays, and give more or less sane looking answers to "zdb -l
/dev/dsk/..."; the whole output of the latter is a bit much to paste here
but I can if somebody thinks it would be useful.  zdb with no arguments also
enumerates the pool in the list and looks saneish.

Other zdb commands apparently do not work, hopefully due to the same root cause.
hydra# zdb -d tank
zdb: can''t open tank: Invalid argument

I really wish I could zdb -c -c right now and at least assess whether or not the
data remains on the platters. :)

If it matters, the pool was not exported from the X86 system prior to migration,
since I didn''t think I''d need to the last time I powered the
machine off.

What am I doing wrong?
Thanks much in advance.
--nwf;
-- 
This message posted from opensolaris.org

Maybe I should have posted the zdb -l output.  Having seen another thread which
suggests that I might be looking at the most recent txg being damaged, I went to
get my pool''s txg counter:

hydra# zdb -l /dev/dsk/c3t10d0s0 | grep txg
    txg=10168474
    txg=10168474
    txg=6324561
    txg=6324561

All of the disks are like this (indeed, the only thing that differs about their
zdb -l output is their own guids, as expected).  Staring at reams of od -x
output, it appears that I have txgs 10168494 through 10168621 in L0 and L1.  L2
and L3 appear to have not been updated in some time!  L0 and L1 are both
"version=14" and have a hostname field.... L2 and L3 are both
"version=3" and do not.  All four labels appear to describe the same
array (guids and devices and all).  The uberblocks in L2 and L3 seem to contain
txgs 6319346 through 6319473.  That''s uh... "funny".

A little bit of dtrace and time travel back to vdev.c as of snv_115 later, I
find that the immediate cause is that vdev_raidz_open is yielding an asize of
1280262012928 but that
when vdev_open called vdev_raidz_open, it thought the asize was 1280272498688. 
(Thus vdev_open and vdev_root_open return EINVAL, and the import fails.)  That
is, the array is actually one megabyte smaller than it thought... which works
out to 256K per disk, which is exactly the size of a label pair and might
explain why L2 and L3 are stale.

Help?
-- 
This message posted from opensolaris.org