Hi,
I have a zfs-pool (unfortunately not setup according to the Best
Practices Guide) that somehow got corrupted after a spontaneous server
reboot. On Solaris 10u4 the machine simply panics when I try to import
the pool. So what I''ve done is taken a dd-image of the whole LUN so
that I have something to lab with without breaking original data. Then
I''ve installed Solaris Express b95 (in a virtual machine) and made the
disk image visible via iSCSI. This in hope that the newer zfs-code in
b95 would be able to import the pool.
Here is the output of some hopefully relevant commands to you zfs-gurus :)
bash-3.2# zpool import
pool: data1
id: 16337833607404088147
state: ONLINE
status: The pool is formatted using an older on-disk version.
action: The pool can be imported using its name or numeric identifier, though
some features will not be available without an explicit ''zpool
upgrade''.
config:
data1 ONLINE
c2t2d0 ONLINE
bash-3.2# zpool import data1
cannot import ''data1'': pool may be in use from other system
use ''-f'' to import anyway
bash-3.2# zpool import -f data1
(when sniffing the iscsi-traffic there is an initial burst of requests
and data, but then the command zpool hangs forever and can''t even be
kill-9''d)
bash-3.2# zdb -l /dev/rdsk/c2t2d0s0
--------------------------------------------
LABEL 0
--------------------------------------------
version=3
name=''data1''
state=0
txg=696136211
pool_guid=16337833607404088147
top_guid=8756997626625498593
guid=8756997626625498593
vdev_tree
type=''disk''
id=0
guid=8756997626625498593
path=''/dev/dsk/c6t213d0s0''
devid=''id1,sd at n4b05a27813001720/a''
whole_disk=1
metaslab_array=13
metaslab_shift=35
ashift=9
asize=6499480109056
DTL=17
(3 more labels with same content follows here)
bash-3.2# zdb -uuu -e data1
Uberblock
magic = 0000000000bab10c
version = 3
txg = 698279317
guid_sum = 6648087160320035124
timestamp = 1214932560 UTC = Tue Jul 1 19:16:00 2008
rootbp = [L0 DMU objset] 400L/200P DVA[0]=<0:56800000200:200>
DVA[1]=<0:3000020200:200> DVA[2]=<0:48800001800:200> fletcher4 lzjb
LE
contiguous birth=698279317 fill=189
cksum=89744d6d8:36e7cf71f81:b1d06b2acd36:1850b4cc5621f3
I have also tried to blindly execute the commands described by
signature kangurek in
<http://www.opensolaris.org/jive/thread.jspa?messageID=220125> but
without success.
Without about knowing much about zfs, I couldn''t help but notice that
the txg ids differs in the label and uberblock. Is this normal?
Any help on moving forward with this recovery process is much appreciated!
Best regards
Erik Gulliksson
Hi Erik, Erik Gulliksson wrote:> Hi, > > I have a zfs-pool (unfortunately not setup according to the Best > Practices Guide) that somehow got corrupted after a spontaneous server > reboot. On Solaris 10u4 the machine simply panics when I try to import > the pool.Panic stack would be useful.> So what I''ve done is taken a dd-image of the whole LUN so > that I have something to lab with without breaking original data. Then > I''ve installed Solaris Express b95 (in a virtual machine) and made the > disk image visible via iSCSI. This in hope that the newer zfs-code in > b95 would be able to import the pool. > > Here is the output of some hopefully relevant commands to you zfs-gurus :) > > bash-3.2# zpool import > pool: data1 > id: 16337833607404088147 > state: ONLINE > status: The pool is formatted using an older on-disk version. > action: The pool can be imported using its name or numeric identifier, though > some features will not be available without an explicit ''zpool upgrade''. > config: > data1 ONLINE > c2t2d0 ONLINE > > bash-3.2# zpool import data1 > cannot import ''data1'': pool may be in use from other system > use ''-f'' to import anyway > > bash-3.2# zpool import -f data1 > (when sniffing the iscsi-traffic there is an initial burst of requests > and data, but then the command zpool hangs forever and can''t even be > kill-9''d)It is apparently blocked somewhere in kernel. Try to do something like this from another window to get better idea: echo "<pid of zpool>::pid2proc|::walk thread|::findtsack -v" | mdb -k echo "::threadlist -v" | mdb -k Some useful information may be logged in FMA, try to see what is available there with fmdump -eV On Nevada you can try the following to (same option repeated several times increases verbosity): zdb -e -bb data1 zdb -e -dddd data1 This may take a while on 6 Tb pool depending on how full it is...> bash-3.2# zdb -l /dev/rdsk/c2t2d0s0 > -------------------------------------------- > LABEL 0 > -------------------------------------------- > version=3 > name=''data1'' > state=0 > txg=696136211 > pool_guid=16337833607404088147 > top_guid=8756997626625498593 > guid=8756997626625498593 > vdev_tree > type=''disk'' > id=0 > guid=8756997626625498593 > path=''/dev/dsk/c6t213d0s0'' > devid=''id1,sd at n4b05a27813001720/a'' > whole_disk=1 > metaslab_array=13 > metaslab_shift=35 > ashift=9 > asize=6499480109056 > DTL=17 > (3 more labels with same content follows here) > > bash-3.2# zdb -uuu -e data1 > Uberblock > magic = 0000000000bab10c > version = 3 > txg = 698279317 > guid_sum = 6648087160320035124 > timestamp = 1214932560 UTC = Tue Jul 1 19:16:00 2008 > rootbp = [L0 DMU objset] 400L/200P DVA[0]=<0:56800000200:200> > DVA[1]=<0:3000020200:200> DVA[2]=<0:48800001800:200> fletcher4 lzjb LE > contiguous birth=698279317 fill=189 > cksum=89744d6d8:36e7cf71f81:b1d06b2acd36:1850b4cc5621f3 > > I have also tried to blindly execute the commands described by > signature kangurek in > <http://www.opensolaris.org/jive/thread.jspa?messageID=220125> but > without success. > > Without about knowing much about zfs, I couldn''t help but notice that > the txg ids differs in the label and uberblock. Is this normal?Yes. Btw, why does timestamp on your uberblock show July 1? wbr, victor
Hi Victor, Thanks for the prompt reply. Here are the results from your suggestions.> Panic stack would be useful.I''m sorry I don''t have this available and I don''t want to cause another panic :)> > It is apparently blocked somewhere in kernel. Try to do something like this > from another window to get better idea: > > echo "<pid of zpool>::pid2proc|::walk thread|::findtsack -v" | mdb -k > echo "::threadlist -v" | mdb -k >window 1: bash-3.2# zpool import -f data1 window 2: bash-3.2# ps -fel | grep zpool 0 S root 897 874 0 40 20 ? 1262 ? 15:03:23 pts/3 0:00 zpool import -f data1 bash-3.2# echo "0t897::pid2proc|::walk thread|::findstack -v" | mdb -k stack pointer for thread ffffff01b57ab840: ffffff00083709f0 [ ffffff00083709f0 _resume_from_idle+0xf1() ] ffffff0008370a30 swtch+0x17f() ffffff0008370a60 cv_wait+0x61(ffffff01b3bd71ca, ffffff01b3bd7188) ffffff0008370ab0 txg_wait_synced+0x81(ffffff01b3bd7000, 299ee597) ffffff0008370b10 spa_config_update_common+0x79(ffffff01b42a8a80, 0, 0) ffffff0008370bb0 spa_import_common+0x36e(ffffff01b5ad4000, ffffff01b5325310, 0 , 0, 0) ffffff0008370be0 spa_import+0x1e(ffffff01b5ad4000, ffffff01b5325310, 0) ffffff0008370c30 zfs_ioc_pool_import+0xad(ffffff01b5ad4000) ffffff0008370cb0 zfsdev_ioctl+0x10d(b600000000, 5a02, 80424f0, 100003, ffffff01b3f181a0, ffffff0008370e9c) ffffff0008370cf0 cdev_ioctl+0x48(b600000000, 5a02, 80424f0, 100003, ffffff01b3f181a0, ffffff0008370e9c) ffffff0008370d30 spec_ioctl+0x86(ffffff01adf41d00, 5a02, 80424f0, 100003, ffffff01b3f181a0, ffffff0008370e9c, 0) ffffff0008370db0 fop_ioctl+0x7b(ffffff01adf41d00, 5a02, 80424f0, 100003, ffffff01b3f181a0, ffffff0008370e9c, 0) ffffff0008370ec0 ioctl+0x174(3, 5a02, 80424f0) ffffff0008370f10 _sys_sysenter_post_swapgs+0x14b() bash-3.2# echo "::threadlist -v" | mdb -k Output a bit too long to post here. Is there anything in particular i should look for in this output?> > Some useful information may be logged in FMA, try to see what is available > there with > > fmdump -eV- I get a few ereport.fs.zfs.checksum reports like this bash-3.2# fmdump -eV Aug 22 2008 15:03:23.203687016 ereport.fs.zfs.checksum nvlist version: 0 class = ereport.fs.zfs.checksum ena = 0x1a77b8287a00001 detector = (embedded nvlist) nvlist version: 0 version = 0x0 scheme = zfs pool = 0xe2bba51ab8c26b53 vdev = 0x79871af1e1f39de1 (end detector) pool = data1 pool_guid = 0xe2bba51ab8c26b53 pool_context = 0 pool_failmode = wait vdev_guid = 0x79871af1e1f39de1 vdev_type = disk vdev_path = /dev/dsk/c2t2d0s0 vdev_devid = id1,sd at t494554000000000000000000020000008300000001008000/a parent_guid = 0xe2bba51ab8c26b53 parent_type = root zio_err = 50 zio_offset = 0x1e800416c00 zio_size = 0x400 zio_objset = 0x0 zio_object = 0xb zio_level = 0 zio_blkid = 0x0 __ttl = 0x1 Aug 22 2008 15:03:23.203687247 ereport.fs.zfs.data nvlist version: 0 class = ereport.fs.zfs.data ena = 0x1a77b8287a00001 detector = (embedded nvlist) nvlist version: 0 version = 0x0 scheme = zfs pool = 0xe2bba51ab8c26b53 (end detector) pool = data1 pool_guid = 0xe2bba51ab8c26b53 pool_context = 0 pool_failmode = wait zio_err = 50 zio_objset = 0x0 zio_object = 0xb zio_level = 0 zio_blkid = 0x0 __ttl = 0x1 __tod = 0x48aeb91b 0xc24054f Aug 22 2008 15:03:23.207225717 ereport.fs.zfs.io_failure nvlist version: 0 class = ereport.fs.zfs.io_failure ena = 0x1a77ee27cb00001 detector = (embedded nvlist) nvlist version: 0 version = 0x0 scheme = zfs pool = 0xe2bba51ab8c26b53 (end detector) pool = data1 pool_guid = 0xe2bba51ab8c26b53 pool_context = 0 pool_failmode = wait __ttl = 0x1 __tod = 0x48aeb91b 0xc5a0375> > On Nevada you can try the following to (same option repeated several times > increases verbosity): > > zdb -e -bb data1 > zdb -e -dddd data1 >bash-3.2# zdb -e -bb data1 Traversing all blocks to verify nothing leaked ... out of memory -- generating core dump Abort Seems i need to get a machine with more ram to do this :) This can be arranged on monday. bash-3.2# zdb -e -dddd data1 Dataset mos [META], ID 0, cr_txg 4, 210M, 189 objects, rootbp [L0 DMU objset] 400L/200P DVA[0]=<0:56800000200:200> DVA[1]=<0:3000020200:200> DVA[2]=<0:48800001800:200> fletcher4 lzjb LE contiguous birth=698279317 fill=189 cksum=89744d6d8:36e7cf71f81:b1d06b2acd36:1850b4cc5621f3 Object lvl iblk dblk lsize asize type 0 2 16K 16K 96.0K 94.5K DMU dnode Object lvl iblk dblk lsize asize type 1 1 16K 512 512 1K object directory microzap: 512 bytes, 6 entries sync_bplist = 12 config = 11 root_dataset = 2 errlog_scrub = 0 deflate = 1 errlog_last = 0 Object lvl iblk dblk lsize asize type 2 1 16K 512 512 0 DSL directory 256 bonus DSL directory creation_time = Tue Mar 18 12:03:05 2008 head_dataset_obj = 5 parent_dir_obj = 0 origin_obj = 0 child_dir_zapobj = 4 used_bytes = 4.41T compressed_bytes = 4.40T uncompressed_bytes = 4.40T quota = 0 reserved = 0 props_zapobj = 3 deleg_zapobj = 0 Object lvl iblk dblk lsize asize type 3 1 16K 16K 32K 7.00K DSL props Fat ZAP stats: Pointer table: 1024 elements zt_blk: 0 zt_numblks: 0 zt_shift: 10 zt_blks_copied: 0 zt_nextblk: 0 ZAP entries: 3 Leaf blocks: 1 Total blocks: 2 zap_block_type: 0x8000000000000001 zap_magic: 0x2f52ab2ab zap_salt: 0x1d0e3ebe7 Leafs with 2^n pointers: 9: 1 * Blocks with n*5 entries: 0: 1 * Blocks n/10 full: 1: 1 * Entries with n chunks: 3: 3 *** Buckets with n entries: 0: 509 **************************************** 1: 3 * atime = 0 exec = 0 mountpoint = legacy Object lvl iblk dblk lsize asize type 4 1 16K 512 512 1K DSL directory child map microzap: 512 bytes, 1 entries $MOS = 8 Object lvl iblk dblk lsize asize type 5 1 16K 512 512 0 DSL dataset 320 bonus DSL dataset dir_obj = 2 prev_snap_obj = 0 prev_snap_txg = 0 next_snap_obj = 0 snapnames_zapobj = 6 num_children = 0 creation_time = Tue Mar 18 12:03:05 2008 creation_txg = 4 deadlist_obj = 7 used_bytes = 4.41T compressed_bytes = 4.40T uncompressed_bytes = 4.40T unique = 4.41T fsid_guid = 63153422916496006 guid = 3076616381815835747 flags = 0 next_clones_obj = 0 bp = [L0 DMU objset] 400L/200P DVA[0]=<0:56800000000:200> DVA[1]=<0:3000020000:200> DVA[2]=<0:48800000a00:200> fletcher4 lzjb LE contiguous birth=698279317 fill=6040564 cksum=f9e53509d:5f89f3d9404:12e97014634a8:294b54e7bf25c2 Object lvl iblk dblk lsize asize type 6 1 16K 512 512 1K DSL dataset snap map microzap: 512 bytes, 0 entries Object lvl iblk dblk lsize asize type 7 1 16K 128K 128K 0 bplist 32 bonus bplist header Object lvl iblk dblk lsize asize type 8 1 16K 512 512 0 DSL directory 256 bonus DSL directory creation_time = Tue Mar 18 12:03:05 2008 head_dataset_obj = 0 parent_dir_obj = 2 origin_obj = 0 child_dir_zapobj = 10 used_bytes = 210M compressed_bytes = 104M uncompressed_bytes = 104M quota = 0 reserved = 0 props_zapobj = 9 deleg_zapobj = 0 Object lvl iblk dblk lsize asize type 9 1 16K 512 512 1K DSL props microzap: 512 bytes, 0 entries Object lvl iblk dblk lsize asize type 10 1 16K 512 512 1K DSL directory child map microzap: 512 bytes, 0 entries Object lvl iblk dblk lsize asize type 11 1 16K 16K 16K 2K packed nvlist 8 bonus packed nvlist size Assertion failed: 0 == dmu_read(os, object, 0, nvsize, packed), file ../zdb.c, line 216, function dump_packed_nvlist> Btw, why does timestamp on your uberblock show July 1?Well, this is about the time when the crash happened. The clock on the server is correct. Once again, thank you for taking the time to look into this. cheers Erik
Erik, could you please provide a little bit more details. Erik Gulliksson wrote:> Hi, > > I have a zfs-pool (unfortunately not setup according to the Best > Practices Guide) that somehow got corrupted after a spontaneous server > reboot.Was it totally spontaneous? What was the uptime before panic? Systems messages on you Solaris 10 machine may have some clues.> On Solaris 10u4 the machine simply panics when I try to import > the pool. So what I''ve done is taken a dd-image of the whole LUN soIs it SAN-based LUN? How many systems can access this LUN? Is there any chance that it might get imported by some other system while being in use on this one? wbr, victor
Erik Gulliksson wrote:> Hi Victor, > > Thanks for the prompt reply. Here are the results from your suggestions. > >> Panic stack would be useful. > I''m sorry I don''t have this available and I don''t want to cause another panic :)It should be saved in system messages on your Solaris 10 machine (unless it''s power was not removed abruptly).>> It is apparently blocked somewhere in kernel. Try to do something like this >> from another window to get better idea: >> >> echo "<pid of zpool>::pid2proc|::walk thread|::findtsack -v" | mdb -k >> echo "::threadlist -v" | mdb -k >> > window 1: > bash-3.2# zpool import -f data1 > window 2: > bash-3.2# ps -fel | grep zpool > 0 S root 897 874 0 40 20 ? 1262 ? 15:03:23 > pts/3 0:00 zpool import -f data1 > bash-3.2# echo "0t897::pid2proc|::walk thread|::findstack -v" | mdb -k > stack pointer for thread ffffff01b57ab840: ffffff00083709f0 > [ ffffff00083709f0 _resume_from_idle+0xf1() ] > ffffff0008370a30 swtch+0x17f() > ffffff0008370a60 cv_wait+0x61(ffffff01b3bd71ca, ffffff01b3bd7188) > ffffff0008370ab0 txg_wait_synced+0x81(ffffff01b3bd7000, 299ee597) > ffffff0008370b10 spa_config_update_common+0x79(ffffff01b42a8a80, 0, 0) > ffffff0008370bb0 spa_import_common+0x36e(ffffff01b5ad4000, ffffff01b5325310, 0 > , 0, 0) > ffffff0008370be0 spa_import+0x1e(ffffff01b5ad4000, ffffff01b5325310, 0) > ffffff0008370c30 zfs_ioc_pool_import+0xad(ffffff01b5ad4000) > ffffff0008370cb0 zfsdev_ioctl+0x10d(b600000000, 5a02, 80424f0, 100003, > ffffff01b3f181a0, ffffff0008370e9c) > ffffff0008370cf0 cdev_ioctl+0x48(b600000000, 5a02, 80424f0, 100003, > ffffff01b3f181a0, ffffff0008370e9c) > ffffff0008370d30 spec_ioctl+0x86(ffffff01adf41d00, 5a02, 80424f0, 100003, > ffffff01b3f181a0, ffffff0008370e9c, 0) > ffffff0008370db0 fop_ioctl+0x7b(ffffff01adf41d00, 5a02, 80424f0, 100003, > ffffff01b3f181a0, ffffff0008370e9c, 0) > ffffff0008370ec0 ioctl+0x174(3, 5a02, 80424f0) > ffffff0008370f10 _sys_sysenter_post_swapgs+0x14b() > > bash-3.2# echo "::threadlist -v" | mdb -k > Output a bit too long to post here. Is there anything in particular i > should look for in this output?Well, since we are talking about ZFS any thread somewhere in ZFS module are interesting, and there should not be too many of them. Though in this case it is clear - it is trying to update config object and waits for the update to sync. There should be another thread with stack similar to this: genunix:cv_wait() zfs:zio_wait() zfs:dbuf_read() zfs:dmu_buf_will_dirty() zfs:dmu_write() zfs:spa_sync_nvlist() zfs:spa_sync_config_object() zfs:spa_sync() zfs:txg_sync_thread() unix:thread_start() It wait due to checksum error detected while reading old config object from disk (call to dmu_read() above). It means that all ditto-blocks of config object got corrupted. On Solaris 10 there''s no>> Some useful information may be logged in FMA, try to see what is available >> there with >> >> fmdump -eV > > - I get a few ereport.fs.zfs.checksum reports like this > bash-3.2# fmdump -eV > Aug 22 2008 15:03:23.203687016 ereport.fs.zfs.checksum > nvlist version: 0 > class = ereport.fs.zfs.checksum > ena = 0x1a77b8287a00001 > detector = (embedded nvlist) > nvlist version: 0 > version = 0x0 > scheme = zfs > pool = 0xe2bba51ab8c26b53 > vdev = 0x79871af1e1f39de1 > (end detector) > > pool = data1 > pool_guid = 0xe2bba51ab8c26b53 > pool_context = 0 > pool_failmode = wait > vdev_guid = 0x79871af1e1f39de1 > vdev_type = disk > vdev_path = /dev/dsk/c2t2d0s0 > vdev_devid = id1,sd at t494554000000000000000000020000008300000001008000/a > parent_guid = 0xe2bba51ab8c26b53 > parent_type = root > zio_err = 50 > zio_offset = 0x1e800416c00 > zio_size = 0x400 > zio_objset = 0x0 > zio_object = 0xb > zio_level = 0 > zio_blkid = 0x0 > __ttl = 0x1 > > Aug 22 2008 15:03:23.203687247 ereport.fs.zfs.data > nvlist version: 0 > class = ereport.fs.zfs.data > ena = 0x1a77b8287a00001 > detector = (embedded nvlist) > nvlist version: 0 > version = 0x0 > scheme = zfs > pool = 0xe2bba51ab8c26b53 > (end detector) > > pool = data1 > pool_guid = 0xe2bba51ab8c26b53 > pool_context = 0 > pool_failmode = wait > zio_err = 50 > zio_objset = 0x0 > zio_object = 0xb > zio_level = 0 > zio_blkid = 0x0 > __ttl = 0x1 > __tod = 0x48aeb91b 0xc24054f > > Aug 22 2008 15:03:23.207225717 ereport.fs.zfs.io_failure > nvlist version: 0 > class = ereport.fs.zfs.io_failure > ena = 0x1a77ee27cb00001 > detector = (embedded nvlist) > nvlist version: 0 > version = 0x0 > scheme = zfs > pool = 0xe2bba51ab8c26b53 > (end detector) > > pool = data1 > pool_guid = 0xe2bba51ab8c26b53 > pool_context = 0 > pool_failmode = wait > __ttl = 0x1 > __tod = 0x48aeb91b 0xc5a0375 > >> On Nevada you can try the following to (same option repeated several times >> increases verbosity): >> >> zdb -e -bb data1 >> zdb -e -dddd data1 >> > > > bash-3.2# zdb -e -bb data1 > Traversing all blocks to verify nothing leaked ... > out of memory -- generating core dump > Abort > > Seems i need to get a machine with more ram to do this :) This can be > arranged on monday.This is not needed now.> bash-3.2# zdb -e -dddd data1 > Dataset mos [META], ID 0, cr_txg 4, 210M, 189 objects, rootbp [L0 DMU > objset] 400L/200P DVA[0]=<0:56800000200:200> DVA[1]=<0:3000020200:200> > DVA[2]=<0:48800001800:200> fletcher4 lzjb LE contiguous > birth=698279317 fill=189 > cksum=89744d6d8:36e7cf71f81:b1d06b2acd36:1850b4cc5621f3 > > Object lvl iblk dblk lsize asize type > 0 2 16K 16K 96.0K 94.5K DMU dnode ><====== lots of output skipped ======>> > Object lvl iblk dblk lsize asize type > 11 1 16K 16K 16K 2K packed nvlist > 8 bonus packed nvlist size > > Assertion failed: 0 == dmu_read(os, object, 0, nvsize, packed), file > ../zdb.c, line 216, function dump_packed_nvlistHere zdb crashes on dmu_read() of config object, since it expects it to return no error.>> Btw, why does timestamp on your uberblock show July 1? > Well, this is about the time when the crash happened. The clock on the > server is correct.Wow! Why did you wait almost two months? Victor
Victor,> Was it totally spontaneous? What was the uptime before panic? Systems > messages on you Solaris 10 machine may have some clues.I actually don''t know exactly what happened (this was during my vacation). Monitoring graphs shows that load was very high on this particular server this day. Approximately the machine had about 20 days uptime when the panic happened (it is a Sun Fire X4100 btw). I can''t find any log entries from this date unfortunately, will investigate it further on monday.> Is it SAN-based LUN? How many systems can access this LUN? Is there any > chance that it might get imported by some other system while being in use on > this one?The LUN (set up as RAID6) is exported from a device that comes from a vendor called easyRAID, we talk to this device with iSCSI. The iSCSI-VLANs is tagged in a way that only the crashing server have access to the box. Do you find anything that indicates that more than one box imported the pool at the same time? Regards Erik
Victor,> Well, since we are talking about ZFS any thread somewhere in ZFS module are > interesting, and there should not be too many of them. Though in this case > it is clear - it is trying to update config object and waits for the update > to sync. There should be another thread with stack similar to this: > > genunix:cv_wait() > zfs:zio_wait() > zfs:dbuf_read() > zfs:dmu_buf_will_dirty() > zfs:dmu_write() > zfs:spa_sync_nvlist() > zfs:spa_sync_config_object() > zfs:spa_sync() > zfs:txg_sync_thread() > unix:thread_start() >Ok, this would be the thread you''re referring to: ffffff000746ec80 fffffffffbc287b0 0 0 60 ffffff01b53ebd88 PC: _resume_from_idle+0xf1 THREAD: txg_sync_thread() stack pointer for thread ffffff000746ec80: ffffff000746e860 [ ffffff000746e860 _resume_from_idle+0xf1() ] swtch+0x17f() cv_wait+0x61() zio_wait+0x5f() dbuf_read+0x1b5() dbuf_will_dirty+0x3d() dmu_write+0xcd() spa_sync_nvlist+0xa7() spa_sync_config_object+0x71() spa_sync+0x20b() txg_sync_thread+0x226() thread_start+8() As you say, there are more zfs/spa/txg-threads that end up in the same wait-state.> It wait due to checksum error detected while reading old config object from > disk (call to dmu_read() above). It means that all ditto-blocks of config > object got corrupted. On Solaris 10 there''s noNow this is getting interesting. Is there any chance to recover from this scenario?>>> Btw, why does timestamp on your uberblock show July 1? >> >> Well, this is about the time when the crash happened. The clock on the >> server is correct. > > Wow! Why did you wait almost two months?There has been a lot on reading up about zfs and finding local recovery-experts. But yes, of course I should have posted to this mailing-list earlier. Again, thanks Victor! Regards Erik
To keep everyone updated - Thanks to Victor we have recovered AND repaired all of the data that was lost in the incident. Victor may be able to explain in detail what he did to accomplish this, I only know it involved loading a patched zfs kernel module. I would like to shout a big thanks to Victor Latushkin, a true hero! Best regards Erik Gulliksson On Fri, Aug 22, 2008 at 6:03 PM, Erik Gulliksson <erik at gulliksson.org> wrote:> Victor, > >> Well, since we are talking about ZFS any thread somewhere in ZFS module are >> interesting, and there should not be too many of them. Though in this case >> it is clear - it is trying to update config object and waits for the update >> to sync. There should be another thread with stack similar to this: >> >> genunix:cv_wait() >> zfs:zio_wait() >> zfs:dbuf_read() >> zfs:dmu_buf_will_dirty() >> zfs:dmu_write() >> zfs:spa_sync_nvlist() >> zfs:spa_sync_config_object() >> zfs:spa_sync() >> zfs:txg_sync_thread() >> unix:thread_start() >> > > Ok, this would be the thread you''re referring to: > > ffffff000746ec80 fffffffffbc287b0 0 0 60 ffffff01b53ebd88 > PC: _resume_from_idle+0xf1 THREAD: txg_sync_thread() > stack pointer for thread ffffff000746ec80: ffffff000746e860 > [ ffffff000746e860 _resume_from_idle+0xf1() ] > swtch+0x17f() > cv_wait+0x61() > zio_wait+0x5f() > dbuf_read+0x1b5() > dbuf_will_dirty+0x3d() > dmu_write+0xcd() > spa_sync_nvlist+0xa7() > spa_sync_config_object+0x71() > spa_sync+0x20b() > txg_sync_thread+0x226() > thread_start+8() > > As you say, there are more zfs/spa/txg-threads that end up in the same > wait-state. > >> It wait due to checksum error detected while reading old config object from >> disk (call to dmu_read() above). It means that all ditto-blocks of config >> object got corrupted. On Solaris 10 there''s no > > Now this is getting interesting. Is there any chance to recover from > this scenario? > >>>> Btw, why does timestamp on your uberblock show July 1? >>> >>> Well, this is about the time when the crash happened. The clock on the >>> server is correct. >> >> Wow! Why did you wait almost two months? > There has been a lot on reading up about zfs and finding local > recovery-experts. But yes, of course I should have posted to this > mailing-list earlier. > > Again, thanks Victor! > > Regards > Erik >
Hi Erik, hi Victor, I have exactly the same problem as you described in your thread. Could you please explain to me what to do to recover the data on the pool? Thanks in advance, Jens Hamisch -- This message posted from opensolaris.org
Hi Jens, Jens Hamisch wrote:> Hi Erik, > hi Victor, > > > I have exactly the same problem as you described in your thread.Exactly same problem would mean that only config object in the pool is corrupted. Are you 100% sure that you have exact same problem?> Could you please explain to me what to do to recover the data > on the pool?If answer is yes, then all you need to do is import your pool with build 99 or later and then export it. This will fix config object. Cheers, Victor
Hi Victor, ok, not exactly ... zdb -e -bb share fails on an assertions as follows: <zfsnix,root> /root 16 # zdb -e -bb share Traversing all blocks to verify nothing leaked ... Assertion failed: space_map_load(&msp->ms_map, &zdb_space_map_ops, 0x0, &msp->ms_smo, spa->spa_meta_objset) == 0, file ../zdb.c, line 1416, function zdb_leak_init Abort We''re already running Solaris Express Build 101. Jens -- This message posted from opensolaris.org
Hello,
I''ve run into a problem with zpool import that seems very similar to
the following thread as far as I can tell:
http://opensolaris.org/jive/thread.jspa?threadID=70205&tstart=15
The suggested solution was to use a later version of open solaris (b99 or later)
but that did not work. I''ve tried the following versions of solaris
without success:
Solaris 10 u4 (original system)
Solaris 10 u6
Opensolaris 2008.11
Opensolaris 2008.11 b99
SXCE b113
Any help with this will be greatly appreciated...my last backup was four months
ago so a lot of my thesis work will be lost. I mistakenly thought a mirrored
zpool on new drives would be good enough for a while.
So here''s what happened: we had a power outage one day and as soon as I
tried to boot the server again it enters an endless reboot cycle. So I thought
the OS drive became corrupted (not mirrored) and reinstalled the OS. Then when I
try zpool import it hangs forever. I even left it going for a couple days in
case it was trying to correct corrupted data. The same thing happens no matter
what version of solaris I use. The symptoms and diagnostic results (see below)
seem to be very similar to the post above but the solution doesn''t
work.
Please let me know if you need any other information.
Thanks,
Brad
bash-3.2# zpool import
pool: tank
id: 4410438565134310480
state: ONLINE
status: The pool is formatted using an older on-disk version.
action: The pool can be imported using its name or numeric identifier, though
some features will not be available without an explicit ''zpool
upgrade''.
config:
tank ONLINE
mirror ONLINE
c2d0p0 ONLINE
c1d0p0 ONLINE
bash-3.2# zpool import tank
cannot import ''tank'': pool may be in use from other system
use ''-f'' to import anyway
bash-3.2# zpool import -f tank
(then it hangs here forever, can''t be killed)
(the following commands were performed while this was running)
bash-3.2# fmdump -eV
TIME CLASS
May 27 2009 22:22:55.308533986 ereport.fs.zfs.checksum
nvlist version: 0
class = ereport.fs.zfs.checksum
ena = 0xd22e37db9000401
detector = (embedded nvlist)
nvlist version: 0
version = 0x0
scheme = zfs
pool = 0x3d350681ea839c50
vdev = 0x85cc302105002c5d
(end detector)
pool = tank
pool_guid = 0x3d350681ea839c50
pool_context = 0
pool_failmode = wait
vdev_guid = 0x85cc302105002c5d
vdev_type = disk
vdev_path = /dev/dsk/c2d0p0
vdev_devid = id1,cmdk at AST3750640AS=____________5QD3MYRH/q
parent_guid = 0x8fb729a008f16e65
parent_type = mirror
zio_err = 50
zio_offset = 0x2500407000
zio_size = 0x1000
zio_objset = 0x0
zio_object = 0x23
zio_level = 0
zio_blkid = 0x0
__ttl = 0x1
__tod = 0x4a1e038f 0x1263dae2
(and many others just like this)
bash-3.2# echo "0t3735::pid2proc|::walk thread|::findtsack -v" | mdb
-k
stack pointer for thread df6ed220: e1ca8c54
e1ca8c94 swtch+0x188()
e1ca8ca4 cv_wait+0x53(e08442aa, e084426c, ffffffff, f9c305ac)
e1ca8ce4 txg_wait_synced+0x90(e0844100, 252b4c, 0, 0)
e1ca8d34 spa_config_update_common+0x88(d6c429c0, 0, 0, e1ca8d68)
e1ca8d84 spa_import_common+0x3cf()
e1ca8db4 spa_import+0x18(dbfcf000, e3c0d018, 0, f9c65810)
e1ca8de4 zfs_ioc_pool_import+0xcd(dbfcf000, 0, 0)
e1ca8e14 zfsdev_ioctl+0x124()
e1ca8e44 cdev_ioctl+0x31(2d80000, 5a02, 80418d0, 100003, dfde91f8, e1ca8f00)
e1ca8e74 spec_ioctl+0x6b(d7a593c0, 5a02, 80418d0, 100003, dfde91f8, e1ca8f00)
e1ca8ec4 fop_ioctl+0x49(d7a593c0, 5a02, 80418d0, 100003, dfde91f8, e1ca8f00)
e1ca8f84 ioctl+0x171()
e1ca8fac sys_sysenter+0x106()
bash-3.2# echo "::threadlist -v" | mdb -k
d4ed8dc0 fec1f558 0 0 60 d5033604
PC: _resume_from_idle+0xb1 THREAD: txg_sync_thread()
stack pointer for thread d4ed8dc0: d4ed8ba8
swtch+0x188()
cv_wait+0x53()
zio_wait+0x55()
vdev_uberblock_sync_list+0x19e()
vdev_config_sync+0x11c()
spa_sync+0x5a5()
txg_sync_thread+0x308()
thread_start+8()
(just chose one seemingly relevant thread from the long list)
bash-3.2# zdb -e -bb tank
Traversing all blocks to verify nothing leaked ...
Assertion failed: space_map_load(&msp->ms_map, &zdb_space_map_ops,
0x0, &msp->ms_smo, spa->spa_meta_objset) == 0, file ../zdb.c, line
1420, function
zdb_leak_init
Abort (core dumped)
bash-3.2# zdb -e -dddd tank
Dataset mos [META], ID 0, cr_txg 4, 10.3M, 137 objects, rootbp [L0 DMU objset]
400L/400P DVA[0]=<0:2500005c00:400> DVA[1]=<0:4400005c00:400>
DVA[2]=<0:6300003000:400> fletcher4 uncompressed LE contiguous
birth=2435914 fill=137 cksum=5224494b4:4524146f316:1d44c6f4690ea:84ef3bd0c105a0
Object lvl iblk dblk lsize asize type
0 2 16K 16K 80.0K 180K DMU dnode
Object lvl iblk dblk lsize asize type
1 1 16K 512 512 1K object directory
microzap: 512 bytes, 5 entries
history = 13
root_dataset = 2
sync_bplist = 12
config = 11
deflate = 1
Object lvl iblk dblk lsize asize type
2 1 16K 512 512 0 DSL directory
256 bonus DSL directory
creation_time = Wed Nov 26 02:07:49 2008
head_dataset_obj = 5
parent_dir_obj = 0
origin_obj = 0
child_dir_zapobj = 4
used_bytes = 402G
compressed_bytes = 401G
uncompressed_bytes = 401G
quota = 0
reserved = 0
props_zapobj = 3
deleg_zapobj = 0
flags = 0
used_breakdown[HEAD] = 0
used_breakdown[SNAP] = 0
used_breakdown[CHILD] = 0
used_breakdown[CHILD_RSRV] = 0
used_breakdown[REFRSRV] = 0
Object lvl iblk dblk lsize asize type
3 1 16K 512 512 1K DSL props
microzap: 512 bytes, 0 entries
Object lvl iblk dblk lsize asize type
4 1 16K 512 512 1K DSL directory child map
microzap: 512 bytes, 1 entries
$MOS = 8
Object lvl iblk dblk lsize asize type
5 1 16K 512 512 0 DSL dataset
320 bonus DSL dataset
dir_obj = 2
prev_snap_obj = 99
prev_snap_txg = 2088469
next_snap_obj = 0
snapnames_zapobj = 6
num_children = 0
creation_time = Wed Nov 26 02:07:49 2008
creation_txg = 4
deadlist_obj = 100
used_bytes = 392G
compressed_bytes = 392G
uncompressed_bytes = 392G
unique = 11.3G
fsid_guid = 43614162550022126
guid = 431952237769608890
flags = 0
next_clones_obj = 0
props_obj = 0
bp = [L0 DMU objset] 400L/200P DVA[0]=<0:2500756400:200>
DVA[1]=<0:44005dce00:200> DVA[2]=<0:630011fe00:200> fletcher4 lzjb
LE contiguous birth=2435911 fill=769038
cksum=b131ea18e:48f57187abe:f4e1282becf2:22e5d4ef0c414d
Object lvl iblk dblk lsize asize type
6 1 16K 512 512 1K DSL dataset snap map
microzap: 512 bytes, 2 entries
tank-2008-12-30 = 97
tank-2009-03-30 = 99
Object lvl iblk dblk lsize asize type
7 1 16K 128K 128K 0 bplist
32 bonus bplist header
Object lvl iblk dblk lsize asize type
8 1 16K 512 512 0 DSL directory
256 bonus DSL directory
creation_time = Wed Nov 26 02:07:49 2008
head_dataset_obj = 0
parent_dir_obj = 2
origin_obj = 0
child_dir_zapobj = 10
used_bytes = 10.3M
compressed_bytes = 5.01M
uncompressed_bytes = 5.01M
quota = 0
reserved = 0
props_zapobj = 9
deleg_zapobj = 0
flags = 0
used_breakdown[HEAD] = 0
used_breakdown[SNAP] = 0
used_breakdown[CHILD] = 0
used_breakdown[CHILD_RSRV] = 0
used_breakdown[REFRSRV] = 0
Object lvl iblk dblk lsize asize type
9 1 16K 512 512 1K DSL props
microzap: 512 bytes, 0 entries
Object lvl iblk dblk lsize asize type
10 1 16K 512 512 1K DSL directory child map
microzap: 512 bytes, 0 entries
Object lvl iblk dblk lsize asize type
11 1 16K 16K 16K 32K packed nvlist
8 bonus packed nvlist size
Assertion failed: 0 == dmu_read(os, object, 0, nvsize, packed), file ../zdb.c,
line 217, function dump_packed_nvlist
Abort (core dumped)
--
This message posted from opensolaris.org
Hi Brad, Brad Reese wrote:> Hello, > > I''ve run into a problem with zpool import that seems very similar to > the following thread as far as I can tell: > > http://opensolaris.org/jive/thread.jspa?threadID=70205&tstart=15 > > The suggested solution was to use a later version of open solaris > (b99 or later) but that did not work. I''ve tried the following > versions of solaris without success: > > Solaris 10 u4 (original system) > Solaris 10 u6 > Opensolaris 2008.11 > Opensolaris 2008.11 b99 > SXCE b113While booted off SXCE b113 bits (or OpenSolaris 2009.06), can you do the following: zdb -e -bcsvL tank zdb -e -u tank Output of various commands below shows that there are some corrupted metadata objects - at least config object and may be some space maps. Option -L for zdb command turns off leak tracking so it will not try to load space maps, and we''ll get better idea what is corrupted and how to proceed. regards, victor> Any help with this will be greatly appreciated...my last backup was > four months ago so a lot of my thesis work will be lost. I mistakenly > thought a mirrored zpool on new drives would be good enough for a > while. > > So here''s what happened: we had a power outage one day and as soon as > I tried to boot the server again it enters an endless reboot cycle. > So I thought the OS drive became corrupted (not mirrored) and > reinstalled the OS. Then when I try zpool import it hangs forever. I > even left it going for a couple days in case it was trying to correct > corrupted data. The same thing happens no matter what version of > solaris I use. The symptoms and diagnostic results (see below) seem > to be very similar to the post above but the solution doesn''t work. > > Please let me know if you need any other information. > > Thanks, > > Brad > > > bash-3.2# zpool import > pool: tank > id: 4410438565134310480 > state: ONLINE > status: The pool is formatted using an older on-disk version. > action: The pool can be imported using its name or numeric identifier, though > some features will not be available without an explicit ''zpool upgrade''. > config: > > tank ONLINE > mirror ONLINE > c2d0p0 ONLINE > c1d0p0 ONLINE > bash-3.2# zpool import tank > cannot import ''tank'': pool may be in use from other system > use ''-f'' to import anyway > bash-3.2# zpool import -f tank > (then it hangs here forever, can''t be killed) > (the following commands were performed while this was running) > > > bash-3.2# fmdump -eV > TIME CLASS > May 27 2009 22:22:55.308533986 ereport.fs.zfs.checksum > nvlist version: 0 > class = ereport.fs.zfs.checksum > ena = 0xd22e37db9000401 > detector = (embedded nvlist) > nvlist version: 0 > version = 0x0 > scheme = zfs > pool = 0x3d350681ea839c50 > vdev = 0x85cc302105002c5d > (end detector) > > pool = tank > pool_guid = 0x3d350681ea839c50 > pool_context = 0 > pool_failmode = wait > vdev_guid = 0x85cc302105002c5d > vdev_type = disk > vdev_path = /dev/dsk/c2d0p0 > vdev_devid = id1,cmdk at AST3750640AS=____________5QD3MYRH/q > parent_guid = 0x8fb729a008f16e65 > parent_type = mirror > zio_err = 50 > zio_offset = 0x2500407000 > zio_size = 0x1000 > zio_objset = 0x0 > zio_object = 0x23 > zio_level = 0 > zio_blkid = 0x0 > __ttl = 0x1 > __tod = 0x4a1e038f 0x1263dae2 > (and many others just like this) > > > > bash-3.2# echo "0t3735::pid2proc|::walk thread|::findtsack -v" | mdb -k > stack pointer for thread df6ed220: e1ca8c54 > e1ca8c94 swtch+0x188() > e1ca8ca4 cv_wait+0x53(e08442aa, e084426c, ffffffff, f9c305ac) > e1ca8ce4 txg_wait_synced+0x90(e0844100, 252b4c, 0, 0) > e1ca8d34 spa_config_update_common+0x88(d6c429c0, 0, 0, e1ca8d68) > e1ca8d84 spa_import_common+0x3cf() > e1ca8db4 spa_import+0x18(dbfcf000, e3c0d018, 0, f9c65810) > e1ca8de4 zfs_ioc_pool_import+0xcd(dbfcf000, 0, 0) > e1ca8e14 zfsdev_ioctl+0x124() > e1ca8e44 cdev_ioctl+0x31(2d80000, 5a02, 80418d0, 100003, dfde91f8, e1ca8f00) > e1ca8e74 spec_ioctl+0x6b(d7a593c0, 5a02, 80418d0, 100003, dfde91f8, e1ca8f00) > e1ca8ec4 fop_ioctl+0x49(d7a593c0, 5a02, 80418d0, 100003, dfde91f8, e1ca8f00) > e1ca8f84 ioctl+0x171() > e1ca8fac sys_sysenter+0x106() > > > > bash-3.2# echo "::threadlist -v" | mdb -k > d4ed8dc0 fec1f558 0 0 60 d5033604 > PC: _resume_from_idle+0xb1 THREAD: txg_sync_thread() > stack pointer for thread d4ed8dc0: d4ed8ba8 > swtch+0x188() > cv_wait+0x53() > zio_wait+0x55() > vdev_uberblock_sync_list+0x19e() > vdev_config_sync+0x11c() > spa_sync+0x5a5() > txg_sync_thread+0x308() > thread_start+8() > (just chose one seemingly relevant thread from the long list) > > > > bash-3.2# zdb -e -bb tank > Traversing all blocks to verify nothing leaked ... > Assertion failed: space_map_load(&msp->ms_map, &zdb_space_map_ops, 0x0, &msp->ms_smo, spa->spa_meta_objset) == 0, file ../zdb.c, line 1420, function > zdb_leak_init > Abort (core dumped) > > > > bash-3.2# zdb -e -dddd tank > Dataset mos [META], ID 0, cr_txg 4, 10.3M, 137 objects, rootbp [L0 DMU objset] 400L/400P DVA[0]=<0:2500005c00:400> DVA[1]=<0:4400005c00:400> DVA[2]=<0:6300003000:400> fletcher4 uncompressed LE contiguous birth=2435914 fill=137 cksum=5224494b4:4524146f316:1d44c6f4690ea:84ef3bd0c105a0 > > Object lvl iblk dblk lsize asize type > 0 2 16K 16K 80.0K 180K DMU dnode > > Object lvl iblk dblk lsize asize type > 1 1 16K 512 512 1K object directory > microzap: 512 bytes, 5 entries > > history = 13 > root_dataset = 2 > sync_bplist = 12 > config = 11 > deflate = 1 > > Object lvl iblk dblk lsize asize type > 2 1 16K 512 512 0 DSL directory > 256 bonus DSL directory > creation_time = Wed Nov 26 02:07:49 2008 > head_dataset_obj = 5 > parent_dir_obj = 0 > origin_obj = 0 > child_dir_zapobj = 4 > used_bytes = 402G > compressed_bytes = 401G > uncompressed_bytes = 401G > quota = 0 > reserved = 0 > props_zapobj = 3 > deleg_zapobj = 0 > flags = 0 > used_breakdown[HEAD] = 0 > used_breakdown[SNAP] = 0 > used_breakdown[CHILD] = 0 > used_breakdown[CHILD_RSRV] = 0 > used_breakdown[REFRSRV] = 0 > > Object lvl iblk dblk lsize asize type > 3 1 16K 512 512 1K DSL props > microzap: 512 bytes, 0 entries > > > Object lvl iblk dblk lsize asize type > 4 1 16K 512 512 1K DSL directory child map > microzap: 512 bytes, 1 entries > > $MOS = 8 > > Object lvl iblk dblk lsize asize type > 5 1 16K 512 512 0 DSL dataset > 320 bonus DSL dataset > dir_obj = 2 > prev_snap_obj = 99 > prev_snap_txg = 2088469 > next_snap_obj = 0 > snapnames_zapobj = 6 > num_children = 0 > creation_time = Wed Nov 26 02:07:49 2008 > creation_txg = 4 > deadlist_obj = 100 > used_bytes = 392G > compressed_bytes = 392G > uncompressed_bytes = 392G > unique = 11.3G > fsid_guid = 43614162550022126 > guid = 431952237769608890 > flags = 0 > next_clones_obj = 0 > props_obj = 0 > bp = [L0 DMU objset] 400L/200P DVA[0]=<0:2500756400:200> DVA[1]=<0:44005dce00:200> DVA[2]=<0:630011fe00:200> fletcher4 lzjb LE contiguous birth=2435911 fill=769038 cksum=b131ea18e:48f57187abe:f4e1282becf2:22e5d4ef0c414d > > Object lvl iblk dblk lsize asize type > 6 1 16K 512 512 1K DSL dataset snap map > microzap: 512 bytes, 2 entries > > tank-2008-12-30 = 97 > tank-2009-03-30 = 99 > > Object lvl iblk dblk lsize asize type > 7 1 16K 128K 128K 0 bplist > 32 bonus bplist header > > Object lvl iblk dblk lsize asize type > 8 1 16K 512 512 0 DSL directory > 256 bonus DSL directory > creation_time = Wed Nov 26 02:07:49 2008 > head_dataset_obj = 0 > parent_dir_obj = 2 > origin_obj = 0 > child_dir_zapobj = 10 > used_bytes = 10.3M > compressed_bytes = 5.01M > uncompressed_bytes = 5.01M > quota = 0 > reserved = 0 > props_zapobj = 9 > deleg_zapobj = 0 > flags = 0 > used_breakdown[HEAD] = 0 > used_breakdown[SNAP] = 0 > used_breakdown[CHILD] = 0 > used_breakdown[CHILD_RSRV] = 0 > used_breakdown[REFRSRV] = 0 > > Object lvl iblk dblk lsize asize type > 9 1 16K 512 512 1K DSL props > microzap: 512 bytes, 0 entries > > > Object lvl iblk dblk lsize asize type > 10 1 16K 512 512 1K DSL directory child map > microzap: 512 bytes, 0 entries > > > Object lvl iblk dblk lsize asize type > 11 1 16K 16K 16K 32K packed nvlist > 8 bonus packed nvlist size > Assertion failed: 0 == dmu_read(os, object, 0, nvsize, packed), file ../zdb.c, line 217, function dump_packed_nvlist > Abort (core dumped)
Hi Victor, zdb -e -bcsvL tank (let this go for a few hours...no output. I will let it go overnight) zdb -e -u tank Uberblock magic = 0000000000bab10c version = 4 txg = 2435914 guid_sum = 16655261404755214374 timestamp = 1240517036 UTC = Thu Apr 23 15:03:56 2009 Thanks for your help, Brad -- This message posted from opensolaris.org
Here''s the output of ''zdb -e -bsvL tank'' (without -c)
in case it helps. I''ll post with -c if it finishes.
Thanks,
Brad
Traversing all blocks ...
block traversal size 431585053184 != alloc 431585209344 (unreachable 156160)
bp count: 4078410
bp logical: 433202894336 avg: 106218
bp physical: 431089822720 avg: 105700 compression: 1.00
bp allocated: 431585053184 avg: 105821 compression: 1.00
SPA allocated: 431585209344 used: 57.75%
Blocks LSIZE PSIZE ASIZE avg comp %Total Type
- - - - - - - deferred free
1 512 512 1K 1K 1.00 0.00 object directory
3 1.50K 1.50K 3.00K 1K 1.00 0.00 object array
1 16K 16K 32K 32K 1.00 0.00 packed nvlist
- - - - - - - packed nvlist size
114 13.9M 1.05M 2.11M 18.9K 13.25 0.00 bplist
- - - - - - - bplist header
- - - - - - - SPA space map header
1.36K 6.57M 3.88M 7.94M 5.82K 1.69 0.00 SPA space map
- - - - - - - ZIL intent log
49.4K 791M 220M 442M 8.95K 3.60 0.11 DMU dnode
4 4K 2.50K 7.50K 1.88K 1.60 0.00 DMU objset
- - - - - - - DSL directory
2 1K 1K 2K 1K 1.00 0.00 DSL directory child map
1 512 512 1K 1K 1.00 0.00 DSL dataset snap map
2 1K 1K 2K 1K 1.00 0.00 DSL props
- - - - - - - DSL dataset
- - - - - - - ZFS znode
- - - - - - - ZFS V0 ACL
3.77M 403G 401G 401G 106K 1.00 99.87 ZFS plain file
72.4K 114M 49.0M 98.8M 1.37K 2.32 0.02 ZFS directory
1 512 512 1K 1K 1.00 0.00 ZFS master node
3 19.5K 1.50K 3.00K 1K 13.00 0.00 ZFS delete queue
- - - - - - - zvol object
- - - - - - - zvol prop
- - - - - - - other uint8[]
- - - - - - - other uint64[]
- - - - - - - other ZAP
- - - - - - - persistent error log
1 128K 5.00K 10.0K 10.0K 25.60 0.00 SPA history
- - - - - - - SPA history offsets
- - - - - - - Pool properties
- - - - - - - DSL permissions
- - - - - - - ZFS ACL
- - - - - - - ZFS SYSACL
- - - - - - - FUID table
- - - - - - - FUID table size
- - - - - - - DSL dataset next clones
- - - - - - - scrub work queue
3.89M 403G 401G 402G 103K 1.00 100.00 Total
capacity operations bandwidth ---- errors ----
description used avail read write read write read write cksum
tank 402G 294G 463 0 1.27M 0 0 0 1
mirror 402G 294G 463 0 1.27M 0 0 0 4
/dev/dsk/c2d0p0 69 0 4.05M 0 0 0 4
/dev/dsk/c1d0p0 67 0 3.96M 0 0 0 4
--
This message posted from opensolaris.org
Hi Victor,
Here''s the output of ''zdb -e -bcsvL tank'' (similar to
above but with -c).
Thanks,
Brad
Traversing all blocks to verify checksums ...
zdb_blkptr_cb: Got error 50 reading <0, 11, 0, 0> [L0 packed nvlist]
4000L/4000P DVA[0]=<0:2500014000:4000> DVA[1]=<0:4400014000:4000>
fletcher4 uncompressed LE contiguous birth=2435914 fill=1
cksum=2cdaaa4db0:2b105dbdf910e:14b020cadaf44446:1720f5444d0b5366 -- skipping
zdb_blkptr_cb: Got error 50 reading <0, 12, 0, 0> [L0 bplist] 4000L/4000P
DVA[0]=<0:2500008000:4000> DVA[1]=<0:4400008000:4000> fletcher4
uncompressed LE contiguous birth=2435914 fill=1
cksum=16b3e12568:143f7c0b11757:91395667fbce35f:9f686628032bddf2 -- skipping
zdb_blkptr_cb: Got error 50 reading <0, 24, 0, 0> [L0 SPA space map]
1000L/1000P DVA[0]=<0:2500024000:1000> DVA[1]=<0:440002c000:1000>
fletcher4 uncompressed LE contiguous birth=2435914 fill=1
cksum=76284dc2e9:1438efeb0fb9b:1d2f57253c8d409:d4a881948152382b -- skipping
zdb_blkptr_cb: Got error 50 reading <0, 30, 0, 4> [L0 SPA space map]
1000L/1000P DVA[0]=<0:2500006000:1000> DVA[1]=<0:4400006000:1000>
fletcher4 uncompressed LE contiguous birth=2435914 fill=1
cksum=5d6df356b5:eed102f1beb0:15280d72b8c8588:5925604865323b6b -- skipping
zdb_blkptr_cb: Got error 50 reading <0, 35, 0, 0> [L0 SPA space map]
1000L/1000P DVA[0]=<0:2500007000:1000> DVA[1]=<0:4400007000:1000>
fletcher4 uncompressed LE contiguous birth=2435914 fill=1
cksum=335f09c578:8b7b18876592:c58c7a556ca72b:c22e91ead638c69c -- skipping
Error counts:
errno count
50 5
block traversal size 431585053184 != alloc 431585209344 (unreachable 156160)
bp count: 4078410
bp logical: 433202894336 avg: 106218
bp physical: 431089822720 avg: 105700 compression: 1.00
bp allocated: 431585053184 avg: 105821 compression: 1.00
SPA allocated: 431585209344 used: 57.75%
Blocks LSIZE PSIZE ASIZE avg comp %Total Type
- - - - - - - deferred free
1 512 512 1K 1K 1.00 0.00 object directory
3 1.50K 1.50K 3.00K 1K 1.00 0.00 object array
1 16K 16K 32K 32K 1.00 0.00 packed nvlist
- - - - - - - packed nvlist size
114 13.9M 1.05M 2.11M 18.9K 13.25 0.00 bplist
- - - - - - - bplist header
- - - - - - - SPA space map header
1.36K 6.57M 3.88M 7.94M 5.82K 1.69 0.00 SPA space map
- - - - - - - ZIL intent log
49.4K 791M 220M 442M 8.95K 3.60 0.11 DMU dnode
4 4K 2.50K 7.50K 1.88K 1.60 0.00 DMU objset
- - - - - - - DSL directory
2 1K 1K 2K 1K 1.00 0.00 DSL directory child map
1 512 512 1K 1K 1.00 0.00 DSL dataset snap map
2 1K 1K 2K 1K 1.00 0.00 DSL props
- - - - - - - DSL dataset
- - - - - - - ZFS znode
- - - - - - - ZFS V0 ACL
3.77M 403G 401G 401G 106K 1.00 99.87 ZFS plain file
72.4K 114M 49.0M 98.8M 1.37K 2.32 0.02 ZFS directory
1 512 512 1K 1K 1.00 0.00 ZFS master node
3 19.5K 1.50K 3.00K 1K 13.00 0.00 ZFS delete queue
- - - - - - - zvol object
- - - - - - - zvol prop
- - - - - - - other uint8[]
- - - - - - - other uint64[]
- - - - - - - other ZAP
- - - - - - - persistent error log
1 128K 5.00K 10.0K 10.0K 25.60 0.00 SPA history
- - - - - - - SPA history offsets
- - - - - - - Pool properties
- - - - - - - DSL permissions
- - - - - - - ZFS ACL
- - - - - - - ZFS SYSACL
- - - - - - - FUID table
- - - - - - - FUID table size
- - - - - - - DSL dataset next clones
- - - - - - - scrub work queue
3.89M 403G 401G 402G 103K 1.00 100.00 Total
capacity operations bandwidth ---- errors ----
description used avail read write read write read write cksum
tank 402G 294G 283 0 26.2M 0 0 0 6
mirror 402G 294G 283 0 26.2M 0 0 0 26
/dev/dsk/c2d0p0 251 0 26.5M 0 0 0 36
/dev/dsk/c1d0p0 251 0 26.5M 0 0 0 26
--
This message posted from opensolaris.org
Brad Reese wrote:> Hi Victor, > > Here''s the output of ''zdb -e -bcsvL tank'' (similar to above but with -c). > > Thanks, > > Brad > > Traversing all blocks to verify checksums ... > zdb_blkptr_cb: Got error 50 reading <0, 11, 0, 0> [L0 packed nvlist] 4000L/4000P DVA[0]=<0:2500014000:4000> DVA[1]=<0:4400014000:4000> fletcher4 uncompressed LE contiguous birth=2435914 fill=1 cksum=2cdaaa4db0:2b105dbdf910e:14b020cadaf44446:1720f5444d0b5366 -- skipping > zdb_blkptr_cb: Got error 50 reading <0, 12, 0, 0> [L0 bplist] 4000L/4000P DVA[0]=<0:2500008000:4000> DVA[1]=<0:4400008000:4000> fletcher4 uncompressed LE contiguous birth=2435914 fill=1 cksum=16b3e12568:143f7c0b11757:91395667fbce35f:9f686628032bddf2 -- skipping > zdb_blkptr_cb: Got error 50 reading <0, 24, 0, 0> [L0 SPA space map] 1000L/1000P DVA[0]=<0:2500024000:1000> DVA[1]=<0:440002c000:1000> fletcher4 uncompressed LE contiguous birth=2435914 fill=1 cksum=76284dc2e9:1438efeb0fb9b:1d2f57253c8d409:d4a881948152382b -- skipping > zdb_blkptr_cb: Got error 50 reading <0, 30, 0, 4> [L0 SPA space map] 1000L/1000P DVA[0]=<0:2500006000:1000> DVA[1]=<0:4400006000:1000> fletcher4 uncompressed LE contiguous birth=2435914 fill=1 cksum=5d6df356b5:eed102f1beb0:15280d72b8c8588:5925604865323b6b -- skipping > zdb_blkptr_cb: Got error 50 reading <0, 35, 0, 0> [L0 SPA space map] 1000L/1000P DVA[0]=<0:2500007000:1000> DVA[1]=<0:4400007000:1000> fletcher4 uncompressed LE contiguous birth=2435914 fill=1 cksum=335f09c578:8b7b18876592:c58c7a556ca72b:c22e91ead638c69c -- skipping >This tells us that there are 5 corrupted objects in the pool, and all of them were born during txg 2435914. Since it is the last synced txg for this pool, it is likely that previous synced txg may be intact. We can try to verify this. To do it could you please run the following while booted off SXCE b113: zdb -e -u -t 2435913 tank zdb -e -bcsv -t 2435913 tank Regards, victor> Error counts: > > errno count > 50 5 > block traversal size 431585053184 != alloc 431585209344 (unreachable 156160) > > bp count: 4078410 > bp logical: 433202894336 avg: 106218 > bp physical: 431089822720 avg: 105700 compression: 1.00 > bp allocated: 431585053184 avg: 105821 compression: 1.00 > SPA allocated: 431585209344 used: 57.75% > > Blocks LSIZE PSIZE ASIZE avg comp %Total Type > - - - - - - - deferred free > 1 512 512 1K 1K 1.00 0.00 object directory > 3 1.50K 1.50K 3.00K 1K 1.00 0.00 object array > 1 16K 16K 32K 32K 1.00 0.00 packed nvlist > - - - - - - - packed nvlist size > 114 13.9M 1.05M 2.11M 18.9K 13.25 0.00 bplist > - - - - - - - bplist header > - - - - - - - SPA space map header > 1.36K 6.57M 3.88M 7.94M 5.82K 1.69 0.00 SPA space map > - - - - - - - ZIL intent log > 49.4K 791M 220M 442M 8.95K 3.60 0.11 DMU dnode > 4 4K 2.50K 7.50K 1.88K 1.60 0.00 DMU objset > - - - - - - - DSL directory > 2 1K 1K 2K 1K 1.00 0.00 DSL directory child map > 1 512 512 1K 1K 1.00 0.00 DSL dataset snap map > 2 1K 1K 2K 1K 1.00 0.00 DSL props > - - - - - - - DSL dataset > - - - - - - - ZFS znode > - - - - - - - ZFS V0 ACL > 3.77M 403G 401G 401G 106K 1.00 99.87 ZFS plain file > 72.4K 114M 49.0M 98.8M 1.37K 2.32 0.02 ZFS directory > 1 512 512 1K 1K 1.00 0.00 ZFS master node > 3 19.5K 1.50K 3.00K 1K 13.00 0.00 ZFS delete queue > - - - - - - - zvol object > - - - - - - - zvol prop > - - - - - - - other uint8[] > - - - - - - - other uint64[] > - - - - - - - other ZAP > - - - - - - - persistent error log > 1 128K 5.00K 10.0K 10.0K 25.60 0.00 SPA history > - - - - - - - SPA history offsets > - - - - - - - Pool properties > - - - - - - - DSL permissions > - - - - - - - ZFS ACL > - - - - - - - ZFS SYSACL > - - - - - - - FUID table > - - - - - - - FUID table size > - - - - - - - DSL dataset next clones > - - - - - - - scrub work queue > 3.89M 403G 401G 402G 103K 1.00 100.00 Total > > capacity operations bandwidth ---- errors ---- > description used avail read write read write read write cksum > tank 402G 294G 283 0 26.2M 0 0 0 6 > mirror 402G 294G 283 0 26.2M 0 0 0 26 > /dev/dsk/c2d0p0 251 0 26.5M 0 0 0 36 > /dev/dsk/c1d0p0 251 0 26.5M 0 0 0 26
Hi Victor,
Sorry it took a while for me to reply, I was traveling and had limited network
access.
''zdb -e -bcsv -t 2435913 tank'' has been running for a few days
with no output...want to try something else?
Here''s the output of ''zdb -e -u -t 2435913 tank'':
Uberblock
magic = 0000000000bab10c
version = 4
txg = 2435911
guid_sum = 16655261404755214374
timestamp = 1240287900 UTC = Mon Apr 20 23:25:00 2009
Thanks,
Brad
--
This message posted from opensolaris.org
Hi Victor, ''zdb -e -bcsv -t 2435913 tank'' ran for about a week with no output. We had yet another brown out and then the comp shut down (have a UPS on the way). A few days before that I started the following commands, which also had no output: zdb -e -bcsv -t 2435911 tank zdb -e -bcsv -t 2435897 tank I''ve given up on these because I don''t think they''ll finish...should I try again? Right now I am trying the following commands which so far have no output: zdb -e -bcsvL -t 2435913 tank zdb -e -bsvL -t 2435913 tank zdb -e -bb -t 2435913 tank ''zdb -e -dddd -t 2435913 tank'' has output and is very long...is there anything I should be looking for? Without -t 243... this command failed on dmu_read, now it just keeps going forever. Your help is much appreciated. Thanks, Brad -- This message posted from opensolaris.org
On 16.06.09 07:57, Brad Reese wrote:> Hi Victor, > > ''zdb -e -bcsv -t 2435913 tank'' ran for about a week with no output. We had yet another brown out and then the comp shut down (have a UPS on the way). A few days before that I started the following commands, which also had no output: > > zdb -e -bcsv -t 2435911 tank > zdb -e -bcsv -t 2435897 tank > > I''ve given up on these because I don''t think they''ll finish...should I try again? > > Right now I am trying the following commands which so far have no output: > > zdb -e -bcsvL -t 2435913 tank > zdb -e -bsvL -t 2435913 tank > zdb -e -bb -t 2435913 tankyou can use something like pstack `pgrep zdb`/1 to check if it is making progress. Is is possible to access your system (e.g. via ssh or Shared Shell)? If yes, please contact me offline to set it up. regards, victor> > ''zdb -e -dddd -t 2435913 tank'' has output and is very long...is there anything I should be looking for? Without -t 243... this command failed on dmu_read, now it just keeps going forever. > > Your help is much appreciated. > > Thanks, > > Brad
Hi Victor, Yes, you may access the system via ssh. Please contact me at bar001 at uark dot edu and I will reply with details of how to connect. Thanks, Brad -- This message posted from opensolaris.org
On Wed, Jun 17, 2009 at 11:49 AM, Brad Reese<no-reply at opensolaris.org> wrote:> Yes, you may access the system via ssh. Please contact me at bar001 at uark dot > edu and I will reply with details of how to connect....and then please tell us what was wrong! :-) -- Kind regards, BM Things, that are stupid at the beginning, rarely ends up wisely.