after several errors on QLogic HBA pool cache was damaged and zfs cannot import
pool, there is no any disk or cpu activity during import...
#uname -a
SunOS orion 5.11 snv_111b i86pc i386 i86pc
# zpool import
pool: data1
id: 6305414271646982336
state: ONLINE
status: The pool was last accessed by another system.
action: The pool can be imported using its name or numeric identifier and
the ''-f'' flag.
see: http://www.sun.com/msg/ZFS-8000-EY
config:
data1 ONLINE
c14t0d0 ONLINE
and after
#zpool import -f data1
terminal still waiting forever.
also after
# zdb -e data1
Uberblock
magic = 0000000000bab10c
version = 6
txg = 2682808
guid_sum = 14250651627001887594
timestamp = 1247866318 UTC = Sat Jul 18 01:31:58 2009
Dataset mos [META], ID 0, cr_txg 4, 27.1M, 3050 objects
Dataset data1 [ZPL], ID 5, cr_txg 4, 5.74T, 52987 objects
terminal still waiting forever too,
and thereis no helpful messages in system log,
and there are no ideas how to recover this pool, any help will be greatly
appreciated...
--
This message posted from opensolaris.org
On 29.07.09 13:04, Pavel Kovalenko wrote:> after several errors on QLogic HBA pool cache was damaged and zfs cannot import pool, there is no any disk or cpu activity during import... > #uname -a > SunOS orion 5.11 snv_111b i86pc i386 i86pc > # zpool import > pool: data1 > id: 6305414271646982336 > state: ONLINE > status: The pool was last accessed by another system. > action: The pool can be imported using its name or numeric identifier and > the ''-f'' flag. > see: http://www.sun.com/msg/ZFS-8000-EY > config: > > data1 ONLINE > c14t0d0 ONLINE > and after > #zpool import -f data1 > terminal still waiting forever.Try to use echo "0t<pid of zpool>::pid2proc|::walk thread|::findstack -v" | mdb -k to find out what it is doing. see fmdump -eV output for fresh error reports from ZFS> also after > # zdb -e data1 > Uberblock > > magic = 0000000000bab10c > version = 6 > txg = 2682808 > guid_sum = 14250651627001887594 > timestamp = 1247866318 UTC = Sat Jul 18 01:31:58 2009 > > Dataset mos [META], ID 0, cr_txg 4, 27.1M, 3050 objects > Dataset data1 [ZPL], ID 5, cr_txg 4, 5.74T, 52987 objects > > terminal still waiting forever too, > and thereis no helpful messages in system log,Does zdb -e -t 2682807 data1 make any difference? victor
fortunately, after several hours terminal went back -->
# zdb -e data1
Uberblock
magic = 0000000000bab10c
version = 6
txg = 2682808
guid_sum = 14250651627001887594
timestamp = 1247866318 UTC = Sat Jul 18 01:31:58 2009
Dataset mos [META], ID 0, cr_txg 4, 27.1M, 3050 objects
Dataset data1 [ZPL], ID 5, cr_txg 4, 5.74T, 52987 objects
capacity operations bandwidth ---- errors ----
description used avail read write read write read write cksum
data1 5.74T 6.99T 772 0 96.0M 0 0 0 91
/dev/dsk/c14t0d0 5.74T 6.99T 772 0 96.0M 0 0 0 223
#
i''ve tried to run zdb -e -t 2682807 data1
and
#echo "0t::pid2proc|::walk thread|::findstack -v" | mdb -k
shows
stack pointer for thread fffffffffbc2cca0: fffffffffbc4d980
[ fffffffffbc4d980 _resume_from_idle+0xf1() ]
fffffffffbc4d9b0 swtch+0x147()
fffffffffbc4da40 sched+0x3fd()
fffffffffbc4da70 main+0x437()
fffffffffbc4da80 _locore_start+0x92()
and
#fmdump -eV
shows checksum errors, such as
Jul 28 2009 11:17:35.386268381 ereport.fs.zfs.checksum
nvlist version: 0
class = ereport.fs.zfs.checksum
ena = 0x1baa23c52ce01c01
detector = (embedded nvlist)
nvlist version: 0
version = 0x0
scheme = zfs
pool = 0x578154df5f3260c0
vdev = 0x6e4327476e17daaa
(end detector)
pool = data1
pool_guid = 0x578154df5f3260c0
pool_context = 2
pool_failmode = wait
vdev_guid = 0x6e4327476e17daaa
vdev_type = disk
vdev_path = /dev/dsk/c14t0d0p0
vdev_devid = id1,sd at n2661000612646364/q
parent_guid = 0x578154df5f3260c0
parent_type = root
zio_err = 50
zio_offset = 0x2313d58000
zio_size = 0x4000
zio_objset = 0x0
zio_object = 0xc
zio_level = 0
zio_blkid = 0x0
__ttl = 0x1
__tod = 0x4a6ea60f 0x1705fcdd
Jul 28 2009 11:17:35.386268179 ereport.fs.zfs.checksum
nvlist version: 0
class = ereport.fs.zfs.checksum
ena = 0x1baa23c52ce01c01
detector = (embedded nvlist)
nvlist version: 0
version = 0x0
scheme = zfs
pool = 0x578154df5f3260c0
vdev = 0x6e4327476e17daaa
(end detector)
pool = data1
pool_guid = 0x578154df5f3260c0
pool_context = 2
pool_failmode = wait
vdev_guid = 0x6e4327476e17daaa
vdev_type = disk
vdev_path = /dev/dsk/c14t0d0p0
vdev_devid = id1,sd at n2661000612646364/q
parent_guid = 0x578154df5f3260c0
parent_type = root
zio_err = 50
zio_offset = 0x5c516eac000
zio_size = 0x4000
zio_objset = 0x0
zio_object = 0xc
zio_level = 0
zio_blkid = 0x0
__ttl = 0x1
__tod = 0x4a6ea60f 0x1705fc13
can i hope, that some data will recover after several hours with zpool import -f
data1 command?
--
This message posted from opensolaris.org
On 29.07.09 14:42, Pavel Kovalenko wrote:> fortunately, after several hours terminal went back --> > # zdb -e data1 > Uberblock > > magic = 0000000000bab10c > version = 6 > txg = 2682808 > guid_sum = 14250651627001887594 > timestamp = 1247866318 UTC = Sat Jul 18 01:31:58 2009 > > Dataset mos [META], ID 0, cr_txg 4, 27.1M, 3050 objects > Dataset data1 [ZPL], ID 5, cr_txg 4, 5.74T, 52987 objects > > capacity operations bandwidth ---- errors ---- > description used avail read write read write read write cksum > data1 5.74T 6.99T 772 0 96.0M 0 0 0 91 > /dev/dsk/c14t0d0 5.74T 6.99T 772 0 96.0M 0 0 0 223 > #So we know that there are some checksum errors there but at least zdb was able to open pool in read-only mode.> i''ve tried to run zdb -e -t 2682807 data1 > and > #echo "0t::pid2proc|::walk thread|::findstack -v" | mdb -kThis is wrong - you need to put PID of the ''zpool import data1'' process right after ''0t''.> and > #fmdump -eV > shows checksum errors, such as > Jul 28 2009 11:17:35.386268381 ereport.fs.zfs.checksum > nvlist version: 0 > class = ereport.fs.zfs.checksum > ena = 0x1baa23c52ce01c01 > detector = (embedded nvlist) > nvlist version: 0 > version = 0x0 > scheme = zfs > pool = 0x578154df5f3260c0 > vdev = 0x6e4327476e17daaa > (end detector) > > pool = data1 > pool_guid = 0x578154df5f3260c0 > pool_context = 2 > pool_failmode = wait > vdev_guid = 0x6e4327476e17daaa > vdev_type = disk > vdev_path = /dev/dsk/c14t0d0p0 > vdev_devid = id1,sd at n2661000612646364/q > parent_guid = 0x578154df5f3260c0 > parent_type = root > zio_err = 50 > zio_offset = 0x2313d58000 > zio_size = 0x4000 > zio_objset = 0x0 > zio_object = 0xc > zio_level = 0 > zio_blkid = 0x0 > __ttl = 0x1 > __tod = 0x4a6ea60f 0x1705fcddThis tells us that object 0xc in metabjset (objset 0x0) is corrupted. So to get more details you can do the following: zdb -e -dddd data1 zdb -e -bbcs data1 victor
I recently noticed that importing larger pools that are occupied by large amounts of data can do zpool import for several hours while zpool iostat only showing some random reads now and then and iostat -xen showing quite busy disk usage, It''s almost it goes thru every bit in pool before it goes thru. Somebody said that zpool import got faster on snv118, but I don''t have real information on that yet. Yours Markus Kovero -----Original Message----- From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss-bounces at opensolaris.org] On Behalf Of Victor Latushkin Sent: 29. hein?kuuta 2009 14:05 To: Pavel Kovalenko Cc: zfs-discuss at opensolaris.org Subject: Re: [zfs-discuss] zpool import hungs up forever... On 29.07.09 14:42, Pavel Kovalenko wrote:> fortunately, after several hours terminal went back --> > # zdb -e data1 > Uberblock > > magic = 0000000000bab10c > version = 6 > txg = 2682808 > guid_sum = 14250651627001887594 > timestamp = 1247866318 UTC = Sat Jul 18 01:31:58 2009 > > Dataset mos [META], ID 0, cr_txg 4, 27.1M, 3050 objects > Dataset data1 [ZPL], ID 5, cr_txg 4, 5.74T, 52987 objects > > capacity operations bandwidth ---- errors ---- > description used avail read write read write read write cksum > data1 5.74T 6.99T 772 0 96.0M 0 0 0 91 > /dev/dsk/c14t0d0 5.74T 6.99T 772 0 96.0M 0 0 0 223 > #So we know that there are some checksum errors there but at least zdb was able to open pool in read-only mode.> i''ve tried to run zdb -e -t 2682807 data1 > and > #echo "0t::pid2proc|::walk thread|::findstack -v" | mdb -kThis is wrong - you need to put PID of the ''zpool import data1'' process right after ''0t''.> and > #fmdump -eV > shows checksum errors, such as > Jul 28 2009 11:17:35.386268381 ereport.fs.zfs.checksum > nvlist version: 0 > class = ereport.fs.zfs.checksum > ena = 0x1baa23c52ce01c01 > detector = (embedded nvlist) > nvlist version: 0 > version = 0x0 > scheme = zfs > pool = 0x578154df5f3260c0 > vdev = 0x6e4327476e17daaa > (end detector) > > pool = data1 > pool_guid = 0x578154df5f3260c0 > pool_context = 2 > pool_failmode = wait > vdev_guid = 0x6e4327476e17daaa > vdev_type = disk > vdev_path = /dev/dsk/c14t0d0p0 > vdev_devid = id1,sd at n2661000612646364/q > parent_guid = 0x578154df5f3260c0 > parent_type = root > zio_err = 50 > zio_offset = 0x2313d58000 > zio_size = 0x4000 > zio_objset = 0x0 > zio_object = 0xc > zio_level = 0 > zio_blkid = 0x0 > __ttl = 0x1 > __tod = 0x4a6ea60f 0x1705fcddThis tells us that object 0xc in metabjset (objset 0x0) is corrupted. So to get more details you can do the following: zdb -e -dddd data1 zdb -e -bbcs data1 victor _______________________________________________ zfs-discuss mailing list zfs-discuss at opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Victor,
after
# ps -ef | grep zdb | grep -v grep
root 3281 1683 1 14:22:09 pts/2 8:57 zdb -e -t 2682807 data1
i''ve inserted pid after 0t:
# echo "0t3281::pid2proc|::walk thread|::findstack -v" | mdb -k
>mdb-k0t3281
and got a couple of records:
stack pointer for thread ffffff02017ad700: ffffff0008ce8a50
[ ffffff0008ce8a50 _resume_from_idle+0xf1() ]
ffffff0008ce8a80 swtch+0x147()
ffffff0008ce8ac0 sema_p+0x1d9(ffffff01df7a9b70)
ffffff0008ce8af0 biowait+0x76(ffffff01df7a9ab0)
ffffff0008ce8bf0 default_physio+0x3d3(fffffffff7a148f0, 0, d700000207, 40,
fffffffff7a14130, ffffff0008ce8e80)
ffffff0008ce8c30 physio+0x25(fffffffff7a148f0, 0, d700000207, 40,
fffffffff7a14130, ffffff0008ce8e80)
ffffff0008ce8c80 sdread+0x150(d700000207, ffffff0008ce8e80, ffffff01e94927c8)
ffffff0008ce8cb0 cdev_read+0x3d(d700000207, ffffff0008ce8e80, ffffff01e94927c8
)
ffffff0008ce8d30 spec_read+0x270(ffffff020de0aa00, ffffff0008ce8e80, 0,
ffffff01e94927c8, 0)
ffffff0008ce8da0 fop_read+0x6b(ffffff020de0aa00, ffffff0008ce8e80, 0,
ffffff01e94927c8, 0)
ffffff0008ce8f00 pread+0x22c(8, 3e98000, 20000, 291cf9a0000)
ffffff0008ce8f10 sys_syscall+0x17b()
stack pointer for thread ffffff01e92dd8a0: ffffff000904cd00
[ ffffff000904cd00 _resume_from_idle+0xf1() ]
ffffff000904cd30 swtch+0x147()
ffffff000904cd90 cv_wait_sig_swap_core+0x170(ffffff01e92dda76,
ffffff01e92dda78, 0)
ffffff000904cdb0 cv_wait_sig_swap+0x18(ffffff01e92dda76, ffffff01e92dda78)
ffffff000904ce20 cv_waituntil_sig+0x135(ffffff01e92dda76, ffffff01e92dda78, 0
, 0)
ffffff000904cec0 lwp_park+0x157(0, 0)
ffffff000904cf00 syslwp_park+0x31(0, 0, 0)
ffffff000904cf10 sys_syscall+0x17b()
stack pointer for thread ffffff01e92c4e60: ffffff0008ceed00
[ ffffff0008ceed00 _resume_from_idle+0xf1() ]
ffffff0008ceed30 swtch+0x147()
ffffff0008ceed90 cv_wait_sig_swap_core+0x170(ffffff01e92c5036,
ffffff01e92c5038, 0)
ffffff0008ceedb0 cv_wait_sig_swap+0x18(ffffff01e92c5036, ffffff01e92c5038)
ffffff0008ceee20 cv_waituntil_sig+0x135(ffffff01e92c5036, ffffff01e92c5038, 0
, 0)
ffffff0008ceeec0 lwp_park+0x157(0, 0)
ffffff0008ceef00 syslwp_park+0x31(0, 0, 0)
ffffff0008ceef10 sys_syscall+0x17b()
and
#zdb -e -dddd data1 >zdb-e-dddd_list.txt
lists a lot of data objects, that were on the pool:
# ls -la zdb-e-dddd_list.txt
-rw-r--r-- 1 root root 28863781 Jul 28 21:41 zdb-e-dddd_list.txt
I can provide more detailed information by email pkovalenko at mtv.ru,
as i don''t know any zfs specialist who can help me recover pool.
--
This message posted from opensolaris.org
On 29.07.09 15:18, Markus Kovero wrote:> I recently noticed that importing larger pools that are occupied by large > amounts of data can do zpool import for several hours while zpool iostat only > showing some random reads now and then and iostat -xen showing quite busy disk > usage, It''s almost it goes thru every bit in pool before it goes thru. > > Somebody said that zpool import got faster on snv118, but I don''t have real > information on that yet.This had nothing to do with speed of ''zpool import''. There was corrupted pool-wide metadata block that prevented pool from importing successfully. Fortunately enough, we found better previous state a few txgs back with txg 2683802 (last synced was 2682802: #zdb -e -ubbcsL -t 2682802 data1 ... 4.25K 19.9M 8.62M 25.8M 6.08K 2.31 0.00 SPA space map 1 128K 128K 128K 128K 1.00 0.00 ZIL intent log 1.77K 28.4M 8.48M 17.3M 9.8K 3.35 0.00 DMU dnode 2 2K 1K 2.50K 1.25K 2.00 0.00 DMU objset - - - - - - - DSL directory 2 1K 1K 3.00K 1.50K 1.00 0.00 DSL directory child map 1 512 512 1.50K 1.50K 1.00 0.00 DSL dataset snap map 2 1K 1K 3.00K 1.50K 1.00 0.00 DSL props - - - - - - - DSL dataset - - - - - - - ZFS znode - - - - - - - ZFS V0 ACL 46.3M 5.74T 5.74T 5.74T 127K 1.00 100.00 ZFS plain file 1.87K 9.04M 2.75M 5.50M 2.94K 3.29 0.00 ZFS directory 1 512 512 1K 1K 1.00 0.00 ZFS master node 1 512 512 1K 1K 1.00 0.00 ZFS delete queue - - - - - - - zvol object - - - - - - - zvol prop - - - - - - - other uint8[] - - - - - - - other uint64[] - - - - - - - other ZAP - - - - - - - persistent error log 1 128K 4.50K 13.5K 13.5K 28.44 0.00 SPA history - - - - - - - SPA history offsets - - - - - - - Pool properties - - - - - - - DSL permissions - - - - - - - ZFS ACL - - - - - - - ZFS SYSACL - - - - - - - FUID table - - - - - - - FUID table size - - - - - - - DSL dataset next clones - - - - - - - scrub work queue 46.3M 5.74T 5.74T 5.74T 127K 1.00 100.00 Total capacity operations bandwidth ---- errors ---- description used avail read write read write read write cksum data1 5.74T 6.99T 523 0 65.1M 0 0 0 1 /dev/dsk/c14t0d0 5.74T 6.99T 523 0 65.1M 0 0 0 17 So we reactivated it and were able to import pool just fine. Subsequent scrub did find couple of errors in metadata. There were no user data error at all: # zpool status -v data1 pool: data1 state: ONLINE status: One or more devices has experienced an unrecoverable error. An attempt was made to correct the error. Applications are unaffected. action: Determine if the device needs to be replaced, and clear the errors using ''zpool clear'' or replace the device with ''zpool replace''. see: http://www.sun.com/msg/ZFS-8000-9P scrub: scrub completed after 12h43m with 0 errors on Thu Aug 6 06:00:11 2009 config: NAME STATE READ WRITE CKSUM data1 ONLINE 0 0 0 c14t0d0 ONLINE 0 0 2 12K repaired errors: No known data errors Upcoming zpool recovery support is going to help perform this kind of recovery in user-friendlier and more automated way. Btw, pool was originally created on FreeBSD, but we performed recovery on Solaris. Pavel said that he was going to stay on OpenSolaris as he learned a lot about it along the way ;-) Cheers, Victor> > Yours > Markus Kovero > > -----Original Message----- > From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss-bounces at opensolaris.org] On Behalf Of Victor Latushkin > Sent: 29. hein?kuuta 2009 14:05 > To: Pavel Kovalenko > Cc: zfs-discuss at opensolaris.org > Subject: Re: [zfs-discuss] zpool import hungs up forever... > > On 29.07.09 14:42, Pavel Kovalenko wrote: >> fortunately, after several hours terminal went back --> >> # zdb -e data1 >> Uberblock >> >> magic = 0000000000bab10c >> version = 6 >> txg = 2682808 >> guid_sum = 14250651627001887594 >> timestamp = 1247866318 UTC = Sat Jul 18 01:31:58 2009 >> >> Dataset mos [META], ID 0, cr_txg 4, 27.1M, 3050 objects >> Dataset data1 [ZPL], ID 5, cr_txg 4, 5.74T, 52987 objects >> >> capacity operations bandwidth ---- errors ---- >> description used avail read write read write read write cksum >> data1 5.74T 6.99T 772 0 96.0M 0 0 0 91 >> /dev/dsk/c14t0d0 5.74T 6.99T 772 0 96.0M 0 0 0 223 >> # > > So we know that there are some checksum errors there but at least zdb > was able to open pool in read-only mode. > >> i''ve tried to run zdb -e -t 2682807 data1 >> and >> #echo "0t::pid2proc|::walk thread|::findstack -v" | mdb -k > > This is wrong - you need to put PID of the ''zpool import data1'' process > right after ''0t''. > >> and >> #fmdump -eV >> shows checksum errors, such as >> Jul 28 2009 11:17:35.386268381 ereport.fs.zfs.checksum >> nvlist version: 0 >> class = ereport.fs.zfs.checksum >> ena = 0x1baa23c52ce01c01 >> detector = (embedded nvlist) >> nvlist version: 0 >> version = 0x0 >> scheme = zfs >> pool = 0x578154df5f3260c0 >> vdev = 0x6e4327476e17daaa >> (end detector) >> >> pool = data1 >> pool_guid = 0x578154df5f3260c0 >> pool_context = 2 >> pool_failmode = wait >> vdev_guid = 0x6e4327476e17daaa >> vdev_type = disk >> vdev_path = /dev/dsk/c14t0d0p0 >> vdev_devid = id1,sd at n2661000612646364/q >> parent_guid = 0x578154df5f3260c0 >> parent_type = root >> zio_err = 50 >> zio_offset = 0x2313d58000 >> zio_size = 0x4000 >> zio_objset = 0x0 >> zio_object = 0xc >> zio_level = 0 >> zio_blkid = 0x0 >> __ttl = 0x1 >> __tod = 0x4a6ea60f 0x1705fcdd > > This tells us that object 0xc in metabjset (objset 0x0) is corrupted. > > So to get more details you can do the following: > > zdb -e -dddd data1 > > zdb -e -bbcs data1 > > victor > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss-- -- Victor Latushkin phone: x11467 / +74959370467 TSC-Kernel EMEA mobile: +78957693012 Sun Services, Moscow blog: http://blogs.sun.com/vlatushkin Sun Microsystems