zfs discuss - Sep 2006 - Need Help: Getting error "zfs:bad checksum (read on <unknown> off...)

All,

Customer who has 15K running Solaris 10 6/06 with 3510 array attached, 
has a disk array failure
and ZFS was mirroring.  The system panic and can''t boot off either in 
single user mode nor in multi-user mode.
The panic error shows:

IMPACT:panic[cpu512]/thread=2a1022a7cc0: ZFS: bad checksum (read on 
<unknown> off 0: zio 3003ca93900 [L0 SPA space map] 1000L/a00P 
DVA[0]=<1:11128e9400:a00> DVA[1]=<2:111204e400:a00> fletcher4 lzjb
BE
contiguous birth=794148 fill=1 
cksum=71ac8b094c:9a49ad9f14f9:7b1154b0bb95af:49201a336bbfbf6a): error 50

000002a1022a7740 zfs:zio_done+284 (3003ca93900, 0, a8, 70785be0, 0, 
3001847d240)
  %l0-3: 0000030019a1fcc0 000003003e8646c0 0000000000000032 0000000000000032
  %l4-7: 0000000000000002 0000000000000001 0000000000000000 0000000000000032
000002a1022a7940 zfs:zio_vdev_io_assess+178 (3003ca93900, 8000, 10, 0, 
0, 10)
  %l0-3: 0000000000000001 0000000000000002 0000000000000000 0000000000000032
  %l4-7: 0000000000000000 ffffffffffffffff 0000000000000000 0000030017a34000
000002a1022a7a00 genunix:taskq_thread+1a4 (30019a2b670, 30019a2b618, 
50001, d548643cc3, 2a1022a7aca, 2a1022a7ac8)
  %l0-3: 0000000000010000 0000030019a2b640 0000030019a2b648 0000030019a2b64a
  %l4-7: 0000030019b0c9e0 0000000000000002 0000000000000000 0000030019a2b638


Customer''s main concern right now is to make the system bootable but it
seems couldn''t do that since the bad disks is part
of the zfs filesystems.  Is there a way to disable or clear out the bad 
zfs filesystem so system can be booted?

Also i did search on the error "ZFS:  bad checksum" error and i found 
some bugs.

Any assistance on this issue is greatly appreciated. Customer''s domain 
is still down at this point.

TIA,
Arlina

NOTE: PLease email me directly as i''m not on this alias.

Arlina Goce-Capiral wrote:
> Customer''s main concern right now is to make the system bootable
but it
> seems couldn''t do that since the bad disks is part
> of the zfs filesystems.  Is there a way to disable or clear out the bad 
> zfs filesystem so system can be booted?
Yes, see this FAQ:

http://opensolaris.org/os/community/zfs/faq/#zfspanic

quote:

What can I do if ZFS panics on every boot?

     ZFS is designed to survive arbitrary hardware failures through the 
use of redundancy (mirroring or RAID-Z). Unfortunately, certain failures 
in non-replicated configurations can cause ZFS to panic when trying to 
load the pool. This is a bug, and will be fixed in the near future 
(along with several other nifty features like background scrubbing and 
the ability to see a list of corrupted files). In the meantime, if you 
find yourself in the situation where you cannot boot due to a corrupt 
pool, do the followng:

        1. boot using ''-m milestone=none''
        2. # mount -o remount /
        3. # rm /etc/zfs/zpool.cache
        4. # reboot

     This will remove all knowledge of pools from your system. You will 
have to re-create your pool and restore from backup.


--matt

Hello Matthew,

Thanks for your very helpful informations.
While waiting for your reply, i checed the ZFS admin Guide and it was in 
there, on page 150. :-)

Have a good weekend.

Thanks again.
Arlina-