zfs discuss - Mar 2008 - ZFS panics solaris while switching a volume to read-only

Hi,

I just found out that ZFS triggers a kernel-panic while switching a mounted
volume
into read-only mode:

The system is attached to a Symmetrix, all zfs-io goes through Powerpath:

I ran some io-intensive stuff on /tank/foo and switched the device into
read-only mode at the same time (symrdf -g bar failover -establish).

ZFS went ''bam'' and triggered a Panic:

WARNING: /pci at 1f,700000/pci at 0/SUNW,emlxs at 2/fp at 0,0/ssd at
w5006048452a4f797,e5 (ssd0):
        Error for Command: write(10)               Error Level: Fatal
        Requested Block: 19062                     Error Block: 19062
        Vendor: EMC                                Serial Number: 900D5008U   
        Sense Key: 
panic[cpu0]/thread=2a10103bcc0: Write Protected
ZFS: I/O failure (write on <unknown> off 0: zio 6001f3f2340 [L0 DMU dnode]
4000L/a00P DVA[0]=<0:54e000:a00> DVA[1]=<0:d000ab800:a00> fletcher4
lzjb BE contiguous birth=597 fill=31
cksum=97d1c4f896:dd5b900a7ec8:b89066e037beda:7021cc17c66f4713): error 6

000002a10103b880 zfs:zio_done+29c (0, 10, 6000e361400, 600202bfb80, 6001f3f2340,
0)
  %l0-3: 0000060003233b40 00000600109c1740 000000007085a218 0000000000000006
  %l4-7: 0000000000000002 0000000000000255 0000000000000006 00000000018bf400
000002a10103b940 zfs:zio_vdev_io_assess+178 (6001f3f2340, 8000, 10, 0, 0, 10)
  %l0-3: 0000000000000002 0000000000000002 0000000000000000 0000000000000006
  %l4-7: 0000000000000010 0000000000000001 0000000000000000 0000000000000000
000002a10103ba00 genunix:taskq_thread+1a4 (60003929228, 600039291d0, 50001,
507428728c, 2a10103baca, 2a10103bac8)
  %l0-3: 0000000000010000 00000600039291f8 0000060003929200 0000060003929202
  %l4-7: 0000060003927a50 0000000000000002 0000000000000000 00000600039291f0

syncing file systems... done



Is there some magic zfs-option such as VxFS''s
''ioerror=disable'' ?



Regards,
 Adrian

You want:

PSARC 2007/567 zpool failmode property

Which went back into build 77 of nevada.

- Eric

On Thu, Mar 20, 2008 at 04:44:43PM +0100, Adrian Ulrich
wrote:
> Hi,
> 
> I just found out that ZFS triggers a kernel-panic while switching a mounted
volume
> into read-only mode:
> 
> The system is attached to a Symmetrix, all zfs-io goes through Powerpath:
> 
> I ran some io-intensive stuff on /tank/foo and switched the device into
> read-only mode at the same time (symrdf -g bar failover -establish).
> 
> ZFS went ''bam'' and triggered a Panic:
> 
> WARNING: /pci at 1f,700000/pci at 0/SUNW,emlxs at 2/fp at 0,0/ssd at
w5006048452a4f797,e5 (ssd0):
>         Error for Command: write(10)               Error Level: Fatal
>         Requested Block: 19062                     Error Block: 19062
>         Vendor: EMC                                Serial Number: 900D5008U
>         Sense Key: 
> panic[cpu0]/thread=2a10103bcc0: Write Protected
> ZFS: I/O failure (write on <unknown> off 0: zio 6001f3f2340 [L0 DMU
dnode] 4000L/a00P DVA[0]=<0:54e000:a00> DVA[1]=<0:d000ab800:a00>
fletcher4 lzjb BE contiguous birth=597 fill=31
cksum=97d1c4f896:dd5b900a7ec8:b89066e037beda:7021cc17c66f4713): error 6
> 
> 000002a10103b880 zfs:zio_done+29c (0, 10, 6000e361400, 600202bfb80,
6001f3f2340, 0)
>   %l0-3: 0000060003233b40 00000600109c1740 000000007085a218
0000000000000006
>   %l4-7: 0000000000000002 0000000000000255 0000000000000006
00000000018bf400
> 000002a10103b940 zfs:zio_vdev_io_assess+178 (6001f3f2340, 8000, 10, 0, 0,
10)
>   %l0-3: 0000000000000002 0000000000000002 0000000000000000
0000000000000006
>   %l4-7: 0000000000000010 0000000000000001 0000000000000000
0000000000000000
> 000002a10103ba00 genunix:taskq_thread+1a4 (60003929228, 600039291d0, 50001,
507428728c, 2a10103baca, 2a10103bac8)
>   %l0-3: 0000000000010000 00000600039291f8 0000060003929200
0000060003929202
>   %l4-7: 0000060003927a50 0000000000000002 0000000000000000
00000600039291f0
> 
> syncing file systems... done
> 
> 
> 
> Is there some magic zfs-option such as VxFS''s
''ioerror=disable'' ?
> 
> 
> 
> Regards,
>  Adrian
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
--
Eric Schrock, Fishworks                        http://blogs.sun.com/eschrock

Hi Eric,
> PSARC 2007/567 zpool failmode property
Thanks, that''s exactly what i''ve been looking for :-)

> Which went back into build 77 of nevada.
Any chance to see this in Solaris-10 ?

We are currently using VxFS on all LUNs ( > 15TB Maildir) and i''d
like
to give ZFS a try on a live system... but this issue is a show-stopper :-/


Regards,
 Adrian

Is there any possibility that the psarc 2007/567 can be made as a patch to
Soalris 10 U5. We are planning to dispose of Veritas as quickly as possible but
since all storage on production machines is on EMC Symmetrix with back-end
mirroring, this panic is a showstopper for us.  Or is it so intertwined that a
back port of this PSARC to U5 is out of the question.

Thanks


Roman
 
 
This message posted from opensolaris.org

The fix is already in Solaris 10 U6. A patch for S10U5 will only be 
available when S10U6 is released.

--
Prabahar.

Veltror wrote:
> Is there any possibility that the psarc 2007/567 can be made as a patch to
Soalris 10 U5. We are planning to dispose of Veritas as quickly as possible but
since all storage on production machines is on EMC Symmetrix with back-end
mirroring, this panic is a showstopper for us.  Or is it so intertwined that a
back port of this PSARC to U5 is out of the question.
> 
> Thanks
> 
> 
> Roman
>  
>  
> This message posted from opensolaris.org
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Why does update 6 have to bve out before a patch can be produced for this? This
is a show-stopper for putting ZFS into production on anything other then local
disks, a production box that panics when a single disk goes offline is  worse
then useless. I cannot see why this is not a high priority bug.
 
 
This message posted from opensolaris.org

| Why does update 6 have to bve out before a patch can be produced
| for this? This is a show-stopper for putting ZFS into production on
| anything other then local disks, a production box that panics when a
| single disk goes offline is worse then useless. I cannot see why this
| is not a high priority bug.

 To be fair to ZFS, ZFS only panics when a pool loses integrity, so
if you have a mirrored or raidz(2) pool that''s drawn from backend SAN
storage you can lose a single volume without having your system keel
over.

 But yes, this is an important bug in a SAN environment, since in
many cases you really want to do much of the storage allocation and
administration in the SAN backend, exporting logical LUNs that the SAN
itself handles redundancy for. We too would like a fix before S10U6.

(Note that answers like ''just export the disks as JBOD and do the
RAID/etc on the ZFS machine'' are not really good ones, because your
system *still* panics if a controller goes down or, worse, is merely
temporarily unreachable.)

	- cks

I am trying to get a commitment to get this fixed, if you have a server with a
whole bunch of SAN attached disks and then use the internals for some sort of
temp space if one of those el cheapo sas disks dies it takes down the whole lot,
not good. This problem is enought o prevent a roll out of zfs in production. ZFS
great for local disks not so great for SAN, sadly all the great features of
snapshots clones etc are being missed whilst this bug is till present. What is
even more annoying is that this is being treated as a RFE.
 
 
This message posted from opensolaris.org