Apr 23 02:02:21 SERVER144 offline or reservation conflict Apr 23 02:02:21 SERVER144 scsi: [ID 107833 kern.warning] WARNING: /scsi_vhci/disk at g60001fe100118db00009119074440055 (sd82): Apr 23 02:02:21 SERVER144 i/o to invalid geometry Apr 23 02:02:21 SERVER144 scsi: [ID 107833 kern.warning] WARNING: /scsi_vhci/disk at g60001fe100118db00009119074440055 (sd82): Apr 23 02:02:21 SERVER144 offline or reservation conflict Apr 23 02:02:21 SERVER144 scsi: [ID 107833 kern.warning] WARNING: /scsi_vhci/disk at g60001fe100118db00009119074440055 (sd82): Apr 23 02:02:21 SERVER144 i/o to invalid geometry Apr 23 02:02:21 SERVER144 scsi: [ID 107833 kern.warning] WARNING: /scsi_vhci/disk at g60001fe100118db00009119074440055 (sd82): Apr 23 02:02:21 SERVER144 offline or reservation conflict Apr 23 02:02:21 SERVER144 scsi: [ID 107833 kern.warning] WARNING: /scsi_vhci/disk at g60001fe100118db00009119074440055 (sd82): Apr 23 02:02:21 SERVER144 i/o to invalid geometry Apr 23 02:02:21 SERVER144 scsi: [ID 107833 kern.warning] WARNING: /scsi_vhci/disk at g60001fe100118db00009119074440055 (sd82): Apr 23 02:02:21 SERVER144 offline or reservation conflict Apr 23 02:02:21 SERVER144 scsi: [ID 107833 kern.warning] WARNING: /scsi_vhci/disk at g60001fe100118db00009119074440055 (sd82): Apr 23 02:02:21 SERVER144 i/o to invalid geometry Apr 23 02:02:21 SERVER144 scsi: [ID 107833 kern.warning] WARNING: /scsi_vhci/disk at g60001fe100118db00009119074440055 (sd82): Apr 23 02:02:21 SERVER144 offline or reservation conflict Apr 23 02:02:21 SERVER144 scsi: [ID 107833 kern.warning] WARNING: /scsi_vhci/disk at g60001fe100118db00009119074440055 (sd82): Apr 23 02:02:21 SERVER144 i/o to invalid geometry Apr 23 02:02:21 SERVER144 scsi: [ID 107833 kern.warning] WARNING: /scsi_vhci/disk at g60001fe100118db00009119074440055 (sd82): Apr 23 02:02:21 SERVER144 offline or reservation conflict Apr 23 02:02:21 SERVER144 scsi: [ID 107833 kern.warning] WARNING: /scsi_vhci/disk at g60001fe100118db00009119074440055 (sd82): Apr 23 02:02:21 SERVER144 i/o to invalid geometry Apr 23 02:02:21 SERVER144 scsi: [ID 107833 kern.warning] WARNING: /scsi_vhci/disk at g60001fe100118db00009119074440055 (sd82): Apr 23 02:02:21 SERVER144 offline or reservation conflict Apr 23 02:02:21 SERVER144 scsi: [ID 107833 kern.warning] WARNING: /scsi_vhci/disk at g60001fe100118db00009119074440055 (sd82): Apr 23 02:02:21 SERVER144 i/o to invalid geometry Apr 23 02:02:21 SERVER144 scsi: [ID 107833 kern.warning] WARNING: /scsi_vhci/disk at g60001fe100118db00009119074440055 (sd82): Apr 23 02:02:21 SERVER144 offline or reservation conflict Apr 23 02:02:21 SERVER144 scsi: [ID 107833 kern.warning] WARNING: /scsi_vhci/disk at g60001fe100118db00009119074440055 (sd82): Apr 23 02:02:21 SERVER144 i/o to invalid geometry Apr 23 02:02:22 SERVER144 unix: [ID 836849 kern.notice] Apr 23 02:02:22 SERVER144 ^Mpanic[cpu1]/thread=ffffff0017fa1c80: Apr 23 02:02:22 SERVER144 genunix: [ID 809409 kern.notice] ZFS: I/O failure (write on <unknown> off 0: zio ffffffff9a5d4cc0 [L0 bplist] 4000L/4000P DVA[0]=<0:770b24 000:4000> DVA[1]=<0:dfa984000:4000> fletcher4 uncompressed LE contiguous birth=260276 fill=1 cksum=1:1000:800800:2ab2ab000): error 5 Apr 23 02:02:22 SERVER144 unix: [ID 100000 kern.notice] Apr 23 02:02:23 SERVER144 genunix: [ID 655072 kern.notice] ffffff0017fa1a40 zfs:zio_done+17c () Apr 23 02:02:23 SERVER144 genunix: [ID 655072 kern.notice] ffffff0017fa1a60 zfs:zio_next_stage+b3 () Apr 23 02:02:23 SERVER144 genunix: [ID 655072 kern.notice] ffffff0017fa1ab0 zfs:zio_wait_for_children+5d () Apr 23 02:02:23 SERVER144 genunix: [ID 655072 kern.notice] ffffff0017fa1ad0 zfs:zio_wait_children_done+20 () Apr 23 02:02:23 SERVER144 genunix: [ID 655072 kern.notice] ffffff0017fa1af0 zfs:zio_next_stage+b3 () Apr 23 02:02:23 SERVER144 genunix: [ID 655072 kern.notice] ffffff0017fa1b40 zfs:zio_vdev_io_assess+129 () Apr 23 02:02:23 SERVER144 genunix: [ID 655072 kern.notice] ffffff0017fa1b60 zfs:zio_next_stage+b3 () Apr 23 02:02:23 SERVER144 genunix: [ID 655072 kern.notice] ffffff0017fa1bb0 zfs:vdev_mirror_io_done+2af () Apr 23 02:02:23 SERVER144 genunix: [ID 655072 kern.notice] ffffff0017fa1bd0 zfs:zio_vdev_io_done+26 () Apr 23 02:02:23 SERVER144 genunix: [ID 655072 kern.notice] ffffff0017fa1c60 genunix:taskq_thread+1a7 () Apr 23 02:02:23 SERVER144 genunix: [ID 655072 kern.notice] ffffff0017fa1c70 unix:thread_start+8 () Apr 23 02:02:23 SERVER144 unix: [ID 100000 kern.notice] Apr 23 02:02:23 SERVER144 genunix: [ID 672855 kern.notice] syncing file systems... Apr 23 02:02:23 SERVER144 genunix: [ID 433738 kern.notice] [1] Apr 23 02:02:53 SERVER144 last message repeated 20 times Apr 23 02:02:54 SERVER144 genunix: [ID 622722 kern.notice] done (not all i/o completed) Apr 23 02:02:55 SERVER144 genunix: [ID 111219 kern.notice] dumping to /dev/dsk/c2t0d0s3, offset 1677983744, content: kernel Apr 23 02:06:43 SERVER144 genunix: [ID 409368 kern.notice] ^M100% done: 1875291 pages dumped, compression ratio 3.34, Apr 23 02:06:43 SERVER144 genunix: [ID 851671 kern.notice] dump succeeded sd82 is a lun used on a zpool that has been exported 2 days ago ... gino This message posted from opensolaris.org
Gino wrote:> Apr 23 02:02:22 SERVER144 ^Mpanic[cpu1]/thread=ffffff0017fa1c80: > Apr 23 02:02:22 SERVER144 genunix: [ID 809409 kern.notice] ZFS: I/O failure (write on <unknown> off 0: zio ffffffff9a5d4cc0 [L0 bplist] 4000L/4000P DVA[0]=<0:770b24 > 000:4000> DVA[1]=<0:dfa984000:4000> fletcher4 uncompressed LE contiguous birth=260276 fill=1 cksum=1:1000:800800:2ab2ab000): error 5 > Apr 23 02:02:22 SERVER144 unix: [ID 100000 kern.notice] > Apr 23 02:02:23 SERVER144 genunix: [ID 655072 kern.notice] ffffff0017fa1a40 zfs:zio_done+17c () > Apr 23 02:02:23 SERVER144 genunix: [ID 655072 kern.notice] ffffff0017fa1a60 zfs:zio_next_stage+b3 () > Apr 23 02:02:23 SERVER144 genunix: [ID 655072 kern.notice] ffffff0017fa1ab0 zfs:zio_wait_for_children+5d () > Apr 23 02:02:23 SERVER144 genunix: [ID 655072 kern.notice] ffffff0017fa1ad0 zfs:zio_wait_children_done+20 () > Apr 23 02:02:23 SERVER144 genunix: [ID 655072 kern.notice] ffffff0017fa1af0 zfs:zio_next_stage+b3 () > Apr 23 02:02:23 SERVER144 genunix: [ID 655072 kern.notice] ffffff0017fa1b40 zfs:zio_vdev_io_assess+129 () > Apr 23 02:02:23 SERVER144 genunix: [ID 655072 kern.notice] ffffff0017fa1b60 zfs:zio_next_stage+b3 () > Apr 23 02:02:23 SERVER144 genunix: [ID 655072 kern.notice] ffffff0017fa1bb0 zfs:vdev_mirror_io_done+2af () > Apr 23 02:02:23 SERVER144 genunix: [ID 655072 kern.notice] ffffff0017fa1bd0 zfs:zio_vdev_io_done+26 () > Apr 23 02:02:23 SERVER144 genunix: [ID 655072 kern.notice] ffffff0017fa1c60 genunix:taskq_thread+1a7 () > Apr 23 02:02:23 SERVER144 genunix: [ID 655072 kern.notice] ffffff0017fa1c70 unix:thread_start+8 () > Apr 23 02:02:23 SERVER144 unix: [ID 100000 kern.notice] > Apr 23 02:02:23 SERVER144 genunix: [ID 672855 kern.notice] syncing file systems... > Apr 23 02:02:23 SERVER144 genunix: [ID 433738 kern.notice] [1] > Apr 23 02:02:53 SERVER144 last message repeated 20 times > Apr 23 02:02:54 SERVER144 genunix: [ID 622722 kern.notice] done (not all i/o completed) > Apr 23 02:02:55 SERVER144 genunix: [ID 111219 kern.notice] dumping to /dev/dsk/c2t0d0s3, offset 1677983744, content: kernel > Apr 23 02:06:43 SERVER144 genunix: [ID 409368 kern.notice] ^M100% done: 1875291 pages dumped, compression ratio 3.34, > Apr 23 02:06:43 SERVER144 genunix: [ID 851671 kern.notice] dump succeeded >The panic looks quite like bug# 6390641 (error# is 5 in this case, though). Bug 6390641 is a dup of 6390236 which is closed as "Not Reproducible". Looks like Gino got a dump and looks like the CR is back in business! ;) It is ''interesting'' however, that 6390236 belongs to the subcategory ''utility:zfs''. Looks like ''kernel:zfs'' to me. The bug comments/evaluation are not public. Unless this is a security issue, (and it doesn''t look like one,) isn''t it time to open the evaluation/comments to the community''s eye? Ah, well... :) -Manoj
Mark Roderick
2007-May-23 16:29 UTC
[zfs-discuss] Re: ZFS panic caused by an exported zpool??
I have a T2000 with an 11/06 release of Solaris 10 installed. I had created a zpool with one LUN in it. Due to an apparent incompatibility with our HBA''s and switches I unplugged the fibre cables from the server''s HBAs. Obviously this is a dev server ;) Two days later an admin logs in on the serial console, types ls, and produces an error similiar to what you are seeing. And the box went down. In this case the zpool had _NOT_ been exported and did not have access to any of the drives in the pool.>From messages:May 23 07:57:21 SCOTLAND unix: [ID 836849 kern.notice] May 23 07:57:21 SCOTLAND ^Mpanic[cpu0]/thread=2a101cddcc0: May 23 07:57:21 SCOTLAND unix: [ID 809409 kern.notice] ZFS: I/O failure (write on <unknown> off 0: zio 6000b7bb400 [L0 unallocated] 4000L/400P DVA[0]=<0:10000:400> DVA[1]=<0:140010000:400> fletcher4 lzjb BE contiguous birth=34165 fill=0 cksum=66c2f1e111:35b2f57598b7:10d666bf86d80f:40518a8662274e3): error 5 May 23 07:57:21 SCOTLAND unix: [ID 100000 kern.notice] May 23 07:57:21 SCOTLAND genunix: [ID 723222 kern.notice] 000002a101cdd740 zfs:zio_done+284 (6000b7bb400, 0, a8, 704a7ca0, 0, 6000603e480) May 23 07:57:21 SCOTLAND genunix: [ID 179002 kern.notice] %l0-3: 000006000b769580 00000000704a7c00 0000000000000005 0000000000000005 May 23 07:57:21 SCOTLAND %l4-7: 0000000000000010 0000000000000002 0000000000008575 0000000000000005 May 23 07:57:21 SCOTLAND genunix: [ID 723222 kern.notice] 000002a101cdd940 zfs:zio_vdev_io_assess+178 (6000b7bb400, 8000, 10, 0, 0, 10) May 23 07:57:22 SCOTLAND genunix: [ID 179002 kern.notice] %l0-3: 0000000000010000 000006000b5cb8b8 0000000000000000 0000000000000005 May 23 07:57:22 SCOTLAND %l4-7: 0000000000000010 0000000000000002 0000000000000000 000006000b5cb8b0 May 23 07:57:22 SCOTLAND genunix: [ID 723222 kern.notice] 000002a101cdda00 genunix:taskq_thread+1a4 (6000b5cb8e8, 6000b5cb890, 50001, 9b9282c16982, 2a101cddaca, 2a101cddac8) May 23 07:57:22 SCOTLAND genunix: [ID 179002 kern.notice] %l0-3: 0000000000010000 000006000b5cb8b8 000006000b5cb8c0 000006000b5cb8c2 May 23 07:57:22 SCOTLAND %l4-7: 000006000b716d98 0000000000000002 0000000000000000 000006000b5cb8b0 May 23 07:57:23 SCOTLAND unix: [ID 100000 kern.notice] May 23 07:57:23 SCOTLAND genunix: [ID 672855 kern.notice] syncing file systems... This message posted from opensolaris.org
ZFS will panic during I/O failure if the zpool is not fully redundant. So you need 2 hba, 2 switches and a RAID10 zpool to keep your server running. Also upgrade to snv_60 or newer. Older release can corrupt your zpool! gino This message posted from opensolaris.org