I have Solaris 10 U2 with all the latest patches that started to crash recently
on regular basis... so I started to dig and see what is causing it and here is
what I found out:
panic[cpu1]/thread=2a1009a7cc0: really out of space
000002a1009a6d70 zfs:zio_write_allocate_gang_members+33c (60017964140, 400, 200,
3, 7b62f380, 7b64d400)
%l0-3: 0000060017964158 000000000000ffff 0000000000000003 0000000000000000
%l4-7: 000003000ff13200 000003000ff13000 0000000000000000 0000000000000001
000002a1009a6e70 zfs:zio_write_allocate_gang_members+2dc (600179643c0, 800, 200,
400, 7b62f380, 7b64d400)
%l0-3: 00000600179643d8 000000000000ffff 0000000000000003 0000000000000400
%l4-7: 000003000ff13000 000003001666d500 0000000000000000 0000000000000001
000002a1009a6f70 zfs:zio_write_allocate_gang_members+2dc (600167e1740, 800, 400,
800, 7b62f380, 7b64d400)
%l0-3: 00000600167e1758 000000000000ffff 0000000000000001 0000000000001a00
%l4-7: 000003001666d500 000003001674b800 0000000000000000 0000000000000001
000002a1009a7070 zfs:zio_write_allocate_gang_members+2dc (600167e19c0, 4e00,
1400, 1a00, 7b62f380, 7b64d400)
%l0-3: 00000600167e19d8 000000000000ffff 0000000000000003 0000000000001a00
%l4-7: 000003001674b800 000003001557db00 0000000000000000 0000000000000001
000002a1009a7170 zfs:zio_write_allocate_gang_members+2dc (6000b690d80, 4e00,
3c00, 4e00, 7b62f380, 7b64d400)
%l0-3: 000006000b690d98 000000000000ffff 0000000000000001 000000000000ee00
%l4-7: 000003001557db00 0000030023477000 0000000000000000 0000000000000001
000002a1009a7270 zfs:zio_write_compress+1ec (6000b690d80, 23e20b, 23e000,
ff00ff, 3, 30023477000)
%l0-3: 000000000000ffff 0000000000000076 0000000000000077 000000000000ee00
%l4-7: 0000000000000000 0000000000ff0000 000000000000fc00 00000000000000ff
000002a1009a7340 zfs:arc_write+e4 (6000b690d80, 60000ebee00, 6, 3, 1, d02188b)
%l0-3: ffffffffffffffff 000000007b605eec 000003002702a328 0000030020863c08
%l4-7: 000002a1009a7548 0000000000000004 0000000000000004 0000030006b703a0
000002a1009a7450 zfs:zfsctl_ops_root+b223eac (3002702a328, 600085451c0, 12f956,
3, 6, d02188b)
%l0-3: 0000060001dccb00 0000000000000000 000006000b514658 000003002702a448
%l4-7: 0000030023477000 0000000000000013 0000000000000d60 0000000000000000
000002a1009a7570 zfs:dnode_sync+35c (0, 0, 600085451c0, 600162ed7d8, 0, 3)
%l0-3: 000003002702a328 000006000b5146b0 000006000b514770 000006000b514710
%l4-7: 0000000000000060 000006000b5146b3 000000000000000c 000003002703d7a0
000002a1009a7630 zfs:dmu_objset_sync_dnodes+6c (60001dccb00, 60001dccc40,
600162ed7d8, 6000b514658, 0, 0)
%l0-3: 00000000703e0438 00000000703e0000 00000000703e0000 0000000000000001
%l4-7: 0000000000000000 00000000703dc000 0000000000000000 00000600085451c0
000002a1009a76e0 zfs:dmu_objset_sync+54 (60001dccb00, 600162ed7d8, 3, 3,
3002331e8f8, d02188b)
%l0-3: 00000000000000bb 000000000000000f 0000000000000000 0000000000410d60
%l4-7: 0000060001dccc40 0000000000000060 0000060001dccbe0 0000060001dccc60
000002a1009a77f0 zfs:dsl_dataset_sync+c (60008aefa40, 600162ed7d8, 60008aefad0,
300000ad278, 300000ad278, 60008aefa40)
%l0-3: 0000000000000001 0000000000000007 00000300000ad2f8 0000000000000003
%l4-7: 0000060008aefac8 0000000000000000 0000000000000000 0000000000000000
000002a1009a78a0 zfs:dsl_pool_sync+104 (300000ad1c0, d02188b, 60008aefa40,
30005fc8148, 60001dcc880, 60001dcc8a8)
%l0-3: 0000000000000000 0000060000ebf1b8 0000000000000006 00000600162ed7d8
%l4-7: 00000300000ad328 00000300000ad2f8 00000300000ad268 0000000000000000
000002a1009a7950 zfs:spa_sync+e4 (60000ebee00, d02188b, 60001dcc8a8,
600162ed7d8, 60000ebef78, 2a1009a7cbc)
%l0-3: 000000000d021889 0000060000ebef40 0000060000ebef08 0000000000000000
%l4-7: 0000060000e4a080 00000300000ad1c0 00000000000001e0 0000000000000000
000002a1009a7a00 zfs:txg_sync_thread+134 (300000ad1c0, d02188b, 0, 2a1009a7ab0,
300000ad2d0, 300000ad2d2)
%l0-3: 00000300000ad2e0 00000300000ad290 0000000000000000 00000300000ad298
%l4-7: 00000300000ad2d6 00000300000ad2d4 00000300000ad288 000000000d021848
syncing file systems... 970 913 893 893 893 893 893 893 893 893 893 893 893 893
893
SC Alert: Failed to send email alert for recent event.
893 893 893 893 893 893 893 893 done (not all i/o completed)
dumping to /dev/dsk/c1t0d0s1, offset 3436969984, content: kernel
59% done
SC Alert: SC Request to XIR Host due to Watchdog
ERROR: Externally Initiated Reset has occurred.
panic[cpu1]/thread=2a1009a7cc0: sync initiated
dump aborted: please record the above information!
rebooting...
SC Alert: Host System has Reset
XIR/Watchdog Reset
Executing Power On Self Test
0>
Any help? I do zfs file system snaps and I am pretty low on disk space at this
moment but I still have like 2 gig free
mypool/d 182163718 180277654 1886065 99% /d/d2
Thanks for help.
Chris