freebsd stable - Jun 2013 - Help! :( ZFS panic on boot, importing pool after server crash.

Hi, I'm a bit at the end of my tether.

We had a ZFS panic last night on a machine that hosts all my mail and web; it
was rebooted and it now panics mounting the ZFS root filesystem.

The call stack info is:

	solaris assert: ss == NULL, file:
/usr/src/sys/modules/zfs/../../cddl/contrib/opensource/uts/common/fs/zfs/space_map.c,
line: 109

	kdb_backtrace
	panic
	space_map_add
	space_map_load
	metaslab_activate
	metaslab_allocate
	zio_dva_allocate	
	zio_execute
	taskqueue_run_locked
	taskqueue_thread_loop
	fork_exit
	fork_trampoline

I can boot from the live DVD filesystem, but I can only mount the pool read-only
without getting the same kernel panic.  This is with FreeBSD 9.0.

The machine is remote, and I don't have access other than through a DRAC
console port (so I can't cut and paste; sorry for the poor stack trace).

Is anyone here in the position to advice me how I might process to get this
machine mounting and running again in multi-user mode?

Thanks so much.
Joe

p.s. the config, btw, is a ZFS mirror on two ad devices. It's got a ZFS root
file system.

14.06.2013 12:55, Dr Josef Karthauser:
> Hi, I'm a bit at the end of my tether.
>
> We had a ZFS panic last night on a machine that hosts all my mail and web;
it was rebooted and it now panics mounting the ZFS root filesystem.
>
> The call stack info is:
>
> 	solaris assert: ss == NULL, file:
/usr/src/sys/modules/zfs/../../cddl/contrib/opensource/uts/common/fs/zfs/space_map.c,
line: 109
>
> 	kdb_backtrace
> 	panic
> 	space_map_add
> 	space_map_load
> 	metaslab_activate
> 	metaslab_allocate
> 	zio_dva_allocate	
> 	zio_execute
> 	taskqueue_run_locked
> 	taskqueue_thread_loop
> 	fork_exit
> 	fork_trampoline
>
> I can boot from the live DVD filesystem, but I can only mount the pool
read-only without getting the same kernel panic.  This is with FreeBSD 9.0.
>
> The machine is remote, and I don't have access other than through a
DRAC console port (so I can't cut and paste; sorry for the poor stack
trace).
>
> Is anyone here in the position to advice me how I might process to get this
machine mounting and running again in multi-user mode?
There's no official way.
> p.s. the config, btw, is a ZFS mirror on two ad devices. It's got a ZFS
root file system.
If you are fairly sure about your devices you can:

1. Remove second disk from pool or create another pool on top of it.

2. Recreate all FS structure on the second disk. You can dump al your FS 
with something like:

zfs list -Ho name | xargs -n1 zfs get -H all | awk 
'BEGIN{shard="";output=""}{if(shard!=$1 &&
shard!=""){output="zfs
create";for(param in params)output=output" -o
"param"="params[param];print
output" "shard;delete 
params;shard=""}}$4~/local/{params[$2]=$3;shard=$1;next}$2~/type/{shard=$1}END{output="zfs
create";for(param in params)output=output" -o 
"param"="params[param];print output" "shard;}'

Be sure to rename the pool and change the first line.

3. Rsync all data to the second disk.

4. Try to boot from the second disk.

If everything worked you are free to attach first disk to second one to 
create a mirror again.

-- 
Sphinx of black quartz, judge my vow.