freebsd stable - Feb 2005 - Lock/reboot, no dump (ugh)

Hi folks;

FreeBSD 5.3-STABLE #1: Wed Feb  2 22:57:48 CST 2005    
karl@FS.denninger.net:/usr/obj/usr/src/sys/KSD-SMP

Sources from January 30th.

Scenario:

1.	Using GEOM_MIRROR to mirror two SATA drives.

2.	Nightly, a third drive is used to back up, as follows:

	a. Check to see if the drive is visible on the SATA interface.
	b. If not, "atacontrol attach 2" to scan the bus it is plugged into
	c. Verify that it is now online.
	d. Use "gmirror insert ...." to insert it into the mirror.
	e. Wait for it to sync.
	f. Stop critical processes (e.g. DBMS, etc)
	g. "gmirror deactivate ...." to remove the backup from the mirror.
	h. "gmirror forget" to clean up the RAID
	i. "atacontrol detach 2" to detach and spin down the disk.

	The disk is now removeable without drama.

Initially, this works fine.

After a couple of days, it gets flakey.  The disk is found during the
attach, but the devices for everything other than the base drive are
MISSING (e.g. the slice and partition table entries in /dev)  Attempts 
to access the base device also fail with "Device not configured" and
of
course the rest of the script aborts since the disk isn't there when it
tries to add it to the mirror.

If you push it by trying to detach and attach a couple more times,
eventually the system will just freeze up.  A minute or so later, it
spontaneously reboots.

During the "freeze" the console is deader than a doornail - the CAPS
key
flips the light, but VT selection is dead, etc.  I/O is completely
quiescent during this time as well, including from the network.

Sorry I don't have a crash dump from this - at least so far I've been 
unable to coax it into producing one.

I'm going to try removing the "attach/detach" stuff and see if
that helps,
in an attempt to figure out where its getting pissed off.

--
-- 
Karl Denninger (karl@denninger.net) Internet Consultant & Kids Rights
Activist
http://www.denninger.net	My home on the net - links to everything I do!
http://scubaforum.org		Your UNCENSORED place to talk about DIVING!
http://www.spamcuda.net		SPAM FREE mailboxes - FREE FOR A LIMITED TIME!
http://genesis3.blogspot.com	Musings Of A Sentient Mind

On Tue, 8 Feb 2005, Karl Denninger wrote:
> Hi folks;
>
> FreeBSD 5.3-STABLE #1: Wed Feb  2 22:57:48 CST 2005    
karl@FS.denninger.net:/usr/obj/usr/src/sys/KSD-SMP
>
> Sources from January 30th.
>
> Scenario:
>
> 1.	Using GEOM_MIRROR to mirror two SATA drives.
>
> 2.	Nightly, a third drive is used to back up, as follows:
>
> 	a. Check to see if the drive is visible on the SATA interface.
> 	b. If not, "atacontrol attach 2" to scan the bus it is plugged
into
> 	c. Verify that it is now online.
> 	d. Use "gmirror insert ...." to insert it into the mirror.
> 	e. Wait for it to sync.
> 	f. Stop critical processes (e.g. DBMS, etc)
> 	g. "gmirror deactivate ...." to remove the backup from the
mirror.
> 	h. "gmirror forget" to clean up the RAID
> 	i. "atacontrol detach 2" to detach and spin down the disk.
This is only safe if you also unmount the filesystem(s) that are stored on
the mirror volume.  If you don't unmount it then you'll get behavior
like
this -- unflushed blocks vanish.  If you insist on doing dirty copies then
you might try mounting the FS sync, but performance will fall through the
floor.

If you're testing simple attach/detach then you might start with a
quiescent volume (unmounted or filled with test patterns) and do mirror
attach/sync/detach in a loop.  THat would take the FS, and any in-memory
filesystem state corruption, out of the picture.

-- 
Doug White                    |  FreeBSD: The Power to Serve
dwhite@gumbysoft.com          |  www.FreeBSD.org