freebsd stable - Apr 2008 - Dreadful gmirror performance, though each half works fine

I am trying to run a system with a pair of drives mirrored under
gmirror, one of thhem being local and the other remote using ggated.

I've measured the performance of both the local and the remote drive
individually, and they are both fine - can get about 70meg/second out
of them. I've also created mirrors with oone drive in of each of thhem, and
these perform fine too - i.e. each mirror with one drive still gives me
around 65-70 meg/second.

When I create a mirror by taking one of them and adding the other the mirror
resyncs at full speed (i.e. I see network traffic of around 70 meg/second
and the time taken to complete the resync correspnds to that rate).

...and then the performance of the drive drops to a few k/second ?! I am
running myself on top of the mirror drives, but it runs like treacle. During
this time there is no intensive CPU activity, no intensive network activits
and no intensive disc activity - the machine just seems completely cuiqre.
but doing an 'ls' on the mirror drive takes forever.

If i then take a drive out using 'gmirror deactivate' the command
freezes
up for a couple of minutes, but enevtiually comes back. I then
do a 'gmirror status' and that also takes a minute or so to come back.
During
the time these commands are running I get 100% disc activity in 'gstat'
on
the drive I took out - though almost no actual data being trasferred.

...at this point I am back to the situation I started with (a mirror with
only one disc in it) but the performance is still horrible. I can stop
the mirror, detach the ggate drive, reattach, restart - still horrible
performance, even though I have now gone back to the situation I
had before.

Has anyone else seen this ? I've tried various configurations, accross
several machines, all with the same result. It looks like ggate gets itself
into a state where it is suddenly very slow, and then cannot recover from
that. This doesnt happen with local gmirrors, nor if I replace the
underlying ggate with iscsi (though that causes kernel panics but that's
another story).

Any advice ?

-pete.

PS: All machines are amd64/7.0-RELEASE

> I have experimented with this rather extensively and have operationally
> decided not to use ggated in combination with gmirror --- it doesn't
appear
> to work as well as one might expect.
Ah, thats unmfortunate :-( I oroginally started off using the
iscsi initiator and target, which did work O.K., but when actually
ran live ended up locking up after several hours,a nd then panicing
the kernel. So not ideal - but when it was working it was fine. ggated
seems the opposite - doesnt crash, but performance is not suitable for any
kind of real use.
> I'm somewhat vaguely wondering if zfs with one local and one ggated
disk
> will work well.
I tried ZFS for a while myself, and it works O.K., but has a tendency
to panic if it wants memory which it can't get. Despite the many different
guides available, I never managed to get it to the point where I would
be happy to use it on a production system without worrying about it
suddenly becomming memory hungry and dieing.

Thanks for the inout though - I am doing some more experimentation
with ggate (basically raing some buffers as per a thread I found) and
seeing if that helps.

BTW, I think ggate is the problem and not gmirror here - gmirror on top
of iscsi works fine as I said.
-pete.

On Thu, Apr 17, 2008 at 7:18 AM, Pete French <petefrench@ticketswitch.com>
wrote:
> I am trying to run a system with a pair of drives mirrored under
> gmirror, one of thhem being local and the other remote using ggated.
>
I have experimented with this rather extensively and have operationally
decided not to use ggated in combination with gmirror --- it doesn't appear
to work as well as one might expect.

Some improvements were made to ggated awhile ago --- to improve error
recovery.  It should have helped, but didn't.

I'm somewhat vaguely wondering if zfs with one local and one ggated disk
will work well.