freebsd stable - Apr 2011 - ZFS HAST config preference

This is more of an proof of concept question:

I am building an redundant cluster of blade servers, and toy with the 
idea to use HAST and ZFS for the storage.

Blades will work in pairs and each pair will provide various services, 
from SQL databases, to hosting virtual machines (jails and otherwise). 
Each pair will use CARP for redundancy.

My original idea was to set up blades so that they run HAST on pairs of 
disks, and run ZFS in number of mirror vdevs on top of HAST. The ZFS 
pool will exist only on the master HAST node. Let's call this setup1.

Or, I could use ZFS volumes and run HAST on top of these. This means, 
that on each blade, I will have an local ZFS pool. Let's call this setup2.

Third idea would be to have the blades completely diskless, boot from 
separate boot/storage server and mount filesystems or iSCSI volumes as 
needed from the storage server. HAST might not be necessary here. ZFS 
pool will exist on the storage server only. Let's call this setup3.

While setup1 is most straightforward, it has some drawbacks:
- disks handled by HAST need to be either identical or have matching 
partitions created;
- the 'spare' blade would do nothing, as it's disk subsystem will be
gone as long as it is HAST slave. As the blades are quite powerful (4x8 
core AMD) that would be wasteful, at least in the beginning.

With setup2, I can get away with different size disks in each blade. All 
blades can also be used for whatever additional processing, shared data 
will be only presented by HAST to whichever node needs it, for 
"important" services. One drawback here:
- can't just pull off one of the blades, without stopping/transferring 
first all of it's services.

It seems that in larger scale, setup3 would be best. I am not yet here, 
although close (the storage server is missing).

HAST replication speed should not be an issue, there is 10Gbit network 
between the blade servers.

Has anyone already setup something similar? What was the experience? 
There were recently some bugs that sort of plagued setup1, but these 
seem to be resolved now.

Daniel

> My original idea was to set up blades so that they run HAST on pairs of 
> disks, and run ZFS in number of mirror vdevs on top of HAST. The ZFS 
> pool will exist only on the master HAST node. Let's call this setup1.
This is exactly how I run things. Personally I think
it is the best solution, as ZFS then knows about the
mirroring of the drives, always a good thing, and you
geta  ZFS filesystem on the top. I also enable compression
to reduce the bandwidth to the drives, as tha reduces the
data flowing across the network. For what I am doing (running
mysql on top) this is actually faster for both selects and
inserts. test with your own application first though.
> Or, I could use ZFS volumes and run HAST on top of these. This means, 
> that on each blade, I will have an local ZFS pool. Let's call this
setup2.
...you would need to put a filesystem on to of the
HAST filesystem though, what would that be ?
> While setup1 is most straightforward, it has some drawbacks:
> - disks handled by HAST need to be either identical or have matching 
> partitions created;
This is true. I run identical machines as primary and secondary.
> - the 'spare' blade would do nothing, as it's disk subsystem
will be
> gone as long as it is HAST slave. As the blades are quite powerful (4x8 
> core AMD) that would be wasteful, at least in the beginning.
If you are keeping a machine as a hot spare then thats something
you just have to live with in my opinion. I've run this
way for several years - before HAST and ZFS we used gmirror and UFS
to do the same thing. It does work very nicely, but you do end up
with a machine idle. 
> HAST replication speed should not be an issue, there is 10Gbit network 
> between the blade servers.
I actually ut separate ether interfaces for each of the hast drives.
So for 2 drives there are 2 space ether ports on the machine, with
a cible between them, deidcated to just that drive. Those are gigabit
cards though, not 10 gig.
> Has anyone already setup something similar? What was the experience? 
very good actually. One thing I would say is to write and test a set of
scvripts to do the failover - avoids shooting yourself in the foot
when trying to do the commands by hand (which is rather easy to do).
I have a script to make the orimary into a secondary,
and one to d the reverse. The first scipt waits unitl the HAST data
is all flushed before changing role, and makes sure the services
are stoped, pool exported before ripping out the disc from underneath.
The script also handles removing a shared IP address from the
interface. 

-pete.