CentOS - May 2011 - scsi3 persistent reservations in cluster storage fencing

I'm interested in the idea of sharing a bunch of SAS JBOD devices 
between two CentOS servers in an active-standby HA cluster sort of 
arrangement, and found something about using scsi3 persistent 
reservations as a fencing method.    I'm not finding a lot of specifics 
about how this works, or how you configure two initiator systems on a 
SAS chain.   I don't have any suitable hardware for running any tests or 
evaluations yet.

general idea:  2 centos servers each with 8 port external SAS cards (2 
x4), cascaded through a SAS box-o-disks with a whole bunch of SAS dual 
ported drives, to implement high availability large nearline 
storage.       all storage configured as JBOD, using linux md raid or 
lvm mirroring. drives should only be accessible by the currently active 
server, with heartbeat managing the fencing.

here's a hardware block diagram
http://freescruz.com/shared-sas.jpg


This is about the only details I've found on scsi persistant 
reservations and linux HA
http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5/html/Configuration_Example_-_Fence_Devices/SCSI_Configuration.html

One question I have is: how well will this scale with several strings of 
100 SAS drives on the same HA pair of servers?

Can SAS storage instead be fenced at the SES/expander level rather than 
having to use reservations with each separate drive?

-- 
john r pierce                            N 37, W 123
santa cruz ca                         mid-left coast

You might be able to go a simpler route, try building a GFBD between
them and sharing the storage that way. I personally have never tried
it, but that might give you the HA you're looking for while staying
withing the CentOS/Upstream packaging. One added benefit is the
ability to scale vertically and horizontally with a setup like that
with failover and quorum. Heck, toss in a little heartbeat scripting
and you could call it an auto-healing storage cloud.

Good luck!

On Fri, May 20, 2011 at 5:28 PM, John R Pierce <pierce at hogranch.com>
wrote:
> I'm interested in the idea of sharing a bunch of SAS JBOD devices
> between two CentOS servers in an active-standby HA cluster sort of
> arrangement, and found something about using scsi3 persistent
> reservations as a fencing method. ? ?I'm not finding a lot of specifics
> about how this works, or how you configure two initiator systems on a
> SAS chain. ? I don't have any suitable hardware for running any tests
or
> evaluations yet.
>
> general idea: ?2 centos servers each with 8 port external SAS cards (2
> x4), cascaded through a SAS box-o-disks with a whole bunch of SAS dual
> ported drives, to implement high availability large nearline
> storage. ? ? ? all storage configured as JBOD, using linux md raid or
> lvm mirroring. drives should only be accessible by the currently active
> server, with heartbeat managing the fencing.
>
> here's a hardware block diagram
> http://freescruz.com/shared-sas.jpg
>
>
> This is about the only details I've found on scsi persistant
> reservations and linux HA
>
http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5/html/Configuration_Example_-_Fence_Devices/SCSI_Configuration.html
>
> One question I have is: how well will this scale with several strings of
> 100 SAS drives on the same HA pair of servers?
>
> Can SAS storage instead be fenced at the SES/expander level rather than
> having to use reservations with each separate drive?
>
> --
> john r pierce ? ? ? ? ? ? ? ? ? ? ? ? ? ?N 37, W 123
> santa cruz ca ? ? ? ? ? ? ? ? ? ? ? ? mid-left coast
>
> _______________________________________________
> CentOS mailing list
> CentOS at centos.org
> http://lists.centos.org/mailman/listinfo/centos
>


-- 
Steven Crothers
steven.crothers at gmail.com

--On Friday, May 20, 2011 02:28:18 PM -0700 John R Pierce
<pierce at hogranch.com> wrote:

[snip]
> One question I have is: how well will this scale with several strings of 
> 100 SAS drives on the same HA pair of servers?
> 
> Can SAS storage instead be fenced at the SES/expander level rather than 
> having to use reservations with each separate drive?
It's been a few years since I've used SCSI reservations as a fencing
method (as in about the time that wide SCSI was taking over from narrow
SCSI, and while it fenced nicely, it sometimes caused bus errors when
you detached the dead node from the bus).

However if you are using a set of disks as a logical unit from
the cluster's perspective (like having set of disks acting as a RAID
device), then IIRC it is sufficient to use a reservation on a single
disk as your fence for the entire disk set (rather than using a 
reservation on every single disk).  Of course, of your storage cabinet
has a few different disk sets in it, you'd need additional reservations
(one per set).

But, as I said, it's been a few years so I'd investigate it more rather
than depending on *just* the above comment.  In recent years I've tended
to use various mechanisms to kill power to the node as my preferred 
fencing method.

The linux-ha and pacemaker lists may be a better source of current 
information.

Devin