zfs discuss - May 2011 - JBOD recommendation for ZFS usage

Dear all

Sorry if it''s kind of off-topic for the list but after talking
to lots of vendors I''m running out of ideas...

We are looking for JBOD systems which

(1) hold 20+ 3.3" SATA drives

(2) are rack mountable

(3) have all the nive hot-swap stuff

(4) allow 2 hosts to connect via SAS (4+ lines per host) and see
    all available drives as disks, no RAID volume.
    In a perfect world both hosts would connect each using
    two independent SAS connectors


The box will be used in a ZFS Solaris/based fileserver in a
fail-over cluster setup. Only one host will access a drive
at any given time.

It seems that a lot of vendors offer JBODs but so far I haven''t found
one in Germany which handles (4).

Any hints?

> Dear all
> 
> Sorry if it''s kind of off-topic for the list but after talking
> to lots of vendors I''m running out of ideas...
> 
> We are looking for JBOD systems which
> 
> (1) hold 20+ 3.3" SATA drives
> 
> (2) are rack mountable
> 
> (3) have all the nive hot-swap stuff
> 
> (4) allow 2 hosts to connect via SAS (4+ lines per host) and see
>     all available drives as disks, no RAID volume.
>     In a perfect world both hosts would connect each using
>     two independent SAS connectors
> 
> 
> The box will be used in a ZFS Solaris/based fileserver in a
> fail-over cluster setup. Only one host will access a drive
> at any given time.
> 
> It seems that a lot of vendors offer JBODs but so far I haven''t
found
> one in Germany which handles (4).
> 
> Any hints?
I know of [1] and [2]. The Promise one''s are available for purchase
through retail channels; don''t know about the Xyratex. Neither do I
know about quality, but AFAIK Xyratex is or was main supplier of
NetApp''s JBOD, which I know to be of good quality.


Regards
Florian Wagner

[1]
http://promise.com/storage/raid_series.aspx?region=en-US&m=577&sub_m=sub_m_2&rsn1=1&rsn3=25
[2] http://www.xyratex.com/products/storage-systems/onestor-SP2424s.aspx
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20110530/fbb596b5/attachment.bin>

Dunno about Germany, but LSI and DataON both have offerings.  (The LSI units are
probably going fast, as LSI exits that business having sold that unit to
NetApp.)

  -- Garrett D''Amore

On May 30, 2011, at 10:08 AM, "Thomas Nau" <Thomas.Nau at
uni-ulm.de> wrote:
> Dear all
> 
> Sorry if it''s kind of off-topic for the list but after talking
> to lots of vendors I''m running out of ideas...
> 
> We are looking for JBOD systems which
> 
> (1) hold 20+ 3.3" SATA drives
> 
> (2) are rack mountable
> 
> (3) have all the nive hot-swap stuff
> 
> (4) allow 2 hosts to connect via SAS (4+ lines per host) and see
>    all available drives as disks, no RAID volume.
>    In a perfect world both hosts would connect each using
>    two independent SAS connectors
> 
> 
> The box will be used in a ZFS Solaris/based fileserver in a
> fail-over cluster setup. Only one host will access a drive
> at any given time.
> 
> It seems that a lot of vendors offer JBODs but so far I haven''t
found
> one in Germany which handles (4).
> 
> Any hints?
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Following up on some of this forum''s discussions, I read the manuals on
SuperMicro''s
SC847E26-RJBOD1 this weekend. 

At the very least, this box provides dual-expander backplanes (2 BPs for a total
of
45 hot-swap disks), so each JBOD has 4 outgoing SFF8087 (4xSATA iPass)
connectors.
However it seems that the backplane chips are doubled for multipathing-failover
within
a single head node with dual connections from same or different HBAs. (Further
on,
backplanes may be daisy-chained to attach other JBODs to the same HBA path -
but if you don''t care much for bandwidth limitations).

According to the docs, each chip addresses all disks on its backplane, and it
seems
implied (but not expressly stated) that either one chip and path works, or
another.

So if your application can live with the unit of failover being a bunch of 21 or
24 disks -
that might be a way to go. However each head would only have one connection to
each backplane, and I''m not sure if you can STONITH the non-leading
head to enforce
failovers (and enable the specific PRI/SEC chip of the backplane).

Also one point was stressed many times in the docs: these failover backplanes
require use of SAS drives, no SATA (while the single-path BPs are okay with both
SAS and SATA). Still, according to the forums, SATA disks on shared backplanes
often give too much headache and may give too little performance in
comparison...

I am not sure if this requirement also implies dual SAS data connectors -
pictures
of HCL HDDs all have one connector... 

Still, I gess my post poses mre questions than answers, but maybe some other 
list readers can reply...

Hint: Nexenta people seem to be good OEM friends with Supermicro, so they 
might know ;)

HTH,
//Jim Klimov

Thanks Jim and all the other who have replied so far


On 05/30/2011 11:37 AM, Jim Klimov wrote:
> ...
> 
> So if your application can live with the unit of failover being a bunch of
21 or 24 disks -
> that might be a way to go. However each head would only have one connection
to
> each backplane, and I''m not sure if you can STONITH the
non-leading head to enforce
> failovers (and enable the specific PRI/SEC chip of the backplane).
That exactly my point. I don''t need any internal failover which
restricts
which disks a host can see. We want to failover between hosts, not connections.
For the later we would use another JBOD and let ZFS do the dirty job or
mirroring.
We run a similar setup for years but with FC connected RAID systems. Over time
they
are kind of limited when it comes to price/performance
> Also one point was stressed many times in the docs: these failover
backplanes
> require use of SAS drives, no SATA (while the single-path BPs are okay with
both
> SAS and SATA). Still, according to the forums, SATA disks on shared
backplanes
> often give too much headache and may give too little performance in
comparison...
I would be fine with SAS as well

Thomas

On Mon, May 30, 2011 at 08:06:31AM +0200, Thomas Nau
wrote:
> 
> We are looking for JBOD systems which
> (1) hold 20+ 3.3" SATA drives
> (2) are rack mountable
> (3) have all the nive hot-swap stuff
> (4) allow 2 hosts to connect via SAS (4+ lines per host) and see
>     all available drives as disks, no RAID volume.
>     In a perfect world both hosts would connect each using
>     two independent SAS connectors
> 
> The box will be used in a ZFS Solaris/based fileserver in a
> fail-over cluster setup. Only one host will access a drive
> at any given time.
I''m using a J4200 array as shared storage for a cluster.  It needs a
SAS HBA in each cluster node.  The disks in the array are visible to
both nodes in the cluster.  Here''s the feature list.  I don''t
know if
it''s still available:

    Sun Storage J4200 Array:
    # Scales up to 48 SAS/SATA disk drives
    # Provides up to 72 Gb/sec of total bandwidth
    * Up to 72 Gb/sec of total bandwidth
    * Four x4-wide 3 Gb/sec SAS host/uplink ports (48 Gb/sec bandwidth)
    * Two x4-wide 3 Gb/sec SAS expansion ports (24 Gb/sec bandwidth)
    * Scales up to 48 drives

-- 
-Gary Mills-        -Unix Group-        -Computer and Network Services-

On May 30, 2011, at 2:37 AM, Jim Klimov wrote:
> Following up on some of this forum''s discussions, I read the
manuals on SuperMicro''s
> SC847E26-RJBOD1 this weekend. 
We see quite a few of these in the NexentaStor installed base. The other
commonly
found 3.5" 24-drive JBOD is the DataON DNS-1600, a product staggeringly
similar to
Sun''s J4400.
> At the very least, this box provides dual-expander backplanes (2 BPs for a
total of
> 45 hot-swap disks), so each JBOD has 4 outgoing SFF8087 (4xSATA iPass)
connectors.
> However it seems that the backplane chips are doubled for
multipathing-failover within
> a single head node with dual connections from same or different HBAs.
(Further on,
> backplanes may be daisy-chained to attach other JBODs to the same HBA path
-
> but if you don''t care much for bandwidth limitations).
We also commonly see the dual-expander backplanes.
> According to the docs, each chip addresses all disks on its backplane, and
it seems
> implied (but not expressly stated) that either one chip and path works, or
another.
For SAS targets, both paths work simultaneously.
> So if your application can live with the unit of failover being a bunch of
21 or 24 disks -
> that might be a way to go. However each head would only have one connection
to
> each backplane, and I''m not sure if you can STONITH the
non-leading head to enforce
> failovers (and enable the specific PRI/SEC chip of the backplane).
The NexentaStor HA-Cluster plugin manages STONITH and reservations.
I do not believe programming expanders or switches for clustering is the best
approach.
It is better to let the higher layers manage this.
> Also one point was stressed many times in the docs: these failover
backplanes
> require use of SAS drives, no SATA (while the single-path BPs are okay with
both
> SAS and SATA). Still, according to the forums, SATA disks on shared
backplanes
> often give too much headache and may give too little performance in
comparison...
The cost of a SATA disk + SATA/SAS interposer is about the same as a native SAS
drive. Native SAS makes a better solution.
> I am not sure if this requirement also implies dual SAS data connectors -
pictures
> of HCL HDDs all have one connector... 
These are dual ported.
> Still, I gess my post poses mre questions than answers, but maybe some
other
> list readers can reply...
> 
> Hint: Nexenta people seem to be good OEM friends with Supermicro, so they 
> might know ;)
Yes :-)
 -- richard

Thanks, now I have someone to interrogate, who seems to have 
seen these boxes live - if you don''t mind ;)

----- Original Message -----
From: Richard Elling <richard.elling at gmail.com>
Date: Monday, May 30, 2011 22:04
> We also commonly see the dual-expander backplanes.
> 
> > According to the docs, each chip addresses all disks on its 
> backplane, and it seems
> > implied (but not expressly stated) that either one chip and 
> path works, or another.
> 
> For SAS targets, both paths work simultaneously.
 
Does this mean that if the J0 uplinks of backplanes are connected 
to HBAs in two different servers, both of these servers can address 
individual disks (and the unit of failover is not a backplane but a disk
after all)?
 
And if both HBAs are in a single server, this doubles the SAS link
throughput by having two paths - and can ZFS somehow balance
among them?
> 
> > So if your application can live with the unit of failover 
> being a bunch of 21 or 24 disks -
> > that might be a way to go. However each head would only have 
> one connection to
> > each backplane, and I''m not sure if you can STONITH the non-
> leading head to enforce
> > failovers (and enable the specific PRI/SEC chip of the backplane).
> 
> The NexentaStor HA-Cluster plugin manages STONITH and reservations.
> I do not believe programming expanders or switches for 
> clustering is the best approach.
> It is better to let the higher layers manage this.
Makes sense.
 
Since I originally thought that only one path works at a given time,
it may be needed to somehow shutdown the competitor HBA/link ;)
 
> > I am not sure if this requirement also implies dual SAS data 
> connectors - pictures
> > of HCL HDDs all have one connector... 
> 
> These are dual ported.
Does this mean mecanically two 7-pin SATA data ports and a wide
power port, for a total of 3 connectors on the back of HDD, as well
as on the backplane sockets? Or does it mean something else?
 
Because I''ve looked up half a dozen of SuperMicro-supported drives
(bold SAS in the list for E2-series chassis), and in the online shops''
images they all have the standard 2 connectors (wide and 7-pin):
http://www.supermicro.com.tr/SAS-1-CompList.pdf
The HCL is rather small, and "other components may work but are
not supported by SuperMicro".
 
And to be more specific, do you know if Hitachi 7K3000 series SAS 
models HUS723020ALS640 (2Tb) or HUS723030ALS640 (3Tb)
are suitable for these boxes?
 
Does it make sense to keep the OS/swap on faster smaller drives like
a mirror of HUS156030VLS600 (300Gb SAS 15kRPM) - or is it
a waste of money? (And are they known to work in these boxes?)
 
> > Hint: Nexenta people seem to be good OEM friends with 
> Supermicro, so they 
> > might know ;)
> 
> Yes :-)
>  -- richard
 
Thanks! 
 
//Jim Klimov
 
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20110530/5f400fa4/attachment.html>

On Mon, May 30, 2011 at 1:35 PM, Jim Klimov <jim at cos.ru> wrote:
> Thanks, now I have someone to interrogate, who seems to have
> seen these boxes live - if you don''t mind ;)
>
> ----- Original Message -----
> From: Richard Elling <richard.elling at gmail.com>
> Date: Monday, May 30, 2011 22:04
>
> > We also commonly see the dual-expander backplanes.
> >
> > > According to the docs, each chip addresses all disks on its
> > backplane, and it seems
> > > implied (but not expressly stated) that either one chip and
> > path works, or another.
> >
> > For SAS targets, both paths work simultaneously.
>
> Does this mean that if the J0 uplinks of backplanes are connected
> to HBAs in two different servers, both of these servers can address
> individual disks (and the unit of failover is not a backplane but a disk
> after all)?
>
> And if both HBAs are in a single server, this doubles the SAS link
> throughput by having two paths - and can ZFS somehow balance
> among them?
>
> >
> > > So if your application can live with the unit of failover
> > being a bunch of 21 or 24 disks -
> > > that might be a way to go. However each head would only have
> > one connection to
> > > each backplane, and I''m not sure if you can STONITH the
non-
> > leading head to enforce
> > > failovers (and enable the specific PRI/SEC chip of the
backplane).
> >
> > The NexentaStor HA-Cluster plugin manages STONITH and reservations.
> > I do not believe programming expanders or switches for
> > clustering is the best approach.
> > It is better to let the higher layers manage this.
> Makes sense.
>
> Since I originally thought that only one path works at a given time,
> it may be needed to somehow shutdown the competitor HBA/link ;)
>
> > > I am not sure if this requirement also implies dual SAS data
> > connectors - pictures
> > > of HCL HDDs all have one connector...
> >
> > These are dual ported.
> Does this mean mecanically two 7-pin SATA data ports and a wide
> power port, for a total of 3 connectors on the back of HDD, as well
> as on the backplane sockets? Or does it mean something else?
>
> Because I''ve looked up half a dozen of SuperMicro-supported drives
> (bold SAS in the list for E2-series chassis), and in the online
shops''
> images they all have the standard 2 connectors (wide and 7-pin):
> http://www.supermicro.com.tr/SAS-1-CompList.pdf
> The HCL is rather small, and "other components may work but are
> not supported by SuperMicro".
>
> And to be more specific, do you know if Hitachi 7K3000 series SAS
> models HUS723020ALS640 (2Tb) or HUS723030ALS640 (3Tb)
> are suitable for these boxes?
>
> Does it make sense to keep the OS/swap on faster smaller drives like
> a mirror of HUS156030VLS600 (300Gb SAS 15kRPM) - or is it
> a waste of money? (And are they known to work in these boxes?)
>
> > > Hint: Nexenta people seem to be good OEM friends with
> > Supermicro, so they
> > > might know ;)
> >
> > Yes :-)
> >  -- richard
>
> Thanks!
>
> //Jim Klimov
>
>
SAS drives are SAS drives, they aren''t like SCSI.  There
aren''t 20 different
versions with different pinouts.

Multipathing is handled by mpxio.

--Tim
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20110530/e3edaf58/attachment-0001.html>

Tim Cook wrote:
 > SAS drives are SAS drives, they aren''t like SCSI.
 > There aren''t 20 different versions with different pinouts.

Uh-huh... Reading some more articles, I think I found the
answer to my question: the SAS connector seems to be
dual-sided (with conductive stripes on both sides of the
plastic) while SATA ports *seem* to be the same (for
backward compatibility) but only have conductive stripes
on one side. Something like that :)

And the SAS connector is notched so the SAS drives can''t
plug in to protocol-incompatible SATA-only controllers.

Also some articles stated that at one time there were
single-port SAS drives, so there are at least two SAS
connectors after all ;)

 > Multipathing is handled by mpxio.

So I configure MPxIO, then feed the "zpool create" device
names of multipathed aggregates, and hopefully failover
should work.

But can two paths work in parallel to double the bandwidth
for ZFS (to  single disk)? Or this depends on particular chips
and drivers?

Rationale: I''ve read a review article of an enterprise SSD
which was sold back in SAS1 times (3Gbit/s) but performed
at over 500Mbyte/s - and the reviewers used parallel MPIO
with an LSI 9211-4i card in order to have so much actual
bandwidth.

Thanks,
//Jim

On Mon, May 30, 2011 at 6:16 PM, Jim Klimov <jimklimov at cos.ru>
wrote:
> Also some articles stated that at one time there were
> single-port SAS drives, so there are at least two SAS
> connectors after all ;)
Nope, only one mechanical connector. A dual port cable can be used
with single- or dual-ported SAS device, or with SATA drives. A single
port cable can be used with a single- or dual-ported SAS device
(although it will only use one port) or with a SATA drive. A SATA
cable can be used with a SATA device.

-B

-- 
Brandon High : bhigh at freaks.com

On May 30, 2011, at 6:16 PM, Jim Klimov wrote:
> > Multipathing is handled by mpxio.
> 
> So I configure MPxIO, then feed the "zpool create" device
> names of multipathed aggregates, and hopefully failover
> should work.
Yes, it is that simple :-)
> But can two paths work in parallel to double the bandwidth
> for ZFS (to  single disk)? Or this depends on particular chips
> and drivers?
It depends more on the end-points than the stuff in the middle.
> Rationale: I''ve read a review article of an enterprise SSD
> which was sold back in SAS1 times (3Gbit/s) but performed
> at over 500Mbyte/s - and the reviewers used parallel MPIO
> with an LSI 9211-4i card in order to have so much actual
> bandwidth.
The highest I''ve seen to a single SAS disk is 780 MB/sec sustained.
Obviously, that wasn''t a HDD :-)
 -- richard

So, if I may, is this the correct summary of the answer to original 
question (on JBOD for a ZFS HA cluster):
 
==  SC847E26-RJBOD1 with dual-ported SAS drives are known
  to work in a failover HA storage scenario, allowing both servers
  (HBAs) access to each single SAS drive individually, so zpools
  can be configured from any disks regardless of which backplane
  they are connected to. HA Clusterware, such as NexentaStor
  HA-Cluster plugin should be used to ensure that only one head
  node actually uses a given disk drive in an imported ZFS pool.
== 
Is the indented statement correct? :)
 
Other questions:
 
What clusterware would be encouraged now for OpenIndiana
boxes?
 
Also, in case of clustered shared filesystems (like VMWare vmfs)
can these JBODs allow two different servers to access one drive 
simultaneously in a safe manner (do they do fencing, reservations 
and other SCSI magic)?
 
> > Following up on some of this forum''s discussions, I read the 
> manuals on SuperMicro''s
> > SC847E26-RJBOD1 this weekend. 
> 
> We see quite a few of these in the NexentaStor installed base. 
> The NexentaStor HA-Cluster plugin manages STONITH and reservations.
> I do not believe programming expanders or switches for 
> clustering is the best approach.
> It is better to let the higher layers manage this.
> The cost of a SATA disk + SATA/SAS interposer is about the same 
> as a native SAS
> drive. Native SAS makes a better solution.
>  
 
-- 

+============================================================+ 
|                                                            | 
| ?????? ???????,                                 Jim Klimov | 
| ??????????? ????????                                   CTO | 
| ??? "??? ? ??"                                  JSC COS&HT | 
|                                                            | 
| +7-903-7705859 (cellular)          mailto:jimklimov at cos.ru | 
|                        CC:admin at cos.ru,jimklimov at gmail.com | 
+============================================================+ 
| ()  ascii ribbon campaign - against html mail              | 
| /\                        - against microsoft attachments  | 
+============================================================+ 
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20110531/84f36cf0/attachment.html>

Thomas,

You can consider the DataON DNS-1600(4U 24 3.5" Bay 6Gb/s SAS JBOD). It is
perfect for ZFS storage as the alternative of J4400.       
http://dataonstorage.com/dns-1600

And we recommend you to use native SAS HD like Seagate Constellation ES 2TB
SAS to connect 2 hosts for fail-over cluster. 

The following is setup diagram of HA failover cluster with Nexenta. Same
configuration can applied to Solaris, OpenSolaris and OpenIndiana  
http://dataonstorage.com/nexentaha

We also have DSM(Disk Shelf Management Tool) available for Solaris 10 and
Nexenta to help identify fail disk and JBOD. You can also check the status
of all FRU  
http://dataonstorage.com/dsm

FYI, we have reseller in Germany. If you need the additional info, you can
let me know!  

Rocky


-----Original Message-----
From: zfs-discuss-bounces at opensolaris.org
[mailto:zfs-discuss-bounces at opensolaris.org] On Behalf Of Thomas Nau
Sent: Sunday, May 29, 2011 11:07 PM
To: zfs-discuss at opensolaris.org
Subject: [zfs-discuss] JBOD recommendation for ZFS usage

Dear all

Sorry if it''s kind of off-topic for the list but after talking
to lots of vendors I''m running out of ideas...

We are looking for JBOD systems which

(1) hold 20+ 3.3" SATA drives

(2) are rack mountable

(3) have all the nive hot-swap stuff

(4) allow 2 hosts to connect via SAS (4+ lines per host) and see
    all available drives as disks, no RAID volume.
    In a perfect world both hosts would connect each using
    two independent SAS connectors


The box will be used in a ZFS Solaris/based fileserver in a
fail-over cluster setup. Only one host will access a drive
at any given time.

It seems that a lot of vendors offer JBODs but so far I haven''t found
one in Germany which handles (4).

Any hints?
_______________________________________________
zfs-discuss mailing list
zfs-discuss at opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss