zfs discuss - Jul 2009 - Understanding SAS/SATA Backplanes and Connectivity

Hello All,

I''m just starting to think about building some mass-storage arrays and
am looking to better understand some of the components involved.

For example, the Supermicro SC826 series of systems is available with  
three backplanes:

1. SAS / SATA Expander Backplane with single LSI SASX28 Expander Chip
2. SAS / SATA Expander Backplane with dual LSI SASX28 Expander Chips
3. SAS / SATA Direct Attached Backplane

Assuming I am using this an external array, connected to a server via  
SAS, how do these fit into my topology? Expander, dual-expanders and  
no expander? Huh?

Thanks for pointing to relevant documentation.

A.


--
Adam Sherman
CTO, Versature Corp.
Tel: +1.877.498.3772 x113

On Thu, Jul 16, 2009 at 17:02, Adam Sherman<asherman at versature.com>
wrote:
> Hello All,
>
> I''m just starting to think about building some mass-storage arrays
and am
> looking to better understand some of the components involved.
>
> For example, the Supermicro SC826 series of systems is available with three
> backplanes:
>
> 1. SAS / SATA Expander Backplane with single LSI SASX28 Expander Chip
> 2. SAS / SATA Expander Backplane with dual LSI SASX28 Expander Chips
> 3. SAS / SATA Direct Attached Backplane
>
> Assuming I am using this an external array, connected to a server via SAS,
> how do these fit into my topology? Expander, dual-expanders and no
expander?
> Huh?
The "direct attached" backplane is right out.  This means that each
drive has its own individual sata port, meaning you''d need three SAS
wide ports just to connect the drives.

The single-expander version has one LSI SAS expander, which connects
to all the drives and has two "upstream" ports.  This means you plug
in one or two servers directly, and they can both see all the disks.
I''ve only tested this with one-server configurations.  It also has one
"downstream" port which you could use to daisy-chain more expanders
(i.e., more 826/846 cases) onto the same server.

We have a SC846E1 at work; it''s the 24-disk, 4u version of the 826e1.
It''s working quite nicely as a SATA JBOD enclosure.  We''ll
probably be
buying another in the coming year to have more capacity.

The dual-expander version has two LSI SAS expanders.  You need
dual-port SAS drives (not SATA).  This lets you have two paths all the
way to each drive; even if one expander fails (this seems pretty
unlikely to me, but if you''re shooting for many nines it''s
worth
considering) you still have access to the disks.
> Thanks for pointing to relevant documentation.
The manual for the Supermicro cases [1, 2] does a pretty good job IMO
explaining the different options.  See page D-14 and on in the 826
manual, or page D-11 and on in the 846 manual.

Will

[1]: http://supermicro.com/manuals/chassis/2U/SC826.pdf
[2]: http://supermicro.com/manuals/chassis/tower/SC846.pdf

On 16-Jul-09, at 18:01 , Will Murnane wrote:
> The "direct attached" backplane is right out.  This means that
each
> drive has its own individual sata port, meaning you''d need three
SAS
> wide ports just to connect the drives.
>
> The single-expander version has one LSI SAS expander, which connects
> to all the drives and has two "upstream" ports.  This means you
plug
> in one or two servers directly, and they can both see all the disks.
> I''ve only tested this with one-server configurations.  It also has
one
> "downstream" port which you could use to daisy-chain more
expanders
> (i.e., more 826/846 cases) onto the same server.
That makes things a heck of a lot clearer, thank you very much for  
taking the time to explain!

Ever seen/read about anyone use this kind of setup for HA clustering?  
I''m getting ideas about Open HA/Solaris Cluster on top of this setup  
with two systems connecting, that would rock!
> We have a SC846E1 at work; it''s the 24-disk, 4u version of the
826e1.
> It''s working quite nicely as a SATA JBOD enclosure. 
We''ll probably be
> buying another in the coming year to have more capacity.
Good to hear. What HBA(s) are you using against it?
>> Thanks for pointing to relevant documentation.
> The manual for the Supermicro cases [1, 2] does a pretty good job IMO
> explaining the different options.  See page D-14 and on in the 826
> manual, or page D-11 and on in the 846 manual.

I''ll read though that, thanks for the detailed pointers.

A.

--
Adam Sherman
CTO, Versature Corp.
Tel: +1.877.498.3772 x113

Another thought in the same vein, I notice many of these systems  
support "SES-2" for management. Does this do anything useful under  
Solaris?

Sorry for these questions, I seem to be having a tough time locating  
relevant information on the web.

Thanks,

A.

--
Adam Sherman
CTO, Versature Corp.
Tel: +1.877.498.3772 x113

> 
> > We have a SC846E1 at work; it''s the 24-disk, 4u
> version of the 826e1.
> > It''s working quite nicely as a SATA JBOD enclosure.
>  We''ll probably be
> buying another in the coming year to have more
>  capacity.
> Good to hear. What HBA(s) are you using against it?
> 
I''ve got one too and it works great. I use the LSI SAS 3442e which also
gives you an external SAS port. You don''t need a fancy HBA with onboard
RAID. Configure to IT mode.
-- 
This message posted from opensolaris.org

On 16-Jul-09, at 20:52 , James C. McPherson wrote:
>> Another thought in the same vein, I notice many of these systems
>> support "SES-2" for management. Does this do anything useful
under
>> Solaris?
>
> We''ve got some integration between FMA and SES devices which
> allows us to to some management tasks.
So that would allow FMA to detect SATA disk failures then?
> libtopo, libscsi and libses are the main methods of getting
> that information out. For an example outside FMA, you could
> have a look into the ses/sgen plugin from pluggable fwflash.
>
> Is there anything you''re specifically interested in wrt management
> uses of SES?
I''m really just exploring. Where can I read about how FMA is going to  
help with failures in my setup?

Thanks,

A.

--
Adam Sherman
CTO, Versature Corp.
Tel: +1.877.498.3772 x113

On Thu, 16 Jul 2009 20:26:17 -0400
Adam Sherman <asherman at versature.com> wrote:
> Another thought in the same vein, I notice many of these systems  
> support "SES-2" for management. Does this do anything useful
under
> Solaris?
We''ve got some integration between FMA and SES devices which
allows us to to some management tasks.

libtopo, libscsi and libses are the main methods of getting
that information out. For an example outside FMA, you could
have a look into the ses/sgen plugin from pluggable fwflash.

Is there anything you''re specifically interested in wrt management
uses of SES?

thanks,
James C. McPherson
--
Senior Kernel Software Engineer, Solaris
Sun Microsystems
http://blogs.sun.com/jmcp	http://www.jmcp.homeunix.com/blog
Kernel Conference Australia - http://au.sun.com/sunnews/events/2009/kernel

On 16-Jul-09, at 18:01 , Will Murnane wrote:
> We have a SC846E1 at work; it''s the 24-disk, 4u version of the
826e1.
> It''s working quite nicely as a SATA JBOD enclosure. 
We''ll probably be
> buying another in the coming year to have more capacity.

I should also ask: any other solutions I should have a look at to get  
 >=12 SATA disks externally attached to my systems?

Thanks!

A.

--
Adam Sherman
CTO, Versature Corp.
Tel: +1.877.498.3772 x113

On Thu, 16 Jul 2009, Adam Sherman wrote:
>
> I should also ask: any other solutions I should have a look at to get
>=12
> SATA disks externally attached to my systems?
Depending on how much failure resiliancy you want and how you plan to 
configure your pool, you may be better off with two independent disk 
trays with 12 disks each.  For example, if you were to use mirrors, 
you could split the mirrors across the disk trays.  If one tray fails, 
then your system still works.

If you are planning to use raidz1 or raidz2 then there is likely no 
benefit to going with two smaller trays.

Bob
--
Bob Friesenhahn
bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/

>> We have a SC846E1 at work; it''s the 24-disk, 4u version of the
826e1.
 >> It''s working quite nicely as a SATA JBOD enclosure.

 > use the LSI SAS 3442e which also gives you an external SAS port.

I''m confused, I though expanders only worked with SAS disk, and SATA
disks
took an entire SAS port. could someone post an output showing more than 4 SATA
drives across one InfiniBand cable (4 SAS ports)

2 % cfgadm | grep sata
sata1/0::dsk/c9t0d0            cd/dvd       connected    configured   ok
sata1/1::dsk/c9t1d0            disk         connected    configured   ok
sata1/2::dsk/c9t2d0            disk         connected    configured   ok
sata1/3                        sata-port    empty        unconfigured ok
sata1/4::dsk/c9t4d0            disk         connected    configured   ok
sata1/5                        sata-port    empty        unconfigured ok
sata2/0::dsk/c7t0d0            disk         connected    configured   ok
sata2/1::dsk/c7t1d0            disk         connected    configured   ok
sata2/2::dsk/c7t2d0            disk         connected    configured   ok
sata2/3                        sata-port    empty        unconfigured ok
sata2/4::dsk/c7t4d0            disk         connected    configured   ok
sata2/5::dsk/c7t5d0            disk         connected    configured   ok
sata2/6                        sata-port    empty        unconfigured ok
sata2/7                        sata-port    empty        unconfigured ok
sata3/0::dsk/c8t0d0            disk         connected    configured   ok
sata3/1::dsk/c8t1d0            disk         connected    configured   ok
sata3/2::dsk/c8t2d0            disk         connected    configured   ok
sata3/3                        sata-port    empty        unconfigured ok
sata3/4::dsk/c8t4d0            disk         connected    configured   ok
sata3/5::dsk/c8t5d0            disk         connected    configured   ok
sata3/6                        sata-port    empty        unconfigured ok
sata3/7                        sata-port    empty        unconfigured ok

On Thu, Jul 16, 2009 at 20:20, Adam Sherman<asherman at versature.com>
wrote:
> Ever seen/read about anyone use this kind of setup for HA clustering?
I''m
> getting ideas about Open HA/Solaris Cluster on top of this setup with two
> systems connecting, that would rock!
It''s possible that this would work with homogeneous hardware, but I
tried with another LSI-based expander and SATA disks, and had no luck.
 Perhaps SAS is necessary?
>> We have a SC846E1 at work; it''s the 24-disk, 4u version of the
826e1.
>> It''s working quite nicely as a SATA JBOD enclosure.
?We''ll probably be
>> buying another in the coming year to have more capacity.
>
> Good to hear. What HBA(s) are you using against it?
LSI 3442E-R.  It''s connected through a Supermicro cable, CBL-0168L, so
it can be attached via an external cable.  There''s a card needed,
CSE-PTJBOD-CB1, that allows the case to run without a motherboard in
it.  There''s no monitoring for the power supplies, but I built one for
it; I can provide schematics and suggested part numbers if you''re
interested.
>> We have a SC846E1 at work; it''s the 24-disk, 4u version of the
826e1.
>> It''s working quite nicely as a SATA JBOD enclosure. 
We''ll probably be
>> buying another in the coming year to have more capacity.
>
>
> I should also ask: any other solutions I should have a look at to get
>=12
> SATA disks externally attached to my systems?
This was the best solution we found for the money.  The 826 is about
$750, while the 846 is $1100 shipped (wiredzone.com).  Per disk, the
846 is almost $20 cheaper.  If you only care for 12 disks, then one
might as well not spend the extra money, but if there''s potential for
expansion it''s a wise investment.

Will

On Thu, Jul 16, 2009 at 21:16, Rob Logan<Rob at logan.com>
wrote:
> I''m confused, I though expanders only worked with SAS disk, and
SATA disks
> took an entire SAS port. could someone post an output showing more than 4
> SATA
> drives across one InfiniBand cable (4 SAS ports)
>
> 2 % cfgadm | grep sata
> sata1/0::dsk/c9t0d0 ? ? ? ? ? ?cd/dvd ? ? ? connected ? ?configured ? ok
> sata1/1::dsk/c9t1d0 ? ? ? ? ? ?disk ? ? ? ? connected ? ?configured ? ok
> sata1/2::dsk/c9t2d0 ? ? ? ? ? ?disk ? ? ? ? connected ? ?configured ? ok
> sata1/3 ? ? ? ? ? ? ? ? ? ? ? ?sata-port ? ?empty ? ? ? ?unconfigured ok
> sata1/4::dsk/c9t4d0 ? ? ? ? ? ?disk ? ? ? ? connected ? ?configured ? ok
> sata1/5 ? ? ? ? ? ? ? ? ? ? ? ?sata-port ? ?empty ? ? ? ?unconfigured ok
> sata2/0::dsk/c7t0d0 ? ? ? ? ? ?disk ? ? ? ? connected ? ?configured ? ok
> sata2/1::dsk/c7t1d0 ? ? ? ? ? ?disk ? ? ? ? connected ? ?configured ? ok
> sata2/2::dsk/c7t2d0 ? ? ? ? ? ?disk ? ? ? ? connected ? ?configured ? ok
> sata2/3 ? ? ? ? ? ? ? ? ? ? ? ?sata-port ? ?empty ? ? ? ?unconfigured ok
> sata2/4::dsk/c7t4d0 ? ? ? ? ? ?disk ? ? ? ? connected ? ?configured ? ok
> sata2/5::dsk/c7t5d0 ? ? ? ? ? ?disk ? ? ? ? connected ? ?configured ? ok
> sata2/6 ? ? ? ? ? ? ? ? ? ? ? ?sata-port ? ?empty ? ? ? ?unconfigured ok
> sata2/7 ? ? ? ? ? ? ? ? ? ? ? ?sata-port ? ?empty ? ? ? ?unconfigured ok
> sata3/0::dsk/c8t0d0 ? ? ? ? ? ?disk ? ? ? ? connected ? ?configured ? ok
> sata3/1::dsk/c8t1d0 ? ? ? ? ? ?disk ? ? ? ? connected ? ?configured ? ok
> sata3/2::dsk/c8t2d0 ? ? ? ? ? ?disk ? ? ? ? connected ? ?configured ? ok
> sata3/3 ? ? ? ? ? ? ? ? ? ? ? ?sata-port ? ?empty ? ? ? ?unconfigured ok
> sata3/4::dsk/c8t4d0 ? ? ? ? ? ?disk ? ? ? ? connected ? ?configured ? ok
> sata3/5::dsk/c8t5d0 ? ? ? ? ? ?disk ? ? ? ? connected ? ?configured ? ok
> sata3/6 ? ? ? ? ? ? ? ? ? ? ? ?sata-port ? ?empty ? ? ? ?unconfigured ok
> sata3/7 ? ? ? ? ? ? ? ? ? ? ? ?sata-port ? ?empty ? ? ? ?unconfigured ok
Here''s the relevant part of cfgadm -al on our machine.  The disks are
all sata.

c4                             scsi-bus     connected    configured   unknown
c4::dsk/c4t15d0                disk         connected    configured   unknown
c4::dsk/c4t17d0                disk         connected    configured   unknown
c4::dsk/c4t18d0                disk         connected    configured   unknown
c4::dsk/c4t19d0                disk         connected    configured   unknown
c4::dsk/c4t20d0                disk         connected    configured   unknown
c4::dsk/c4t21d0                disk         connected    configured   unknown
c4::dsk/c4t22d0                disk         connected    configured   unknown
c4::dsk/c4t23d0                disk         connected    configured   unknown
c4::dsk/c4t24d0                disk         connected    configured   unknown
c4::dsk/c4t25d0                disk         connected    configured   unknown
c4::dsk/c4t26d0                disk         connected    configured   unknown
c4::dsk/c4t27d0                disk         connected    configured   unknown
c4::dsk/c4t28d0                disk         connected    configured   unknown
c4::dsk/c4t29d0                disk         connected    configured   unknown
c4::dsk/c4t30d0                disk         connected    configured   unknown
c4::dsk/c4t31d0                disk         connected    configured   unknown
c4::dsk/c4t32d0                disk         connected    configured   unknown
c4::dsk/c4t33d0                disk         connected    configured   unknown
c4::es/ses0                    ESI          connected    configured   unknown

Will

> c4                             scsi-bus     connected    configured  
unknown
 > c4::dsk/c4t15d0                disk         connected    configured  
unknown
  :
 > c4::dsk/c4t33d0                disk         connected    configured  
unknown
 > c4::es/ses0                    ESI          connected    configured  
unknown

thanks! so SATA disks show up JBOD in IT mode.. Is there some magic that
load balances the 4 SAS ports as this shows up as one "scsi-bus"?

On 16-Jul-09, at 21:17 , Will Murnane wrote:
>> Good to hear. What HBA(s) are you using against it?
> LSI 3442E-R.  It''s connected through a Supermicro cable,
CBL-0168L, so
> it can be attached via an external cable.

I''m looking at the LSI SAS3801X because it seems to be what Sun OEMs  
for my X4100s:

http://sunsolve.sun.com/handbook_private/validateUser.do?target=Devices/SCSI/SCSI_PCIX_SAS_SATA_HBA

$280 or so, looks like. Might be overkill for me though.

A.

--
Adam Sherman
CTO, Versature Corp.
Tel: +1.877.498.3772 x113

On Thu, Jul 16, 2009 at 21:35, Adam Sherman<asherman at versature.com>
wrote:
> I''m looking at the LSI SAS3801X because it seems to be what Sun
OEMs for my
> X4100s:
If you''re given the choice (i.e., you have the M2 revision), PCI
Express is probably the bus to go with.  It''s basically the same card,
but on a faster bus.  But there''s nothing wrong with the PCI-X
version.
http://www.lsi.com/storage_home/products_home/host_bus_adapters/sas_hbas/lsisas3801e/index.html
> $280 or so, looks like. Might be overkill for me though.
The 3442X-R is a little cheaper: $205 from Provantage.
http://www.provantage.com/lsi-logic-lsi00164~7LSIG06K.htm

Will

On Thu, Jul 16, 2009 at 21:30, Rob Logan<Rob at logan.com>
wrote:
>> c4 ? ? ? ? ? ? ? ? ? ? ? ? ? ? scsi-bus ? ? connected ? ?configured
>> unknown
>> c4::dsk/c4t15d0 ? ? ? ? ? ? ? ?disk ? ? ? ? connected ? ?configured
>> unknown
> ?:
>> c4::dsk/c4t33d0 ? ? ? ? ? ? ? ?disk ? ? ? ? connected ? ?configured
>> unknown
>> c4::es/ses0 ? ? ? ? ? ? ? ? ? ?ESI ? ? ? ? ?connected ? ?configured
>> unknown
>
> thanks! so SATA disks show up JBOD in IT mode.. Is there some magic that
> load balances the 4 SAS ports as this shows up as one "scsi-bus"?
Hypothetically, yes.  In practical terms, though, I''ve seen more than
300 MB/s of I/O over it:
                capacity     operations    bandwidth
pool          used  avail   read  write   read  write
-----------  -----  -----  -----  -----  -----  -----
data1        1.06T  1.21T      1  1.61K  2.49K   200M
  mirror      460G   236G      0    522  1.15K  63.8M
    c4t18d0      -      -      0    518  6.38K  63.6M
    c4t21d0      -      -      0    518  12.8K  63.8M
  mirror      467G   229G      0    533    306  64.8M
    c4t23d0      -      -      0    523  6.38K  64.3M
    c4t25d0      -      -      0    529      0  65.0M
  mirror      153G   775G      0    597  1.05K  71.8M
    c4t20d0      -      -      0    589  12.8K  72.5M
    c4t22d0      -      -      0    584      0  71.8M
-----------  -----  -----  -----  -----  -----  -----

Note that the pool is only doing 200 MB/s, but the individual devices
are doing a total of 400 MB/s.  It''s not possible to put more than 300
MB/s into or out of a single device, so there''s no "link
aggregation"
to worry about.

Will

On 17-Jul-09, at 1:45 , Will Murnane wrote:
>> I''m looking at the LSI SAS3801X because it seems to be what
Sun
>> OEMs for my
>> X4100s:
> If you''re given the choice (i.e., you have the M2 revision), PCI
> Express is probably the bus to go with.  It''s basically the same
card,
> but on a faster bus.  But there''s nothing wrong with the PCI-X
> version.
I have a stack of the original X4100s.
>> $280 or so, looks like. Might be overkill for me though.
> The 3442X-R is a little cheaper: $205 from Provantage.
> http://www.provantage.com/lsi-logic-lsi00164~7LSIG06K.htm

I don''t get it, why is that one cheaper than:

http://www.provantage.com/lsi-logic-lsi00124~7LSIG03W.htm

Just newer?

A.

--
Adam Sherman
CTO, Versature Corp.
Tel: +1.877.498.3772 x113

On Fri, Jul 17, 2009 at 11:43, Adam Sherman<asherman at versature.com>
wrote:
> On 17-Jul-09, at 1:45 , Will Murnane wrote:
>>>
>>> I''m looking at the LSI SAS3801X because it seems to be
what Sun OEMs for
>>> my
>>> X4100s:
>>
>> If you''re given the choice (i.e., you have the M2 revision),
PCI
>> Express is probably the bus to go with. ?It''s basically the
same card,
>> but on a faster bus. ?But there''s nothing wrong with the PCI-X
>> version.
>
> I have a stack of the original X4100s.
Ah, okay.  PCI-X it is.
>>> $280 or so, looks like. Might be overkill for me though.
>>
>> The 3442X-R is a little cheaper: $205 from Provantage.
>> http://www.provantage.com/lsi-logic-lsi00164~7LSIG06K.htm
>
>
> I don''t get it, why is that one cheaper than:
>
> http://www.provantage.com/lsi-logic-lsi00124~7LSIG03W.htm
When I understand LSI''s pricing, I will let you know.  It''s
got a
different connector (8470 instead of 8088) on the outside, and one
internal port.

On the other hand, that card only gets you one external port.  If you
plan to have more than one JBOD per host, the 3801 makes more sense.

Will

>>>>> "rl" == Rob Logan <Rob at Logan.com>
writes:
    rl> Is there some magic that load balances the 4 SAS ports as this
    rl> shows up as one "scsi-bus"?

The LSI card is not SATA framework.  I''ve the impression drive
enumeration and topology is handled by the proprietary firmware on the
card, so it''s likely there isn''t any explicit support for SAS
expanders inside solaris''s binary mpt driver at all.  If you have x86
I think you can explore topology using the bootup Blue Screens of
Setup, but I don''t have anything with SAS expander to test it.

I think the SAS standard itself has a concept of ``wide
ports'''' like
infiniband or PCIe, so I would speculate the 4 pairs are treated as
lanes rather than ports.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 304 bytes
Desc: not available
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20090717/6bad96bb/attachment.bin>

Also in reply to the previous email by Will.

Can anyone shed more light on the combination lsi sas hba , the lsisasx36
expander chip (or it''s relatives) and sata disks.
I''m investigating a migration from discrete channels (like in the
thumper) to a multiplexed solution via a sas expander.
I''m aware that there''s a penalty but for my workload
that''s acceptable. Only when a wide sas port (4 links) is used, which
should result in a theoretical throughput of 1.2GB/s. But Will reported only
300MB/s, which corresponds to a single link.
>From the limited information available regarding the lsi expanders, I found
that support for the SataTunnelProtocol is present. From the expander to the hba
only sas packets will go and therefore I see obstacle for not using a wide port.
Can this be a cable problem or a hba problem or firmware?
Perhaps readers would like to share their experience with a similar setup and
what setup they used. What''s the throughput when only using sas disks?

Regards,

Frederik
-- 
This message posted from opensolaris.org

On Mon, Jul 20, 2009 at 05:44, F. Wessels<no-reply at opensolaris.org>
wrote:
> Also in reply to the previous email by Will.
>
> Can anyone shed more light on the combination lsi sas hba , the lsisasx36
expander chip (or it''s relatives) and sata disks.
> I''m investigating a migration from discrete channels (like in the
thumper) to a multiplexed solution via a sas expander.
> I''m aware that there''s a penalty but for my workload
that''s acceptable. Only when a wide sas port (4 links) is used, which
should result in a theoretical throughput of 1.2GB/s. But Will reported only
300MB/s, which corresponds to a single link.
That''s not what I intended to report.  I saw 400 MB/s of write I/O to
my six-disk mirrored zpool, and a little over 300 MB/s read.  However,
the write I/O is reported as half that because it''s a mirrored setup,
so the userspace app writes 200 MB in a second and the disks have to
write that twice in the same period.

In short, I have seen more than 300 MB/s over a SAS expander.  I just
don''t have enough non-production disks to show much more than that.

Will

F. Wessels wrote:
> Also in reply to the previous email by Will.
>
> Can anyone shed more light on the combination lsi sas hba , the lsisasx36
expander chip (or it''s relatives) and sata disks.
> I''m investigating a migration from discrete channels (like in the
thumper) to a multiplexed solution via a sas expander.
>   
You are describing a Sun J4500, a thumper with SAS expanders instead of 
a computer.
http://www.sun.com/storage/disk_systems/expansion/4500/
 -- richard

So to wrap it up. According to Will, a supermicro chassis using a single lsi
expander connected to sata disks can utilize the wide sas port between hba and
the chassis. (like a J4500 Richard mentioned. How much I like these systems
(thumper etc), they''re way out of my budget.) Will did see more
throughput than a single link could handle. But due to insufficient disks no
more i/o could be generated to better demonstrate the availability off the wide
sas port.
I assume the J4500 is using dual expanders. Richard, can you confirm that the
J4500 is using an active/active sata mux per drive to allow failover between
expanders? Aside from this can you also confirm that a wide sas link between a
hba and a single expander can be fully utilized when using sata drives? To
rephrase, utilization is independent of the disk type (sas or sata)
All disclaimers apply. 

Regards Frederik
-- 
This message posted from opensolaris.org

On 21-Jul-09, at 9:25 , F. Wessels wrote:
> So to wrap it up. According to Will, a supermicro chassis using a  
> single lsi expander connected to sata disks can utilize the wide sas  
> port between hba and the chassis. (like a J4500 Richard mentioned.  
> How much I like these systems (thumper etc), they''re way out of my
> budget.) Will did see more throughput than a single link could  
> handle. But due to insufficient disks no more i/o could be generated  
> to better demonstrate the availability off the wide sas port.
> I assume the J4500 is using dual expanders. Richard, can you confirm  
> that the J4500 is using an active/active sata mux per drive to allow  
> failover between expanders? Aside from this can you also confirm  
> that a wide sas link between a hba and a single expander can be  
> fully utilized when using sata drives? To rephrase, utilization is  
> independent of the disk type (sas or sata)

The J series JBODs aren''t overly expensive, it''s the darn
drives for
them that break the budget.

And, on that subject, is there truly a difference between Seagate''s  
line-up of 7200 RPM drives? They seem to now have a bunch:

Model	Bus		Capacity	SKU		Cost (USD)
7200.12	SATA 3.0Gb/s	1TB		ST31000528AS	$90
ES.2	SATA 3.0Gb/s	1TB		ST31000340NS	$158
ES.2	SAS 3Gb/s	1TB		ST31000640SS	$215

Other manufacturers seem to have similar lineups. Is the difference  
going to matter to me when putting a mess of them into a SAS JBOD with  
an expander?

Thanks for everyone''s great feedback, this thread has been highly  
educating.

A.

--
Adam Sherman
CTO, Versature Corp.
Tel: +1.877.498.3772 x113

On Jul 21, 2009, at 6:25 AM, F. Wessels wrote:
> So to wrap it up. According to Will, a supermicro chassis using a  
> single lsi expander connected to sata disks can utilize the wide sas  
> port between hba and the chassis. (like a J4500 Richard mentioned.  
> How much I like these systems (thumper etc), they''re way out of my
> budget.) Will did see more throughput than a single link could  
> handle. But due to insufficient disks no more i/o could be generated  
> to better demonstrate the availability off the wide sas port.
> I assume the J4500 is using dual expanders. Richard, can you confirm  
> that the J4500 is using an active/active sata mux per drive to allow  
> failover between expanders?
Yes, this is seen in Figure 1-1 of the Sun Storage J4500 Array
System Overview manual.  Look for the Active-Active MUX (AAMUX).
http://dlc.sun.com/pdf/820-3163-11/820-3163-11.pdf

> Aside from this can you also confirm that a wide sas link between a  
> hba and a single expander can be fully utilized when using sata  
> drives? To rephrase, utilization is independent of the disk type  
> (sas or sata)
It is a simple architecture.  Think of expanders (and AAMUXes for this  
case)
as switches, and it might seem more familiar. For a dual-port SAS disk,
that Sun only sells in 2.5" form factor (IIRC), you don''t need the
AAMUX.
For example, the J4400 is basically the same architecture as the J4500,
except that it accepts SAS and SATA disks. Since there is a mix, it is  
not
feasible to put the AAMUX on the system board, so they are placed in
the disk carrier. If you look at the exploded part numbers, you''ll see
a
SATA interposer card (PN 570-1181).
http://sunsolve.sun.com/handbook_pub/validateUser.do?target=Systems/J4400/components
  -- richard


> All disclaimers apply.
>
> Regards Frederik
> -- 
> This message posted from opensolaris.org
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

asherman at versature.com said:
> And, on that subject, is there truly a difference between
Seagate''s line-up
> of 7200 RPM drives? They seem to now have a bunch: 
> . . .
> Other manufacturers seem to have similar lineups. Is the difference going
to
> matter to me when putting a mess of them into a SAS JBOD with an expander? 
There are differences even within the lineup of Sun-supplied SATA drives.
Some support multipathing, and some do not.  Even some that are reported
(in Sun docs) to support it, do not.

  http://opensolaris.org/jive/thread.jspa?threadID=107049&tstart=30
  http://opensolaris.org/jive/thread.jspa?threadID=107057&tstart=15

Regards,

Marion

On Fri, 17 Jul 2009 14:16:32 -0400
Miles Nordin <carton at Ivy.NET> wrote:
> >>>>> "rl" == Rob Logan <Rob at Logan.com>
writes:
> 
>     rl> Is there some magic that load balances the 4 SAS ports as this
>     rl> shows up as one "scsi-bus"?
> 
> The LSI card is not SATA framework.  I''ve the impression drive
> enumeration and topology is handled by the proprietary firmware on the
> card, so it''s likely there isn''t any explicit support for
SAS
> expanders inside solaris''s binary mpt driver at all. 
There kinda is - mpt(7d) detects SAS expanders as SCSI Enclosure
Services devices (which is what the spec says), and passes the
enumeration off to ses(7d) or sgen(7d), depending on what you''ve
got as a device alias for scsiclass,0d.

We also (in NV since build 81, S10 Update 6) detect and correcly
handle Serial Management Protocol instances, which SAS expanders
hook into. The SAS HBA chip passes SMP frames to and from the 
expander.

> If you have x86
> I think you can explore topology using the bootup Blue Screens of
> Setup, but I don''t have anything with SAS expander to test it.
Yes, that''s correct, the bluescreenofsetup allows you to do some
minimal viewing of the config.
> I think the SAS standard itself has a concept of ``wide
ports'''' like
> infiniband or PCIe, so I would speculate the 4 pairs are treated as
> lanes rather than ports.
mpt(7d) bundles the phys and only shows one controller for internal
and one controller for external connections - on a physical hba basis.

cheers,
James C. McPherson
--
Senior Kernel Software Engineer, Solaris
Sun Microsystems
http://blogs.sun.com/jmcp	http://www.jmcp.homeunix.com/blog
Kernel Conference Australia - http://au.sun.com/sunnews/events/2009/kernel

Will,

sorry for picking an old thread, but you mentioned a psu monitor to supplement
the CSE-PTJBOD-CB1.
I have two of these and am interested in your design.
Oddly, the LSI backplane chipset supports 2 x i2c busses that Supermicro
didn''t make use of for monitoring the psu''s.

Mark.
-- 
This message posted from opensolaris.org

On Wed, Jan 6, 2010 at 20:41, Mark Bennett <mark.bennett at public.co.nz>
wrote:
> Will,
>
> sorry for picking an old thread,
That''s okay---I liked this thread ;)
> but you mentioned a psu monitor to supplement the CSE-PTJBOD-CB1.
> I have two of these and am interested in your design.
> Oddly, the LSI backplane chipset supports 2 x i2c busses that Supermicro
didn''t make use of for monitoring the psu''s.
Okay, here goes.  Basically what I found is that the 4-pin cable which
comes out of the power supply gives 5 volts (at a very low amperage)
when both power supplies are present, and nothing otherwise (on either
the leftmost or two rightmost pins; I don''t remember and it depends on
which way you define as up anyways.  Dig out the voltmeter and plug
and unplug power supplies until you find it).  So the idea is to use
that tiny supply to turn on a transistor, and send high-current (well,
relatively) 5v from a normal Molex connector to operate a relay.  This
relay is connected in series with a serial cable, and the machine on
the other end runs a small daemon which writes some stuff to the
serial port (in this case, a multi-port serial interface left over
from our SPARC boxes) and waits to see if it gets stuff back.  The
source code that I use (which you should probably rewrite, now that I
look at it) is attached.  The whole thing is a hack, but I can say it
works fine in production.

I can also draw a simple schematic and recommend parts from Radio
Shack if you would like, but I don''t have access to that machine until
next week and I don''t remember exactly how it went.  Ask your friendly
neighborhood EE and you should be able to get there quickly.

Will
-------------- next part --------------
A non-text attachment was scrubbed...
Name: serial-monitor.c
Type: text/x-csrc
Size: 2049 bytes
Desc: not available
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100106/aa6560b0/attachment.bin>

Thanks Will,

I thought it might be an i2c interface port to the psu, but obviously much
simpler.
I''ll probably use a small picaxe micro, since I have a few here &
have used them before.
I used them to ''translate'' the replacement fans clock pulse to
what the monitoring circuit needed in a few V240 PSU''s.
Much cheaper than replacing the whole psu due to poor fan lifespan.

Mark.
-- 
This message posted from opensolaris.org