zfs discuss - Jan 2009 - Looking for new SATA/SAS HBA; JBOD is not always JBOD

Why do they throw these fancy RAID-controllers at us when we have plenty CPU
force to do zfs mirror and even raidz1 and raidz2?

I have 12 SATA disks and would like to prepare to add 12 new internal SATA disks
to my home server.
The cabinet (Lian Li Modular Cube
http://www.microplex.no/aspx/produkt/prdinfovnet.aspx?plid=33415#) takes 24
3.5" or (insane 72 2.5") in front with suitable HDD frames/backplanes.

I have used two Supermicro AOC-SAT2-MV8 PCI-X true 8-port JBOD SATA HBA and
I''m quite happy with them; They are cheap and works in plain PCI
(limited by the PCI-speed aof course, but for my 12-disk home NAS it''s
good enough)
The only problem is that no mainstream motherboards come with plenty PCI-X
slots. My MB has none and PCI/PCI-X is not future proof. So adding a third
AOC-SAT2-MV8 seems awkward

So I''m looking for a PCIe HBA. The Adaptec 31605  or 52445  seems
tempting as I could reuse the old AOC-SAT2-MV8 elsewhere. Or I could add an
8-port Adaptec 3805  or an LSI SAS3081E-R.

What scares me off (aside the price for these) is that I''ve bumped into
both Sun OEM Adaptec and HP OEM Megaraid at work and none of them would do true
JBOD; No disks showed up in format. I had to configure 1-disk volumes in BIOS to
simulate JBOD, and from what I understand this writes this config to disk
destroying any existing data and partitioning, and what shoes up in format is
*not* the disks (i.e. SEAGATE/IBM/whatever) but the logical volumes (i.e.
Sun-STK RAID EXT or HP-LOGICAL VOLUME)

On the other hand the Sun X4xxx series uses, AFAIK, some LSI chipset for the
boot disks and they show the physical disks unless you deliberately configure a
RAID0.

Also Adaptec claims the 3- and 5-series can do JBOD, as do LSI for the Advanced
Connectivity Line, though
http://www.lsi.com/DistributionSystem/AssetDocument/LSI_HBA_v_Adaptec_WP_072108.pdf
was more confusing than clarifying on to me on this.

My almost absolute requirement is that I *must* be able to move disks with data
from one controller to another of different brands (and back!), only doing zpool
export and import, which implies the HBA must be able to run in JBOD-mode
without storing or modify anything on the disks.

So could anyone confirm whether any of these adapters can operate in true
JBOD-mode by my definition of JBOD; i.e. Just a Bunch Of Disks and not a Bunch
of Volumes that looks like or simulates JBOD!?

I would like to have a HBA that is just (like the AOC-SAT2-MV8):
+ Plug the pieces together
+ power on
+ devfsadm
+ zpool create

(Though my brain-dead MSI BIOS thinks any added or replaced disk would be a
better boot-candidate than the existing one :/ but I think that''s a MB
problem only..)

P?l
-- 
This message posted from opensolaris.org

I''m pretty darned sure that the LSI 1068-based HBAs will do
"true"
JBOD.  Supermicro makes two such beasts:  AOC-USAS-L8i  and 
AOC-USASLP-L8i     Both are 8-port PCI-Express x4 cards.

The "low-cost" LSI SATA HBAs should also give you what you want.  E.g.
LSISAS3081E-R     These use the same family of chips that are on the 
X4xxx-series machines from us (Sun).


That said, there are still a number of available PCI-X motherboards out 
there, particularly if you are looking at a socket-940 Opteron. 
There''s
very little PCI-X support for single-socket systems of any sort (I think 
I''ve seen  2-slot PCI-X boards for both socket-940 and LGA775, but 
that''s about it).

Supermicro has a nice matrix which shows their various motherboards and 
the slots available:

Intel-based:    
http://www.supermicro.com/products/motherboard/matrix/index.cfm
AMD-based:    http://www.supermicro.com/Aplus/motherboard/matrix/


Tyan''s complete motherboard matrix is here:    
http://www.tyan.com/tech/product_matrix.aspx


Additionally, I''d look at Asus''s Workstation and Server
motherboard
selection - they have some interestingly weird configurations.

    http://usa.asus.com/products.aspx?l1=9&l2=39
and
    http://usa.asus.com/products.aspx?l1=3&l2=82





P?l Baltzersen wrote:
> Why do they throw these fancy RAID-controllers at us when we have plenty
CPU force to do zfs mirror and even raidz1 and raidz2?
>
> I have 12 SATA disks and would like to prepare to add 12 new internal SATA
disks to my home server.
> The cabinet (Lian Li Modular Cube
http://www.microplex.no/aspx/produkt/prdinfovnet.aspx?plid=33415#) takes 24
3.5" or (insane 72 2.5") in front with suitable HDD frames/backplanes.
>
> I have used two Supermicro AOC-SAT2-MV8 PCI-X true 8-port JBOD SATA HBA and
I''m quite happy with them; They are cheap and works in plain PCI
(limited by the PCI-speed aof course, but for my 12-disk home NAS it''s
good enough)
> The only problem is that no mainstream motherboards come with plenty PCI-X
slots. My MB has none and PCI/PCI-X is not future proof. So adding a third
AOC-SAT2-MV8 seems awkward
>
> So I''m looking for a PCIe HBA. The Adaptec 31605  or 52445  seems
tempting as I could reuse the old AOC-SAT2-MV8 elsewhere. Or I could add an
8-port Adaptec 3805  or an LSI SAS3081E-R.
>
> What scares me off (aside the price for these) is that I''ve bumped
into both Sun OEM Adaptec and HP OEM Megaraid at work and none of them would do
true JBOD; No disks showed up in format. I had to configure 1-disk volumes in
BIOS to simulate JBOD, and from what I understand this writes this config to
disk destroying any existing data and partitioning, and what shoes up in format
is *not* the disks (i.e. SEAGATE/IBM/whatever) but the logical volumes (i.e.
Sun-STK RAID EXT or HP-LOGICAL VOLUME)
>
> On the other hand the Sun X4xxx series uses, AFAIK, some LSI chipset for
the boot disks and they show the physical disks unless you deliberately
configure a RAID0.
>
> Also Adaptec claims the 3- and 5-series can do JBOD, as do LSI for the
Advanced Connectivity Line, though
http://www.lsi.com/DistributionSystem/AssetDocument/LSI_HBA_v_Adaptec_WP_072108.pdf
was more confusing than clarifying on to me on this.
>
> My almost absolute requirement is that I *must* be able to move disks with
data from one controller to another of different brands (and back!), only doing
zpool export and import, which implies the HBA must be able to run in JBOD-mode
without storing or modify anything on the disks.
>
> So could anyone confirm whether any of these adapters can operate in true
JBOD-mode by my definition of JBOD; i.e. Just a Bunch Of Disks and not a Bunch
of Volumes that looks like or simulates JBOD!?
>
> I would like to have a HBA that is just (like the AOC-SAT2-MV8):
> + Plug the pieces together
> + power on
> + devfsadm
> + zpool create
>
> (Though my brain-dead MSI BIOS thinks any added or replaced disk would be a
better boot-candidate than the existing one :/ but I think that''s a MB
problem only..)
>
> P?l
>   

-- 
Erik Trimble
Java System Support
Mailstop:  usca22-123
Phone:  x17195
Santa Clara, CA

On Fri, Jan 9, 2009 at 09:28, Erik Trimble <Erik.Trimble at sun.com>
wrote:
> I''m pretty darned sure that the LSI 1068-based HBAs will do
"true"
> JBOD.  Supermicro makes two such beasts:  AOC-USAS-L8i  and
> AOC-USASLP-L8i     Both are 8-port PCI-Express x4 cards.
No, both are UIO cards.  They''re compatible with PCI express x8 (not
x4) but they''re mirrored top-for-bottom.  Take a closer look at the
pictures [1] [2]; these cards are designed to share a slot with a
standard one (think of the PCI+ISA boards that used to exist).  On a
standard PCI device, the PCB is toward the top when mounted in a
standard case, but these cards go in the other way around.  I haven''t
heard anything definitive about whether these work in a standard x8
slot, but I wouldn''t assume that they work.
>> ... I''ve bumped into both Sun OEM Adaptec and HP OEM Megaraid
at work and none of them would do true JBOD; No disks showed up in format. I had
to configure 1-disk volumes in BIOS to simulate JBOD, and from what I understand
this writes this config to disk destroying any existing data and partitioning,
and what shoes up in format is *not* the disks (i.e. SEAGATE/IBM/whatever) but
the logical volumes (i.e. Sun-STK RAID EXT or HP-LOGICAL VOLUME)
This is still the case, judging from my x4150s.
>> I would like to have a HBA that is just (like the AOC-SAT2-MV8):
>> + Plug the pieces together
>> + power on
>> + devfsadm
>> + zpool create
The LSI cards fulfill this requirement.

You might consider a case with a SAS expander in it; you can plug sata
disks into it, and it eliminates the need for a large number of
controller ports.  The Supermicro SC846e1 [3] does this; you can plug
24 disks into one SAS controller.  The power supply fans are fairly
noisy, though, for a home box.  You could also take a look at a
standalone SAS expander; Chenbro sells one called the CK12801 [4] that
would suit the purpose, and might be cheaper than buying additional
controller ports.  An 8-port LSI card with two of those expanders
would run about $750, and support 32 drives.  Compared to an Areca
card of that size, that''s a good deal.  I haven''t seen anyone
running
the Chenbro expanders, but if they perform as specified they''re
convenient.

Will

[1]: http://www.supermicro.com/products/accessories/addon/AOC-USAS-L8i.cfm
[2]: http://www.supermicro.com/products/accessories/addon/AOC-USASLP-L8i.cfm
[3]: http://www.supermicro.com/products/chassis/4U/846/SC846E1-R900.cfm
[4]: http://www.valleyseek.com/product.action?itemID=83883

On 1/9/2009 8:21 AM, Will Murnane wrote:
> You might consider a case with a SAS expander in it; you can plug sata
Except Solaris still lacks support for port expanders, or did last I 
checked. Has this changed?

-- 
Carson

On Fri, Jan 9, 2009 at 9:45 AM, Carson Gaspar <carson at taltos.org>
wrote:
> On 1/9/2009 8:21 AM, Will Murnane wrote:
>
>> You might consider a case with a SAS expander in it; you can plug sata
>
> Except Solaris still lacks support for port expanders, or did last I
> checked. Has this changed?
A SAS expander is different than a SATA  port multiplier (PMP). I''m
not sure if the SAS expander is supported, but it might be.

-B

-- 
Brandon High : bhigh at freaks.com

Brandon High wrote:
> On Fri, Jan 9, 2009 at 9:45 AM, Carson Gaspar <carson at taltos.org>
wrote:
>   
>> On 1/9/2009 8:21 AM, Will Murnane wrote:
>>
>>     
>>> You might consider a case with a SAS expander in it; you can plug
sata
>>>       
>> Except Solaris still lacks support for port expanders, or did last I
>> checked. Has this changed?
>>     
>
> A SAS expander is different than a SATA  port multiplier (PMP).
I''m
> not sure if the SAS expander is supported, but it might be
>   
Sun sells many products which use SAS expanders.
 -- richard

Hi B, thx for hitting on the expander issue that I need to finish your job -

On top of as Rich said, Sun sells many expander-based stuff,

please also understand why you even need that --
the LSI SAS HBA that Sun resells does up to over 200 JBOD disks by itself.

But the new HDS AMS2000 SAS implementation uses the expander technology for 
some some..., just that you have to buy the whole storage controller (a not 
so open storage server if you can''t make the connection).

So, expander or not to expander or how to expander is debate-able in my 
view.
[and yea Orvar, they are all pretty safe for your baby silo data management 
approach...]
;-)

best,
z

----- Original Message ----- 
From: "Richard Elling" <Richard.Elling at Sun.COM>
To: "Brandon High" <bhigh at freaks.com>
Cc: <zfs-discuss at opensolaris.org>
Sent: Friday, January 09, 2009 4:23 PM
Subject: Re: [zfs-discuss] Looking for new SATA/SAS HBA;JBOD is not always 
JBOD

> Brandon High wrote:
>> On Fri, Jan 9, 2009 at 9:45 AM, Carson Gaspar <carson at
taltos.org> wrote:
>>
>>> On 1/9/2009 8:21 AM, Will Murnane wrote:
>>>
>>>
>>>> You might consider a case with a SAS expander in it; you can
plug sata
>>>>
>>> Except Solaris still lacks support for port expanders, or did last
I
>>> checked. Has this changed?
>>>
>>
>> A SAS expander is different than a SATA  port multiplier (PMP).
I''m
>> not sure if the SAS expander is supported, but it might be
>>
>
> Sun sells many products which use SAS expanders.
> -- richard
>
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

On Jan 9, 2009, at 9:28 AM, Erik Trimble wrote:
> I''m pretty darned sure that the LSI 1068-based HBAs will do
"true"
> JBOD.
Indeed they do, and the mpt driver works fine with these cards.

/dale