zfs discuss - Jan 2010 - New Supermicro SAS/SATA controller: AOC-USAS2-L8e in SOHO NAS and HD HTPC

Supermicro AOC-USASLP-L8i has been confirmed to work flawlessly on OpenSolaris
by several users on these forums already and is AFAIK widely considered the best
controller for building custom Opensolaris ZFS NAS servers
price/performance/compatibility wise in both SOHO NAS and HD HTPC scenarios.


Now Supermicro is shipping a new controller type with 8 internal 6Gb/s SAS/SATA
ports which seems to be an even better investment:

http://www.supermicro.com/products/accessories/addon/AOC-USAS2-L8i.cfm?TYP=E


1) Can anyone confirm it is fully compatible with Opensolaris?

2) Is it as stable as AOC-USAS2-L8e? Are there any known drawbacks compared to
AOC-USAS2-L8e?

3) Does hot-swap cause any problems?

4) Does it support staggered spin-up?

5) The spec states it "Supports 122 devices as HBA in IT mode". Does
this only imply you may connect more than 8 SAS drives via SAS Expanders or can
you as well connect more than 8 SATA drives (e.g. via SATA Port Multipliers)?

6) Which ZFS pool configurations and connection schemes would you recommend to
consider on a single server with 1-2 of these controllers in SOHO NAS and HD
HTPC usage scenarios considering differences in performance characteristics,
single raidz array expansion/upgrade CAPEX, HBA failure/corruption resiliency,
etc.? (e.g. 4-disc raidz1 arrays vs. 8-disc raidz2 arrays vs. ...; all discs in
the same raidz array connected to same controller vs. half of them connected to
each controller)

7) Any further suggestions on choosing Opensolaris ZFS SOHO NAS and HD HTPC
related hardware are welcome (suitable SSDs for ZIL, better 5-to-3 hot-swap
internal backplane HDD cage than ICY DOCK MB455SPF-B, consumer HDDs more
suitable than Samsung HD103UJ [1TB, 7200rpm, NCQ], recommended RAM size/type,
...)


-dusan
-- 
This message posted from opensolaris.org

Yes, this model looks to be interesting.

SuperMicro seem to have produced two models new models that satisfy the SATA III
requirements of 6Gbps per channel:

1. AOC-USAS2-L8e:
http://www.supermicro.com/products/accessories/addon/AOC-USAS2-L8i.cfm?TYP=E
2. AOC-USAS2-L8i:
http://www.supermicro.com/products/accessories/addon/AOC-USAS2-L8i.cfm?TYP=I

The main difference appears to be that the L8i model has RAID capabilities,
whereas the L8e model does not.

As ZFS does its own RAID calculations in software it needs JBOD, and
doesn''t need the adapter to have RAID capabilities, so the
AOC-USAS2-L8e model looks to be ideal. If we''re lucky maybe
it''s also a little cheaper too.

Sorry I can''t help you with your questions though. Hopefully someone
else will be able to help. I will also be interested to hear any further info on
this card.

Cheers,
Simon

http://breden.org.uk/2008/03/02/a-home-fileserver-using-zfs/
-- 
This message posted from opensolaris.org

hello

i have basically tested supermicro mainboard x8dth-6f together with nexenta
http://www.supermicro.com/products/motherboard/QPI/5500/X8DTH-6F.cfm

(same sas-II lsi-2008 chipset)

nexenta 2: did not work
nexenta 3: (snv 124+) install without problem, but no further testing

see also my reference hardware for nexenta:
http://www.napp-it.org/hardware/index.html


gea
-- 
This message posted from opensolaris.org

That looks promising.

As the main thing here is that OpenSolaris supports the LSI SAS2008 controller,
I have created a new post to ask for confirmation of driver support -- see here:
http://opensolaris.org/jive/thread.jspa?threadID=122156&tstart=0

Cheers,
Simon

http://breden.org.uk/2008/03/02/a-home-fileserver-using-zfs/
-- 
This message posted from opensolaris.org

Thanks for the feedback and hardware recommendation gea.

Thus driver support is there out-of-the-box at least in snv_124+.

-dusan
-- 
This message posted from opensolaris.org

Thank you for the effort Simon.

Good to know from the feedback in your thread that the mpt_sas(7d) driver is
actually responsible for the SuperMicro AOC-USAS2-L8e support.

As far as the support for the other RAID capable lsi-2008 variants in the
mr_sas(7d) driver goes, it is but of little concern for ZFS employment in SOHO
NAS and HD HTPC scenarios. Apart from increasing costs, I don''t see
them bring in any advantages, that''s why I left them out -- as you self
point out, ZFS works best on top of JBOD.

Although I agree that driver support and compatibility are of the greatest
concern, the other yet unanswered questions are IMHO nevertheless all of
fundamental importance for SOHO NAS and HD HTPC builders which is a niche that
OpenSolaris and ZFS fits quite well and easily could dominate.

With more feedback targeting the other questions as well, this thread could
become a good kick-start reference for future OpenSolaris/ZFS SOHO NAS and HD
HTPC builders -- something currently AFAICT sadly missing -- and hopefully
significantly boost OpenSolaris adoption.

-dusan
-- 
This message posted from opensolaris.org

> Thank you for the effort Simon.
Thank you too Dusan, for creating this post that made me aware of this new card
-- it looks like a good one, and doesn''t have the unnecessary RAID
stuff included :)
> Good to know from the feedback in your thread that the mpt_sas(7d) driver
is actually responsible for the SuperMicro AOC-USAS2-L8e support.
Yes, the next step will be to find out from people using the mpt_sas driver how
well it''s been working for them, although nobody has volunteered any
feedback yet, probably because the card is so new. Maybe we''ll see some
posts appear on this card and the mpt_sas driver over the next few weeks and
months?

Cheers,
Simon

http://breden.org.uk/2008/03/02/a-home-fileserver-using-zfs/
-- 
This message posted from opensolaris.org

Has anyone bought one of these cards recently? It seems to list for around $170
at various places, which seems like quite a decent deal. But no well-known
reputable vendor I know seems to sell these, and I want to be able to have
someone backing the sale if something isn''t perfect. Where do you all
recommend buying this card from?
-- 
This message posted from opensolaris.org

Not myself yet - but here is some really interesting reading on it:

http://hardforum.com/showthread.php?p=1035820555


On Aug 12, 2010, at 7:03 PM, valrhona at gmail.com wrote:
> Has anyone bought one of these cards recently? It seems to list for around
$170 at various places, which seems like quite a decent deal. But no well-known
reputable vendor I know seems to sell these, and I want to be able to have
someone backing the sale if something isn''t perfect. Where do you all
recommend buying this card from?
> -- 
> This message posted from opensolaris.org
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Thanks for the link; as a result, I learned how to use dd to get some better
data on transfer rates, which was extremely helpful. I guess you can fit the
card in standard PCIe slot with some spacers, but does anyone have any specific
info on this?
-- 
This message posted from opensolaris.org

On Aug 12, 2010, at 7:03 PM, valrhona at gmail.com wrote:
> Has anyone bought one of these cards recently? It seems to list for
> around $170 at various places, which seems like quite a decent deal. But
> no well-known reputable vendor I know seems to sell these, and I want to
> be able to have someone backing the sale if something isn''t
perfect.
> Where do you all recommend buying this card from?
I put something very similar in -- same number with an ''i''
suffix instead
of the ''e''.  I remember seeing both existed at the time, and
that the i
was what I needed.  I''m using SATA cables, and no expanders (each cable
goes directly to a drive), maybe the ''e'' has more advanced
features (that
I knew I didn''t need).

I can''t imagine the retailer would be of any value for support on such
a
card; perhaps, in the worst case, they  might possibly take it back. 
Selling it on Ebay is often more profitable, since the buyer pays shipping
:-).
-- 
David Dyer-Bennet, dd-b at dd-b.net; http://dd-b.net/
Snapshots: http://dd-b.net/dd-b/SnapshotAlbum/data/
Photos: http://dd-b.net/photography/gallery/
Dragaera: http://dragaera.info

What I would really like to know is why do pci-e raid controller cards cost more
than an entire motherboard with processor. Some cards can cost over $1,000
dollars, for what.
-- 
This message posted from opensolaris.org

On Mon, Aug 16, 2010 at 7:13 AM, Mike DeMarco <mikejd42 at yahoo.com>
wrote:
> What I would really like to know is why do pci-e raid controller cards cost
more than an entire motherboard with processor. Some cards can cost over $1,000
dollars, for what.
Because they include a motherboard and processor.  :)  The high-end
RAID controllers include their own CPUs and RAM for doing all the RAID
stuff in hardware.

The low-end RAID controllers (if you can even really call them RAID
controllers) do all the RAID stuff in software via a driver installed
in the OS, running on the host computer''s CPU.

And the ones in the middle have "simple" XOR engines for doing the
RAID.stuff in hardware.

-- 
Freddie Cash
fjwcash at gmail.com

On 08/16/2010 10:35 AM, Freddie Cash wrote:
> On Mon, Aug 16, 2010 at 7:13 AM, Mike DeMarco<mikejd42 at yahoo.com> 
wrote:
>> What I would really like to know is why do pci-e raid controller cards
cost more than an entire motherboard with processor. Some cards can cost over
$1,000 dollars, for what.
>
> Because they include a motherboard and processor.  :)  The high-end
> RAID controllers include their own CPUs and RAM for doing all the RAID
> stuff in hardware.
>
> The low-end RAID controllers (if you can even really call them RAID
> controllers) do all the RAID stuff in software via a driver installed
> in the OS, running on the host computer''s CPU.
>
> And the ones in the middle have "simple" XOR engines for doing
the
> RAID.stuff in hardware.
>

And the irony is that the expensive hardware RAID controllers really
aren''t a
good idea for ZFS. For a ZFS application, you''re far better off to use
a simple
HBA in JBOD mode, and such HBAs can be had in the $100-$200 range.

On 8/16/2010 7:13 AM, Mike DeMarco wrote:
> What I would really like to know is why do pci-e raid controller cards cost
more than an entire motherboard with processor. Some cards can cost over $1,000
dollars, for what.
Well...

Part of it is positioning. Most RAID controllers are targeted at the 
server market, where there is significantly lower volume than the 
desktop.  Yes, they can be expensive. However, there are *very* few raid 
controllers that cost less than a server-class motherboard and CPU.  
Remember, real server-class MB+CPU is going to be a minimum of $500, and 
more likely $1000 to start.

Also, if you look at what''s in a PCI-E RAID controller these days,
it''s
a mini-motherboard. SAS controller chip, plus a dedicated CPU and DRAM, 
plus a battery, plus possibly a SAS expander. Plus custom firmware for a 
driver to use.  Personally, I''m surprised that I can get a SAS RAID 
controller for under $300 at all these days.


Note that practically no-one makes a SATA-only RAID controller -
they''re
all either "fake RAID" where the hardware is really just a SATA HBA
with
the RAID in the software driver, or they''re really a SAS controller
that
can do SATA also.  In the first case, they''re cheap, because there is
no
hardware intelligence at all. In the second case, they''re just as 
expensive as a SAS RAID controller, because the *are* a SAS RAID controller.


Note that the AOC-USAS2-L8e is a SAS HBA, not a RAID controller.

-- 
Erik Trimble
Java System Support
Mailstop:  usca22-123
Phone:  x17195
Santa Clara, CA
Timezone: US/Pacific (GMT-0800)

On 8/16/2010 3:57 PM, Russ Price wrote:
> On 08/16/2010 10:35 AM, Freddie Cash wrote:
>> On Mon, Aug 16, 2010 at 7:13 AM, Mike DeMarco<mikejd42 at
yahoo.com>
>> wrote:
>>> What I would really like to know is why do pci-e raid controller 
>>> cards cost more than an entire motherboard with processor. Some 
>>> cards can cost over $1,000 dollars, for what.
>>
>> Because they include a motherboard and processor.  :)  The high-end
>> RAID controllers include their own CPUs and RAM for doing all the RAID
>> stuff in hardware.
>>
>> The low-end RAID controllers (if you can even really call them RAID
>> controllers) do all the RAID stuff in software via a driver installed
>> in the OS, running on the host computer''s CPU.
>>
>> And the ones in the middle have "simple" XOR engines for
doing the
>> RAID.stuff in hardware.
>>
>
>
> And the irony is that the expensive hardware RAID controllers really 
> aren''t a good idea for ZFS. For a ZFS application, you''re
far better
> off to use a simple HBA in JBOD mode, and such HBAs can be had in the 
> $100-$200 range.
>

Yep, though, honestly, the best thing for ZFS would be some sort of 
enclosure that has a redundant "controller" connection that does 
*no*RAID or other device manipulation at all, but DOES have a large 
NVRAM cache.  I get this currently by running all my array enclosures 
either in JBOD mode, or, more likely, as single-disk RAID0 volumes.  But 
I''m overpaying for all that nice RAID controller hardware I''m
not using,
so it would be nice to just see someone make such an enclosure.  Call it 
a "caching JBOD".

:-)


-- 
Erik Trimble
Java System Support
Mailstop:  usca22-123
Phone:  x17195
Santa Clara, CA
Timezone: US/Pacific (GMT-0800)