zfs discuss - Nov 2007 - Recommended many-port SATA controllers for budget ZFS

Hello,

Short version: Can anyone recommend a *many port* (8 or more) SATA/SAS 
controller (RAID or otherwise) that will allow *SAFE* use of ZFS, including 
honoring cache flush commands in the sense of submitting them to the 
underlying device, that is also *low budget* (suitable for personal use; say 
in the <= $250 range)?

Long version:

I am having difficulty getting reliable information on SATA controllers for 
use with ZFS (that also work with FreeBSD in this case; though I magine the 
same problem applies with Solaris).

The problem is that most non-RAID controllers do not have enough ports. In 
fact the only one I have found that is decent is the Supermicro Marvell card; 
but that is PCI-X rather than PCI (or PCI Express). Works in one machine, 
doesn''t in another (presumably because of PCI-X; only have PCI slots).
And
even if it does work, you are rather limited in bandwidth. Not that I really 
care about the latter for low budget use.

You might say that just get a RAID controller and configure it for JBOD. Well, 
I was assuming that would be aafe bet, but apparanly you cannot trust them to 
behave correctly with respect to write caching and cache flushing.

I recently found out that the Dell supplied LSI MegaRaid derived Perc 5/i RAID 
controllers will not honor cache flush requests (according to Dell technical 
support, after quite some time trying to explain to them what I wanted to 
know). So assuming this information is correct (I never saw the actual 
response from the "behind the lines" tech support that my tech support
contact in turn asked), it means that running without battery backup, you 
actually negate the safety offered by ZFS with respect to write caching, 
making the pool less reliable than it would be with a cheap non-RAID card. 

Right now I have noticed that LSI has recently began offering some 
lower-budget stuff; specifically I am looking at the MegaRAID SAS 
8208ELP/XLP, which are very reasonably priced.

The problem again is that, while they are cheap raid cards without cache, my 
understanding is that it is still primarily intended for RAID rather than 
plain SATA "pass-through". As a result I am worried about the same
problem as
with the Perc 5/i.

Of course, LSI being a large corporation, it is seemingly impossible to obtain 
contact information for them, or find any technical specifications that would 
contain information as specific as what it will do in response to cache flush 
requests, so I am at a loss. Unless you''re buying 10 000 cards
it''s difficult
to get answers.
 
Does anyone have suggestions on what to choose, that will actually work the 
way you want it for JBOD use with ZFS? Or avenus of investigation? Is there 
any chance of a lowly consumer getting any information out of LSI? Is there 
some other manufacturer that provide low-budget stuff that you can get some 
technical information about? Does anyone have some specific knowledge of a 
suitable product?

-- 
/ Peter Schuller

PGP userID: 0xE9758B7D or ''Peter Schuller <peter.schuller at
infidyne.com>''
Key retrieval: Send an E-Mail to getpgpkey at scode.org
E-Mail: peter.schuller at infidyne.com Web: http://www.scode.org

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 187 bytes
Desc: This is a digitally signed message part.
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20071102/730cbdd7/attachment.bin>

Marvell controllers work great with solaris.

Supermicro AOC-SAT2-MV8 is what I currently use.  I bought it on
recommendation from this list actually. I think I paid 110$ for mine.

-Andy


On 11/2/07 4:10 PM, "Peter Schuller" <peter.schuller at
infidyne.com> wrote:
> Hello,
> 
> Short version: Can anyone recommend a *many port* (8 or more) SATA/SAS
> controller (RAID or otherwise) that will allow *SAFE* use of ZFS, including
> honoring cache flush commands in the sense of submitting them to the
> underlying device, that is also *low budget* (suitable for personal use;
say
> in the <= $250 range)?
> 
> Long version:
> 
> I am having difficulty getting reliable information on SATA controllers for
> use with ZFS (that also work with FreeBSD in this case; though I magine the
> same problem applies with Solaris).
> 
> The problem is that most non-RAID controllers do not have enough ports. In
> fact the only one I have found that is decent is the Supermicro Marvell
card;
> but that is PCI-X rather than PCI (or PCI Express). Works in one machine,
> doesn''t in another (presumably because of PCI-X; only have PCI
slots). And
> even if it does work, you are rather limited in bandwidth. Not that I
really
> care about the latter for low budget use.
> 
> You might say that just get a RAID controller and configure it for JBOD.
Well,
> I was assuming that would be aafe bet, but apparanly you cannot trust them
to
> behave correctly with respect to write caching and cache flushing.
> 
> I recently found out that the Dell supplied LSI MegaRaid derived Perc 5/i
RAID
> controllers will not honor cache flush requests (according to Dell
technical
> support, after quite some time trying to explain to them what I wanted to
> know). So assuming this information is correct (I never saw the actual
> response from the "behind the lines" tech support that my tech
support
> contact in turn asked), it means that running without battery backup, you
> actually negate the safety offered by ZFS with respect to write caching,
> making the pool less reliable than it would be with a cheap non-RAID card.
> 
> Right now I have noticed that LSI has recently began offering some
> lower-budget stuff; specifically I am looking at the MegaRAID SAS
> 8208ELP/XLP, which are very reasonably priced.
> 
> The problem again is that, while they are cheap raid cards without cache,
my
> understanding is that it is still primarily intended for RAID rather than
> plain SATA "pass-through". As a result I am worried about the
same problem as
> with the Perc 5/i.
> 
> Of course, LSI being a large corporation, it is seemingly impossible to
obtain
> contact information for them, or find any technical specifications that
would
> contain information as specific as what it will do in response to cache
flush
> requests, so I am at a loss. Unless you''re buying 10 000 cards
it''s difficult
> to get answers.
>  
> Does anyone have suggestions on what to choose, that will actually work the
> way you want it for JBOD use with ZFS? Or avenus of investigation? Is there
> any chance of a lowly consumer getting any information out of LSI? Is there
> some other manufacturer that provide low-budget stuff that you can get some
> technical information about? Does anyone have some specific knowledge of a
> suitable product?

On Fri, 2 Nov 2007, Peter Schuller wrote:

.... snip ....
> Does anyone have suggestions on what to choose, that will actually work the
> way you want it for JBOD use with ZFS? Or avenus of investigation? Is there
> any chance of a lowly consumer getting any information out of LSI? Is there
                                                          ^^^^^^^^^^
Your best bet is to call Tech Support and not Sales.  I''ve found LSI 
tech support to be very responsive to individual customers.
> some other manufacturer that provide low-budget stuff that you can get some
> technical information about? Does anyone have some specific knowledge of a
> suitable product?
I recommend the SuperMicro card - but that is PCI-X and I think you''re 
looking for PCI-Express?  I''ve used the older LSI 4-port (internal) 
PCI Express SAS3041E card which is still available for around $165 and 
works well with ZFS (SATA or SAS drives).  The newer cards are less 
expensive - but its not clear from the LSI website if they support 
JBOD operation or if you can form a "mirror" or "stripe"
using only
one drive and present it to ZFS as a single drive.

Please let us know what you find out...

Regards,

Al Hopper  Logical Approach Inc, Plano, TX.  al at logical-approach.com
            Voice: 972.379.2133 Fax: 972.379.2134  Timezone: US CDT
OpenSolaris Governing Board (OGB) Member - Apr 2005 to Mar 2007
http://www.opensolaris.org/os/community/ogb/ogb_2005-2007/
Graduate from "sugar-coating school"?  Sorry - I never attended! :)

> Marvell controllers work great with solaris.
>
> Supermicro AOC-SAT2-MV8 is what I currently use.  I bought it on
> recommendation from this list actually. I think I paid 110$ for mine.
Yeah I have one of these; and they''re nice. Problem is (1) they are
PCI-X
(thus not compatible with all PCI slots/motherboards), and (2) they are not 
PCI Express ;)

The one I am using has been great so far though (on FreeBSD; never got a 
chance to try on Solaris).

-- 
/ Peter Schuller

PGP userID: 0xE9758B7D or ''Peter Schuller <peter.schuller at
infidyne.com>''
Key retrieval: Send an E-Mail to getpgpkey at scode.org
E-Mail: peter.schuller at infidyne.com Web: http://www.scode.org

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 187 bytes
Desc: This is a digitally signed message part.
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20071104/14cbc188/attachment.bin>

> Your best bet is to call Tech Support and not Sales.  I''ve found
LSI
> tech support to be very responsive to individual customers.
Thanks. I''ll try them. I eventually noticed you could actually get the
number
to them under the "LSI offices" category of their find-your-contact
web form
system which otherwise looked like a re-seller inventory.
> I recommend the SuperMicro card - but that is PCI-X and I think
you''re
> looking for PCI-Express? 
PCI is okay and nice, PCI-Express is nicer. PCI-X I don''t want since it
is
only semi-compatible with PCI. E.g. the Marvell I have now works in one 
machine, not in another.
> works well with ZFS (SATA or SAS drives).  The newer cards are less
> expensive - but its not clear from the LSI website if they support
> JBOD operation or if you can form a "mirror" or
"stripe" using only
> one drive and present it to ZFS as a single drive.
I am okay with a one-disk mirror/stripe in the worse case, as long as cache 
flushes and such get passed through.

Would definitely prefer JBOD though since single-disk virtual volumes tend to 
cause some additional headachages (like having two levels of volumen 
management).
> Please let us know what you find out...
If I get anything confirmed from LSI I''ll post an update.

-- 
/ Peter Schuller

PGP userID: 0xE9758B7D or ''Peter Schuller <peter.schuller at
infidyne.com>''
Key retrieval: Send an E-Mail to getpgpkey at scode.org
E-Mail: peter.schuller at infidyne.com Web: http://www.scode.org

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 187 bytes
Desc: This is a digitally signed message part.
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20071104/31163cdd/attachment.bin>

> Right now I have noticed that LSI has recently began offering some
> lower-budget stuff; specifically I am looking at the MegaRAID SAS
> 8208ELP/XLP, which are very reasonably priced.
I was pleasantly surprised to find that their tech support was very willing to 
look into queries by end-users without any kind of fuss. This will be a 
reason for me to prefer LSI in the future.

I got the following response regarding cache flushing on the 8208:

====Yes the 8208 does honor OS flush cache commands. Disk cache would be
flushed with every flush cache from the OS. There is an out of resource
condition though, which can cause the flush cache not to happen.

 

The out of resource condition that you refer to can be seen under heavy
I/O. The problem can cause *some* cache flush commands to be missed.
There is currently no quantitative data on this issue.  This issue is
being looked at in the next firmware release.  
====
-- 
/ Peter Schuller

PGP userID: 0xE9758B7D or ''Peter Schuller <peter.schuller at
infidyne.com>''
Key retrieval: Send an E-Mail to getpgpkey at scode.org
E-Mail: peter.schuller at infidyne.com Web: http://www.scode.org

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 187 bytes
Desc: This is a digitally signed message part.
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20071118/c9b5691a/attachment.bin>

On Sun, Nov 18, 2007 at 02:18:21PM +0100, Peter Schuller
wrote:
> > Right now I have noticed that LSI has recently began offering some
> > lower-budget stuff; specifically I am looking at the MegaRAID SAS
> > 8208ELP/XLP, which are very reasonably priced.
I looked up the 8204XLP, which is really quite expensive compared to
the Supermicro MV based card.

That being said, for a small 1U box that is only going to have two SATA
disks, the Supermicro card is way overkill/overpriced for my needs.

Does anyone know if there are any PCI-X cards based on the MV88SX6041?

I''m not having much luck finding any.

Thanks,

-brian
-- 
"Perl can be fast and elegant as much as J2EE can be fast and elegant.
In the hands of a skilled artisan, it can and does happen; it''s just
that most of the shit out there is built by people who''d be better
suited to making sure that my burger is cooked thoroughly."  -- Jonathan
Patschke

On Mon, 19 Nov 2007, Brian Hechinger wrote:
> On Sun, Nov 18, 2007 at 02:18:21PM +0100, Peter Schuller wrote:
>>> Right now I have noticed that LSI has recently began offering some
>>> lower-budget stuff; specifically I am looking at the MegaRAID SAS
>>> 8208ELP/XLP, which are very reasonably priced.
>
> I looked up the 8204XLP, which is really quite expensive compared to
> the Supermicro MV based card.
>
> That being said, for a small 1U box that is only going to have two SATA
> disks, the Supermicro card is way overkill/overpriced for my needs.
>
> Does anyone know if there are any PCI-X cards based on the MV88SX6041?
>
> I''m not having much luck finding any.
A couple of options:

a) the SuperMicro AOC-SAT2-MV8 is an 8-port SATA card available for 
around $110 IIRC.

b) There is also a PCI-X version of the older LSI 4-port (internal) 
PCI Express SAS3041E card which is still available for around $165 and 
works well with ZFS (SATA or SAS drives).

c) Any card based on the SiliconImage 3124/3132 chips will work.  But, 
ensure you''re running an OS with the latest version of the si3124 
drivers - or - you can swap out the older drivers using the files 
from:

http://www.opensolaris.org/jive/servlet/JiveServlet/download/80-32437-138083-3390/si3124.tar.gz

Note: if these drives are your boot drives, you''ll need to do this 
after booting from a CDROM/DVD disk, otherwise you can unload the 
driver and swap out the files.

Regards,

Al Hopper  Logical Approach Inc, Plano, TX.  al at logical-approach.com
            Voice: 972.379.2133 Fax: 972.379.2134  Timezone: US CDT
OpenSolaris Governing Board (OGB) Member - Apr 2005 to Mar 2007
http://www.opensolaris.org/os/community/ogb/ogb_2005-2007/
Graduate from "sugar-coating school"?  Sorry - I never attended! :)

On Tue, Nov 20, 2007 at 02:01:34PM -0600, Al Hopper
wrote:
> 
> a) the SuperMicro AOC-SAT2-MV8 is an 8-port SATA card available for 
> around $110 IIRC.
Yeah, I''d like to spend a lot less than that, especially as I only need
2 ports. :)
> b) There is also a PCI-X version of the older LSI 4-port (internal) 
> PCI Express SAS3041E card which is still available for around $165 and 
> works well with ZFS (SATA or SAS drives).
I actually just picked up a SAS3080X for my Ultra80 on ebay for $30. I
guess I can always scour ebay for something similar.
> c) Any card based on the SiliconImage 3124/3132 chips will work.  But, 
> ensure you''re running an OS with the latest version of the si3124 
> drivers - or - you can swap out the older drivers using the files 
> from:
the 3124 looks perfect. The only problem is the only thing I found on ebay
was for the 3132, which is PCIe, which doesn''t help me. :) I''m
not finding
anything for 3124 other than the data on silicon image''s site.  Do you
know
of any cards I should be looking for that uses this chip?

These will be OS disks, and I''m willing to run whichever version is
best
for this hardware and ZFS (I''m going to try the most recent SXCE once
I have all the hardware together).  Any recommendations as related to
this card?

-brian
-- 
"Perl can be fast and elegant as much as J2EE can be fast and elegant.
In the hands of a skilled artisan, it can and does happen; it''s just
that most of the shit out there is built by people who''d be better
suited to making sure that my burger is cooked thoroughly."  -- Jonathan
Patschke

>the 3124 looks perfect. The only problem is the only thing I found on ebay
>was for the 3132, which is PCIe, which doesn''t help me. :)
I''m not finding
>anything for 3124 other than the data on silicon image''s site.  Do
you know
>of any cards I should be looking for that uses this chip?
http://www.cooldrives.com/sata-cards.html

There are a couple on there for about $80.  Not quite where you want to get I am
sure but it is an option.

On Tue, 20 Nov 2007, Jason P. Warr wrote:
>
>
>> the 3124 looks perfect. The only problem is the only thing I found on
ebay
>> was for the 3132, which is PCIe, which doesn''t help me. :)
I''m not finding
>> anything for 3124 other than the data on silicon image''s site.
Do you know
>> of any cards I should be looking for that uses this chip?
>
> http://www.cooldrives.com/sata-cards.html
>
> There are a couple on there for about $80.  Not quite where you want to get
I am sure but it is an option.
Yep - I see: http://www.cooldrives.com/saiiraco2esa.html for $60.

Regards,

Al Hopper  Logical Approach Inc, Plano, TX.  al at logical-approach.com
            Voice: 972.379.2133 Fax: 972.379.2134  Timezone: US CDT
OpenSolaris Governing Board (OGB) Member - Apr 2005 to Mar 2007
http://www.opensolaris.org/os/community/ogb/ogb_2005-2007/
Graduate from "sugar-coating school"?  Sorry - I never attended! :)

>>> the 3124 looks perfect. The only problem is the only thing I found
on
>>> ebay
>>> was for the 3132, which is PCIe, which doesn''t help me. :)
I''m not
>>> finding
>>> anything for 3124 other than the data on silicon image''s
site.  Do you
>>> know
>>> of any cards I should be looking for that uses this chip?
>>
>> http://www.cooldrives.com/sata-cards.html
>>
>> There are a couple on there for about $80.  Not quite where you want to
>> get I am sure but it is an option.
>
> Yep - I see: http://www.cooldrives.com/saiiraco2esa.html for $60.
I got a Sil3114 (4 internal ports) off ebay for $AU30 including postage. 
Didnt look at any PCIe stuff since I''m building up from old parts.
> Regards,
>
> Al Hopper  Logical Approach Inc, Plano, TX.  al at logical-approach.com
>            Voice: 972.379.2133 Fax: 972.379.2134  Timezone: US CDT
> OpenSolaris Governing Board (OGB) Member - Apr 2005 to Mar 2007
> http://www.opensolaris.org/os/community/ogb/ogb_2005-2007/
> Graduate from "sugar-coating school"?  Sorry - I never attended!
:)
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss