I'm not on the -STABLE list so please reply to me. I'm using an Intel Core i3-530 on a Gigabyte H55M-D2H motherboard with 8 x 2TB drives & 2 x 1TB drives. The plan is to have the 1 TB drives in a zmirror and the 8 in a raidz2. Now the Intel chipset has only 6 on board SATA II ports so ideally I'm looking for a non RAID SATA II HBA to give me 6 extra ports (4 min). Why 6 extra ? Well the case I'm using has 2 x eSATA ports so 6 would be ideal, 5 OK, and 4 the minimum I need to do the job. So... What do people recommend for 8-STABLE as a PCIe SATA II HBA for someone using ZFS ? Not wanting to break the bank. Not interested in SATA III 6GB at this time... though it could be useful if I add an SSD for... (is it ZIL ?). Can this be added at any time ? The main issue is I need at least 10 ports total for all existing drives... ZIL would require 11 so ideally we are talking a 6 port HBA.
Areca's work well. The ARC-1220 (8 ports) should do you, not the cheapest
but
good support and performance.
Regards
Steve
----- Original Message -----
From: "Matt Thyer" <matt.thyer@gmail.com>
To: <stable@freebsd.org>
Sent: Tuesday, May 31, 2011 1:48 PM
Subject: PCIe SATA HBA for ZFS on -STABLE
> I'm not on the -STABLE list so please reply to me.
>
> I'm using an Intel Core i3-530 on a Gigabyte H55M-D2H motherboard with
8 x
> 2TB drives & 2 x 1TB drives.
> The plan is to have the 1 TB drives in a zmirror and the 8 in a raidz2.
>
> Now the Intel chipset has only 6 on board SATA II ports so ideally I'm
> looking for a non RAID SATA II HBA to give me 6 extra ports (4 min).
> Why 6 extra ?
> Well the case I'm using has 2 x eSATA ports so 6 would be ideal, 5 OK,
and 4
> the minimum I need to do the job.
>
> So...
>
> What do people recommend for 8-STABLE as a PCIe SATA II HBA for someone
> using ZFS ?
>
> Not wanting to break the bank.
> Not interested in SATA III 6GB at this time... though it could be useful if
> I add an SSD for... (is it ZIL ?).
> Can this be added at any time ?
>
> The main issue is I need at least 10 ports total for all existing drives...
> ZIL would require 11 so ideally we are talking a 6 port HBA.
> _______________________________________________
> freebsd-stable@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to
"freebsd-stable-unsubscribe@freebsd.org"
>
===============================================This e.mail is private and
confidential between Multiplay (UK) Ltd. and the person or entity to whom it is
addressed. In the event of misdirection, the recipient is prohibited from using,
copying, printing or otherwise disseminating it or any information contained in
it.
In the event of misdirection, illegible or incomplete transmission please
telephone +44 845 868 1337
or return the E.mail to postmaster@multiplay.co.uk.
On Tue, May 31, 2011 at 5:48 AM, Matt Thyer <matt.thyer@gmail.com> wrote:> What do people recommend for 8-STABLE as a PCIe SATA II HBA for someone > using ZFS ? > > Not wanting to break the bank. > Not interested in SATA III 6GB at this time... though it could be useful if > I add an SSD for... (is it ZIL ?). > Can this be added at any time ? > > The main issue is I need at least 10 ports total for all existing drives... > ZIL would require 11 so ideally we are talking a 6 port HBA. >SuperMicro AOC-USAS2-L8i works exceptionally well. These are 8-port HBAs using the LSI1068 chipset, supported by the mpt(4) driver. Support 3 Gpbs SATA/SAS, using multi-lane cables (2 connectors on the card, each connector supports 4 SATA ports), hot-plug, hot-swap. These are UIO cards, so the bracket that comes with it doesn't work with normal cases (the bracket is on the wrong side of the card; they're made for SuperMicro's UIO-based motherboards). However, these are normal PCIe cards and work in any PCIe slot. You either have to remove the bracket, or you can purchase separate brackets online. These cards are recommended on the zfs-discuss mailing list. They are only ~$120 CDN at places like cdw.ca and newegg.ca. -- Freddie Cash fjwcash@gmail.com
On Tue, May 31, 2011 at 10:31 PM, Freddie Cash <fjwcash@gmail.com> wrote:> On Tue, May 31, 2011 at 5:48 AM, Matt Thyer <matt.thyer@gmail.com> wrote: > > > What do people recommend for 8-STABLE as a PCIe SATA II HBA for someone > > using ZFS ? > > > > Not wanting to break the bank. > > Not interested in SATA III 6GB at this time... though it could be useful > if > > I add an SSD for... (is it ZIL ?). > > Can this be added at any time ? > > > > The main issue is I need at least 10 ports total for all existing > drives... > > ZIL would require 11 so ideally we are talking a 6 port HBA. > > > > SuperMicro AOC-USAS2-L8i works exceptionally well. These are 8-port HBAs > using the LSI1068 chipset, supported by the mpt(4) driver. Support 3 Gpbs > SATA/SAS, using multi-lane cables (2 connectors on the card, each connector > supports 4 SATA ports), hot-plug, hot-swap. > >The USAS2 (6Gbps) is supported by the mps driver (on -CURRENT, not sure if it's in 8-STABLE yet). Perhaps you're referring to the earlier USAS which does 3Gbps and is supported by the mpt driver. -- TJ
:I'm not on the -STABLE list so please reply to me.
:
:I'm using an Intel Core i3-530 on a Gigabyte H55M-D2H motherboard with 8 x
:2TB drives & 2 x 1TB drives.
:The plan is to have the 1 TB drives in a zmirror and the 8 in a raidz2.
:
:Now the Intel chipset has only 6 on board SATA II ports so ideally I'm
:looking for a non RAID SATA II HBA to give me 6 extra ports (4 min).
:Why 6 extra ?
:Well the case I'm using has 2 x eSATA ports so 6 would be ideal, 5 OK, and
4
:the minimum I need to do the job.
:
:So...
:
:What do people recommend for 8-STABLE as a PCIe SATA II HBA for someone
:using ZFS ?
:
:Not wanting to break the bank.
:Not interested in SATA III 6GB at this time... though it could be useful if
:I add an SSD for... (is it ZIL ?).
:Can this be added at any time ?
:
:The main issue is I need at least 10 ports total for all existing drives...
:ZIL would require 11 so ideally we are talking a 6 port HBA.
The absolute cheapest solution is to buy a Sil-3132 PCIe card
(providing 2 E-SATA ports), and then connect an external port multiplier
to each port. External port multiplier enclosures typically support
5 drives each so that would give you your 10 drives.
Even the 3132 is a piss-ant little card it does support FIS-Based
switching so performance will be very good... it will just be limited
to SATA-II speeds is all.
Motherboard AHCI-based SATA ports typically do NOT have FIS-Based
switching support (this would be the FBSS capability flag when the AHCI
driver probes the chipset). This means that while you can attach an
external port multiplier enclosure to mobo SATA ports (see later
on E-SATA vs SATA), read performance from multiple drives concurrently
will be horrible. Write performance will still be decent due to drive
write caches despite being serialized.
On E-SATA vs SATA. Essentially there are only two differences between
E-SATA and SATA. One is the cable and connector format. The other is
hot swap detection. Most mobo SATA ports can be strung out to E-SATA
with an appropriate adapter. High-end Intel ASUS mobos often come with
such adapters (this is why they usually don't sport an actual E-Sata
port on the backplane) and the BIOS has setup features to specify E-SATA
on a port-by-port basis.
--
For SSDs you want to directly connect the SSD to a mobo SATA port and
then either mount the SSD in the case or mount it in a hot-swap gadget
that you can screw into a PCI slot (it doesn't actually use the PCI
connector, just the slot). A SATA-III port with a SATA-III SSD really
shines here and 400-500 MBytes/sec random read performance from a single
SSD is possible, but it isn't an absolute requirement. A SATA-II port
will still work fine as long as you don't mind maxing out the bandwidth
at 250 MBytes/sec.
--
I can't help with any of the other questions. Someone also suggested
the MPS driver for FreeBSD, with caveats.
I'll add a caveat on the port multiplier enclosures. Nearly all such
enclosures use another SIL chipset internally and it works pretty well
EXCEPT that it isn't 100% dependable if you try to hot-swap drives in
the
enclosure while other drives in the enclosure are active. So with that
caveat, I recommend the port multiplier enclosure as the cheapest solution.
To get robust hot-swap enclosures you either need to go with SAS or you
need to go with discrete SATA ports (no port multiplication), and the
ports have to support hot-swap. The best hot-swap support for an AHCI
port is if the AHCI chipset supports cold-presence-detect (CPD), and
again Mobo AHCI chipsets usually don't. Hot-swap is a bit hit or miss
without CPD because power savings modes can effectively prevent hot-swap
detect from working properly. Drive disconnects will always be detected
but drive connects might not be.
And even with discrete SATA ports the AHCI firmware on mobos does not
necessarily handle hot-swap properly. For example my Intel-I7 ASUS mobo
will generate spurious interrupts and status on a DIFFERENT discrete
SATA port when I hot swap on some other discrete SATA port, in addition
to generating the status interrupt on the correct port. So then it comes
down to the driver in the operating system properly handling the
spurious status and properly stopping and restarting pending commands
when necessary. So, again, it is best for the machine to be idle before
attempting a hot-swap.
Lots of caveats. Sorry... you can blame Intel for all the blasted issues
with AHCI and SATA. Intel didn't produce a very good chipset spec and
vendors took all sorts of liberties.
-Matt
Matthew Dillon
<dillon@backplane.com>
On 07.06.2011 05:33, Matthew Dillon wrote:> The absolute cheapest solution is to buy a Sil-3132 PCIe card > (providing 2 E-SATA ports), and then connect an external port multiplier > to each port. External port multiplier enclosures typically support > 5 drives each so that would give you your 10 drives. > > Even the 3132 is a piss-ant little card it does support FIS-Based > switching so performance will be very good... it will just be limited > to SATA-II speeds is all.SiI3132 is indeed good for it's price and it is quite good for random I/O. But at burst speeds it is limited lower then SATA-II. Even lower then PCIe 1.0 x1 it uses. IIRC I've seen about 150MB/s from one port and about 170MB/s from two. If burst rate is important, SiI3124 chip is much better -- up to about 900MB/s measured from 4 ports. The only issue is PCI-X interface: either motherboard with PCI-X needed, or card with PCIe x8 bridge (like these http://www.addonics.com/products/host_controller/adsa3gpx8-4e.asp), but last case is too expensive. There are also much cheaper (~$50) PCIe x1 bridge SiI3124 cards (http://www.sybausa.com/productInfo.php?iid=537). They are not so fast -- about 200MB/s, but still more then SiI3132. And they still have 4 SATA ports.> For SSDs you want to directly connect the SSD to a mobo SATA port and > then either mount the SSD in the case or mount it in a hot-swap gadget > that you can screw into a PCI slot (it doesn't actually use the PCI > connector, just the slot). A SATA-III port with a SATA-III SSD really > shines here and 400-500 MBytes/sec random read performance from a single > SSD is possible, but it isn't an absolute requirement. A SATA-II port > will still work fine as long as you don't mind maxing out the bandwidth > at 250 MBytes/sec.Agree. Intel on-board ports rock! Recently I've built new system with two OCZ Vertex 3 SSDs connected to 6Gbps SATA ports on Intel Sandy Bridge class motherboard. UFS on top of graid RAID0 volume gives me about 950MB/s on both read and write!> To get robust hot-swap enclosures you either need to go with SAS or you > need to go with discrete SATA ports (no port multiplication), and the > ports have to support hot-swap. The best hot-swap support for an AHCI > port is if the AHCI chipset supports cold-presence-detect (CPD), and > again Mobo AHCI chipsets usually don't. Hot-swap is a bit hit or miss > without CPD because power savings modes can effectively prevent hot-swap > detect from working properly. Drive disconnects will always be detected > but drive connects might not be.I would say it depends. In some cases it is easier to detect hot-plug then hot-unplug, as device sends COMINIT that should wake up port even from power-save state. With ICH10, for example, I've managed to make both hot plug and unplug work even with power-management enabled: hot-plug via tracking COMINIT, unplug via it's CPD capability. Without PM it "just works". :) -- Alexander Motin