zfs discuss - Feb 2011 - External SATA drive enclosures + ZFS?

Hi all,

Space is starting to get a bit tight here, so I''m looking at adding
a couple of TB to my home server.  I''m considering external USB or
FireWire attached drive enclosures.  Cost is a real issue, but I also
want the data to be managed by ZFS--so enclosures without a JBOD option
have been disgarded (i.e., I don''t want to use any internal HW RAID
controllers).

One product that seems to fit the bill is the StarTech.com S352U2RER,
an external dual SATA disk enclosure with USB and eSATA connectivity
(I''d be using the USB port).  Here''s a link to the specific
product
I''m considering:

http://ca.startech.com/product/S352U2RER-35in-eSATA-USB-Dual-SATA-Hot-Swap-External-RAID-Hard-Drive-Enclosure

The intent would be put two 1TB or 2TB drives in the enclosure and use
ZFS to create a mirrored pool out of them.  Assuming this enclosure is
set to JBOD mode, would I be able to use this with ZFS?  The enclosure
would be connected to either my Sun Blade 1000 or an Ultra 20.  The
SB 1000 is currently running SXCE b130; the Ultra 20 would either run
SXCE b130 or the latest version of Solaris 11 Express (or whatever its
called!).

I can''t think of a reason why it wouldn''t work, but I also
have exactly
zero experience with this kind of set up!

Assuming this would work, given that I can''t see to find a 4-drive
version of it, would I be correct in thinking that I could buy two of
the above enclosures and connect them to two different USB ports?
Presumably, if that is the case, I could set them up as a RAID 10
pool controlled by ZFS?

This would be replacing a D1000 array, which is mostly empty (I think
I''m
only using one pair of 10K RPM 143 GB disks at the moment!).  I could add
extra disks to the D1000 but appropriate disks seem to be rare/expensive
especially when $/GB is factored in to the equation.

Assuming my proposed enclosure would work, and assuming the use of
reasonable quality 7200 RPM disks, how would you expect the performance
to compare with the differential UltraSCSI set up I''m currently using?
I think the DWIS is rated at either 20MB/sec or 40MB/sec, so on the
surface, the USB attached drives would seem to be MUCH faster...

Many thanks for any pointers received,

-- 
Rich Teer, Publisher
Vinylphile Magazine

www.vinylphilemag.com

On 2/25/2011 7:34 PM, Rich Teer wrote:
> 
> One product that seems to fit the bill is the StarTech.com S352U2RER,
> an external dual SATA disk enclosure with USB and eSATA connectivity
> (I''d be using the USB port).  Here''s a link to the
specific product
> I''m considering:
> 
>
http://ca.startech.com/product/S352U2RER-35in-eSATA-USB-Dual-SATA-Hot-Swap-External-RAID-Hard-Drive-Enclosure
I have had mixed results with their 4 bay version.  When they work, they
are great, but we have had a number of DOA/almost DOA units.  I have had
good luck with products from
http://www.addonics.com/
(They ship to Canada as well without issue)

Why use USB ? You wll get much better performance/throughput on eSata
(if you have good drivers of course). I use their sil3124 eSata
controller on FreeBSD as well as a number of PM units and they work great.

	---Mike


-- 
-------------------
Mike Tancsa, tel +1 519 651 3400
Sentex Communications, mike at sentex.net
Providing Internet services since 1994 www.sentex.net
Cambridge, Ontario Canada   http://www.tancsa.com/

On Fri, Feb 25, 2011 at 4:34 PM, Rich Teer <rich.teer at rite-group.com>
wrote:
> Space is starting to get a bit tight here, so I''m looking at
adding
> a couple of TB to my home server. ?I''m considering external USB or
> FireWire attached drive enclosures. ?Cost is a real issue, but I also
I would avoid USB, since it can be less reliable than other connection
methods. That''s the impression I get from older posts made by Sun
devs, at least. I''m not sure how well Firewire 400 is supported, let
alone Firewire 800.

You might want to consider eSATA. Port multipliers are supported in
recent builds (128+ I think), and will give better performance than
USB. I''m not sure if PMP are supported on Sparc though., since it
requires support in both the controller and PMP.

Consider enclosures from other manufacturers as well. I''ve heard good
things about Sans Digital, but I''ve never used them. The 2-drive
enclosure has the same components as the item you linked but 1/2 the
cost via Newegg.
> The intent would be put two 1TB or 2TB drives in the enclosure and use
> ZFS to create a mirrored pool out of them. ?Assuming this enclosure is
> set to JBOD mode, would I be able to use this with ZFS? ?The enclosure
Yes, but I think the enclosure has a SiI5744 inside it, so you''ll
still have one connection from the computer to the enclosure. If that
goes, you''ll lose both drives. If you''re just using two
drives, two
separate enclosures on separate buses may be better. Look at
http://www.sansdigital.com/towerstor/ts1ut.html for instance. There
are also larger enclosures with up to 8 drives.
> I can''t think of a reason why it wouldn''t work, but I
also have exactly
> zero experience with this kind of set up!
Like I mentioned, USB is prone to some flakiness.
> Assuming this would work, given that I can''t see to find a 4-drive
> version of it, would I be correct in thinking that I could buy two of
You might be better off using separate enclosures for reliability.
Make sure to split the mirrors across the two devices. Use separate
USB controllers if possible, so a bus reset doesn''t affect both sides.
> Assuming my proposed enclosure would work, and assuming the use of
> reasonable quality 7200 RPM disks, how would you expect the performance
> to compare with the differential UltraSCSI set up I''m currently
using?
> I think the DWIS is rated at either 20MB/sec or 40MB/sec, so on the
> surface, the USB attached drives would seem to be MUCH faster...
USB 2.0 is about 30-40MB/s under ideal conditions, but doesn''t support
any of the command queuing that SCSI does. I''d expect performance to
be slightly lower, and to use slightly more CPU. Most USB controllers
don''t support DMA, so all I/O requires CPU time.

What about an inexpensive SAS card (eg: Supermicro AOC-USAS-L4i) and
external SAS enclosure (eg: Sans Digital TowerRAID TR4X). It would
cost about $350 for the setup.

-B

-- 
Brandon High : bhigh at freaks.com

I''m with the gang on this one as far as USB being the spawn of the 
devil for mass storage you want to depend on. I''d rather scoop my eyes 
out with a red hot spoon than depend on permanently attached USB 
storage... And - don''t even start me on SPARC and USB storage...
It''s
like watching pitch flow... (see 
http://en.wikipedia.org/wiki/Pitch_drop_experiment). I never spent too 
much time working out why - but I never seen to get better than about 
10MB/s with SPARC+USB...

When it comes to cheap... I use cheap external SATA/USB combo enclosures 
(single drive ones) as I like the flexibility of being able to use them 
in eSATA mode nice and fast (and reliable considering the $$) or in USB 
mode should I need to split a mirror off and read it on my laptop, which 
has no esata port...

Also - using the single drive enclosures is by far the cheapest (at 
least here in Oz), and you get redundant power supplies, as they use 
their own mini brick AC/DC units. I''m currently very happy using 2TB 
disks in the external eSATA+USB thingies.

I had been using ASTONE external eSATA/USB units - though it seems my 
local shop has stopped carrying them... I liked them as they had 
perforated side panels, which allow the disk to stay much cooler than 
some of my other enclosures... (And have a better ''vertical''
stand if
you want the disks to stand up, rather than lie on their side.)

If your box has PCI-e slots, grab one or two $20 Silicon Image 3132 
controllers with eSATA ports and you should be golden... You will then 
be able to run between 2 and 4 disks - easily pushing them to their 
maximum platter speed - which for most of the 2TB disks is near enough 
to 100M/s at the outer edges. You will also get considerably higher IOPS 
- particularly when they are sequential - using eSATA.

Note: All of this is with the ''cheap'' view... You can most
certainly buy
much better hardware... But bang for buck - I have been happy with the 
above.

Cheers!

Nathan.

On 02/26/11 01:58 PM, Brandon High wrote:
> On Fri, Feb 25, 2011 at 4:34 PM, Rich Teer<rich.teer at
rite-group.com>  wrote:
>> Space is starting to get a bit tight here, so I''m looking at
adding
>> a couple of TB to my home server.  I''m considering external
USB or
>> FireWire attached drive enclosures.  Cost is a real issue, but I also
> I would avoid USB, since it can be less reliable than other connection
> methods. That''s the impression I get from older posts made by Sun
> devs, at least. I''m not sure how well Firewire 400 is supported,
let
> alone Firewire 800.
>
> You might want to consider eSATA. Port multipliers are supported in
> recent builds (128+ I think), and will give better performance than
> USB. I''m not sure if PMP are supported on Sparc though., since it
> requires support in both the controller and PMP.
>
> Consider enclosures from other manufacturers as well. I''ve heard
good
> things about Sans Digital, but I''ve never used them. The 2-drive
> enclosure has the same components as the item you linked but 1/2 the
> cost via Newegg.
>
>> The intent would be put two 1TB or 2TB drives in the enclosure and use
>> ZFS to create a mirrored pool out of them.  Assuming this enclosure is
>> set to JBOD mode, would I be able to use this with ZFS?  The enclosure
> Yes, but I think the enclosure has a SiI5744 inside it, so you''ll
> still have one connection from the computer to the enclosure. If that
> goes, you''ll lose both drives. If you''re just using two
drives, two
> separate enclosures on separate buses may be better. Look at
> http://www.sansdigital.com/towerstor/ts1ut.html for instance. There
> are also larger enclosures with up to 8 drives.
>
>> I can''t think of a reason why it wouldn''t work, but I
also have exactly
>> zero experience with this kind of set up!
> Like I mentioned, USB is prone to some flakiness.
>
>> Assuming this would work, given that I can''t see to find a
4-drive
>> version of it, would I be correct in thinking that I could buy two of
> You might be better off using separate enclosures for reliability.
> Make sure to split the mirrors across the two devices. Use separate
> USB controllers if possible, so a bus reset doesn''t affect both
sides.
>
>> Assuming my proposed enclosure would work, and assuming the use of
>> reasonable quality 7200 RPM disks, how would you expect the performance
>> to compare with the differential UltraSCSI set up I''m
currently using?
>> I think the DWIS is rated at either 20MB/sec or 40MB/sec, so on the
>> surface, the USB attached drives would seem to be MUCH faster...
> USB 2.0 is about 30-40MB/s under ideal conditions, but doesn''t
support
> any of the command queuing that SCSI does. I''d expect performance
to
> be slightly lower, and to use slightly more CPU. Most USB controllers
> don''t support DMA, so all I/O requires CPU time.
>
> What about an inexpensive SAS card (eg: Supermicro AOC-USAS-L4i) and
> external SAS enclosure (eg: Sans Digital TowerRAID TR4X). It would
> cost about $350 for the setup.
>
> -B
>

--- rich.teer at rite-group.com wrote:
>Space is starting to get a bit tight here, so I''m looking at adding
>a couple of TB to my home server.  I''m considering external USB or
>FireWire attached drive enclosures.  Cost is a real issue, but I also
>want the data to be managed by ZFS--so enclosures without a JBOD option
>have been disgarded (i.e., I don''t want to use any internal HW RAID
>controllers).
"tank" on my home file server is a raidz3 with all six drives hooked
up via USB. Across 2 expansion card controllers. (Leaving the motherboard
controller of mouse/keyboard, and hooking up a a fresh drive during capacity
expansions.)
>The intent would be put two 1TB or 2TB drives in the enclosure and use
>ZFS to create a mirrored pool out of them. 
I''d mirror across enclosures. As a home setup, even if I label things,
3 more cables will appear before I want to plug or unplug. I want my single
points of failure to be "the tower" and "the UPS" and
"the guy in the mirror who can type zpool destroy" not any individual
cable.
>I can''t think of a reason why it wouldn''t work, but I also
have exactly
>zero experience with this kind of set up!
I appears to work fine with my commodity parts setup.  I can''t speak to
the reliability of eSATA or FireWire as they fall in the "impossible to
find" category.
>would I be correct in thinking that I could buy two of
>the above enclosures and connect them to two different USB ports?
Don''t see why not, but if you still want the single cable to
accidentally disconnect, you could hook the enclosures up through a hub, and
then use one port on the system.
>Presumably, if that is the case, I could set them up as a RAID 10
>pool controlled by ZFS?
Sure, and since you left ZFS in charge, you can upgrade them to three-way
mirrors in the future if you desire.
>Assuming my proposed enclosure would work, and assuming the use of
>reasonable quality 7200 RPM disks, how would you expect the performance
>to compare with the differential UltraSCSI set up I''m currently
using?
>I think the DWIS is rated at either 20MB/sec or 40MB/sec, so on the
>surface, the USB attached drives would seem to be MUCH faster...
Performance is one thing I don''t know. My solution works for me. 
Lurking here I haven''t heard enough of people talking in consistent
terms to know where the bottleneck is in my system, and if it is something to
worry about.  That changes the moment I start talking to the server from more
than one system at a time.
And all this is with snv_134 should that make any difference.

> From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss-
> bounces at opensolaris.org] On Behalf Of Brandon High
> 
> I would avoid USB, since it can be less reliable than other connection
> methods. That''s the impression I get from older posts made by Sun
Take that a step further.  Anything external is unreliable.  I have used
USB, eSATA, and Firewire external devices.  They all work.  The only
question is for how long.  

If it''s external, you''ve got yet another controller on the
motherboard (or
PCIe) being used...  Yet another data circuit being used inside the external
enclosure....  Now in addition to your internal power supply, you''ve
got an
external power supply.  With power & data wires that can be bumped or
knocked off a shelf...  

The system is up 24/7, and external enclosures aren''t well built for
that
type of usage.  If it''s all internal it''s all enclosed and
it''s all securely
attached and built to stay on 24/7.  With fewer circuits involved to
possibly fail.

I think you should consider the possibility of upgrading the size of your
internal disks before adding external disks.

On 28 February 2011 02:06, Edward Ned Harvey <
opensolarisisdeadlongliveopensolaris at nedharvey.com> wrote:
> Take that a step further.  Anything external is unreliable.  I have used
> USB, eSATA, and Firewire external devices.  They all work.  The only
> question is for how long.

eSATA has no need for any interposer chips between a modern SATA chipset on
the motherboard and a SATA hard drive. You can buy cables with appropriate
ends for this. There is no reason why the data side of an eSATA drive should
be any more likely to fail than SATA. (within bounds, for cable lengths,
etc) At least you can be assured that the drive will receive a flush request
at appropriate times.

I can''t argue about the external power supplies, other than to say that
many
external cases these days use a single +12V rail, and have a +5V regulator
on board. These are a lot better because they allow for easy replacement of
the power supply. External units which use a combined +12V/+5V power supply
are often rendered useless by a power supply failure.

Cheers,
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20110228/58085e04/attachment.html>

On Sun, Feb 27 at 10:06, Edward Ned Harvey wrote:
>> From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss-
>> bounces at opensolaris.org] On Behalf Of Brandon High
>>
>> I would avoid USB, since it can be less reliable than other connection
>> methods. That''s the impression I get from older posts made by
Sun
>
>Take that a step further.  Anything external is unreliable.  I have used
>USB, eSATA, and Firewire external devices.  They all work.  The only
>question is for how long.
>
>If it''s external, you''ve got yet another controller on the
motherboard (or
>PCIe) being used...  Yet another data circuit being used inside the external
>enclosure....  Now in addition to your internal power supply,
you''ve got an
>external power supply.  With power & data wires that can be bumped or
>knocked off a shelf...
>
>The system is up 24/7, and external enclosures aren''t well built
for that
>type of usage.  If it''s all internal it''s all enclosed and
it''s all securely
>attached and built to stay on 24/7.  With fewer circuits involved to
>possibly fail.
I think this will depend a lot on the enclosure itself, not all
enclosures are cheap pieces of doo-doo.
>I think you should consider the possibility of upgrading the size of your
>internal disks before adding external disks.
If space is generally available in the chassis, sure, but I don''t see
*that* much difference between 4 internal controller cards in a
Thumper versus 4 controller cards talking to direct-attached SAS
JBODs.  If you do it right, while your chances of any single hardware
failure occurring goes up with the # of components in the whole
system, your probability of a failure taking you offline should be <the
unified solution.


-- 
Eric D. Mudama
edmudama at bounceswoosh.org

On Fri, 25 Feb 2011, Brandon High wrote:
> You might want to consider eSATA. Port multipliers are supported in
> recent builds (128+ I think), and will give better performance than
> USB. I''m not sure if PMP are supported on Sparc though., since it
> requires support in both the controller and PMP.
OK, the main thing I''m getting here is that USB isn''t a
particularly
reliable connection, and that eSATA would be a better way to go.  I
only mentioned USB/FireWire because those ports are already on the HW
and I wouldn''t have to buy an extra HBA.  But if the cost
isn''t too
great, I''d be happy to look at eSATA-connected external drives too (and
would proabbyl go that route if I can find appropriate HW).

So the question is, what eSATA non-RAID HBA do people recommend?  Bear
in mind that I''m looking for something with driver support "out of
the
box" with either the latest Solaris 10, or Solaris 11 Express.

The SB1000 has only PCI and PCI-X slots IIRC, but the Ultra 20 (or Ultra
20 M2; I have one of each) also has PCI Express.

Assuming the use of eSATA enclosures do do people recommend?  I don''t
need huge amounts of space; two drives should be enough and four will
be plenty and allow for expansion.  Again, I''m looking for a JBOD coz
I want ZFS do all the work.
> What about an inexpensive SAS card (eg: Supermicro AOC-USAS-L4i) and
> external SAS enclosure (eg: Sans Digital TowerRAID TR4X). It would
> cost about $350 for the setup.
Oooh, that''s worth looking at too.  Thanks!

-- 
Rich Teer, Publisher
Vinylphile Magazine

www.vinylphilemag.com

On Sun, Feb 27, 2011 at 4:15 PM, Rich Teer <rich.teer at rite-group.com>
wrote:
> So the question is, what eSATA non-RAID HBA do people recommend? ?Bear
> in mind that I''m looking for something with driver support
"out of the
> box" with either the latest Solaris 10, or Solaris 11 Express.
The SiI3124 (PCI / PCI-X) and SiI3132 (PCIe) based cards can be picked
up for about $20-$30. They''re supported, and support PMPs in Solaris.
I don''t know about support on Sparc though.

http://www.newegg.com/Product/Product.aspx?Item=N82E16816132021
http://www.newegg.com/Product/Product.aspx?Item=N82E16816132027
> Assuming the use of eSATA enclosures do do people recommend? ?I
don''t
> need huge amounts of space; two drives should be enough and four will
> be plenty and allow for expansion. ?Again, I''m looking for a JBOD
coz
> I want ZFS do all the work.
Something similar to the Sans Digital enclosures would probably work.
They use a PMP to make all the drives available via one eSATA, which
may or may not work. It''s supposed to, but there are hardware
blacklists in the drivers that may cause you trouble.

Another thought is to ditch the Sun boxes and use a HP ProLiant
Microserver. It''s about $320 and holds 4 drives, with an expansion
slot for an additional controller. I think some people have reported
success with these on the list.

-B

-- 
Brandon High : bhigh at freaks.com

I can tell you specifically that the 3124 will not work in Sparc
equipment.  I specifically purchased a 3124 after seeing glowing reviews
in the archives.  I needed it for a low end project using a V120 or
Netra T1.  What I didn''t pick up from reviewing the archives was all of
the glowing reviews where from x86 users.

I posted my dilemma to the list a couple of weeks ago and it was
suggested that I probably wouldn''t find an eSATA controller that would
work in a Sparc system and that I would have better luck with a SAS
controller in Sparc.

So now I am looking for a PCI SAS controller.

Jerry


On 02/27/11 20:46, Brandon High wrote:

STUFF DELETED HERE
> 
> The SiI3124 (PCI / PCI-X) and SiI3132 (PCIe) based cards can be picked
> up for about $20-$30. They''re supported, and support PMPs in
Solaris.
> I don''t know about support on Sparc though.
> 
> http://www.newegg.com/Product/Product.aspx?Item=N82E16816132021
> http://www.newegg.com/Product/Product.aspx?Item=N82E16816132027
>

On Sun, Feb 27, 2011 at 7:48 AM, taemun <taemun at gmail.com>
wrote:
> eSATA has no need for any interposer chips between a modern SATA chipset on
> the motherboard and a SATA hard drive. You can buy cables with appropriate
eSATA has different electrical specifications, namely higher minimum
transmit power and lower minimum receive power. An internal power
might work with a SATA to eSATA cable or adapter, but it''s not
guaranteed to.

-B

-- 
Brandon High : bhigh at freaks.com

On Feb 27, 2011, at 10:48 , taemun wrote:
> 
> eSATA has no need for any interposer chips between a modern SATA chipset on
the motherboard and a SATA hard drive. You can buy cables with appropriate ends
for this. There is no reason why the data side of an eSATA drive should be any
more likely to fail than SATA. (within bounds, for cable lengths, etc) At least
you can be assured that the drive will receive a flush request at appropriate
times.
Intel''s platform design guide (at least for its mobile platforms) calls
for a SATA repeater/redriver chip immediately before the eSATA connector (or
docking connector). It is however "passive" in the sense that is
redrives the signal without appearing to the system whatsoever (just a receiver
and re-driver inside the IC). I''d think that an eSATA drive with a
stable power supply + a cable length within spec would be reliable enough for
basic home use.

--khd