zfs discuss - Jun 2011 - Should Intel X25-E not be used with a SAS Expander?

I was just doing some storage research and came across this 
http://www.nexenta.com/corp/images/stories/pdfs/hardware-supported.pdf. 
In that document for Nexenta (an opensolaris variant) it states that you 
should not use Intel X25-E SSDSA2SH032G1 SSD with a SAS Expander. Can 
anyone tell me why?

This seems to be a very common drive people deploy in ZFS pools.

Thanks,

Josh Simon

2011-06-02 18:40, Josh Simon ?????:
> I was just doing some storage research and came across this 
> http://www.nexenta.com/corp/images/stories/pdfs/hardware-supported.pdf. In 
> that document for Nexenta (an opensolaris variant) it states that you 
> should not use Intel X25-E SSDSA2SH032G1 SSD with a SAS Expander. Can 
> anyone tell me why?
>
> This seems to be a very common drive people deploy in ZFS pools.
I believe one reason is that these are single-port devices - and as such
do not support failover to another SAS path.

Prooflink: 
http://www.infostor.com/index/articles/display/0774287313/articles/infostor/disk-arrays/ssd-drives/pliant-delivers_efd.html

EFDs are also expected to compete to a degree with SSDs from Intel, 
although some analysts -- such as Objective Analysis'' Handy -- do not 
consider Intel''s SATA-based  (single-port, half-duplex) X25-E SSDs to
be
"enterprise class" SSDs. (SAS drives are dual-port, full-duplex
devices.)



Another reason *may be* (maybe not, speculative) if these drives have
a SATA protocol firmware instead of a SAS one - resulting in general
feature sets...

//Jim

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20110602/ce05c814/attachment.html>

I don''t believe this to be the reason since there are other SATA 
(single-port) SSD drives listed as approved in that same document.

Upon further research I found some interesting links that may point to a 
potentially different reason for not using the Intel X25-E with a SAS 
Expander:

http://gdamore.blogspot.com/2010/08/why-sas-sata-is-not-such-great-idea.html

"Update: At a significant account, I can say that we (meaning Nexenta) 
have verified that SAS/SATA expanders combined with high loads of ZFS 
activity have proven conclusively to be highly toxic. So, if you''re 
designing an enterprise storage solution, please consider using SAS all 
the way to the disk drives, and just skip those cheaper SATA options. 
You may think SATA looks like a bargain, but when your array goes 
offline during ZFS scrub or resilver operations because the expander is 
choking on cache sync commands, you''ll really wish you had spent the 
extra cash up front. Really."

and

http://gdamore.blogspot.com/2010/12/update-on-sata-expanders.html

This sounds like it will affect a lot of people since so many are using 
SATA SSD for their log devices connected to SAS expanders.

Thanks,

Josh Simon


On 06/02/2011 01:25 PM, Jim Klimov wrote:
> 2011-06-02 18:40, Josh Simon ?????:
>> I was just doing some storage research and came across this
>> http://www.nexenta.com/corp/images/stories/pdfs/hardware-supported.pdf.
In
>> that document for Nexenta (an opensolaris variant) it states that you
>> should not use Intel X25-E SSDSA2SH032G1 SSD with a SAS Expander. Can
>> anyone tell me why?
>>
>> This seems to be a very common drive people deploy in ZFS pools.
>
> I believe one reason is that these are single-port devices - and as such
> do not support failover to another SAS path.
>
>
> //Jim
>

On Thu, Jun 02, 2011 at 11:19:25AM -0700, Josh Simon
wrote:
> I don''t believe this to be the reason since there are other SATA 
> (single-port) SSD drives listed as approved in that same document.
> 
> Upon further research I found some interesting links that may point to a 
> potentially different reason for not using the Intel X25-E with a SAS 
> Expander:
> 
>
http://gdamore.blogspot.com/2010/08/why-sas-sata-is-not-such-great-idea.html
> 
> "Update: At a significant account, I can say that we (meaning Nexenta)
> have verified that SAS/SATA expanders combined with high loads of ZFS 
> activity have proven conclusively to be highly toxic. So, if
you''re
> designing an enterprise storage solution, please consider using SAS all 
> the way to the disk drives, and just skip those cheaper SATA options. 
> You may think SATA looks like a bargain, but when your array goes 
> offline during ZFS scrub or resilver operations because the expander is 
> choking on cache sync commands, you''ll really wish you had spent
the
> extra cash up front. Really."
> 
> and
> 
> http://gdamore.blogspot.com/2010/12/update-on-sata-expanders.html
> 
> This sounds like it will affect a lot of people since so many are using 
> SATA SSD for their log devices connected to SAS expanders.
> 
> Thanks,
> 
> Josh Simon
> 
Yup; reset storms affected us as well (we were using the X-25 series
for ZIL/L2ARC).  Only the ZIL drives were impacted, but it was a large
impact :)

Our solution was to move the SSD''s off of the expander and remount
internally attached via one of the LSI SAS ports directly (we also had
problems with running the drives directly off the on-board SATA ports
on our SuperMicro motherboards -- occasionally the entire zpool would
freeze up).

Ray
> 
> On 06/02/2011 01:25 PM, Jim Klimov wrote:
> > 2011-06-02 18:40, Josh Simon ?????:
> >> I was just doing some storage research and came across this
> >>
http://www.nexenta.com/corp/images/stories/pdfs/hardware-supported.pdf. In
> >> that document for Nexenta (an opensolaris variant) it states that
you
> >> should not use Intel X25-E SSDSA2SH032G1 SSD with a SAS Expander.
Can
> >> anyone tell me why?
> >>
> >> This seems to be a very common drive people deploy in ZFS pools.
> >
> > I believe one reason is that these are single-port devices - and as
such
> > do not support failover to another SAS path.
> >
> >
> > //Jim

> Yup; reset storms affected us as well (we were using the X-25 series
> for ZIL/L2ARC). ?Only the ZIL drives were impacted, but it was a large
> impact :)
What did you see with your reset storm? Were there log errors in
/var/adm/messages or did you need to check the controller loogs with
something like lsi util?

Did the reset workaround in the blog post help?

The expanders you were using were SAS/SATA expanders? Or SAS expanders
with adapters on the drive to allow the use of SATA disks?

I''ve been using 4 X-25E''s with Promise J610sD SAS shelves and
the
AAMUX adapters and have yet to have a problem.
> Our solution was to move the SSD''s off of the expander and remount
> internally attached via one of the LSI SAS ports directly (we also had
> problems with running the drives directly off the on-board SATA ports
> on our SuperMicro motherboards -- occasionally the entire zpool would
> freeze up).
I''m surprised you had problems with the internal SATA ports as well-
any idea what was causing the problems there?

-Don

On Jun 2, 2011, at 7:40 AM, Josh Simon wrote:
> I was just doing some storage research and came across this
http://www.nexenta.com/corp/images/stories/pdfs/hardware-supported.pdf. In that
document for Nexenta (an opensolaris variant) it states that you should not use
Intel X25-E SSDSA2SH032G1 SSD with a SAS Expander. Can anyone tell me why?
Erhhhmm... it doesn''t work. There is a firmware bug that can cause
reset storms,
killing performance and, in extreme cases, affecting all other devices on the
expander.
Please contact your Intel FAE for more information.
> This seems to be a very common drive people deploy in ZFS pools.
Yes.

On Jun 2, 2011, at 11:39 AM, Donald Stahl wrote:
>> Yup; reset storms affected us as well (we were using the X-25 series
>> for ZIL/L2ARC).  Only the ZIL drives were impacted, but it was a large
>> impact :)
> What did you see with your reset storm? Were there log errors in
> /var/adm/messages or did you need to check the controller loogs with
> something like lsi util?
Resets are normally logged to syslog. You will also see transport errors
logged in iostat -e or kstat. For more detail, you will see error reports in
FMA: fmdump -e.

Note: resets are a reaction to a symptom, not a cause. There are many
pathologies that are indirectly exposed by the presence of resets.
 -- richard

On Thu, Jun 02, 2011 at 11:39:13AM -0700, Donald Stahl
wrote:
> > Yup; reset storms affected us as well (we were using the X-25 series
> > for ZIL/L2ARC). ?Only the ZIL drives were impacted, but it was a large
> > impact :)
> What did you see with your reset storm? Were there log errors in
> /var/adm/messages or did you need to check the controller loogs with
> something like lsi util?
Yep, /var/adm/messages had Unit Attention errors.  Ref:

    http://markmail.org/message/5rmfzvqwlmosh2oh
> Did the reset workaround in the blog post help?
We re-architected before reading the blog post, so I''m unsure if it
would have helped or not.  In any case, moving the SSD''s internal lets
us use additional hot-swappable data disks, so it was beneficial in
other areas as well.
> 
> The expanders you were using were SAS/SATA expanders? Or SAS expanders
> with adapters on the drive to allow the use of SATA disks?
The expander was a SuperMicro SAS-846EL1 which is a SAS expander but has
SFF-8482 connectors to provide compatability with SATA drives.
> 
> I''ve been using 4 X-25E''s with Promise J610sD SAS shelves
and the
> AAMUX adapters and have yet to have a problem.
It definitely seemed itermittent, and various suggestions we received
indicated we might need to downgrade our backplane/expander''s firmware.
Never did try that, but it wouldn''t surprise me if behavior was
better/worse on different backplanes...
> 
> > Our solution was to move the SSD''s off of the expander and
remount
> > internally attached via one of the LSI SAS ports directly (we also had
> > problems with running the drives directly off the on-board SATA ports
> > on our SuperMicro motherboards -- occasionally the entire zpool would
> > freeze up).
>
> I''m surprised you had problems with the internal SATA ports as
well-
> any idea what was causing the problems there?
Nope.  I posted this:

    http://mail.opensolaris.org/pipermail/zfs-discuss/2010-October/045625.html

But got no responses.  We resolved the NFS errors (which I believe were
coincidental), but the watchdog port issues kept reoccurring without
rhyme or reason.  The box itself wouldn''t lock up, but the zpool would
become non-resopnsive and we''d have to hard reset.

This was all production stuff, so as soon as we were able to, we
ditched using the SATA ports entirely instead of pursuing a fix with
Sun.

Ray

Josh, I don''t know the internals of the device but I have heard reports
of SSDs that would ignore flush write cache commands _and_ wouldn''t
have a supercap protection (nor battery).
Such devices are subject to dataloss. 

Did you also catch this thread 
	http://opensolaris.org/jive/thread.jspa?threadID=121424

-r

Le 2 juin 2011 ? 16:40, Josh Simon a ?crit :
> I was just doing some storage research and came across this
http://www.nexenta.com/corp/images/stories/pdfs/hardware-supported.pdf. In that
document for Nexenta (an opensolaris variant) it states that you should not use
Intel X25-E SSDSA2SH032G1 SSD with a SAS Expander. Can anyone tell me why?
> 
> This seems to be a very common drive people deploy in ZFS pools.
> 
> Thanks,
> 
> Josh Simon
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss