zfs discuss - Aug 2011 - Question about WD drives with Super Micro systems

Hi all

We have a few servers with WD Black (and some green) drives on Super Micro
systems. We''ve seen both drives work well with direct attach, but with
LSI controllers and Super Micro''s SAS expanders, well, that''s
another story. With those SAS expanders, we''ve seen numerous drives
being kicked out and flagged as bad during high load (typically scrub/resilver).
We have not seen this on the units we have with Hitachi or Seagate drives. After
a drive is kicked out, we run a test on it, using WDs tool, and in many (or
most) cases, we find the drive being error free. We''ve seen these
issues on several machines, so hardware failure seem not to be the case.

Have anyone here used WD drives with LSI controllers (3801/3081/9211) with Super
Micro machines? Any success stories?

Vennlige hilsener / Best regards

roy
--
Roy Sigurd Karlsbakk
(+47) 97542685
roy at karlsbakk.net
http://blogg.karlsbakk.net/
--
I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det er
et element?rt imperativ for alle pedagoger ? unng? eksessiv anvendelse av
idiomer med fremmed opprinnelse. I de fleste tilfeller eksisterer adekvate og
relevante synonymer p? norsk.

On Aug 6, 2011, at 9:45 AM, Roy Sigurd Karlsbakk wrote:
> Hi all
> 
> We have a few servers with WD Black (and some green) drives on Super Micro
systems. We''ve seen both drives work well with direct attach, but with
LSI controllers and Super Micro''s SAS expanders, well, that''s
another story. With those SAS expanders, we''ve seen numerous drives
being kicked out and flagged as bad during high load (typically scrub/resilver).
We have not seen this on the units we have with Hitachi or Seagate drives. After
a drive is kicked out, we run a test on it, using WDs tool, and in many (or
most) cases, we find the drive being error free. We''ve seen these
issues on several machines, so hardware failure seem not to be the case.
> 
> Have anyone here used WD drives with LSI controllers (3801/3081/9211) with
Super Micro machines? Any success stories?
In my experience, SATA drives behind SAS expanders just don''t work.
They "fail" in the manner you
describe, sooner or later. Use SAS and be happy.
 -- richard

> In my experience, SATA drives behind SAS expanders just don''t
work.
> They "fail" in the manner you
> describe, sooner or later. Use SAS and be happy.
Funny thing is Hitachi and Seagate drives work stably, whereas WD drives tend to
fail rather quickly

Vennlige hilsener / Best regards

roy
--
Roy Sigurd Karlsbakk
(+47) 97542685
roy at karlsbakk.net
http://blogg.karlsbakk.net/
--
I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det er
et element?rt imperativ for alle pedagoger ? unng? eksessiv anvendelse av
idiomer med fremmed opprinnelse. I de fleste tilfeller eksisterer adekvate og
relevante synonymer p? norsk.

WD''s drives have gotten better the last few years but their quality is
still not very good. I doubt they test their drives extensively for heavy duty
server configs, particularly since you don''t see them inside any of the
major server manufactures'' boxes.

Hitachi in particular does well in mass storage configs. 

-J

Sent via iPhone

Is your email Premiere?

On Aug 6, 2011, at 10:45, Roy Sigurd Karlsbakk <roy at karlsbakk.net>
wrote:
> Hi all
> 
> We have a few servers with WD Black (and some green) drives on Super Micro
systems. We''ve seen both drives work well with direct attach, but with
LSI controllers and Super Micro''s SAS expanders, well, that''s
another story. With those SAS expanders, we''ve seen numerous drives
being kicked out and flagged as bad during high load (typically scrub/resilver).
We have not seen this on the units we have with Hitachi or Seagate drives. After
a drive is kicked out, we run a test on it, using WDs tool, and in many (or
most) cases, we find the drive being error free. We''ve seen these
issues on several machines, so hardware failure seem not to be the case.
> 
> Have anyone here used WD drives with LSI controllers (3801/3081/9211) with
Super Micro machines? Any success stories?
> 
> Vennlige hilsener / Best regards
> 
> roy
> --
> Roy Sigurd Karlsbakk
> (+47) 97542685
> roy at karlsbakk.net
> http://blogg.karlsbakk.net/
> --
> I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det
er et element?rt imperativ for alle pedagoger ? unng? eksessiv anvendelse av
idiomer med fremmed opprinnelse. I de fleste tilfeller eksisterer adekvate og
relevante synonymer p? norsk.
> 
> _______________________________________________
> OpenIndiana-discuss mailing list
> OpenIndiana-discuss at openindiana.org
> http://openindiana.org/mailman/listinfo/openindiana-discuss

On Aug 6, 2011, at 9:56 AM, Roy Sigurd Karlsbakk wrote:
>> In my experience, SATA drives behind SAS expanders just don''t
work.
>> They "fail" in the manner you
>> describe, sooner or later. Use SAS and be happy.
> 
> Funny thing is Hitachi and Seagate drives work stably, whereas WD drives
tend to fail rather quickly
For reference, there are no WD disks qualified for use on NexentaStor platforms.
 -- richard

This might be related to your issue:

http://blog.mpecsinc.ca/2010/09/western-digital-re3-series-sata-drives.html

On Saturday, August 6, 2011, Roy Sigurd Karlsbakk <roy at karlsbakk.net>
wrote:
>> In my experience, SATA drives behind SAS expanders just don''t
work.
>> They "fail" in the manner you
>> describe, sooner or later. Use SAS and be happy.
>
> Funny thing is Hitachi and Seagate drives work stably, whereas WD drives
tend to fail rather quickly
>
> Vennlige hilsener / Best regards
>
> roy
> --
> Roy Sigurd Karlsbakk
> (+47) 97542685
> roy at karlsbakk.net
> http://blogg.karlsbakk.net/
> --
> I all pedagogikk er det essensielt at pensum presenteres intelligibelt.
Det er et element?rt imperativ for alle pedagoger ? unng? eksessiv
anvendelse av idiomer med fremmed opprinnelse. I de fleste tilfeller
eksisterer adekvate og relevante synonymer p? norsk.
>
> _______________________________________________
> OpenIndiana-discuss mailing list
> OpenIndiana-discuss at openindiana.org
> http://openindiana.org/mailman/listinfo/openindiana-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20110806/c8f29e55/attachment.html>

On Aug 6, 2011, at 9:56 AM, Roy Sigurd Karlsbakk wrote:
>> In my experience, SATA drives behind SAS expanders just don''t
work.
>> They "fail" in the manner you
>> describe, sooner or later. Use SAS and be happy.
> 
> Funny thing is Hitachi and Seagate drives work stably, whereas WD drives
tend to fail rather quickly
> 
> Vennlige hilsener / Best regards
Might this be the SATA drives taking too long to reallocate bad sectors? This is
a common problem "desktop" drives have, they will stop and basically
focus on reallocating the bad sector as long as it takes, which causes the raid
setup to time out the operation and flag the drive as failed. The
"enterprise" sata drives, typically the same as the high performing
desktop drive, only they have a short timeout on how long they are allowed to
try and reallocate a bad sector so they don''t hit the failed drive
timeout. Some drive firmwares, such as older WD blacks if memory serves, had the
ability to be forced to behave like the enterprise drive, but WD updated the
firmware so this is longer possible.

This is why you see SATA drives that typically have almost identical specs, but
one will be $69 and the other $139 - the former is a "desktop" model
while the latter is an "enterprise" or "raid" specific
model. I believe it''s called different things by different brands:
TLER, ERC, and CCTL (?).

Greg

> Might this be the SATA drives taking too long to reallocate bad
> sectors? This is a common problem "desktop" drives have, they
will
> stop and basically focus on reallocating the bad sector as long as it
> takes, which causes the raid setup to time out the operation and flag
> the drive as failed. The "enterprise" sata drives, typically the
same
> as the high performing desktop drive, only they have a short timeout
> on how long they are allowed to try and reallocate a bad sector so
> they don''t hit the failed drive timeout. Some drive firmwares,
such as
> older WD blacks if memory serves, had the ability to be forced to
> behave like the enterprise drive, but WD updated the firmware so this
> is longer possible.
> 
> This is why you see SATA drives that typically have almost identical
> specs, but one will be $69 and the other $139 - the former is a
> "desktop" model while the latter is an "enterprise" or
"raid" specific
> model. I believe it''s called different things by different brands:
> TLER, ERC, and CCTL (?).
I doubt this is about the lack of TLER et al. Some, or most, of the drives
ditched by ZFS have shown to be quite good indeed. I guess this is a WD vs Intel
SAS expanders issue

Vennlige hilsener / Best regards

roy
--
Roy Sigurd Karlsbakk
(+47) 97542685
roy at karlsbakk.net
http://blogg.karlsbakk.net/
--
I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det er
et element?rt imperativ for alle pedagoger ? unng? eksessiv anvendelse av
idiomer med fremmed opprinnelse. I de fleste tilfeller eksisterer adekvate og
relevante synonymer p? norsk.

On Sat, Aug 06, 2011 at 06:45:05PM +0200, Roy Sigurd Karlsbakk
wrote:
> Have anyone here used WD drives with LSI controllers (3801/3081/9211)
> with Super Micro machines? Any success stories?
I''m using 4 x WD RE3 1TB drives with a Supermicro X7SB3 mobo with a
builtin LSI 1068E controller and a CSE-SAS-833TQ SAS backplane.

Have run ZFS with both Solaris and FreeBSD without a problem for a
couple years now.  Had one drive go bad, but it was caught early by
running periodic scrubs.

-- 
Jason Fortezzo
fortezza at mechanicalism.net

> I''m using 4 x WD RE3 1TB drives with a Supermicro X7SB3 mobo with
a
> builtin LSI 1068E controller and a CSE-SAS-833TQ SAS backplane.
> 
> Have run ZFS with both Solaris and FreeBSD without a problem for a
> couple years now. Had one drive go bad, but it was caught early by
> running periodic scrubs.
We have some 180 drives on these systems, with 1068-based chipsets, and it seems
problems arouse when traffic is high.

Vennlige hilsener / Best regards

roy
--
Roy Sigurd Karlsbakk
(+47) 97542685
roy at karlsbakk.net
http://blogg.karlsbakk.net/
--
I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det er
et element?rt imperativ for alle pedagoger ? unng? eksessiv anvendelse av
idiomer med fremmed opprinnelse. I de fleste tilfeller eksisterer adekvate og
relevante synonymer p? norsk.

On Sat, Aug 06, 2011 at 07:45:31PM +0200, Roy Sigurd Karlsbakk
wrote:
> > Might this be the SATA drives taking too long to reallocate bad
> > sectors? This is a common problem "desktop" drives have,
they will
> > stop and basically focus on reallocating the bad sector as long as it
> > takes, which causes the raid setup to time out the operation and flag
> > the drive as failed. The "enterprise" sata drives, typically
the same
> > as the high performing desktop drive, only they have a short timeout
> > on how long they are allowed to try and reallocate a bad sector so
> > they don''t hit the failed drive timeout. Some drive
firmwares, such as
> > older WD blacks if memory serves, had the ability to be forced to
> > behave like the enterprise drive, but WD updated the firmware so this
> > is longer possible.
> > 
> > This is why you see SATA drives that typically have almost identical
> > specs, but one will be $69 and the other $139 - the former is a
> > "desktop" model while the latter is an
"enterprise" or "raid" specific
> > model. I believe it''s called different things by different
brands:
> > TLER, ERC, and CCTL (?).
> 
> I doubt this is about the lack of TLER et al. Some, or most, of the drives
ditched by ZFS have shown to be quite good indeed. I guess this is a WD vs Intel
SAS expanders issue
> 
What exact chassis / backplane / SAS-expander is that? (with Intel SAS
expander).

-- Pasi