zfs discuss - Aug 2010 - PowerEdge R510 with PERC H200/H700 with ZFS

Anyone have any experience with a R510 with the PERC H200/H700 controller
with ZFS?

My perception is that Dell doesn''t play well with OpenSolaris. 

Thanks,

Geoff

>-----Original Message-----
>From: Brian Hechinger [mailto:wonko at 4amlunch.net]
>Sent: Saturday, August 07, 2010 8:10 AM
>To: Geoff Nordli
>Subject: Re: [zfs-discuss] PowerEdge R510 with PERC H200/H700 with ZFS
>
>On Sat, Aug 07, 2010 at 08:00:11AM -0700, Geoff Nordli wrote:
>> Anyone have any experience with a R510 with the PERC H200/H700
>> controller with ZFS?
>
>Not that particular setup, but I do run Solaris on a Precision 690 with
PERC 6i
>controllers.
>
>> My perception is that Dell doesn''t play well with OpenSolaris.
>
>What makes you say that?  I''ve run Solaris on quite a few Dell
boxes and
have
>never had any issues.
>
>-brian
>--
 
Hi Brian.

I am glad to hear that, because I would prefer to use a dell box.  

Is there a JBOD mode with the PERC 6i? 

It is funny how sometimes one forms these views as you gather information.  

Geoff

> From: Geoff Nordli <geoffn at gnaa.net>
> Date: Sat, 7 Aug 2010 08:39:46 -0700
> To: <zfs-discuss at opensolaris.org>
> Subject: Re: [zfs-discuss] PowerEdge R510 with PERC H200/H700 with ZFS
> 
> 
> 
>> -----Original Message-----
>> From: Brian Hechinger [mailto:wonko at 4amlunch.net]
>> Sent: Saturday, August 07, 2010 8:10 AM
>> To: Geoff Nordli
>> Subject: Re: [zfs-discuss] PowerEdge R510 with PERC H200/H700 with ZFS
>> 
>> On Sat, Aug 07, 2010 at 08:00:11AM -0700, Geoff Nordli wrote:
>>> Anyone have any experience with a R510 with the PERC H200/H700
>>> controller with ZFS?
>> 
>> Not that particular setup, but I do run Solaris on a Precision 690 with
> PERC 6i
>> controllers.
>> 
>>> My perception is that Dell doesn''t play well with
OpenSolaris.
>> 
>> What makes you say that?  I''ve run Solaris on quite a few Dell
boxes and
> have
>> never had any issues.
>> 
>> -brian
>> --
>  
> Hi Brian.
> 
> I am glad to hear that, because I would prefer to use a dell box.
> 
> Is there a JBOD mode with the PERC 6i?
> 
> It is funny how sometimes one forms these views as you gather information.
> 
> Geoff   
It is just that lots of the PERC controllers do not do JBOD very well. 
I''ve
done it several times making a RAID 0 for each drive.  Unfortunately, that
means the server has lots of RAID hardware that is not utilized very well.
Also, ZFS loves to see lots of spindles, and Dell boxes tend not to have
lots of drive bays in comparison to what you can build at a given price
point.   Of course then you have warranty / service issues to consider.

--
Terry Hull
Network Resource Group, Inc.

>-----Original Message-----
>From: Terry Hull [mailto:tah at nrg-inc.com]
>Sent: Saturday, August 07, 2010 1:12 PM
>
>> From: Geoff Nordli <geoffn at gnaa.net>
>> Date: Sat, 7 Aug 2010 08:39:46 -0700
>>
>>> From: Brian Hechinger [mailto:wonko at 4amlunch.net]
>>> Sent: Saturday, August 07, 2010 8:10 AM
>>>
>>> On Sat, Aug 07, 2010 at 08:00:11AM -0700, Geoff Nordli wrote:
>>>> Anyone have any experience with a R510 with the PERC H200/H700
>>>> controller with ZFS?
>>>
>>> Not that particular setup, but I do run Solaris on a Precision 690
>>> with
>> PERC 6i
>>> controllers.
>>>
>>>> My perception is that Dell doesn''t play well with
OpenSolaris.
>>>
>>> What makes you say that?  I''ve run Solaris on quite a few
Dell boxes
>>> and
>> have
>>> never had any issues.
>>>
>>> -brian
>>
>> Hi Brian.
>>
>> I am glad to hear that, because I would prefer to use a dell box.
>>
>> Is there a JBOD mode with the PERC 6i?
>>
>> It is funny how sometimes one forms these views as you gather
information.
>>
>> Geoff
>
>It is just that lots of the PERC controllers do not do JBOD very well.
I''ve done it
>several times making a RAID 0 for each drive.  Unfortunately, that means
the
>server has lots of RAID hardware that is not utilized very well.
>Also, ZFS loves to see lots of spindles, and Dell boxes tend not to have
lots of
>drive bays in comparison to what you can build at a given price
>point.   Of course then you have warranty / service issues to consider.
>
>--
>Terry Hull
Terry, you are right, the part that was really missing with the Dell was the
lack of spindles.  It seems the R510 can have up to 12 spindles.  

The online configurator only allows you to select SLC SSDs, which are a lot
more expensive than the MLC versions.  It would be nice to do MLC since that
works fine for L2ARC.  

I believe they have an onboard SD flash connector too.  It would be great to
be able to install the base OS onto a flash card and not waste two drives.  

Are you using the Broadcom or Intel NICs?  
 
For sure the benefit of buying name brand is the warranty/service side of
things, which is important to me.  I don''t want to spend any time
worrying/fixing boxes.  

Thanks,

Geoff

> From: Geoff Nordli <geoffn at gnaa.net>
> Date: Sat, 7 Aug 2010 14:11:37 -0700
> To: Terry Hull <tah at nrg-inc.com>, <zfs-discuss at
opensolaris.org>
> Subject: RE: [zfs-discuss] PowerEdge R510 with PERC H200/H700 with ZFS
[stuff deleted]
> 
> Terry, you are right, the part that was really missing with the Dell was
the
> lack of spindles.  It seems the R510 can have up to 12 spindles.
> 
> The online configurator only allows you to select SLC SSDs, which are a lot
> more expensive than the MLC versions.  It would be nice to do MLC since
that
> works fine for L2ARC.
> 
> I believe they have an onboard SD flash connector too.  It would be great
to
> be able to install the base OS onto a flash card and not waste two drives.
> 
> Are you using the Broadcom or Intel NICs?
>  
> For sure the benefit of buying name brand is the warranty/service side of
> things, which is important to me.  I don''t want to spend any time
> worrying/fixing boxes.
I understand that one.

I have been using both Intel and Broadcom NICs successfully.   My gut tells
me I like the Intel better, but I can''t say that is because I have had
trouble with the Broadcom.  It is just a personal preference that I probably
can''t justify.   

--
Terry

I''ve been running OpenSolaris on my Dell Precision Workstation T7500
for 9 months, and it works great. It''s my default desktop operating
system, and I''ve had zero problems with hardware compatibility.

I also have installed EON 0.600 on my Dell PowerEdge T410 (not so different from
your R510). A few words of caution:

1. Beware of the onboard controllers. The "RAID" controller on that
motherboard only works in Windows; neither Linux nor OpenSolaris can recognize
drives attached to it at all. So I was stuck running in "ATA" mode at
the beginning, which is awful in terms of performance.

2. I''d also recommend avoiding the PERC cards, in particular since it
makes drives attached to it impossible to transport to another system. Instead,
I use the SAS 6i/R controller. That''s built into the motherboard on the
PW T7500, and I got one separately for the PE T410. That works well, and is
completely fine with OpenSolaris. I''d recommend those, because then you
can be sure to get the cabling from Dell (which in the case of the PowerEdge,
was completely nonstandard). And if the card fails, they''ll replace it
ASAP, which isn''t necessarily the case with other vendors''
cards.

So aside from the RAID controller and cabling issues on the PE T410,
I''ve had nothing but good experiences in terms of Dell Precision
workstations and PowerEdge servers, running OpenSolaris.
-- 
This message posted from opensolaris.org

On 08/08/2010, at 07.23, valrhona at gmail.com wrote:
> 2. I''d also recommend avoiding the PERC cards, in particular since
it makes drives attached to it impossible to transport to another system.
Instead, I use the SAS 6i/R controller. That''s built into the
motherboard on the PW T7500, and I got one separately for the PE T410. That
works well, and is completely fine with OpenSolaris. I''d recommend
those, because then you can be sure to get the cabling from Dell (which in the
case of the PowerEdge, was completely nonstandard). And if the card fails,
they''ll replace it ASAP, which isn''t necessarily the case with
other vendors'' cards.
The PERC H200''s have full pass-through, completely bypassing the raid
controller. http://tinyurl.com/yl6zupj

 - Lasse

> From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss-
> bounces at opensolaris.org] On Behalf Of Geoff Nordli
> 
> Anyone have any experience with a R510 with the PERC H200/H700
> controller
> with ZFS?
> 
> My perception is that Dell doesn''t play well with OpenSolaris.
I have an R710...  Not quite the same, but similar.

When you say "doesn''t play nice with opensolaris,"
you''re hitting a grain of
truth but blowing it out of proportion.  It''s true that some stuff
might not
brainlessly work the way you''d like.  And it''s true that
you''ll be lacking
some functionality that you''d probably like.  But it''s easy to
quantify and
describe, and easy to describe the options of dealing with it, too:

#1  Optical drive, by default, in AHCI mode.  For some reason, apparently
using a chipset that osol can''t handle.  So put the optical drive into
ATA
or Legacy mode before booting.  Or use an external optical drive for
installation.

#2  On my R710, the perc driver wasn''t included in the sol10 DVD by
default,
so I had to download a special one from support.dell.com, and hit "5 -
insert driver disk" or whatever, during sol10 installation.  This was not
an
issue for osol installation.

#3  OMSA provides the web-based gui to manage your raid card.  Not available
in sol10/osol.  Instead, find the appropriate MegaCLI for your device ...
which is really tough to do.  This is needed if you want the ability to
replace a failed drive without rebooting.

Once you overcome these things, it works great.

> From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss-
> bounces at opensolaris.org] On Behalf Of valrhona at gmail.com
> 
> 2. I''d also recommend avoiding the PERC cards, in particular since
it
> makes drives attached to it impossible to transport to another system.
This is not true.  You can transport these drives to another system with
either a Perc, or any other form of LSI MegaRaid.

What''s more, I''ll advise staying away from generic SATA,
because they''re
impossible to transport to any system which is more advanced than generic
SATA.  ;-)  See?  It works two-ways.  And the same is true of all the other
HBA''s out there which have any intelligence in them, too ... RealTek
and so
forth.  It''s all not compatible with each other.  Realtek drives have
to
stay with realtek controllers.  LSI drives with LSI controllers.  Generic
SATA with generic controllers.  Etc.

One advantage of using the Perc, or some other HBA with battery backed
write-back cache, is a really big performance benefit for sync writes...
But not quite as big a benefit as using the dedicated log device ... But the
HBA is far lower cost than the SSD solution, and doesn''t take up a
drive
slot (or two).

>From: Edward Ned Harvey [mailto:shill at nedharvey.com]
>Sent: Sunday, August 08, 2010 8:34 PM
>
>> bounces at opensolaris.org] On Behalf Of Geoff Nordli
>>
>> Anyone have any experience with a R510 with the PERC H200/H700
>> controller with ZFS?
>>
>> My perception is that Dell doesn''t play well with OpenSolaris.
>
>I have an R710...  Not quite the same, but similar.
>
>When you say "doesn''t play nice with opensolaris,"
you''re hitting a grain
of truth
>but blowing it out of proportion.  It''s true that some stuff might
not
brainlessly
>work the way you''d like.  And it''s true that
you''ll be lacking some
functionality
>that you''d probably like.  But it''s easy to quantify and
describe, and easy
to
>describe the options of dealing with it, too:
>
>#1  Optical drive, by default, in AHCI mode.  For some reason, apparently
using a
>chipset that osol can''t handle.  So put the optical drive into ATA
or
Legacy mode
>before booting.  Or use an external optical drive for installation.
>
>#2  On my R710, the perc driver wasn''t included in the sol10 DVD by
default, so I
>had to download a special one from support.dell.com, and hit "5 -
insert
driver
>disk" or whatever, during sol10 installation.  This was not an issue
for
osol
>installation.
>
>#3  OMSA provides the web-based gui to manage your raid card.  Not
available in
>sol10/osol.  Instead, find the appropriate MegaCLI for your device ...
>which is really tough to do.  This is needed if you want the ability to
replace a
>failed drive without rebooting.
>
>Once you overcome these things, it works great.
Thanks Edward.

What did you end up using for the L2ARC?   The SSDs shown in the online
configurator are SLC based.  

Did you use the Broadcom or the optional Intel NIC?

Geoff

> From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss-
> bounces at opensolaris.org] On Behalf Of Geoff Nordli
> >
> >I have an R710...  Not quite the same, but similar.
> >
> Thanks Edward.
> 
> What did you end up using for the L2ARC?   The SSDs shown in the online
> configurator are SLC based.
> 
> Did you use the Broadcom or the optional Intel NIC?
We opted not to use L2ARC in the dell server.  Because the Sun server is our
primary server, and the Dell is only for backup.  That being said, for most
purposes I would usually recommend investing in a log device before a cache
device, and that''s what we''re using on the Sun server.  But we
did some
performance benchmarks on the Dell including the SSD before putting it into
production.  

Do you have something against SLC?

We got the broadcom based nic.  I think it was the default.  No thought was
put into this decision - If the intel nic has some advantage to offer, you
might consider it.  In my experience, as long as a nic works, there''s
very
little difference from one to the next.

>From: Edward Ned Harvey [mailto:shill at nedharvey.com]
>Sent: Monday, August 09, 2010 4:25 AM
>
>> bounces at opensolaris.org] On Behalf Of Geoff Nordli
>> >
>> >I have an R710...  Not quite the same, but similar.
>> >
>> Thanks Edward.
>>
>> What did you end up using for the L2ARC?   The SSDs shown in the online
>> configurator are SLC based.
>>
>> Did you use the Broadcom or the optional Intel NIC?
>
>We opted not to use L2ARC in the dell server.  Because the Sun server is
our
>primary server, and the Dell is only for backup.  That being said, for most
>purposes I would usually recommend investing in a log device before a cache
>device, and that''s what we''re using on the Sun server. 
But we did some
>performance benchmarks on the Dell including the SSD before putting it into
>production.
>
>Do you have something against SLC?
 
Cost of SLC is considerably more than an MLC version, which I don''t
think is
justified for an L2ARC.

This server is going to be used to host VMs via iSCSI in a lab/test system. 

We are going to use dedup with the table stored on SSD.  

The strategy I am looking at is using slower 7200 RPM disks to hold rarely
used data, and then get the "working" set running from SSDs.  
>
>We got the broadcom based nic.  I think it was the default.  No thought was
put
>into this decision - If the intel nic has some advantage to offer, you
might
>consider it.  In my experience, as long as a nic works, there''s
very little
difference
>from one to the next.
Thanks for the feedback.

Geoff

Hi.
I install solaris 10 x86 on PowerEdge R510 with PERC H700 without problem. 
8HDD configured with RAID 6.
Only question is how to monitor this controller?

Do you have any tools which allow you to monitor this controller? 
Get HDD status.

Thank you for help.

PS.
I know this is OpenSolaris not solaris group but maybe I get help here.
-- 
This message posted from opensolaris.org

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
 


On 8/7/2010 4:11 PM, Terry Hull wrote:
>
> It is just that lots of the PERC controllers do not do JBOD very well.
I''ve
> done it several times making a RAID 0 for each drive. Unfortunately, that
> means the server has lots of RAID hardware that is not utilized very well.
Doing that lets you use the cache, which is the only part of the RAID
HW that I''d worry about wasting.
> Also, ZFS loves to see lots of spindles, and Dell boxes tend not to have
> lots of drive bays in comparison to what you can build at a given price
> point.
I''ve found the R515 (the R510''s cousing with AMD processors)
to be
very interesting in this regard. It has many more drive bays than most
Dell boxes.

I''ve also priced out the IBM x3630 M3, even more drive bays in this
one.
for about %20 more.
> Of course then you have warranty / service issues to consider.
>
I don''t know what you''re needs are, but I found
dell''s 5yr onsite 10x5
NBD support to be priced very attractively. But I can live with a
machine being down till the next day, or through a weekend.

 -Kyle
> --
> Terry Hull
> Network Resource Group, Inc.
>
>
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.14 (MingW32)
 
iQEcBAEBAgAGBQJMyZFEAAoJEEADRM+bKN5wTr4IAIh1LzIcm346TVcRZdKwkbgW
EkFux2ZT8uzk/v1lXqgiDCkO0zQ/Bwpk9SsSa0KOblOxKRWPYQwj2pO30syX/QnR
82aFfhcJaWmf0H3aphoowqTTDhKRefYXgbPINaVafDV8JY8tN9d0+Tcnhv03n3pq
7Eafg+RbjaZPceZxDuNQ0xJFw+cpXvOYSFAcCB+E49actOqDIErf4A2xGL96PK7k
POu1bHN5qyIsca6t76nvuR7w8+yq6FfM4HY0KahyPhx/MXjp01N7vFyQKdLF5rGU
ByliQedo7r8OsLl6BxeMwv+SBNxab4sjqWpWfTzniLk1Ng6aG3mm5YQ7/iAUZ+0=FDkN
-----END PGP SIGNATURE-----