Gluster users - Feb 2018 - Gluster performance / Dell Idrac enterprise conflict

I've tested about 12 different Dell servers.  Ony a couple of them have
Idrac express and all the others have Idrac Enterprise.  All the boxes with
Enterprise perform poorly and the couple that have express perform well.  I
use the disks in raid mode on all of them.  I've tried a few non-Dell boxes
and they all perform well even though some of them are very old.  I've also
tried disabling Idrac, the Idrac nic, virtual storage for Idrac with no
sucess..

On Mon, Feb 26, 2018 at 9:28 AM, Serkan ?oban <cobanserkan at gmail.com>
wrote:
> I don't think it is related with iDRAC itself but some configuration
> is wrong or there is some hw error.
> Did you check battery of raid controller? Do you use disks in jbod
> mode or raid mode?
>
> On Mon, Feb 26, 2018 at 6:12 PM, Ryan Wilkinson <ryanwilk at
gmail.com>
> wrote:
> > Thanks for the suggestion.  I tried both of these with no difference
in
> > performance.I have tried several other Dell hosts with Idrac
Enterprise
> and
> > getting the same results.  I also tried a new Dell T130 with Idrac
> express
> > and was getting over 700 MB/s.  Any other users had this issues with
> Idrac
> > Enterprise??
> >
> >
> > On Thu, Feb 22, 2018 at 12:16 AM, Serkan ?oban <cobanserkan at
gmail.com>
> > wrote:
> >>
> >> "Did you check the BIOS/Power settings? They should be set
for high
> >> performance.
> >> Also you can try to boot "intel_idle.max_cstate=0"
kernel command line
> >> option to be sure CPUs not entering power saving states.
> >>
> >> On Thu, Feb 22, 2018 at 9:59 AM, Ryan Wilkinson <ryanwilk at
gmail.com>
> >> wrote:
> >> >
> >> >
> >> > I have a 3 host gluster replicated cluster that is providing
storage
> for
> >> > our
> >> > RHEV environment.  We've been having issues with
inconsistent
> >> > performance
> >> > from the VMs depending on which Hypervisor they are running
on.  I've
> >> > confirmed throughput to be ~9Gb/s to each of the storage
hosts from
> the
> >> > hypervisors.  I'm getting ~300MB/s disk read spead when
our test vm is
> >> > on
> >> > the slow Hypervisors and over 500 on the faster ones.  The
performance
> >> > doesn't seem to be affected much by the cpu, memory that
are in the
> >> > hypervisors.  I have tried a couple of really old boxes and
got over
> 500
> >> > MB/s.  The common thread seems to be that the poorly
perfoming hosts
> all
> >> > have Dell's Idrac 7 Enterprise.  I have one Hypervisor
that has Idrac
> 7
> >> > express and it performs well.  We've compared system
packages and
> >> > versions
> >> > til we're blue in the face and have been struggling with
this for a
> >> > couple
> >> > months but that seems to be the only common denominator. 
I've tried
> on
> >> > one
> >> > of those Idrac 7 hosts to disable the nic, virtual drive,
etc, etc.
> but
> >> > no
> >> > change in performance.  In addition, I tried 5 new hosts and
all are
> >> > complying to the Idrac enterprise theory.  Anyone else had
this
> issue?!
> >> >
> >> >
> >> >
> >> > _______________________________________________
> >> > Gluster-users mailing list
> >> > Gluster-users at gluster.org
> >> > http://lists.gluster.org/mailman/listinfo/gluster-users
> >
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20180226/e1239fba/attachment.html>

I would be really supprised if the problem was related to Idrac.

The Idrac processor is a stand alone cpu with its own nic and runs 
independent of the main CPU.

That being said it does have visibility into the whole system.

try using dmidecode to compare the systems and take a close look at the 
raid controllers and what size and form of cache they have.


On 02/26/2018 11:34 AM, Ryan Wilkinson wrote:
> I've tested about 12 different Dell servers.? Ony a couple of them 
> have Idrac express and all the others have Idrac Enterprise.? All the 
> boxes with Enterprise perform poorly and the couple that have express 
> perform well.? I use the disks in raid mode on all of them.? I've 
> tried a few non-Dell boxes and they all perform well even though some 
> of them are very old. I've also tried disabling Idrac, the Idrac nic, 
> virtual storage for Idrac with no sucess..
>
> On Mon, Feb 26, 2018 at 9:28 AM, Serkan ?oban <cobanserkan at gmail.com 
> <mailto:cobanserkan at gmail.com>> wrote:
>
>     I don't think it is related with iDRAC itself but some
configuration
>     is wrong or there is some hw error.
>     Did you check battery of raid controller? Do you use disks in jbod
>     mode or raid mode?
>
>     On Mon, Feb 26, 2018 at 6:12 PM, Ryan Wilkinson
>     <ryanwilk at gmail.com <mailto:ryanwilk at gmail.com>>
wrote:
>     > Thanks for the suggestion.? I tried both of these with no
>     difference in
>     > performance.I have tried several other Dell hosts with Idrac
>     Enterprise and
>     > getting the same results.? I also tried a new Dell T130 with
>     Idrac express
>     > and was getting over 700 MB/s.? Any other users had this issues
>     with Idrac
>     > Enterprise??
>     >
>     >
>     > On Thu, Feb 22, 2018 at 12:16 AM, Serkan ?oban
>     <cobanserkan at gmail.com <mailto:cobanserkan at
gmail.com>>
>     > wrote:
>     >>
>     >> "Did you check the BIOS/Power settings? They should be
set for high
>     >> performance.
>     >> Also you can try to boot "intel_idle.max_cstate=0"
kernel
>     command line
>     >> option to be sure CPUs not entering power saving states.
>     >>
>     >> On Thu, Feb 22, 2018 at 9:59 AM, Ryan Wilkinson
>     <ryanwilk at gmail.com <mailto:ryanwilk at gmail.com>>
>     >> wrote:
>     >> >
>     >> >
>     >> > I have a 3 host gluster replicated cluster that is
providing
>     storage for
>     >> > our
>     >> > RHEV environment.? We've been having issues with
inconsistent
>     >> > performance
>     >> > from the VMs depending on which Hypervisor they are
running
>     on.? I've
>     >> > confirmed throughput to be ~9Gb/s to each of the storage
>     hosts from the
>     >> > hypervisors.? I'm getting ~300MB/s disk read spead
when our
>     test vm is
>     >> > on
>     >> > the slow Hypervisors and over 500 on the faster ones.?
The
>     performance
>     >> > doesn't seem to be affected much by the cpu, memory
that are
>     in the
>     >> > hypervisors.? I have tried a couple of really old boxes
and
>     got over 500
>     >> > MB/s.? The common thread seems to be that the poorly
>     perfoming hosts all
>     >> > have Dell's Idrac 7 Enterprise.? I have one
Hypervisor that
>     has Idrac 7
>     >> > express and it performs well.? We've compared system
packages and
>     >> > versions
>     >> > til we're blue in the face and have been struggling
with this
>     for a
>     >> > couple
>     >> > months but that seems to be the only common denominator.?
>     I've tried on
>     >> > one
>     >> > of those Idrac 7 hosts to disable the nic, virtual drive,
>     etc, etc. but
>     >> > no
>     >> > change in performance.? In addition, I tried 5 new hosts
and
>     all are
>     >> > complying to the Idrac enterprise theory.? Anyone else
had
>     this issue?!
>     >> >
>     >> >
>     >> >
>     >> > _______________________________________________
>     >> > Gluster-users mailing list
>     >> > Gluster-users at gluster.org <mailto:Gluster-users at
gluster.org>
>     >> > http://lists.gluster.org/mailman/listinfo/gluster-users
>     <http://lists.gluster.org/mailman/listinfo/gluster-users>
>     >
>     >
>
>
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
-- 
Alvin Starr                   ||   land:  (905)513-7688
Netvel Inc.                   ||   Cell:  (416)806-0133
alvin at netvel.net              ||

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20180226/a7b2d546/attachment.html>

Here is info. about the Raid controllers.  Doesn't seem to be the culprit.

Slow host:
Name PERC H710 Mini (Embedded)
Firmware Version 21.3.4-0001
Cache Memory Size 512 MB
Fast Host:

Name PERC H310 Mini (Embedded)
Firmware Version 20.12.1-0002
Cache Memory Size 0 MB
Slow host:
Name PERC H310 Mini (Embedded)
Firmware Version 20.13.1-0002
Cache Memory Size 0 MB
Slow host:
Name PERC H310 Mini (Embedded)
Firmware Version 20.13.3-0001 Cache Memory Size 0 MB
Slow Host:
Name PERC H710 Mini (Embedded)
Firmware Version 21.3.5-0002
Cache Memory Size 512 MB
Fast Host
Perc H730
Cache Memory Size 1GB

On Mon, Feb 26, 2018 at 9:42 AM, Alvin Starr <alvin at netvel.net> wrote:
> I would be really supprised if the problem was related to Idrac.
>
> The Idrac processor is a stand alone cpu with its own nic and runs
> independent of the main CPU.
>
> That being said it does have visibility into the whole system.
>
> try using dmidecode to compare the systems and take a close look at the
> raid controllers and what size and form of cache they have.
>
> On 02/26/2018 11:34 AM, Ryan Wilkinson wrote:
>
> I've tested about 12 different Dell servers.  Ony a couple of them have
> Idrac express and all the others have Idrac Enterprise.  All the boxes with
> Enterprise perform poorly and the couple that have express perform well.  I
> use the disks in raid mode on all of them.  I've tried a few non-Dell
boxes
> and they all perform well even though some of them are very old.  I've
also
> tried disabling Idrac, the Idrac nic, virtual storage for Idrac with no
> sucess..
>
> On Mon, Feb 26, 2018 at 9:28 AM, Serkan ?oban <cobanserkan at
gmail.com>
> wrote:
>
>> I don't think it is related with iDRAC itself but some
configuration
>> is wrong or there is some hw error.
>> Did you check battery of raid controller? Do you use disks in jbod
>> mode or raid mode?
>>
>> On Mon, Feb 26, 2018 at 6:12 PM, Ryan Wilkinson <ryanwilk at
gmail.com>
>> wrote:
>> > Thanks for the suggestion.  I tried both of these with no
difference in
>> > performance.I have tried several other Dell hosts with Idrac
Enterprise
>> and
>> > getting the same results.  I also tried a new Dell T130 with Idrac
>> express
>> > and was getting over 700 MB/s.  Any other users had this issues
with
>> Idrac
>> > Enterprise??
>> >
>> >
>> > On Thu, Feb 22, 2018 at 12:16 AM, Serkan ?oban <cobanserkan at
gmail.com>
>> > wrote:
>> >>
>> >> "Did you check the BIOS/Power settings? They should be
set for high
>> >> performance.
>> >> Also you can try to boot "intel_idle.max_cstate=0"
kernel command line
>> >> option to be sure CPUs not entering power saving states.
>> >>
>> >> On Thu, Feb 22, 2018 at 9:59 AM, Ryan Wilkinson <ryanwilk
at gmail.com>
>> >> wrote:
>> >> >
>> >> >
>> >> > I have a 3 host gluster replicated cluster that is
providing storage
>> for
>> >> > our
>> >> > RHEV environment.  We've been having issues with
inconsistent
>> >> > performance
>> >> > from the VMs depending on which Hypervisor they are
running on.  I've
>> >> > confirmed throughput to be ~9Gb/s to each of the storage
hosts from
>> the
>> >> > hypervisors.  I'm getting ~300MB/s disk read spead
when our test vm
>> is
>> >> > on
>> >> > the slow Hypervisors and over 500 on the faster ones. 
The
>> performance
>> >> > doesn't seem to be affected much by the cpu, memory
that are in the
>> >> > hypervisors.  I have tried a couple of really old boxes
and got over
>> 500
>> >> > MB/s.  The common thread seems to be that the poorly
perfoming hosts
>> all
>> >> > have Dell's Idrac 7 Enterprise.  I have one
Hypervisor that has
>> Idrac 7
>> >> > express and it performs well.  We've compared system
packages and
>> >> > versions
>> >> > til we're blue in the face and have been struggling
with this for a
>> >> > couple
>> >> > months but that seems to be the only common denominator. 
I've tried
>> on
>> >> > one
>> >> > of those Idrac 7 hosts to disable the nic, virtual drive,
etc, etc.
>> but
>> >> > no
>> >> > change in performance.  In addition, I tried 5 new hosts
and all are
>> >> > complying to the Idrac enterprise theory.  Anyone else
had this
>> issue?!
>> >> >
>> >> >
>> >> >
>> >> > _______________________________________________
>> >> > Gluster-users mailing list
>> >> > Gluster-users at gluster.org
>> >> > http://lists.gluster.org/mailman/listinfo/gluster-users
>> >
>> >
>>
>
>
>
> _______________________________________________
> Gluster-users mailing listGluster-users at
gluster.orghttp://lists.gluster.org/mailman/listinfo/gluster-users
>
>
> --
> Alvin Starr                   ||   land:  (905)513-7688
<(905)%20513-7688>
> Netvel Inc.                   ||   Cell:  (416)806-0133
<(416)%20806-0133>alvin at netvel.net              ||
>
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20180226/0eb9e554/attachment.html>