After lot of digging today finaly figured out that it's not really
using PERC controller but some Fusion MPT. Then it wasn't clear which
tool it supports. Finally I installed lsiutil and was able to change
the cache size.
[root at dsdb1 ~]# lspci|grep LSI
02:00.0 SCSI storage controller: LSI Logic / Symbios Logic SAS1068E
PCI-Express Fusion-MPT SAS (rev 08)
[root at dsdb1 ~]# dd if=/dev/zero of=/data/big.file bs=128k count=40k
oflag=direct
1024+0 records in
1024+0 records out
134217728 bytes (134 MB) copied, 0.742517 seconds, 181 MB/s
I compared this with SW RAID mdadm that I created yesterday on one of
the servers and I get around 300MB/s. I will test out first with what
we have before destroying and testing with mdadm.
Thanks for your help that led me to this path. Another question I had
was when creating mdadm RAID does it make sense to use multipathing?
On Thu, Apr 21, 2011 at 10:30 AM, Joe Landman
<landman at scalableinformatics.com> wrote:> In the lab working on fixing a unit will be about an hour
> Please pardon brevity and spelling errors, sent from my BlackBerry
>
> -----Original Message-----
> From: Mohit Anchlia <mohitanchlia at gmail.com>
> Date: Thu, 21 Apr 2011 10:29:52
> To: <landman at scalableinformatics.com>; <gluster-users at
gluster.org>
> Subject: Re: [Gluster-users] Performance
>
> Hi Joe,
>
> When you get chance can you please at my mail? It will be helpful to
> get your advise.
>
>
> On Wed, Apr 20, 2011 at 6:16 PM, Mohit Anchlia <mohitanchlia at
gmail.com> wrote:
>> Yes indeed. As soon as I sent you last email I realized that and set
>> the write back option. Now I get 130MB/s better but still nowhere
>> close to 600MB/s as advertised or what others say one should see.
>>
>> What are your recommendations about HW choice? What is more
>> preferrable and better?
>>
>> Another question do I need to set WCE on all the disks first before
>> creating RAID0? Or can I do that after creating RAID0? I tried to set
>> WCE on existing RAID0 but it fails "change_mode_page: failed
fetching
>> page: Caching (SBC)".
>>
>> ?sdparm --set=WCE --save /dev/sda
>> ? ?/dev/sda: FUJITSU ? MBD2300RC ? ? ? ? D809
>>
>> dd if=/dev/zero of=/dev/sda bs=128k count=80k oflag=direct
>> 81920+0 records in
>> 81920+0 records out
>> 10737418240 bytes (11 GB) copied, 82.8041 seconds, 130 MB/s
>>
>> Thanks for your help as always.
>>
>> On Wed, Apr 20, 2011 at 5:53 PM, Joe Landman
>> <landman at scalableinformatics.com> wrote:
>>> On 04/20/2011 07:50 PM, Mohit Anchlia wrote:
>>>>
>>>> I did that but it looks the same. I did get an error even
though it
>>>> says write-caching is on.
>>>>
>>>> [root at dslg1 ~]# hdparm -W1 /dev/sda
>>>>
>>>> /dev/sda:
>>>> ?setting drive write-caching to 1 (on)
>>>> ?HDIO_DRIVE_CMD(setcache) failed: Invalid argument
>>>> [root at dslg1 ~]# hdparm /dev/sda
>>>
>>> You might need sdparm
>>>
>>> ? ? ? ?sdparm -a /dev/sda | grep WCE
>>>
>>> With WCE on I see
>>>
>>> ? ? ? ?[root at smash ~]# sdparm -a /dev/sda | grep WCE
>>> ? ? ? ? ?WCE ? ? ? ? 1
>>>
>>> and with it off, I see
>>>
>>> ? ? ? ?[root at smash ~]# hdparm -W0 /dev/sda
>>>
>>> ? ? ? ?/dev/sda:
>>> ? ? ? ? setting drive write-caching to 0 (off)
>>> ? ? ? ? write-caching = ?0 (off)
>>>
>>> ? ? ? ?[root at smash ~]# sdparm -a /dev/sda | grep WCE
>>> ? ? ? ? ?WCE ? ? ? ? 0
>>>
>>> You might need to change WCE using
>>>
>>> ? ? ? ?sdparm --set=WCE -a /dev/sda
>>>
>>> or similar ...
>>>
>>>>
>>>> /dev/sda:
>>>> ?readonly ? ? = ?0 (off)
>>>> ?readahead ? ?= 256 (on)
>>>> ?geometry ? ? = 36472/255/63, sectors = 585937500, start = 0
>>>> [root at dslg1 ~]# [A
>>>> [root at dslg1 ~]# dd if=/dev/zero of=/dev/sda bs=128k count=1k
oflag=direct
>>>> 1024+0 records in
>>>> 1024+0 records out
>>>> 134217728 bytes (134 MB) copied, 8.10005 seconds, 16.6 MB/s
>>>>
>>>>
>>>> On Wed, Apr 20, 2011 at 5:45 PM, Joe Landman
>>>> <landman at scalableinformatics.com> ?wrote:
>>>>>
>>>>> On 04/20/2011 07:28 PM, Mohit Anchlia wrote:
>>>>>>
>>>>>> dd of=/dev/null if=/dev/sda bs=128k count=80k
iflag=direct
>>>>>> 81920+0 records in
>>>>>> 81920+0 records out
>>>>>> 10737418240 bytes (11 GB) copied, 83.8293 seconds, 128
MB/s
>>>>>
>>>>> Ok, this is closer to what I was expecting (really ~150
MB/s would make
>>>>> more
>>>>> sense to me, but I can live with 128 MB/s).
>>>>>
>>>>> The write speed is definitely problematic. ?I am wondering
if write cache
>>>>> is
>>>>> off, and other features are turned off in strange ways.
>>>>>
>>>>> This is a 2 year old SATA disk
>>>>>
>>>>> [root at smash ~]# dd if=/dev/zero of=/dev/sda2 bs=128k
oflag=direct
>>>>> dd: writing `/dev/sda2': No space left on device
>>>>> 16379+0 records in
>>>>> 16378+0 records out
>>>>> 2146798080 bytes (2.1 GB) copied, 20.8322 s, 103 MB/s
>>>>>
>>>>> Write cache is enabled. ?Turning write cache off (might not
be so
>>>>> relevant
>>>>> for a RAID0),
>>>>>
>>>>> [root at smash ~]# hdparm -W /dev/sda
>>>>>
>>>>> /dev/sda:
>>>>> ?write-caching = ?1 (on)
>>>>> [root at smash ~]# hdparm -W0 /dev/sda
>>>>>
>>>>> /dev/sda:
>>>>> ?setting drive write-caching to 0 (off)
>>>>> ?write-caching = ?0 (off)
>>>>>
>>>>> [root at smash ~]# dd if=/dev/zero of=/dev/sda2 bs=128k
oflag=direct
>>>>> dd: writing `/dev/sda2': No space left on device
>>>>> 16379+0 records in
>>>>> 16378+0 records out
>>>>> 2146798080 bytes (2.1 GB) copied, 155.636 s, 13.8 MB/s
>>>>>
>>>>> See if you can do an
>>>>>
>>>>> ? ? ? ?hdparm -W1 /dev/sda
>>>>>
>>>>> and see if it has any impact on the write speed. ?If you
are using a
>>>>> RAID0,
>>>>> safety isn't so much on your mind anyway, so you can
see if you can
>>>>> adjust
>>>>> your cache settings. ?If this doesn't work, you might
need to get to the
>>>>> console and tell it to allow caching.
>>>>>
>>>>>
>>>>> --
>>>>> Joseph Landman, Ph.D
>>>>> Founder and CEO
>>>>> Scalable Informatics, Inc.
>>>>> email: landman at scalableinformatics.com
>>>>> web ?: http://scalableinformatics.com
>>>>> ? ? ? http://scalableinformatics.com/sicluster
>>>>> phone: +1 734 786 8423 x121
>>>>> fax ?: +1 866 888 3112
>>>>> cell : +1 734 612 4615
>>>>>
>>>
>>>
>>> --
>>> Joseph Landman, Ph.D
>>> Founder and CEO
>>> Scalable Informatics, Inc.
>>> email: landman at scalableinformatics.com
>>> web ?: http://scalableinformatics.com
>>> ? ? ? http://scalableinformatics.com/sicluster
>>> phone: +1 734 786 8423 x121
>>> fax ?: +1 866 888 3112
>>> cell : +1 734 612 4615
>>>
>>
>