Pranith Kumar Karampuri
2017-May-05 14:58 UTC
[Gluster-users] Slow write times to gluster disk
hi Pat,
Let us concentrate on the performance numbers part for now. We will
look at the permissions one after this?
As per the profile info, only 2.6% of the work-load is writes. There are
too many Lookups.
Would it be possible to get the data for just the dd test you were doing
earlier?
On Fri, May 5, 2017 at 8:14 PM, Pat Haley <phaley at mit.edu> wrote:
>
> Hi Pranith & Ravi,
>
> A couple of quick questions
>
> We have profile turned on. Are there specific queries we should make that
> would help debug our configuration? (The default profile info was
> previously sent in http://lists.gluster.org/pipermail/gluster-users/2017-
> May/030840.html but I'm not sure if that is what you were looking for.)
>
> We also started to do a test on serving gluster over NFS. We rediscovered
> an issue we previously reported ( http://lists.gluster.org/
> pipermail/gluster-users/2016-September/028289.html ) in that the NFS
> mounted version was ignoring the group write permissions. What specific
> information would be useful in debugging this?
>
> Thanks
>
> Pat
>
>
>
> On 04/14/2017 03:01 AM, Ravishankar N wrote:
>
> On 04/14/2017 12:20 PM, Pranith Kumar Karampuri wrote:
>
>
>
> On Sat, Apr 8, 2017 at 10:28 AM, Ravishankar N <ravishankar at
redhat.com>
> wrote:
>
>> Hi Pat,
>>
>> I'm assuming you are using gluster native (fuse mount). If it
helps, you
>> could try mounting it via gluster NFS (gnfs) and then see if there is
an
>> improvement in speed. Fuse mounts are slower than gnfs mounts but you
get
>> the benefit of avoiding a single point of failure. Unlike fuse mounts,
if
>> the gluster node containing the gnfs server goes down, all mounts done
>> using that node will fail). For fuse mounts, you could try tweaking the
>> write-behind xlator settings to see if it helps. See the
>> performance.write-behind and performance.write-behind-window-size
>> options in `gluster volume set help`. Of course, even for gnfs mounts,
you
>> can achieve fail-over by using CTDB.
>>
>
> Ravi,
> Do you have any data that suggests fuse mounts are slower than gNFS
> servers?
>
> I have heard anecdotal evidence time and again on the ML and IRC, which is
> why I wanted to compare it with NFS numbers on his setup.
>
>
> Pat,
> I see that I am late to the thread, but do you happen to have
> "profile info" of the workload?
>
> You can follow https://gluster.readthedocs.io/en/latest/Administrator%
> 20Guide/Monitoring%20Workload/ to get the information.
>
> Yeah, Let's see if profile info shows up anything interesting.
> -Ravi
>
>
>
>>
>> Thanks,
>> Ravi
>>
>>
>> On 04/08/2017 12:07 AM, Pat Haley wrote:
>>
>>
>> Hi,
>>
>> We noticed a dramatic slowness when writing to a gluster disk when
>> compared to writing to an NFS disk. Specifically when using dd (data
>> duplicator) to write a 4.3 GB file of zeros:
>>
>> - on NFS disk (/home): 9.5 Gb/s
>> - on gluster disk (/gdata): 508 Mb/s
>>
>> The gluser disk is 2 bricks joined together, no replication or anything
>> else. The hardware is (literally) the same:
>>
>> - one server with 70 hard disks and a hardware RAID card.
>> - 4 disks in a RAID-6 group (the NFS disk)
>> - 32 disks in a RAID-6 group (the max allowed by the card,
>> /mnt/brick1)
>> - 32 disks in another RAID-6 group (/mnt/brick2)
>> - 2 hot spare
>>
>> Some additional information and more tests results (after changing the
>> log level):
>>
>> glusterfs 3.7.11 built on Apr 27 2016 14:09:22
>> CentOS release 6.8 (Final)
>> RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS-3 3108
>> [Invader] (rev 02)
>>
>>
>>
>> *Create the file to /gdata (gluster)*
>> [root at mseas-data2 gdata]# dd if=/dev/zero of=/gdata/zero1 bs=1M
>> count=1000
>> 1000+0 records in
>> 1000+0 records out
>> 1048576000 bytes (1.0 GB) copied, 1.91876 s, *546 MB/s*
>>
>> *Create the file to /home (ext4)*
>> [root at mseas-data2 gdata]# dd if=/dev/zero of=/home/zero1 bs=1M
count=1000
>> 1000+0 records in
>> 1000+0 records out
>> 1048576000 bytes (1.0 GB) copied, 0.686021 s, *1.5 GB/s - *3 times as
>> fast
>>
>>
>>
>> * Copy from /gdata to /gdata (gluster to gluster) *[root at mseas-data2
>> gdata]# dd if=/gdata/zero1 of=/gdata/zero2
>> 2048000+0 records in
>> 2048000+0 records out
>> 1048576000 bytes (1.0 GB) copied, 101.052 s, *10.4 MB/s* - realllyyy
>> slooowww
>>
>>
>> *Copy from /gdata to /gdata* *2nd time (gluster to gluster)*
>> [root at mseas-data2 gdata]# dd if=/gdata/zero1 of=/gdata/zero2
>> 2048000+0 records in
>> 2048000+0 records out
>> 1048576000 bytes (1.0 GB) copied, 92.4904 s, *11.3 MB/s* - realllyyy
>> slooowww again
>>
>>
>>
>> *Copy from /home to /home (ext4 to ext4)*
>> [root at mseas-data2 gdata]# dd if=/home/zero1 of=/home/zero2
>> 2048000+0 records in
>> 2048000+0 records out
>> 1048576000 bytes (1.0 GB) copied, 3.53263 s, *297 MB/s *30 times as
fast
>>
>>
>> *Copy from /home to /home (ext4 to ext4)*
>> [root at mseas-data2 gdata]# dd if=/home/zero1 of=/home/zero3
>> 2048000+0 records in
>> 2048000+0 records out
>> 1048576000 bytes (1.0 GB) copied, 4.1737 s, *251 MB/s* - 30 times as
fast
>>
>>
>> As a test, can we copy data directly to the xfs mountpoint
(/mnt/brick1)
>> and bypass gluster?
>>
>>
>> Any help you could give us would be appreciated.
>>
>> Thanks
>>
>> --
>>
>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>> Pat Haley Email: phaley at mit.edu
>> Center for Ocean Engineering Phone: (617) 253-6824
>> Dept. of Mechanical Engineering Fax: (617) 253-8125
>> MIT, Room 5-213 http://web.mit.edu/phaley/www/
>> 77 Massachusetts Avenue
>> Cambridge, MA 02139-4301
>>
>> _______________________________________________
>> Gluster-users mailing listGluster-users at
gluster.orghttp://lists.gluster.org/mailman/listinfo/gluster-users
>>
>> _______________________________________________ Gluster-users mailing
>> list Gluster-users at gluster.org http://lists.gluster.org/mailm
>> an/listinfo/gluster-users
>
> --
> Pranith
>
> --
>
> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
> Pat Haley Email: phaley at mit.edu
> Center for Ocean Engineering Phone: (617) 253-6824
> Dept. of Mechanical Engineering Fax: (617) 253-8125
> MIT, Room 5-213 http://web.mit.edu/phaley/www/
> 77 Massachusetts Avenue
> Cambridge, MA 02139-4301
>
>
--
Pranith
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20170505/bb14d44f/attachment.html>
Hi Pranith, I presume you are asking for some version of the profile data that just shows the dd test (or a repeat of the dd test). If yes, how do I extract just that data? Thanks Pat On 05/05/2017 10:58 AM, Pranith Kumar Karampuri wrote:> hi Pat, > Let us concentrate on the performance numbers part for now. We > will look at the permissions one after this? > > As per the profile info, only 2.6% of the work-load is writes. There > are too many Lookups. > > Would it be possible to get the data for just the dd test you were > doing earlier? > > > On Fri, May 5, 2017 at 8:14 PM, Pat Haley <phaley at mit.edu > <mailto:phaley at mit.edu>> wrote: > > > Hi Pranith & Ravi, > > A couple of quick questions > > We have profile turned on. Are there specific queries we should > make that would help debug our configuration? (The default > profile info was previously sent in > http://lists.gluster.org/pipermail/gluster-users/2017-May/030840.html > <http://lists.gluster.org/pipermail/gluster-users/2017-May/030840.html> > but I'm not sure if that is what you were looking for.) > > We also started to do a test on serving gluster over NFS. We > rediscovered an issue we previously reported ( > http://lists.gluster.org/pipermail/gluster-users/2016-September/028289.html > <http://lists.gluster.org/pipermail/gluster-users/2016-September/028289.html> > ) in that the NFS mounted version was ignoring the group write > permissions. What specific information would be useful in > debugging this? > > Thanks > > Pat > > > > On 04/14/2017 03:01 AM, Ravishankar N wrote: >> On 04/14/2017 12:20 PM, Pranith Kumar Karampuri wrote: >>> >>> >>> On Sat, Apr 8, 2017 at 10:28 AM, Ravishankar N >>> <ravishankar at redhat.com <mailto:ravishankar at redhat.com>> wrote: >>> >>> Hi Pat, >>> >>> I'm assuming you are using gluster native (fuse mount). If >>> it helps, you could try mounting it via gluster NFS (gnfs) >>> and then see if there is an improvement in speed. Fuse >>> mounts are slower than gnfs mounts but you get the benefit >>> of avoiding a single point of failure. Unlike fuse mounts, >>> if the gluster node containing the gnfs server goes down, >>> all mounts done using that node will fail). For fuse mounts, >>> you could try tweaking the write-behind xlator settings to >>> see if it helps. See the performance.write-behind and >>> performance.write-behind-window-size options in `gluster >>> volume set help`. Of course, even for gnfs mounts, you can >>> achieve fail-over by using CTDB. >>> >>> >>> Ravi, >>> Do you have any data that suggests fuse mounts are slower >>> than gNFS servers? >> I have heard anecdotal evidence time and again on the ML and IRC, >> which is why I wanted to compare it with NFS numbers on his setup. >>> >>> Pat, >>> I see that I am late to the thread, but do you happen to >>> have "profile info" of the workload? >>> >>> You can follow >>> https://gluster.readthedocs.io/en/latest/Administrator%20Guide/Monitoring%20Workload/ >>> <https://gluster.readthedocs.io/en/latest/Administrator%20Guide/Monitoring%20Workload/> >>> to get the information. >> Yeah, Let's see if profile info shows up anything interesting. >> -Ravi >>> >>> >>> Thanks, >>> Ravi >>> >>> >>> On 04/08/2017 12:07 AM, Pat Haley wrote: >>>> >>>> Hi, >>>> >>>> We noticed a dramatic slowness when writing to a gluster >>>> disk when compared to writing to an NFS disk. Specifically >>>> when using dd (data duplicator) to write a 4.3 GB file of >>>> zeros: >>>> >>>> * on NFS disk (/home): 9.5 Gb/s >>>> * on gluster disk (/gdata): 508 Mb/s >>>> >>>> The gluser disk is 2 bricks joined together, no replication >>>> or anything else. The hardware is (literally) the same: >>>> >>>> * one server with 70 hard disks and a hardware RAID card. >>>> * 4 disks in a RAID-6 group (the NFS disk) >>>> * 32 disks in a RAID-6 group (the max allowed by the >>>> card, /mnt/brick1) >>>> * 32 disks in another RAID-6 group (/mnt/brick2) >>>> * 2 hot spare >>>> >>>> Some additional information and more tests results (after >>>> changing the log level): >>>> >>>> glusterfs 3.7.11 built on Apr 27 2016 14:09:22 >>>> CentOS release 6.8 (Final) >>>> RAID bus controller: LSI Logic / Symbios Logic MegaRAID >>>> SAS-3 3108 [Invader] (rev 02) >>>> >>>> >>>> >>>> *Create the file to /gdata (gluster)* >>>> [root at mseas-data2 gdata]# dd if=/dev/zero of=/gdata/zero1 >>>> bs=1M count=1000 >>>> 1000+0 records in >>>> 1000+0 records out >>>> 1048576000 bytes (1.0 GB) copied, 1.91876 s, *546 MB/s* >>>> >>>> *Create the file to /home (ext4)* >>>> [root at mseas-data2 gdata]# dd if=/dev/zero of=/home/zero1 >>>> bs=1M count=1000 >>>> 1000+0 records in >>>> 1000+0 records out >>>> 1048576000 bytes (1.0 GB) copied, 0.686021 s, *1.5 GB/s - >>>> *3 times as fast* >>>> >>>> >>>> Copy from /gdata to /gdata (gluster to gluster) >>>> *[root at mseas-data2 gdata]# dd if=/gdata/zero1 of=/gdata/zero2 >>>> 2048000+0 records in >>>> 2048000+0 records out >>>> 1048576000 bytes (1.0 GB) copied, 101.052 s, *10.4 MB/s* - >>>> realllyyy slooowww >>>> >>>> >>>> *Copy from /gdata to /gdata* *2nd time *(gluster to gluster)** >>>> [root at mseas-data2 gdata]# dd if=/gdata/zero1 of=/gdata/zero2 >>>> 2048000+0 records in >>>> 2048000+0 records out >>>> 1048576000 bytes (1.0 GB) copied, 92.4904 s, *11.3 MB/s* - >>>> realllyyy slooowww again >>>> >>>> >>>> >>>> *Copy from /home to /home (ext4 to ext4)* >>>> [root at mseas-data2 gdata]# dd if=/home/zero1 of=/home/zero2 >>>> 2048000+0 records in >>>> 2048000+0 records out >>>> 1048576000 bytes (1.0 GB) copied, 3.53263 s, *297 MB/s *30 >>>> times as fast >>>> >>>> >>>> *Copy from /home to /home (ext4 to ext4)* >>>> [root at mseas-data2 gdata]# dd if=/home/zero1 of=/home/zero3 >>>> 2048000+0 records in >>>> 2048000+0 records out >>>> 1048576000 bytes (1.0 GB) copied, 4.1737 s, *251 MB/s* - 30 >>>> times as fast >>>> >>>> >>>> As a test, can we copy data directly to the xfs mountpoint >>>> (/mnt/brick1) and bypass gluster? >>>> >>>> >>>> Any help you could give us would be appreciated. >>>> >>>> Thanks >>>> >>>> -- >>>> >>>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- >>>> Pat Haley Email:phaley at mit.edu <mailto:phaley at mit.edu> >>>> Center for Ocean Engineering Phone: (617) 253-6824 >>>> Dept. of Mechanical Engineering Fax: (617) 253-8125 >>>> MIT, Room 5-213http://web.mit.edu/phaley/www/ >>>> 77 Massachusetts Avenue >>>> Cambridge, MA 02139-4301 >>>> >>>> _______________________________________________ >>>> Gluster-users mailing list >>>> Gluster-users at gluster.org <mailto:Gluster-users at gluster.org> >>>> http://lists.gluster.org/mailman/listinfo/gluster-users >>>> <http://lists.gluster.org/mailman/listinfo/gluster-users> >>> >>> _______________________________________________ >>> Gluster-users mailing list Gluster-users at gluster.org >>> <mailto:Gluster-users at gluster.org> >>> http://lists.gluster.org/mailman/listinfo/gluster-users >>> <http://lists.gluster.org/mailman/listinfo/gluster-users> >>> >>> -- >>> Pranith >> > -- > > -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- > Pat Haley Email:phaley at mit.edu <mailto:phaley at mit.edu> > Center for Ocean Engineering Phone: (617) 253-6824 > Dept. of Mechanical Engineering Fax: (617) 253-8125 > MIT, Room 5-213http://web.mit.edu/phaley/www/ > 77 Massachusetts Avenue > Cambridge, MA 02139-4301 > > -- > Pranith-- -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Pat Haley Email: phaley at mit.edu Center for Ocean Engineering Phone: (617) 253-6824 Dept. of Mechanical Engineering Fax: (617) 253-8125 MIT, Room 5-213 http://web.mit.edu/phaley/www/ 77 Massachusetts Avenue Cambridge, MA 02139-4301 -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170505/a94b18f6/attachment.html>