Gluster users - Mar 2018 - Gluster very poor performance when copying small files (1x (2+1) = 3, SSD)

On Tue, Mar 20, 2018 at 1:55 AM, TomK <tomkcpr at mdevsys.com> wrote:
> On 3/19/2018 10:52 AM, Rik Theys wrote:
>
>> Hi,
>>
>> On 03/19/2018 03:42 PM, TomK wrote:
>>
>>> On 3/19/2018 5:42 AM, Ondrej Valousek wrote:
>>> Removing NFS or NFS Ganesha from the equation, not very impressed
on my
>>> own setup either.  For the writes it's doing, that's alot
of CPU usage
>>> in top. Seems bottle-necked via a single execution core somewhere
trying
>>> to facilitate read / writes to the other bricks.
>>>
>>> Writes to the gluster FS from within one of the gluster
participating
>>> bricks:
>>>
>>> [root at nfs01 n]# dd if=/dev/zero of=./some-file.bin
>>>
>>> 393505+0 records in
>>> 393505+0 records out
>>> 201474560 bytes (201 MB) copied, 50.034 s, 4.0 MB/s
>>>
>>
>> That's not really a fare comparison as you don't specify a
blocksize.
>> What does
>>
>> dd if=/dev/zero of=./some-file.bin bs=1M count=1000 oflag=direct
>>
>> give?
>>
>>
>> Rik
>>
>> Correct.  Higher block sizes gave me better numbers earlier.  Curious
> about improving the small file size performance though, preferrably via
> gluster tunables, if possible.
>
> Though it could be said I guess that compressing a set of large files and
> transferring them over that way is one solution.  However needed the small
> block size on dd to perhaps quickly simulate alot of small requests in a
> somewhat ok-ish way.
>
Aggregating large number of small writes by write-behind into large writes
has been merged on master:
https://github.com/gluster/glusterfs/issues/364

Would like to know whether it helps for this usecase. Note that its not
part of any release yet. So you've to build and install from repo.

Another suggestion is to run tests with turning off option
performance.write-behind-trickling-writes.

# gluster volume set <volname> performance.write-behind-trickling-writes
off

A word of caution though is if your files are too small, these suggestions
may not have much impact.

> Here's the numbers from the VM:
>
> [ Via Gluster ]
> [root at nfs01 n]# dd if=/dev/zero of=./some-file.bin bs=1M count=10000
> oflag=direct
> 10000+0 records in
> 10000+0 records out
> 10485760000 bytes (10 GB) copied, 96.3228 s, 109 MB/s
> [root at nfs01 n]# rm some-file.bin
> rm: remove regular file ?some-file.bin?? y
>
> [ Via XFS ]
> [root at nfs01 n]# cd /bricks/0/gv01/
> [root at nfs01 gv01]# dd if=/dev/zero of=./some-file.bin bs=1M count=10000
> oflag=direct
> 10000+0 records in
> 10000+0 records out
> 10485760000 bytes (10 GB) copied, 44.79 s, 234 MB/s
> [root at nfs01 gv01]#
>
>
>
> top - 12:49:48 up 1 day,  9:39,  2 users,  load average: 0.66, 1.15, 1.82
> Tasks: 165 total,   1 running, 164 sleeping,   0 stopped,   0 zombie
> %Cpu0  : 10.3 us,  9.6 sy,  0.0 ni, 28.0 id, 50.4 wa,  0.0 hi,  1.8 si,
> 0.0 st
> %Cpu1  : 13.8 us, 13.8 sy,  0.0 ni, 38.6 id, 30.0 wa,  0.0 hi,  3.8 si,
> 0.0 st
> %Cpu2  :  8.7 us,  6.9 sy,  0.0 ni, 48.7 id, 34.9 wa,  0.0 hi,  0.7 si,
> 0.0 st
> %Cpu3  : 10.6 us,  7.8 sy,  0.0 ni, 57.1 id, 24.1 wa,  0.0 hi,  0.4 si,
> 0.0 st
> KiB Mem :  3881708 total,  3543280 free,   224008 used,   114420 buff/cache
> KiB Swap:  4063228 total,  3836612 free,   226616 used.  3457708 avail Mem
>
>   PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
> 14115 root      20   0 2504832  27640   2612 S  43.5  0.7 432:10.35
> glusterfsd
>  1319 root      20   0 1269620  23780   2636 S  38.9  0.6 752:44.78
> glusterfs
>  1334 root      20   0 2694264  56988   1672 S  16.3  1.5 311:20.90
> ganesha.nfsd
> 27458 root      20   0  108984   1404    540 D   3.0  0.0   0:00.24 dd
> 14127 root      20   0 1164720   4860   1960 S   0.7  0.1   1:47.59
> glusterfs
>   750 root      20   0  389864   5528   3988 S   0.3  0.1   0:08.77 sssd_be
>
> --
> Cheers,
> Tom K.
> ------------------------------------------------------------
> -------------------------
>
> Living on earth is expensive, but it includes a free trip around the sun.
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20180320/1f7b47ea/attachment.html>

Hi Raghavendra,

> On 20 Mar 2018, at 1:55 pm, Raghavendra Gowdappa <rgowdapp at
redhat.com> wrote:
> 
> Aggregating large number of small writes by write-behind into large writes
has been merged on master:
> https://github.com/gluster/glusterfs/issues/364
<https://github.com/gluster/glusterfs/issues/364>
> 
> Would like to know whether it helps for this usecase. Note that its not
part of any release yet. So you've to build and install from repo.
Sounds interesting, not too keen to build packages at the moment but I've
added myself as a watcher to that issue on Github and once it's in a 3.x
release I'll try it and let you know.
> Another suggestion is to run tests with turning off option
performance.write-behind-trickling-writes.
> 
> # gluster volume set <volname>
performance.write-behind-trickling-writes off
> 
> A word of caution though is if your files are too small, these suggestions
may not have much impact.
I'm looking for documentation on this option but all I could really find is
in the source for write-behind.c:

if is enabled (which it is), do not hold back writes if there are no outstanding
requests.


and a note on aggregate-size stating that 

"aggregation won't happen if performance.write-behind-trickling-writes
is turned on"


What are the potentially negative performance impacts of disabling this?

--
Sam McLeod (protoporpoise on IRC)
https://smcleod.net
https://twitter.com/s_mcleod

Words are my own opinions and do not necessarily represent those of my employer
or partners.

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20180320/78b9ec28/attachment.html>

On Tue, Mar 20, 2018 at 8:57 AM, Sam McLeod <mailinglists at smcleod.net>
wrote:
> Hi Raghavendra,
>
>
> On 20 Mar 2018, at 1:55 pm, Raghavendra Gowdappa <rgowdapp at
redhat.com>
> wrote:
>
> Aggregating large number of small writes by write-behind into large writes
> has been merged on master:
> https://github.com/gluster/glusterfs/issues/364
>
> Would like to know whether it helps for this usecase. Note that its not
> part of any release yet. So you've to build and install from repo.
>
>
> Sounds interesting, not too keen to build packages at the moment but
I've
> added myself as a watcher to that issue on Github and once it's in a
3.x
> release I'll try it and let you know.
>
> Another suggestion is to run tests with turning off option
> performance.write-behind-trickling-writes.
>
> # gluster volume set <volname>
performance.write-behind-trickling-writes
> off
>
> A word of caution though is if your files are too small, these suggestions
> may not have much impact.
>
>
> I'm looking for documentation on this option but all I could really
find
> is in the source for write-behind.c:
>
> if is enabled (which it is), do not hold back writes if there are no
> outstanding requests.
>
Till recently this functionality though was available, couldn't be
configured from cli. One could change this option by editing volume
configuration file. However, now its configurable through cli:

https://review.gluster.org/#/c/18719/

>
> and a note on aggregate-size stating that
>
> *"aggregation won't happen if
performance.write-behind-trickling-writes is
> turned on"*
>
>
> What are the potentially negative performance impacts of disabling this?
>
Even if aggregation option is turned off, write-behind has the capacity to
aggregate till a size of 128KB. But, to completely make use of this in case
of small write workloads write-behind has to wait for sometime so that
there are enough number of write-requests to fill the capacity. With this
option enabled, write-behind though aggregates existing requests, won't
wait for future writes. This means descendant xlators of write-behind can
see writes smaller than 128K. So, for a scenario where small number of
large writes are preferred over large number of small sized writes, this
can be a problem.

> --
> Sam McLeod (protoporpoise on IRC)
> https://smcleod.net
> https://twitter.com/s_mcleod
>
> Words are my own opinions and do not necessarily represent those of
> my employer or partners.
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20180320/1ccd96ef/attachment.html>