Gluster users - May 2017 - Bad perf for small files on large EC volume

Hi

We've got 3 servers with 60 drives each setup with an EC volume running on
gluster 3.10.0
The servers are connected via 10gigE.

We've done the changes recommended here :
https://bugzilla.redhat.com/show_bug.cgi?id=1349953#c17 and we're able to
max out the network with the iozone tests referenced in the same ticket.

However for small files we are getting 3-5 MB/s with the smallfile_cli.py
tool. For instance:
python smallfile_cli.py --operation create --threads 32 --file-size 100
--files 1000 --top /tmp/dfs-archive-001/
.
.
total threads = 32
total files = 31294
total data =     2.984 GB
 97.79% of requested files processed, minimum is  90.00
785.542908 sec elapsed time
39.837416 files/sec
39.837416 IOPS
3.890373 MB/sec
.

We're going to use these servers for archive purposes, so the files will be
moved there and accessed very little. After noticing our migration tool
performing very badly we did some analyses on the data actually being moved
:

Bucket 31808791 (16.27 GB) :: 0 bytes - 1.00 KB
Bucket 49448258 (122.89 GB) :: 1.00 KB - 5.00 KB
Bucket 13382242 (96.92 GB) :: 5.00 KB - 10.00 KB
Bucket 13557684 (195.15 GB) :: 10.00 KB - 20.00 KB
Bucket 22735245 (764.96 GB) :: 20.00 KB - 50.00 KB
Bucket 15101878 (1041.56 GB) :: 50.00 KB - 100.00 KB
Bucket 10734103 (1558.35 GB) :: 100.00 KB - 200.00 KB
Bucket 17695285 (5773.74 GB) :: 200.00 KB - 500.00 KB
Bucket 13632394 (10039.92 GB) :: 500.00 KB - 1.00 MB
Bucket 21815815 (32641.81 GB) :: 1.00 MB - 2.00 MB
Bucket 36940815 (117683.33 GB) :: 2.00 MB - 5.00 MB
Bucket 13580667 (91899.10 GB) :: 5.00 MB - 10.00 MB
Bucket 10945768 (232316.33 GB) :: 10.00 MB - 50.00 MB
Bucket 1723848 (542581.89 GB) :: 50.00 MB - 9223372036.85 GB

So it turns out we've got a very large number of very small files being
written to this volume.
I've attached the volume config and 2 profiling runs so if someone wants to
take a look and maybe give us some hints in terms of what volume settings
will be best for writing a lot of small files that would be much
appreciated.

kind regards
ingard
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20170508/9c57a708/attachment.html>

[This email is either empty or too large to be displayed at this time]