Gluster users - Apr 2017 - Small files performance on SSD

Dear Gluster community,

I have problems with tuning Gluster for small files performance on SSD.

My usage scenario is, as I've learned, worst possible scenario, but
it's not up to me to change it:
- small 1KB files
- at least 20M of those
- approx. 10 files/directory
- mostly writes
- average speed 1000 files/sec with peaks up to 10K files/sec.

I'm doing something wrong, because I cannot get past performance metrics
4K files/sec for distributed volume (2 bricks)
2K files/sec for replicated volume (2 bricks).
I've been experimenting with various XFS formatting and mounting
options and with Gluster tuning (md-cache, lookup optimize, thread,
writeback, tiering), but no luck.

I've learned that it's not disk IO that is the bottleneck (direct
tests on mounted XFS partition show waaay better results, like 100K
files/sec).

As I've learned from
http://blog.gluster.org/2014/03/experiments-using-ssds-with-gluster/
it's possible to get 24K files/sec performance (and that was three years
ago).

Test I'm using, run on one server (2 x Xeons, 256 GB RAM, 10GbE network):
smallfile_cli.py --operation create --threads 32 --file-size 1 --files
15625 --top /mnt/testdir/test

Setup:
2 servers with 2 Xeons each, 256GB RAM, 8 x 800GB SSD drives in RAID6,
10GbE network
Ubuntu 14.04
Gluster 3.7.3

Do you have any hints where I should start investigating for the bottleneck?

Best regards,
Szymon

On Fri, Apr 28, 2017 at 3:44 PM, Szymon Miotk <szymon.miotk at gmail.com>
wrote:
> Dear Gluster community,
>
> I have problems with tuning Gluster for small files performance on SSD.
>
> My usage scenario is, as I've learned, worst possible scenario, but
> it's not up to me to change it:
> - small 1KB files
> - at least 20M of those
> - approx. 10 files/directory
> - mostly writes
> - average speed 1000 files/sec with peaks up to 10K files/sec.
>
> I'm doing something wrong, because I cannot get past performance
metrics
> 4K files/sec for distributed volume (2 bricks)
> 2K files/sec for replicated volume (2 bricks).
> I've been experimenting with various XFS formatting and mounting
> options and with Gluster tuning (md-cache, lookup optimize, thread,
> writeback, tiering), but no luck.
>
> I've learned that it's not disk IO that is the bottleneck (direct
> tests on mounted XFS partition show waaay better results, like 100K
> files/sec).
>
> As I've learned from
> http://blog.gluster.org/2014/03/experiments-using-ssds-with-gluster/
> it's possible to get 24K files/sec performance (and that was three
years
> ago).
>
>
How many clients are you running? Considering Gluster is a distributed
solution, the performance should be measured as aggregate of all the
clients.

> Test I'm using, run on one server (2 x Xeons, 256 GB RAM, 10GbE
network):
> smallfile_cli.py --operation create --threads 32 --file-size 1 --files
> 15625 --top /mnt/testdir/test
>
> Setup:
> 2 servers with 2 Xeons each, 256GB RAM, 8 x 800GB SSD drives in RAID6,
> 10GbE network
> Ubuntu 14.04
> Gluster 3.7.3
>
> Do you have any hints where I should start investigating for the
> bottleneck?
>
>
Currently just the fuse mount may be the bottleneck, but I recommend
running multiple clients (from different machines) doing these operations
in parallel to get the best results out of Gluster.

-Amar

Best regards,
> Szymon
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
>


-- 
Amar Tumballi (amarts)
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20170428/9af0f914/attachment.html>