Gluster users - Sep 2017 - 3.10.5 vs 3.12.0 huge performance loss

Hi,

Just do some ingestion tests to 40 node 16+4EC 19PB single volume.
100 clients are writing each has 5 threads total 500 threads.
With 3.10.5 each server has 800MB/s network traffic, cluster total is 32GB/s
With 3.12.0 each server has 200MB/s network traffic, cluster total is 8GB/s
I did not change any volume options in both configs.

Any thoughts?
Serkan

On 09/06/2017 05:48 AM, Serkan ?oban wrote:
> Hi,
> 
> Just do some ingestion tests to 40 node 16+4EC 19PB single volume.
> 100 clients are writing each has 5 threads total 500 threads.
> With 3.10.5 each server has 800MB/s network traffic, cluster total is
32GB/s
> With 3.12.0 each server has 200MB/s network traffic, cluster total is 8GB/s
> I did not change any volume options in both configs.
I just performed some *basic* IOZone tests on a 6 x (4+2) disperse 
volume and compared this against 3.10.5 and 3.12.0. The tests are no 
where near your capacity, but I do not see anything alarming in the 
results. (4 server, 4 clients, 4 worker thread per client)

I do notice a 6% drop in Sequential and random write performance, and 
gains in the sequential and random reads.

I need to improve the test to do larger files and for a longer duration, 
hence not reporting any numbers as yet.

Tests were against 3.10.5 and then a down server upgrade to 3.12.0 and 
remounting on the clients (after the versions were upgraded there as well).

I guess your test can be characterized as a sequential write workload 
(ingestion of data). What is the average file size being ingested? I can 
mimic something equivalent to that to look at this further.

I would like to ensure there are no evident performance regressions as 
you report.

Shyam

It is sequential write with file size 2GB. Same behavior observed with
3.11.3 too.

On Thu, Sep 7, 2017 at 12:43 AM, Shyam Ranganathan <srangana at
redhat.com> wrote:
> On 09/06/2017 05:48 AM, Serkan ?oban wrote:
>>
>> Hi,
>>
>> Just do some ingestion tests to 40 node 16+4EC 19PB single volume.
>> 100 clients are writing each has 5 threads total 500 threads.
>> With 3.10.5 each server has 800MB/s network traffic, cluster total is
>> 32GB/s
>> With 3.12.0 each server has 200MB/s network traffic, cluster total is
>> 8GB/s
>> I did not change any volume options in both configs.
>
>
> I just performed some *basic* IOZone tests on a 6 x (4+2) disperse volume
> and compared this against 3.10.5 and 3.12.0. The tests are no where near
> your capacity, but I do not see anything alarming in the results. (4
server,
> 4 clients, 4 worker thread per client)
>
> I do notice a 6% drop in Sequential and random write performance, and gains
> in the sequential and random reads.
>
> I need to improve the test to do larger files and for a longer duration,
> hence not reporting any numbers as yet.
>
> Tests were against 3.10.5 and then a down server upgrade to 3.12.0 and
> remounting on the clients (after the versions were upgraded there as well).
>
> I guess your test can be characterized as a sequential write workload
> (ingestion of data). What is the average file size being ingested? I can
> mimic something equivalent to that to look at this further.
>
> I would like to ensure there are no evident performance regressions as you
> report.
>
> Shyam