Gluster users - Sep 2017 - 3.10.5 vs 3.12.0 huge performance loss

It is sequential write with file size 2GB. Same behavior observed with
3.11.3 too.

On Thu, Sep 7, 2017 at 12:43 AM, Shyam Ranganathan <srangana at
redhat.com> wrote:
> On 09/06/2017 05:48 AM, Serkan ?oban wrote:
>>
>> Hi,
>>
>> Just do some ingestion tests to 40 node 16+4EC 19PB single volume.
>> 100 clients are writing each has 5 threads total 500 threads.
>> With 3.10.5 each server has 800MB/s network traffic, cluster total is
>> 32GB/s
>> With 3.12.0 each server has 200MB/s network traffic, cluster total is
>> 8GB/s
>> I did not change any volume options in both configs.
>
>
> I just performed some *basic* IOZone tests on a 6 x (4+2) disperse volume
> and compared this against 3.10.5 and 3.12.0. The tests are no where near
> your capacity, but I do not see anything alarming in the results. (4
server,
> 4 clients, 4 worker thread per client)
>
> I do notice a 6% drop in Sequential and random write performance, and gains
> in the sequential and random reads.
>
> I need to improve the test to do larger files and for a longer duration,
> hence not reporting any numbers as yet.
>
> Tests were against 3.10.5 and then a down server upgrade to 3.12.0 and
> remounting on the clients (after the versions were upgraded there as well).
>
> I guess your test can be characterized as a sequential write workload
> (ingestion of data). What is the average file size being ingested? I can
> mimic something equivalent to that to look at this further.
>
> I would like to ensure there are no evident performance regressions as you
> report.
>
> Shyam

Here are my results:

Summary: I am not able to reproduce the problem, IOW I get relatively 
equivalent numbers for sequential IO when going against 3.10.5 or 3.12.0

Next steps:
- Could you pass along your volfile (both for a brick and also the 
client vol file (from 
/var/lib/glusterd/vols/<yourvolname>/patchy.tcp-fuse.vol and a brick vol 
file from the same place)
   - I want to check what options are in use in your setup as compared 
to mine and see if that makes a difference

- Is it possible for you to run the IOZone test, as below (it needs more 
clarification in case you have not used IOZone before, so reach out in 
that case) in your setup and report the results?

- Details:

Test: IOZone iozone -+m /<clientlistfile> -+h <hostaddr> -C -w -c -e
-i
0 -+n -r 256k -s 2g -t 4/8/16/32
   - Sequential IOZone write tests, with -t number of threads (files) 
per client, across 4 clients, and using a 256k record size
   - Essentially 8/16/32 threads per client, which are 4 in total

Volume: 6x(4+2) disperse, on 36 disks, each disk is a SAS 10k JBOD, 
configured with the defaults, when creating the volume using 3.10.5 as a 
start point.

Server network saturation expectation: The brick nodes will get 1.5 
times the data that the client generated (4+2). As a result, aggregate 
IOZone results should be seen as, 1.5XAggregate/4 per server network 
utilization.

Results:
Threads/client	3.10.5 		3.12.0
		 (bytes/sec aggregate)
8		1938564		1922249
16		2044677		2082817
32		2465990		2435669

The brick nodes (which are separate from the client nodes) have a 
(greater than) 10G interface.

At best (32 threads/client case), I see the server link getting utilized as,
3.10.5: (2465990*1.5)/(4*1024) = 903MB/sec
3.12.0: (2465990*1.5)/(4*1024) = 892MB/sec

Shyam
On 09/07/2017 12:07 AM, Serkan ?oban wrote:
> It is sequential write with file size 2GB. Same behavior observed with
> 3.11.3 too.
> 
> On Thu, Sep 7, 2017 at 12:43 AM, Shyam Ranganathan <srangana at
redhat.com> wrote:
>> On 09/06/2017 05:48 AM, Serkan ?oban wrote:
>>>
>>> Hi,
>>>
>>> Just do some ingestion tests to 40 node 16+4EC 19PB single volume.
>>> 100 clients are writing each has 5 threads total 500 threads.
>>> With 3.10.5 each server has 800MB/s network traffic, cluster total
is
>>> 32GB/s
>>> With 3.12.0 each server has 200MB/s network traffic, cluster total
is
>>> 8GB/s
>>> I did not change any volume options in both configs.
>>
>>
>> I just performed some *basic* IOZone tests on a 6 x (4+2) disperse
volume
>> and compared this against 3.10.5 and 3.12.0. The tests are no where
near
>> your capacity, but I do not see anything alarming in the results. (4
server,
>> 4 clients, 4 worker thread per client)
>>
>> I do notice a 6% drop in Sequential and random write performance, and
gains
>> in the sequential and random reads.
>>
>> I need to improve the test to do larger files and for a longer
duration,
>> hence not reporting any numbers as yet.
>>
>> Tests were against 3.10.5 and then a down server upgrade to 3.12.0 and
>> remounting on the clients (after the versions were upgraded there as
well).
>>
>> I guess your test can be characterized as a sequential write workload
>> (ingestion of data). What is the average file size being ingested? I
can
>> mimic something equivalent to that to look at this further.
>>
>> I would like to ensure there are no evident performance regressions as
you
>> report.
>>
>> Shyam

Hi,
Servers are in production with 3.10.5, so I cannot provide 3.12
related information anymore.
Thanks for help, sorry for inconvenience.