It is sequential write with file size 2GB. Same behavior observed with 3.11.3 too. On Thu, Sep 7, 2017 at 12:43 AM, Shyam Ranganathan <srangana at redhat.com> wrote:> On 09/06/2017 05:48 AM, Serkan ?oban wrote: >> >> Hi, >> >> Just do some ingestion tests to 40 node 16+4EC 19PB single volume. >> 100 clients are writing each has 5 threads total 500 threads. >> With 3.10.5 each server has 800MB/s network traffic, cluster total is >> 32GB/s >> With 3.12.0 each server has 200MB/s network traffic, cluster total is >> 8GB/s >> I did not change any volume options in both configs. > > > I just performed some *basic* IOZone tests on a 6 x (4+2) disperse volume > and compared this against 3.10.5 and 3.12.0. The tests are no where near > your capacity, but I do not see anything alarming in the results. (4 server, > 4 clients, 4 worker thread per client) > > I do notice a 6% drop in Sequential and random write performance, and gains > in the sequential and random reads. > > I need to improve the test to do larger files and for a longer duration, > hence not reporting any numbers as yet. > > Tests were against 3.10.5 and then a down server upgrade to 3.12.0 and > remounting on the clients (after the versions were upgraded there as well). > > I guess your test can be characterized as a sequential write workload > (ingestion of data). What is the average file size being ingested? I can > mimic something equivalent to that to look at this further. > > I would like to ensure there are no evident performance regressions as you > report. > > Shyam
Shyam Ranganathan
2017-Sep-11 14:44 UTC
[Gluster-users] 3.10.5 vs 3.12.0 huge performance loss
Here are my results: Summary: I am not able to reproduce the problem, IOW I get relatively equivalent numbers for sequential IO when going against 3.10.5 or 3.12.0 Next steps: - Could you pass along your volfile (both for a brick and also the client vol file (from /var/lib/glusterd/vols/<yourvolname>/patchy.tcp-fuse.vol and a brick vol file from the same place) - I want to check what options are in use in your setup as compared to mine and see if that makes a difference - Is it possible for you to run the IOZone test, as below (it needs more clarification in case you have not used IOZone before, so reach out in that case) in your setup and report the results? - Details: Test: IOZone iozone -+m /<clientlistfile> -+h <hostaddr> -C -w -c -e -i 0 -+n -r 256k -s 2g -t 4/8/16/32 - Sequential IOZone write tests, with -t number of threads (files) per client, across 4 clients, and using a 256k record size - Essentially 8/16/32 threads per client, which are 4 in total Volume: 6x(4+2) disperse, on 36 disks, each disk is a SAS 10k JBOD, configured with the defaults, when creating the volume using 3.10.5 as a start point. Server network saturation expectation: The brick nodes will get 1.5 times the data that the client generated (4+2). As a result, aggregate IOZone results should be seen as, 1.5XAggregate/4 per server network utilization. Results: Threads/client 3.10.5 3.12.0 (bytes/sec aggregate) 8 1938564 1922249 16 2044677 2082817 32 2465990 2435669 The brick nodes (which are separate from the client nodes) have a (greater than) 10G interface. At best (32 threads/client case), I see the server link getting utilized as, 3.10.5: (2465990*1.5)/(4*1024) = 903MB/sec 3.12.0: (2465990*1.5)/(4*1024) = 892MB/sec Shyam On 09/07/2017 12:07 AM, Serkan ?oban wrote:> It is sequential write with file size 2GB. Same behavior observed with > 3.11.3 too. > > On Thu, Sep 7, 2017 at 12:43 AM, Shyam Ranganathan <srangana at redhat.com> wrote: >> On 09/06/2017 05:48 AM, Serkan ?oban wrote: >>> >>> Hi, >>> >>> Just do some ingestion tests to 40 node 16+4EC 19PB single volume. >>> 100 clients are writing each has 5 threads total 500 threads. >>> With 3.10.5 each server has 800MB/s network traffic, cluster total is >>> 32GB/s >>> With 3.12.0 each server has 200MB/s network traffic, cluster total is >>> 8GB/s >>> I did not change any volume options in both configs. >> >> >> I just performed some *basic* IOZone tests on a 6 x (4+2) disperse volume >> and compared this against 3.10.5 and 3.12.0. The tests are no where near >> your capacity, but I do not see anything alarming in the results. (4 server, >> 4 clients, 4 worker thread per client) >> >> I do notice a 6% drop in Sequential and random write performance, and gains >> in the sequential and random reads. >> >> I need to improve the test to do larger files and for a longer duration, >> hence not reporting any numbers as yet. >> >> Tests were against 3.10.5 and then a down server upgrade to 3.12.0 and >> remounting on the clients (after the versions were upgraded there as well). >> >> I guess your test can be characterized as a sequential write workload >> (ingestion of data). What is the average file size being ingested? I can >> mimic something equivalent to that to look at this further. >> >> I would like to ensure there are no evident performance regressions as you >> report. >> >> Shyam