Strahil
2019-Jul-05 19:13 UTC
[Gluster-users] Extremely low performance - am I doing somethingwrong?
I don't know what you are trying to test, but I'm sure this test doesn't show anything meaningful. Have you tested with your apps' workload ? I have done your test and I get aprox 20MB/s, but I can asure you that the performance is way better in my VMs. Best Regards, Strahil NikolovOn Jul 5, 2019 20:17, wkmail <wkmail at bneit.com> wrote:> > > On 7/4/2019 2:28 AM, Vladimir Melnik wrote: > > So, the disk is OK and the network is OK, I'm 100% sure. > > > > Seems to be a GlusterFS-related issue. Either something needs to be > > tweaked or it's a normal performance for a replica-3 cluster. > > There is more to it than Gluster on that particular test. > > I have some addititional datapoints, since those numbers seemed low > given the long time I have played with Gluster (first install was 3.3) > > So I ran that exact test on some locally mounted hard drive sets (mdadm > RAID1- spinning metal) on Cent7 (stock)? and U18(stock) and got the > following: > > No Gluster involved. > > # for i in {1..5}; do { dd if=/dev/zero of=./test.tmp bs=1M count=10 > oflag=sync; rm -f ./test.tmp; } done > 10+0 records in > 10+0 records out > 10485760 bytes (10 MB) copied, 1.0144 s, 10.3 MB/s > 10+0 records in > 10+0 records out > 10485760 bytes (10 MB) copied, 0.791071 s, 13.3 MB/s > 10+0 records in > 10+0 records out > 10485760 bytes (10 MB) copied, 0.832186 s, 12.6 MB/s > 10+0 records in > 10+0 records out > 10485760 bytes (10 MB) copied, 0.80427 s, 13.0 MB/s > 10+0 records in > > That was reproducible over several machines with different CPUs that we > have in production. > > Performance is about 20%? better when 7200 rpm drives were involved or > when no RAID was involved but never above 18 MB/s > > Performance is also MUCH better when I use oflag=direct (roughly 2x) > > However, on a U18 VM Host testbed machine that has a seperate SSD swap > disk I get the following, even though I am writing the test.tmp file to > the metal. > > # for i in {1..5}; do { dd if=/dev/zero of=./test.tmp bs=1M count=10 > oflag=sync; rm -f ./test.tmp; } done > > 10+0 records in > 10+0 records out > 10485760 bytes (10 MB, 10 MiB) copied, 0.0949153 s, 110 MB/s > 10+0 records in > 10+0 records out > 10485760 bytes (10 MB, 10 MiB) copied, 0.0605883 s, 173 MB/s > 10+0 records in > 10+0 records out > 10485760 bytes (10 MB, 10 MiB) copied, 0.0582863 s, 180 MB/s > 10+0 records in > 10+0 records out > 10485760 bytes (10 MB, 10 MiB) copied, 0.0604369 s, 173 MB/s > 10+0 records in > 10+0 records out > 10485760 bytes (10 MB, 10 MiB) copied, 0.0598746 s, 175 MB/s > > So something else is going on with that particular test. Clearly, > buffers, elevators, cache etc count despite the oflag setting. > > For the record on the Gluster Fuse Mount (2x+1arb Volume)? on that VM > Host I do get reduced performance > > part of that is due to the gluster network being 2x1G using teaming on > that testbed, so there is a network bottleneck. > > # for i in {1..5}; do { dd if=/dev/zero of=./test.tmp bs=1M count=10 > oflag=sync; rm -f ./test.tmp; } done > 10+0 records in > 10+0 records out > 10485760 bytes (10 MB, 10 MiB) copied, 0.693351 s, 15.1 MB/s > 10+0 records in > 10+0 records out > 10485760 bytes (10 MB, 10 MiB) copied, 0.349881 s, 30.0 MB/s > 10+0 records in > 10+0 records out > 10485760 bytes (10 MB, 10 MiB) copied, 0.339699 s, 30.9 MB/s > 10+0 records in > 10+0 records out > 10485760 bytes (10 MB, 10 MiB) copied, 0.34202 s, 30.7 MB/s > 10+0 records in > 10+0 records out > 10485760 bytes (10 MB, 10 MiB) copied, 0.337904 s, 31.0 MB/s > > So the gluster fuse mount negates the advantage of that SSD swap disk > along with the obvious network bottleneck. > > But clearly we have to all agree on same valid test. > > -wk > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users
wkmail
2019-Jul-05 19:28 UTC
[Gluster-users] Extremely low performance - am I doing somethingwrong?
well if you are addressing me, that was the point of my post re the
original posters complaint.
If his chosen test gets lousy or inconsistent results on non-gluster
setups then its hard to complain about gluster absent the known Gluster
issues (i.e. network bandwidth, fuse context switching, etc)
there is more involved there.
and yes, my performance IS better inside the VMs because even though you
use the oflag for sync or direct, KVM/Qemu still caches stuff underneath
the qcow2 image.
So this is hist test on an active Gluster Rep2 + arb KVM setup run
within a qcow2 image that is doing real work.
# for i in {1..5}; do { dd if=/dev/zero of=./test.tmp bs=1M count=10
oflag=sync; rm -f ./test.tmp; } done
10+0 records in
10+0 records out
10485760 bytes (10 MB, 10 MiB) copied, 0.0206228 s, 508 MB/s
10+0 records in
10+0 records out
10485760 bytes (10 MB, 10 MiB) copied, 0.0152477 s, 688 MB/s
10+0 records in
10+0 records out
10485760 bytes (10 MB, 10 MiB) copied, 0.0149008 s, 704 MB/s
10+0 records in
10+0 records out
10485760 bytes (10 MB, 10 MiB) copied, 0.014808 s, 708 MB/s
10+0 records in
10+0 records out
10485760 bytes (10 MB, 10 MiB) copied, 0.0147982 s, 709 MB/s
On 7/5/2019 12:13 PM, Strahil wrote:> I don't know what you are trying to test, but I'm sure this test
doesn't show anything meaningful.
> Have you tested with your apps' workload ?
>
> I have done your test and I get aprox 20MB/s, but I can asure you that the
performance is way better in my VMs.
>
> Best Regards,
> Strahil NikolovOn Jul 5, 2019 20:17, wkmail <wkmail at bneit.com>
wrote:
>>
>> On 7/4/2019 2:28 AM, Vladimir Melnik wrote:
>>> So, the disk is OK and the network is OK, I'm 100% sure.
>>>
>>> Seems to be a GlusterFS-related issue. Either something needs to be
>>> tweaked or it's a normal performance for a replica-3 cluster.
>> There is more to it than Gluster on that particular test.
>>
>> I have some addititional datapoints, since those numbers seemed low
>> given the long time I have played with Gluster (first install was 3.3)
>>
>> So I ran that exact test on some locally mounted hard drive sets (mdadm
>> RAID1- spinning metal) on Cent7 (stock)? and U18(stock) and got the
>> following:
>>
>> No Gluster involved.
>>
>> # for i in {1..5}; do { dd if=/dev/zero of=./test.tmp bs=1M count=10
>> oflag=sync; rm -f ./test.tmp; } done
>> 10+0 records in
>> 10+0 records out
>> 10485760 bytes (10 MB) copied, 1.0144 s, 10.3 MB/s
>> 10+0 records in
>> 10+0 records out
>> 10485760 bytes (10 MB) copied, 0.791071 s, 13.3 MB/s
>> 10+0 records in
>> 10+0 records out
>> 10485760 bytes (10 MB) copied, 0.832186 s, 12.6 MB/s
>> 10+0 records in
>> 10+0 records out
>> 10485760 bytes (10 MB) copied, 0.80427 s, 13.0 MB/s
>> 10+0 records in
>>
>> That was reproducible over several machines with different CPUs that we
>> have in production.
>>
>> Performance is about 20%? better when 7200 rpm drives were involved or
>> when no RAID was involved but never above 18 MB/s
>>
>> Performance is also MUCH better when I use oflag=direct (roughly 2x)
>>
>> However, on a U18 VM Host testbed machine that has a seperate SSD swap
>> disk I get the following, even though I am writing the test.tmp file to
>> the metal.
>>
>> # for i in {1..5}; do { dd if=/dev/zero of=./test.tmp bs=1M count=10
>> oflag=sync; rm -f ./test.tmp; } done
>>
>> 10+0 records in
>> 10+0 records out
>> 10485760 bytes (10 MB, 10 MiB) copied, 0.0949153 s, 110 MB/s
>> 10+0 records in
>> 10+0 records out
>> 10485760 bytes (10 MB, 10 MiB) copied, 0.0605883 s, 173 MB/s
>> 10+0 records in
>> 10+0 records out
>> 10485760 bytes (10 MB, 10 MiB) copied, 0.0582863 s, 180 MB/s
>> 10+0 records in
>> 10+0 records out
>> 10485760 bytes (10 MB, 10 MiB) copied, 0.0604369 s, 173 MB/s
>> 10+0 records in
>> 10+0 records out
>> 10485760 bytes (10 MB, 10 MiB) copied, 0.0598746 s, 175 MB/s
>>
>> So something else is going on with that particular test. Clearly,
>> buffers, elevators, cache etc count despite the oflag setting.
>>
>> For the record on the Gluster Fuse Mount (2x+1arb Volume)? on that VM
>> Host I do get reduced performance
>>
>> part of that is due to the gluster network being 2x1G using teaming on
>> that testbed, so there is a network bottleneck.
>>
>> # for i in {1..5}; do { dd if=/dev/zero of=./test.tmp bs=1M count=10
>> oflag=sync; rm -f ./test.tmp; } done
>> 10+0 records in
>> 10+0 records out
>> 10485760 bytes (10 MB, 10 MiB) copied, 0.693351 s, 15.1 MB/s
>> 10+0 records in
>> 10+0 records out
>> 10485760 bytes (10 MB, 10 MiB) copied, 0.349881 s, 30.0 MB/s
>> 10+0 records in
>> 10+0 records out
>> 10485760 bytes (10 MB, 10 MiB) copied, 0.339699 s, 30.9 MB/s
>> 10+0 records in
>> 10+0 records out
>> 10485760 bytes (10 MB, 10 MiB) copied, 0.34202 s, 30.7 MB/s
>> 10+0 records in
>> 10+0 records out
>> 10485760 bytes (10 MB, 10 MiB) copied, 0.337904 s, 31.0 MB/s
>>
>> So the gluster fuse mount negates the advantage of that SSD swap disk
>> along with the obvious network bottleneck.
>>
>> But clearly we have to all agree on same valid test.
>>
>> -wk
>>
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> https://lists.gluster.org/mailman/listinfo/gluster-users