Gluster users - May 2017 - Poor read performance and other funny performance results

Hi, I'm trying to set up a very simple glusterfs based system, but I'm
getting some strange performance numbers.

To keep the test simple I've:
* Set up a volume on a single server
* Mounted the volume only via localhost
* Kept the files small (1-10G) so they always fit in page cache.

I hope this means that I'm testing only glusterfs performance and not other
aspects of my machine.

The host has two E5-2687W v3 @ 3.10GHz CPUs and 377 GB of ram and nothing
else running.

I'm running ubuntu server 16.04 with the glusterfs 3.10 from:
https://launchpad.net/~gluster/+archive/ubuntu/glusterfs-3.10

The script I'm using is:
#!/bin/bash


set -x

# Volume was created
using:

# gluster volume create testvol transport tcp localhost:/mnt
force

# gluster volume start
testvol




BS=1M
BC=1k


umount hest
mount -t glusterfs localhost:/testvol hest
dd if=/dev/zero of=hest/test/10g bs=$BS count=$BC
dd if=/mnt/test/10g of=/dev/null bs=$BS
dd if=/mnt/test/10g of=/dev/null bs=$BS
dd if=hest/test/10g of=/dev/null bs=$BS
dd if=hest/test/1g of=/dev/null bs=$BS


echo "Purely local test"
dd if=/dev/zero of=/mnt/fest/10g bs=$BS count=$BC
dd if=/mnt/fest/10g of=/dev/null bs=$BS
dd if=/mnt/fest/10g of=/dev/null bs=$BS


The typical results I get are:
+ BS=1M
+ BC=1k
+ umount hest
+ mount -t glusterfs localhost:/testvol hest
+ dd if=/dev/zero of=hest/test/10g bs=1M count=1k
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.824452 s, 1.3 GB/s
+ dd if=/mnt/test/10g of=/dev/null bs=1M
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.374057 s, 2.9 GB/s
+ dd if=/mnt/test/10g of=/dev/null bs=1M
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.202726 s, 5.3 GB/s
+ dd if=hest/test/10g of=/dev/null bs=1M
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 1.38545 s, 775 MB/s
+ dd if=hest/test/1g of=/dev/null bs=1M
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 1.26429 s, 849 MB/s


Purely local test
+ dd if=/dev/zero of=/mnt/fest/10g bs=1M count=1k
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.957372 s, 1.1 GB/s
+ dd if=/mnt/fest/10g of=/dev/null bs=1M
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.245052 s, 4.4 GB/s
+ dd if=/mnt/fest/10g of=/dev/null bs=1M
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.229069 s, 4.7 GB/s


I have a few questions:

1: Why is read performance so horrendously bad at 849 MB/s when I read the
file back via glusterfs after I have just read the file from the local
filesystem at 5.3 GB/s and thus ensuring that the file is in page cache for
the glusterfs server to read from?

2: When I write the file locally and read it locally I can read the file
out of page cache at more than 4 GB/s the first time I read it as well as
the second, but when I write the file via glusterfs and read it locally the
first read is much slower, indicating that the glusterfs server does
something strange wrt. caching, what's going on here?


I'm aiming to run the real system on 40Gb/s IB, so I'm a bit annoyed
that
glusterfs seems to bottleneck at 22% of line speed even on localhost,
without any real IO going on.

-- 
Flemming Frandsen - YAPH - http://osaa.dk - http://dren.dk/
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20170503/b5097427/attachment.html>