thr3ads.net - Gluster users - [Gluster-users] gluster client performance [Jul 2011]

If this information is useful, please help other people find it:
Share via:

John Lalande

2011-Jul-25 22:12 UTC

[Gluster-users] gluster client performance

Hi-

I'm new to Gluster, but am trying to get it set up on a new compute
cluster we're building. We picked Gluster for one of our cluster file
systems (we're also using Lustre for fast scratch space), but the
Gluster performance has been so bad that I think maybe we have a
configuration problem -- perhaps we're missing a tuning parameter that
would help, but I can't find anything in the Gluster documentation --
all the tuning info I've found seems geared toward Gluster 2.x.

For some background, our compute cluster has 64 compute nodes. The
gluster storage pool has 10 Dell PowerEdge R515 servers, each with 12 x
2 TB disks. We have another 16 Dell PowerEdge R515s used as Lustre
storage servers. The compute and storage nodes are all connected via QDR
Infiniband. Both Gluster and Lustre are set to use RDMA over Infiniband.
We are using OFED version 1.5.2-20101219, Gluster 3.2.2 and CentOS 5.5
on both the compute and storage nodes.

Oddly, it seems like there's some sort of bottleneck on the client side
-- for example, we're only seeing about 50 MB/s write throughput from a
single compute node when writing a 10GB file. But, if we run multiple
simultaneous writes from multiple compute nodes to the same Gluster
volume, we get 50 MB/s from each compute node. However, running multiple
writes from the same compute node does not increase throughput. The
compute nodes have 48 cores and 128 GB RAM, so I don't think the issue
is with the compute node hardware.

With Lustre, on the same hardware, with the same version of OFED, we're
seeing write throughput on that same 10 GB file as follows: 476 MB/s
single stream write from a single compute node and aggregate performance
of more like 2.4 GB/s if we run simultaneous writes. That leads me to
believe that we don't have a problem with RDMA, otherwise Lustre, which
is also using RDMA, should be similarly affected.

We have tried both xfs and ext4 for the backend file system on the
Gluster storage nodes (we're currently using ext4). We went with
distributed (not distributed striped) for the Gluster volume -- the
thought was that if there was a catastrophic failure of one of the
storage nodes, we'd only lose the data on that node; presumably with
distributed striped you'd lose any data striped across that volume,
unless I have misinterpreted the documentation.

So ... what's expected/normal throughput for Gluster over QDR IB to a
relatively large storage pool (10 servers / 120 disks)? Does anyone have
suggested tuning tips for improving performance?

Thanks!

John

________________________________________________________

John Lalande
University of Wisconsin-Madison
Space Science & Engineering Center
1225 W. Dayton Street, Room 439, Madison, WI 53706
608-263-2268 / john.lalande at ssec.wisc.edu

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 9431 bytes
Desc: S/MIME Cryptographic Signature
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20110725/a2058196/attachment.bin>

Sabuj Pattanayek

2011-Jul-25 23:15 UTC

head link

[Gluster-users] gluster client performance

Hi,

Here's our QDR IB gluster setup:

http://piranha.structbio.vanderbilt.edu

We're still using gluster 3.0 on all our servers and clients as well
as CENTOS5.6 kernels and ofed 1.4. To simulate a single stream I use
this nfsSpeedTest script I wrote :

http://code.google.com/p/nfsspeedtest/
>From a single QDR IB connected client to our /pirstripe directorywhich is a stripe of the gluster storage servers, this is the
performance I get (note use a file size > amount of RAM on client and
server systems, 13GB in this case) :

4k block size :

111 pir4:/pirstripe% /sb/admin/scripts/nfsSpeedTest -s 13g -y
pir4: Write test (dd): 142.281 MB/s 1138.247 mbps 93.561 seconds
pir4: Read test (dd): 274.321 MB/s 2194.570 mbps 48.527 seconds

testing from 8k - 128k block size on the dd, best performance was
achieved at 64k block sizes:

114 pir4:/pirstripe% /sb/admin/scripts/nfsSpeedTest -s 13g -b 64k -y
pir4: Write test (dd): 213.344 MB/s 1706.750 mbps 62.397 seconds
pir4: Read test (dd): 955.328 MB/s 7642.620 mbps 13.934 seconds

This is to the /pirdist directories which are mounted in distribute
mode (file is written to only one of the gluster servers) :

105 pir4:/pirdist% /sb/admin/scripts/nfsSpeedTest -s 13g -y
pir4: Write test (dd): 182.410 MB/s 1459.281 mbps 72.978 seconds
pir4: Read test (dd): 244.379 MB/s 1955.033 mbps 54.473 seconds

106 pir4:/pirdist% /sb/admin/scripts/nfsSpeedTest -s 13g -y -b 64k
pir4: Write test (dd): 204.297 MB/s 1634.375 mbps 65.160 seconds
pir4: Read test (dd): 340.427 MB/s 2723.419 mbps 39.104 seconds

For reference/control, here's the same test writing straight to the
XFS filesystem on one of the gluster storage nodes:

[sabujp at gluster1 tmp]$ /sb/admin/scripts/nfsSpeedTest -s 13g -y
gluster1: Write test (dd): 398.971 MB/s 3191.770 mbps 33.366 seconds
gluster1: Read test (dd): 234.563 MB/s 1876.501 mbps 56.752 seconds

[sabujp at gluster1 tmp]$ /sb/admin/scripts/nfsSpeedTest -s 13g -y -b 64k
gluster1: Write test (dd): 442.251 MB/s 3538.008 mbps 30.101 seconds
gluster1: Read test (dd): 219.708 MB/s 1757.660 mbps 60.590 seconds

The read test seems to scale linearly with the # of storage servers
(almost 1GB/s!). Interestingly, the /pirdist read test at 64k block
size was 120MB/s faster than the read test straight from XFS, however,
it could have been that gluster1 was busy and when I read from
/pirdist the file was actually being read from one of the other 4 less
busy storage nodes.

Here's our storage node setup (many of these settings may not apply to v3.2)
:

####

volume posix-stripe
  type storage/posix
  option directory /export/gluster1/stripe
end-volume

volume posix-distribute
        type storage/posix
        option directory /export/gluster1/distribute
end-volume

volume locks
  type features/locks
  subvolumes posix-stripe
end-volume

volume locks-dist
  type features/locks
  subvolumes posix-distribute
end-volume

volume iothreads
  type performance/io-threads
  option thread-count 16
  subvolumes locks
end-volume

volume iothreads-dist
  type performance/io-threads
  option thread-count 16
  subvolumes locks-dist
end-volume

volume server
  type protocol/server
  option transport-type ib-verbs
  option auth.addr.iothreads.allow 10.2.178.*
  option auth.addr.iothreads-dist.allow 10.2.178.*
  option auth.addr.locks.allow 10.2.178.*
  option auth.addr.posix-stripe.allow 10.2.178.*
  subvolumes iothreads iothreads-dist locks posix-stripe
end-volume

####

Here's our stripe client setup :

####

volume client-stripe-1
  type protocol/client
  option transport-type ib-verbs
  option remote-host gluster1
  option remote-subvolume iothreads
end-volume

volume client-stripe-2
  type protocol/client
  option transport-type ib-verbs
  option remote-host gluster2
  option remote-subvolume iothreads
end-volume

volume client-stripe-3
  type protocol/client
  option transport-type ib-verbs
  option remote-host gluster3
  option remote-subvolume iothreads
end-volume

volume client-stripe-4
  type protocol/client
  option transport-type ib-verbs
  option remote-host gluster4
  option remote-subvolume iothreads
end-volume

volume client-stripe-5
  type protocol/client
  option transport-type ib-verbs
  option remote-host gluster5
  option remote-subvolume iothreads
end-volume

volume readahead-gluster1
  type performance/read-ahead
  option page-count 4           # 2 is default
  option force-atime-update off # default is off
  subvolumes client-stripe-1
end-volume

volume readahead-gluster2
  type performance/read-ahead
  option page-count 4           # 2 is default
  option force-atime-update off # default is off
  subvolumes client-stripe-2
end-volume

volume readahead-gluster3
  type performance/read-ahead
  option page-count 4           # 2 is default
  option force-atime-update off # default is off
  subvolumes client-stripe-3
end-volume

volume readahead-gluster4
  type performance/read-ahead
  option page-count 4           # 2 is default
  option force-atime-update off # default is off
  subvolumes client-stripe-4
end-volume
	
volume readahead-gluster5
  type performance/read-ahead
  option page-count 4           # 2 is default
  option force-atime-update off # default is off
  subvolumes client-stripe-5
end-volume

volume writebehind-gluster1
  type performance/write-behind
  option flush-behind on
  subvolumes readahead-gluster1
end-volume

volume writebehind-gluster2
  type performance/write-behind
  option flush-behind on
  subvolumes readahead-gluster2
end-volume

volume writebehind-gluster3
  type performance/write-behind
  option flush-behind on
  subvolumes readahead-gluster3
end-volume

volume writebehind-gluster4
  type performance/write-behind
  option flush-behind on
  subvolumes readahead-gluster4
end-volume

volume writebehind-gluster5
  type performance/write-behind
  option flush-behind on
  subvolumes readahead-gluster5
end-volume

volume quick-read-gluster1
	type performance/quick-read
	subvolumes writebehind-gluster1
end-volume

volume quick-read-gluster2
	type performance/quick-read
	subvolumes writebehind-gluster2
end-volume

volume quick-read-gluster3
	type performance/quick-read
	subvolumes writebehind-gluster3
end-volume

volume quick-read-gluster4
	type performance/quick-read
	subvolumes writebehind-gluster4
end-volume

volume quick-read-gluster5
	type performance/quick-read
	subvolumes writebehind-gluster5
end-volume

volume stat-prefetch-gluster1
	type performance/stat-prefetch
    #subvolumes quick-read-gluster1
    subvolumes writebehind-gluster1
end-volume

volume stat-prefetch-gluster2
	type performance/stat-prefetch
    #subvolumes quick-read-gluster2
    subvolumes writebehind-gluster2
end-volume

volume stat-prefetch-gluster3
	type performance/stat-prefetch
    #subvolumes quick-read-gluster3
    subvolumes writebehind-gluster3
end-volume

volume stat-prefetch-gluster4
	type performance/stat-prefetch
    #subvolumes quick-read-gluster4
    subvolumes writebehind-gluster4
end-volume

volume stat-prefetch-gluster5
	type performance/stat-prefetch
    #subvolumes quick-read-gluster5
    subvolumes writebehind-gluster5
end-volume

volume stripe
  type cluster/stripe
  option block-size 2MB
  #subvolumes client-stripe-1 client-stripe-2 client-stripe-3
client-stripe-4 client-stripe-5
  #subvolumes readahead-gluster1 readahead-gluster2 readahead-gluster3
readahead-gluster4 readahead-gluster5
  #subvolumes writebehind-gluster1 writebehind-gluster2
writebehind-gluster3 writebehind-gluster4 writebehind-gluster5
  #subvolumes quick-read-gluster1 quick-read-gluster2
quick-read-gluster3 quick-read-gluster4 quick-read-gluster5
  subvolumes stat-prefetch-gluster1 stat-prefetch-gluster2
stat-prefetch-gluster3 stat-prefetch-gluster4 stat-prefetch-gluster5
end-volume

####

Quick read was disabled because there was a bug that causes a crash
when that's enabled. This has been fixed in more recent versions but I
haven't upgraded. Here's our client distribute setup :

####

volume client-distribute-1
  type protocol/client
  option transport-type ib-verbs
  option remote-host gluster1
  option remote-subvolume iothreads-dist
end-volume

volume client-distribute-2
  type protocol/client
  option transport-type ib-verbs
  option remote-host gluster2
  option remote-subvolume iothreads-dist
end-volume

volume client-distribute-3
  type protocol/client
  option transport-type ib-verbs
  option remote-host gluster3
  option remote-subvolume iothreads-dist
end-volume

volume client-distribute-4
  type protocol/client
  option transport-type ib-verbs
  option remote-host gluster4
  option remote-subvolume iothreads-dist
end-volume

volume client-distribute-5
  type protocol/client
  option transport-type ib-verbs
  option remote-host gluster5
  option remote-subvolume iothreads-dist
end-volume

volume readahead-gluster1
  type performance/read-ahead
  option page-count 4           # 2 is default
  option force-atime-update off # default is off
  subvolumes client-distribute-1
end-volume

volume readahead-gluster2
  type performance/read-ahead
  option page-count 4           # 2 is default
  option force-atime-update off # default is off
  subvolumes client-distribute-2
end-volume

volume readahead-gluster3
  type performance/read-ahead
  option page-count 4           # 2 is default
  option force-atime-update off # default is off
  subvolumes client-distribute-3
end-volume

volume readahead-gluster4
  type performance/read-ahead
  option page-count 4           # 2 is default
  option force-atime-update off # default is off
  subvolumes client-distribute-4
end-volume
	
volume readahead-gluster5
  type performance/read-ahead
  option page-count 4           # 2 is default
  option force-atime-update off # default is off
  subvolumes client-distribute-5
end-volume

volume writebehind-gluster1
  type performance/write-behind
  option flush-behind on
  subvolumes readahead-gluster1
end-volume

volume writebehind-gluster2
  type performance/write-behind
  option flush-behind on
  subvolumes readahead-gluster2
end-volume

volume writebehind-gluster3
  type performance/write-behind
  option flush-behind on
  subvolumes readahead-gluster3
end-volume

volume writebehind-gluster4
  type performance/write-behind
  option flush-behind on
  subvolumes readahead-gluster4
end-volume

volume writebehind-gluster5
  type performance/write-behind
  option flush-behind on
  subvolumes readahead-gluster5
end-volume

volume quick-read-gluster1
	type performance/quick-read
	subvolumes writebehind-gluster1
end-volume

volume quick-read-gluster2
	type performance/quick-read
	subvolumes writebehind-gluster2
end-volume

volume quick-read-gluster3
	type performance/quick-read
	subvolumes writebehind-gluster3
end-volume

volume quick-read-gluster4
	type performance/quick-read
	subvolumes writebehind-gluster4
end-volume

volume quick-read-gluster5
	type performance/quick-read
	subvolumes writebehind-gluster5
end-volume

volume stat-prefetch-gluster1
	type performance/stat-prefetch
    subvolumes quick-read-gluster1
end-volume

volume stat-prefetch-gluster2
	type performance/stat-prefetch
    subvolumes quick-read-gluster2
end-volume

volume stat-prefetch-gluster3
	type performance/stat-prefetch
    subvolumes quick-read-gluster3
end-volume

volume stat-prefetch-gluster4
	type performance/stat-prefetch
    subvolumes quick-read-gluster4
end-volume

volume stat-prefetch-gluster5
	type performance/stat-prefetch
    subvolumes quick-read-gluster5
end-volume

volume distribute
  type cluster/distribute
  #option block-size 2MB
  #subvolumes client-distribute-1 client-distribute-2
client-distribute-3 client-distribute-4 client-distribute-5
  option min-free-disk 1%
  #subvolumes writebehind-gluster1 writebehind-gluster2
writebehind-gluster3 writebehind-gluster4 writebehind-gluster5
  subvolumes stat-prefetch-gluster1 stat-prefetch-gluster2
stat-prefetch-gluster3 stat-prefetch-gluster4 stat-prefetch-gluster5
end-volume

####

I don't know why my writes are so slow compared to reads. Let me know
if you're able to get better write speeds with the newer version of
gluster and any of the configurations (if they apply) that I've
posted. It might compel me to upgrade.

HTH,
Sabuj Pattanayek
> For some background, our compute cluster has 64 compute nodes. The gluster
> storage pool has 10 Dell PowerEdge R515 servers, each with 12 x 2 TB disks.
> We have another 16 Dell PowerEdge R515s used as Lustre storage servers. The
> compute and storage nodes are all connected via QDR Infiniband. Both
Gluster
> and Lustre are set to use RDMA over Infiniband. We are using OFED version
> 1.5.2-20101219, Gluster 3.2.2 and CentOS 5.5 on both the compute and
storage
> nodes.
>
> Oddly, it seems like there's some sort of bottleneck on the client side
--
> for example, we're only seeing about 50 MB/s write throughput from a
single
> compute node when writing a 10GB file. But, if we run multiple simultaneous
> writes from multiple compute nodes to the same Gluster volume, we get 50
> MB/s from each compute node. However, running multiple writes from the same
> compute node does not increase throughput. The compute nodes have 48 cores
> and 128 GB RAM, so I don't think the issue is with the compute node
> hardware.
>
> With Lustre, on the same hardware, with the same version of OFED, we're
> seeing write throughput on that same 10 GB file as follows: 476 MB/s single
> stream write from a single compute node and aggregate performance of more
> like 2.4 GB/s if we run simultaneous writes. That leads me to believe that
> we don't have a problem with RDMA, otherwise Lustre, which is also
using
> RDMA, should be similarly affected.
>
> We have tried both xfs and ext4 for the backend file system on the Gluster
> storage nodes (we're currently using ext4). We went with distributed
(not
> distributed striped) for the Gluster volume -- the thought was that if
there
> was a catastrophic failure of one of the storage nodes, we'd only lose
the
> data on that node; presumably with distributed striped you'd lose any
data
> striped across that volume, unless I have misinterpreted the documentation.
>
> So ... what's expected/normal throughput for Gluster over QDR IB to a
> relatively large storage pool (10 servers / 120 disks)? Does anyone have
> suggested tuning tips for improving performance?
>
> Thanks!
>
> John
>
> --
>
> ________________________________________________________
>
> John Lalande
> University of Wisconsin-Madison
> Space Science & Engineering Center
> 1225 W. Dayton Street, Room 439, Madison, WI 53706
> 608-263-2268?/ john.lalande at ssec.wisc.edu
>
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>
>

Pavan T C

2011-Jul-26 09:03 UTC

head link

[Gluster-users] gluster client performance

On Tuesday 26 July 2011 03:42 AM, John Lalande wrote:> Hi-
>
> I'm new to Gluster, but am trying to get it set up on a new compute
> cluster we're building. We picked Gluster for one of our cluster file
> systems (we're also using Lustre for fast scratch space), but the
> Gluster performance has been so bad that I think maybe we have a
> configuration problem -- perhaps we're missing a tuning parameter that
> would help, but I can't find anything in the Gluster documentation --
> all the tuning info I've found seems geared toward Gluster 2.x.
>
> For some background, our compute cluster has 64 compute nodes. The
> gluster storage pool has 10 Dell PowerEdge R515 servers, each with 12 x
> 2 TB disks. We have another 16 Dell PowerEdge R515s used as Lustre
> storage servers. The compute and storage nodes are all connected via QDR
> Infiniband. Both Gluster and Lustre are set to use RDMA over Infiniband.
> We are using OFED version 1.5.2-20101219, Gluster 3.2.2 and CentOS 5.5
> on both the compute and storage nodes.
Hi John,

I would need some more information about your setup to estimate the 
performance you should get with your gluster setup.

1. Can you provide the details of how disks are connected to the storage 
boxes ? Is it via FC ? What raid configuration is it using (if at all any) ?

2. What is the disk bandwidth you are getting on the local filesystem on 
a given storage node ? I mean, pick any of the 10 storage servers 
dedicated for Gluster Storage and perform a dd as below:

Write bandwidth measurement:
dd if=/dev/zero of=/export_directory/10g_file bs=128K count=80000 
oflag=direct

Read bandwidth measurement:
dd if=/export_directory/10g_file of=/dev/null bs=128K count=80000 
iflag=direct

[The above command is doing a direct IO of 10GB via your backend FS - 
ext4/xfs.]

3. What is the IB bandwidth that you are getting between the compute 
node and the glusterfs storage node? You can run the tool "rdma_bw" to
get the details:

On the server, run:
# rdma_bw -b
[ -b measures bi-directional bandwidth]

On the compute node, run,
# rdma_bw -b <server>

[If you have not already installed it, rdma_bw is available via -
http://mirror.centos.org/centos/5/os/x86_64/CentOS/perftest-1.2.3-1.el5.x86_64.rpm]

Lets start with this, and I will ask for more if necessary.

Pavan
>
> Oddly, it seems like there's some sort of bottleneck on the client side
> -- for example, we're only seeing about 50 MB/s write throughput from a
> single compute node when writing a 10GB file. But, if we run multiple
> simultaneous writes from multiple compute nodes to the same Gluster
> volume, we get 50 MB/s from each compute node. However, running multiple
> writes from the same compute node does not increase throughput. The
> compute nodes have 48 cores and 128 GB RAM, so I don't think the issue
> is with the compute node hardware.
>
> With Lustre, on the same hardware, with the same version of OFED, we're
> seeing write throughput on that same 10 GB file as follows: 476 MB/s
> single stream write from a single compute node and aggregate performance
> of more like 2.4 GB/s if we run simultaneous writes. That leads me to
> believe that we don't have a problem with RDMA, otherwise Lustre, which
> is also using RDMA, should be similarly affected.
>
> We have tried both xfs and ext4 for the backend file system on the
> Gluster storage nodes (we're currently using ext4). We went with
> distributed (not distributed striped) for the Gluster volume -- the
> thought was that if there was a catastrophic failure of one of the
> storage nodes, we'd only lose the data on that node; presumably with
> distributed striped you'd lose any data striped across that volume,
> unless I have misinterpreted the documentation.
>
> So ... what's expected/normal throughput for Gluster over QDR IB to a
> relatively large storage pool (10 servers / 120 disks)? Does anyone have
> suggested tuning tips for improving performance?
>
> Thanks!
>
> John
>
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Jesse Stroik

2011-Aug-09 18:41 UTC

head link

[Gluster-users] gluster client performance

Pavan,

Thank you for your help.  We wanted to get back to you with our results 
and observations.  I'm cc'ing gluster-users for posterity.

We did experiment with enable-trickling-writes.  That was one of the 
translator tunables we wanted to know the precise syntax for so that we 
could be certain we were disabling it.  As hoped, disabling trickling 
writes improved performance somewhat.

We are definitely interested in any other undocumented write-buffer 
related tunables.  We've tested the documented tuning parameters.

Performance improved significantly when we switched clients to mainline 
kernel (2.5.35-13).  We also updated to OFED 1.5.3 but it wasn't 
responsible for the performance improvement.

Our findings with 32KB block size (cp) write performance:

250-300MB/sec single stream performance
400MB/sec multiple-stream per client performance

This is much higher than we observed with kernel 2.6.18 series.  Using 
the 2.6.18 line, we also observed virtually no difference between 
running single stream tests and multi stream tests suggesting a 
bottleneck with the fabric.

Both 2.6.18 and 2.6.35-13 performed very well (about 600MB/sec) when 
writing 128KB blocks.

When I disabled write-behind on the 2.6.18 series of kernels as a test, 
performance plummeted to a few MB/sec when writing blocks sizes smaller 
than 128KB.  We did not test this extensively.

Disabling enable-trickling-writes gave us approximately a 20% boost, 
reflected in the numbers above, for single-stream writes.  We observed 
no significant difference with several streams per client due to 
disabling that tunable.

For reference, we are running another cluster file system on the same 
underlying hardware/software.  With both the old kernel (2.6.18.x) and 
the new kernel (2.6.35-13) we get approximately:

450-550MB/sec single stream performance
1200MB+/sec multiple stream per client performance

We set the test directory to write entire files to a single LUN which is 
how we configured gluster in an effort to mitigate differences.

It is treacherous to speculate why we might be more limited with gluster 
over RDMA than the other cluster file system without spending a 
significant amount of analysis.  That said, I wonder if there may be an 
issue with the way in which fuse handles write buffers causing a 
bottleneck for RMDA.

The bottom line is that our observed performance was poor using the 
2.6.18 RHEL 5 kernel line relative to the mainline (2.6.35) kernels. 
Updating to the newer kernels was well worth the testing and downtime. 
Hopefully this information can help others.

Best,
Jesse Stroik

Possibly Parallel Threads

Search for more apparently analagous threads

Gluster users - Jul 2011 - gluster client performance

[Gluster-users] gluster client performance

[Gluster-users] gluster client performance

[Gluster-users] gluster client performance

[Gluster-users] gluster client performance

Possibly Parallel Threads