In some basic testing using 2 RDMA gluster servers configured as a replicated pair, I'm seeing better significantly better performance with IPoIB versus RDMA only volumes (almost 2x in write performance and ~10-20% in read). Has anyone had a similar experience? glusterfs 3.4.0 built on Jul 12 2013 20:36:46 Repository revision: git://git.gluster.com/glusterfs.git Copyright (c) 2006-2011 Gluster Inc. <http://www.gluster.com> GlusterFS comes with ABSOLUTELY NO WARRANTY. You may redistribute copies of GlusterFS under the terms of the GNU General Public License. Volume Name: perftest Type: Replicate Volume ID: 10578349-a466-497a-b7de-1fcd34f967da Status: Started Number of Bricks: 1 x 2 = 2 Transport-type: rdma Bricks: Brick1: cs1-i:/gluster/perftest Brick2: cs2-i:/gluster/perftest cs1-i:/perftest.rdma 3.2T 13G 3.2T 1% /tmp/perftest Run began: Wed Jul 17 08:31:39 2013 Record Size 1024 KB File size set to 5242880 KB Command line used: iozone -i0 -i1 -r 1024#k -s 5#g Output is in Kbytes/sec Time Resolution = 0.000001 seconds. Processor cache size set to 1024 Kbytes. Processor cache line size set to 32 bytes. File stride size set to 17 * record size. random random bkwd record stride KB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread 5242880 1024 422514 496985 364946 324213 Volume Name: perftest Type: Replicate Volume ID: 10578349-a466-497a-b7de-1fcd34f967da Status: Started Number of Bricks: 1 x 2 = 2 Transport-type: tcp Bricks: Brick1: cs1-i:/gluster/perftest Brick2: cs2-i:/gluster/perftest cs1-i:/perftest.tcp 3.2T 13G 3.2T 1% /tmp/perftest Run began: Wed Jul 17 08:38:23 2013 Record Size 1024 KB File size set to 5242880 KB Command line used: iozone -i0 -i1 -r 1024#k -s 5#g Output is in Kbytes/sec Time Resolution = 0.000001 seconds. Processor cache size set to 1024 Kbytes. Processor cache line size set to 32 bytes. File stride size set to 17 * record size. random random bkwd record stride KB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread 5242880 1024 557479 549629 667576 666251 -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20130717/8e0d2b35/attachment.html>
On 17/07/2013, at 2:46 PM, Ryan Aydelott wrote:> In some basic testing using 2 RDMA gluster servers configured as a replicated pair, I'm seeing better significantly better performance with IPoIB versus RDMA only volumes (almost 2x in write performance and ~10-20% in read). > > Has anyone had a similar experience?For me, using stripe 2 and not in depth testing at the time (a few weeks ago with the new RDMA code merged), I was getting good write performance (500MB/s) but very bad read performance (80-90MB/s). That was just on two initial gluster boxes, with the volume mounted to a third one. The volumes were RDMA only. The 80-90MS/s figure seems awfully similar to GigE kind of line speed though, so I wonder if something was somehow sending the data down the wrong link. (no idea, might not be) I plan on looking at this stuff properly again in a while, but want to get a proper multi-node regression testing suite set up and working first (so we can all use it). Maybe based on Autotest or similar (not sure yet). Not sure if this info all helps... ;) Regards and best wishes, Justin Clift -- Open Source and Standards @ Red Hat twitter.com/realjustinclift