Hi all, I'm using 2.0.0rc4 on a pair of infiniband connected servers. Both servers are running 2.6.28.7, and are using ext4 on a local raid set as the storage for Gluster. The speeds that I get writing to the aforementioned ext4 mount, depending on benchmark and method, vary around 400MB/s (with dd) to 550MB/s+ (with iozone, bonnie, and dd directio). Next, I used the Gluster server and client on both servers, with ib-verbs and replicate (see attached configuration files). The best speeds that I get when writing to the Gluster mount with this setup is about 150MB/s, going up to about 200MB/s when doing simultaneous writes from both nodes. I tried a couple of configuration tweaks such as writeback, but they don't seem to be having much effect. Is there something I can do to bring the performance closer to the native disk speed, or are there some inherent limitations somewhere that I should be aware of? Thanks! -Gennadiy -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20090323/1bef674f/attachment.html> -------------- next part -------------- A non-text attachment was scrubbed... Name: glusterfs.vol Type: application/octet-stream Size: 1138 bytes Desc: not available URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20090323/1bef674f/attachment.obj> -------------- next part -------------- A non-text attachment was scrubbed... Name: glusterfsd.vol Type: application/octet-stream Size: 622 bytes Desc: not available URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20090323/1bef674f/attachment-0001.obj>
On Mon, Mar 23, 2009 at 10:55 AM, Gennadiy Nerubayev <parakie at gmail.com>wrote:> Hi all, > > I'm using 2.0.0rc4 on a pair of infiniband connected servers. Both servers > are running 2.6.28.7, and are using ext4 on a local raid set as the storage > for Gluster. The speeds that I get writing to the aforementioned ext4 mount, > depending on benchmark and method, vary around 400MB/s (with dd) to 550MB/s+ > (with iozone, bonnie, and dd directio). Next, I used the Gluster server and > client on both servers, with ib-verbs and replicate (see attached > configuration files). The best speeds that I get when writing to the Gluster > mount with this setup is about 150MB/s, going up to about 200MB/s when doing > simultaneous writes from both nodes. I tried a couple of configuration > tweaks such as writeback, but they don't seem to be having much effect. > > Is there something I can do to bring the performance closer to the native > disk speed, or are there some inherent limitations somewhere that I should > be aware of?Woops, was making some interim changes to the server names and forgot to change that in the saved config file for the client. Reattached, sorry for the self reply. -Gennadiy -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20090323/7c39bd94/attachment.html> -------------- next part -------------- A non-text attachment was scrubbed... Name: glusterfs.vol Type: application/octet-stream Size: 1138 bytes Desc: not available URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20090323/7c39bd94/attachment.obj>
On Mon, 23 Mar 2009, Gennadiy Nerubayev wrote:> Hi all, > > I'm using 2.0.0rc4 on a pair of infiniband connected servers. Both servers > are running 2.6.28.7, and are using ext4 on a local raid set as the storage > for Gluster. The speeds that I get writing to the aforementioned ext4 mount, > depending on benchmark and method, vary around 400MB/s (with dd) to 550MB/s+ > (with iozone, bonnie, and dd directio). Next, I used the Gluster server and > client on both servers, with ib-verbs and replicate (see attached > configuration files). The best speeds that I get when writing to the Gluster > mount with this setup is about 150MB/s, going up to about 200MB/s when doing > simultaneous writes from both nodes. I tried a couple of configuration > tweaks such as writeback, but they don't seem to be having much effect.That is about what I am getting: Raw disk: 8589934592 bytes (8.6 GB) copied, 21.0523 seconds, 408 MB/s Gluster: 8589934592 bytes (8.6 GB) copied, 47.4356 seconds, 181 MB/s Gluster --disable-direct-io-mode 8589934592 bytes (8.6 GB) copied, 336.514 seconds, 25.5 MB/ The part that is most concerning to me is the --disable-direct-io-mode stuff since I am trying to use this with xen tap:aio.> Is there something I can do to bring the performance closer to the native > disk speed, or are there some inherent limitations somewhere that I should > be aware of?You can try some caching, but overall it did not help me out a lot.><>Nathan Stratton CTO, BlinkMind, Inc. nathan at robotics.net nathan at blinkmind.com http://www.robotics.net http://www.blinkmind.com