Gluster users - Jan 2013 - Gluster-users Digest, Vol 57, Issue 32

About the following issue, anyone could help us? Why using VM mounted by NFS
V3(which is also provided by gluster) could approach full
bandwidth(100MB/s),while VM mounted by gluster only reach half of bandwidth
(50MB/s).

The following is the mount command:
1)gluster native client: 
mount -t glusterfs  server1:/test-volume  /var/lib/nova/instance/
2) NFS V3:
mount -t nfs -o vers=3 server1:/test-volume /var/lib/nova/instance/

/var/lib/nova/instance/ is the file directory storing VMs

Thanks. 



zhxue

------------------------------

Message: 6
Date: Mon, 14 Jan 2013 23:13:30 +0800
From: glusterzhxue <glusterzhxue at 163.com>
To: gluster-users <gluster-users at gluster.org>
Subject: Re: [Gluster-users] Gluster-users Digest, Vol 57, Issue 31
Message-ID: <201301142313289494315 at 163.com>
Content-Type: text/plain; charset="gb2312"

Thanks all the responses.
It seems that I need to furtherly describe our question. If our VM is mounted on
gluster via NFS(V3), writing big data on VM reach the full bandwidth. However,
if VM is mounted by gluster client, it has half bandwidth when writing data on
VM (mounted on gluster).
Considering we don't want VM users see our gluster file system, mount
gluster file on VM is not allowed.
By the way, Where can I get the 3.4a version of Gluster? 

  
Regards,



zhxue
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20130115/2504b695/attachment.html>

On 01/15/13 07:06, glusterzhxue wrote:
> About the following issue, anyone could help us? Why using VM mounted by
> NFS V3(which is also provided by gluster) could approach full
> bandwidth(100MB/s),while VM mounted by gluster only reach half of
> bandwidth (50MB/s).
> The following is the mount command:
> 1)gluster native client:
> mount -t glusterfs server1:/test-volume /var/lib/nova/instance/
> 2) NFS V3:
> mount -t nfs -o vers=3 server1:/test-volume /var/lib/nova/instance/
> /var/lib/nova/instance/ is the file directory storing VMs
> Thanks.
> ------------------------------------------------------------------------
> zhxue
What is your volume layout?
The output of "gluster volume info test-volume"

What type of network hardware do you have?
What type of ethernet ports on the host running the VMs?
What type of ethernet ports on the hosts containing the storage (the 
gluster servers)?
How are they configured?

My configuration has 4 1Gb ports LACP bonded on the VM host and single 
10Gb ports on the the storage nodes (gluster servers.)

Our volume is: (4 bricks per gluster server)
Volume Name: vol01
Type: Distributed-Replicate
Volume ID: 5ebcaed8-dd9b-4c48-be75-46457fe2e978
Status: Started
Number of Bricks: 24 x 2 = 48
Transport-type: tcp

With the iozone testing on a FUSE mount, I am currently doing, I get:

~100MB/s writes if the replicated bricks chosen hash out in the LACP 
bonding to the different 1Gb ethernet ports.

~50MB/s writes if the replicated bricks chosen hash out in the LACP 
bonding to the same 1Gb ethernet ports.

~20MB/s writes for the test using the O_SYNC option for iozone, no 
matter what 1Gb ethernet ports end up being used.

Reads are always ~100MB/s as it comes from one brick.

If you have a replica volume, then mount using FUSE, the writes are 
duplicated on the client.  If you mount via NFS, the writes are 
duplicated on the server.

Also of note, with the iozone going full speed, the FUSE client takes 
50% of a CPU core.

-- 
Mr. Flibble
King of the Potato People