Gurdeep Singh (Guru)
2014-Jul-06 13:28 UTC
[Gluster-users] Gluster high RPC calls and reply
Hello, I have setup gluster in replicate type and its working fine. I am seeing a constant chatting between the hosts for lookup call and lookup reply. I am trying to understand as to why this traffic is being initiated constantly. Please look at the attached image. This traffic is using around 200KB/s of constant bandwidth and is exhausting our allocated monthly bandwidth on our 2 VPS. The configuration I have for Gluster is: [guru at srv1 ~]$ sudo gluster volume info [sudo] password for guru: Volume Name: gv0 Type: Replicate Volume ID: dc8dc3f2-f5bd-4047-9101-acad04695442 Status: Started Number of Bricks: 1 x 2 = 2 Transport-type: tcp Bricks: Brick1: srv1:/root/gluster-vol0 Brick2: srv2:/root/gluster-vol0 Options Reconfigured: cluster.lookup-unhashed: on performance.cache-refresh-timeout: 60 performance.cache-size: 1GB storage.health-check-interval: 30 Please suggest how to fine tune the RPC calls/reply. Thanks, Gurdeep. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140706/7b597415/attachment.html> -------------- next part -------------- A non-text attachment was scrubbed... Name: Screen Shot 2014-07-06 at 11.21.55 pm.png Type: image/png Size: 35435 bytes Desc: not available URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140706/7b597415/attachment.png>
Pranith Kumar Karampuri
2014-Jul-07 09:19 UTC
[Gluster-users] Gluster high RPC calls and reply
On 07/06/2014 06:58 PM, Gurdeep Singh (Guru) wrote:> Hello, > > I have setup gluster in replicate type and its working fine. > > I am seeing a constant chatting between the hosts for lookup call and > lookup reply. I am trying to understand as to why this traffic is > being initiated constantly. Please look at the attached image. This > traffic is using around 200KB/s of constant bandwidth and is > exhausting our allocated monthly bandwidth on our 2 VPS.Self-heal-daemon tries to check if there is anything that needs healing every 10 minutes. Those RPCs are lookups. Could you check from the tcpdump if it is generating from self-heal-daemon? Which version of gluster are you using? I think there is a way to increase this 'time' in latest versions. Pranith> > The configuration I have for Gluster is: > > [guru at srv1 ~]$ sudo gluster volume info > [sudo] password for guru: > > > Volume Name: gv0 > Type: Replicate > Volume ID: dc8dc3f2-f5bd-4047-9101-acad04695442 > Status: Started > Number of Bricks: 1 x 2 = 2 > Transport-type: tcp > Bricks: > Brick1: srv1:/root/gluster-vol0 > Brick2: srv2:/root/gluster-vol0 > Options Reconfigured: > cluster.lookup-unhashed: on > performance.cache-refresh-timeout: 60 > performance.cache-size: 1GB > storage.health-check-interval: 30 > > > Please suggest how to fine tune the RPC calls/reply. > > Thanks, > Gurdeep. > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://supercolony.gluster.org/mailman/listinfo/gluster-users-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140707/1e8fd007/attachment.html> -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/png Size: 35435 bytes Desc: not available URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140707/1e8fd007/attachment.png>
On Sun, Jul 06, 2014 at 11:28:51PM +1000, Gurdeep Singh (Guru) wrote:> Hello, > > I have setup gluster in replicate type and its working fine. > > I am seeing a constant chatting between the hosts for lookup call and > lookup reply. I am trying to understand as to why this traffic is > being initiated constantly. Please look at the attached image. This > traffic is using around 200KB/s of constant bandwidth and is > exhausting our allocated monthly bandwidth on our 2 VPS.You can use Wireshark to identify which process does the LOOKUP calls. For this, do the following: 1. select a LOOKUP Call 2. enable the 'packet details' pane (found in the main menu, 'view') 3. expand the 'Transmission Control Protocol' tree 4. check the 'Source port' of the LOOKUP Call Together with the 'Source' and the 'Source port' you can go to the server that matches the 'Source' address. A command like this would give you the PID of the process in the right column: # netstat -tpn | grep $SOURCE_PORT And with 'ps -v $PID' you can check which process is responsible for the LOOKUP. This process can be a fuse-mount, self-heal-daemon or any other glusterfs-client. Depending on the type of client, you maybe can tune the workload or other options a little. In Wireshark you can also check what filename is LOOKUP'd, just expand the 'GlusterFS' part in the 'packet details' and check the 'Basename'. Maybe this filename (without directory structure) does give you any ideas of which activity is causing the LOOKUPs. HTH, Niels> > The configuration I have for Gluster is: > > [guru at srv1 ~]$ sudo gluster volume info > [sudo] password for guru: > > Volume Name: gv0 > Type: Replicate > Volume ID: dc8dc3f2-f5bd-4047-9101-acad04695442 > Status: Started > Number of Bricks: 1 x 2 = 2 > Transport-type: tcp > Bricks: > Brick1: srv1:/root/gluster-vol0 > Brick2: srv2:/root/gluster-vol0 > Options Reconfigured: > cluster.lookup-unhashed: on > performance.cache-refresh-timeout: 60 > performance.cache-size: 1GB > storage.health-check-interval: 30 > > > > Please suggest how to fine tune the RPC calls/reply.