Gluster users - Nov 2017 - glusterfs brick server use too high memory

Hi all,
I'm glad to add glusterfs community.


I have a glusterfs cluster:
Nodes: 4

System: Centos7.1
Glusterfs: 3.8.9
Each Node:
CPU: 48 core
Mem: 128GB
Disk: 1*4T


There is one Distributed Replicated volume. There are ~160 k8s pods as clients
connecting to glusterfs. But, the memory of  glusterfsd process is too high,
gradually increase to 100G every node.
Then, I reboot the glusterfsd process. But the memory increase during
approximate a week.
How can I debug the problem?


Thanks.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20171108/95f39341/attachment.html>

On 8 November 2017 at 17:16, Yao Guotao <yaoguo_tao at 163.com> wrote:
> Hi all,
> I'm glad to add glusterfs community.
>
> I have a glusterfs cluster:
> Nodes: 4
> System: Centos7.1
> Glusterfs: 3.8.9
> Each Node:
> CPU: 48 core
> Mem: 128GB
> Disk: 1*4T
>
> There is one Distributed Replicated volume. There are ~160 k8s pods as
> clients connecting to glusterfs. But, the memory of  glusterfsd process is
> too high, gradually increase to 100G every node.
> Then, I reboot the glusterfsd process. But the memory increase during
> approximate a week.
> How can I debug the problem?
>
> Hi,
Please take statedumps at intervals (a minimum of 2 at intervals of an
hour) of a brick process for which you see the memory increasing and send
them to us.  [1] describes how to take statedumps.

Regards,
Nithya

[1] http://docs.gluster.org/en/latest/Troubleshooting/statedump/
<http://docs.gluster.org/en/latest/Troubleshooting/statedump/>

<http://docs.gluster.org/en/latest/Troubleshooting/statedump/>
> Thanks.
>
>
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20171109/5ec1273c/attachment.html>

Thank you very much for your reply.


I think I found the key to the problem. But, I'd like you to confirm whether
this is the issue.
My gluster version is 3.8.9, And I found there is a bug in it, and fixed in
3.8.11:
https://bugzilla.redhat.com/show_bug.cgi?id=1431592



A large amount of memory allocated for agf_common_mt_strdup. Everythig else seem
to be all right.



My cluster:


Yesterday Afternoon??I'm sorry I forgot the specific time.?

[features/locks.www-volume-locks - usage-type gf_common_mt_strdup memusage]
size=1941483443
num_allocs=617382139
max_size=1941483443
max_num_allocs=617382139
total_allocs=661873332



TIme: 2017.11.9 17:15 (Today)
[features/locks.www-volume-locks - usage-type gf_common_mt_strdup memusage]
size=792538295
num_allocs=752904534
max_size=792538295
max_num_allocs=752904534
total_allocs=800889589



The statedump files are in the attachment.


Thanks again.





? 2017-11-09 17:22:03?"Nithya Balachandran" <nbalacha at
redhat.com> ???





On 8 November 2017 at 17:16, Yao Guotao <yaoguo_tao at 163.com> wrote:

Hi all,
I'm glad to add glusterfs community.


I have a glusterfs cluster:
Nodes: 4

System: Centos7.1
Glusterfs: 3.8.9
Each Node:
CPU: 48 core
Mem: 128GB
Disk: 1*4T


There is one Distributed Replicated volume. There are ~160 k8s pods as clients
connecting to glusterfs. But, the memory of  glusterfsd process is too high,
gradually increase to 100G every node.
Then, I reboot the glusterfsd process. But the memory increase during
approximate a week.
How can I debug the problem?


Hi,


Please take statedumps at intervals (a minimum of 2 at intervals of an hour) of
a brick process for which you see the memory increasing and send them to us. 
[1] describes how to take statedumps.


Regards,
Nithya


[1] http://docs.gluster.org/en/latest/Troubleshooting/statedump/ 


 
Thanks.





 


_______________________________________________
Gluster-users mailing list
Gluster-users at gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20171109/a394c932/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: home-www-log.152061.dump.20171108
Type: application/octet-stream
Size: 806070 bytes
Desc: not available
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20171109/a394c932/attachment-0002.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: home-www-log.152061.dump.20171109
Type: application/octet-stream
Size: 739337 bytes
Desc: not available
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20171109/a394c932/attachment-0003.obj>