moka111 at 21p.de
2016-Jun-02 11:22 UTC
[Gluster-users] moka111 - memory leak problem as written in IRC
Hi, here is the collected data requested by atinm in irc @ #gluster-dev. Additionally I can say, that all 5 nodes have 32GB of RAM installed. Regards moka111 -------------- next part -------------- A non-text attachment was scrubbed... Name: collected.tgz Type: application/x-compressed Size: 555132 bytes Desc: not available URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160602/e6cf1709/attachment-0001.bin> -------------- next part -------------- A non-text attachment was scrubbed... Name: glusterdump.1359.dump.1464859580 Type: application/octet-stream Size: 20600 bytes Desc: not available URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160602/e6cf1709/attachment-0001.obj>
Atin Mukherjee
2016-Jun-15 05:45 UTC
[Gluster-users] moka111 - memory leak problem as written in IRC
A little late as it went out of my radar, apologies.>From st02's glusterd log, it looks like profile commands were executedvery frequently. [2016-06-02 10:35:02.232596] I [MSGID: 106494] [glusterd-handler.c:3070:__glusterd_handle_cli_profile_volume] 0-management: Received volume profile req for volume webroot [2016-06-02 10:35:10.315758] I [MSGID: 106494] [glusterd-handler.c:3070:__glusterd_handle_cli_profile_volume] 0-management: Received volume profile req for volume webroot [2016-06-02 10:35:10.487004] I [MSGID: 106494] [glusterd-handler.c:3070:__glusterd_handle_cli_profile_volume] 0-management: Received volume profile req for volume webroot The thing which worries me here is the following log: [2016-06-02 10:40:08.113106] W [glusterd-locks.c:577:glusterd_mgmt_v3_lock] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.7.11/xlator/mgmt/glusterd.so(glusterd_op_sm+0x266) [0x7fa5ad038106] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.7.11/xlator/mgmt/glusterd.so(+0x5aab5) [0x7fa5ad02aab5] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.7.11/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_lock+0x54d) [0x7fa5ad0c097d] ) 0-management: Lock for webroot held by 86cc4dab-c13a-40b1-ac40-0f6a98c1b46c [2016-06-02 10:40:08.113123] E [MSGID: 106119] [glusterd-op-sm.c:3711:glusterd_op_ac_lock] 0-management: Unable to acquire lock for webroot 1758 [2016-06-02 10:40:08.113151] E [MSGID: 106376] [glusterd-op-sm.c:7591:glusterd_op_sm] 0-management: handler returned: -1 This indicates that op-sm framework couldn't process the event and I guess didn't free up the transaction object which got malloced. From glusterd statedump it does look like there are so many txn objects which are not freed up. [mgmt/glusterd.management - usage-type gf_common_mt_txn_opinfo_obj_t memusage] size=3759056 *num_allocs=33563* max_size=3759168 max_num_allocs=33564 total_allocs=167742 We are aware of some memory leak for gluster profile command but not at this level and this needs a thorough investigation. Do you mind filing up a bug for this and we pick that up for further analysis. Thanks! ~Atin On 06/02/2016 04:52 PM, moka111 at 21p.de wrote:> Hi, > > here is the collected data requested by atinm in irc @ #gluster-dev. > Additionally I can say, that all 5 nodes have 32GB of RAM installed. > > Regards > moka111 > > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users >