On 01/27/2018 02:29 AM, Dan Ragle wrote:> > On 1/25/2018 8:21 PM, Ravishankar N wrote: >> >> >> On 01/25/2018 11:04 PM, Dan Ragle wrote: >>> *sigh* trying again to correct formatting ... apologize for the >>> earlier mess. >>> >>> Having a memory issue with Gluster 3.12.4 and not sure how to >>> troubleshoot. I don't *think* this is expected behavior. >>> >>> This is on an updated CentOS 7 box. The setup is a simple two node >>> replicated layout where the two nodes act as both server and >>> client. >>> >>> The volume in question: >>> >>> Volume Name: GlusterWWW >>> Type: Replicate >>> Volume ID: 8e9b0e79-f309-4d9b-a5bb-45d065faaaa3 >>> Status: Started >>> Snapshot Count: 0 >>> Number of Bricks: 1 x 2 = 2 >>> Transport-type: tcp >>> Bricks: >>> Brick1: vs1dlan.mydomain.com:/glusterfs_bricks/brick1/www >>> Brick2: vs2dlan.mydomain.com:/glusterfs_bricks/brick1/www >>> Options Reconfigured: >>> nfs.disable: on >>> cluster.favorite-child-policy: mtime >>> transport.address-family: inet >>> >>> I had some other performance options in there, (increased >>> cache-size, md invalidation, etc) but stripped them out in an >>> attempt to >>> isolate the issue. Still got the problem without them. >>> >>> The volume currently contains over 1M files. >>> >>> When mounting the volume, I get (among other things) a process as such: >>> >>> /usr/sbin/glusterfs --volfile-server=localhost >>> --volfile-id=/GlusterWWW /var/www >>> >>> This process begins with little memory, but then as files are >>> accessed in the volume the memory increases. I setup a script that >>> simply reads the files in the volume one at a time (no writes). It's >>> been running on and off about 12 hours now and the resident >>> memory of the above process is already at 7.5G and continues to grow >>> slowly. If I stop the test script the memory stops growing, >>> but does not reduce. Restart the test script and the memory begins >>> slowly growing again. >>> >>> This is obviously a contrived app environment. With my intended >>> application load it takes about a week or so for the memory to get >>> high enough to invoke the oom killer. >> >> Can you try debugging with the statedump >> (https://gluster.readthedocs.io/en/latest/Troubleshooting/statedump/#read-a-statedump) >> of >> the fuse mount process and see what member is leaking? Take the >> statedumps in succession, maybe once initially during the I/O and >> once the memory gets high enough to hit the OOM mark. >> Share the dumps here. >> >> Regards, >> Ravi > > Thanks for the reply. I noticed yesterday that an update (3.12.5) had > been posted so I went ahead and updated and repeated the test > overnight. The memory usage does not appear to be growing as quickly > as is was with 3.12.4, but does still appear to be growing. > > I should also mention that there is another process beyond my test app > that is reading the files from the volume. Specifically, there is an > rsync that runs from the second node 2-4 times an hour that reads from > the GlusterWWW volume mounted on node 1. Since none of the files in > that mount are changing it doesn't actually rsync anything, but > nonetheless it is running and reading the files in addition to my test > script. (It's a part of my intended production setup that I forgot was > still running.) > > The mount process appears to be gaining memory at a rate of about 1GB > every 4 hours or so. At that rate it'll take several days before it > runs the box out of memory. But I took your suggestion and made some > statedumps today anyway, about 2 hours apart, 4 total so far. It looks > like there may already be some actionable information. These are the > only registers where the num_allocs have grown with each of the four > samples: > > [mount/fuse.fuse - usage-type gf_fuse_mt_gids_t memusage] > ?---> num_allocs at Fri Jan 26 08:57:31 2018: 784 > ?---> num_allocs at Fri Jan 26 10:55:50 2018: 831 > ?---> num_allocs at Fri Jan 26 12:55:15 2018: 877 > ?---> num_allocs at Fri Jan 26 14:58:27 2018: 908 > > [mount/fuse.fuse - usage-type gf_common_mt_fd_lk_ctx_t memusage] > ?---> num_allocs at Fri Jan 26 08:57:31 2018: 5 > ?---> num_allocs at Fri Jan 26 10:55:50 2018: 10 > ?---> num_allocs at Fri Jan 26 12:55:15 2018: 15 > ?---> num_allocs at Fri Jan 26 14:58:27 2018: 17 > > [cluster/distribute.GlusterWWW-dht - usage-type gf_dht_mt_dht_layout_t > memusage] > ?---> num_allocs at Fri Jan 26 08:57:31 2018: 24243596 > ?---> num_allocs at Fri Jan 26 10:55:50 2018: 27902622 > ?---> num_allocs at Fri Jan 26 12:55:15 2018: 30678066 > ?---> num_allocs at Fri Jan 26 14:58:27 2018: 33801036 > > Not sure the best way to get you the full dumps. They're pretty big, > over 1G for all four. Also, I noticed some filepath information in > there that I'd rather not share. What's the recommended next step?I've CC'd the fuse/ dht devs to see if these data types have potential leaks. Could you raise a bug with the volume info and a (dropbox?) link from which we can download the dumps? You can remove/replace the filepaths from them. Regards. Ravi> > Cheers! > > Dan > >>> >>> Is there potentially something misconfigured here? >>> >>> I did see a reference to a memory leak in another thread in this >>> list, but that had to do with the setting of quotas, I don't have >>> any quotas set on my system. >>> >>> Thanks, >>> >>> Dan Ragle >>> daniel at Biblestuph.com >>> >>> On 1/25/2018 11:04 AM, Dan Ragle wrote: >>>> Having a memory issue with Gluster 3.12.4 and not sure how to >>>> troubleshoot. I don't *think* this is expected behavior. This is on an >>>> updated CentOS 7 box. The setup is a simple two node replicated layout >>>> where the two nodes act as both server and client. The volume in >>>> question: Volume Name: GlusterWWW Type: Replicate Volume ID: >>>> 8e9b0e79-f309-4d9b-a5bb-45d065faaaa3 Status: Started Snapshot Count: 0 >>>> Number of Bricks: 1 x 2 = 2 Transport-type: tcp Bricks: Brick1: >>>> vs1dlan.mydomain.com:/glusterfs_bricks/brick1/www Brick2: >>>> vs2dlan.mydomain.com:/glusterfs_bricks/brick1/www Options >>>> Reconfigured: >>>> nfs.disable: on cluster.favorite-child-policy: mtime >>>> transport.address-family: inet I had some other performance options in >>>> there, (increased cache-size, md invalidation, etc) but stripped them >>>> out in an attempt to isolate the issue. Still got the problem without >>>> them. The volume currently contains over 1M files. When mounting the >>>> volume, I get (among other things) a process as such: >>>> /usr/sbin/glusterfs --volfile-server=localhost >>>> --volfile-id=/GlusterWWW >>>> /var/www This process begins with little memory, but then as files are >>>> accessed in the volume the memory increases. I setup a script that >>>> simply reads the files in the volume one at a time (no writes). It's >>>> been running on and off about 12 hours now and the resident memory of >>>> the above process is already at 7.5G and continues to grow slowly. >>>> If I >>>> stop the test script the memory stops growing, but does not reduce. >>>> Restart the test script and the memory begins slowly growing again. >>>> This >>>> is obviously a contrived app environment. With my intended application >>>> load it takes about a week or so for the memory to get high enough to >>>> invoke the oom killer. Is there potentially something misconfigured >>>> here? Thanks, Dan Ragle daniel at Biblestuph.com >>>> >>>> >>>> >>>> >>>> _______________________________________________ >>>> Gluster-users mailing list >>>> Gluster-users at gluster.org >>>> http://lists.gluster.org/mailman/listinfo/gluster-users >>>> >>> _______________________________________________ >>> Gluster-users mailing list >>> Gluster-users at gluster.org >>> http://lists.gluster.org/mailman/listinfo/gluster-users >> > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://lists.gluster.org/mailman/listinfo/gluster-users
Nithya Balachandran
2018-Jan-29 05:19 UTC
[Gluster-users] Run away memory with gluster mount
Csaba, Could this be the problem of the inodes not getting freed in the fuse process? Daniel, as Ravi requested, please provide access to the statedumps. You can strip out the filepath information. Does your data set include a lot of directories? Thanks, Nithya On 27 January 2018 at 10:23, Ravishankar N <ravishankar at redhat.com> wrote:> > > On 01/27/2018 02:29 AM, Dan Ragle wrote: > >> >> On 1/25/2018 8:21 PM, Ravishankar N wrote: >> >>> >>> >>> On 01/25/2018 11:04 PM, Dan Ragle wrote: >>> >>>> *sigh* trying again to correct formatting ... apologize for the earlier >>>> mess. >>>> >>>> Having a memory issue with Gluster 3.12.4 and not sure how to >>>> troubleshoot. I don't *think* this is expected behavior. >>>> >>>> This is on an updated CentOS 7 box. The setup is a simple two node >>>> replicated layout where the two nodes act as both server and >>>> client. >>>> >>>> The volume in question: >>>> >>>> Volume Name: GlusterWWW >>>> Type: Replicate >>>> Volume ID: 8e9b0e79-f309-4d9b-a5bb-45d065faaaa3 >>>> Status: Started >>>> Snapshot Count: 0 >>>> Number of Bricks: 1 x 2 = 2 >>>> Transport-type: tcp >>>> Bricks: >>>> Brick1: vs1dlan.mydomain.com:/glusterfs_bricks/brick1/www >>>> Brick2: vs2dlan.mydomain.com:/glusterfs_bricks/brick1/www >>>> Options Reconfigured: >>>> nfs.disable: on >>>> cluster.favorite-child-policy: mtime >>>> transport.address-family: inet >>>> >>>> I had some other performance options in there, (increased cache-size, >>>> md invalidation, etc) but stripped them out in an attempt to >>>> isolate the issue. Still got the problem without them. >>>> >>>> The volume currently contains over 1M files. >>>> >>>> When mounting the volume, I get (among other things) a process as such: >>>> >>>> /usr/sbin/glusterfs --volfile-server=localhost --volfile-id=/GlusterWWW >>>> /var/www >>>> >>>> This process begins with little memory, but then as files are accessed >>>> in the volume the memory increases. I setup a script that >>>> simply reads the files in the volume one at a time (no writes). It's >>>> been running on and off about 12 hours now and the resident >>>> memory of the above process is already at 7.5G and continues to grow >>>> slowly. If I stop the test script the memory stops growing, >>>> but does not reduce. Restart the test script and the memory begins >>>> slowly growing again. >>>> >>>> This is obviously a contrived app environment. With my intended >>>> application load it takes about a week or so for the memory to get >>>> high enough to invoke the oom killer. >>>> >>> >>> Can you try debugging with the statedump (https://gluster.readthedocs.i >>> o/en/latest/Troubleshooting/statedump/#read-a-statedump) of >>> the fuse mount process and see what member is leaking? Take the >>> statedumps in succession, maybe once initially during the I/O and >>> once the memory gets high enough to hit the OOM mark. >>> Share the dumps here. >>> >>> Regards, >>> Ravi >>> >> >> Thanks for the reply. I noticed yesterday that an update (3.12.5) had >> been posted so I went ahead and updated and repeated the test overnight. >> The memory usage does not appear to be growing as quickly as is was with >> 3.12.4, but does still appear to be growing. >> >> I should also mention that there is another process beyond my test app >> that is reading the files from the volume. Specifically, there is an rsync >> that runs from the second node 2-4 times an hour that reads from the >> GlusterWWW volume mounted on node 1. Since none of the files in that mount >> are changing it doesn't actually rsync anything, but nonetheless it is >> running and reading the files in addition to my test script. (It's a part >> of my intended production setup that I forgot was still running.) >> >> The mount process appears to be gaining memory at a rate of about 1GB >> every 4 hours or so. At that rate it'll take several days before it runs >> the box out of memory. But I took your suggestion and made some statedumps >> today anyway, about 2 hours apart, 4 total so far. It looks like there may >> already be some actionable information. These are the only registers where >> the num_allocs have grown with each of the four samples: >> >> [mount/fuse.fuse - usage-type gf_fuse_mt_gids_t memusage] >> ---> num_allocs at Fri Jan 26 08:57:31 2018: 784 >> ---> num_allocs at Fri Jan 26 10:55:50 2018: 831 >> ---> num_allocs at Fri Jan 26 12:55:15 2018: 877 >> ---> num_allocs at Fri Jan 26 14:58:27 2018: 908 >> >> [mount/fuse.fuse - usage-type gf_common_mt_fd_lk_ctx_t memusage] >> ---> num_allocs at Fri Jan 26 08:57:31 2018: 5 >> ---> num_allocs at Fri Jan 26 10:55:50 2018: 10 >> ---> num_allocs at Fri Jan 26 12:55:15 2018: 15 >> ---> num_allocs at Fri Jan 26 14:58:27 2018: 17 >> >> [cluster/distribute.GlusterWWW-dht - usage-type gf_dht_mt_dht_layout_t >> memusage] >> ---> num_allocs at Fri Jan 26 08:57:31 2018: 24243596 >> ---> num_allocs at Fri Jan 26 10:55:50 2018: 27902622 >> ---> num_allocs at Fri Jan 26 12:55:15 2018: 30678066 >> ---> num_allocs at Fri Jan 26 14:58:27 2018: 33801036 >> >> Not sure the best way to get you the full dumps. They're pretty big, over >> 1G for all four. Also, I noticed some filepath information in there that >> I'd rather not share. What's the recommended next step? >> > > I've CC'd the fuse/ dht devs to see if these data types have potential > leaks. Could you raise a bug with the volume info and a (dropbox?) link > from which we can download the dumps? You can remove/replace the filepaths > from them. > > Regards. > Ravi > > > >> Cheers! >> >> Dan >> >> >>>> Is there potentially something misconfigured here? >>>> >>>> I did see a reference to a memory leak in another thread in this list, >>>> but that had to do with the setting of quotas, I don't have >>>> any quotas set on my system. >>>> >>>> Thanks, >>>> >>>> Dan Ragle >>>> daniel at Biblestuph.com >>>> >>>> On 1/25/2018 11:04 AM, Dan Ragle wrote: >>>> >>>>> Having a memory issue with Gluster 3.12.4 and not sure how to >>>>> troubleshoot. I don't *think* this is expected behavior. This is on an >>>>> updated CentOS 7 box. The setup is a simple two node replicated layout >>>>> where the two nodes act as both server and client. The volume in >>>>> question: Volume Name: GlusterWWW Type: Replicate Volume ID: >>>>> 8e9b0e79-f309-4d9b-a5bb-45d065faaaa3 Status: Started Snapshot Count: 0 >>>>> Number of Bricks: 1 x 2 = 2 Transport-type: tcp Bricks: Brick1: >>>>> vs1dlan.mydomain.com:/glusterfs_bricks/brick1/www Brick2: >>>>> vs2dlan.mydomain.com:/glusterfs_bricks/brick1/www Options >>>>> Reconfigured: >>>>> nfs.disable: on cluster.favorite-child-policy: mtime >>>>> transport.address-family: inet I had some other performance options in >>>>> there, (increased cache-size, md invalidation, etc) but stripped them >>>>> out in an attempt to isolate the issue. Still got the problem without >>>>> them. The volume currently contains over 1M files. When mounting the >>>>> volume, I get (among other things) a process as such: >>>>> /usr/sbin/glusterfs --volfile-server=localhost --volfile-id=/GlusterWWW >>>>> /var/www This process begins with little memory, but then as files are >>>>> accessed in the volume the memory increases. I setup a script that >>>>> simply reads the files in the volume one at a time (no writes). It's >>>>> been running on and off about 12 hours now and the resident memory of >>>>> the above process is already at 7.5G and continues to grow slowly. If I >>>>> stop the test script the memory stops growing, but does not reduce. >>>>> Restart the test script and the memory begins slowly growing again. >>>>> This >>>>> is obviously a contrived app environment. With my intended application >>>>> load it takes about a week or so for the memory to get high enough to >>>>> invoke the oom killer. Is there potentially something misconfigured >>>>> here? Thanks, Dan Ragle daniel at Biblestuph.com >>>>> >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> Gluster-users mailing list >>>>> Gluster-users at gluster.org >>>>> http://lists.gluster.org/mailman/listinfo/gluster-users >>>>> >>>>> _______________________________________________ >>>> Gluster-users mailing list >>>> Gluster-users at gluster.org >>>> http://lists.gluster.org/mailman/listinfo/gluster-users >>>> >>> >>> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org >> http://lists.gluster.org/mailman/listinfo/gluster-users >> > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20180129/4340de6d/attachment.html>
Raghavendra Gowdappa
2018-Jan-29 07:30 UTC
[Gluster-users] Run away memory with gluster mount
----- Original Message -----> From: "Nithya Balachandran" <nbalacha at redhat.com> > To: "Ravishankar N" <ravishankar at redhat.com> > Cc: "Csaba Henk" <chenk at redhat.com>, "gluster-users" <gluster-users at gluster.org> > Sent: Monday, January 29, 2018 10:49:43 AM > Subject: Re: [Gluster-users] Run away memory with gluster mount > > Csaba, > > Could this be the problem of the inodes not getting freed in the fuse > process?We can answer that question only after looking into statedumps. If we find too many inodes in fuse inode table's lru list (with refcount 0, lookup count > 0), it could be because sub-optimal garbage collection of inodes.> > Daniel, > as Ravi requested, please provide access to the statedumps. You can strip out > the filepath information. > Does your data set include a lot of directories? > > > Thanks, > Nithya > > On 27 January 2018 at 10:23, Ravishankar N < ravishankar at redhat.com > wrote: > > > > > > On 01/27/2018 02:29 AM, Dan Ragle wrote: > > > > On 1/25/2018 8:21 PM, Ravishankar N wrote: > > > > > On 01/25/2018 11:04 PM, Dan Ragle wrote: > > > *sigh* trying again to correct formatting ... apologize for the earlier mess. > > Having a memory issue with Gluster 3.12.4 and not sure how to troubleshoot. I > don't *think* this is expected behavior. > > This is on an updated CentOS 7 box. The setup is a simple two node replicated > layout where the two nodes act as both server and > client. > > The volume in question: > > Volume Name: GlusterWWW > Type: Replicate > Volume ID: 8e9b0e79-f309-4d9b-a5bb-45d065faaaa3 > Status: Started > Snapshot Count: 0 > Number of Bricks: 1 x 2 = 2 > Transport-type: tcp > Bricks: > Brick1: vs1dlan.mydomain.com:/glusterfs_bricks/brick1/www > Brick2: vs2dlan.mydomain.com:/glusterfs_bricks/brick1/www > Options Reconfigured: > nfs.disable: on > cluster.favorite-child-policy: mtime > transport.address-family: inet > > I had some other performance options in there, (increased cache-size, md > invalidation, etc) but stripped them out in an attempt to > isolate the issue. Still got the problem without them. > > The volume currently contains over 1M files. > > When mounting the volume, I get (among other things) a process as such: > > /usr/sbin/glusterfs --volfile-server=localhost --volfile-id=/GlusterWWW > /var/www > > This process begins with little memory, but then as files are accessed in the > volume the memory increases. I setup a script that > simply reads the files in the volume one at a time (no writes). It's been > running on and off about 12 hours now and the resident > memory of the above process is already at 7.5G and continues to grow slowly. > If I stop the test script the memory stops growing, > but does not reduce. Restart the test script and the memory begins slowly > growing again. > > This is obviously a contrived app environment. With my intended application > load it takes about a week or so for the memory to get > high enough to invoke the oom killer. > > Can you try debugging with the statedump ( > https://gluster.readthedocs.io/en/latest/Troubleshooting/statedump/#read-a-statedump > ) of > the fuse mount process and see what member is leaking? Take the statedumps in > succession, maybe once initially during the I/O and > once the memory gets high enough to hit the OOM mark. > Share the dumps here. > > Regards, > Ravi > > Thanks for the reply. I noticed yesterday that an update (3.12.5) had been > posted so I went ahead and updated and repeated the test overnight. The > memory usage does not appear to be growing as quickly as is was with 3.12.4, > but does still appear to be growing. > > I should also mention that there is another process beyond my test app that > is reading the files from the volume. Specifically, there is an rsync that > runs from the second node 2-4 times an hour that reads from the GlusterWWW > volume mounted on node 1. Since none of the files in that mount are changing > it doesn't actually rsync anything, but nonetheless it is running and > reading the files in addition to my test script. (It's a part of my intended > production setup that I forgot was still running.) > > The mount process appears to be gaining memory at a rate of about 1GB every 4 > hours or so. At that rate it'll take several days before it runs the box out > of memory. But I took your suggestion and made some statedumps today anyway, > about 2 hours apart, 4 total so far. It looks like there may already be some > actionable information. These are the only registers where the num_allocs > have grown with each of the four samples: > > [mount/fuse.fuse - usage-type gf_fuse_mt_gids_t memusage] > ---> num_allocs at Fri Jan 26 08:57:31 2018: 784 > ---> num_allocs at Fri Jan 26 10:55:50 2018: 831 > ---> num_allocs at Fri Jan 26 12:55:15 2018: 877 > ---> num_allocs at Fri Jan 26 14:58:27 2018: 908 > > [mount/fuse.fuse - usage-type gf_common_mt_fd_lk_ctx_t memusage] > ---> num_allocs at Fri Jan 26 08:57:31 2018: 5 > ---> num_allocs at Fri Jan 26 10:55:50 2018: 10 > ---> num_allocs at Fri Jan 26 12:55:15 2018: 15 > ---> num_allocs at Fri Jan 26 14:58:27 2018: 17 > > [cluster/distribute.GlusterWWW-dht - usage-type gf_dht_mt_dht_layout_t > memusage] > ---> num_allocs at Fri Jan 26 08:57:31 2018: 24243596 > ---> num_allocs at Fri Jan 26 10:55:50 2018: 27902622 > ---> num_allocs at Fri Jan 26 12:55:15 2018: 30678066 > ---> num_allocs at Fri Jan 26 14:58:27 2018: 33801036 > > Not sure the best way to get you the full dumps. They're pretty big, over 1G > for all four. Also, I noticed some filepath information in there that I'd > rather not share. What's the recommended next step? > > I've CC'd the fuse/ dht devs to see if these data types have potential leaks. > Could you raise a bug with the volume info and a (dropbox?) link from which > we can download the dumps? You can remove/replace the filepaths from them. > > Regards. > Ravi > > > > > > Cheers! > > Dan > > > > > > > Is there potentially something misconfigured here? > > I did see a reference to a memory leak in another thread in this list, but > that had to do with the setting of quotas, I don't have > any quotas set on my system. > > Thanks, > > Dan Ragle > daniel at Biblestuph.com > > On 1/25/2018 11:04 AM, Dan Ragle wrote: > > > Having a memory issue with Gluster 3.12.4 and not sure how to > troubleshoot. I don't *think* this is expected behavior. This is on an > updated CentOS 7 box. The setup is a simple two node replicated layout > where the two nodes act as both server and client. The volume in > question: Volume Name: GlusterWWW Type: Replicate Volume ID: > 8e9b0e79-f309-4d9b-a5bb-45d065faaaa3 Status: Started Snapshot Count: 0 > Number of Bricks: 1 x 2 = 2 Transport-type: tcp Bricks: Brick1: > vs1dlan.mydomain.com:/glusterfs_bricks/brick1/www Brick2: > vs2dlan.mydomain.com:/glusterfs_bricks/brick1/www Options Reconfigured: > nfs.disable: on cluster.favorite-child-policy: mtime > transport.address-family: inet I had some other performance options in > there, (increased cache-size, md invalidation, etc) but stripped them > out in an attempt to isolate the issue. Still got the problem without > them. The volume currently contains over 1M files. When mounting the > volume, I get (among other things) a process as such: > /usr/sbin/glusterfs --volfile-server=localhost --volfile-id=/GlusterWWW > /var/www This process begins with little memory, but then as files are > accessed in the volume the memory increases. I setup a script that > simply reads the files in the volume one at a time (no writes). It's > been running on and off about 12 hours now and the resident memory of > the above process is already at 7.5G and continues to grow slowly. If I > stop the test script the memory stops growing, but does not reduce. > Restart the test script and the memory begins slowly growing again. This > is obviously a contrived app environment. With my intended application > load it takes about a week or so for the memory to get high enough to > invoke the oom killer. Is there potentially something misconfigured > here? Thanks, Dan Ragle daniel at Biblestuph.com > > > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://lists.gluster.org/mailman/listinfo/gluster-users > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://lists.gluster.org/mailman/listinfo/gluster-users > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://lists.gluster.org/mailman/listinfo/gluster-users > > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://lists.gluster.org/mailman/listinfo/gluster-users
Raghavendra Gowdappa
2018-Jan-29 07:36 UTC
[Gluster-users] Run away memory with gluster mount
----- Original Message -----> From: "Ravishankar N" <ravishankar at redhat.com> > To: "Dan Ragle" <daniel at Biblestuph.com>, gluster-users at gluster.org > Cc: "Csaba Henk" <chenk at redhat.com>, "Niels de Vos" <ndevos at redhat.com>, "Nithya Balachandran" <nbalacha at redhat.com>, > "Raghavendra Gowdappa" <rgowdapp at redhat.com> > Sent: Saturday, January 27, 2018 10:23:38 AM > Subject: Re: [Gluster-users] Run away memory with gluster mount > > > > On 01/27/2018 02:29 AM, Dan Ragle wrote: > > > > On 1/25/2018 8:21 PM, Ravishankar N wrote: > >> > >> > >> On 01/25/2018 11:04 PM, Dan Ragle wrote: > >>> *sigh* trying again to correct formatting ... apologize for the > >>> earlier mess. > >>> > >>> Having a memory issue with Gluster 3.12.4 and not sure how to > >>> troubleshoot. I don't *think* this is expected behavior. > >>> > >>> This is on an updated CentOS 7 box. The setup is a simple two node > >>> replicated layout where the two nodes act as both server and > >>> client. > >>> > >>> The volume in question: > >>> > >>> Volume Name: GlusterWWW > >>> Type: Replicate > >>> Volume ID: 8e9b0e79-f309-4d9b-a5bb-45d065faaaa3 > >>> Status: Started > >>> Snapshot Count: 0 > >>> Number of Bricks: 1 x 2 = 2 > >>> Transport-type: tcp > >>> Bricks: > >>> Brick1: vs1dlan.mydomain.com:/glusterfs_bricks/brick1/www > >>> Brick2: vs2dlan.mydomain.com:/glusterfs_bricks/brick1/www > >>> Options Reconfigured: > >>> nfs.disable: on > >>> cluster.favorite-child-policy: mtime > >>> transport.address-family: inet > >>> > >>> I had some other performance options in there, (increased > >>> cache-size, md invalidation, etc) but stripped them out in an > >>> attempt to > >>> isolate the issue. Still got the problem without them. > >>> > >>> The volume currently contains over 1M files. > >>> > >>> When mounting the volume, I get (among other things) a process as such: > >>> > >>> /usr/sbin/glusterfs --volfile-server=localhost > >>> --volfile-id=/GlusterWWW /var/www > >>> > >>> This process begins with little memory, but then as files are > >>> accessed in the volume the memory increases. I setup a script that > >>> simply reads the files in the volume one at a time (no writes). It's > >>> been running on and off about 12 hours now and the resident > >>> memory of the above process is already at 7.5G and continues to grow > >>> slowly. If I stop the test script the memory stops growing, > >>> but does not reduce. Restart the test script and the memory begins > >>> slowly growing again. > >>> > >>> This is obviously a contrived app environment. With my intended > >>> application load it takes about a week or so for the memory to get > >>> high enough to invoke the oom killer. > >> > >> Can you try debugging with the statedump > >> (https://gluster.readthedocs.io/en/latest/Troubleshooting/statedump/#read-a-statedump) > >> of > >> the fuse mount process and see what member is leaking? Take the > >> statedumps in succession, maybe once initially during the I/O and > >> once the memory gets high enough to hit the OOM mark. > >> Share the dumps here. > >> > >> Regards, > >> Ravi > > > > Thanks for the reply. I noticed yesterday that an update (3.12.5) had > > been posted so I went ahead and updated and repeated the test > > overnight. The memory usage does not appear to be growing as quickly > > as is was with 3.12.4, but does still appear to be growing. > > > > I should also mention that there is another process beyond my test app > > that is reading the files from the volume. Specifically, there is an > > rsync that runs from the second node 2-4 times an hour that reads from > > the GlusterWWW volume mounted on node 1. Since none of the files in > > that mount are changing it doesn't actually rsync anything, but > > nonetheless it is running and reading the files in addition to my test > > script. (It's a part of my intended production setup that I forgot was > > still running.) > > > > The mount process appears to be gaining memory at a rate of about 1GB > > every 4 hours or so. At that rate it'll take several days before it > > runs the box out of memory. But I took your suggestion and made some > > statedumps today anyway, about 2 hours apart, 4 total so far. It looks > > like there may already be some actionable information. These are the > > only registers where the num_allocs have grown with each of the four > > samples: > > > > [mount/fuse.fuse - usage-type gf_fuse_mt_gids_t memusage] > > ?---> num_allocs at Fri Jan 26 08:57:31 2018: 784 > > ?---> num_allocs at Fri Jan 26 10:55:50 2018: 831 > > ?---> num_allocs at Fri Jan 26 12:55:15 2018: 877 > > ?---> num_allocs at Fri Jan 26 14:58:27 2018: 908 > > > > [mount/fuse.fuse - usage-type gf_common_mt_fd_lk_ctx_t memusage] > > ?---> num_allocs at Fri Jan 26 08:57:31 2018: 5 > > ?---> num_allocs at Fri Jan 26 10:55:50 2018: 10 > > ?---> num_allocs at Fri Jan 26 12:55:15 2018: 15 > > ?---> num_allocs at Fri Jan 26 14:58:27 2018: 17 > > > > [cluster/distribute.GlusterWWW-dht - usage-type gf_dht_mt_dht_layout_t > > memusage] > > ?---> num_allocs at Fri Jan 26 08:57:31 2018: 24243596 > > ?---> num_allocs at Fri Jan 26 10:55:50 2018: 27902622 > > ?---> num_allocs at Fri Jan 26 12:55:15 2018: 30678066 > > ?---> num_allocs at Fri Jan 26 14:58:27 2018: 33801036 > > > > Not sure the best way to get you the full dumps. They're pretty big, > > over 1G for all four. Also, I noticed some filepath information in > > there that I'd rather not share. What's the recommended next step?Please run the following query on statedump files and report us the results: # grep itable <client-statedump> | grep active | wc -l # grep itable <client-statedump> | grep active_size # grep itable <client-statedump> | grep lru | wc -l # grep itable <client-statedump> | grep lru_size # grep itable <client-statedump> | grep purge | wc -l # grep itable <client-statedump> | grep purge_size> > I've CC'd the fuse/ dht devs to see if these data types have potential > leaks. Could you raise a bug with the volume info and a (dropbox?) link > from which we can download the dumps? You can remove/replace the > filepaths from them. > > Regards. > Ravi > > > > > Cheers! > > > > Dan > > > >>> > >>> Is there potentially something misconfigured here? > >>> > >>> I did see a reference to a memory leak in another thread in this > >>> list, but that had to do with the setting of quotas, I don't have > >>> any quotas set on my system. > >>> > >>> Thanks, > >>> > >>> Dan Ragle > >>> daniel at Biblestuph.com > >>> > >>> On 1/25/2018 11:04 AM, Dan Ragle wrote: > >>>> Having a memory issue with Gluster 3.12.4 and not sure how to > >>>> troubleshoot. I don't *think* this is expected behavior. This is on an > >>>> updated CentOS 7 box. The setup is a simple two node replicated layout > >>>> where the two nodes act as both server and client. The volume in > >>>> question: Volume Name: GlusterWWW Type: Replicate Volume ID: > >>>> 8e9b0e79-f309-4d9b-a5bb-45d065faaaa3 Status: Started Snapshot Count: 0 > >>>> Number of Bricks: 1 x 2 = 2 Transport-type: tcp Bricks: Brick1: > >>>> vs1dlan.mydomain.com:/glusterfs_bricks/brick1/www Brick2: > >>>> vs2dlan.mydomain.com:/glusterfs_bricks/brick1/www Options > >>>> Reconfigured: > >>>> nfs.disable: on cluster.favorite-child-policy: mtime > >>>> transport.address-family: inet I had some other performance options in > >>>> there, (increased cache-size, md invalidation, etc) but stripped them > >>>> out in an attempt to isolate the issue. Still got the problem without > >>>> them. The volume currently contains over 1M files. When mounting the > >>>> volume, I get (among other things) a process as such: > >>>> /usr/sbin/glusterfs --volfile-server=localhost > >>>> --volfile-id=/GlusterWWW > >>>> /var/www This process begins with little memory, but then as files are > >>>> accessed in the volume the memory increases. I setup a script that > >>>> simply reads the files in the volume one at a time (no writes). It's > >>>> been running on and off about 12 hours now and the resident memory of > >>>> the above process is already at 7.5G and continues to grow slowly. > >>>> If I > >>>> stop the test script the memory stops growing, but does not reduce. > >>>> Restart the test script and the memory begins slowly growing again. > >>>> This > >>>> is obviously a contrived app environment. With my intended application > >>>> load it takes about a week or so for the memory to get high enough to > >>>> invoke the oom killer. Is there potentially something misconfigured > >>>> here? Thanks, Dan Ragle daniel at Biblestuph.com > >>>> > >>>> > >>>> > >>>> > >>>> _______________________________________________ > >>>> Gluster-users mailing list > >>>> Gluster-users at gluster.org > >>>> http://lists.gluster.org/mailman/listinfo/gluster-users > >>>> > >>> _______________________________________________ > >>> Gluster-users mailing list > >>> Gluster-users at gluster.org > >>> http://lists.gluster.org/mailman/listinfo/gluster-users > >> > > _______________________________________________ > > Gluster-users mailing list > > Gluster-users at gluster.org > > http://lists.gluster.org/mailman/listinfo/gluster-users > >
On 1/29/2018 12:19 AM, Nithya Balachandran wrote:> Csaba, > > Could this be the problem of the inodes not getting freed in the fuse > process? > > Daniel, > as Ravi requested, please provide access to the statedumps. You can strip > out the filepath information.Working on filing a bug report and getting you the dumps now. Will update soon.> Does your data set include a lot of directories?The volume in question has 1M+ files and 77k+ directories. Cheers! Dan> > > Thanks, > Nithya > > On 27 January 2018 at 10:23, Ravishankar N <ravishankar at redhat.com> wrote: > >> >> >> On 01/27/2018 02:29 AM, Dan Ragle wrote: >> >>> >>> On 1/25/2018 8:21 PM, Ravishankar N wrote: >>> >>>> >>>> >>>> On 01/25/2018 11:04 PM, Dan Ragle wrote: >>>> >>>>> *sigh* trying again to correct formatting ... apologize for the earlier >>>>> mess. >>>>> >>>>> Having a memory issue with Gluster 3.12.4 and not sure how to >>>>> troubleshoot. I don't *think* this is expected behavior. >>>>> >>>>> This is on an updated CentOS 7 box. The setup is a simple two node >>>>> replicated layout where the two nodes act as both server and >>>>> client. >>>>> >>>>> The volume in question: >>>>> >>>>> Volume Name: GlusterWWW >>>>> Type: Replicate >>>>> Volume ID: 8e9b0e79-f309-4d9b-a5bb-45d065faaaa3 >>>>> Status: Started >>>>> Snapshot Count: 0 >>>>> Number of Bricks: 1 x 2 = 2 >>>>> Transport-type: tcp >>>>> Bricks: >>>>> Brick1: vs1dlan.mydomain.com:/glusterfs_bricks/brick1/www >>>>> Brick2: vs2dlan.mydomain.com:/glusterfs_bricks/brick1/www >>>>> Options Reconfigured: >>>>> nfs.disable: on >>>>> cluster.favorite-child-policy: mtime >>>>> transport.address-family: inet >>>>> >>>>> I had some other performance options in there, (increased cache-size, >>>>> md invalidation, etc) but stripped them out in an attempt to >>>>> isolate the issue. Still got the problem without them. >>>>> >>>>> The volume currently contains over 1M files. >>>>> >>>>> When mounting the volume, I get (among other things) a process as such: >>>>> >>>>> /usr/sbin/glusterfs --volfile-server=localhost --volfile-id=/GlusterWWW >>>>> /var/www >>>>> >>>>> This process begins with little memory, but then as files are accessed >>>>> in the volume the memory increases. I setup a script that >>>>> simply reads the files in the volume one at a time (no writes). It's >>>>> been running on and off about 12 hours now and the resident >>>>> memory of the above process is already at 7.5G and continues to grow >>>>> slowly. If I stop the test script the memory stops growing, >>>>> but does not reduce. Restart the test script and the memory begins >>>>> slowly growing again. >>>>> >>>>> This is obviously a contrived app environment. With my intended >>>>> application load it takes about a week or so for the memory to get >>>>> high enough to invoke the oom killer. >>>>> >>>> >>>> Can you try debugging with the statedump (https://gluster.readthedocs.i >>>> o/en/latest/Troubleshooting/statedump/#read-a-statedump) of >>>> the fuse mount process and see what member is leaking? Take the >>>> statedumps in succession, maybe once initially during the I/O and >>>> once the memory gets high enough to hit the OOM mark. >>>> Share the dumps here. >>>> >>>> Regards, >>>> Ravi >>>> >>> >>> Thanks for the reply. I noticed yesterday that an update (3.12.5) had >>> been posted so I went ahead and updated and repeated the test overnight. >>> The memory usage does not appear to be growing as quickly as is was with >>> 3.12.4, but does still appear to be growing. >>> >>> I should also mention that there is another process beyond my test app >>> that is reading the files from the volume. Specifically, there is an rsync >>> that runs from the second node 2-4 times an hour that reads from the >>> GlusterWWW volume mounted on node 1. Since none of the files in that mount >>> are changing it doesn't actually rsync anything, but nonetheless it is >>> running and reading the files in addition to my test script. (It's a part >>> of my intended production setup that I forgot was still running.) >>> >>> The mount process appears to be gaining memory at a rate of about 1GB >>> every 4 hours or so. At that rate it'll take several days before it runs >>> the box out of memory. But I took your suggestion and made some statedumps >>> today anyway, about 2 hours apart, 4 total so far. It looks like there may >>> already be some actionable information. These are the only registers where >>> the num_allocs have grown with each of the four samples: >>> >>> [mount/fuse.fuse - usage-type gf_fuse_mt_gids_t memusage] >>> ---> num_allocs at Fri Jan 26 08:57:31 2018: 784 >>> ---> num_allocs at Fri Jan 26 10:55:50 2018: 831 >>> ---> num_allocs at Fri Jan 26 12:55:15 2018: 877 >>> ---> num_allocs at Fri Jan 26 14:58:27 2018: 908 >>> >>> [mount/fuse.fuse - usage-type gf_common_mt_fd_lk_ctx_t memusage] >>> ---> num_allocs at Fri Jan 26 08:57:31 2018: 5 >>> ---> num_allocs at Fri Jan 26 10:55:50 2018: 10 >>> ---> num_allocs at Fri Jan 26 12:55:15 2018: 15 >>> ---> num_allocs at Fri Jan 26 14:58:27 2018: 17 >>> >>> [cluster/distribute.GlusterWWW-dht - usage-type gf_dht_mt_dht_layout_t >>> memusage] >>> ---> num_allocs at Fri Jan 26 08:57:31 2018: 24243596 >>> ---> num_allocs at Fri Jan 26 10:55:50 2018: 27902622 >>> ---> num_allocs at Fri Jan 26 12:55:15 2018: 30678066 >>> ---> num_allocs at Fri Jan 26 14:58:27 2018: 33801036 >>> >>> Not sure the best way to get you the full dumps. They're pretty big, over >>> 1G for all four. Also, I noticed some filepath information in there that >>> I'd rather not share. What's the recommended next step? >>> >> >> I've CC'd the fuse/ dht devs to see if these data types have potential >> leaks. Could you raise a bug with the volume info and a (dropbox?) link >> from which we can download the dumps? You can remove/replace the filepaths >> from them. >> >> Regards. >> Ravi >> >> >> >>> Cheers! >>> >>> Dan >>> >>> >>>>> Is there potentially something misconfigured here? >>>>> >>>>> I did see a reference to a memory leak in another thread in this list, >>>>> but that had to do with the setting of quotas, I don't have >>>>> any quotas set on my system. >>>>> >>>>> Thanks, >>>>> >>>>> Dan Ragle >>>>> daniel at Biblestuph.com >>>>> >>>>> On 1/25/2018 11:04 AM, Dan Ragle wrote: >>>>> >>>>>> Having a memory issue with Gluster 3.12.4 and not sure how to >>>>>> troubleshoot. I don't *think* this is expected behavior. This is on an >>>>>> updated CentOS 7 box. The setup is a simple two node replicated layout >>>>>> where the two nodes act as both server and client. The volume in >>>>>> question: Volume Name: GlusterWWW Type: Replicate Volume ID: >>>>>> 8e9b0e79-f309-4d9b-a5bb-45d065faaaa3 Status: Started Snapshot Count: 0 >>>>>> Number of Bricks: 1 x 2 = 2 Transport-type: tcp Bricks: Brick1: >>>>>> vs1dlan.mydomain.com:/glusterfs_bricks/brick1/www Brick2: >>>>>> vs2dlan.mydomain.com:/glusterfs_bricks/brick1/www Options >>>>>> Reconfigured: >>>>>> nfs.disable: on cluster.favorite-child-policy: mtime >>>>>> transport.address-family: inet I had some other performance options in >>>>>> there, (increased cache-size, md invalidation, etc) but stripped them >>>>>> out in an attempt to isolate the issue. Still got the problem without >>>>>> them. The volume currently contains over 1M files. When mounting the >>>>>> volume, I get (among other things) a process as such: >>>>>> /usr/sbin/glusterfs --volfile-server=localhost --volfile-id=/GlusterWWW >>>>>> /var/www This process begins with little memory, but then as files are >>>>>> accessed in the volume the memory increases. I setup a script that >>>>>> simply reads the files in the volume one at a time (no writes). It's >>>>>> been running on and off about 12 hours now and the resident memory of >>>>>> the above process is already at 7.5G and continues to grow slowly. If I >>>>>> stop the test script the memory stops growing, but does not reduce. >>>>>> Restart the test script and the memory begins slowly growing again. >>>>>> This >>>>>> is obviously a contrived app environment. With my intended application >>>>>> load it takes about a week or so for the memory to get high enough to >>>>>> invoke the oom killer. Is there potentially something misconfigured >>>>>> here? Thanks, Dan Ragle daniel at Biblestuph.com >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> Gluster-users mailing list >>>>>> Gluster-users at gluster.org >>>>>> http://lists.gluster.org/mailman/listinfo/gluster-users >>>>>> >>>>>> _______________________________________________ >>>>> Gluster-users mailing list >>>>> Gluster-users at gluster.org >>>>> http://lists.gluster.org/mailman/listinfo/gluster-users >>>>> >>>> >>>> _______________________________________________ >>> Gluster-users mailing list >>> Gluster-users at gluster.org >>> http://lists.gluster.org/mailman/listinfo/gluster-users >>> >> >> >
On 1/29/2018 2:36 AM, Raghavendra Gowdappa wrote:> > > ----- Original Message ----- >> From: "Ravishankar N" <ravishankar at redhat.com> >> To: "Dan Ragle" <daniel at Biblestuph.com>, gluster-users at gluster.org >> Cc: "Csaba Henk" <chenk at redhat.com>, "Niels de Vos" <ndevos at redhat.com>, "Nithya Balachandran" <nbalacha at redhat.com>, >> "Raghavendra Gowdappa" <rgowdapp at redhat.com> >> Sent: Saturday, January 27, 2018 10:23:38 AM >> Subject: Re: [Gluster-users] Run away memory with gluster mount >> >> >> >> On 01/27/2018 02:29 AM, Dan Ragle wrote: >>> >>> On 1/25/2018 8:21 PM, Ravishankar N wrote: >>>> >>>> >>>> On 01/25/2018 11:04 PM, Dan Ragle wrote: >>>>> *sigh* trying again to correct formatting ... apologize for the >>>>> earlier mess. >>>>> >>>>> Having a memory issue with Gluster 3.12.4 and not sure how to >>>>> troubleshoot. I don't *think* this is expected behavior. >>>>> >>>>> This is on an updated CentOS 7 box. The setup is a simple two node >>>>> replicated layout where the two nodes act as both server and >>>>> client. >>>>> >>>>> The volume in question: >>>>> >>>>> Volume Name: GlusterWWW >>>>> Type: Replicate >>>>> Volume ID: 8e9b0e79-f309-4d9b-a5bb-45d065faaaa3 >>>>> Status: Started >>>>> Snapshot Count: 0 >>>>> Number of Bricks: 1 x 2 = 2 >>>>> Transport-type: tcp >>>>> Bricks: >>>>> Brick1: vs1dlan.mydomain.com:/glusterfs_bricks/brick1/www >>>>> Brick2: vs2dlan.mydomain.com:/glusterfs_bricks/brick1/www >>>>> Options Reconfigured: >>>>> nfs.disable: on >>>>> cluster.favorite-child-policy: mtime >>>>> transport.address-family: inet >>>>> >>>>> I had some other performance options in there, (increased >>>>> cache-size, md invalidation, etc) but stripped them out in an >>>>> attempt to >>>>> isolate the issue. Still got the problem without them. >>>>> >>>>> The volume currently contains over 1M files. >>>>> >>>>> When mounting the volume, I get (among other things) a process as such: >>>>> >>>>> /usr/sbin/glusterfs --volfile-server=localhost >>>>> --volfile-id=/GlusterWWW /var/www >>>>> >>>>> This process begins with little memory, but then as files are >>>>> accessed in the volume the memory increases. I setup a script that >>>>> simply reads the files in the volume one at a time (no writes). It's >>>>> been running on and off about 12 hours now and the resident >>>>> memory of the above process is already at 7.5G and continues to grow >>>>> slowly. If I stop the test script the memory stops growing, >>>>> but does not reduce. Restart the test script and the memory begins >>>>> slowly growing again. >>>>> >>>>> This is obviously a contrived app environment. With my intended >>>>> application load it takes about a week or so for the memory to get >>>>> high enough to invoke the oom killer. >>>> >>>> Can you try debugging with the statedump >>>> (https://gluster.readthedocs.io/en/latest/Troubleshooting/statedump/#read-a-statedump) >>>> of >>>> the fuse mount process and see what member is leaking? Take the >>>> statedumps in succession, maybe once initially during the I/O and >>>> once the memory gets high enough to hit the OOM mark. >>>> Share the dumps here. >>>> >>>> Regards, >>>> Ravi >>> >>> Thanks for the reply. I noticed yesterday that an update (3.12.5) had >>> been posted so I went ahead and updated and repeated the test >>> overnight. The memory usage does not appear to be growing as quickly >>> as is was with 3.12.4, but does still appear to be growing. >>> >>> I should also mention that there is another process beyond my test app >>> that is reading the files from the volume. Specifically, there is an >>> rsync that runs from the second node 2-4 times an hour that reads from >>> the GlusterWWW volume mounted on node 1. Since none of the files in >>> that mount are changing it doesn't actually rsync anything, but >>> nonetheless it is running and reading the files in addition to my test >>> script. (It's a part of my intended production setup that I forgot was >>> still running.) >>> >>> The mount process appears to be gaining memory at a rate of about 1GB >>> every 4 hours or so. At that rate it'll take several days before it >>> runs the box out of memory. But I took your suggestion and made some >>> statedumps today anyway, about 2 hours apart, 4 total so far. It looks >>> like there may already be some actionable information. These are the >>> only registers where the num_allocs have grown with each of the four >>> samples: >>> >>> [mount/fuse.fuse - usage-type gf_fuse_mt_gids_t memusage] >>> ?---> num_allocs at Fri Jan 26 08:57:31 2018: 784 >>> ?---> num_allocs at Fri Jan 26 10:55:50 2018: 831 >>> ?---> num_allocs at Fri Jan 26 12:55:15 2018: 877 >>> ?---> num_allocs at Fri Jan 26 14:58:27 2018: 908 >>> >>> [mount/fuse.fuse - usage-type gf_common_mt_fd_lk_ctx_t memusage] >>> ?---> num_allocs at Fri Jan 26 08:57:31 2018: 5 >>> ?---> num_allocs at Fri Jan 26 10:55:50 2018: 10 >>> ?---> num_allocs at Fri Jan 26 12:55:15 2018: 15 >>> ?---> num_allocs at Fri Jan 26 14:58:27 2018: 17 >>> >>> [cluster/distribute.GlusterWWW-dht - usage-type gf_dht_mt_dht_layout_t >>> memusage] >>> ?---> num_allocs at Fri Jan 26 08:57:31 2018: 24243596 >>> ?---> num_allocs at Fri Jan 26 10:55:50 2018: 27902622 >>> ?---> num_allocs at Fri Jan 26 12:55:15 2018: 30678066 >>> ?---> num_allocs at Fri Jan 26 14:58:27 2018: 33801036 >>> >>> Not sure the best way to get you the full dumps. They're pretty big, >>> over 1G for all four. Also, I noticed some filepath information in >>> there that I'd rather not share. What's the recommended next step? > > Please run the following query on statedump files and report us the results: > # grep itable <client-statedump> | grep active | wc -l > # grep itable <client-statedump> | grep active_size > # grep itable <client-statedump> | grep lru | wc -l > # grep itable <client-statedump> | grep lru_size > # grep itable <client-statedump> | grep purge | wc -l > # grep itable <client-statedump> | grep purge_sizeHad to restart the test and have been running for 36 hours now. RSS is currently up to 23g. Working on getting a bug report with link to the dumps. In the mean time, I'm including the results of your above queries for the first dump, the 18 hour dump, and the 36 hour dump: # grep itable glusterdump.153904.dump.1517104561 | grep active | wc -l 53865 # grep itable glusterdump.153904.dump.1517169361 | grep active | wc -l 53864 # grep itable glusterdump.153904.dump.1517234161 | grep active | wc -l 53864 # grep itable glusterdump.153904.dump.1517104561 | grep active_size xlator.mount.fuse.itable.active_size=53864 # grep itable glusterdump.153904.dump.1517169361 | grep active_size xlator.mount.fuse.itable.active_size=53863 # grep itable glusterdump.153904.dump.1517234161 | grep active_size xlator.mount.fuse.itable.active_size=53863 # grep itable glusterdump.153904.dump.1517104561 | grep lru | wc -l 998510 # grep itable glusterdump.153904.dump.1517169361 | grep lru | wc -l 998510 # grep itable glusterdump.153904.dump.1517234161 | grep lru | wc -l 995992 # grep itable glusterdump.153904.dump.1517104561 | grep lru_size xlator.mount.fuse.itable.lru_size=998508 # grep itable glusterdump.153904.dump.1517169361 | grep lru_size xlator.mount.fuse.itable.lru_size=998508 # grep itable glusterdump.153904.dump.1517234161 | grep lru_size xlator.mount.fuse.itable.lru_size=995990 # grep itable glusterdump.153904.dump.1517104561 | grep purge | wc -l 1 # grep itable glusterdump.153904.dump.1517169361 | grep purge | wc -l 1 # grep itable glusterdump.153904.dump.1517234161 | grep purge | wc -l 1 # grep itable glusterdump.153904.dump.1517104561 | grep purge_size xlator.mount.fuse.itable.purge_size=0 # grep itable glusterdump.153904.dump.1517169361 | grep purge_size xlator.mount.fuse.itable.purge_size=0 # grep itable glusterdump.153904.dump.1517234161 | grep purge_size xlator.mount.fuse.itable.purge_size=0 Cheers, Dan> >> >> I've CC'd the fuse/ dht devs to see if these data types have potential >> leaks. Could you raise a bug with the volume info and a (dropbox?) link >> from which we can download the dumps? You can remove/replace the >> filepaths from them. >> >> Regards. >> Ravi >> >>> >>> Cheers! >>> >>> Dan >>> >>>>> >>>>> Is there potentially something misconfigured here? >>>>> >>>>> I did see a reference to a memory leak in another thread in this >>>>> list, but that had to do with the setting of quotas, I don't have >>>>> any quotas set on my system. >>>>> >>>>> Thanks, >>>>> >>>>> Dan Ragle >>>>> daniel at Biblestuph.com >>>>> >>>>> On 1/25/2018 11:04 AM, Dan Ragle wrote: >>>>>> Having a memory issue with Gluster 3.12.4 and not sure how to >>>>>> troubleshoot. I don't *think* this is expected behavior. This is on an >>>>>> updated CentOS 7 box. The setup is a simple two node replicated layout >>>>>> where the two nodes act as both server and client. The volume in >>>>>> question: Volume Name: GlusterWWW Type: Replicate Volume ID: >>>>>> 8e9b0e79-f309-4d9b-a5bb-45d065faaaa3 Status: Started Snapshot Count: 0 >>>>>> Number of Bricks: 1 x 2 = 2 Transport-type: tcp Bricks: Brick1: >>>>>> vs1dlan.mydomain.com:/glusterfs_bricks/brick1/www Brick2: >>>>>> vs2dlan.mydomain.com:/glusterfs_bricks/brick1/www Options >>>>>> Reconfigured: >>>>>> nfs.disable: on cluster.favorite-child-policy: mtime >>>>>> transport.address-family: inet I had some other performance options in >>>>>> there, (increased cache-size, md invalidation, etc) but stripped them >>>>>> out in an attempt to isolate the issue. Still got the problem without >>>>>> them. The volume currently contains over 1M files. When mounting the >>>>>> volume, I get (among other things) a process as such: >>>>>> /usr/sbin/glusterfs --volfile-server=localhost >>>>>> --volfile-id=/GlusterWWW >>>>>> /var/www This process begins with little memory, but then as files are >>>>>> accessed in the volume the memory increases. I setup a script that >>>>>> simply reads the files in the volume one at a time (no writes). It's >>>>>> been running on and off about 12 hours now and the resident memory of >>>>>> the above process is already at 7.5G and continues to grow slowly. >>>>>> If I >>>>>> stop the test script the memory stops growing, but does not reduce. >>>>>> Restart the test script and the memory begins slowly growing again. >>>>>> This >>>>>> is obviously a contrived app environment. With my intended application >>>>>> load it takes about a week or so for the memory to get high enough to >>>>>> invoke the oom killer. Is there potentially something misconfigured >>>>>> here? Thanks, Dan Ragle daniel at Biblestuph.com >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> Gluster-users mailing list >>>>>> Gluster-users at gluster.org >>>>>> http://lists.gluster.org/mailman/listinfo/gluster-users >>>>>> >>>>> _______________________________________________ >>>>> Gluster-users mailing list >>>>> Gluster-users at gluster.org >>>>> http://lists.gluster.org/mailman/listinfo/gluster-users >>>> >>> _______________________________________________ >>> Gluster-users mailing list >>> Gluster-users at gluster.org >>> http://lists.gluster.org/mailman/listinfo/gluster-users >> >> >
On 1/26/2018 11:53 PM, Ravishankar N wrote:> > > On 01/27/2018 02:29 AM, Dan Ragle wrote: >> >> On 1/25/2018 8:21 PM, Ravishankar N wrote: >>> >>> >>> On 01/25/2018 11:04 PM, Dan Ragle wrote: >>>> *sigh* trying again to correct formatting ... apologize for the >>>> earlier mess. >>>> >>>> Having a memory issue with Gluster 3.12.4 and not sure how to >>>> troubleshoot. I don't *think* this is expected behavior. >>>> >>>> This is on an updated CentOS 7 box. The setup is a simple two node >>>> replicated layout where the two nodes act as both server and >>>> client. >>>> >>>> The volume in question: >>>> >>>> Volume Name: GlusterWWW >>>> Type: Replicate >>>> Volume ID: 8e9b0e79-f309-4d9b-a5bb-45d065faaaa3 >>>> Status: Started >>>> Snapshot Count: 0 >>>> Number of Bricks: 1 x 2 = 2 >>>> Transport-type: tcp >>>> Bricks: >>>> Brick1: vs1dlan.mydomain.com:/glusterfs_bricks/brick1/www >>>> Brick2: vs2dlan.mydomain.com:/glusterfs_bricks/brick1/www >>>> Options Reconfigured: >>>> nfs.disable: on >>>> cluster.favorite-child-policy: mtime >>>> transport.address-family: inet >>>> >>>> I had some other performance options in there, (increased >>>> cache-size, md invalidation, etc) but stripped them out in an >>>> attempt to >>>> isolate the issue. Still got the problem without them. >>>> >>>> The volume currently contains over 1M files. >>>> >>>> When mounting the volume, I get (among other things) a process as such: >>>> >>>> /usr/sbin/glusterfs --volfile-server=localhost >>>> --volfile-id=/GlusterWWW /var/www >>>> >>>> This process begins with little memory, but then as files are >>>> accessed in the volume the memory increases. I setup a script that >>>> simply reads the files in the volume one at a time (no writes). It's >>>> been running on and off about 12 hours now and the resident >>>> memory of the above process is already at 7.5G and continues to grow >>>> slowly. If I stop the test script the memory stops growing, >>>> but does not reduce. Restart the test script and the memory begins >>>> slowly growing again. >>>> >>>> This is obviously a contrived app environment. With my intended >>>> application load it takes about a week or so for the memory to get >>>> high enough to invoke the oom killer. >>> >>> Can you try debugging with the statedump >>> (https://gluster.readthedocs.io/en/latest/Troubleshooting/statedump/#read-a-statedump) >>> of >>> the fuse mount process and see what member is leaking? Take the >>> statedumps in succession, maybe once initially during the I/O and >>> once the memory gets high enough to hit the OOM mark. >>> Share the dumps here. >>> >>> Regards, >>> Ravi >> >> Thanks for the reply. I noticed yesterday that an update (3.12.5) had >> been posted so I went ahead and updated and repeated the test >> overnight. The memory usage does not appear to be growing as quickly >> as is was with 3.12.4, but does still appear to be growing. >> >> I should also mention that there is another process beyond my test app >> that is reading the files from the volume. Specifically, there is an >> rsync that runs from the second node 2-4 times an hour that reads from >> the GlusterWWW volume mounted on node 1. Since none of the files in >> that mount are changing it doesn't actually rsync anything, but >> nonetheless it is running and reading the files in addition to my test >> script. (It's a part of my intended production setup that I forgot was >> still running.) >> >> The mount process appears to be gaining memory at a rate of about 1GB >> every 4 hours or so. At that rate it'll take several days before it >> runs the box out of memory. But I took your suggestion and made some >> statedumps today anyway, about 2 hours apart, 4 total so far. It looks >> like there may already be some actionable information. These are the >> only registers where the num_allocs have grown with each of the four >> samples: >> >> [mount/fuse.fuse - usage-type gf_fuse_mt_gids_t memusage] >> ?---> num_allocs at Fri Jan 26 08:57:31 2018: 784 >> ?---> num_allocs at Fri Jan 26 10:55:50 2018: 831 >> ?---> num_allocs at Fri Jan 26 12:55:15 2018: 877 >> ?---> num_allocs at Fri Jan 26 14:58:27 2018: 908 >> >> [mount/fuse.fuse - usage-type gf_common_mt_fd_lk_ctx_t memusage] >> ?---> num_allocs at Fri Jan 26 08:57:31 2018: 5 >> ?---> num_allocs at Fri Jan 26 10:55:50 2018: 10 >> ?---> num_allocs at Fri Jan 26 12:55:15 2018: 15 >> ?---> num_allocs at Fri Jan 26 14:58:27 2018: 17 >> >> [cluster/distribute.GlusterWWW-dht - usage-type gf_dht_mt_dht_layout_t >> memusage] >> ?---> num_allocs at Fri Jan 26 08:57:31 2018: 24243596 >> ?---> num_allocs at Fri Jan 26 10:55:50 2018: 27902622 >> ?---> num_allocs at Fri Jan 26 12:55:15 2018: 30678066 >> ?---> num_allocs at Fri Jan 26 14:58:27 2018: 33801036 >> >> Not sure the best way to get you the full dumps. They're pretty big, >> over 1G for all four. Also, I noticed some filepath information in >> there that I'd rather not share. What's the recommended next step? > > I've CC'd the fuse/ dht devs to see if these data types have potential > leaks. Could you raise a bug with the volume info and a (dropbox?) link > from which we can download the dumps? You can remove/replace the > filepaths from them. > > Regards. > RaviFiled this bug with links to the tar balled statedumps: https://bugs.centos.org/view.php?id=14428 Since my testing platform is CentOS I didn't know if it would be appropriate to report it in the RedHat bugzilla. I found in further testing this weekend that I could reproduce the problem easily with this: while true; do /usr/bin/ls -R /var/www > /dev/null 2>&1; sleep 10; done & If anyone wants the statedumps directly they are here. The first (600MB+) has all 18 dumps while the second has only three (start, 18 hours, 36 hours; 110MB+). The file paths are obfuscated. https://drive.google.com/file/d/1e9UfkHwWZMhHPC480GYLr0gERV985134/view?usp=sharing https://drive.google.com/file/d/1HhJ-FTamd44XKZ5NDxaE4IjVtSOUFO83/view?usp=sharing Reviewing the num_allocs, the gf_dht_mt_dht_layout_t entry still appears to be the biggest standout: [cluster/distribute.GlusterWWW-dht - usage-type gf_dht_mt_dht_layout_t memusage] ---> num_allocs at Sat Jan 27 20:56:01 2018: 371847 ---> num_allocs at Sat Jan 27 22:56:01 2018: 6480129 ---> num_allocs at Sun Jan 28 00:56:01 2018: 13036666 ---> num_allocs at Sun Jan 28 02:56:01 2018: 19623082 ---> num_allocs at Sun Jan 28 04:56:01 2018: 26221450 ---> num_allocs at Sun Jan 28 06:56:01 2018: 32760766 ---> num_allocs at Sun Jan 28 08:56:01 2018: 39312051 ---> num_allocs at Sun Jan 28 10:56:01 2018: 45881896 ---> num_allocs at Sun Jan 28 12:56:01 2018: 52456557 ---> num_allocs at Sun Jan 28 14:56:01 2018: 59017422 ---> num_allocs at Sun Jan 28 16:56:01 2018: 65552767 ---> num_allocs at Sun Jan 28 18:56:01 2018: 72061290 ---> num_allocs at Sun Jan 28 20:56:01 2018: 78743296 ---> num_allocs at Sun Jan 28 22:56:01 2018: 85297985 ---> num_allocs at Mon Jan 29 00:56:01 2018: 91800496 ---> num_allocs at Mon Jan 29 02:56:01 2018: 98306935 ---> num_allocs at Mon Jan 29 04:56:01 2018: 104802985 ---> num_allocs at Mon Jan 29 06:56:01 2018: 111385407 ---> num_allocs at Mon Jan 29 08:56:01 2018: 117901593 Let me know if I can provide anything further. The test is actually still running, currently with RSS memory at 25GB+. Cheers! Dan> >> >> Cheers! >> >> Dan >> >>>> >>>> Is there potentially something misconfigured here? >>>> >>>> I did see a reference to a memory leak in another thread in this >>>> list, but that had to do with the setting of quotas, I don't have >>>> any quotas set on my system. >>>> >>>> Thanks, >>>> >>>> Dan Ragle >>>> daniel at Biblestuph.com >>>> >>>> On 1/25/2018 11:04 AM, Dan Ragle wrote: >>>>> Having a memory issue with Gluster 3.12.4 and not sure how to >>>>> troubleshoot. I don't *think* this is expected behavior. This is on an >>>>> updated CentOS 7 box. The setup is a simple two node replicated layout >>>>> where the two nodes act as both server and client. The volume in >>>>> question: Volume Name: GlusterWWW Type: Replicate Volume ID: >>>>> 8e9b0e79-f309-4d9b-a5bb-45d065faaaa3 Status: Started Snapshot Count: 0 >>>>> Number of Bricks: 1 x 2 = 2 Transport-type: tcp Bricks: Brick1: >>>>> vs1dlan.mydomain.com:/glusterfs_bricks/brick1/www Brick2: >>>>> vs2dlan.mydomain.com:/glusterfs_bricks/brick1/www Options >>>>> Reconfigured: >>>>> nfs.disable: on cluster.favorite-child-policy: mtime >>>>> transport.address-family: inet I had some other performance options in >>>>> there, (increased cache-size, md invalidation, etc) but stripped them >>>>> out in an attempt to isolate the issue. Still got the problem without >>>>> them. The volume currently contains over 1M files. When mounting the >>>>> volume, I get (among other things) a process as such: >>>>> /usr/sbin/glusterfs --volfile-server=localhost >>>>> --volfile-id=/GlusterWWW >>>>> /var/www This process begins with little memory, but then as files are >>>>> accessed in the volume the memory increases. I setup a script that >>>>> simply reads the files in the volume one at a time (no writes). It's >>>>> been running on and off about 12 hours now and the resident memory of >>>>> the above process is already at 7.5G and continues to grow slowly. >>>>> If I >>>>> stop the test script the memory stops growing, but does not reduce. >>>>> Restart the test script and the memory begins slowly growing again. >>>>> This >>>>> is obviously a contrived app environment. With my intended application >>>>> load it takes about a week or so for the memory to get high enough to >>>>> invoke the oom killer. Is there potentially something misconfigured >>>>> here? Thanks, Dan Ragle daniel at Biblestuph.com >>>>> >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> Gluster-users mailing list >>>>> Gluster-users at gluster.org >>>>> http://lists.gluster.org/mailman/listinfo/gluster-users >>>>> >>>> _______________________________________________ >>>> Gluster-users mailing list >>>> Gluster-users at gluster.org >>>> http://lists.gluster.org/mailman/listinfo/gluster-users >>> >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org >> http://lists.gluster.org/mailman/listinfo/gluster-users >