Poornima Gurusiddaiah
2019-Mar-02 04:07 UTC
[Gluster-users] Gluster eating up a lot of ram
This high memory consumption is not normal. Looks like it's a memory leak. Is it possible to try it on test setup with gluster-6rc? What is the kind of workload that goes into fuse mount? Large files or small files? We need the following information to debug further: - Gluster volume info output - Statedump of the Gluster fuse mount process consuming 44G ram. Regards, Poornima On Sat, Mar 2, 2019, 3:40 AM Diego Remolina <dijuremo at gmail.com> wrote:> I am using glusterfs with two servers as a file server sharing files via > samba and ctdb. I cannot use samba vfs gluster plugin, due to bug in > current Centos version of samba. So I am mounting via fuse and exporting > the volume to samba from the mount point. > > Upon initial boot, the server where samba is exporting files climbs up to > ~10GB RAM within a couple hours of use. From then on, it is a constant slow > memory increase. In the past with gluster 3.8.x we had to reboot the > servers at around 30 days . With gluster 4.1.6 we are getting up to 48 > days, but RAM use is at 48GB out of 64GB. Is this normal? > > The particular versions are below, > > [root at ysmha01 home]# uptime > 16:59:39 up 48 days, 9:56, 1 user, load average: 3.75, 3.17, 3.00 > [root at ysmha01 home]# rpm -qa | grep gluster > centos-release-gluster41-1.0-3.el7.centos.noarch > glusterfs-server-4.1.6-1.el7.x86_64 > glusterfs-api-4.1.6-1.el7.x86_64 > centos-release-gluster-legacy-4.0-2.el7.centos.noarch > glusterfs-4.1.6-1.el7.x86_64 > glusterfs-client-xlators-4.1.6-1.el7.x86_64 > libvirt-daemon-driver-storage-gluster-3.9.0-14.el7_5.8.x86_64 > glusterfs-fuse-4.1.6-1.el7.x86_64 > glusterfs-libs-4.1.6-1.el7.x86_64 > glusterfs-rdma-4.1.6-1.el7.x86_64 > glusterfs-cli-4.1.6-1.el7.x86_64 > samba-vfs-glusterfs-4.8.3-4.el7.x86_64 > [root at ysmha01 home]# rpm -qa | grep samba > samba-common-tools-4.8.3-4.el7.x86_64 > samba-client-libs-4.8.3-4.el7.x86_64 > samba-libs-4.8.3-4.el7.x86_64 > samba-4.8.3-4.el7.x86_64 > samba-common-libs-4.8.3-4.el7.x86_64 > samba-common-4.8.3-4.el7.noarch > samba-vfs-glusterfs-4.8.3-4.el7.x86_64 > [root at ysmha01 home]# cat /etc/redhat-release > CentOS Linux release 7.6.1810 (Core) > > RAM view using top > Tasks: 398 total, 1 running, 397 sleeping, 0 stopped, 0 zombie > %Cpu(s): 7.0 us, 9.3 sy, 1.7 ni, 71.6 id, 9.7 wa, 0.0 hi, 0.8 si, > 0.0 st > KiB Mem : 65772000 total, 1851344 free, 60487404 used, 3433252 buff/cache > KiB Swap: 0 total, 0 free, 0 used. 3134316 avail Mem > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 9953 root 20 0 3727912 946496 3196 S 150.2 1.4 38626:27 > glusterfsd > 9634 root 20 0 48.1g 47.2g 3184 S 96.3 75.3 29513:55 > glusterfs > 14485 root 20 0 3404140 63780 2052 S 80.7 0.1 1590:13 > glusterfs > > [root at ysmha01 ~]# gluster v status export > Status of volume: export > Gluster process TCP Port RDMA Port Online > Pid > > ------------------------------------------------------------------------------ > Brick 10.0.1.7:/bricks/hdds/brick 49157 0 Y > 13986 > Brick 10.0.1.6:/bricks/hdds/brick 49153 0 Y > 9953 > Self-heal Daemon on localhost N/A N/A Y > 14485 > Self-heal Daemon on 10.0.1.7 N/A N/A Y > 21934 > Self-heal Daemon on 10.0.1.5 N/A N/A Y > 4598 > > Task Status of Volume export > > ------------------------------------------------------------------------------ > There are no active volume tasks > > > > > > <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=icon> Virus-free. > www.avast.com > <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=link> > <#m_5816452762692804512_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20190302/f3f42356/attachment.html>
Hi, I will not be able to test gluster-6rc because this is a production environment and it takes several days for memory to grow a lot. The Samba server is hosting all types of files, small and large from small roaming profile type files to bigger files like adobe suite, autodesk Revit (file sizes in the hundreds of megabytes). As I stated before, this same issue was present back with 3.8.x which I was running before. The information you requested: [root at ysmha02 ~]# gluster v info export Volume Name: export Type: Replicate Volume ID: b4353b3f-6ef6-4813-819a-8e85e5a95cff Status: Started Snapshot Count: 0 Number of Bricks: 1 x 2 = 2 Transport-type: tcp Bricks: Brick1: 10.0.1.7:/bricks/hdds/brick Brick2: 10.0.1.6:/bricks/hdds/brick Options Reconfigured: performance.stat-prefetch: on performance.cache-min-file-size: 0 network.inode-lru-limit: 65536 performance.cache-invalidation: on features.cache-invalidation: on performance.md-cache-timeout: 600 features.cache-invalidation-timeout: 600 performance.cache-samba-metadata: on transport.address-family: inet server.allow-insecure: on performance.cache-size: 10GB cluster.server-quorum-type: server nfs.disable: on performance.io-thread-count: 64 performance.io-cache: on cluster.lookup-optimize: on cluster.readdir-optimize: on server.event-threads: 5 client.event-threads: 5 performance.cache-max-file-size: 256MB diagnostics.client-log-level: INFO diagnostics.brick-log-level: INFO cluster.server-quorum-ratio: 51% <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=icon> Virus-free. www.avast.com <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=link> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> On Fri, Mar 1, 2019 at 11:07 PM Poornima Gurusiddaiah <pgurusid at redhat.com> wrote:> This high memory consumption is not normal. Looks like it's a memory leak. > Is it possible to try it on test setup with gluster-6rc? What is the kind > of workload that goes into fuse mount? Large files or small files? We need > the following information to debug further: > - Gluster volume info output > - Statedump of the Gluster fuse mount process consuming 44G ram. > > Regards, > Poornima > > > On Sat, Mar 2, 2019, 3:40 AM Diego Remolina <dijuremo at gmail.com> wrote: > >> I am using glusterfs with two servers as a file server sharing files via >> samba and ctdb. I cannot use samba vfs gluster plugin, due to bug in >> current Centos version of samba. So I am mounting via fuse and exporting >> the volume to samba from the mount point. >> >> Upon initial boot, the server where samba is exporting files climbs up to >> ~10GB RAM within a couple hours of use. From then on, it is a constant slow >> memory increase. In the past with gluster 3.8.x we had to reboot the >> servers at around 30 days . With gluster 4.1.6 we are getting up to 48 >> days, but RAM use is at 48GB out of 64GB. Is this normal? >> >> The particular versions are below, >> >> [root at ysmha01 home]# uptime >> 16:59:39 up 48 days, 9:56, 1 user, load average: 3.75, 3.17, 3.00 >> [root at ysmha01 home]# rpm -qa | grep gluster >> centos-release-gluster41-1.0-3.el7.centos.noarch >> glusterfs-server-4.1.6-1.el7.x86_64 >> glusterfs-api-4.1.6-1.el7.x86_64 >> centos-release-gluster-legacy-4.0-2.el7.centos.noarch >> glusterfs-4.1.6-1.el7.x86_64 >> glusterfs-client-xlators-4.1.6-1.el7.x86_64 >> libvirt-daemon-driver-storage-gluster-3.9.0-14.el7_5.8.x86_64 >> glusterfs-fuse-4.1.6-1.el7.x86_64 >> glusterfs-libs-4.1.6-1.el7.x86_64 >> glusterfs-rdma-4.1.6-1.el7.x86_64 >> glusterfs-cli-4.1.6-1.el7.x86_64 >> samba-vfs-glusterfs-4.8.3-4.el7.x86_64 >> [root at ysmha01 home]# rpm -qa | grep samba >> samba-common-tools-4.8.3-4.el7.x86_64 >> samba-client-libs-4.8.3-4.el7.x86_64 >> samba-libs-4.8.3-4.el7.x86_64 >> samba-4.8.3-4.el7.x86_64 >> samba-common-libs-4.8.3-4.el7.x86_64 >> samba-common-4.8.3-4.el7.noarch >> samba-vfs-glusterfs-4.8.3-4.el7.x86_64 >> [root at ysmha01 home]# cat /etc/redhat-release >> CentOS Linux release 7.6.1810 (Core) >> >> RAM view using top >> Tasks: 398 total, 1 running, 397 sleeping, 0 stopped, 0 zombie >> %Cpu(s): 7.0 us, 9.3 sy, 1.7 ni, 71.6 id, 9.7 wa, 0.0 hi, 0.8 si, >> 0.0 st >> KiB Mem : 65772000 total, 1851344 free, 60487404 used, 3433252 >> buff/cache >> KiB Swap: 0 total, 0 free, 0 used. 3134316 avail Mem >> >> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ >> COMMAND >> 9953 root 20 0 3727912 946496 3196 S 150.2 1.4 38626:27 >> glusterfsd >> 9634 root 20 0 48.1g 47.2g 3184 S 96.3 75.3 29513:55 >> glusterfs >> 14485 root 20 0 3404140 63780 2052 S 80.7 0.1 1590:13 >> glusterfs >> >> [root at ysmha01 ~]# gluster v status export >> Status of volume: export >> Gluster process TCP Port RDMA Port Online >> Pid >> >> ------------------------------------------------------------------------------ >> Brick 10.0.1.7:/bricks/hdds/brick 49157 0 Y >> 13986 >> Brick 10.0.1.6:/bricks/hdds/brick 49153 0 Y >> 9953 >> Self-heal Daemon on localhost N/A N/A Y >> 14485 >> Self-heal Daemon on 10.0.1.7 N/A N/A Y >> 21934 >> Self-heal Daemon on 10.0.1.5 N/A N/A Y >> 4598 >> >> Task Status of Volume export >> >> ------------------------------------------------------------------------------ >> There are no active volume tasks >> >> >> >> >> >> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=icon> Virus-free. >> www.avast.com >> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=link> >> <#m_1092070095161815064_m_5816452762692804512_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org >> https://lists.gluster.org/mailman/listinfo/gluster-users > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20190303/ff20c03e/attachment.html>