Strahil Nikolov
2022-Feb-05 09:23 UTC
[Gluster-users] GlusterFS 9.5 fuse mount excessive memory usage
Can you generate a statedump before and after the tar ?
For statedump generation , you can follow
https://github.com/gluster/glusterfs/issues/1440#issuecomment-674051243 .
Best Regards,
Strahil Nikolov
? ??????, 5 ???????? 2022 ?., 07:54:22 ???????+2, Zakhar Kirpichenko
<zakhar at gmail.com> ??????:
Hi!
I opened a Github issue https://github.com/gluster/glusterfs/issues/3206 but not
sure how much attention they get there, so re-posting here just in case someone
has any ideas.
Description of problem:
GlusterFS 9.5, 3-node cluster (2 bricks + arbiter), an attempt to tar the whole
filesystem (35-40 GB, 1.6 million files) on a client succeeds but causes the
glusterfs fuse mount process to consume 0.5+ GB of RAM. The usage never goes
down after tar exits.
The exact command to reproduce the issue:
/usr/bin/tar --use-compress-program="/bin/pigz" -cf
/path/to/archive.tar.gz --warning=no-file-changed /glusterfsmount
The output of the gluster volume info command:
Volume Name: gvol1
Type: Replicate
Volume ID: 0292ac43-89bd-45a4-b91d-799b49613e60
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x (2 + 1) = 3
Transport-type: tcp
Bricks:
Brick1: 192.168.0.31:/gluster/brick1/gvol1
Brick2: 192.168.0.32:/gluster/brick1/gvol1
Brick3: 192.168.0.5:/gluster/brick1/gvol1 (arbiter)
Options Reconfigured:
performance.open-behind: off
cluster.readdir-optimize: off
cluster.consistent-metadata: on
features.cache-invalidation: on
diagnostics.count-fop-hits: on
diagnostics.latency-measurement: on
storage.fips-mode-rchecksum: on
performance.cache-size: 256MB
client.event-threads: 8
server.event-threads: 4
storage.reserve: 1
performance.cache-invalidation: on
cluster.lookup-optimize: on
transport.address-family: inet
nfs.disable: on
performance.client-io-threads: on
features.cache-invalidation-timeout: 600
performance.md-cache-timeout: 600
network.inode-lru-limit: 50000
cluster.shd-max-threads: 4
cluster.self-heal-window-size: 8
performance.enable-least-priority: off
performance.cache-max-file-size: 2MB
The output of the gluster volume status command:
Status of volume: gvol1
Gluster process ? ? ? ? ? ? ? ? ? ? ? ? ? ? TCP Port ?RDMA Port ?Online ?Pid
------------------------------------------------------------------------------
Brick 192.168.0.31:/gluster/brick1/gvol1 ? ?49152 ? ? 0 ? ? ? ? ?Y ? ? ? 1767
Brick 192.168.0.32:/gluster/brick1/gvol1 ? ?49152 ? ? 0 ? ? ? ? ?Y ? ? ? 1696
Brick 192.168.0.5:/gluster/brick1/gvol1 ? ? 49152 ? ? 0 ? ? ? ? ?Y ? ? ? 1318
Self-heal Daemon on localhost ? ? ? ? ? ? ? N/A ? ? ? N/A ? ? ? ?Y ? ? ? 1329
Self-heal Daemon on 192.168.0.31 ? ? ? ? ? ?N/A ? ? ? N/A ? ? ? ?Y ? ? ? 1778
Self-heal Daemon on 192.168.0.32 ? ? ? ? ? ?N/A ? ? ? N/A ? ? ? ?Y ? ? ? 1707
Task Status of Volume gvol1
------------------------------------------------------------------------------
There are no active volume tasks
The output of the gluster volume heal command:
Brick 192.168.0.31:/gluster/brick1/gvol1
Status: Connected
Number of entries: 0
Brick 192.168.0.32:/gluster/brick1/gvol1
Status: Connected
Number of entries: 0
Brick 192.168.0.5:/gluster/brick1/gvol1
Status: Connected
Number of entries: 0
The operating system / glusterfs version:
CentOS Linux release 7.9.2009 (Core), fully up to date
glusterfs 9.5
kernel 3.10.0-1160.53.1.el7.x86_64
The logs are basically empty since the last mount except for the mount-related
messages.
Additional info: a statedump from the client is attached to the Github issue,
https://github.com/gluster/glusterfs/files/8004792/glusterdump.18906.dump.1643991007.gz,
in case someone wants to have a look.
There was also an issue with other clients, running PHP applications with lots
of small files, where glusterfs fuse mount process would very quickly balloon to
~2 GB over the course of 24 hours and its performance would slow to a crawl.
This happened very consistently with glusterfs 8.x and 9.5, I managed to resolve
it at least partially with disabling performance.open-behind: the memory usage
either remains consistent or increases at a much slower rate, which is
acceptable for this use case.
Now the issue remains on this single client, which doesn't do much other
than reading and archiving all files from the gluster volume once per day. The
glusterfs fuse mount process balloons to 0.5+ GB during the first tar run and
remains more or less consistent afterwards, including subsequent tar runs.
I would very much appreciate any advice or suggestions.
Best regards,
Zakhar
________
Community Meeting Calendar:
Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://meet.google.com/cpu-eiue-hvk
Gluster-users mailing list
Gluster-users at gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20220205/aa73acb1/attachment.html>
Zakhar Kirpichenko
2022-Feb-05 18:45 UTC
[Gluster-users] GlusterFS 9.5 fuse mount excessive memory usage
Hi Strahil, Many thanks for your reply! I've updated the Github issue with statedump files taken before and after the tar operation: https://github.com/gluster/glusterfs/files/8008635/glusterdump.19102.dump.zip Please disregard that path= entries are empty, in the original dumps there are real paths but I deleted them as they might contain sensitive information. The odd thing is that the dump file is full of: 1) xlator.performance.write-behind.wb_inode entries, but the tar operation does not write to these files. The whole backup process is read-only. 2) xlator.performance.quick-read.inodectx entries, which never go away. None of this happens on other clients, which read and write from/to the same volume in a much more intense manner. Best regards, Z On Sat, Feb 5, 2022 at 11:23 AM Strahil Nikolov <hunter86_bg at yahoo.com> wrote:> Can you generate a statedump before and after the tar ? > For statedump generation , you can follow > https://github.com/gluster/glusterfs/issues/1440#issuecomment-674051243 . > > Best Regards, > Strahil Nikolov > > > ? ??????, 5 ???????? 2022 ?., 07:54:22 ???????+2, Zakhar Kirpichenko < > zakhar at gmail.com> ??????: > > > Hi! > > I opened a Github issue https://github.com/gluster/glusterfs/issues/3206 > but not sure how much attention they get there, so re-posting here just in > case someone has any ideas. > > Description of problem: > > GlusterFS 9.5, 3-node cluster (2 bricks + arbiter), an attempt to tar the > whole filesystem (35-40 GB, 1.6 million files) on a client succeeds but > causes the glusterfs fuse mount process to consume 0.5+ GB of RAM. The > usage never goes down after tar exits. > > The exact command to reproduce the issue: > > /usr/bin/tar --use-compress-program="/bin/pigz" -cf > /path/to/archive.tar.gz --warning=no-file-changed /glusterfsmount > > The output of the gluster volume info command: > > Volume Name: gvol1 > Type: Replicate > Volume ID: 0292ac43-89bd-45a4-b91d-799b49613e60 > Status: Started > Snapshot Count: 0 > Number of Bricks: 1 x (2 + 1) = 3 > Transport-type: tcp > Bricks: > Brick1: 192.168.0.31:/gluster/brick1/gvol1 > Brick2: 192.168.0.32:/gluster/brick1/gvol1 > Brick3: 192.168.0.5:/gluster/brick1/gvol1 (arbiter) > Options Reconfigured: > performance.open-behind: off > cluster.readdir-optimize: off > cluster.consistent-metadata: on > features.cache-invalidation: on > diagnostics.count-fop-hits: on > diagnostics.latency-measurement: on > storage.fips-mode-rchecksum: on > performance.cache-size: 256MB > client.event-threads: 8 > server.event-threads: 4 > storage.reserve: 1 > performance.cache-invalidation: on > cluster.lookup-optimize: on > transport.address-family: inet > nfs.disable: on > performance.client-io-threads: on > features.cache-invalidation-timeout: 600 > performance.md-cache-timeout: 600 > network.inode-lru-limit: 50000 > cluster.shd-max-threads: 4 > cluster.self-heal-window-size: 8 > performance.enable-least-priority: off > performance.cache-max-file-size: 2MB > > The output of the gluster volume status command: > > Status of volume: gvol1 > Gluster process TCP Port RDMA Port Online > Pid > > ------------------------------------------------------------------------------ > Brick 192.168.0.31:/gluster/brick1/gvol1 49152 0 Y > 1767 > Brick 192.168.0.32:/gluster/brick1/gvol1 49152 0 Y > 1696 > Brick 192.168.0.5:/gluster/brick1/gvol1 49152 0 Y > 1318 > Self-heal Daemon on localhost N/A N/A Y > 1329 > Self-heal Daemon on 192.168.0.31 N/A N/A Y > 1778 > Self-heal Daemon on 192.168.0.32 N/A N/A Y > 1707 > > Task Status of Volume gvol1 > > ------------------------------------------------------------------------------ > There are no active volume tasks > > The output of the gluster volume heal command: > > Brick 192.168.0.31:/gluster/brick1/gvol1 > Status: Connected > Number of entries: 0 > > Brick 192.168.0.32:/gluster/brick1/gvol1 > Status: Connected > Number of entries: 0 > > Brick 192.168.0.5:/gluster/brick1/gvol1 > Status: Connected > Number of entries: 0 > > The operating system / glusterfs version: > > CentOS Linux release 7.9.2009 (Core), fully up to date > glusterfs 9.5 > kernel 3.10.0-1160.53.1.el7.x86_64 > > The logs are basically empty since the last mount except for the > mount-related messages. > > Additional info: a statedump from the client is attached to the Github > issue, > https://github.com/gluster/glusterfs/files/8004792/glusterdump.18906.dump.1643991007.gz, > in case someone wants to have a look. > > There was also an issue with other clients, running PHP applications with > lots of small files, where glusterfs fuse mount process would very quickly > balloon to ~2 GB over the course of 24 hours and its performance would slow > to a crawl. This happened very consistently with glusterfs 8.x and 9.5, I > managed to resolve it at least partially with disabling > performance.open-behind: the memory usage either remains consistent or > increases at a much slower rate, which is acceptable for this use case. > > Now the issue remains on this single client, which doesn't do much other > than reading and archiving all files from the gluster volume once per day. > The glusterfs fuse mount process balloons to 0.5+ GB during the first tar > run and remains more or less consistent afterwards, including subsequent > tar runs. > > I would very much appreciate any advice or suggestions. > > Best regards, > Zakhar > ________ > > > > Community Meeting Calendar: > > Schedule - > Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC > Bridge: https://meet.google.com/cpu-eiue-hvk > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20220205/d118f4fe/attachment.html>