Carlo Rodrigues
2024-Jan-03 19:54 UTC
[Gluster-users] Files exist, but sometimes are not seen by the clients: "No such file or directory"
Hello all, We're having problems with files that suddenly stop being seen on the fuse clients. I couldn't yet find a way to reproduce this. It happens every once in a while. Sometimes you try to ls some file and it can't be found. When you run ls on the parent directory, it is shown on the output, and, after that, you can access it. I'm mentioning ls, but the problem also manifests when trying to access the file in any other way, not only when listing it with the ls command. Example: (The file names and location were edited for brevity) 1. ls file returns "No such file or directory" $ ls -l /home/RESEARCH/user.x/some_path/idtracker.log ls: cannot access /home/RESEARCH/user.x/some_path/idtracker.log: No such file or directory 2. ls parent directory. The file is shown in the results. $ ls -l /home/RESEARCH/user.x/some_path/ total 334500 -rw-r--r-- 1 user.x group_x 348521772 Jan 19 15:26 file_17.avi -rw-r--r-- 1 user.x group_x 978252 Jan 19 15:26 file_17.csv -rw-r--r-- 1 user.x group_x 1185 Jun 5 10:05 idtracker.log drwxr-xr-x 2 user.x group_x 4096 Jun 2 21:17 segm 3. try to ls file again. This time, it is found. $ ls -l /home/RESEARCH/user.x/some_path/idtracker.log -rw-r--r-- 1 user.x group_x 1185 Jun 5 10:05 /home/RESEARCH/user.x/some_path/idtracker.log The glusterfs filesystem is mounted like this storage:/tank on /nfs/tank type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072) At the moment, cluster.readdir-optimize was on, cluster.lookup-optimize was on and cluster.lookup-unhashed was off. Now I've changed those 3 options to check if the problem still appears. Current configuration: $ sudo gluster volume info tank Volume Name: tank Type: Distribute Volume ID: 2c3ca623-c1df-4b5f-b20e-5222d189f834 Status: Started Snapshot Count: 0 Number of Bricks: 24 Transport-type: tcp Bricks: Brick1: swp-gluster-05:/tank/volume1/brick Brick2: swp-gluster-06:/tank/volume1/brick Brick3: swp-gluster-07:/tank/volume1/brick Brick4: swp-gluster-05:/tank/volume2/brick Brick5: swp-gluster-06:/tank/volume2/brick Brick6: swp-gluster-07:/tank/volume2/brick Brick7: swp-gluster-05:/tank/volume3/brick Brick8: swp-gluster-06:/tank/volume3/brick Brick9: swp-gluster-07:/tank/volume3/brick Brick10: swp-gluster-05:/tank/volume4/brick Brick11: swp-gluster-06:/tank/volume4/brick Brick12: swp-gluster-07:/tank/volume4/brick Brick13: swp-gluster-01:/tank/volume1/brick Brick14: swp-gluster-02:/tank/volume1/brick Brick15: swp-gluster-03:/tank/volume1/brick Brick16: swp-gluster-04:/tank/volume1/brick Brick17: swp-gluster-01:/tank/volume2/brick Brick18: swp-gluster-02:/tank/volume2/brick Brick19: swp-gluster-03:/tank/volume2/brick Brick20: swp-gluster-04:/tank/volume2/brick Brick21: swp-gluster-01:/tank/volume3/brick Brick22: swp-gluster-02:/tank/volume3/brick Brick23: swp-gluster-03:/tank/volume3/brick Brick24: swp-gluster-04:/tank/volume3/brick Options Reconfigured: cluster.lookup-unhashed: on cluster.lookup-optimize: off performance.aggregate-size: 1MB performance.read-ahead-page-count: 16 performance.nl-cache-timeout: 600 performance.nl-cache: on network.inode-lru-limit: 200000 performance.md-cache-timeout: 600 performance.cache-invalidation: on performance.cache-samba-metadata: on features.cache-invalidation-timeout: 600 features.cache-invalidation: on performance.write-behind: off performance.cache-size: 128MB storage.fips-mode-rchecksum: on transport.address-family: inet nfs.disable: on features.inode-quota: on features.quota: on server.event-threads: 32 client.event-threads: 16 cluster.readdir-optimize: off performance.io-thread-count: 64 performance.readdir-ahead: on performance.client-io-threads: on performance.parallel-readdir: disable performance.read-ahead: on performance.stat-prefetch: on performance.open-behind: off features.quota-deem-statfs: on Has anyone gone through this before? Thanks in advance. Carlo Rodrigues -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20240103/f1a488ce/attachment.html>
Aravinda
2024-Jan-04 08:43 UTC
[Gluster-users] Files exist, but sometimes are not seen by the clients: "No such file or directory"
I see the volume tank is of type distribute only. If a brick disconnected, then the file present in that brick will not be available. Do you see any connection errors in the mount log?? --- Aravinda Kadalu Technologies ---- On Thu, 04 Jan 2024 01:24:55 +0530 Carlo Rodrigues <carlo.rodrigues at pt.clara.net> wrote --- Hello all, ? We're having problems with files that suddenly stop being seen on the fuse clients. I couldn't yet find a way to reproduce this. It happens every once in a while. ? Sometimes you try to ls some file and it can't be found. When you run ls on the parent directory, it is shown on the output, and, after that, you can access it. I'm mentioning ls, but the problem also manifests when trying to access the file in any other way, not only when listing it with the ls command. ? Example: (The file names and location were edited for brevity) ? 1. ls file returns "No such file or directory" ? $ ls -l /home/RESEARCH/user.x/some_path/idtracker.log ls: cannot access /home/RESEARCH/user.x/some_path/idtracker.log: No such file or directory ? 2. ls parent directory. The file is shown in the results. ? $ ls -l /home/RESEARCH/user.x/some_path/ total 334500 -rw-r--r-- 1 user.x group_x 348521772 Jan 19 15:26 file_17.avi -rw-r--r-- 1 user.x group_x??? 978252 Jan 19 15:26 file_17.csv -rw-r--r-- 1 user.x group_x????? 1185 Jun? 5 10:05 idtracker.log drwxr-xr-x 2 user.x group_x????? 4096 Jun? 2 21:17 segm ? 3. try to ls file again. This time, it is found. ? $ ls -l /home/RESEARCH/user.x/some_path/idtracker.log -rw-r--r-- 1 user.x group_x 1185 Jun? 5 10:05 /home/RESEARCH/user.x/some_path/idtracker.log ? ? The glusterfs filesystem is mounted like this storage:/tank on /nfs/tank type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072) ? At the moment, cluster.readdir-optimize was on, cluster.lookup-optimize was on and cluster.lookup-unhashed was off. ? Now I've changed those 3 options to check if the problem still appears. ? Current configuration: ? $ sudo gluster volume info tank ? Volume Name: tank Type: Distribute Volume ID: 2c3ca623-c1df-4b5f-b20e-5222d189f834 Status: Started Snapshot Count: 0 Number of Bricks: 24 Transport-type: tcp Bricks: Brick1: swp-gluster-05:/tank/volume1/brick Brick2: swp-gluster-06:/tank/volume1/brick Brick3: swp-gluster-07:/tank/volume1/brick Brick4: swp-gluster-05:/tank/volume2/brick Brick5: swp-gluster-06:/tank/volume2/brick Brick6: swp-gluster-07:/tank/volume2/brick Brick7: swp-gluster-05:/tank/volume3/brick Brick8: swp-gluster-06:/tank/volume3/brick Brick9: swp-gluster-07:/tank/volume3/brick Brick10: swp-gluster-05:/tank/volume4/brick Brick11: swp-gluster-06:/tank/volume4/brick Brick12: swp-gluster-07:/tank/volume4/brick Brick13: swp-gluster-01:/tank/volume1/brick Brick14: swp-gluster-02:/tank/volume1/brick Brick15: swp-gluster-03:/tank/volume1/brick Brick16: swp-gluster-04:/tank/volume1/brick Brick17: swp-gluster-01:/tank/volume2/brick Brick18: swp-gluster-02:/tank/volume2/brick Brick19: swp-gluster-03:/tank/volume2/brick Brick20: swp-gluster-04:/tank/volume2/brick Brick21: swp-gluster-01:/tank/volume3/brick Brick22: swp-gluster-02:/tank/volume3/brick Brick23: swp-gluster-03:/tank/volume3/brick Brick24: swp-gluster-04:/tank/volume3/brick Options Reconfigured: cluster.lookup-unhashed: on cluster.lookup-optimize: off performance.aggregate-size: 1MB performance.read-ahead-page-count: 16 performance.nl-cache-timeout: 600 performance.nl-cache: on network.inode-lru-limit: 200000 performance.md-cache-timeout: 600 performance.cache-invalidation: on performance.cache-samba-metadata: on features.cache-invalidation-timeout: 600 features.cache-invalidation: on performance.write-behind: off performance.cache-size: 128MB storage.fips-mode-rchecksum: on transport.address-family: inet nfs.disable: on features.inode-quota: on features.quota: on server.event-threads: 32 client.event-threads: 16 cluster.readdir-optimize: off performance.io-thread-count: 64 performance.readdir-ahead: on performance.client-io-threads: on performance.parallel-readdir: disable performance.read-ahead: on performance.stat-prefetch: on performance.open-behind: off features.quota-deem-statfs: on ? Has anyone gone through this before? Thanks in advance. ? Carlo Rodrigues ________ Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://meet.google.com/cpu-eiue-hvk Gluster-users mailing list Gluster-users at gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20240104/c42e3c47/attachment.html>