Gluster users - Jun 2021 - File not found errors

Hi,

I've got an interesting issue with files not being found when accessed
directly.

When accessing a file like so:

  #ls /mnt/path/to/some/file/file.json
  ls: cannot access '/mnt/path/to/some/file/file.json': No such file or
directory

I can wait, try again, and same result. I could put this in a loop with a delay
between read attempts, same result.

However, if I do an "ls" of the directory first then everything works.
That is:

  # ls /mnt/path/to/some/file/file.json
  ls: cannot access '/mnt/path/to/some/file/file.json': No such file or
directory

  # ls /mnt/path/to/some/file/
  dir1  dir2  file1.txt  file2.txt  file3.txt  file.json

At this point I can successfully "ls" the file:

  # ls /mnt/path/to/some/file/file.json
  /mnt/path/to/some/file/file.json

It is curious, but seems to be isolated to certain directories/mount points.

Looking at the mount logs there is absolutely nothing printed to the mnt log on
the client with regards to the failure or the eventual success.

The configuration:

  Distributed-Replicate
  4 x 2 = 8
  Gluster version 9.2

Clients are using native FUSE mounts (also version 9.2)

I did recently add the 4th brick, and the system is currently undergoing a
rebalance (the fix-layout rebalance already completed). I am throttling the full
rebalance as 'lazy'.

I looked at the bricks on the server and the files still exists on the older
bricks, so I don't think the files in question got rebalanced (yet).

The system load isn't too high, and this doesn't seem to be a consistent
error as I don't have this issue in other directories, but I consistently
have the issue on this directory as well as a few others.

Adding some additional background/history:

  Cluster was setup a couple of years ago on gluster 6.x. 
  Did an expansion (two replica pairs to three replica pairs) and rebalance
while on 6.x, no issues.
  Upgraded to gluster 7.x -> 8.x -> 9.x
  Ops version has been upgraded to 90000
  Doing another expansion (three replica pairs to four replica pairs) and
rebalance now, and seeing some interesting issues, including this one.

There is 0 entries under "heal info" or "info split-brain".

I'm just wondering if anyone on this list has seen anything similar, or has
any suggestions.

Il 24/06/21 03:45, cfeller at rocketmail.com ha scritto:

Probably related to the problem I reported some time ago (apr, 22nd), 
thread "quantum directories?". Got no answer :(
> Hi,
> 
> I've got an interesting issue with files not being found when accessed
directly.
> 
> When accessing a file like so:
> 
>    #ls /mnt/path/to/some/file/file.json
>    ls: cannot access '/mnt/path/to/some/file/file.json': No such
file or directory
> 
> I can wait, try again, and same result. I could put this in a loop with a
delay between read attempts, same result.
> 
> However, if I do an "ls" of the directory first then everything
works. That is:
> 
>    # ls /mnt/path/to/some/file/file.json
>    ls: cannot access '/mnt/path/to/some/file/file.json': No such
file or directory
> 
>    # ls /mnt/path/to/some/file/
>    dir1  dir2  file1.txt  file2.txt  file3.txt  file.json
> 
> At this point I can successfully "ls" the file:
> 
>    # ls /mnt/path/to/some/file/file.json
>    /mnt/path/to/some/file/file.json
> 
> It is curious, but seems to be isolated to certain directories/mount
points.
> 
> Looking at the mount logs there is absolutely nothing printed to the mnt
log on the client with regards to the failure or the eventual success.
> 
> The configuration:
> 
>    Distributed-Replicate
>    4 x 2 = 8
>    Gluster version 9.2
> 
> Clients are using native FUSE mounts (also version 9.2)
> 
> I did recently add the 4th brick, and the system is currently undergoing a
rebalance (the fix-layout rebalance already completed). I am throttling the full
rebalance as 'lazy'.
> 
> I looked at the bricks on the server and the files still exists on the
older bricks, so I don't think the files in question got rebalanced (yet).
> 
> The system load isn't too high, and this doesn't seem to be a
consistent error as I don't have this issue in other directories, but I
consistently have the issue on this directory as well as a few others.
> 
> Adding some additional background/history:
> 
>    Cluster was setup a couple of years ago on gluster 6.x.
>    Did an expansion (two replica pairs to three replica pairs) and
rebalance while on 6.x, no issues.
>    Upgraded to gluster 7.x -> 8.x -> 9.x
>    Ops version has been upgraded to 90000
>    Doing another expansion (three replica pairs to four replica pairs) and
rebalance now, and seeing some interesting issues, including this one.
> 
> There is 0 entries under "heal info" or "info
split-brain".
> 
> I'm just wondering if anyone on this list has seen anything similar, or
has any suggestions.
> ________
> 
> 
> 
> Community Meeting Calendar:
> 
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://meet.google.com/cpu-eiue-hvk
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
> 

-- 
Diego Zuccato
DIFA - Dip. di Fisica e Astronomia
Servizi Informatici
Alma Mater Studiorum - Universit? di Bologna
V.le Berti-Pichat 6/2 - 40127 Bologna - Italy
tel.: +39 051 20 95786