Gluster users - Sep 2019 - 'ls' some file will hang, and log will keep growing with the same error

Hi,

 ??? Under some folder, running 'ls' will hang. And the client log will 
keep growing until there is no space left or the glusterfs process is 
killed.

[2019-09-10 12:46:30.032136] I [dict.c:559:dict_get] 
(-->/usr/lib/glusterfs/6.5/xlator/protocol/client.so(+0x5c96e) 
[0x7f7c6b62a96e] 
-->/usr/lib/glusterfs/6.5/xlator/cluster/distribute.so(+0x41f01) 
[0x7f7c6b54ef01] -->/usr/lib/libglusterfs.so.0(dict_get+0x8c) 
[0x7f7c71194b2c] ) 0-dict: !this || key=trusted.glusterfs.dht.mds 
[Invalid argument]
[2019-09-10 12:46:30.033706] I [dict.c:559:dict_get] 
(-->/usr/lib/glusterfs/6.5/xlator/protocol/client.so(+0x5c96e) 
[0x7f7c6b62a96e] 
-->/usr/lib/glusterfs/6.5/xlator/cluster/distribute.so(+0x41f01) 
[0x7f7c6b54ef01] -->/usr/lib/libglusterfs.so.0(dict_get+0x8c) 
[0x7f7c71194b2c] ) 0-dict: !this || key=trusted.glusterfs.dht.mds 
[Invalid argument]
[2019-09-10 12:46:30.035249] I [dict.c:559:dict_get] 
(-->/usr/lib/glusterfs/6.5/xlator/protocol/client.so(+0x5c96e) 
[0x7f7c6b62a96e] 
-->/usr/lib/glusterfs/6.5/xlator/cluster/distribute.so(+0x41f01) 
[0x7f7c6b54ef01] -->/usr/lib/libglusterfs.so.0(dict_get+0x8c) 
[0x7f7c71194b2c] ) 0-dict: !this || key=trusted.glusterfs.dht.mds 
[Invalid argument]
[2019-09-10 12:46:30.036739] I [dict.c:559:dict_get] 
(-->/usr/lib/glusterfs/6.5/xlator/protocol/client.so(+0x5c96e) 
[0x7f7c6b62a96e] 
-->/usr/lib/glusterfs/6.5/xlator/cluster/distribute.so(+0x41f01) 
[0x7f7c6b54ef01] -->/usr/lib/libglusterfs.so.0(dict_get+0x8c) 
[0x7f7c71194b2c] ) 0-dict: !this || key=trusted.glusterfs.dht.mds 
[Invalid argument]

 ??? The gluster version is 6.5. Running on an Arch Linux, which was 
updated to the latest version yesterday.

Best,

Manhong

I made some progress with these tests.

1, a fuse mount with gluster version 5.1 doesn't have this issue. All 
files can listed with 'ls', 'ls -l', 'find . -ls ', etc.

2, The directories after running 'find . -ls' with gluster fuse client 
version 5.1 don't have this issue on gluster fuse client version 6.5 
anymore. Actually I am using this trick now to 'fix' the whole
filesystem.

3, 'ls -f', 'find . ' always work on both? version 5.1? and 6.5.
but
'ls', 'ls -l', 'find . -ls' is the trouble-maker

It seems to me that version 6.5 fuse mount has a bug.


Best,

Manhong

On 9/10/19 8:59 AM, Manhong Dai wrote:
> Hi,
>
> ??? Under some folder, running 'ls' will hang. And the client log
will
> keep growing until there is no space left or the glusterfs process is 
> killed.
>
> [2019-09-10 12:46:30.032136] I [dict.c:559:dict_get] 
> (-->/usr/lib/glusterfs/6.5/xlator/protocol/client.so(+0x5c96e) 
> [0x7f7c6b62a96e] 
> -->/usr/lib/glusterfs/6.5/xlator/cluster/distribute.so(+0x41f01) 
> [0x7f7c6b54ef01] -->/usr/lib/libglusterfs.so.0(dict_get+0x8c) 
> [0x7f7c71194b2c] ) 0-dict: !this || key=trusted.glusterfs.dht.mds 
> [Invalid argument]
> [2019-09-10 12:46:30.033706] I [dict.c:559:dict_get] 
> (-->/usr/lib/glusterfs/6.5/xlator/protocol/client.so(+0x5c96e) 
> [0x7f7c6b62a96e] 
> -->/usr/lib/glusterfs/6.5/xlator/cluster/distribute.so(+0x41f01) 
> [0x7f7c6b54ef01] -->/usr/lib/libglusterfs.so.0(dict_get+0x8c) 
> [0x7f7c71194b2c] ) 0-dict: !this || key=trusted.glusterfs.dht.mds 
> [Invalid argument]
> [2019-09-10 12:46:30.035249] I [dict.c:559:dict_get] 
> (-->/usr/lib/glusterfs/6.5/xlator/protocol/client.so(+0x5c96e) 
> [0x7f7c6b62a96e] 
> -->/usr/lib/glusterfs/6.5/xlator/cluster/distribute.so(+0x41f01) 
> [0x7f7c6b54ef01] -->/usr/lib/libglusterfs.so.0(dict_get+0x8c) 
> [0x7f7c71194b2c] ) 0-dict: !this || key=trusted.glusterfs.dht.mds 
> [Invalid argument]
> [2019-09-10 12:46:30.036739] I [dict.c:559:dict_get] 
> (-->/usr/lib/glusterfs/6.5/xlator/protocol/client.so(+0x5c96e) 
> [0x7f7c6b62a96e] 
> -->/usr/lib/glusterfs/6.5/xlator/cluster/distribute.so(+0x41f01) 
> [0x7f7c6b54ef01] -->/usr/lib/libglusterfs.so.0(dict_get+0x8c) 
> [0x7f7c71194b2c] ) 0-dict: !this || key=trusted.glusterfs.dht.mds 
> [Invalid argument]
>
> ??? The gluster version is 6.5. Running on an Arch Linux, which was 
> updated to the latest version yesterday.
>
> Best,
>
> Manhong
>