thr3ads.net - Gluster users - [Gluster-users] Gluster clients can't see directories that exist or are created within a mounted volume, but can enter them. [Nov 2017]

If this information is useful, please help other people find it:
Share via:

Nithya Balachandran

2017-Nov-08 10:03 UTC

[Gluster-users] Gluster clients can't see directories that exist or are created within a mounted volume, but can enter them.

On 8 November 2017 at 02:47, Sam McLeod <mailinglists at smcleod.net>
wrote:
>
> On 6 Nov 2017, at 3:32 pm, Laura Bailey <lbailey at redhat.com>
wrote:
>
> Do the users have permission to see/interact with the directories, in
> addition to the files?
>
>
> Yes, full access to directories and files.
> Also testing using the root user.
>
>
> On Mon, Nov 6, 2017 at 1:55 PM, Nithya Balachandran <nbalacha at
redhat.com>
> wrote:
>
>> Hi,
>>
>> Please provide the gluster volume info. Do you see any errors in the
>> client mount log file (/var/log/glusterfs/var-lib-mountedgluster.log)?
>>
>
>
> root at int-gluster-01:/var/log/glusterfs  # grep 'dev_static'
*.log|grep -v
> cmd_history
>
> glusterd.log:[2017-11-05 22:37:06.934787] W
[glusterd-locks.c:675:glusterd_mgmt_v3_unlock]
> (-->/usr/lib64/glusterfs/3.12.2/xlator/mgmt/glusterd.so(+0x22e5a)
> [0x7f5047169e5a]
-->/usr/lib64/glusterfs/3.12.2/xlator/mgmt/glusterd.so(+0x2cdc8)
> [0x7f5047173dc8]
-->/usr/lib64/glusterfs/3.12.2/xlator/mgmt/glusterd.so(+0xe372a)
> [0x7f504722a72a] ) 0-management: Lock for vol dev_static not held
> glusterd.log:[2017-11-05 22:37:06.934806] W [MSGID: 106118]
> [glusterd-handler.c:6309:__glusterd_peer_rpc_notify] 0-management: Lock
> not released for dev_static
> glusterd.log:[2017-11-05 22:39:49.924472] W
[glusterd-locks.c:675:glusterd_mgmt_v3_unlock]
> (-->/usr/lib64/glusterfs/3.12.2/xlator/mgmt/glusterd.so(+0x22e5a)
> [0x7fde97921e5a]
-->/usr/lib64/glusterfs/3.12.2/xlator/mgmt/glusterd.so(+0x2cdc8)
> [0x7fde9792bdc8]
-->/usr/lib64/glusterfs/3.12.2/xlator/mgmt/glusterd.so(+0xe372a)
> [0x7fde979e272a] ) 0-management: Lock for vol dev_static not held
> glusterd.log:[2017-11-05 22:39:49.924494] W [MSGID: 106118]
> [glusterd-handler.c:6309:__glusterd_peer_rpc_notify] 0-management: Lock
> not released for dev_static
> glusterd.log:[2017-11-05 22:41:42.565123] W
[glusterd-locks.c:675:glusterd_mgmt_v3_unlock]
> (-->/usr/lib64/glusterfs/3.12.2/xlator/mgmt/glusterd.so(+0x22e5a)
> [0x7fde97921e5a]
-->/usr/lib64/glusterfs/3.12.2/xlator/mgmt/glusterd.so(+0x2cdc8)
> [0x7fde9792bdc8]
-->/usr/lib64/glusterfs/3.12.2/xlator/mgmt/glusterd.so(+0xe372a)
> [0x7fde979e272a] ) 0-management: Lock for vol dev_static not held
> glusterd.log:[2017-11-05 22:41:42.565227] W [MSGID: 106118]
> [glusterd-handler.c:6309:__glusterd_peer_rpc_notify] 0-management: Lock
> not released for dev_static
> glusterd.log:[2017-11-05 22:42:06.931060] W
[glusterd-locks.c:675:glusterd_mgmt_v3_unlock]
> (-->/usr/lib64/glusterfs/3.12.2/xlator/mgmt/glusterd.so(+0x22e5a)
> [0x7fde97921e5a]
-->/usr/lib64/glusterfs/3.12.2/xlator/mgmt/glusterd.so(+0x2cdc8)
> [0x7fde9792bdc8]
-->/usr/lib64/glusterfs/3.12.2/xlator/mgmt/glusterd.so(+0xe372a)
> [0x7fde979e272a] ) 0-management: Lock for vol dev_static not held
> glusterd.log:[2017-11-05 22:42:06.931090] W [MSGID: 106118]
> [glusterd-handler.c:6309:__glusterd_peer_rpc_notify] 0-management: Lock
> not released for dev_static
>
>
>>That is not the log for the mount. Please check
 /var/log/glusterfs/var-lib-mountedgluster.log on the system on which you
are running the mount process.

Please provide the volume config details as well (gluster volume info) from
one of the server nodes.


>> Thanks,
>> Nithya
>>
>> On 6 November 2017 at 05:13, Sam McLeod <mailinglists at
smcleod.net> wrote:
>>
>>> We've got an issue with Gluster (3.12.x) where clients
can't see
>>> directories that exist or are created within a mounted volume.
>>>
>>>
>>> We can see files, but not directories (with ls, find etc...)
>>> We can enter (cd) into directories, even if we can't see them.
>>>
>>> - Host typology is: 2 replica, 1 arbiter.
>>> - Volumes are: replicated and running on XFS on the hosts.
>>> - Clients are: GlusterFS native fuse client (mount.glusterfs), the
same
>>> version and op-version as the hosts.
>>> - Gluster server and client version: 3.12.2 (also found on 3.12.1,
>>> unsure about previous versions) running on CentOS 7.
>>>
>>>
>>> Examples:
>>>
>>>
>>> mount:
>>> 192.168.0.151:/gluster_vol on /var/lib/mountedgluster type
>>> fuse.glusterfs (rw,relatime,user_id=0,group_i
>>> d=0,default_permissions,allow_other,max_read=131072)
>>>
>>> root at gluster-client:/var/lib/mountedgluster  # ls -la
>>> total 0
>>>
>>> (note no . or .. directories)
>>>
>>> root at gluster-client:/var/lib/mountedgluster  # touch test
>>> root at gluster-client:/var/lib/mountedgluster  # ls -la
>>> total 0
>>> -rw-r--r--. 1 root root 0 Nov  6 10:10 test
>>>
>>> ("test" file shows up. Still no . or .. directories.)
>>>
>>> root at gluster-client:/var/lib/mountedgluster  # mkdir testdir
>>> root at gluster-client:/var/lib/mountedgluster  # ls -la
>>> total 0
>>> -rw-r--r--. 1 root root 0 Nov  6 10:10 test
>>>
>>> (directory was made, but doesn't show in ls)
>>>
>>> root at gluster-client:/var/lib/mountedgluster  # cd testdir
>>> root at gluster-client:/var/lib/mountedgluster/testdir  # ls -la
>>> total 0
>>>
>>> (cd works, no . or .. shown in ls though)
>>>
>>> root at gluster-client:/var/lib/mountedgluster/testdir  # touch
test
>>> root at gluster-client:/var/lib/mountedgluster/testdir  # ls -la
>>> total 0
>>> -rw-r--r--. 1 root root 0 Nov  6 10:10 test
>>>
>>> (can create test file in testdir)
>>>
>>>
>>> root at gluster-client:/var/lib/mountedgluster/testdir  # cd ..
>>> root at gluster-client:/var/lib/mountedgluster  # ls -ld testdir
>>> drwxr-xr-x. 2 root root 4096 Nov  6 10:10 testdir
>>>
>>> (going back to parent directory, although ls -la doesn't show
the
>>> directory, if I specify the name to ls -ld it does show)
>>>
>>>
>>>
>>> --
>>> Sam McLeod
>>> https://twitter.com/s_mcleod
>>> https://smcleod.net
>>>
>>>
>>> _______________________________________________
>>> Gluster-users mailing list
>>> Gluster-users at gluster.org
>>> http://lists.gluster.org/mailman/listinfo/gluster-users
>>>
>>
>>
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://lists.gluster.org/mailman/listinfo/gluster-users
>>
>
>
>
> --
> Laura Bailey
> Senior Technical Writer
> Customer Content Services BNE
>
>
>
>-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20171108/c7b856d2/attachment.html>

Sam McLeod

2017-Nov-08 21:57 UTC

head link

[Gluster-users] Gluster clients can't see directories that exist or are created within a mounted volume, but can enter them.

> On 8 Nov 2017, at 9:03 pm, Nithya Balachandran <nbalacha at
redhat.com> wrote:
> 
> 
> That is not the log for the mount. Please check 
/var/log/glusterfs/var-lib-mountedgluster.log on the system on which you are
running the mount process.
> 
> Please provide the volume config details as well (gluster volume info) from
one of the server nodes.
> 
Oh I'm sorry, I totally misread that - didn't realise it was on the
client.

Clarification for below logs:

- 'dev_static' is the gluster volume.
- 'int-kube-01' is the gluster client.
- '10.51.70.151' is the first node in a three node (2 replica, 1
arbiter) gluster cluster.
- '/var/lib/kubelet/...../iss3dev-static' is a directory on the client
that should be mounting '10.51.70.151:/dev_static/iss3dev-static', where
'iss3dev-static' is a directory inside the gluster mount
'dev_static'.
- FYI only, we are essentially mounting directories inside gluster volumes as
per Kubernetes example:
https://github.com/kubernetes/examples/blob/master/staging/volumes/glusterfs/README.md
<https://github.com/kubernetes/examples/blob/master/staging/volumes/glusterfs/README.md>

root at int-kube-01:/var/log/glusterfs  # tail -20 mnt.log

[2017-11-07 21:15:03.561470] I [MSGID: 100030] [glusterfsd.c:2524:main]
0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.12.2 (args:
/usr/sbin/glusterfs --volfile-server=10.51.70.151 --volfile-id=/dev_static /mnt)
[2017-11-07 21:15:03.571205] W [MSGID: 101002] [options.c:995:xl_opt_validate]
0-glusterfs: option 'address-family' is deprecated, preferred is
'transport.address-family', continuing with correction
[2017-11-07 21:15:03.584098] I [MSGID: 101190]
[event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started thread with
index 1


No new log entries appear after executing the following to demonstrate the
problem:


root at
int-kube-01:/var/lib/kubelet/pods/434cba8e-bf87-11e7-8389-1aa903709357/volumes/kubernetes.io~glusterfs/iss3dev-static
# touch test
root at
int-kube-01:/var/lib/kubelet/pods/434cba8e-bf87-11e7-8389-1aa903709357/volumes/kubernetes.io~glusterfs/iss3dev-static
# mkdir testdir2

root at
int-kube-01:/var/lib/kubelet/pods/434cba8e-bf87-11e7-8389-1aa903709357/volumes/kubernetes.io~glusterfs/iss3dev-static
# ls -ltar
total 0
-rw-r--r--. 1 root root 0 Nov  6 10:10 test

root at
int-kube-01:/var/lib/kubelet/pods/434cba8e-bf87-11e7-8389-1aa903709357/volumes/kubernetes.io~glusterfs/iss3dev-static
# cd testdir2
root at
int-kube-01:/var/lib/kubelet/pods/434cba8e-bf87-11e7-8389-1aa903709357/volumes/kubernetes.io~glusterfs/iss3dev-static/testdir2
# ls -la
total 0

root at
int-kube-01:/var/lib/kubelet/pods/434cba8e-bf87-11e7-8389-1aa903709357/volumes/kubernetes.io~glusterfs/iss3dev-static/testdir2
# cd ..
root at
int-kube-01:/var/lib/kubelet/pods/434cba8e-bf87-11e7-8389-1aa903709357/volumes/kubernetes.io~glusterfs/iss3dev-static#
ls -ltar
total 0
-rw-r--r--. 1 root root 0 Nov  6 10:10 test


The mount entry looks like:

root at int-kube-01:~ # mount | grep iss3dev-static
10.51.70.151:/dev_static on
/var/lib/kubelet/pods/434cba8e-bf87-11e7-8389-1aa903709357/volumes/kubernetes.io~glusterfs/iss3dev-static
type fuse.glusterfs
(rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20171109/de6d59f4/attachment.html>

Sam McLeod

2017-Nov-12 21:27 UTC

head link

[Gluster-users] Gluster clients can't see directories that exist or are created within a mounted volume, but can enter them.

> Clarification for below logs:
> 
> - 'dev_static' is the gluster volume.
> - 'int-kube-01' is the gluster client.
> - '10.51.70.151' is the first node in a three node (2 replica, 1
arbiter) gluster cluster.
> - '/var/lib/kubelet/...../iss3dev-static' is a directory on the
client that should be mounting
'10.51.70.151:/dev_static/iss3dev-static', where
'iss3dev-static' is a directory inside the gluster mount
'dev_static'.
> - FYI only, we are essentially mounting directories inside gluster volumes
as per Kubernetes example:
https://github.com/kubernetes/examples/blob/master/staging/volumes/glusterfs/README.md
<https://github.com/kubernetes/examples/blob/master/staging/volumes/glusterfs/README.md>
> 
> root at int-kube-01:/var/log/glusterfs  # tail -20 mnt.log
> 
> [2017-11-07 21:15:03.561470] I [MSGID: 100030] [glusterfsd.c:2524:main]
0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.12.2 (args:
/usr/sbin/glusterfs --volfile-server=10.51.70.151 --volfile-id=/dev_static /mnt)
> [2017-11-07 21:15:03.571205] W [MSGID: 101002]
[options.c:995:xl_opt_validate] 0-glusterfs: option 'address-family' is
deprecated, preferred is 'transport.address-family', continuing with
correction
> [2017-11-07 21:15:03.584098] I [MSGID: 101190]
[event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started thread with
index 1
> 
> 
> No new log entries appear after executing the following to demonstrate the
problem:
> 
> 
> root at
int-kube-01:/var/lib/kubelet/pods/434cba8e-bf87-11e7-8389-1aa903709357/volumes/kubernetes.io
<http://kubernetes.io/>~glusterfs/iss3dev-static # touch test
> root at
int-kube-01:/var/lib/kubelet/pods/434cba8e-bf87-11e7-8389-1aa903709357/volumes/kubernetes.io
<http://kubernetes.io/>~glusterfs/iss3dev-static # mkdir testdir2
> 
> root at
int-kube-01:/var/lib/kubelet/pods/434cba8e-bf87-11e7-8389-1aa903709357/volumes/kubernetes.io
<http://kubernetes.io/>~glusterfs/iss3dev-static  # ls -ltar
> total 0
> -rw-r--r--. 1 root root 0 Nov  6 10:10 test
> 
> root at
int-kube-01:/var/lib/kubelet/pods/434cba8e-bf87-11e7-8389-1aa903709357/volumes/kubernetes.io
<http://kubernetes.io/>~glusterfs/iss3dev-static  # cd testdir2
> root at
int-kube-01:/var/lib/kubelet/pods/434cba8e-bf87-11e7-8389-1aa903709357/volumes/kubernetes.io
<http://kubernetes.io/>~glusterfs/iss3dev-static/testdir2  # ls -la
> total 0
> 
> root at
int-kube-01:/var/lib/kubelet/pods/434cba8e-bf87-11e7-8389-1aa903709357/volumes/kubernetes.io
<http://kubernetes.io/>~glusterfs/iss3dev-static/testdir2  # cd ..
> root at
int-kube-01:/var/lib/kubelet/pods/434cba8e-bf87-11e7-8389-1aa903709357/volumes/kubernetes.io
<http://kubernetes.io/>~glusterfs/iss3dev-static# ls -ltar
> total 0
> -rw-r--r--. 1 root root 0 Nov  6 10:10 test
> 
> 
> The mount entry looks like:
> 
> root at int-kube-01:~ # mount | grep iss3dev-static
> 10.51.70.151:/dev_static on
/var/lib/kubelet/pods/434cba8e-bf87-11e7-8389-1aa903709357/volumes/kubernetes.io
<http://kubernetes.io/>~glusterfs/iss3dev-static type fuse.glusterfs
(rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)
> _______________________________________________

Does anyone have any ideas with this?

It's essentially stopped us from being able to roll out Gluster as even the
most simple mount using the standard gluster native client with default options
means that clients can't see directories.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.gluster.org/pipermail/gluster-users/attachments/20171113/44340426/attachment.html>

Maybe Matching Threads

Search for more maybe matching threads

Gluster users - Nov 2017 - Gluster clients can't see directories that exist or are created within a mounted volume, but can enter them.

[Gluster-users] Gluster clients can't see directories that exist or are created within a mounted volume, but can enter them.

[Gluster-users] Gluster clients can't see directories that exist or are created within a mounted volume, but can enter them.

[Gluster-users] Gluster clients can't see directories that exist or are created within a mounted volume, but can enter them.

Maybe Matching Threads