Nithya Balachandran
2017-Nov-08 10:03 UTC
[Gluster-users] Gluster clients can't see directories that exist or are created within a mounted volume, but can enter them.
On 8 November 2017 at 02:47, Sam McLeod <mailinglists at smcleod.net> wrote:> > On 6 Nov 2017, at 3:32 pm, Laura Bailey <lbailey at redhat.com> wrote: > > Do the users have permission to see/interact with the directories, in > addition to the files? > > > Yes, full access to directories and files. > Also testing using the root user. > > > On Mon, Nov 6, 2017 at 1:55 PM, Nithya Balachandran <nbalacha at redhat.com> > wrote: > >> Hi, >> >> Please provide the gluster volume info. Do you see any errors in the >> client mount log file (/var/log/glusterfs/var-lib-mountedgluster.log)? >> > > > root at int-gluster-01:/var/log/glusterfs # grep 'dev_static' *.log|grep -v > cmd_history > > glusterd.log:[2017-11-05 22:37:06.934787] W [glusterd-locks.c:675:glusterd_mgmt_v3_unlock] > (-->/usr/lib64/glusterfs/3.12.2/xlator/mgmt/glusterd.so(+0x22e5a) > [0x7f5047169e5a] -->/usr/lib64/glusterfs/3.12.2/xlator/mgmt/glusterd.so(+0x2cdc8) > [0x7f5047173dc8] -->/usr/lib64/glusterfs/3.12.2/xlator/mgmt/glusterd.so(+0xe372a) > [0x7f504722a72a] ) 0-management: Lock for vol dev_static not held > glusterd.log:[2017-11-05 22:37:06.934806] W [MSGID: 106118] > [glusterd-handler.c:6309:__glusterd_peer_rpc_notify] 0-management: Lock > not released for dev_static > glusterd.log:[2017-11-05 22:39:49.924472] W [glusterd-locks.c:675:glusterd_mgmt_v3_unlock] > (-->/usr/lib64/glusterfs/3.12.2/xlator/mgmt/glusterd.so(+0x22e5a) > [0x7fde97921e5a] -->/usr/lib64/glusterfs/3.12.2/xlator/mgmt/glusterd.so(+0x2cdc8) > [0x7fde9792bdc8] -->/usr/lib64/glusterfs/3.12.2/xlator/mgmt/glusterd.so(+0xe372a) > [0x7fde979e272a] ) 0-management: Lock for vol dev_static not held > glusterd.log:[2017-11-05 22:39:49.924494] W [MSGID: 106118] > [glusterd-handler.c:6309:__glusterd_peer_rpc_notify] 0-management: Lock > not released for dev_static > glusterd.log:[2017-11-05 22:41:42.565123] W [glusterd-locks.c:675:glusterd_mgmt_v3_unlock] > (-->/usr/lib64/glusterfs/3.12.2/xlator/mgmt/glusterd.so(+0x22e5a) > [0x7fde97921e5a] -->/usr/lib64/glusterfs/3.12.2/xlator/mgmt/glusterd.so(+0x2cdc8) > [0x7fde9792bdc8] -->/usr/lib64/glusterfs/3.12.2/xlator/mgmt/glusterd.so(+0xe372a) > [0x7fde979e272a] ) 0-management: Lock for vol dev_static not held > glusterd.log:[2017-11-05 22:41:42.565227] W [MSGID: 106118] > [glusterd-handler.c:6309:__glusterd_peer_rpc_notify] 0-management: Lock > not released for dev_static > glusterd.log:[2017-11-05 22:42:06.931060] W [glusterd-locks.c:675:glusterd_mgmt_v3_unlock] > (-->/usr/lib64/glusterfs/3.12.2/xlator/mgmt/glusterd.so(+0x22e5a) > [0x7fde97921e5a] -->/usr/lib64/glusterfs/3.12.2/xlator/mgmt/glusterd.so(+0x2cdc8) > [0x7fde9792bdc8] -->/usr/lib64/glusterfs/3.12.2/xlator/mgmt/glusterd.so(+0xe372a) > [0x7fde979e272a] ) 0-management: Lock for vol dev_static not held > glusterd.log:[2017-11-05 22:42:06.931090] W [MSGID: 106118] > [glusterd-handler.c:6309:__glusterd_peer_rpc_notify] 0-management: Lock > not released for dev_static > > >>That is not the log for the mount. Please check /var/log/glusterfs/var-lib-mountedgluster.log on the system on which you are running the mount process. Please provide the volume config details as well (gluster volume info) from one of the server nodes.>> Thanks, >> Nithya >> >> On 6 November 2017 at 05:13, Sam McLeod <mailinglists at smcleod.net> wrote: >> >>> We've got an issue with Gluster (3.12.x) where clients can't see >>> directories that exist or are created within a mounted volume. >>> >>> >>> We can see files, but not directories (with ls, find etc...) >>> We can enter (cd) into directories, even if we can't see them. >>> >>> - Host typology is: 2 replica, 1 arbiter. >>> - Volumes are: replicated and running on XFS on the hosts. >>> - Clients are: GlusterFS native fuse client (mount.glusterfs), the same >>> version and op-version as the hosts. >>> - Gluster server and client version: 3.12.2 (also found on 3.12.1, >>> unsure about previous versions) running on CentOS 7. >>> >>> >>> Examples: >>> >>> >>> mount: >>> 192.168.0.151:/gluster_vol on /var/lib/mountedgluster type >>> fuse.glusterfs (rw,relatime,user_id=0,group_i >>> d=0,default_permissions,allow_other,max_read=131072) >>> >>> root at gluster-client:/var/lib/mountedgluster # ls -la >>> total 0 >>> >>> (note no . or .. directories) >>> >>> root at gluster-client:/var/lib/mountedgluster # touch test >>> root at gluster-client:/var/lib/mountedgluster # ls -la >>> total 0 >>> -rw-r--r--. 1 root root 0 Nov 6 10:10 test >>> >>> ("test" file shows up. Still no . or .. directories.) >>> >>> root at gluster-client:/var/lib/mountedgluster # mkdir testdir >>> root at gluster-client:/var/lib/mountedgluster # ls -la >>> total 0 >>> -rw-r--r--. 1 root root 0 Nov 6 10:10 test >>> >>> (directory was made, but doesn't show in ls) >>> >>> root at gluster-client:/var/lib/mountedgluster # cd testdir >>> root at gluster-client:/var/lib/mountedgluster/testdir # ls -la >>> total 0 >>> >>> (cd works, no . or .. shown in ls though) >>> >>> root at gluster-client:/var/lib/mountedgluster/testdir # touch test >>> root at gluster-client:/var/lib/mountedgluster/testdir # ls -la >>> total 0 >>> -rw-r--r--. 1 root root 0 Nov 6 10:10 test >>> >>> (can create test file in testdir) >>> >>> >>> root at gluster-client:/var/lib/mountedgluster/testdir # cd .. >>> root at gluster-client:/var/lib/mountedgluster # ls -ld testdir >>> drwxr-xr-x. 2 root root 4096 Nov 6 10:10 testdir >>> >>> (going back to parent directory, although ls -la doesn't show the >>> directory, if I specify the name to ls -ld it does show) >>> >>> >>> >>> -- >>> Sam McLeod >>> https://twitter.com/s_mcleod >>> https://smcleod.net >>> >>> >>> _______________________________________________ >>> Gluster-users mailing list >>> Gluster-users at gluster.org >>> http://lists.gluster.org/mailman/listinfo/gluster-users >>> >> >> >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org >> http://lists.gluster.org/mailman/listinfo/gluster-users >> > > > > -- > Laura Bailey > Senior Technical Writer > Customer Content Services BNE > > > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20171108/c7b856d2/attachment.html>
Sam McLeod
2017-Nov-08 21:57 UTC
[Gluster-users] Gluster clients can't see directories that exist or are created within a mounted volume, but can enter them.
> On 8 Nov 2017, at 9:03 pm, Nithya Balachandran <nbalacha at redhat.com> wrote: > > > That is not the log for the mount. Please check /var/log/glusterfs/var-lib-mountedgluster.log on the system on which you are running the mount process. > > Please provide the volume config details as well (gluster volume info) from one of the server nodes. >Oh I'm sorry, I totally misread that - didn't realise it was on the client. Clarification for below logs: - 'dev_static' is the gluster volume. - 'int-kube-01' is the gluster client. - '10.51.70.151' is the first node in a three node (2 replica, 1 arbiter) gluster cluster. - '/var/lib/kubelet/...../iss3dev-static' is a directory on the client that should be mounting '10.51.70.151:/dev_static/iss3dev-static', where 'iss3dev-static' is a directory inside the gluster mount 'dev_static'. - FYI only, we are essentially mounting directories inside gluster volumes as per Kubernetes example: https://github.com/kubernetes/examples/blob/master/staging/volumes/glusterfs/README.md <https://github.com/kubernetes/examples/blob/master/staging/volumes/glusterfs/README.md> root at int-kube-01:/var/log/glusterfs # tail -20 mnt.log [2017-11-07 21:15:03.561470] I [MSGID: 100030] [glusterfsd.c:2524:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.12.2 (args: /usr/sbin/glusterfs --volfile-server=10.51.70.151 --volfile-id=/dev_static /mnt) [2017-11-07 21:15:03.571205] W [MSGID: 101002] [options.c:995:xl_opt_validate] 0-glusterfs: option 'address-family' is deprecated, preferred is 'transport.address-family', continuing with correction [2017-11-07 21:15:03.584098] I [MSGID: 101190] [event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 No new log entries appear after executing the following to demonstrate the problem: root at int-kube-01:/var/lib/kubelet/pods/434cba8e-bf87-11e7-8389-1aa903709357/volumes/kubernetes.io~glusterfs/iss3dev-static # touch test root at int-kube-01:/var/lib/kubelet/pods/434cba8e-bf87-11e7-8389-1aa903709357/volumes/kubernetes.io~glusterfs/iss3dev-static # mkdir testdir2 root at int-kube-01:/var/lib/kubelet/pods/434cba8e-bf87-11e7-8389-1aa903709357/volumes/kubernetes.io~glusterfs/iss3dev-static # ls -ltar total 0 -rw-r--r--. 1 root root 0 Nov 6 10:10 test root at int-kube-01:/var/lib/kubelet/pods/434cba8e-bf87-11e7-8389-1aa903709357/volumes/kubernetes.io~glusterfs/iss3dev-static # cd testdir2 root at int-kube-01:/var/lib/kubelet/pods/434cba8e-bf87-11e7-8389-1aa903709357/volumes/kubernetes.io~glusterfs/iss3dev-static/testdir2 # ls -la total 0 root at int-kube-01:/var/lib/kubelet/pods/434cba8e-bf87-11e7-8389-1aa903709357/volumes/kubernetes.io~glusterfs/iss3dev-static/testdir2 # cd .. root at int-kube-01:/var/lib/kubelet/pods/434cba8e-bf87-11e7-8389-1aa903709357/volumes/kubernetes.io~glusterfs/iss3dev-static# ls -ltar total 0 -rw-r--r--. 1 root root 0 Nov 6 10:10 test The mount entry looks like: root at int-kube-01:~ # mount | grep iss3dev-static 10.51.70.151:/dev_static on /var/lib/kubelet/pods/434cba8e-bf87-11e7-8389-1aa903709357/volumes/kubernetes.io~glusterfs/iss3dev-static type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20171109/de6d59f4/attachment.html>
Sam McLeod
2017-Nov-12 21:27 UTC
[Gluster-users] Gluster clients can't see directories that exist or are created within a mounted volume, but can enter them.
> Clarification for below logs: > > - 'dev_static' is the gluster volume. > - 'int-kube-01' is the gluster client. > - '10.51.70.151' is the first node in a three node (2 replica, 1 arbiter) gluster cluster. > - '/var/lib/kubelet/...../iss3dev-static' is a directory on the client that should be mounting '10.51.70.151:/dev_static/iss3dev-static', where 'iss3dev-static' is a directory inside the gluster mount 'dev_static'. > - FYI only, we are essentially mounting directories inside gluster volumes as per Kubernetes example: https://github.com/kubernetes/examples/blob/master/staging/volumes/glusterfs/README.md <https://github.com/kubernetes/examples/blob/master/staging/volumes/glusterfs/README.md> > > root at int-kube-01:/var/log/glusterfs # tail -20 mnt.log > > [2017-11-07 21:15:03.561470] I [MSGID: 100030] [glusterfsd.c:2524:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.12.2 (args: /usr/sbin/glusterfs --volfile-server=10.51.70.151 --volfile-id=/dev_static /mnt) > [2017-11-07 21:15:03.571205] W [MSGID: 101002] [options.c:995:xl_opt_validate] 0-glusterfs: option 'address-family' is deprecated, preferred is 'transport.address-family', continuing with correction > [2017-11-07 21:15:03.584098] I [MSGID: 101190] [event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 > > > No new log entries appear after executing the following to demonstrate the problem: > > > root at int-kube-01:/var/lib/kubelet/pods/434cba8e-bf87-11e7-8389-1aa903709357/volumes/kubernetes.io <http://kubernetes.io/>~glusterfs/iss3dev-static # touch test > root at int-kube-01:/var/lib/kubelet/pods/434cba8e-bf87-11e7-8389-1aa903709357/volumes/kubernetes.io <http://kubernetes.io/>~glusterfs/iss3dev-static # mkdir testdir2 > > root at int-kube-01:/var/lib/kubelet/pods/434cba8e-bf87-11e7-8389-1aa903709357/volumes/kubernetes.io <http://kubernetes.io/>~glusterfs/iss3dev-static # ls -ltar > total 0 > -rw-r--r--. 1 root root 0 Nov 6 10:10 test > > root at int-kube-01:/var/lib/kubelet/pods/434cba8e-bf87-11e7-8389-1aa903709357/volumes/kubernetes.io <http://kubernetes.io/>~glusterfs/iss3dev-static # cd testdir2 > root at int-kube-01:/var/lib/kubelet/pods/434cba8e-bf87-11e7-8389-1aa903709357/volumes/kubernetes.io <http://kubernetes.io/>~glusterfs/iss3dev-static/testdir2 # ls -la > total 0 > > root at int-kube-01:/var/lib/kubelet/pods/434cba8e-bf87-11e7-8389-1aa903709357/volumes/kubernetes.io <http://kubernetes.io/>~glusterfs/iss3dev-static/testdir2 # cd .. > root at int-kube-01:/var/lib/kubelet/pods/434cba8e-bf87-11e7-8389-1aa903709357/volumes/kubernetes.io <http://kubernetes.io/>~glusterfs/iss3dev-static# ls -ltar > total 0 > -rw-r--r--. 1 root root 0 Nov 6 10:10 test > > > The mount entry looks like: > > root at int-kube-01:~ # mount | grep iss3dev-static > 10.51.70.151:/dev_static on /var/lib/kubelet/pods/434cba8e-bf87-11e7-8389-1aa903709357/volumes/kubernetes.io <http://kubernetes.io/>~glusterfs/iss3dev-static type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072) > _______________________________________________Does anyone have any ideas with this? It's essentially stopped us from being able to roll out Gluster as even the most simple mount using the standard gluster native client with default options means that clients can't see directories. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20171113/44340426/attachment.html>
Seemingly Similar Threads
- Gluster clients can't see directories that exist or are created within a mounted volume, but can enter them.
- Gluster clients can't see directories that exist or are created within a mounted volume, but can enter them.
- Kubernetes v1.7.3 and GlusterFS Plugin
- Kubernetes v1.7.3 and GlusterFS Plugin
- Kubernetes v1.7.3 and GlusterFS Plugin