Jan Písačka (@nsa.gov)
2015-Sep-02 09:35 UTC
[Gluster-users] Some files inaccessible via NFS
Hi everybody, we are experiencing random problems accessing some files on a volume via NFS. Any help is highly appreciated. The setup: CentOS release 6.5 glusterfs-libs-3.4.7-1.el6.x86_64 glusterfs-fuse-3.4.7-1.el6.x86_64 glusterfs-server-3.4.7-1.el6.x86_64 glusterfs-3.4.7-1.el6.x86_64 glusterfs-cli-3.4.7-1.el6.x86_64 Volume Name: repsilo Type: Distributed-Replicate Volume ID: 94d8f999-bf1e-4f99-a96f-5bb6564222d9 Status: Started Number of Bricks: 10 x 2 = 20 Transport-type: tcp Bricks: Brick1: gls07.srv:/mnt/glusterRawL Brick2: gls08.srv:/mnt/glusterRawL Brick3: gls07.srv:/mnt/glusterRawR Brick4: gls08.srv:/mnt/glusterRawR Brick5: gls14.srv:/mnt/glusterRawL Brick6: gls15.srv:/mnt/glusterRawL Brick7: gls14.srv:/mnt/glusterRawR Brick8: gls15.srv:/mnt/glusterRawR Brick9: gls12.srv:/mnt/glusterRawL Brick10: gls13.srv:/mnt/glusterRawL Brick11: gls12.srv:/mnt/glusterRawR Brick12: gls13.srv:/mnt/glusterRawR Brick13: gls01.srv:/mnt/glusterRawL Brick14: gls02.srv:/mnt/glusterRawL Brick15: gls01.srv:/mnt/glusterRawR Brick16: gls02.srv:/mnt/glusterRawR Brick17: gls03.srv:/mnt/glusterRawL Brick18: gls04.srv:/mnt/glusterRawL Brick19: gls03.srv:/mnt/glusterRawR Brick20: gls04.srv:/mnt/glusterRawR While the files are readable through the native client, trying to read them via NFS fails on some servers. This is an example - relevant content of nfs.log on the gls04 server: [2015-09-02 08:54:51.916981] W [client-rpc-fops.c:259:client3_3_mknod_cbk] 0-repsilo-client-18: remote operation failed: Permission denied. Path: /CDB_data/7907/RAW_DATA/MARTE_NODE.FeedbackDataCollection.EFPSCurrentFeedbackReq uest.1.h5 (e3a07a5a-9ef4-4bbf-a2a9-86b1dd078a12) [2015-09-02 08:54:51.917051] W [client-rpc-fops.c:259:client3_3_mknod_cbk] 0-repsilo-client-19: remote operation failed: Permission denied. Path: /CDB_data/7907/RAW_DATA/MARTE_NODE.FeedbackDataCollection.EFPSCurrentFeedbackRequest.1.h5 (e3a07a5a-9ef4-4bbf-a2a9-86b1dd078a12) [2015-09-02 08:54:51.917109] E [dht-helper.c:1259:dht_inode_ctx_get] (-->/usr/lib64/glusterfs/3.4.7/xlator/cluster/distribute.so(dht_lookup_linkfile_create_cbk+0x6d) [0x7fbfd8398f6d] (-->/usr/lib64/glusterfs/3.4.7/xlator/cluster/distribute.so(dht_layout_preset+0x5e) [0x7fbfd837be6e] (-->/usr/lib64/glusterfs/3.4.7/xlator/cluster/distribute.so(dht_inode_ctx_layout_set+0x34) [0x7fbfd837de84]))) 0-repsilo-dht: invalid argument: inode [2015-09-02 08:54:51.917140] E [dht-helper.c:1278:dht_inode_ctx_set] (-->/usr/lib64/glusterfs/3.4.7/xlator/cluster/distribute.so(dht_lookup_linkfile_create_cbk+0x6d) [0x7fbfd8398f6d] (-->/usr/lib64/glusterfs/3.4.7/xlator/cluster/distribute.so(dht_layout_preset+0x5e) [0x7fbfd837be6e] (-->/usr/lib64/glusterfs/3.4.7/xlator/cluster/distribute.so(dht_inode_ctx_layout_set+0x52) [0x7fbfd837dea2]))) 0-repsilo-dht: invalid argument: inode [2015-09-02 08:54:51.918177] W [nfs3.c:726:nfs3svc_getattr_stat_cbk] 0-nfs: a4fcea5f: /CDB_data/7907/RAW_DATA/MARTE_NODE.FeedbackDataCollection.EFPSCurrentFeedbackRequest.1.h5 => -1 (Invalid argument) [2015-09-02 08:54:51.918225] W [nfs3-helpers.c:3380:nfs3_log_common_res] 0-nfs-nfsv3: XID: a4fcea5f, GETATTR: NFS: 22(Invalid argument for operation), POSIX: 22(Invalid argument) [2015-09-02 08:54:51.920121] W [client-rpc-fops.c:259:client3_3_mknod_cbk] 0-repsilo-client-19: remote operation failed: Permission denied. Path: /CDB_data/7907/RAW_DATA/MARTE_NODE.FeedbackDataCollection.EFPSCurrentFeedbackRequest.1.h5 (e3a07a5a-9ef4-4bbf-a2a9-86b1dd078a12) [2015-09-02 08:54:51.920208] W [client-rpc-fops.c:259:client3_3_mknod_cbk] 0-repsilo-client-18: remote operation failed: Permission denied. Path: /CDB_data/7907/RAW_DATA/MARTE_NODE.FeedbackDataCollection.EFPSCurrentFeedbackRequest.1.h5 (e3a07a5a-9ef4-4bbf-a2a9-86b1dd078a12) [2015-09-02 08:54:51.920267] E [dht-helper.c:1259:dht_inode_ctx_get] (-->/usr/lib64/glusterfs/3.4.7/xlator/cluster/distribute.so(dht_lookup_linkfile_create_cbk+0x6d) [0x7fbfd8398f6d] (-->/usr/lib64/glusterfs/3.4.7/xlator/cluster/distribute.so(dht_layout_preset+0x5e) [0x7fbfd837be6e] (-->/usr/lib64/glusterfs/3.4.7/xlator/cluster/distribute.so(dht_inode_ctx_layout_set+0x34) [0x7fbfd837de84]))) 0-repsilo-dht: invalid argument: inode [2015-09-02 08:54:51.920298] E [dht-helper.c:1278:dht_inode_ctx_set] (-->/usr/lib64/glusterfs/3.4.7/xlator/cluster/distribute.so(dht_lookup_linkfile_create_cbk+0x6d) [0x7fbfd8398f6d] (-->/usr/lib64/glusterfs/3.4.7/xlator/cluster/distribute.so(dht_layout_preset+0x5e) [0x7fbfd837be6e] (-->/usr/lib64/glusterfs/3.4.7/xlator/cluster/distribute.so(dht_inode_ctx_layout_set+0x52) [0x7fbfd837dea2]))) 0-repsilo-dht: invalid argument: inode [2015-09-02 08:54:51.920679] W [nfs3.c:1512:nfs3svc_access_cbk] 0-nfs: a6fcea5f: /CDB_data/7907/RAW_DATA/MARTE_NODE.FeedbackDataCollection.EFPSCurrentFeedbackRequest.1.h5 => -1 (Invalid argument) [2015-09-02 08:54:51.920703] W [nfs3-helpers.c:3380:nfs3_log_common_res] 0-nfs-nfsv3: XID: a6fcea5f, ACCESS: NFS: 22(Invalid argument for operation), POSIX: 22(Invalid argument) [2015-09-02 08:54:51.921374] W [nfs3.c:1512:nfs3svc_access_cbk] 0-nfs: a8fcea5f: /CDB_data/7907/RAW_DATA/MARTE_NODE.FeedbackDataCollection.EFPSCurrentFeedbackRequest.1.h5 => -1 (Invalid argument) [2015-09-02 08:54:51.921391] W [nfs3-helpers.c:3380:nfs3_log_common_res] 0-nfs-nfsv3: XID: a8fcea5f, ACCESS: NFS: 22(Invalid argument for operation), POSIX: 22(Invalid argument) [2015-09-02 08:55:07.041430] W [nfs3.c:726:nfs3svc_getattr_stat_cbk] 0-nfs: a9fcea5f: /CDB_data/7907/RAW_DATA/MARTE_NODE.FeedbackDataCollection.EFPSCurrentFeedbackRequest.1.h5 => -1 (Invalid argument) [2015-09-02 08:55:07.041493] W [nfs3-helpers.c:3380:nfs3_log_common_res] 0-nfs-nfsv3: XID: a9fcea5f, GETATTR: NFS: 22(Invalid argument for operation), POSIX: 22(Invalid argument) gluster volume heal repsilo info heal-failed and gluster volume heal repsilo info split-brain both give Number of entries: 0 on all bricks Thanks. Regards, Jan