Hello all, I've already tried to post this, but i'm unsure it arrived to the mailing list. I have some issues regarding my nfs mounts. My setup is as follows: Rhel 6.4, gluster 3.4.2-1 running on a vm (4 cores, 8GB ram) attached to a san. I have one disk on which are all the bricks (formatted ext4 in 64 bit mode) of 25TB. On the gluster side of things, everything works without issues. The trouble starts when I mount a volume as an nfs mount. Lots of volumes work without issues, but others behave strangely. The volumes that act weird generally contain many files (can be accidental?). The volumes in question mount without issues, but when I try to go into any subdirectory sometimes it works, sometimes I get errors. On windows with nfs client: access denied In nfslog: [2014-03-31 13:57:58.771241] I [dht-layout.c:638:dht_layout_normalize] 0-caviar_data11-dht: found anomalies in <gfid:c8d94120-6851-46ea-9f28-c629a44b1015>. holes=1 overlaps=0 [2014-03-31 13:57:58.771348] E [nfs3-helpers.c:3595:nfs3_fh_resolve_inode_lookup_cbk] 0-nfs-nfsv3: Lookup failed: <gfid:c8d94120-6851-46ea-9f28-c629a44b1015>: Invalid argument [2014-03-31 13:57:58.771380] E [nfs3.c:1380:nfs3_lookup_resume] 0-nfs-nfsv3: Unable to resolve FH: (192.168.148.46:984) caviar_data11 : c8d94120-6851-46ea-9f28-c629a44b1015 [2014-03-31 13:57:58.771819] W [nfs3-helpers.c:3380:nfs3_log_common_res] 0-nfs-nfsv3: XID: 1ec28530, LOOKUP: NFS: 22(Invalid argument for operation), POSIX: 14(Bad address) [2014-03-31 13:57:58.798967] I [dht-layout.c:638:dht_layout_normalize] 0-caviar_data11-dht: found anomalies in <gfid:14972193-1039-4d7a-aed5-0d7e7eccf57b>. holes=1 overlaps=0 [2014-03-31 13:57:58.799039] E [nfs3-helpers.c:3595:nfs3_fh_resolve_inode_lookup_cbk] 0-nfs-nfsv3: Lookup failed: <gfid:14972193-1039-4d7a-aed5-0d7e7eccf57b>: Invalid argument [2014-03-31 13:57:58.799056] E [nfs3.c:1380:nfs3_lookup_resume] 0-nfs-nfsv3: Unable to resolve FH: (192.168.148.46:984) caviar_data11 : 14972193-1039-4d7a-aed5-0d7e7eccf57b [2014-03-31 13:57:58.799088] W [nfs3-helpers.c:3380:nfs3_log_common_res] 0-nfs-nfsv3: XID: 1ec28531, LOOKUP: NFS: 22(Invalid argument for operation), POSIX: 14(Bad address) .... On linux: [root at lpr-nas01 brick-xiv2]# ll /media/2011/201105/20110530/ ls: /media/2011/201105/20110530/37: No such file or directory total 332 ... drwxrwsr-x 2 nfsnobody 1003 4096 Jun 6 2011 32 drwxrwsr-x 2 nfsnobody 1003 4096 Jun 6 2011 34 drwxrwsr-x 2 nfsnobody 1003 4096 Jun 6 2011 35 drwxrwsr-x 2 nfsnobody 1003 4096 Jun 6 2011 36 drwxrwsr-x 2 nfsnobody 1003 4096 Jun 6 2011 37 ... [root at lpr-nas01 brick-xiv2]# ll /media/2011/201105/20110530/37 ls: /media/2011/201105/20110530/37/NN.0000073824357.00001.tif: No such file or directory ls: /media/2011/201105/20110530/37/NN.0000073824357.00003.tif: No such file or directory total 54 -rwxrwxr-x 0 nfsnobody 1003 9340 Jun 6 2011 NN.0000073824357.00001.tif -rwxrwxr-x 1 nfsnobody 1003 35312 Jun 6 2011 NN.0000073824357.00002.tif -rwxrwxr-x 0 nfsnobody 1003 9340 Jun 6 2011 NN.0000073824357.00003.tif I see in the nfslog: ... [2014-03-31 12:44:18.941083] I [dht-layout.c:638:dht_layout_normalize] 0-caviar_data11-dht: found anomalies in /2011/201107/20110716/55. holes=1 overlaps=0 [2014-03-31 12:44:18.958078] I [dht-layout.c:638:dht_layout_normalize] 0-caviar_data11-dht: found anomalies in /2011/201107/20110716/30. holes=1 overlaps=0 [2014-03-31 12:44:18.959980] I [dht-layout.c:638:dht_layout_normalize] 0-caviar_data11-dht: found anomalies in /2011/201107/20110716/90. holes=1 overlaps=0 [2014-03-31 12:44:18.961094] E [dht-helper.c:429:dht_subvol_get_hashed] (-->/usr/lib64/glusterfs/3.4.2/xlator/debug/io-stats.so(io_stats_lookup+0x157) [0x7fd6a61282e7] (-->/usr/lib64/libglusterfs.so.0(default_lookup+0x6d) [0x3dfe01c03d] (-->/usr/lib64/glusterfs/3.4.2/xlator/cluster/distribute.so(dht_lookup+0xa7e) [0x7fd6a656af2e]))) 0-caviar_data11-dht: invalid argument: loc->parent [2014-03-31 12:44:18.961283] W [client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-caviar_data11-client-0: remote operation failed: Invalid argument. Path: <gfid:00000000-0000-0000-0000-000000000000> (00000000-0000-0000-0000-000000000000) [2014-03-31 12:44:18.961319] E [acl3.c:334:acl3_getacl_resume] 0-nfs-ACL: Unable to resolve FH: (192.168.151.21:740) caviar_data11 : 00000000-0000-0000-0000-000000000000 [2014-03-31 12:44:18.961338] E [acl3.c:342:acl3_getacl_resume] 0-nfs-ACL: unable to open_and_resume ... The weirdest thing is it changes from time to time which files and directories work and which don't Any ideas? Thanks! -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140331/c8dc9857/attachment.html>
Well, saying your client-side is "linux" does not help much. Distro, flavor, etc helps a lot, but I'll take a wild guess here. First, force your NFS mount (client) to use nfs version 3. The same for Microsoft. (It is fair to say I have no idea if the MS client supports v4 or not). Additionally, check that firewalls are disabled on both sides, just for testing. The same goes for SElinux. Windows and ACL, and user mapping is something that might be in your way too. There is a Technet document that describes how to handle this mapping if I am not wrong. Just for testing, mount your nfs share you your own server, using localhost:/nfs_share and see how it goes. It is a good start. Kr, Carlos On Mon, Mar 31, 2014 at 3:58 PM, VAN CAUSBROECK Wannes < Wannes.VANCAUSBROECK at onprvp.fgov.be> wrote:> Hello all, > > > > I've already tried to post this, but i'm unsure it arrived to the mailing > list. > > > > I have some issues regarding my nfs mounts. My setup is as follows: > > Rhel 6.4, gluster 3.4.2-1 running on a vm (4 cores, 8GB ram) attached to a > san. I have one disk on which are all the bricks (formatted ext4 in 64 bit > mode) of 25TB. > > On the gluster side of things, everything works without issues. The > trouble starts when I mount a volume as an nfs mount. > > Lots of volumes work without issues, but others behave strangely. The > volumes that act weird generally contain many files (can be accidental?). > > The volumes in question mount without issues, but when I try to go into > any subdirectory sometimes it works, sometimes I get errors. > > > > On windows with nfs client: access denied > > > > In nfslog: > > [2014-03-31 13:57:58.771241] I [dht-layout.c:638:dht_layout_normalize] > 0-caviar_data11-dht: found anomalies in > <gfid:c8d94120-6851-46ea-9f28-c629a44b1015>. holes=1 overlaps=0 > > [2014-03-31 13:57:58.771348] E > [nfs3-helpers.c:3595:nfs3_fh_resolve_inode_lookup_cbk] 0-nfs-nfsv3: Lookup > failed: <gfid:c8d94120-6851-46ea-9f28-c629a44b1015>: Invalid argument > > [2014-03-31 13:57:58.771380] E [nfs3.c:1380:nfs3_lookup_resume] > 0-nfs-nfsv3: Unable to resolve FH: (192.168.148.46:984) caviar_data11 : > c8d94120-6851-46ea-9f28-c629a44b1015 > > [2014-03-31 13:57:58.771819] W [nfs3-helpers.c:3380:nfs3_log_common_res] > 0-nfs-nfsv3: XID: 1ec28530, LOOKUP: NFS: 22(Invalid argument for > operation), POSIX: 14(Bad address) > > [2014-03-31 13:57:58.798967] I [dht-layout.c:638:dht_layout_normalize] > 0-caviar_data11-dht: found anomalies in > <gfid:14972193-1039-4d7a-aed5-0d7e7eccf57b>. holes=1 overlaps=0 > > [2014-03-31 13:57:58.799039] E > [nfs3-helpers.c:3595:nfs3_fh_resolve_inode_lookup_cbk] 0-nfs-nfsv3: Lookup > failed: <gfid:14972193-1039-4d7a-aed5-0d7e7eccf57b>: Invalid argument > > [2014-03-31 13:57:58.799056] E [nfs3.c:1380:nfs3_lookup_resume] > 0-nfs-nfsv3: Unable to resolve FH: (192.168.148.46:984) caviar_data11 : > 14972193-1039-4d7a-aed5-0d7e7eccf57b > > [2014-03-31 13:57:58.799088] W [nfs3-helpers.c:3380:nfs3_log_common_res] > 0-nfs-nfsv3: XID: 1ec28531, LOOKUP: NFS: 22(Invalid argument for > operation), POSIX: 14(Bad address) > > .... > > > > > > On linux: > > [root at lpr-nas01 brick-xiv2]# ll /media/2011/201105/20110530/ > > ls: /media/2011/201105/20110530/37: No such file or directory > > total 332 > > ... > > drwxrwsr-x 2 nfsnobody 1003 4096 Jun 6 2011 32 > > drwxrwsr-x 2 nfsnobody 1003 4096 Jun 6 2011 34 > > drwxrwsr-x 2 nfsnobody 1003 4096 Jun 6 2011 35 > > drwxrwsr-x 2 nfsnobody 1003 4096 Jun 6 2011 36 > > drwxrwsr-x 2 nfsnobody 1003 4096 Jun 6 2011 37 > > ... > > > > [root at lpr-nas01 brick-xiv2]# ll /media/2011/201105/20110530/37 > > ls: /media/2011/201105/20110530/37/NN.0000073824357.00001.tif: No such > file or directory > > ls: /media/2011/201105/20110530/37/NN.0000073824357.00003.tif: No such > file or directory > > total 54 > > -rwxrwxr-x 0 nfsnobody 1003 9340 Jun 6 2011 NN.0000073824357.00001.tif > > -rwxrwxr-x 1 nfsnobody 1003 35312 Jun 6 2011 NN.0000073824357.00002.tif > > -rwxrwxr-x 0 nfsnobody 1003 9340 Jun 6 2011 NN.0000073824357.00003.tif > > > > > > I see in the nfslog: > > ... > > [2014-03-31 12:44:18.941083] I [dht-layout.c:638:dht_layout_normalize] > 0-caviar_data11-dht: found anomalies in /2011/201107/20110716/55. holes=1 > overlaps=0 > > [2014-03-31 12:44:18.958078] I [dht-layout.c:638:dht_layout_normalize] > 0-caviar_data11-dht: found anomalies in /2011/201107/20110716/30. holes=1 > overlaps=0 > > [2014-03-31 12:44:18.959980] I [dht-layout.c:638:dht_layout_normalize] > 0-caviar_data11-dht: found anomalies in /2011/201107/20110716/90. holes=1 > overlaps=0 > > [2014-03-31 12:44:18.961094] E [dht-helper.c:429:dht_subvol_get_hashed] > (-->/usr/lib64/glusterfs/3.4.2/xlator/debug/io-stats.so(io_stats_lookup+0x157) > [0x7fd6a61282e7] (-->/usr/lib64/libglusterfs.so.0(default_lookup+0x6d) > [0x3dfe01c03d] > (-->/usr/lib64/glusterfs/3.4.2/xlator/cluster/distribute.so(dht_lookup+0xa7e) > [0x7fd6a656af2e]))) 0-caviar_data11-dht: invalid argument: loc->parent > > [2014-03-31 12:44:18.961283] W > [client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-caviar_data11-client-0: > remote operation failed: Invalid argument. Path: > <gfid:00000000-0000-0000-0000-000000000000> > (00000000-0000-0000-0000-000000000000) > > [2014-03-31 12:44:18.961319] E [acl3.c:334:acl3_getacl_resume] 0-nfs-ACL: > Unable to resolve FH: (192.168.151.21:740) caviar_data11 : > 00000000-0000-0000-0000-000000000000 > > [2014-03-31 12:44:18.961338] E [acl3.c:342:acl3_getacl_resume] 0-nfs-ACL: > unable to open_and_resume > > ... > > > > The weirdest thing is it changes from time to time which files and > directories work and which don't > > Any ideas? > > > > Thanks! > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://supercolony.gluster.org/mailman/listinfo/gluster-users >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140331/75ca62ce/attachment.html>
[2014-03-31 12:44:18.941083] I [dht-layout.c:638:dht_layout_normalize] 0-caviar_data11-dht: found anomalies in /2011/201107/20110716/55. holes=1 overlaps=0 This tells us that there is a chunk of hash map missing. My guess would be that the server you're nfs mounting from cannot connect to one of the bricks. On March 31, 2014 6:58:53 AM PDT, VAN CAUSBROECK Wannes <Wannes.VANCAUSBROECK at onprvp.fgov.be> wrote:>Hello all, > >I've already tried to post this, but i'm unsure it arrived to the >mailing list. > >I have some issues regarding my nfs mounts. My setup is as follows: >Rhel 6.4, gluster 3.4.2-1 running on a vm (4 cores, 8GB ram) attached >to a san. I have one disk on which are all the bricks (formatted ext4 >in 64 bit mode) of 25TB. >On the gluster side of things, everything works without issues. The >trouble starts when I mount a volume as an nfs mount. >Lots of volumes work without issues, but others behave strangely. The >volumes that act weird generally contain many files (can be >accidental?). >The volumes in question mount without issues, but when I try to go into >any subdirectory sometimes it works, sometimes I get errors. > >On windows with nfs client: access denied > >In nfslog: >[2014-03-31 13:57:58.771241] I [dht-layout.c:638:dht_layout_normalize] >0-caviar_data11-dht: found anomalies in ><gfid:c8d94120-6851-46ea-9f28-c629a44b1015>. holes=1 overlaps=0 >[2014-03-31 13:57:58.771348] E >[nfs3-helpers.c:3595:nfs3_fh_resolve_inode_lookup_cbk] 0-nfs-nfsv3: >Lookup failed: <gfid:c8d94120-6851-46ea-9f28-c629a44b1015>: Invalid >argument >[2014-03-31 13:57:58.771380] E [nfs3.c:1380:nfs3_lookup_resume] >0-nfs-nfsv3: Unable to resolve FH: (192.168.148.46:984) caviar_data11 : >c8d94120-6851-46ea-9f28-c629a44b1015 >[2014-03-31 13:57:58.771819] W >[nfs3-helpers.c:3380:nfs3_log_common_res] 0-nfs-nfsv3: XID: 1ec28530, >LOOKUP: NFS: 22(Invalid argument for operation), POSIX: 14(Bad address) >[2014-03-31 13:57:58.798967] I [dht-layout.c:638:dht_layout_normalize] >0-caviar_data11-dht: found anomalies in ><gfid:14972193-1039-4d7a-aed5-0d7e7eccf57b>. holes=1 overlaps=0 >[2014-03-31 13:57:58.799039] E >[nfs3-helpers.c:3595:nfs3_fh_resolve_inode_lookup_cbk] 0-nfs-nfsv3: >Lookup failed: <gfid:14972193-1039-4d7a-aed5-0d7e7eccf57b>: Invalid >argument >[2014-03-31 13:57:58.799056] E [nfs3.c:1380:nfs3_lookup_resume] >0-nfs-nfsv3: Unable to resolve FH: (192.168.148.46:984) caviar_data11 : >14972193-1039-4d7a-aed5-0d7e7eccf57b >[2014-03-31 13:57:58.799088] W >[nfs3-helpers.c:3380:nfs3_log_common_res] 0-nfs-nfsv3: XID: 1ec28531, >LOOKUP: NFS: 22(Invalid argument for operation), POSIX: 14(Bad address) >.... > > >On linux: >[root at lpr-nas01 brick-xiv2]# ll /media/2011/201105/20110530/ >ls: /media/2011/201105/20110530/37: No such file or directory >total 332 >... >drwxrwsr-x 2 nfsnobody 1003 4096 Jun 6 2011 32 >drwxrwsr-x 2 nfsnobody 1003 4096 Jun 6 2011 34 >drwxrwsr-x 2 nfsnobody 1003 4096 Jun 6 2011 35 >drwxrwsr-x 2 nfsnobody 1003 4096 Jun 6 2011 36 >drwxrwsr-x 2 nfsnobody 1003 4096 Jun 6 2011 37 >... > >[root at lpr-nas01 brick-xiv2]# ll /media/2011/201105/20110530/37 >ls: /media/2011/201105/20110530/37/NN.0000073824357.00001.tif: No such >file or directory >ls: /media/2011/201105/20110530/37/NN.0000073824357.00003.tif: No such >file or directory >total 54 >-rwxrwxr-x 0 nfsnobody 1003 9340 Jun 6 2011 >NN.0000073824357.00001.tif >-rwxrwxr-x 1 nfsnobody 1003 35312 Jun 6 2011 >NN.0000073824357.00002.tif >-rwxrwxr-x 0 nfsnobody 1003 9340 Jun 6 2011 >NN.0000073824357.00003.tif > > >I see in the nfslog: >... >[2014-03-31 12:44:18.941083] I [dht-layout.c:638:dht_layout_normalize] >0-caviar_data11-dht: found anomalies in /2011/201107/20110716/55. >holes=1 overlaps=0 >[2014-03-31 12:44:18.958078] I [dht-layout.c:638:dht_layout_normalize] >0-caviar_data11-dht: found anomalies in /2011/201107/20110716/30. >holes=1 overlaps=0 >[2014-03-31 12:44:18.959980] I [dht-layout.c:638:dht_layout_normalize] >0-caviar_data11-dht: found anomalies in /2011/201107/20110716/90. >holes=1 overlaps=0 >[2014-03-31 12:44:18.961094] E [dht-helper.c:429:dht_subvol_get_hashed] >(-->/usr/lib64/glusterfs/3.4.2/xlator/debug/io-stats.so(io_stats_lookup+0x157) >[0x7fd6a61282e7] (-->/usr/lib64/libglusterfs.so.0(default_lookup+0x6d) >[0x3dfe01c03d] >(-->/usr/lib64/glusterfs/3.4.2/xlator/cluster/distribute.so(dht_lookup+0xa7e) >[0x7fd6a656af2e]))) 0-caviar_data11-dht: invalid argument: loc->parent >[2014-03-31 12:44:18.961283] W >[client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-caviar_data11-client-0: >remote operation failed: Invalid argument. Path: ><gfid:00000000-0000-0000-0000-000000000000> >(00000000-0000-0000-0000-000000000000) >[2014-03-31 12:44:18.961319] E [acl3.c:334:acl3_getacl_resume] >0-nfs-ACL: Unable to resolve FH: (192.168.151.21:740) caviar_data11 : >00000000-0000-0000-0000-000000000000 >[2014-03-31 12:44:18.961338] E [acl3.c:342:acl3_getacl_resume] >0-nfs-ACL: unable to open_and_resume >... > >The weirdest thing is it changes from time to time which files and >directories work and which don't >Any ideas? > >Thanks! > > >------------------------------------------------------------------------ > >_______________________________________________ >Gluster-users mailing list >Gluster-users at gluster.org >http://supercolony.gluster.org/mailman/listinfo/gluster-users-- Sent from my Android device with K-9 Mail. Please excuse my brevity. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140401/58a5da78/attachment.html>