Neil Van Lysel
2013-Oct-15 20:44 UTC
[Gluster-users] Remote operation failed: Stale NFS file handle
Hello! Many of our Gluster client nodes are seeing a lot of these errors in their log files: [2013-10-15 06:48:59.467263] W [client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-home-client-6: remote operation failed: Stale NFS file handle. Path: /path (3cfbebf4-40e4-4300-aa6e-bd43b4310b94) [2013-10-15 06:48:59.467331] W [client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-home-client-7: remote operation failed: Stale NFS file handle. Path: /path (3cfbebf4-40e4-4300-aa6e-bd43b4310b94) [2013-10-15 06:48:59.470554] W [client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-home-client-0: remote operation failed: Stale NFS file handle. Path: /path (d662e7db-7864-4b18-b587-bdc5e8756076) [2013-10-15 06:48:59.470624] W [client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-home-client-1: remote operation failed: Stale NFS file handle. Path: /path (d662e7db-7864-4b18-b587-bdc5e8756076) [2013-10-15 06:49:04.537548] W [client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-home-client-3: remote operation failed: Stale NFS file handle. Path: /path (a4ea32e0-25f8-440d-b258-23430490624d) [2013-10-15 06:49:04.537651] W [client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-home-client-2: remote operation failed: Stale NFS file handle. Path: /path (a4ea32e0-25f8-440d-b258-23430490624d) [2013-10-15 06:49:14.380551] W [client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-home-client-0: remote operation failed: Stale NFS file handle. Path: /path (669a2d6b-2998-48b2-8f3f-93d5f65cdd87) [2013-10-15 06:49:14.380663] W [client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-home-client-1: remote operation failed: Stale NFS file handle. Path: /path (669a2d6b-2998-48b2-8f3f-93d5f65cdd87) [2013-10-15 06:49:14.386390] W [client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-home-client-4: remote operation failed: Stale NFS file handle. Path: /path (016aafa9-35ac-4f6f-90bd-b4ac5d435ad0) [2013-10-15 06:49:14.386471] W [client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-home-client-5: remote operation failed: Stale NFS file handle. Path: /path (016aafa9-35ac-4f6f-90bd-b4ac5d435ad0) [2013-10-15 18:28:10.630357] W [client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-scratch-client-2: remote operation failed: Stale NFS file handle. Path: /path (5d6153cc-64b3-4151-85cd-2646c33c6918) [2013-10-15 18:28:10.630425] W [client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-scratch-client-3: remote operation failed: Stale NFS file handle. Path: /path (5d6153cc-64b3-4151-85cd-2646c33c6918) [2013-10-15 18:28:10.636301] W [client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-scratch-client-4: remote operation failed: Stale NFS file handle. Path: /path (2f64b9fe-02a0-408b-9edb-0c5e5bf0ed0e) [2013-10-15 18:28:10.636377] W [client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-scratch-client-5: remote operation failed: Stale NFS file handle. Path: /path (2f64b9fe-02a0-408b-9edb-0c5e5bf0ed0e) [2013-10-15 18:28:10.638574] W [client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-scratch-client-5: remote operation failed: Stale NFS file handle. Path: /path (990de721-1fc9-461d-8412-8c17c23ebbbd) [2013-10-15 18:28:10.638647] W [client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-scratch-client-4: remote operation failed: Stale NFS file handle. Path: /path (990de721-1fc9-461d-8412-8c17c23ebbbd) [2013-10-15 18:28:10.645043] W [client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-scratch-client-7: remote operation failed: Stale NFS file handle. Path: /path (0d8d3c5a-d26e-4c15-a8d5-987a4033a6d0) [2013-10-15 18:28:10.645157] W [client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-scratch-client-6: remote operation failed: Stale NFS file handle. Path: /path (0d8d3c5a-d26e-4c15-a8d5-987a4033a6d0) [2013-10-15 18:28:10.648126] W [client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-scratch-client-6: remote operation failed: Stale NFS file handle. Path: /path (c1c84d57-f54d-4dc1-a5df-9be563da78fb) [2013-10-15 18:28:10.648276] W [client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-scratch-client-7: remote operation failed: Stale NFS file handle. Path: /path (c1c84d57-f54d-4dc1-a5df-9be563da78fb) How can I resolve these errors? *gluster --version: glusterfs 3.4.0 built on Jul 25 2013 04:12:27 *gluster volume info: Volume Name: scratch Type: Distributed-Replicate Volume ID: 198b9d77-96e6-4c7f-9f0c-3618cbcaa940 Status: Started Number of Bricks: 4 x 2 = 8 Transport-type: tcp Bricks: Brick1: 10.129.40.21:/data/glusterfs/brick1/scratch Brick2: 10.129.40.22:/data/glusterfs/brick1/scratch Brick3: 10.129.40.23:/data/glusterfs/brick1/scratch Brick4: 10.129.40.24:/data/glusterfs/brick1/scratch Brick5: 10.129.40.21:/data/glusterfs/brick2/scratch Brick6: 10.129.40.22:/data/glusterfs/brick2/scratch Brick7: 10.129.40.23:/data/glusterfs/brick2/scratch Brick8: 10.129.40.24:/data/glusterfs/brick2/scratch Options Reconfigured: features.quota: off Volume Name: home Type: Distributed-Replicate Volume ID: 0d8ebafc-471e-4b16-a4a9-787ce8616225 Status: Started Number of Bricks: 4 x 2 = 8 Transport-type: tcp Bricks: Brick1: 10.129.40.21:/data/glusterfs/brick1/home Brick2: 10.129.40.22:/data/glusterfs/brick1/home Brick3: 10.129.40.23:/data/glusterfs/brick1/home Brick4: 10.129.40.24:/data/glusterfs/brick1/home Brick5: 10.129.40.21:/data/glusterfs/brick2/home Brick6: 10.129.40.22:/data/glusterfs/brick2/home Brick7: 10.129.40.23:/data/glusterfs/brick2/home Brick8: 10.129.40.24:/data/glusterfs/brick2/home Options Reconfigured: features.quota: off *gluster volume status: Status of volume: scratch Gluster process Port Online Pid ------------------------------------------------------------------------------ Brick 10.129.40.21:/data/glusterfs/brick1/scratch 49154 Y 7536 Brick 10.129.40.22:/data/glusterfs/brick1/scratch 49154 Y 27976 Brick 10.129.40.23:/data/glusterfs/brick1/scratch 49154 Y 7436 Brick 10.129.40.24:/data/glusterfs/brick1/scratch 49154 Y 19773 Brick 10.129.40.21:/data/glusterfs/brick2/scratch 49155 Y 7543 Brick 10.129.40.22:/data/glusterfs/brick2/scratch 49155 Y 27982 Brick 10.129.40.23:/data/glusterfs/brick2/scratch 49155 Y 7442 Brick 10.129.40.24:/data/glusterfs/brick2/scratch 49155 Y 19778 NFS Server on localhost 2049 Y 7564 Self-heal Daemon on localhost N/A Y 7569 NFS Server on 10.129.40.24 2049 Y 19788 Self-heal Daemon on 10.129.40.24 N/A Y 19792 NFS Server on 10.129.40.23 2049 Y 7464 Self-heal Daemon on 10.129.40.23 N/A Y 7468 NFS Server on 10.129.40.22 2049 Y 28004 Self-heal Daemon on 10.129.40.22 N/A Y 28008 There are no active volume tasks Status of volume: home Gluster process Port Online Pid ------------------------------------------------------------------------------ Brick 10.129.40.21:/data/glusterfs/brick1/home 49152 Y 7549 Brick 10.129.40.22:/data/glusterfs/brick1/home 49152 Y 27989 Brick 10.129.40.23:/data/glusterfs/brick1/home 49152 Y 7449 Brick 10.129.40.24:/data/glusterfs/brick1/home 49152 Y 19760 Brick 10.129.40.21:/data/glusterfs/brick2/home 49153 Y 7554 Brick 10.129.40.22:/data/glusterfs/brick2/home 49153 Y 27994 Brick 10.129.40.23:/data/glusterfs/brick2/home 49153 Y 7454 Brick 10.129.40.24:/data/glusterfs/brick2/home 49153 Y 19766 NFS Server on localhost 2049 Y 7564 Self-heal Daemon on localhost N/A Y 7569 NFS Server on 10.129.40.24 2049 Y 19788 Self-heal Daemon on 10.129.40.24 N/A Y 19792 NFS Server on 10.129.40.22 2049 Y 28004 Self-heal Daemon on 10.129.40.22 N/A Y 28008 NFS Server on 10.129.40.23 2049 Y 7464 Self-heal Daemon on 10.129.40.23 N/A Y 7468 There are no active volume tasks *The gluster volumes are mounted using the glusterfs-fuse package (glusterfs-fuse-3.4.0-3.el6.x86_64) on the clients like so: /sbin/mount.glusterfs 10.129.40.21:home /home /sbin/mount.glusterfs 10.129.40.21:scratch /scratch *Gluster packages on Gluster servers: glusterfs-server-3.4.0-3.el6.x86_64 glusterfs-libs-3.4.0-8.el6.x86_64 glusterfs-3.4.0-3.el6.x86_64 glusterfs-geo-replication-3.4.0-3.el6.x86_64 glusterfs-fuse-3.4.0-3.el6.x86_64 glusterfs-rdma-3.4.0-3.el6.x86_64 *Gluster packages on clients: glusterfs-fuse-3.4.0-3.el6.x86_64 glusterfs-3.4.0-3.el6.x86_64 All clients and servers are running the same OS and kernel: *uname -a: Linux <hostname> 2.6.32-358.6.1.el6.x86_64 #1 SMP Tue Apr 23 16:15:13 CDT 2013 x86_64 x86_64 x86_64 GNU/Linux *cat /etc/redhat-release : Scientific Linux release 6.3 (Carbon) Thanks for your help, Neil Van Lysel UNIX Systems Administrator Center for High Throughput Computing University of Wisconsin - Madison -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 3528 bytes Desc: S/MIME Cryptographic Signature URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20131015/cc6e29da/attachment.p7s>
Justin Dossey
2013-Oct-15 21:54 UTC
[Gluster-users] Remote operation failed: Stale NFS file handle
I've seen these errors too on GlusterFS 3.3.1 nodes with glusterfs-fuse mounts. It's particularly strange because we're not using NFS to mount the volumes. On Tue, Oct 15, 2013 at 1:44 PM, Neil Van Lysel <van-lyse at cs.wisc.edu>wrote:> Hello! > > Many of our Gluster client nodes are seeing a lot of these errors in their > log files: > > [2013-10-15 06:48:59.467263] W [client-rpc-fops.c:2624:**client3_3_lookup_cbk] > 0-home-client-6: remote operation failed: Stale NFS file handle. Path: > /path (3cfbebf4-40e4-4300-aa6e-**bd43b4310b94) > [2013-10-15 06:48:59.467331] W [client-rpc-fops.c:2624:**client3_3_lookup_cbk] > 0-home-client-7: remote operation failed: Stale NFS file handle. Path: > /path (3cfbebf4-40e4-4300-aa6e-**bd43b4310b94) > [2013-10-15 06:48:59.470554] W [client-rpc-fops.c:2624:**client3_3_lookup_cbk] > 0-home-client-0: remote operation failed: Stale NFS file handle. Path: > /path (d662e7db-7864-4b18-b587-**bdc5e8756076) > [2013-10-15 06:48:59.470624] W [client-rpc-fops.c:2624:**client3_3_lookup_cbk] > 0-home-client-1: remote operation failed: Stale NFS file handle. Path: > /path (d662e7db-7864-4b18-b587-**bdc5e8756076) > [2013-10-15 06:49:04.537548] W [client-rpc-fops.c:2624:**client3_3_lookup_cbk] > 0-home-client-3: remote operation failed: Stale NFS file handle. Path: > /path (a4ea32e0-25f8-440d-b258-**23430490624d) > [2013-10-15 06:49:04.537651] W [client-rpc-fops.c:2624:**client3_3_lookup_cbk] > 0-home-client-2: remote operation failed: Stale NFS file handle. Path: > /path (a4ea32e0-25f8-440d-b258-**23430490624d) > [2013-10-15 06:49:14.380551] W [client-rpc-fops.c:2624:**client3_3_lookup_cbk] > 0-home-client-0: remote operation failed: Stale NFS file handle. Path: > /path (669a2d6b-2998-48b2-8f3f-**93d5f65cdd87) > [2013-10-15 06:49:14.380663] W [client-rpc-fops.c:2624:**client3_3_lookup_cbk] > 0-home-client-1: remote operation failed: Stale NFS file handle. Path: > /path (669a2d6b-2998-48b2-8f3f-**93d5f65cdd87) > [2013-10-15 06:49:14.386390] W [client-rpc-fops.c:2624:**client3_3_lookup_cbk] > 0-home-client-4: remote operation failed: Stale NFS file handle. Path: > /path (016aafa9-35ac-4f6f-90bd-**b4ac5d435ad0) > [2013-10-15 06:49:14.386471] W [client-rpc-fops.c:2624:**client3_3_lookup_cbk] > 0-home-client-5: remote operation failed: Stale NFS file handle. Path: > /path (016aafa9-35ac-4f6f-90bd-**b4ac5d435ad0) > [2013-10-15 18:28:10.630357] W [client-rpc-fops.c:2624:**client3_3_lookup_cbk] > 0-scratch-client-2: remote operation failed: Stale NFS file handle. Path: > /path (5d6153cc-64b3-4151-85cd-**2646c33c6918) > [2013-10-15 18:28:10.630425] W [client-rpc-fops.c:2624:**client3_3_lookup_cbk] > 0-scratch-client-3: remote operation failed: Stale NFS file handle. Path: > /path (5d6153cc-64b3-4151-85cd-**2646c33c6918) > [2013-10-15 18:28:10.636301] W [client-rpc-fops.c:2624:**client3_3_lookup_cbk] > 0-scratch-client-4: remote operation failed: Stale NFS file handle. Path: > /path (2f64b9fe-02a0-408b-9edb-**0c5e5bf0ed0e) > [2013-10-15 18:28:10.636377] W [client-rpc-fops.c:2624:**client3_3_lookup_cbk] > 0-scratch-client-5: remote operation failed: Stale NFS file handle. Path: > /path (2f64b9fe-02a0-408b-9edb-**0c5e5bf0ed0e) > [2013-10-15 18:28:10.638574] W [client-rpc-fops.c:2624:**client3_3_lookup_cbk] > 0-scratch-client-5: remote operation failed: Stale NFS file handle. Path: > /path (990de721-1fc9-461d-8412-**8c17c23ebbbd) > [2013-10-15 18:28:10.638647] W [client-rpc-fops.c:2624:**client3_3_lookup_cbk] > 0-scratch-client-4: remote operation failed: Stale NFS file handle. Path: > /path (990de721-1fc9-461d-8412-**8c17c23ebbbd) > [2013-10-15 18:28:10.645043] W [client-rpc-fops.c:2624:**client3_3_lookup_cbk] > 0-scratch-client-7: remote operation failed: Stale NFS file handle. Path: > /path (0d8d3c5a-d26e-4c15-a8d5-**987a4033a6d0) > [2013-10-15 18:28:10.645157] W [client-rpc-fops.c:2624:**client3_3_lookup_cbk] > 0-scratch-client-6: remote operation failed: Stale NFS file handle. Path: > /path (0d8d3c5a-d26e-4c15-a8d5-**987a4033a6d0) > [2013-10-15 18:28:10.648126] W [client-rpc-fops.c:2624:**client3_3_lookup_cbk] > 0-scratch-client-6: remote operation failed: Stale NFS file handle. Path: > /path (c1c84d57-f54d-4dc1-a5df-**9be563da78fb) > [2013-10-15 18:28:10.648276] W [client-rpc-fops.c:2624:**client3_3_lookup_cbk] > 0-scratch-client-7: remote operation failed: Stale NFS file handle. Path: > /path (c1c84d57-f54d-4dc1-a5df-**9be563da78fb) > > > How can I resolve these errors? > > > *gluster --version: > glusterfs 3.4.0 built on Jul 25 2013 04:12:27 > > > *gluster volume info: > Volume Name: scratch > Type: Distributed-Replicate > Volume ID: 198b9d77-96e6-4c7f-9f0c-**3618cbcaa940 > Status: Started > Number of Bricks: 4 x 2 = 8 > Transport-type: tcp > Bricks: > Brick1: 10.129.40.21:/data/glusterfs/**brick1/scratch > Brick2: 10.129.40.22:/data/glusterfs/**brick1/scratch > Brick3: 10.129.40.23:/data/glusterfs/**brick1/scratch > Brick4: 10.129.40.24:/data/glusterfs/**brick1/scratch > Brick5: 10.129.40.21:/data/glusterfs/**brick2/scratch > Brick6: 10.129.40.22:/data/glusterfs/**brick2/scratch > Brick7: 10.129.40.23:/data/glusterfs/**brick2/scratch > Brick8: 10.129.40.24:/data/glusterfs/**brick2/scratch > Options Reconfigured: > features.quota: off > > Volume Name: home > Type: Distributed-Replicate > Volume ID: 0d8ebafc-471e-4b16-a4a9-**787ce8616225 > Status: Started > Number of Bricks: 4 x 2 = 8 > Transport-type: tcp > Bricks: > Brick1: 10.129.40.21:/data/glusterfs/**brick1/home > Brick2: 10.129.40.22:/data/glusterfs/**brick1/home > Brick3: 10.129.40.23:/data/glusterfs/**brick1/home > Brick4: 10.129.40.24:/data/glusterfs/**brick1/home > Brick5: 10.129.40.21:/data/glusterfs/**brick2/home > Brick6: 10.129.40.22:/data/glusterfs/**brick2/home > Brick7: 10.129.40.23:/data/glusterfs/**brick2/home > Brick8: 10.129.40.24:/data/glusterfs/**brick2/home > Options Reconfigured: > features.quota: off > > > *gluster volume status: > Status of volume: scratch > Gluster process Port Online Pid > ------------------------------**------------------------------** > ------------------ > Brick 10.129.40.21:/data/glusterfs/**brick1/scratch 49154 Y > 7536 > Brick 10.129.40.22:/data/glusterfs/**brick1/scratch 49154 Y > 27976 > Brick 10.129.40.23:/data/glusterfs/**brick1/scratch 49154 Y > 7436 > Brick 10.129.40.24:/data/glusterfs/**brick1/scratch 49154 Y > 19773 > Brick 10.129.40.21:/data/glusterfs/**brick2/scratch 49155 Y > 7543 > Brick 10.129.40.22:/data/glusterfs/**brick2/scratch 49155 Y > 27982 > Brick 10.129.40.23:/data/glusterfs/**brick2/scratch 49155 Y > 7442 > Brick 10.129.40.24:/data/glusterfs/**brick2/scratch 49155 Y > 19778 > NFS Server on localhost 2049 Y 7564 > Self-heal Daemon on localhost N/A Y 7569 > NFS Server on 10.129.40.24 2049 Y 19788 > Self-heal Daemon on 10.129.40.24 N/A Y 19792 > NFS Server on 10.129.40.23 2049 Y 7464 > Self-heal Daemon on 10.129.40.23 N/A Y 7468 > NFS Server on 10.129.40.22 2049 Y 28004 > Self-heal Daemon on 10.129.40.22 N/A Y 28008 > > There are no active volume tasks > Status of volume: home > Gluster process Port Online Pid > ------------------------------**------------------------------** > ------------------ > Brick 10.129.40.21:/data/glusterfs/**brick1/home 49152 Y > 7549 > Brick 10.129.40.22:/data/glusterfs/**brick1/home 49152 Y > 27989 > Brick 10.129.40.23:/data/glusterfs/**brick1/home 49152 Y > 7449 > Brick 10.129.40.24:/data/glusterfs/**brick1/home 49152 Y > 19760 > Brick 10.129.40.21:/data/glusterfs/**brick2/home 49153 Y > 7554 > Brick 10.129.40.22:/data/glusterfs/**brick2/home 49153 Y > 27994 > Brick 10.129.40.23:/data/glusterfs/**brick2/home 49153 Y > 7454 > Brick 10.129.40.24:/data/glusterfs/**brick2/home 49153 Y > 19766 > NFS Server on localhost 2049 Y 7564 > Self-heal Daemon on localhost N/A Y 7569 > NFS Server on 10.129.40.24 2049 Y 19788 > Self-heal Daemon on 10.129.40.24 N/A Y 19792 > NFS Server on 10.129.40.22 2049 Y 28004 > Self-heal Daemon on 10.129.40.22 N/A Y 28008 > NFS Server on 10.129.40.23 2049 Y 7464 > Self-heal Daemon on 10.129.40.23 N/A Y 7468 > > There are no active volume tasks > > > *The gluster volumes are mounted using the glusterfs-fuse package > (glusterfs-fuse-3.4.0-3.el6.**x86_64) on the clients like so: > /sbin/mount.glusterfs 10.129.40.21:home /home > /sbin/mount.glusterfs 10.129.40.21:scratch /scratch > > > *Gluster packages on Gluster servers: > glusterfs-server-3.4.0-3.el6.**x86_64 > glusterfs-libs-3.4.0-8.el6.**x86_64 > glusterfs-3.4.0-3.el6.x86_64 > glusterfs-geo-replication-3.4.**0-3.el6.x86_64 > glusterfs-fuse-3.4.0-3.el6.**x86_64 > glusterfs-rdma-3.4.0-3.el6.**x86_64 > > > *Gluster packages on clients: > glusterfs-fuse-3.4.0-3.el6.**x86_64 > glusterfs-3.4.0-3.el6.x86_64 > > > All clients and servers are running the same OS and kernel: > > *uname -a: > Linux <hostname> 2.6.32-358.6.1.el6.x86_64 #1 SMP Tue Apr 23 16:15:13 CDT > 2013 x86_64 x86_64 x86_64 GNU/Linux > > *cat /etc/redhat-release : > Scientific Linux release 6.3 (Carbon) > > > Thanks for your help, > > Neil Van Lysel > UNIX Systems Administrator > Center for High Throughput Computing > University of Wisconsin - Madison > > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://supercolony.gluster.org/mailman/listinfo/gluster-users >-- Justin Dossey CTO, PodOmatic -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20131015/7d962882/attachment.html>
Vijay Bellur
2013-Oct-16 12:26 UTC
[Gluster-users] Remote operation failed: Stale NFS file handle
On 10/16/2013 02:14 AM, Neil Van Lysel wrote:> Hello! > > Many of our Gluster client nodes are seeing a lot of these errors in > their log files:Are multiple clients attempting to create and delete the same files/directories simultaneously? Are these errors being observed by applications running on the mount point or are these messages being seen only in the logs? -Vijay