Heiko Schröter
2011-Mar-16 12:26 UTC
[Gluster-users] STALE NFS Handle (Transport endpoint is not connected)
Hello, the following produces STALE NFS Handle (Transport endpoint is not connected) Scenario: 2 bricks, 1 client Version: 3.1.2 1) Bring gluster system up rd32 ~ # gluster volume info Volume Name: test-volume Type: Distribute Status: Started Number of Bricks: 2 Transport-type: tcp Bricks: Brick1: rd32:/mnt/gluster1 Brick2: rd33:/mnt/gluster1 1a) mount /mnt/gluster on client. 2) Bring bricks down 3) Try to access /mnt/gluster on client: client# ls -la /mnt/gluster ls: cannot access /mnt/gluster: Transport endpoint is not connected 3) Start glusterd on Brick1: client# ls -la /mnt/gluster ls: cannot access /mnt/gluster: Stale NFS file handle 4) Start glusterd on Brick2: client# ls -la /mnt/gluster total 102408 drwxr-xr-x 2 root root 58 Mar 14 15:48 . drwxr-xr-x 7 root root 4096 Mar 2 10:21 .. -rw-r--r-- 1 root root 1 Mar 14 14:36 .version -rw-r--r-- 1 root root 104857600 Mar 14 15:48 willi.dat STALE NFS Handle only vanishs when ALL bricks are up and running, when started as above. This is reproducable. Expected behaviour: Client should never ever see a STALE NFS Handle. i.e If no brick is running -> /mnt/gluster should be just empty. If at least one brick is running -> /mnt/gluster should show the files on that single brick. Thanks and Regards Heiko
Burnash, James
2011-Mar-16 12:33 UTC
[Gluster-users] [SPAM?] STALE NFS Handle (Transport endpoint is not connected)
Hi Heiko. Since you have your bricks setup as Distribute, both bricks participate in the file system - if you look at the underlying directories that make up each brick, you will find some files on each server that appear to be under the same directory when mounted by a client. You must have both bricks up to present a valid filesystem. To get the scenario you are describing, you would have to delete and recreate your Gluster volume as Replicated-Distribute, using the arguments "replica 2" in your gluster volume create stanza. James Burnash, Unix Engineering -----Original Message----- From: gluster-users-bounces at gluster.org [mailto:gluster-users-bounces at gluster.org] On Behalf Of Heiko Schr?ter Sent: Wednesday, March 16, 2011 8:26 AM To: gluster-users at gluster.org Subject: [SPAM?] [Gluster-users] STALE NFS Handle (Transport endpoint is not connected) Importance: Low Hello, the following produces STALE NFS Handle (Transport endpoint is not connected) Scenario: 2 bricks, 1 client Version: 3.1.2 1) Bring gluster system up rd32 ~ # gluster volume info Volume Name: test-volume Type: Distribute Status: Started Number of Bricks: 2 Transport-type: tcp Bricks: Brick1: rd32:/mnt/gluster1 Brick2: rd33:/mnt/gluster1 1a) mount /mnt/gluster on client. 2) Bring bricks down 3) Try to access /mnt/gluster on client: client# ls -la /mnt/gluster ls: cannot access /mnt/gluster: Transport endpoint is not connected 3) Start glusterd on Brick1: client# ls -la /mnt/gluster ls: cannot access /mnt/gluster: Stale NFS file handle 4) Start glusterd on Brick2: client# ls -la /mnt/gluster total 102408 drwxr-xr-x 2 root root 58 Mar 14 15:48 . drwxr-xr-x 7 root root 4096 Mar 2 10:21 .. -rw-r--r-- 1 root root 1 Mar 14 14:36 .version -rw-r--r-- 1 root root 104857600 Mar 14 15:48 willi.dat STALE NFS Handle only vanishs when ALL bricks are up and running, when started as above. This is reproducable. Expected behaviour: Client should never ever see a STALE NFS Handle. i.e If no brick is running -> /mnt/gluster should be just empty. If at least one brick is running -> /mnt/gluster should show the files on that single brick. Thanks and Regards Heiko _______________________________________________ Gluster-users mailing list Gluster-users at gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users DISCLAIMER: This e-mail, and any attachments thereto, is intended only for use by the addressee(s) named herein and may contain legally privileged and/or confidential information. If you are not the intended recipient of this e-mail, you are hereby notified that any dissemination, distribution or copying of this e-mail, and any attachments thereto, is strictly prohibited. If you have received this in error, please immediately notify me and permanently delete the original and any copy of any e-mail and any printout thereof. E-mail transmission cannot be guaranteed to be secure or error-free. The sender therefore does not accept liability for any errors or omissions in the contents of this message which arise as a result of e-mail transmission. NOTICE REGARDING PRIVACY AND CONFIDENTIALITY Knight Capital Group may, at its discretion, monitor and review the content of all e-mail communications. http://www.knight.com
Here you go - thanks for the assistance: root at jc1letgfs5:/etc/glusterd/vols# netstat -anlp | grep glusterd tcp 0 0 10.20.72.156:1023 10.20.72.158:24007 ESTABLISHED 12645/glusterd tcp 0 0 10.20.72.156:1019 10.20.72.159:24007 ESTABLISHED 12645/glusterd tcp 0 0 :::24007 :::* LISTEN 12645/glusterd tcp 0 0 ::ffff:10.20.72.156:24007 ::ffff:10.20.72.158:1021 ESTABLISHED 12645/glusterd tcp 0 0 ::ffff:127.0.0.1:24007 ::ffff:127.0.0.1:1013 ESTABLISHED 12645/glusterd tcp 0 0 ::ffff:127.0.0.1:24007 ::ffff:127.0.0.1:1014 ESTABLISHED 12645/glusterd tcp 0 0 ::ffff:10.20.72.156:24007 ::ffff:10.20.243.188:1022 ESTABLISHED 12645/glusterd tcp 0 0 ::ffff:10.20.72.156:24007 ::ffff:10.20.72.159:1021 ESTABLISHED 12645/glusterd tcp 0 0 ::ffff:10.20.72.156:24007 ::ffff:10.20.72.157:1023 ESTABLISHED 12645/glusterd root at jc1letgfs6:~# netstat -anlp | grep glusterd tcp 0 0 10.20.72.157:1023 10.20.72.156:24007 ESTABLISHED 3853/glusterd tcp 0 0 :::24007 :::* LISTEN 3853/glusterd tcp 0 0 ::ffff:10.20.72.157:24007 ::ffff:10.20.72.159:1023 ESTABLISHED 3853/glusterd tcp 0 0 ::ffff:127.0.0.1:24007 ::ffff:127.0.0.1:1021 ESTABLISHED 3853/glusterd tcp 0 0 ::ffff:127.0.0.1:24007 ::ffff:127.0.0.1:1022 ESTABLISHED 3853/glusterd tcp 0 0 ::ffff:10.20.72.157:24007 ::ffff:10.20.72.158:1023 ESTABLISHED 3853/glusterd root at jc1letgfs7:~# netstat -anlp | grep glusterd tcp 0 0 10.20.72.158:1021 10.20.72.156:24007 ESTABLISHED 3860/glusterd tcp 0 0 10.20.72.158:1009 10.20.72.159:24007 ESTABLISHED 3860/glusterd tcp 0 0 10.20.72.158:1023 10.20.72.157:24007 ESTABLISHED 3860/glusterd tcp 0 0 :::24007 :::* LISTEN 3860/glusterd tcp 0 0 ::ffff:10.20.72.158:24007 ::ffff:10.20.72.156:1023 ESTABLISHED 3860/glusterd tcp 0 0 ::ffff:127.0.0.1:24007 ::ffff:127.0.0.1:1020 ESTABLISHED 3860/glusterd tcp 0 0 ::ffff:127.0.0.1:24007 ::ffff:127.0.0.1:1019 ESTABLISHED 3860/glusterd tcp 0 0 ::ffff:10.20.72.158:24007 ::ffff:10.20.72.159:1022 ESTABLISHED 3860/glusterd root at jc1letgfs8:~# netstat -anlp | grep glusterd tcp 0 0 10.20.72.159:1023 10.20.72.157:24007 ESTABLISHED 3851/glusterd tcp 0 0 10.20.72.159:1022 10.20.72.158:24007 ESTABLISHED 3851/glusterd tcp 0 0 10.20.72.159:1021 10.20.72.156:24007 ESTABLISHED 3851/glusterd tcp 0 0 :::24007 :::* LISTEN 3851/glusterd tcp 0 0 ::ffff:127.0.0.1:24007 ::ffff:127.0.0.1:1020 ESTABLISHED 3851/glusterd tcp 0 0 ::ffff:127.0.0.1:24007 ::ffff:127.0.0.1:1019 ESTABLISHED 3851/glusterd tcp 0 0 ::ffff:10.20.72.159:24007 ::ffff:10.20.72.156:1019 ESTABLISHED 3851/glusterd tcp 0 0 ::ffff:10.20.72.159:24007 ::ffff:10.20.72.158:1009 ESTABLISHED 3851/glusterd -----Original Message----- From: Vijay Bellur [mailto:vijay at gluster.com] Sent: Monday, March 21, 2011 12:51 PM To: Burnash, James Subject: Re: [Gluster-users] What does this error mean? On Monday 21 March 2011 10:04 PM, Burnash, James wrote:> root at jc1letgfs5:/etc/glusterd/vols# gluster peer status > Number of Peers: 3 > > Hostname: jc1letgfs6 > Uuid: cd590fad-022c-4b9a-97f5-3262080d772d > State: Peer in Cluster (Disconnected) > > Hostname: jc1letgfs7 > Uuid: c5f40de4-9bb1-47ad-93b6-d52c6689ee29 > State: Peer in Cluster (Disconnected) >Can you please provide the o/p of "netstat -anlp | grep glusterd" from all the 4 nodes? Has there been any change in IP address of any of the nodes from the time you set up the cluster? Regards, Vijay DISCLAIMER: This e-mail, and any attachments thereto, is intended only for use by the addressee(s) named herein and may contain legally privileged and/or confidential information. If you are not the intended recipient of this e-mail, you are hereby notified that any dissemination, distribution or copying of this e-mail, and any attachments thereto, is strictly prohibited. If you have received this in error, please immediately notify me and permanently delete the original and any copy of any e-mail and any printout thereof. E-mail transmission cannot be guaranteed to be secure or error-free. The sender therefore does not accept liability for any errors or omissions in the contents of this message which arise as a result of e-mail transmission. NOTICE REGARDING PRIVACY AND CONFIDENTIALITY Knight Capital Group may, at its discretion, monitor and review the content of all e-mail communications. http://www.knight.com