Uwe Weiss
2011-Jul-12 14:16 UTC
[Gluster-users] glusterfs 3.2.1 nfs hangs on opensuse 11.4
Hi list, I have some trouble with glusterfs 3.2.1. compiled from tarball in two opensue 11.4 boxes. Compiling an Installing of glusterfs went fine. Afterwards I have configured a distributed replicated volume named gluster_vol1 (peers are of course connected to each other) As long as I only use the native gluster-client everything works fine an I am reallay happy about glusterfs. But if I try to use nfs I am running in trouble. In details: I have two hosts called vmhost1 (192.168.50.1) and vmhost2(192.168.50.2). On both hosts glusterfs is started. On both hosts I have mounted a directory named /nfs_mount with: mount -t nfs -o mountproto=tcp 192.168.50.1:/gluster_vol1 /nfs_mount Afterward I can ls, df, du and so on with this directory on both hosts. Even copying of small files works mostly but not every times. But if I try to create bigfiles with dd for example [], the connections hangs after one or two minutes. Sometimes it helps to restart glusterfs on the mounted node, sometimes only a reset of the master node solves the problem. This happens if I run the copying or the creation of bigfile on any node that has mounted the gluster nfs share independently if I run this command from a single node or concurrently from multiple nodes. Here is my logfile from my masternode after starting glusterfs on both nodes: cat etc-glusterfs-glusterd.vol.log [2011-07-12 15:19:08.455046] I [glusterd.c:564:init] 0-management: Using /etc/glusterd as working directory [2011-07-12 15:19:08.467129] E [rpc-transport.c:676:rpc_transport_load] 0-rpc-transport: /usr/local/lib/glusterfs/3.2.1/rpc-transport/rdma.so: cannot open shared object file: No such file or directory [2011-07-12 15:19:08.467151] E [rpc-transport.c:680:rpc_transport_load] 0-rpc-transport: volume 'rdma.management': transport-type 'rdma' is not valid or not found on this machine [2011-07-12 15:19:08.467164] W [rpcsvc.c:1288:rpcsvc_transport_create] 0-rpc-service: cannot create listener, initing the transport failed [2011-07-12 15:19:08.475337] I [glusterd.c:88:glusterd_uuid_init] 0-glusterd: retrieved UUID: 358770a7-d1ee-4fbd-8552-b354a2295d79 [2011-07-12 15:19:09.852048] E [glusterd-store.c:1567:glusterd_store_retrieve_volume] 0-: Unknown key: brick-0 [2011-07-12 15:19:09.852093] E [glusterd-store.c:1567:glusterd_store_retrieve_volume] 0-: Unknown key: brick-1 [2011-07-12 15:19:10.72382] I [glusterd-handler.c:3399:glusterd_friend_add] 0-glusterd: connect returned 0 [2011-07-12 15:19:10.74324] I [glusterd-utils.c:1092:glusterd_volume_start_glusterfs] 0-: About to start glusterfs for brick 192.168.50.1:/sp1/glusterfs [2011-07-12 15:19:10.98987] I [glusterd-utils.c:2217:glusterd_nfs_pmap_deregister] 0-: De-registered MOUNTV3 successfully [2011-07-12 15:19:10.100043] I [glusterd-utils.c:2222:glusterd_nfs_pmap_deregister] 0-: De-registered MOUNTV1 successfully [2011-07-12 15:19:10.101123] I [glusterd-utils.c:2227:glusterd_nfs_pmap_deregister] 0-: De-registered NFSV3 successfully Given volfile: +--------------------------------------------------------------------------- ---+ 1: volume management 2: type mgmt/glusterd 3: option working-directory /etc/glusterd 4: option transport-type socket,rdma 5: option transport.socket.keepalive-time 10 6: option transport.socket.keepalive-interval 2 7: end-volume 8: +--------------------------------------------------------------------------- ---+ [2011-07-12 15:19:10.115334] E [socket.c:1685:socket_connect_finish] 0-management: connection to failed (Connection refused) [2011-07-12 15:19:10.193987] I [glusterd-pmap.c:237:pmap_registry_bind] 0-pmap: adding brick /sp1/glusterfs on port 24010 [2011-07-12 15:19:10.224800] W [socket.c:1494:__socket_proto_state_machine] 0-socket.management: reading from socket failed. Error (Transport endpoint is not connected), peer (192.168.50.1:1023) [2011-07-12 15:21:39.983189] W [socket.c:1494:__socket_proto_state_machine] 0-socket.management: reading from socket failed. Error (Transport endpoint is not connected), peer (192.168.50.2:1021) [2011-07-12 15:21:40.318042] I [glusterd-handshake.c:317:glusterd_set_clnt_mgmt_program] 0-: Using Program glusterd clnt mgmt, Num (1238433), Version (1) Here is my nfs.log before mounting anything: cat nfs.log [2011-07-12 15:19:10.213621] I [nfs.c:675:init] 0-nfs: NFS service started [2011-07-12 15:19:10.213743] W [write-behind.c:3029:init] 0-gluster_vol1-write-behind: disabling write-behind for first 0 bytes [2011-07-12 15:19:10.216610] I [client.c:1935:notify] 0-gluster_vol1-client-0: parent translators are ready, attempting connect on transport [2011-07-12 15:19:10.220675] I [client.c:1935:notify] 0-gluster_vol1-client-1: parent translators are ready, attempting connect on transport Given volfile: +--------------------------------------------------------------------------- ---+ 1: volume gluster_vol1-client-0 2: type protocol/client 3: option remote-host 192.168.50.1 4: option remote-subvolume /sp1/glusterfs 5: option transport-type tcp 6: end-volume 7: 8: volume gluster_vol1-client-1 9: type protocol/client 10: option remote-host 192.168.50.2 11: option remote-subvolume /sp1/glusterfs 12: option transport-type tcp 13: end-volume 14: 15: volume gluster_vol1-replicate-0 16: type cluster/replicate 17: subvolumes gluster_vol1-client-0 gluster_vol1-client-1 18: end-volume 19: 20: volume gluster_vol1-write-behind 21: type performance/write-behind 22: subvolumes gluster_vol1-replicate-0 23: end-volume 24: 25: volume gluster_vol1-read-ahead 26: type performance/read-ahead 27: subvolumes gluster_vol1-write-behind 28: end-volume 29: 30: volume gluster_vol1-io-cache 31: type performance/io-cache 32: subvolumes gluster_vol1-read-ahead 33: end-volume 34: 35: volume gluster_vol1-quick-read 36: type performance/quick-read 37: subvolumes gluster_vol1-io-cache 38: end-volume 39: 40: volume gluster_vol1 41: type debug/io-stats 42: option latency-measurement off 43: option count-fop-hits off 44: subvolumes gluster_vol1-quick-read 45: end-volume 46: 47: volume nfs-server 48: type nfs/server 49: option nfs.dynamic-volumes on 50: option rpc-auth.addr.gluster_vol1.allow * 51: option nfs3.gluster_vol1.volume-id 41ec51e6-e67a-45d4-bc80-be06c2453333 52: subvolumes gluster_vol1 53: end-volume +--------------------------------------------------------------------------- ---+ [2011-07-12 15:19:10.224263] E [socket.c:1685:socket_connect_finish] 0-gluster_vol1-client-1: connection to failed (Connection refused) [2011-07-12 15:19:10.224670] I [rpc-clnt.c:1531:rpc_clnt_reconfig] 0-gluster_vol1-client-0: changing port to 24010 (from 0) [2011-07-12 15:19:14.229314] I [client-handshake.c:1082:select_server_supported_programs] 0-gluster_vol1-client-0: Using Program GlusterFS-3.1.0, Num (1298437), Version (310) [2011-07-12 15:19:14.231655] I [client-handshake.c:913:client_setvolume_cbk] 0-gluster_vol1-client-0: Connected to 192.168.50.1:24010, attached to remote volume '/sp1/glusterfs'. [2011-07-12 15:19:14.231690] I [afr-common.c:2591:afr_notify] 0-gluster_vol1-replicate-0: Subvolume 'gluster_vol1-client-0' came back up; going online. [2011-07-12 15:19:14.232730] I [afr-common.c:902:afr_fresh_lookup_cbk] 0-gluster_vol1-replicate-0: added root inode [2011-07-12 15:21:39.850087] E [client-handshake.c:1163:client_query_portmap_cbk] 0-gluster_vol1-client-1: failed to get the port number for remote subvolume [2011-07-12 15:21:39.850177] I [client.c:1883:client_rpc_notify] 0-gluster_vol1-client-1: disconnected [2011-07-12 15:21:43.473577] I [rpc-clnt.c:1531:rpc_clnt_reconfig] 0-gluster_vol1-client-1: changing port to 24010 (from 0) [2011-07-12 15:21:47.478306] I [client-handshake.c:1082:select_server_supported_programs] 0-gluster_vol1-client-1: Using Program GlusterFS-3.1.0, Num (1298437), Version (310) [2011-07-12 15:21:47.478804] I [client-handshake.c:913:client_setvolume_cbk] 0-gluster_vol1-client-1: Connected to 192.168.50.2:24010, attached to remote volume '/sp1/glusterfs'. This is the bricks log after starting glusterfs on both nodes cat bricks/sp1-glusterfs.log [2011-07-12 15:19:10.109639] W [socket.c:419:__socket_keepalive] 0-socket: failed to set keep idle on socket 8 [2011-07-12 15:19:10.109714] W [socket.c:1846:socket_server_event_handler] 0-socket.glusterfsd: Failed to set keep-alive: Operation not supported [2011-07-12 15:19:10.175912] W [graph.c:274:gf_add_cmdline_options] 0-gluster_vol1-server: adding option 'listen-port' for volume 'gluster_vol1-server' with value '24010' [2011-07-12 15:19:10.176601] W [rpc-transport.c:447:validate_volume_options] 0-tcp.gluster_vol1-server: option 'listen-port' is deprecated, preferred is 'transport.socket.listen-port', continuing with correction Given volfile: +--------------------------------------------------------------------------- ---+ 1: volume gluster_vol1-posix 2: type storage/posix 3: option directory /sp1/glusterfs 4: end-volume 5: 6: volume gluster_vol1-access-control 7: type features/access-control 8: subvolumes gluster_vol1-posix 9: end-volume 10: 11: volume gluster_vol1-locks 12: type features/locks 13: subvolumes gluster_vol1-access-control 14: end-volume 15: 16: volume gluster_vol1-io-threads 17: type performance/io-threads 18: subvolumes gluster_vol1-locks 19: end-volume 20: 21: volume gluster_vol1-marker 22: type features/marker 23: option volume-uuid 41ec51e6-e67a-45d4-bc80-be06c2453333 24: option timestamp-file /etc/glusterd/vols/gluster_vol1/marker.tstamp 25: option xtime off 26: option quota off 27: subvolumes gluster_vol1-io-threads 28: end-volume 29: 30: volume /sp1/glusterfs 31: type debug/io-stats 32: option latency-measurement off 33: option count-fop-hits off 34: subvolumes gluster_vol1-marker 35: end-volume 36: 37: volume gluster_vol1-server 38: type protocol/server 39: option transport-type tcp 40: option auth.addr./sp1/glusterfs.allow * 41: subvolumes /sp1/glusterfs 42: end-volume +--------------------------------------------------------------------------- ---+ [2011-07-12 15:19:14.229546] I [server-handshake.c:534:server_setvolume] 0-gluster_vol1-server: accepted client from 192.168.50.1:1022 [2011-07-12 15:21:43.990367] I [server-handshake.c:534:server_setvolume] 0-gluster_vol1-server: accepted client from 192.168.50.2:1019 After a successful mount of vmhost1 from vmhost2 I see the following from the mount command: 192.168.50.1:/gluster_vol1 on /nfs_mount type nfs (rw,relatime,vers=3,rsize=65536,wsize=65536,namlen=255,hard,proto=tcp,port=3 8467,timeo=600,retrans=2,sec=sys,mountaddr=192.168.50.1,mountvers=3,mountpor t=38465,mountproto=tcp,local_lock=none,addr=192.168.50.1) Afterwards I tried to create a bigfile in /nfs_mount with: dd < /dev/zero > /nfs_mount/bigfile2.img bs=1M count=20480 The command runs a few seconds and than it hangs. I found the following entrys in nfs log on the master masterserver (vmhost1) [2011-07-12 15:54:40.840976] C [client-handshake.c:121:rpc_client_ping_timer_expired] 0-gluster_vol1-client-1: server 192.168.50.2:24010 has not responded in the last 42 seconds, disconnecting. [2011-07-12 15:54:40.841670] E [rpc-clnt.c:338:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$ [2011-07-12 15:54:40.841839] I [client3_1-fops.c:683:client3_1_writev_cbk] 0-gluster_vol1-client-1: remote operation failed: Transport endpoint is not connected [2011-07-12 15:54:40.846839] I [socket.c:2275:socket_submit_request] 0-gluster_vol1-client-1: not connected (priv->connected = 0) [2011-07-12 15:54:40.847046] W [rpc-clnt.c:1411:rpc_clnt_submit] 0-gluster_vol1-client-1: failed to submit rpc-request (XID: 0x263239x Program: GlusterFS 3.1, ProgVers: 310, Proc: 34) to rpc-transport (gluster_vol1-$ [2011-07-12 15:54:40.847091] W [rpc-clnt.c:1411:rpc_clnt_submit] 0-gluster_vol1-client-1: failed to submit rpc-request (XID: 0x263240x Program: GlusterFS 3.1, ProgVers: 310, Proc: 30) to rpc-transport (gluster_vol1-$ [2011-07-12 15:54:40.847130] E [rpc-clnt.c:338:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$ [2011-07-12 15:54:40.847159] I [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1: remote operation failed: Transport endpoint is not connected [2011-07-12 15:54:40.847193] I [client3_1-fops.c:683:client3_1_writev_cbk] 0-gluster_vol1-client-1: remote operation failed: Transport endpoint is not connected [2011-07-12 15:54:40.847423] W [rpc-clnt.c:1411:rpc_clnt_submit] 0-gluster_vol1-client-1: failed to submit rpc-request (XID: 0x263241x Program: GlusterFS 3.1, ProgVers: 310, Proc: 34) to rpc-transport (gluster_vol1-$ [2011-07-12 15:54:40.847477] E [rpc-clnt.c:338:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$ [2011-07-12 15:54:40.847495] I [client3_1-fops.c:683:client3_1_writev_cbk] 0-gluster_vol1-client-1: remote operation failed: Transport endpoint is not connected [2011-07-12 15:54:40.847571] W [rpc-clnt.c:1411:rpc_clnt_submit] 0-gluster_vol1-client-1: failed to submit rpc-request (XID: 0x263242x Program: GlusterFS 3.1, ProgVers: 310, Proc: 34) to rpc-transport (gluster_vol1-$ [2011-07-12 15:54:40.847626] E [rpc-clnt.c:338:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$ [2011-07-12 15:54:40.847643] I [client3_1-fops.c:683:client3_1_writev_cbk] 0-gluster_vol1-client-1: remote operation failed: Transport endpoint is not connected [2011-07-12 15:54:40.847716] W [rpc-clnt.c:1411:rpc_clnt_submit] 0-gluster_vol1-client-1: failed to submit rpc-request (XID: 0x263243x Program: GlusterFS 3.1, ProgVers: 310, Proc: 34) to rpc-transport (gluster_vol1-$ [2011-07-12 15:54:40.847771] E [rpc-clnt.c:338:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$ [2011-07-12 15:54:40.847798] I [client3_1-fops.c:683:client3_1_writev_cbk] 0-gluster_vol1-client-1: remote operation failed: Transport endpoint is not connected [2011-07-12 15:54:40.847874] W [rpc-clnt.c:1411:rpc_clnt_submit] 0-gluster_vol1-client-1: failed to submit rpc-request (XID: 0x263244x Program: GlusterFS 3.1, ProgVers: 310, Proc: 34) to rpc-transport (gluster_vol1-$ [2011-07-12 15:54:40.847984] E [rpc-clnt.c:338:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$ [2011-07-12 15:54:40.848006] I [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1: remote operation failed: Transport endpoint is not connected [2011-07-12 15:54:40.848071] E [rpc-clnt.c:338:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$ [2011-07-12 15:54:40.848106] I [client3_1-fops.c:683:client3_1_writev_cbk] 0-gluster_vol1-client-1: remote operation failed: Transport endpoint is not connected [2011-07-12 15:54:40.848199] W [rpc-clnt.c:1411:rpc_clnt_submit] 0-gluster_vol1-client-1: failed to submit rpc-request (XID: 0x263245x Program: GlusterFS 3.1, ProgVers: 310, Proc: 34) to rpc-transport (gluster_vol1-$ [2011-07-12 15:54:40.848265] E [rpc-clnt.c:338:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$ [2011-07-12 15:54:40.848283] I [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1: remote operation failed: Transport endpoint is not connected [2011-07-12 15:54:40.848320] E [rpc-clnt.c:338:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$ [2011-07-12 15:54:40.848338] I [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1: remote operation failed: Transport endpoint is not connected [2011-07-12 15:54:40.848435] E [rpc-clnt.c:338:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$ [2011-07-12 15:54:40.848466] I [client3_1-fops.c:683:client3_1_writev_cbk] 0-gluster_vol1-client-1: remote operation failed: Transport endpoint is not connected [2011-07-12 15:54:40.848578] W [rpc-clnt.c:1411:rpc_clnt_submit] 0-gluster_vol1-client-1: failed to submit rpc-request (XID: 0x263246x Program: GlusterFS 3.1, ProgVers: 310, Proc: 34) to rpc-transport (gluster_vol1-$ [2011-07-12 15:54:40.848630] E [rpc-clnt.c:338:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$ [2011-07-12 15:54:40.848657] I [client3_1-fops.c:683:client3_1_writev_cbk] 0-gluster_vol1-client-1: remote operation failed: Transport endpoint is not connected [2011-07-12 15:54:40.848738] W [rpc-clnt.c:1411:rpc_clnt_submit] 0-gluster_vol1-client-1: failed to submit rpc-request (XID: 0x263247x Program: GlusterFS 3.1, ProgVers: 310, Proc: 34) to rpc-transport (gluster_vol1-$ [2011-07-12 15:54:40.848808] E [rpc-clnt.c:338:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$ [2011-07-12 15:54:40.848820] W [rpc-clnt.c:1411:rpc_clnt_submit] 0-gluster_vol1-client-1: failed to submit rpc-request (XID: 0x263248x Program: GlusterFS 3.1, ProgVers: 310, Proc: 30) to rpc-transport (gluster_vol1-$ [2011-07-12 15:54:40.848848] I [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1: remote operation failed: Transport endpoint is not connected [2011-07-12 15:54:40.849145] I [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1: remote operation failed: Transport endpoint is not connected [2011-07-12 15:54:40.849261] E [rpc-clnt.c:338:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$ [2011-07-12 15:54:40.849304] I [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1: remote operation failed: Transport endpoint is not connected [2011-07-12 15:54:40.849403] E [rpc-clnt.c:338:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$ [2011-07-12 15:54:40.849429] I [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1: remote operation failed: Transport endpoint is not connected [2011-07-12 15:54:40.849513] E [rpc-clnt.c:338:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$ [2011-07-12 15:54:40.849545] I [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1: remote operation failed: Transport endpoint is not connected [2011-07-12 15:54:40.849605] E [rpc-clnt.c:338:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$ [2011-07-12 15:54:40.849613] W [rpc-clnt.c:1411:rpc_clnt_submit] 0-gluster_vol1-client-1: failed to submit rpc-request (XID: 0x263249x Program: GlusterFS 3.1, ProgVers: 310, Proc: 30) to rpc-transport (gluster_vol1-$ [2011-07-12 15:54:40.849678] I [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1: remote operation failed: Transport endpoint is not connected [2011-07-12 15:54:40.849649] I [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1: remote operation failed: Transport endpoint is not connected [2011-07-12 15:54:40.849938] E [rpc-clnt.c:338:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$ [2011-07-12 15:54:40.849974] I [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1: remote operation failed: Transport endpoint is not connected [2011-07-12 15:54:40.850083] E [rpc-clnt.c:338:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$ [2011-07-12 15:54:40.850114] W [rpc-clnt.c:1411:rpc_clnt_submit] 0-gluster_vol1-client-1: failed to submit rpc-request (XID: 0x263250x Program: GlusterFS 3.1, ProgVers: 310, Proc: 30) to rpc-transport (gluster_vol1-$ [2011-07-12 15:54:40.850151] I [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1: remote operation failed: Transport endpoint is not connected [2011-07-12 15:54:40.850116] I [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1: remote operation failed: Transport endpoint is not connected [2011-07-12 15:54:40.850285] E [rpc-clnt.c:338:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$ [2011-07-12 15:54:40.850324] W [client-handshake.c:264:client_ping_cbk] 0-gluster_vol1-client-1: timer must have expired [2011-07-12 15:54:40.850388] E [rpc-clnt.c:338:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$ [2011-07-12 15:54:40.850416] I [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1: remote operation failed: Transport endpoint is not connected [2011-07-12 15:54:40.850557] E [rpc-clnt.c:338:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$ [2011-07-12 15:54:40.850630] I [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1: remote operation failed: Transport endpoint is not connected [2011-07-12 15:54:40.850694] W [rpc-clnt.c:1411:rpc_clnt_submit] 0-gluster_vol1-client-1: failed to submit rpc-request (XID: 0x263251x Program: GlusterFS 3.1, ProgVers: 310, Proc: 30) to rpc-transport (gluster_vol1-$ [2011-07-12 15:54:40.850745] I [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1: remote operation failed: Transport endpoint is not connected [2011-07-12 15:54:40.850767] E [rpc-clnt.c:338:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$ [2011-07-12 15:54:40.850855] I [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1: remote operation failed: Transport endpoint is not connected [2011-07-12 15:54:40.850984] E [rpc-clnt.c:338:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$ [2011-07-12 15:54:40.851055] I [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1: remote operation failed: Transport endpoint is not connected [2011-07-12 15:54:40.851152] W [rpc-clnt.c:1411:rpc_clnt_submit] 0-gluster_vol1-client-1: failed to submit rpc-request (XID: 0x263252x Program: GlusterFS 3.1, ProgVers: 310, Proc: 30) to rpc-transport (gluster_vol1-$ [2011-07-12 15:54:40.851186] I [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1: remote operation failed: Transport endpoint is not connected [2011-07-12 15:54:40.851163] E [rpc-clnt.c:338:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$ [2011-07-12 15:54:40.851309] I [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1: remote operation failed: Transport endpoint is not connected [2011-07-12 15:54:40.851407] E [rpc-clnt.c:338:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$ [2011-07-12 15:54:40.851436] I [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1: remote operation failed: Transport endpoint is not connected [2011-07-12 15:54:40.851544] E [rpc-clnt.c:338:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$ [2011-07-12 15:54:40.851573] I [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1: remote operation failed: Transport endpoint is not connected [2011-07-12 15:54:40.851648] E [rpc-clnt.c:338:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$ [2011-07-12 15:54:40.851670] I [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1: remote operation failed: Transport endpoint is not connected [2011-07-12 15:54:40.851742] E [rpc-clnt.c:338:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$ [2011-07-12 15:54:40.851822] I [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1: remote operation failed: Transport endpoint is not connected [2011-07-12 15:54:40.851899] W [rpc-clnt.c:1411:rpc_clnt_submit] 0-gluster_vol1-client-1: failed to submit rpc-request (XID: 0x263253x Program: GlusterFS 3.1, ProgVers: 310, Proc: 30) to rpc-transport (gluster_vol1-$ [2011-07-12 15:54:40.851935] I [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1: remote operation failed: Transport endpoint is not connected [2011-07-12 15:54:40.851944] E [rpc-clnt.c:338:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$ [2011-07-12 15:54:40.852080] I [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1: remote operation failed: Transport endpoint is not connected [2011-07-12 15:54:40.852224] W [rpc-clnt.c:1411:rpc_clnt_submit] 0-gluster_vol1-client-1: failed to submit rpc-request (XID: 0x263254x Program: GlusterFS 3.1, ProgVers: 310, Proc: 30) to rpc-transport (gluster_vol1-$ [2011-07-12 15:54:40.852255] I [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1: remote operation failed: Transport endpoint is not connected [2011-07-12 15:54:40.852227] E [rpc-clnt.c:338:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$ [2011-07-12 15:54:40.852302] I [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1: remote operation failed: Transport endpoint is not connected [2011-07-12 15:54:40.852379] E [rpc-clnt.c:338:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$ [2011-07-12 15:54:40.852402] I [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1: remote operation failed: Transport endpoint is not connected [2011-07-12 15:54:40.852477] E [rpc-clnt.c:338:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$ [2011-07-12 15:54:40.852501] I [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1: remote operation failed: Transport endpoint is not connected [2011-07-12 15:54:40.852573] E [rpc-clnt.c:338:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$ [2011-07-12 15:54:40.852594] I [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1: remote operation failed: Transport endpoint is not connected [2011-07-12 15:54:40.852665] E [rpc-clnt.c:338:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$ [2011-07-12 15:54:40.852693] I [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1: remote operation failed: Transport endpoint is not connected [2011-07-12 15:54:40.852769] E [rpc-clnt.c:338:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$ [2011-07-12 15:54:40.852792] I [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1: remote operation failed: Transport endpoint is not connected [2011-07-12 15:54:40.852839] I [client.c:1883:client_rpc_notify] 0-gluster_vol1-client-1: disconnected [2011-07-12 15:54:40.852935] W [client3_1-fops.c:4777:client3_1_finodelk] 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD [2011-07-12 15:54:40.852972] W [client3_1-fops.c:4824:client3_1_finodelk] 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad state [2011-07-12 15:54:40.853082] W [client3_1-fops.c:4777:client3_1_finodelk] 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD [2011-07-12 15:54:40.853112] W [client3_1-fops.c:4824:client3_1_finodelk] 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad state [2011-07-12 15:54:40.853228] W [client3_1-fops.c:4777:client3_1_finodelk] 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD [2011-07-12 15:54:40.853262] W [client3_1-fops.c:4824:client3_1_finodelk] 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad state [2011-07-12 15:54:40.853324] W [client3_1-fops.c:4777:client3_1_finodelk] 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD [2011-07-12 15:54:40.853347] W [client3_1-fops.c:4824:client3_1_finodelk] 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad state [2011-07-12 15:54:40.853423] W [client3_1-fops.c:4777:client3_1_finodelk] 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD [2011-07-12 15:54:40.853460] W [client3_1-fops.c:4824:client3_1_finodelk] 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad state [2011-07-12 15:54:40.853509] W [client3_1-fops.c:4777:client3_1_finodelk] 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD [2011-07-12 15:54:40.853528] W [client3_1-fops.c:4824:client3_1_finodelk] 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad state [2011-07-12 15:54:40.853552] I [client-handshake.c:1082:select_server_supported_programs] 0-gluster_vol1-client-1: Using Program GlusterFS-3.1.0, Num (1298437), Version (310) [2011-07-12 15:54:40.853637] W [client3_1-fops.c:4777:client3_1_finodelk] 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD[2011-07-12 15:54:40.854121] I [client-handshake.c:913:client_setvolume_cbk] 0-gluster_vol1-client-1: Connected to 192.168.50.2:24010, attached to remote volume '/sp1/glusterfs'. [2011-07-12 15:54:40.854150] I [client-handshake.c:779:client_post_handshake] 0-gluster_vol1-client-1: 1 fds open - Delaying child_up until they are re-opened [2011-07-12 15:54:40.854228] W [client3_1-fops.c:4777:client3_1_finodelk] 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD [2011-07-12 15:54:40.854255] W [client3_1-fops.c:4824:client3_1_finodelk] 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad state [2011-07-12 15:54:40.854348] W [client3_1-fops.c:4777:client3_1_finodelk] 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD [2011-07-12 15:54:40.854373] W [client3_1-fops.c:4824:client3_1_finodelk] 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad state [2011-07-12 15:54:40.854473] W [client3_1-fops.c:4777:client3_1_finodelk] 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD [2011-07-12 15:54:40.854499] W [client3_1-fops.c:4824:client3_1_finodelk] 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad state [2011-07-12 15:54:40.854720] W [client3_1-fops.c:4777:client3_1_finodelk] 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD [2011-07-12 15:54:40.854750] W [client3_1-fops.c:4824:client3_1_finodelk] 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad state [2011-07-12 15:54:40.854868] W [client3_1-fops.c:4777:client3_1_finodelk] 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD [2011-07-12 15:54:40.854897] W [client3_1-fops.c:4824:client3_1_finodelk] 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad state [2011-07-12 15:54:40.855076] W [client3_1-fops.c:4777:client3_1_finodelk] 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD [2011-07-12 15:54:40.855107] W [client3_1-fops.c:4824:client3_1_finodelk] 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad state [2011-07-12 15:54:40.855241] W [client3_1-fops.c:4777:client3_1_finodelk] 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD [2011-07-12 15:54:40.855277] W [client3_1-fops.c:4824:client3_1_finodelk] 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad state [2011-07-12 15:54:40.855453] W [client3_1-fops.c:4777:client3_1_finodelk] 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD [2011-07-12 15:54:40.855488] W [client3_1-fops.c:4824:client3_1_finodelk] 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad state [2011-07-12 15:54:40.855665] W [client3_1-fops.c:4777:client3_1_finodelk] 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD [2011-07-12 15:54:40.855690] W [client3_1-fops.c:4824:client3_1_finodelk] 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad state [2011-07-12 15:54:40.855813] W [client3_1-fops.c:4777:client3_1_finodelk] 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD [2011-07-12 15:54:40.855842] W [client3_1-fops.c:4824:client3_1_finodelk] 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad state [2011-07-12 15:54:40.856037] W [client3_1-fops.c:4777:client3_1_finodelk] 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD [2011-07-12 15:54:40.856067] W [client3_1-fops.c:4824:client3_1_finodelk] 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad state [2011-07-12 15:54:40.856181] W [client3_1-fops.c:4777:client3_1_finodelk] 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD [2011-07-12 15:54:40.856208] W [client3_1-fops.c:4824:client3_1_finodelk] 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad state [2011-07-12 15:54:40.856307] W [client3_1-fops.c:4777:client3_1_finodelk] 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD [2011-07-12 15:54:40.856335] W [client3_1-fops.c:4824:client3_1_finodelk] 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad state [2011-07-12 15:54:40.856447] W [client3_1-fops.c:4777:client3_1_finodelk] 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD [2011-07-12 15:54:40.856474] W [client3_1-fops.c:4824:client3_1_finodelk] 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad state [2011-07-12 15:54:40.856579] W [client3_1-fops.c:4777:client3_1_finodelk] 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD [2011-07-12 15:54:40.856607] W [client3_1-fops.c:4824:client3_1_finodelk] 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad state [2011-07-12 15:54:40.857560] W [client3_1-fops.c:4777:client3_1_finodelk] 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD [2011-07-12 15:54:40.857590] W [client3_1-fops.c:4824:client3_1_finodelk] 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad state [2011-07-12 15:54:40.858438] W [client3_1-fops.c:4777:client3_1_finodelk] 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD [2011-07-12 15:54:40.858470] W [client3_1-fops.c:4824:client3_1_finodelk] 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad state [2011-07-12 15:54:40.858570] W [client3_1-fops.c:4777:client3_1_finodelk] 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD [2011-07-12 15:54:40.858599] W [client3_1-fops.c:4824:client3_1_finodelk] 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad state [2011-07-12 15:54:40.858830] W [client3_1-fops.c:4777:client3_1_finodelk] 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD [2011-07-12 15:54:40.858861] W [client3_1-fops.c:4824:client3_1_finodelk] 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad state [2011-07-12 15:54:40.859057] W [client3_1-fops.c:4777:client3_1_finodelk] 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD [2011-07-12 15:54:40.853659] W [client3_1-fops.c:4824:client3_1_finodelk] 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad state [2011-07-12 15:54:40.853702] W [client3_1-fops.c:4777:client3_1_finodelk] 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD [2011-07-12 15:54:40.853720] W [client3_1-fops.c:4824:client3_1_finodelk] 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad state [2011-07-12 15:54:40.853893] W [client3_1-fops.c:4777:client3_1_finodelk] 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD [2011-07-12 15:54:40.853912] W [client3_1-fops.c:4824:client3_1_finodelk] 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad state [2011-07-12 15:54:40.859086] W [client3_1-fops.c:4824:client3_1_finodelk] 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad state [2011-07-12 15:54:40.859198] W [client3_1-fops.c:4777:client3_1_finodelk] 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD [2011-07-12 15:54:40.859226] W [client3_1-fops.c:4824:client3_1_finodelk] 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad state [2011-07-12 15:54:40.860050] W [client3_1-fops.c:4777:client3_1_finodelk] 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD [2011-07-12 15:54:40.860084] W [client3_1-fops.c:4824:client3_1_finodelk] 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad state [2011-07-12 15:54:40.860196] W [client3_1-fops.c:4777:client3_1_finodelk] 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD [2011-07-12 15:54:40.860226] W [client3_1-fops.c:4824:client3_1_finodelk] 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad state [2011-07-12 15:54:40.861396] W [client3_1-fops.c:4777:client3_1_finodelk] 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD [2011-07-12 15:54:40.861428] W [client3_1-fops.c:4824:client3_1_finodelk] 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad state [2011-07-12 15:54:40.861516] W [client3_1-fops.c:4777:client3_1_finodelk] 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD [2011-07-12 15:54:40.861538] W [client3_1-fops.c:4824:client3_1_finodelk] 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad state [2011-07-12 15:54:40.861732] W [client3_1-fops.c:4777:client3_1_finodelk] 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD [2011-07-12 15:54:40.861754] W [client3_1-fops.c:4824:client3_1_finodelk] 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad state [2011-07-12 15:54:40.862049] W [client3_1-fops.c:4777:client3_1_finodelk] 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD [2011-07-12 15:54:40.862077] W [client3_1-fops.c:4824:client3_1_finodelk] 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad state [2011-07-12 15:54:40.862165] W [client3_1-fops.c:4777:client3_1_finodelk] 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD [2011-07-12 15:54:40.862187] W [client3_1-fops.c:4824:client3_1_finodelk] 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad state [2011-07-12 15:54:40.862334] W [client3_1-fops.c:4777:client3_1_finodelk] 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD [2011-07-12 15:54:40.862358] W [client3_1-fops.c:4824:client3_1_finodelk] 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad state [2011-07-12 15:54:40.862579] W [client3_1-fops.c:4777:client3_1_finodelk] 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD [2011-07-12 15:54:40.862602] W [client3_1-fops.c:4824:client3_1_finodelk] 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad state [2011-07-12 15:54:40.862874] W [client3_1-fops.c:4777:client3_1_finodelk] 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD [2011-07-12 15:54:40.862909] W [client3_1-fops.c:4824:client3_1_finodelk] 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad state [2011-07-12 15:54:41.21179] I [client-lk.c:617:decrement_reopen_fd_count] 0-gluster_vol1-client-1: last fd open'd/lock-self-heal'd - notifying CHILD-UP [2011-07-12 15:54:41.21567] W [afr-open.c:624:afr_openfd_flush] 0-gluster_vol1-replicate-0: fd not open on any subvolume 0x7f7ba7dd1024 (/bigfile2.img) [2011-07-12 15:54:41.22347] W [afr-open.c:624:afr_openfd_flush] 0-gluster_vol1-replicate-0: fd not open on any subvolume 0x7f7ba7dd1024 (/bigfile2.img) [2011-07-12 15:54:41.22831] W [afr-open.c:624:afr_openfd_flush] 0-gluster_vol1-replicate-0: fd not open on any subvolume 0x7f7ba7dd1024 (/bigfile2.img) [2011-07-12 15:54:41.23295] W [afr-open.c:624:afr_openfd_flush] 0-gluster_vol1-replicate-0: fd not open on any subvolume 0x7f7ba7dd1024 (/bigfile2.img) [2011-07-12 15:54:41.23775] W [afr-open.c:624:afr_openfd_flush] 0-gluster_vol1-replicate-0: fd not open on any subvolume 0x7f7ba7dd1024 (/bigfile2.img) [2011-07-12 15:54:41.24338] W [afr-open.c:624:afr_openfd_flush] 0-gluster_vol1-replicate-0: fd not open on any subvolume 0x7f7ba7dd1024 (/bigfile2.img) [2011-07-12 15:54:41.24781] W [afr-open.c:624:afr_openfd_flush] 0-gluster_vol1-replicate-0: fd not open on any subvolume 0x7f7ba7dd1024 (/bigfile2.img) [2011-07-12 15:54:41.25357] W [afr-open.c:624:afr_openfd_flush] 0-gluster_vol1-replicate-0: fd not open on any subvolume 0x7f7ba7dd1024 (/bigfile2.img) [2011-07-12 15:54:41.26039] W [afr-open.c:624:afr_openfd_flush] 0-gluster_vol1-replicate-0: fd not open on any subvolume 0x7f7ba7dd1024 (/bigfile2.img) [2011-07-12 15:54:41.26628] W [afr-open.c:624:afr_openfd_flush] 0-gluster_vol1-replicate-0: fd not open on any subvolume 0x7f7ba7dd1024 (/bigfile2.img) [2011-07-12 15:54:41.27038] W [afr-open.c:624:afr_openfd_flush] 0-gluster_vol1-replicate-0: fd not open on any subvolume 0x7f7ba7dd1024 (/bigfile2.img) [2011-07-12 15:54:41.27602] W [afr-open.c:624:afr_openfd_flush] 0-gluster_vol1-replicate-0: fd not open on any subvolume 0x7f7ba7dd1024 (/bigfile2.img) [2011-07-12 15:54:41.28089] W [afr-open.c:624:afr_openfd_flush] 0-gluster_vol1-replicate-0: fd not open on any subvolume 0x7f7ba7dd1024 (/bigfile2.img) [2011-07-12 15:54:41.28577] W [afr-open.c:624:afr_openfd_flush] 0-gluster_vol1-replicate-0: fd not open on any subvolume 0x7f7ba7dd1024 (/bigfile2.img) [2011-07-12 15:54:41.29033] W [afr-open.c:624:afr_openfd_flush] 0-gluster_vol1-replicate-0: fd not open on any subvolume 0x7f7ba7dd1024 (/bigfile2.img) [2011-07-12 15:54:41.29601] W [afr-open.c:624:afr_openfd_flush] 0-gluster_vol1-replicate-0: fd not open on any subvolume 0x7f7ba7dd1024 (/bigfile2.img) Has anyone an ideas? Help is appreciated Thanks ewuewu
Christopher Anderlik
2011-Jul-12 14:50 UTC
[Gluster-users] glusterfs 3.2.1 nfs hangs on opensuse 11.4
hi. I don't know if this is a similar issue - but try to compile glusterfs with debugging enabled: #export CFLAGS='-g3 -O0' http://gluster.org/pipermail/gluster-users/2011-January/006530.html cheers, christopher Am 12.07.2011 16:16, schrieb Uwe Weiss:> Hi list, > > I have some trouble with glusterfs 3.2.1. compiled from tarball in two > opensue 11.4 boxes. > Compiling an Installing of glusterfs went fine. > Afterwards I have configured a distributed replicated volume named > gluster_vol1 (peers are of course connected to each other) > As long as I only use the native gluster-client everything works fine an I > am reallay happy about glusterfs. > But if I try to use nfs I am running in trouble. > In details: I have two hosts called vmhost1 (192.168.50.1) and > vmhost2(192.168.50.2). On both hosts glusterfs is started. > On both hosts I have mounted a directory named /nfs_mount with: > mount -t nfs -o mountproto=tcp 192.168.50.1:/gluster_vol1 /nfs_mount > Afterward I can ls, df, du and so on with this directory on both hosts. Even > copying of small files works mostly but not every times. > > But if I try to create bigfiles with dd for example [], the connections > hangs after one or two minutes. Sometimes it helps to restart glusterfs on > the mounted node, sometimes only a reset of the master node solves the > problem. > This happens if I run the copying or the creation of bigfile on any node > that has mounted the gluster nfs share independently if I run this command > from a single node or concurrently from multiple nodes. > > Here is my logfile from my masternode after starting glusterfs on both > nodes: > > cat etc-glusterfs-glusterd.vol.log > [2011-07-12 15:19:08.455046] I [glusterd.c:564:init] 0-management: Using > /etc/glusterd as working directory > [2011-07-12 15:19:08.467129] E [rpc-transport.c:676:rpc_transport_load] > 0-rpc-transport: /usr/local/lib/glusterfs/3.2.1/rpc-transport/rdma.so: > cannot open shared object file: No such file or directory > [2011-07-12 15:19:08.467151] E [rpc-transport.c:680:rpc_transport_load] > 0-rpc-transport: volume 'rdma.management': transport-type 'rdma' is not > valid or not found on this machine > [2011-07-12 15:19:08.467164] W [rpcsvc.c:1288:rpcsvc_transport_create] > 0-rpc-service: cannot create listener, initing the transport failed > [2011-07-12 15:19:08.475337] I [glusterd.c:88:glusterd_uuid_init] > 0-glusterd: retrieved UUID: 358770a7-d1ee-4fbd-8552-b354a2295d79 > [2011-07-12 15:19:09.852048] E > [glusterd-store.c:1567:glusterd_store_retrieve_volume] 0-: Unknown key: > brick-0 > [2011-07-12 15:19:09.852093] E > [glusterd-store.c:1567:glusterd_store_retrieve_volume] 0-: Unknown key: > brick-1 > [2011-07-12 15:19:10.72382] I [glusterd-handler.c:3399:glusterd_friend_add] > 0-glusterd: connect returned 0 > [2011-07-12 15:19:10.74324] I > [glusterd-utils.c:1092:glusterd_volume_start_glusterfs] 0-: About to start > glusterfs for brick 192.168.50.1:/sp1/glusterfs > [2011-07-12 15:19:10.98987] I > [glusterd-utils.c:2217:glusterd_nfs_pmap_deregister] 0-: De-registered > MOUNTV3 successfully > [2011-07-12 15:19:10.100043] I > [glusterd-utils.c:2222:glusterd_nfs_pmap_deregister] 0-: De-registered > MOUNTV1 successfully > [2011-07-12 15:19:10.101123] I > [glusterd-utils.c:2227:glusterd_nfs_pmap_deregister] 0-: De-registered NFSV3 > successfully > Given volfile: > +--------------------------------------------------------------------------- > ---+ > 1: volume management > 2: type mgmt/glusterd > 3: option working-directory /etc/glusterd > 4: option transport-type socket,rdma > 5: option transport.socket.keepalive-time 10 > 6: option transport.socket.keepalive-interval 2 > 7: end-volume > 8: > > +--------------------------------------------------------------------------- > ---+ > [2011-07-12 15:19:10.115334] E [socket.c:1685:socket_connect_finish] > 0-management: connection to failed (Connection refused) > [2011-07-12 15:19:10.193987] I [glusterd-pmap.c:237:pmap_registry_bind] > 0-pmap: adding brick /sp1/glusterfs on port 24010 > [2011-07-12 15:19:10.224800] W [socket.c:1494:__socket_proto_state_machine] > 0-socket.management: reading from socket failed. Error (Transport endpoint > is not connected), peer (192.168.50.1:1023) > [2011-07-12 15:21:39.983189] W [socket.c:1494:__socket_proto_state_machine] > 0-socket.management: reading from socket failed. Error (Transport endpoint > is not connected), peer (192.168.50.2:1021) > [2011-07-12 15:21:40.318042] I > [glusterd-handshake.c:317:glusterd_set_clnt_mgmt_program] 0-: Using Program > glusterd clnt mgmt, Num (1238433), Version (1) > > Here is my nfs.log before mounting anything: > cat nfs.log > [2011-07-12 15:19:10.213621] I [nfs.c:675:init] 0-nfs: NFS service started > [2011-07-12 15:19:10.213743] W [write-behind.c:3029:init] > 0-gluster_vol1-write-behind: disabling write-behind for first 0 bytes > [2011-07-12 15:19:10.216610] I [client.c:1935:notify] > 0-gluster_vol1-client-0: parent translators are ready, attempting connect on > transport > [2011-07-12 15:19:10.220675] I [client.c:1935:notify] > 0-gluster_vol1-client-1: parent translators are ready, attempting connect on > transport > Given volfile: > +--------------------------------------------------------------------------- > ---+ > 1: volume gluster_vol1-client-0 > 2: type protocol/client > 3: option remote-host 192.168.50.1 > 4: option remote-subvolume /sp1/glusterfs > 5: option transport-type tcp > 6: end-volume > 7: > 8: volume gluster_vol1-client-1 > 9: type protocol/client > 10: option remote-host 192.168.50.2 > 11: option remote-subvolume /sp1/glusterfs > 12: option transport-type tcp > 13: end-volume > 14: > 15: volume gluster_vol1-replicate-0 > 16: type cluster/replicate > 17: subvolumes gluster_vol1-client-0 gluster_vol1-client-1 > 18: end-volume > 19: > 20: volume gluster_vol1-write-behind > 21: type performance/write-behind > 22: subvolumes gluster_vol1-replicate-0 > 23: end-volume > 24: > 25: volume gluster_vol1-read-ahead > 26: type performance/read-ahead > 27: subvolumes gluster_vol1-write-behind > 28: end-volume > 29: > 30: volume gluster_vol1-io-cache > 31: type performance/io-cache > 32: subvolumes gluster_vol1-read-ahead > 33: end-volume > 34: > 35: volume gluster_vol1-quick-read > 36: type performance/quick-read > 37: subvolumes gluster_vol1-io-cache > 38: end-volume > 39: > 40: volume gluster_vol1 > 41: type debug/io-stats > 42: option latency-measurement off > 43: option count-fop-hits off > 44: subvolumes gluster_vol1-quick-read > 45: end-volume > 46: > 47: volume nfs-server > 48: type nfs/server > 49: option nfs.dynamic-volumes on > 50: option rpc-auth.addr.gluster_vol1.allow * > 51: option nfs3.gluster_vol1.volume-id > 41ec51e6-e67a-45d4-bc80-be06c2453333 > 52: subvolumes gluster_vol1 > 53: end-volume > > +--------------------------------------------------------------------------- > ---+ > [2011-07-12 15:19:10.224263] E [socket.c:1685:socket_connect_finish] > 0-gluster_vol1-client-1: connection to failed (Connection refused) > [2011-07-12 15:19:10.224670] I [rpc-clnt.c:1531:rpc_clnt_reconfig] > 0-gluster_vol1-client-0: changing port to 24010 (from 0) > [2011-07-12 15:19:14.229314] I > [client-handshake.c:1082:select_server_supported_programs] > 0-gluster_vol1-client-0: Using Program GlusterFS-3.1.0, Num (1298437), > Version (310) > [2011-07-12 15:19:14.231655] I [client-handshake.c:913:client_setvolume_cbk] > 0-gluster_vol1-client-0: Connected to 192.168.50.1:24010, attached to remote > volume '/sp1/glusterfs'. > [2011-07-12 15:19:14.231690] I [afr-common.c:2591:afr_notify] > 0-gluster_vol1-replicate-0: Subvolume 'gluster_vol1-client-0' came back up; > going online. > [2011-07-12 15:19:14.232730] I [afr-common.c:902:afr_fresh_lookup_cbk] > 0-gluster_vol1-replicate-0: added root inode > [2011-07-12 15:21:39.850087] E > [client-handshake.c:1163:client_query_portmap_cbk] 0-gluster_vol1-client-1: > failed to get the port number for remote subvolume > [2011-07-12 15:21:39.850177] I [client.c:1883:client_rpc_notify] > 0-gluster_vol1-client-1: disconnected > [2011-07-12 15:21:43.473577] I [rpc-clnt.c:1531:rpc_clnt_reconfig] > 0-gluster_vol1-client-1: changing port to 24010 (from 0) > [2011-07-12 15:21:47.478306] I > [client-handshake.c:1082:select_server_supported_programs] > 0-gluster_vol1-client-1: Using Program GlusterFS-3.1.0, Num (1298437), > Version (310) > [2011-07-12 15:21:47.478804] I [client-handshake.c:913:client_setvolume_cbk] > 0-gluster_vol1-client-1: Connected to 192.168.50.2:24010, attached to remote > volume '/sp1/glusterfs'. > > This is the bricks log after starting glusterfs on both nodes > cat bricks/sp1-glusterfs.log > [2011-07-12 15:19:10.109639] W [socket.c:419:__socket_keepalive] 0-socket: > failed to set keep idle on socket 8 > [2011-07-12 15:19:10.109714] W [socket.c:1846:socket_server_event_handler] > 0-socket.glusterfsd: Failed to set keep-alive: Operation not supported > [2011-07-12 15:19:10.175912] W [graph.c:274:gf_add_cmdline_options] > 0-gluster_vol1-server: adding option 'listen-port' for volume > 'gluster_vol1-server' with value '24010' > [2011-07-12 15:19:10.176601] W [rpc-transport.c:447:validate_volume_options] > 0-tcp.gluster_vol1-server: option 'listen-port' is deprecated, preferred is > 'transport.socket.listen-port', continuing with correction > Given volfile: > +--------------------------------------------------------------------------- > ---+ > 1: volume gluster_vol1-posix > 2: type storage/posix > 3: option directory /sp1/glusterfs > 4: end-volume > 5: > 6: volume gluster_vol1-access-control > 7: type features/access-control > 8: subvolumes gluster_vol1-posix > 9: end-volume > 10: > 11: volume gluster_vol1-locks > 12: type features/locks > 13: subvolumes gluster_vol1-access-control > 14: end-volume > 15: > 16: volume gluster_vol1-io-threads > 17: type performance/io-threads > 18: subvolumes gluster_vol1-locks > 19: end-volume > 20: > 21: volume gluster_vol1-marker > 22: type features/marker > 23: option volume-uuid 41ec51e6-e67a-45d4-bc80-be06c2453333 > 24: option timestamp-file /etc/glusterd/vols/gluster_vol1/marker.tstamp > 25: option xtime off > 26: option quota off > 27: subvolumes gluster_vol1-io-threads > 28: end-volume > 29: > 30: volume /sp1/glusterfs > 31: type debug/io-stats > 32: option latency-measurement off > 33: option count-fop-hits off > 34: subvolumes gluster_vol1-marker > 35: end-volume > 36: > 37: volume gluster_vol1-server > 38: type protocol/server > 39: option transport-type tcp > 40: option auth.addr./sp1/glusterfs.allow * > 41: subvolumes /sp1/glusterfs > 42: end-volume > > +--------------------------------------------------------------------------- > ---+ > [2011-07-12 15:19:14.229546] I [server-handshake.c:534:server_setvolume] > 0-gluster_vol1-server: accepted client from 192.168.50.1:1022 > [2011-07-12 15:21:43.990367] I [server-handshake.c:534:server_setvolume] > 0-gluster_vol1-server: accepted client from 192.168.50.2:1019 > > > After a successful mount of vmhost1 from vmhost2 I see the following from > the mount command: > > > 192.168.50.1:/gluster_vol1 on /nfs_mount type nfs > (rw,relatime,vers=3,rsize=65536,wsize=65536,namlen=255,hard,proto=tcp,port=3 > 8467,timeo=600,retrans=2,sec=sys,mountaddr=192.168.50.1,mountvers=3,mountpor > t=38465,mountproto=tcp,local_lock=none,addr=192.168.50.1) > > Afterwards I tried to create a bigfile in /nfs_mount with: > > dd< /dev/zero> /nfs_mount/bigfile2.img bs=1M count=20480 > > The command runs a few seconds and than it hangs. > I found the following entrys in nfs log on the master masterserver > (vmhost1) > > [2011-07-12 15:54:40.840976] C > [client-handshake.c:121:rpc_client_ping_timer_expired] > 0-gluster_vol1-client-1: server 192.168.50.2:24010 has not responded in the > last 42 seconds, disconnecting. > [2011-07-12 15:54:40.841670] E [rpc-clnt.c:338:saved_frames_unwind] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$ > [2011-07-12 15:54:40.841839] I [client3_1-fops.c:683:client3_1_writev_cbk] > 0-gluster_vol1-client-1: remote operation failed: Transport endpoint is not > connected > [2011-07-12 15:54:40.846839] I [socket.c:2275:socket_submit_request] > 0-gluster_vol1-client-1: not connected (priv->connected = 0) > [2011-07-12 15:54:40.847046] W [rpc-clnt.c:1411:rpc_clnt_submit] > 0-gluster_vol1-client-1: failed to submit rpc-request (XID: 0x263239x > Program: GlusterFS 3.1, ProgVers: 310, Proc: 34) to rpc-transport > (gluster_vol1-$ > [2011-07-12 15:54:40.847091] W [rpc-clnt.c:1411:rpc_clnt_submit] > 0-gluster_vol1-client-1: failed to submit rpc-request (XID: 0x263240x > Program: GlusterFS 3.1, ProgVers: 310, Proc: 30) to rpc-transport > (gluster_vol1-$ > [2011-07-12 15:54:40.847130] E [rpc-clnt.c:338:saved_frames_unwind] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$ > [2011-07-12 15:54:40.847159] I > [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1: > remote operation failed: Transport endpoint is not connected > [2011-07-12 15:54:40.847193] I [client3_1-fops.c:683:client3_1_writev_cbk] > 0-gluster_vol1-client-1: remote operation failed: Transport endpoint is not > connected > [2011-07-12 15:54:40.847423] W [rpc-clnt.c:1411:rpc_clnt_submit] > 0-gluster_vol1-client-1: failed to submit rpc-request (XID: 0x263241x > Program: GlusterFS 3.1, ProgVers: 310, Proc: 34) to rpc-transport > (gluster_vol1-$ > [2011-07-12 15:54:40.847477] E [rpc-clnt.c:338:saved_frames_unwind] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$ > [2011-07-12 15:54:40.847495] I [client3_1-fops.c:683:client3_1_writev_cbk] > 0-gluster_vol1-client-1: remote operation failed: Transport endpoint is not > connected > [2011-07-12 15:54:40.847571] W [rpc-clnt.c:1411:rpc_clnt_submit] > 0-gluster_vol1-client-1: failed to submit rpc-request (XID: 0x263242x > Program: GlusterFS 3.1, ProgVers: 310, Proc: 34) to rpc-transport > (gluster_vol1-$ > [2011-07-12 15:54:40.847626] E [rpc-clnt.c:338:saved_frames_unwind] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$ > [2011-07-12 15:54:40.847643] I [client3_1-fops.c:683:client3_1_writev_cbk] > 0-gluster_vol1-client-1: remote operation failed: Transport endpoint is not > connected > [2011-07-12 15:54:40.847716] W [rpc-clnt.c:1411:rpc_clnt_submit] > 0-gluster_vol1-client-1: failed to submit rpc-request (XID: 0x263243x > Program: GlusterFS 3.1, ProgVers: 310, Proc: 34) to rpc-transport > (gluster_vol1-$ > [2011-07-12 15:54:40.847771] E [rpc-clnt.c:338:saved_frames_unwind] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$ > [2011-07-12 15:54:40.847798] I [client3_1-fops.c:683:client3_1_writev_cbk] > 0-gluster_vol1-client-1: remote operation failed: Transport endpoint is not > connected > [2011-07-12 15:54:40.847874] W [rpc-clnt.c:1411:rpc_clnt_submit] > 0-gluster_vol1-client-1: failed to submit rpc-request (XID: 0x263244x > Program: GlusterFS 3.1, ProgVers: 310, Proc: 34) to rpc-transport > (gluster_vol1-$ > [2011-07-12 15:54:40.847984] E [rpc-clnt.c:338:saved_frames_unwind] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$ > [2011-07-12 15:54:40.848006] I > [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1: > remote operation failed: Transport endpoint is not connected > [2011-07-12 15:54:40.848071] E [rpc-clnt.c:338:saved_frames_unwind] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$ > [2011-07-12 15:54:40.848106] I [client3_1-fops.c:683:client3_1_writev_cbk] > 0-gluster_vol1-client-1: remote operation failed: Transport endpoint is not > connected > [2011-07-12 15:54:40.848199] W [rpc-clnt.c:1411:rpc_clnt_submit] > 0-gluster_vol1-client-1: failed to submit rpc-request (XID: 0x263245x > Program: GlusterFS 3.1, ProgVers: 310, Proc: 34) to rpc-transport > (gluster_vol1-$ > [2011-07-12 15:54:40.848265] E [rpc-clnt.c:338:saved_frames_unwind] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$ > [2011-07-12 15:54:40.848283] I > [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1: > remote operation failed: Transport endpoint is not connected > [2011-07-12 15:54:40.848320] E [rpc-clnt.c:338:saved_frames_unwind] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$ > [2011-07-12 15:54:40.848338] I > [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1: > remote operation failed: Transport endpoint is not connected > [2011-07-12 15:54:40.848435] E [rpc-clnt.c:338:saved_frames_unwind] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$ > [2011-07-12 15:54:40.848466] I [client3_1-fops.c:683:client3_1_writev_cbk] > 0-gluster_vol1-client-1: remote operation failed: Transport endpoint is not > connected > [2011-07-12 15:54:40.848578] W [rpc-clnt.c:1411:rpc_clnt_submit] > 0-gluster_vol1-client-1: failed to submit rpc-request (XID: 0x263246x > Program: GlusterFS 3.1, ProgVers: 310, Proc: 34) to rpc-transport > (gluster_vol1-$ > [2011-07-12 15:54:40.848630] E [rpc-clnt.c:338:saved_frames_unwind] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$ > [2011-07-12 15:54:40.848657] I [client3_1-fops.c:683:client3_1_writev_cbk] > 0-gluster_vol1-client-1: remote operation failed: Transport endpoint is not > connected > [2011-07-12 15:54:40.848738] W [rpc-clnt.c:1411:rpc_clnt_submit] > 0-gluster_vol1-client-1: failed to submit rpc-request (XID: 0x263247x > Program: GlusterFS 3.1, ProgVers: 310, Proc: 34) to rpc-transport > (gluster_vol1-$ > [2011-07-12 15:54:40.848808] E [rpc-clnt.c:338:saved_frames_unwind] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$ > [2011-07-12 15:54:40.848820] W [rpc-clnt.c:1411:rpc_clnt_submit] > 0-gluster_vol1-client-1: failed to submit rpc-request (XID: 0x263248x > Program: GlusterFS 3.1, ProgVers: 310, Proc: 30) to rpc-transport > (gluster_vol1-$ > [2011-07-12 15:54:40.848848] I > [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1: > remote operation failed: Transport endpoint is not connected > [2011-07-12 15:54:40.849145] I > [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1: > remote operation failed: Transport endpoint is not connected > [2011-07-12 15:54:40.849261] E [rpc-clnt.c:338:saved_frames_unwind] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$ > [2011-07-12 15:54:40.849304] I > [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1: > remote operation failed: Transport endpoint is not connected > [2011-07-12 15:54:40.849403] E [rpc-clnt.c:338:saved_frames_unwind] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$ > [2011-07-12 15:54:40.849429] I > [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1: > remote operation failed: Transport endpoint is not connected > [2011-07-12 15:54:40.849513] E [rpc-clnt.c:338:saved_frames_unwind] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$ > [2011-07-12 15:54:40.849545] I > [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1: > remote operation failed: Transport endpoint is not connected > [2011-07-12 15:54:40.849605] E [rpc-clnt.c:338:saved_frames_unwind] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$ > [2011-07-12 15:54:40.849613] W [rpc-clnt.c:1411:rpc_clnt_submit] > 0-gluster_vol1-client-1: failed to submit rpc-request (XID: 0x263249x > Program: GlusterFS 3.1, ProgVers: 310, Proc: 30) to rpc-transport > (gluster_vol1-$ > [2011-07-12 15:54:40.849678] I > [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1: > remote operation failed: Transport endpoint is not connected > [2011-07-12 15:54:40.849649] I > [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1: > remote operation failed: Transport endpoint is not connected > [2011-07-12 15:54:40.849938] E [rpc-clnt.c:338:saved_frames_unwind] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$ > [2011-07-12 15:54:40.849974] I > [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1: > remote operation failed: Transport endpoint is not connected > [2011-07-12 15:54:40.850083] E [rpc-clnt.c:338:saved_frames_unwind] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$ > [2011-07-12 15:54:40.850114] W [rpc-clnt.c:1411:rpc_clnt_submit] > 0-gluster_vol1-client-1: failed to submit rpc-request (XID: 0x263250x > Program: GlusterFS 3.1, ProgVers: 310, Proc: 30) to rpc-transport > (gluster_vol1-$ > [2011-07-12 15:54:40.850151] I > [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1: > remote operation failed: Transport endpoint is not connected > [2011-07-12 15:54:40.850116] I > [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1: > remote operation failed: Transport endpoint is not connected > [2011-07-12 15:54:40.850285] E [rpc-clnt.c:338:saved_frames_unwind] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$ > [2011-07-12 15:54:40.850324] W [client-handshake.c:264:client_ping_cbk] > 0-gluster_vol1-client-1: timer must have expired > [2011-07-12 15:54:40.850388] E [rpc-clnt.c:338:saved_frames_unwind] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$ > [2011-07-12 15:54:40.850416] I > [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1: > remote operation failed: Transport endpoint is not connected > [2011-07-12 15:54:40.850557] E [rpc-clnt.c:338:saved_frames_unwind] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$ > [2011-07-12 15:54:40.850630] I > [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1: > remote operation failed: Transport endpoint is not connected > [2011-07-12 15:54:40.850694] W [rpc-clnt.c:1411:rpc_clnt_submit] > 0-gluster_vol1-client-1: failed to submit rpc-request (XID: 0x263251x > Program: GlusterFS 3.1, ProgVers: 310, Proc: 30) to rpc-transport > (gluster_vol1-$ > [2011-07-12 15:54:40.850745] I > [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1: > remote operation failed: Transport endpoint is not connected > [2011-07-12 15:54:40.850767] E [rpc-clnt.c:338:saved_frames_unwind] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$ > [2011-07-12 15:54:40.850855] I > [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1: > remote operation failed: Transport endpoint is not connected > [2011-07-12 15:54:40.850984] E [rpc-clnt.c:338:saved_frames_unwind] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$ > [2011-07-12 15:54:40.851055] I > [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1: > remote operation failed: Transport endpoint is not connected > [2011-07-12 15:54:40.851152] W [rpc-clnt.c:1411:rpc_clnt_submit] > 0-gluster_vol1-client-1: failed to submit rpc-request (XID: 0x263252x > Program: GlusterFS 3.1, ProgVers: 310, Proc: 30) to rpc-transport > (gluster_vol1-$ > [2011-07-12 15:54:40.851186] I > [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1: > remote operation failed: Transport endpoint is not connected > [2011-07-12 15:54:40.851163] E [rpc-clnt.c:338:saved_frames_unwind] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$ > [2011-07-12 15:54:40.851309] I > [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1: > remote operation failed: Transport endpoint is not connected > [2011-07-12 15:54:40.851407] E [rpc-clnt.c:338:saved_frames_unwind] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$ > [2011-07-12 15:54:40.851436] I > [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1: > remote operation failed: Transport endpoint is not connected > [2011-07-12 15:54:40.851544] E [rpc-clnt.c:338:saved_frames_unwind] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$ > [2011-07-12 15:54:40.851573] I > [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1: > remote operation failed: Transport endpoint is not connected > [2011-07-12 15:54:40.851648] E [rpc-clnt.c:338:saved_frames_unwind] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$ > [2011-07-12 15:54:40.851670] I > [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1: > remote operation failed: Transport endpoint is not connected > [2011-07-12 15:54:40.851742] E [rpc-clnt.c:338:saved_frames_unwind] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$ > [2011-07-12 15:54:40.851822] I > [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1: > remote operation failed: Transport endpoint is not connected > [2011-07-12 15:54:40.851899] W [rpc-clnt.c:1411:rpc_clnt_submit] > 0-gluster_vol1-client-1: failed to submit rpc-request (XID: 0x263253x > Program: GlusterFS 3.1, ProgVers: 310, Proc: 30) to rpc-transport > (gluster_vol1-$ > [2011-07-12 15:54:40.851935] I > [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1: > remote operation failed: Transport endpoint is not connected > [2011-07-12 15:54:40.851944] E [rpc-clnt.c:338:saved_frames_unwind] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$ > [2011-07-12 15:54:40.852080] I > [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1: > remote operation failed: Transport endpoint is not connected > [2011-07-12 15:54:40.852224] W [rpc-clnt.c:1411:rpc_clnt_submit] > 0-gluster_vol1-client-1: failed to submit rpc-request (XID: 0x263254x > Program: GlusterFS 3.1, ProgVers: 310, Proc: 30) to rpc-transport > (gluster_vol1-$ > [2011-07-12 15:54:40.852255] I > [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1: > remote operation failed: Transport endpoint is not connected > [2011-07-12 15:54:40.852227] E [rpc-clnt.c:338:saved_frames_unwind] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$ > [2011-07-12 15:54:40.852302] I > [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1: > remote operation failed: Transport endpoint is not connected > [2011-07-12 15:54:40.852379] E [rpc-clnt.c:338:saved_frames_unwind] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$ > [2011-07-12 15:54:40.852402] I > [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1: > remote operation failed: Transport endpoint is not connected > [2011-07-12 15:54:40.852477] E [rpc-clnt.c:338:saved_frames_unwind] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$ > [2011-07-12 15:54:40.852501] I > [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1: > remote operation failed: Transport endpoint is not connected > [2011-07-12 15:54:40.852573] E [rpc-clnt.c:338:saved_frames_unwind] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$ > [2011-07-12 15:54:40.852594] I > [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1: > remote operation failed: Transport endpoint is not connected > [2011-07-12 15:54:40.852665] E [rpc-clnt.c:338:saved_frames_unwind] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$ > [2011-07-12 15:54:40.852693] I > [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1: > remote operation failed: Transport endpoint is not connected > [2011-07-12 15:54:40.852769] E [rpc-clnt.c:338:saved_frames_unwind] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x6d) [0x7f7bac47abfd] > (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7d) [0x7f7ba$ > [2011-07-12 15:54:40.852792] I > [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-gluster_vol1-client-1: > remote operation failed: Transport endpoint is not connected > [2011-07-12 15:54:40.852839] I [client.c:1883:client_rpc_notify] > 0-gluster_vol1-client-1: disconnected > [2011-07-12 15:54:40.852935] W [client3_1-fops.c:4777:client3_1_finodelk] > 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD > [2011-07-12 15:54:40.852972] W [client3_1-fops.c:4824:client3_1_finodelk] > 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad > state > [2011-07-12 15:54:40.853082] W [client3_1-fops.c:4777:client3_1_finodelk] > 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD > [2011-07-12 15:54:40.853112] W [client3_1-fops.c:4824:client3_1_finodelk] > 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad > state > [2011-07-12 15:54:40.853228] W [client3_1-fops.c:4777:client3_1_finodelk] > 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD > [2011-07-12 15:54:40.853262] W [client3_1-fops.c:4824:client3_1_finodelk] > 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad > state > [2011-07-12 15:54:40.853324] W [client3_1-fops.c:4777:client3_1_finodelk] > 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD > [2011-07-12 15:54:40.853347] W [client3_1-fops.c:4824:client3_1_finodelk] > 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad > state > [2011-07-12 15:54:40.853423] W [client3_1-fops.c:4777:client3_1_finodelk] > 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD > [2011-07-12 15:54:40.853460] W [client3_1-fops.c:4824:client3_1_finodelk] > 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad > state > [2011-07-12 15:54:40.853509] W [client3_1-fops.c:4777:client3_1_finodelk] > 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD > [2011-07-12 15:54:40.853528] W [client3_1-fops.c:4824:client3_1_finodelk] > 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad > state > [2011-07-12 15:54:40.853552] I > [client-handshake.c:1082:select_server_supported_programs] > 0-gluster_vol1-client-1: Using Program GlusterFS-3.1.0, Num (1298437), > Version (310) > [2011-07-12 15:54:40.853637] W [client3_1-fops.c:4777:client3_1_finodelk] > 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. > EBADFD[2011-07-12 15:54:40.854121] I > [client-handshake.c:913:client_setvolume_cbk] 0-gluster_vol1-client-1: > Connected to 192.168.50.2:24010, attached to remote volume '/sp1/glusterfs'. > [2011-07-12 15:54:40.854150] I > [client-handshake.c:779:client_post_handshake] 0-gluster_vol1-client-1: 1 > fds open - Delaying child_up until they are re-opened > [2011-07-12 15:54:40.854228] W [client3_1-fops.c:4777:client3_1_finodelk] > 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD > [2011-07-12 15:54:40.854255] W [client3_1-fops.c:4824:client3_1_finodelk] > 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad > state > [2011-07-12 15:54:40.854348] W [client3_1-fops.c:4777:client3_1_finodelk] > 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD > [2011-07-12 15:54:40.854373] W [client3_1-fops.c:4824:client3_1_finodelk] > 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad > state > [2011-07-12 15:54:40.854473] W [client3_1-fops.c:4777:client3_1_finodelk] > 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD > [2011-07-12 15:54:40.854499] W [client3_1-fops.c:4824:client3_1_finodelk] > 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad > state > [2011-07-12 15:54:40.854720] W [client3_1-fops.c:4777:client3_1_finodelk] > 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD > [2011-07-12 15:54:40.854750] W [client3_1-fops.c:4824:client3_1_finodelk] > 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad > state > [2011-07-12 15:54:40.854868] W [client3_1-fops.c:4777:client3_1_finodelk] > 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD > [2011-07-12 15:54:40.854897] W [client3_1-fops.c:4824:client3_1_finodelk] > 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad > state > [2011-07-12 15:54:40.855076] W [client3_1-fops.c:4777:client3_1_finodelk] > 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD > [2011-07-12 15:54:40.855107] W [client3_1-fops.c:4824:client3_1_finodelk] > 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad > state > [2011-07-12 15:54:40.855241] W [client3_1-fops.c:4777:client3_1_finodelk] > 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD > [2011-07-12 15:54:40.855277] W [client3_1-fops.c:4824:client3_1_finodelk] > 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad > state > [2011-07-12 15:54:40.855453] W [client3_1-fops.c:4777:client3_1_finodelk] > 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD > [2011-07-12 15:54:40.855488] W [client3_1-fops.c:4824:client3_1_finodelk] > 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad > state > [2011-07-12 15:54:40.855665] W [client3_1-fops.c:4777:client3_1_finodelk] > 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD > [2011-07-12 15:54:40.855690] W [client3_1-fops.c:4824:client3_1_finodelk] > 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad > state > [2011-07-12 15:54:40.855813] W [client3_1-fops.c:4777:client3_1_finodelk] > 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD > [2011-07-12 15:54:40.855842] W [client3_1-fops.c:4824:client3_1_finodelk] > 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad > state > [2011-07-12 15:54:40.856037] W [client3_1-fops.c:4777:client3_1_finodelk] > 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD > [2011-07-12 15:54:40.856067] W [client3_1-fops.c:4824:client3_1_finodelk] > 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad > state > [2011-07-12 15:54:40.856181] W [client3_1-fops.c:4777:client3_1_finodelk] > 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD > [2011-07-12 15:54:40.856208] W [client3_1-fops.c:4824:client3_1_finodelk] > 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad > state > [2011-07-12 15:54:40.856307] W [client3_1-fops.c:4777:client3_1_finodelk] > 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD > [2011-07-12 15:54:40.856335] W [client3_1-fops.c:4824:client3_1_finodelk] > 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad > state > [2011-07-12 15:54:40.856447] W [client3_1-fops.c:4777:client3_1_finodelk] > 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD > [2011-07-12 15:54:40.856474] W [client3_1-fops.c:4824:client3_1_finodelk] > 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad > state > [2011-07-12 15:54:40.856579] W [client3_1-fops.c:4777:client3_1_finodelk] > 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD > [2011-07-12 15:54:40.856607] W [client3_1-fops.c:4824:client3_1_finodelk] > 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad > state > [2011-07-12 15:54:40.857560] W [client3_1-fops.c:4777:client3_1_finodelk] > 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD > [2011-07-12 15:54:40.857590] W [client3_1-fops.c:4824:client3_1_finodelk] > 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad > state > [2011-07-12 15:54:40.858438] W [client3_1-fops.c:4777:client3_1_finodelk] > 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD > [2011-07-12 15:54:40.858470] W [client3_1-fops.c:4824:client3_1_finodelk] > 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad > state > [2011-07-12 15:54:40.858570] W [client3_1-fops.c:4777:client3_1_finodelk] > 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD > [2011-07-12 15:54:40.858599] W [client3_1-fops.c:4824:client3_1_finodelk] > 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad > state > [2011-07-12 15:54:40.858830] W [client3_1-fops.c:4777:client3_1_finodelk] > 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD > [2011-07-12 15:54:40.858861] W [client3_1-fops.c:4824:client3_1_finodelk] > 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad > state > [2011-07-12 15:54:40.859057] W [client3_1-fops.c:4777:client3_1_finodelk] > 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD > [2011-07-12 15:54:40.853659] W [client3_1-fops.c:4824:client3_1_finodelk] > 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad > state > [2011-07-12 15:54:40.853702] W [client3_1-fops.c:4777:client3_1_finodelk] > 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD > [2011-07-12 15:54:40.853720] W [client3_1-fops.c:4824:client3_1_finodelk] > 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad > state > [2011-07-12 15:54:40.853893] W [client3_1-fops.c:4777:client3_1_finodelk] > 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD > [2011-07-12 15:54:40.853912] W [client3_1-fops.c:4824:client3_1_finodelk] > 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad > state > [2011-07-12 15:54:40.859086] W [client3_1-fops.c:4824:client3_1_finodelk] > 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad > state > [2011-07-12 15:54:40.859198] W [client3_1-fops.c:4777:client3_1_finodelk] > 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD > [2011-07-12 15:54:40.859226] W [client3_1-fops.c:4824:client3_1_finodelk] > 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad > state > [2011-07-12 15:54:40.860050] W [client3_1-fops.c:4777:client3_1_finodelk] > 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD > [2011-07-12 15:54:40.860084] W [client3_1-fops.c:4824:client3_1_finodelk] > 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad > state > [2011-07-12 15:54:40.860196] W [client3_1-fops.c:4777:client3_1_finodelk] > 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD > [2011-07-12 15:54:40.860226] W [client3_1-fops.c:4824:client3_1_finodelk] > 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad > state > [2011-07-12 15:54:40.861396] W [client3_1-fops.c:4777:client3_1_finodelk] > 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD > [2011-07-12 15:54:40.861428] W [client3_1-fops.c:4824:client3_1_finodelk] > 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad > state > [2011-07-12 15:54:40.861516] W [client3_1-fops.c:4777:client3_1_finodelk] > 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD > [2011-07-12 15:54:40.861538] W [client3_1-fops.c:4824:client3_1_finodelk] > 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad > state > [2011-07-12 15:54:40.861732] W [client3_1-fops.c:4777:client3_1_finodelk] > 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD > [2011-07-12 15:54:40.861754] W [client3_1-fops.c:4824:client3_1_finodelk] > 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad > state > [2011-07-12 15:54:40.862049] W [client3_1-fops.c:4777:client3_1_finodelk] > 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD > [2011-07-12 15:54:40.862077] W [client3_1-fops.c:4824:client3_1_finodelk] > 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad > state > [2011-07-12 15:54:40.862165] W [client3_1-fops.c:4777:client3_1_finodelk] > 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD > [2011-07-12 15:54:40.862187] W [client3_1-fops.c:4824:client3_1_finodelk] > 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad > state > [2011-07-12 15:54:40.862334] W [client3_1-fops.c:4777:client3_1_finodelk] > 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD > [2011-07-12 15:54:40.862358] W [client3_1-fops.c:4824:client3_1_finodelk] > 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad > state > [2011-07-12 15:54:40.862579] W [client3_1-fops.c:4777:client3_1_finodelk] > 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD > [2011-07-12 15:54:40.862602] W [client3_1-fops.c:4824:client3_1_finodelk] > 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad > state > [2011-07-12 15:54:40.862874] W [client3_1-fops.c:4777:client3_1_finodelk] > 0-gluster_vol1-client-1: (140509202): failed to get fd ctx. EBADFD > [2011-07-12 15:54:40.862909] W [client3_1-fops.c:4824:client3_1_finodelk] > 0-gluster_vol1-client-1: failed to send the fop: File descriptor in bad > state > [2011-07-12 15:54:41.21179] I [client-lk.c:617:decrement_reopen_fd_count] > 0-gluster_vol1-client-1: last fd open'd/lock-self-heal'd - notifying > CHILD-UP > [2011-07-12 15:54:41.21567] W [afr-open.c:624:afr_openfd_flush] > 0-gluster_vol1-replicate-0: fd not open on any subvolume 0x7f7ba7dd1024 > (/bigfile2.img) > [2011-07-12 15:54:41.22347] W [afr-open.c:624:afr_openfd_flush] > 0-gluster_vol1-replicate-0: fd not open on any subvolume 0x7f7ba7dd1024 > (/bigfile2.img) > [2011-07-12 15:54:41.22831] W [afr-open.c:624:afr_openfd_flush] > 0-gluster_vol1-replicate-0: fd not open on any subvolume 0x7f7ba7dd1024 > (/bigfile2.img) > [2011-07-12 15:54:41.23295] W [afr-open.c:624:afr_openfd_flush] > 0-gluster_vol1-replicate-0: fd not open on any subvolume 0x7f7ba7dd1024 > (/bigfile2.img) > [2011-07-12 15:54:41.23775] W [afr-open.c:624:afr_openfd_flush] > 0-gluster_vol1-replicate-0: fd not open on any subvolume 0x7f7ba7dd1024 > (/bigfile2.img) > [2011-07-12 15:54:41.24338] W [afr-open.c:624:afr_openfd_flush] > 0-gluster_vol1-replicate-0: fd not open on any subvolume 0x7f7ba7dd1024 > (/bigfile2.img) > [2011-07-12 15:54:41.24781] W [afr-open.c:624:afr_openfd_flush] > 0-gluster_vol1-replicate-0: fd not open on any subvolume 0x7f7ba7dd1024 > (/bigfile2.img) > [2011-07-12 15:54:41.25357] W [afr-open.c:624:afr_openfd_flush] > 0-gluster_vol1-replicate-0: fd not open on any subvolume 0x7f7ba7dd1024 > (/bigfile2.img) > [2011-07-12 15:54:41.26039] W [afr-open.c:624:afr_openfd_flush] > 0-gluster_vol1-replicate-0: fd not open on any subvolume 0x7f7ba7dd1024 > (/bigfile2.img) > [2011-07-12 15:54:41.26628] W [afr-open.c:624:afr_openfd_flush] > 0-gluster_vol1-replicate-0: fd not open on any subvolume 0x7f7ba7dd1024 > (/bigfile2.img) > [2011-07-12 15:54:41.27038] W [afr-open.c:624:afr_openfd_flush] > 0-gluster_vol1-replicate-0: fd not open on any subvolume 0x7f7ba7dd1024 > (/bigfile2.img) > [2011-07-12 15:54:41.27602] W [afr-open.c:624:afr_openfd_flush] > 0-gluster_vol1-replicate-0: fd not open on any subvolume 0x7f7ba7dd1024 > (/bigfile2.img) > [2011-07-12 15:54:41.28089] W [afr-open.c:624:afr_openfd_flush] > 0-gluster_vol1-replicate-0: fd not open on any subvolume 0x7f7ba7dd1024 > (/bigfile2.img) > [2011-07-12 15:54:41.28577] W [afr-open.c:624:afr_openfd_flush] > 0-gluster_vol1-replicate-0: fd not open on any subvolume 0x7f7ba7dd1024 > (/bigfile2.img) > [2011-07-12 15:54:41.29033] W [afr-open.c:624:afr_openfd_flush] > 0-gluster_vol1-replicate-0: fd not open on any subvolume 0x7f7ba7dd1024 > (/bigfile2.img) > [2011-07-12 15:54:41.29601] W [afr-open.c:624:afr_openfd_flush] > 0-gluster_vol1-replicate-0: fd not open on any subvolume 0x7f7ba7dd1024 > (/bigfile2.img) > > Has anyone an ideas? > Help is appreciated > > Thanks > ewuewu > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users >