Were you automounting your glusterfs?
This is extremely similar to what I described previously:
<http://comments.gmane.org/gmane.comp.file-systems.gluster.user/9241>
In my case the servers were running SL6.2; clients were running
CentOS 5.7 and were automounting the glusterfs with IPoIB. When we
switched to hard mounts, the problem seems to have gone away, but I'd
be interested in seeing this resolved. The "Client lk-version numbers
are not same, reopening the fds" strings in the logs were also
identical, even when I compiled new client versions that matched the
server version.
If this does not have a trivial answer, I'll be happy to file an
official bug report. One report could be a twit :). second one could
be a bug.
hjm
On Fri, Jun 22, 2012 at 7:49 AM, Nathan Stratton <nathan at robotics.net>
wrote:>
> When I do a NFS mount and do a ls I get:
>
> [root at ovirt share]# ls
> ls: reading directory .: Too many levels of symbolic links
>
> [root at ovirt share]# ls -fl
> ls: reading directory .: Too many levels of symbolic links
> total 3636
> drwxr-xr-x ? 3 root root 16384 Jun 21 19:34 .
> dr-xr-xr-x. 21 root root ?4096 Jun 21 19:29 ..
> drwxr-xr-x ? 3 root root 16384 Jun 21 19:34 .
> dr-xr-xr-x. 21 root root ?4096 Jun 21 19:29 ..
> drwxr-xr-x ? 3 root root 16384 Jun 21 19:34 .
> dr-xr-xr-x. 21 root root ?4096 Jun 21 19:29 ..
> drwxr-xr-x ? 3 root root 16384 Jun 21 19:34 .
> dr-xr-xr-x. 21 root root ?4096 Jun 21 19:29 ..
> drwxr-xr-x ? 3 root root 16384 Jun 21 19:34 .
>
> {and so on}
>
> When I try a fuse mount I get:
> [client-handshake.c:1445:client_setvolume_cbk] 0-share-client-0: Server and
> Client lk-version numbers are not same, reopening the fds
>
> I have tried the fedora 16 RPMs and also built new fedora 17 RPMs.
>
> Full Log:
>
> [2012-06-21 19:24:35.633510] I [glusterfsd.c:1666:main]
> 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.3.0
> [2012-06-21 19:24:35.646832] I [io-cache.c:1549:check_cache_size_ok]
> 0-share-quick-read: Max cache size is 16825450496
> [2012-06-21 19:24:35.646916] I [io-cache.c:1549:check_cache_size_ok]
> 0-share-io-cache: Max cache size is 16825450496
> [2012-06-21 19:24:35.660807] I [client.c:2142:notify] 0-share-client-0:
> parent translators are ready, attempting connect on transport
> [2012-06-21 19:24:35.664026] I [client.c:2142:notify] 0-share-client-1:
> parent translators are ready, attempting connect on transport
> [2012-06-21 19:24:35.666801] I [client.c:2142:notify] 0-share-client-2:
> parent translators are ready, attempting connect on transport
> [2012-06-21 19:24:35.669385] I [client.c:2142:notify] 0-share-client-3:
> parent translators are ready, attempting connect on transport
> [2012-06-21 19:24:35.671951] I [client.c:2142:notify] 0-share-client-4:
> parent translators are ready, attempting connect on transport
> [2012-06-21 19:24:35.674514] I [client.c:2142:notify] 0-share-client-5:
> parent translators are ready, attempting connect on transport
> [2012-06-21 19:24:35.677093] I [client.c:2142:notify] 0-share-client-6:
> parent translators are ready, attempting connect on transport
> [2012-06-21 19:24:35.679652] I [client.c:2142:notify] 0-share-client-7:
> parent translators are ready, attempting connect on transport
> Given volfile:
>
+------------------------------------------------------------------------------+
> ?1: volume share-client-0
> ?2: ? ? type protocol/client
> ?3: ? ? option remote-host virt01.casabi.net
> ?4: ? ? option remote-subvolume /export
> ?5: ? ? option transport-type tcp
> ?6: ? ? option username f0a71b3a-1cb2-4a38-8620-f3853aa90798
> ?7: ? ? option password 9641cac2-de93-4621-b94b-9320ada20fd7
> ?8: end-volume
> ?9:
> ?10: volume share-client-1
> ?11: ? ? type protocol/client
> ?12: ? ? option remote-host virt02.casabi.net
> ?13: ? ? option remote-subvolume /export
> ?14: ? ? option transport-type tcp
> ?15: ? ? option username f0a71b3a-1cb2-4a38-8620-f3853aa90798
> ?16: ? ? option password 9641cac2-de93-4621-b94b-9320ada20fd7
> ?17: end-volume
> ?18:
> ?19: volume share-client-2
> ?20: ? ? type protocol/client
> ?21: ? ? option remote-host virt03.casabi.net
> ?22: ? ? option remote-subvolume /export
> ?23: ? ? option transport-type tcp
> ?24: ? ? option username f0a71b3a-1cb2-4a38-8620-f3853aa90798
> ?25: ? ? option password 9641cac2-de93-4621-b94b-9320ada20fd7
> ?26: end-volume
> ?27:
> ?28: volume share-client-3
> ?29: ? ? type protocol/client
> ?30: ? ? option remote-host virt04.casabi.net
> ?31: ? ? option remote-subvolume /export
> ?32: ? ? option transport-type tcp
> ?33: ? ? option username f0a71b3a-1cb2-4a38-8620-f3853aa90798
> ?34: ? ? option password 9641cac2-de93-4621-b94b-9320ada20fd7
> ?35: end-volume
> ?36:
> ?37: volume share-client-4
> ?38: ? ? type protocol/client
> ?39: ? ? option remote-host virt05.casabi.net
> ?40: ? ? option remote-subvolume /export
> ?41: ? ? option transport-type tcp
> ?42: ? ? option username f0a71b3a-1cb2-4a38-8620-f3853aa90798
> ?43: ? ? option password 9641cac2-de93-4621-b94b-9320ada20fd7
> ?44: end-volume
> ?45:
> ?46: volume share-client-5
> ?47: ? ? type protocol/client
> ?48: ? ? option remote-host virt06.casabi.net
> ?49: ? ? option remote-subvolume /export
> ?50: ? ? option transport-type tcp
> ?51: ? ? option username f0a71b3a-1cb2-4a38-8620-f3853aa90798
> ?52: ? ? option password 9641cac2-de93-4621-b94b-9320ada20fd7
> ?53: end-volume
> ?54:
> ?55: volume share-client-6
> ?56: ? ? type protocol/client
> ?57: ? ? option remote-host virt07.casabi.net
> ?58: ? ? option remote-subvolume /export
> ?59: ? ? option transport-type tcp
> ?60: ? ? option username f0a71b3a-1cb2-4a38-8620-f3853aa90798
> ?61: ? ? option password 9641cac2-de93-4621-b94b-9320ada20fd7
> ?62: end-volume
> ?63:
> ?64: volume share-client-7
> ?65: ? ? type protocol/client
> ?66: ? ? option remote-host virt08.casabi.net
> ?67: ? ? option remote-subvolume /export
> ?68: ? ? option transport-type tcp
> ?69: ? ? option username f0a71b3a-1cb2-4a38-8620-f3853aa90798
> ?70: ? ? option password 9641cac2-de93-4621-b94b-9320ada20fd7
> ?71: end-volume
> ?72:
> ?73: volume share-replicate-0
> ?74: ? ? type cluster/replicate
> ?75: ? ? subvolumes share-client-0 share-client-1
> ?76: end-volume
> ?77:
> ?78: volume share-replicate-1
> ?79: ? ? type cluster/replicate
> ?80: ? ? subvolumes share-client-2 share-client-3
> ?81: end-volume
> ?82:
> ?83: volume share-replicate-2
> ?84: ? ? type cluster/replicate
> ?85: ? ? subvolumes share-client-4 share-client-5
> ?86: end-volume
> ?87:
> ?88: volume share-replicate-3
> ?89: ? ? type cluster/replicate
> ?90: ? ? subvolumes share-client-6 share-client-7
> ?91: end-volume
> ?92:
> ?93: volume share-dht
> ?94: ? ? type cluster/distribute
> ?95: ? ? subvolumes share-replicate-0 share-replicate-1 share-replicate-2
> share-replicate-3
> ?96: end-volume
> ?97:
> ?98: volume share-write-behind
> ?99: ? ? type performance/write-behind
> 100: ? ? subvolumes share-dht
> 101: end-volume
> 102:
> 103: volume share-read-ahead
> 104: ? ? type performance/read-ahead
> 105: ? ? subvolumes share-write-behind
> 106: end-volume
> 107:
> 108: volume share-io-cache
> 109: ? ? type performance/io-cache
> 110: ? ? subvolumes share-read-ahead
> 111: end-volume
> 112:
> 113: volume share-quick-read
> 114: ? ? type performance/quick-read
> 115: ? ? subvolumes share-io-cache
> 116: end-volume
> 117:
> 118: volume share-md-cache
> 119: ? ? type performance/md-cache
> 120: ? ? subvolumes share-quick-read
> 121: end-volume
> 122:
> 123: volume share
> 124: ? ? type debug/io-stats
> 125: ? ? option latency-measurement off
> 126: ? ? option count-fop-hits off
> 127: ? ? subvolumes share-md-cache
> 128: end-volume
>
>
+------------------------------------------------------------------------------+
> [2012-06-21 19:24:35.682819] I [rpc-clnt.c:1660:rpc_clnt_reconfig]
> 0-share-client-0: changing port to 24009 (from 0)
> [2012-06-21 19:24:35.682973] I [rpc-clnt.c:1660:rpc_clnt_reconfig]
> 0-share-client-1: changing port to 24010 (from 0)
> [2012-06-21 19:24:35.683014] I [rpc-clnt.c:1660:rpc_clnt_reconfig]
> 0-share-client-2: changing port to 24010 (from 0)
> [2012-06-21 19:24:35.683039] I [rpc-clnt.c:1660:rpc_clnt_reconfig]
> 0-share-client-3: changing port to 24010 (from 0)
> [2012-06-21 19:24:35.683121] I [rpc-clnt.c:1660:rpc_clnt_reconfig]
> 0-share-client-5: changing port to 24010 (from 0)
> [2012-06-21 19:24:35.683162] I [rpc-clnt.c:1660:rpc_clnt_reconfig]
> 0-share-client-4: changing port to 24010 (from 0)
> [2012-06-21 19:24:35.683206] I [rpc-clnt.c:1660:rpc_clnt_reconfig]
> 0-share-client-7: changing port to 24010 (from 0)
> [2012-06-21 19:24:35.683246] I [rpc-clnt.c:1660:rpc_clnt_reconfig]
> 0-share-client-6: changing port to 24010 (from 0)
> [2012-06-21 19:24:39.644198] I
> [client-handshake.c:1636:select_server_supported_programs]
0-share-client-0:
> Using Program GlusterFS 3.3.0, Num (1298437), Version (330)
> [2012-06-21 19:24:39.644493] I
> [client-handshake.c:1433:client_setvolume_cbk] 0-share-client-0: Connected
> to 10.59.0.11:24009, attached to remote volume '/export'.
> [2012-06-21 19:24:39.644512] I
> [client-handshake.c:1445:client_setvolume_cbk] 0-share-client-0: Server and
> Client lk-version numbers are not same, reopening the fds
> [2012-06-21 19:24:39.644579] I [afr-common.c:3627:afr_notify]
> 0-share-replicate-0: Subvolume 'share-client-0' came back up; going
online.
> [2012-06-21 19:24:39.644621] I
> [client-handshake.c:453:client_set_lk_version_cbk] 0-share-client-0: Server
> lk version = 1
> [2012-06-21 19:24:39.646923] I
> [client-handshake.c:1636:select_server_supported_programs]
0-share-client-1:
> Using Program GlusterFS 3.3.0, Num (1298437), Version (330)
> [2012-06-21 19:24:39.647207] I
> [client-handshake.c:1433:client_setvolume_cbk] 0-share-client-1: Connected
> to 10.59.0.12:24010, attached to remote volume '/export'.
> [2012-06-21 19:24:39.647227] I
> [client-handshake.c:1445:client_setvolume_cbk] 0-share-client-1: Server and
> Client lk-version numbers are not same, reopening the fds
> [2012-06-21 19:24:39.647371] I
> [client-handshake.c:453:client_set_lk_version_cbk] 0-share-client-1: Server
> lk version = 1
> [2012-06-21 19:24:39.649587] I
> [client-handshake.c:1636:select_server_supported_programs]
0-share-client-2:
> Using Program GlusterFS 3.3.0, Num (1298437), Version (330)
> [2012-06-21 19:24:39.649806] I
> [client-handshake.c:1433:client_setvolume_cbk] 0-share-client-2: Connected
> to 10.59.0.13:24010, attached to remote volume '/export'.
> [2012-06-21 19:24:39.649826] I
> [client-handshake.c:1445:client_setvolume_cbk] 0-share-client-2: Server and
> Client lk-version numbers are not same, reopening the fds
> [2012-06-21 19:24:39.649865] I [afr-common.c:3627:afr_notify]
> 0-share-replicate-1: Subvolume 'share-client-2' came back up; going
online.
> [2012-06-21 19:24:39.649935] I
> [client-handshake.c:453:client_set_lk_version_cbk] 0-share-client-2: Server
> lk version = 1
> [2012-06-21 19:24:39.652192] I
> [client-handshake.c:1636:select_server_supported_programs]
0-share-client-3:
> Using Program GlusterFS 3.3.0, Num (1298437), Version (330)
> [2012-06-21 19:24:39.652428] I
> [client-handshake.c:1433:client_setvolume_cbk] 0-share-client-3: Connected
> to 10.59.0.14:24010, attached to remote volume '/export'.
> [2012-06-21 19:24:39.652448] I
> [client-handshake.c:1445:client_setvolume_cbk] 0-share-client-3: Server and
> Client lk-version numbers are not same, reopening the fds
> [2012-06-21 19:24:39.652566] I
> [client-handshake.c:453:client_set_lk_version_cbk] 0-share-client-3: Server
> lk version = 1
> [2012-06-21 19:24:39.654706] I
> [client-handshake.c:1636:select_server_supported_programs]
0-share-client-5:
> Using Program GlusterFS 3.3.0, Num (1298437), Version (330)
> [2012-06-21 19:24:39.654910] I
> [client-handshake.c:1433:client_setvolume_cbk] 0-share-client-5: Connected
> to 10.59.0.16:24010, attached to remote volume '/export'.
> [2012-06-21 19:24:39.654926] I
> [client-handshake.c:1445:client_setvolume_cbk] 0-share-client-5: Server and
> Client lk-version numbers are not same, reopening the fds
> [2012-06-21 19:24:39.654961] I [afr-common.c:3627:afr_notify]
> 0-share-replicate-2: Subvolume 'share-client-5' came back up; going
online.
> [2012-06-21 19:24:39.655035] I
> [client-handshake.c:453:client_set_lk_version_cbk] 0-share-client-5: Server
> lk version = 1
> [2012-06-21 19:24:39.657289] I
> [client-handshake.c:1636:select_server_supported_programs]
0-share-client-4:
> Using Program GlusterFS 3.3.0, Num (1298437), Version (330)
> [2012-06-21 19:24:39.657511] I
> [client-handshake.c:1433:client_setvolume_cbk] 0-share-client-4: Connected
> to 10.59.0.15:24010, attached to remote volume '/export'.
> [2012-06-21 19:24:39.657526] I
> [client-handshake.c:1445:client_setvolume_cbk] 0-share-client-4: Server and
> Client lk-version numbers are not same, reopening the fds
> [2012-06-21 19:24:39.657646] I
> [client-handshake.c:453:client_set_lk_version_cbk] 0-share-client-4: Server
> lk version = 1
> [2012-06-21 19:24:39.659884] I
> [client-handshake.c:1636:select_server_supported_programs]
0-share-client-7:
> Using Program GlusterFS 3.3.0, Num (1298437), Version (330)
> [2012-06-21 19:24:39.660075] I
> [client-handshake.c:1433:client_setvolume_cbk] 0-share-client-7: Connected
> to 10.59.0.18:24010, attached to remote volume '/export'.
> [2012-06-21 19:24:39.660091] I
> [client-handshake.c:1445:client_setvolume_cbk] 0-share-client-7: Server and
> Client lk-version numbers are not same, reopening the fds
> [2012-06-21 19:24:39.660127] I [afr-common.c:3627:afr_notify]
> 0-share-replicate-3: Subvolume 'share-client-7' came back up; going
online.
> [2012-06-21 19:24:39.660202] I
> [client-handshake.c:453:client_set_lk_version_cbk] 0-share-client-7: Server
> lk version = 1
> [2012-06-21 19:24:39.662579] I
> [client-handshake.c:1636:select_server_supported_programs]
0-share-client-6:
> Using Program GlusterFS 3.3.0, Num (1298437), Version (330)
> [2012-06-21 19:24:39.662908] I
> [client-handshake.c:1433:client_setvolume_cbk] 0-share-client-6: Connected
> to 10.59.0.17:24010, attached to remote volume '/export'.
> [2012-06-21 19:24:39.662931] I
> [client-handshake.c:1445:client_setvolume_cbk] 0-share-client-6: Server and
> Client lk-version numbers are not same, reopening the fds
> [2012-06-21 19:24:39.668028] I [fuse-bridge.c:4193:fuse_graph_setup]
0-fuse:
> switched to graph 0
> [2012-06-21 19:24:39.668234] I
> [client-handshake.c:453:client_set_lk_version_cbk] 0-share-client-6: Server
> lk version = 1
> [2012-06-21 19:24:39.668287] I [fuse-bridge.c:3376:fuse_init]
> 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.13 kernel
> 7.18
> [2012-06-21 19:24:39.668739] I
> [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 0-share-replicate-0:
> added root inode
> [2012-06-21 19:24:39.669232] I
> [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 0-share-replicate-1:
> added root inode
> [2012-06-21 19:24:39.669321] I
> [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 0-share-replicate-2:
> added root inode
> [2012-06-21 19:24:39.669379] I
> [afr-common.c:1964:afr_set_root_inode_on_first_lookup] 0-share-replicate-3:
> added root inode
>
>
>
>> <>
>
> Nathan Stratton
> nathan at robotics.net
> http://www.robotics.net
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users