Taylor Lewick
2015-Aug-13 18:47 UTC
[Gluster-users] After Centos 6 yum update client fails to mount glusterfs volume
Ok, Thank you, I had read online where gluster supported backwards compatibility, with respect to newer client versions working with older server versions. Thanks for the clarification? From: Atin Mukherjee [mailto:atin.mukherjee83 at gmail.com] Sent: Thursday, August 13, 2015 10:46 AM To: Taylor Lewick Cc: gluster-users at gluster.org Subject: Re: [Gluster-users] After Centos 6 yum update client fails to mount glusterfs volume -Atin Sent from one plus one On Aug 13, 2015 9:11 PM, "Taylor Lewick" <tlewick at adknowledge.com<mailto:tlewick at adknowledge.com>> wrote:> > As a follow up, I created a test gluster cluster, installed 3.6, and upgraded it to 3.7. I verified client machines running glusterfs 3.6 and 3.7 could mount the volume via glusterfs? > > > > So currently, clients running 3.7 can?t mount a gluster volume via glusterfs if the gluster servers are running 3.6.x. I didn?t test mounting the volume via NFS.I think I didn't understood your problem earlier. If you are trying to connect a client which is running higher version than the server then its expected as glusterd will reject the mount request. You should first upgrade your servers followed by the clients.> > When it fails it still returns an authentication error. > > > > Not sure if this is a bug or just requires specifically setting some authentication parameters on the volume/cluster. > > > > From: gluster-users-bounces at gluster.org<mailto:gluster-users-bounces at gluster.org> [mailto:gluster-users-bounces at gluster.org<mailto:gluster-users-bounces at gluster.org>] On Behalf Of Taylor Lewick > Sent: Wednesday, August 12, 2015 10:52 AM > To: gluster-users at gluster.org<mailto:gluster-users at gluster.org> > Subject: [Gluster-users] After Centos 6 yum update client fails to mount glusterfs volume > > > > I have an 8 node gluster cluster running 3.6.2. Yesterday one of our admin ran a yum update on his Centos 6.6 machine, and it updated the gluster client to 3.7.3. (Client update only on his machine, not the gluster cluster servers). > > After the update he was not able to mount the gluster volume. > > He then removed and reinstalled the 3.6 gluster client, and he was able to mount the volume again. > > > > So to test/confirm, I did the same thing. Prior to the running the upgrade I was running 3.6.4 glusterfs and glusterfs-fuse, and I had the mountpoint mounted on my test server. > > I ran the yum update glusterfs, it completed successfully, and I could access the mount. I then unmounted the gluster mount, and when I tried to mount it, and I received the following error: > > Mount failed. Please check the log file for more details. > > Since I could access the mountpoint, it appears something has happened in this update to change default behavior from a client mount request. Any ideas what changes I need to make to be able to mount the volume? > > With 3.7 version of client, do you have to specifically set auth.allow options for volumes? > > Below is the logfile from the client > > [2015-08-12 15:47:10.982346] I [MSGID: 100030] [glusterfsd.c:2301:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.7.3 (args: /usr/sbin/glusterfs --volfile-server=infarchive01 --volfile-id=/archives /media) > > [2015-08-12 15:47:11.029757] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 > > [2015-08-12 15:47:11.058367] I [MSGID: 114020] [client.c:2118:notify] 0-archives-client-0: parent translators are ready, attempting connect on transport > > [2015-08-12 15:47:11.058529] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2 > > [2015-08-12 15:47:11.065361] I [MSGID: 114020] [client.c:2118:notify] 0-archives-client-1: parent translators are ready, attempting connect on transport > > [2015-08-12 15:47:11.066220] I [rpc-clnt.c:1819:rpc_clnt_reconfig] 0-archives-client-0: changing port to 49152 (from 0) > > [2015-08-12 15:47:11.073949] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-archives-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) > > [2015-08-12 15:47:11.074260] I [MSGID: 114020] [client.c:2118:notify] 0-archives-client-2: parent translators are ready, attempting connect on transport > > [2015-08-12 15:47:11.075504] W [MSGID: 114043] [client-handshake.c:1114:client_setvolume_cbk] 0-archives-client-0: failed to set the volume [Permission denied] > > [2015-08-12 15:47:11.075671] W [MSGID: 114007] [client-handshake.c:1143:client_setvolume_cbk] 0-archives-client-0: failed to get 'process-uuid' from reply dict [Invalid argument] > > [2015-08-12 15:47:11.075761] E [MSGID: 114044] [client-handshake.c:1149:client_setvolume_cbk] 0-archives-client-0: SETVOLUME on remote-host failed [Permission denied] > > [2015-08-12 15:47:11.075801] I [MSGID: 114049] [client-handshake.c:1240:client_setvolume_cbk] 0-archives-client-0: sending AUTH_FAILED event > > [2015-08-12 15:47:11.075840] E [fuse-bridge.c:5130:notify] 0-fuse: Server authenication failed. Shutting down. > > [2015-08-12 15:47:11.075892] I [fuse-bridge.c:5595:fini] 0-fuse: Unmounting '/media'. > > [2015-08-12 15:47:11.082675] I [MSGID: 114020] [client.c:2118:notify] 0-archives-client-3: parent translators are ready, attempting connect on transport > > [2015-08-12 15:47:11.084889] W [glusterfsd.c:1219:cleanup_and_exit] (-->/lib64/libpthread.so.0() [0x3fe5a07a51] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xcd) [0x405e4d] -->/usr/sbin/glusterfs(cleanup_and_exit+0x65) [0x4059b5] ) 0-: received signum (15), shutting down > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org<mailto:Gluster-users at gluster.org> > http://www.gluster.org/mailman/listinfo/gluster-users-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150813/3a744b21/attachment.html>
Joe Julian
2015-Aug-13 20:23 UTC
[Gluster-users] After Centos 6 yum update client fails to mount glusterfs volume
"... a client which is running higher version than the server [...] glusterd will reject the mount request." is incorrect. You are correct in your understanding that the rpc is supposed to be cross-version compatible with newer features not being allowed as long as older versions are in the cluster as either servers or clients. The problem seems to have come from a policy shift between 3.7.2 and 3.7.3 where 3.7.3 uses unprivileged ports by default (there was supposed to be an email set to gluster-announce but I can't find that nor hardly any other announcements this year). To solve this, add "option rpc-auth-allow-insecure on" to "/etc/glusterfs/glusterd.vol" and set "server.allow-insecure on" for your volumes. On 08/13/2015 11:47 AM, Taylor Lewick wrote:> > Ok, Thank you, I had read online where gluster supported backwards > compatibility, with respect to newer client versions working with > older server versions. Thanks for the clarification? > > *From:*Atin Mukherjee [mailto:atin.mukherjee83 at gmail.com] > *Sent:* Thursday, August 13, 2015 10:46 AM > *To:* Taylor Lewick > *Cc:* gluster-users at gluster.org > *Subject:* Re: [Gluster-users] After Centos 6 yum update client fails > to mount glusterfs volume > > -Atin > Sent from one plus one > On Aug 13, 2015 9:11 PM, "Taylor Lewick" <tlewick at adknowledge.com > <mailto:tlewick at adknowledge.com>> wrote: > > > > As a follow up, I created a test gluster cluster, installed 3.6, and > upgraded it to 3.7. I verified client machines running glusterfs 3.6 > and 3.7 could mount the volume via glusterfs? > > > > > > > > So currently, clients running 3.7 can?t mount a gluster volume via > glusterfs if the gluster servers are running 3.6.x. I didn?t test > mounting the volume via NFS. > I think I didn't understood your problem earlier. If you are trying to > connect a client which is running higher version than the server then > its expected as glusterd will reject the mount request. You should > first upgrade your servers followed by the clients. > > > > When it fails it still returns an authentication error. > > > > > > > > Not sure if this is a bug or just requires specifically setting some > authentication parameters on the volume/cluster. > > > > > > > > From: gluster-users-bounces at gluster.org > <mailto:gluster-users-bounces at gluster.org> > [mailto:gluster-users-bounces at gluster.org > <mailto:gluster-users-bounces at gluster.org>] On Behalf Of Taylor Lewick > > Sent: Wednesday, August 12, 2015 10:52 AM > > To: gluster-users at gluster.org <mailto:gluster-users at gluster.org> > > Subject: [Gluster-users] After Centos 6 yum update client fails to > mount glusterfs volume > > > > > > > > I have an 8 node gluster cluster running 3.6.2. Yesterday one of our > admin ran a yum update on his Centos 6.6 machine, and it updated the > gluster client to 3.7.3. (Client update only on his machine, not the > gluster cluster servers). > > > > After the update he was not able to mount the gluster volume. > > > > He then removed and reinstalled the 3.6 gluster client, and he was > able to mount the volume again. > > > > > > > > So to test/confirm, I did the same thing. Prior to the running the > upgrade I was running 3.6.4 glusterfs and glusterfs-fuse, and I had > the mountpoint mounted on my test server. > > > > I ran the yum update glusterfs, it completed successfully, and I > could access the mount. I then unmounted the gluster mount, and when > I tried to mount it, and I received the following error: > > > > Mount failed. Please check the log file for more details. > > > > Since I could access the mountpoint, it appears something has > happened in this update to change default behavior from a client mount > request. Any ideas what changes I need to make to be able to mount > the volume? > > > > With 3.7 version of client, do you have to specifically set > auth.allow options for volumes? > > > > Below is the logfile from the client > > > > [2015-08-12 15:47:10.982346] I [MSGID: 100030] > [glusterfsd.c:2301:main] 0-/usr/sbin/glusterfs: Started running > /usr/sbin/glusterfs version 3.7.3 (args: /usr/sbin/glusterfs > --volfile-server=infarchive01 --volfile-id=/archives /media) > > > > [2015-08-12 15:47:11.029757] I [MSGID: 101190] > [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started > thread with index 1 > > > > [2015-08-12 15:47:11.058367] I [MSGID: 114020] > [client.c:2118:notify] 0-archives-client-0: parent translators are > ready, attempting connect on transport > > > > [2015-08-12 15:47:11.058529] I [MSGID: 101190] > [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started > thread with index 2 > > > > [2015-08-12 15:47:11.065361] I [MSGID: 114020] > [client.c:2118:notify] 0-archives-client-1: parent translators are > ready, attempting connect on transport > > > > [2015-08-12 15:47:11.066220] I [rpc-clnt.c:1819:rpc_clnt_reconfig] > 0-archives-client-0: changing port to 49152 (from 0) > > > > [2015-08-12 15:47:11.073949] I [MSGID: 114057] > [client-handshake.c:1437:select_server_supported_programs] > 0-archives-client-0: Using Program GlusterFS 3.3, Num (1298437), > Version (330) > > > > [2015-08-12 15:47:11.074260] I [MSGID: 114020] > [client.c:2118:notify] 0-archives-client-2: parent translators are > ready, attempting connect on transport > > > > [2015-08-12 15:47:11.075504] W [MSGID: 114043] > [client-handshake.c:1114:client_setvolume_cbk] 0-archives-client-0: > failed to set the volume [Permission denied] > > > > [2015-08-12 15:47:11.075671] W [MSGID: 114007] > [client-handshake.c:1143:client_setvolume_cbk] 0-archives-client-0: > failed to get 'process-uuid' from reply dict [Invalid argument] > > > > [2015-08-12 15:47:11.075761] E [MSGID: 114044] > [client-handshake.c:1149:client_setvolume_cbk] 0-archives-client-0: > SETVOLUME on remote-host failed [Permission denied] > > > > [2015-08-12 15:47:11.075801] I [MSGID: 114049] > [client-handshake.c:1240:client_setvolume_cbk] 0-archives-client-0: > sending AUTH_FAILED event > > > > [2015-08-12 15:47:11.075840] E [fuse-bridge.c:5130:notify] 0-fuse: > Server authenication failed. Shutting down. > > > > [2015-08-12 15:47:11.075892] I [fuse-bridge.c:5595:fini] 0-fuse: > Unmounting '/media'. > > > > [2015-08-12 15:47:11.082675] I [MSGID: 114020] > [client.c:2118:notify] 0-archives-client-3: parent translators are > ready, attempting connect on transport > > > > [2015-08-12 15:47:11.084889] W [glusterfsd.c:1219:cleanup_and_exit] > (-->/lib64/libpthread.so.0() [0x3fe5a07a51] > -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xcd) [0x405e4d] > -->/usr/sbin/glusterfs(cleanup_and_exit+0x65) [0x4059b5] ) 0-: > received signum (15), shutting down > > > > > > _______________________________________________ > > Gluster-users mailing list > > Gluster-users at gluster.org <mailto:Gluster-users at gluster.org> > > http://www.gluster.org/mailman/listinfo/gluster-users > > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150813/e1f4677b/attachment.html>