Hi Diego, Just to clarify, so did you do an offline upgrade with an existing cluster (3.6.x => 3.10.x)? Thanks. On Fri, Aug 25, 2017 at 8:42 PM, Diego Remolina <dijuremo at gmail.com> wrote:> I was never able to go from 3.6.x to 3.7.x without downtime. Then > 3.7.x did not work well for me, so I stuck with 3.6.x until recently. > I went from 3.6.x to 3.10.x but downtime was scheduled. > > Diego > > On Fri, Aug 25, 2017 at 8:25 AM, Yong Tseng <yongtw123 at gmail.com> wrote: > > Hi Diego, > > > > Thanks for the information. I tried only setting 'allow-insecure on' but > > nada. > > The sentence "If you are using GlusterFS version 3.4.x or below, you can > > upgrade it to following" in documentation is surely misleading. > > So would you suggest creating a new 3.10 cluster from scratch then > rsync(?) > > the data from old cluster to the new? > > > > > > On Fri, Aug 25, 2017 at 7:53 PM, Diego Remolina <dijuremo at gmail.com> > wrote: > >> > >> You cannot do a rolling upgrade from 3.6.x to 3.10.x You will need > >> downtime. > >> > >> Even 3.6 to 3.7 was not possible... see some references to it below: > >> > >> https://marc.info/?l=gluster-users&m=145136214452772&w=2 > >> https://gluster.readthedocs.io/en/latest/release-notes/3.7.1/ > >> > >> # gluster volume set <volname> server.allow-insecure on Edit > >> /etc/glusterfs/glusterd.vol to contain this line: option > >> rpc-auth-allow-insecure on > >> > >> Post 1, restarting the volume would be necessary: > >> > >> # gluster volume stop <volname> > >> # gluster volume start <volname> > >> > >> > >> HTH, > >> > >> Diego > >> > >> On Fri, Aug 25, 2017 at 7:46 AM, Yong Tseng <yongtw123 at gmail.com> > wrote: > >> > Hi all, > >> > > >> > I'm currently in process of upgrading a replicated cluster (1 x 4) > from > >> > 3.6.3 to 3.10.5. The nodes run CentOS 6. However after upgrading the > >> > first > >> > node, the said node fails to connect to other peers (as seen via > >> > 'gluster > >> > peer status'), but somehow other non-upgraded peers can still see the > >> > upgraded peer as connected. > >> > > >> > Writes to the Gluster volume via local mounts of non-upgraded peers > are > >> > replicated to the upgraded peer, but I can't write via the upgraded > peer > >> > as > >> > the local mount seems forced to read-only. > >> > > >> > Launching heal operations from non-upgraded peers will output 'Commit > >> > failed > >> > on <upgraded peer IP>. Please check log for details'. > >> > > >> > In addition, during upgrade process there were warning messages about > my > >> > old > >> > vol files renamed with .rpmsave extension. I tried starting Gluster > with > >> > my > >> > old vol files but the problem persisted. I tried generating new vol > >> > files > >> > with 'glusterd --xlator-option "*.upgrade=on" -N', still no avail. > >> > > >> > Also I checked the brick log it had several messages about "failed to > >> > get > >> > client opversion". I don't know if this is pertinent. Could it be that > >> > the > >> > upgraded node cannot connect to older nodes but still can receive > >> > instructions from them? > >> > > >> > Below are command outputs; some data are masked. > >> > I'd provide more information if required. > >> > Thanks in advance. > >> > > >> > ===> 'gluster volume status' ran on non-upgraded peers > >> > > >> > Status of volume: gsnfs > >> > Gluster process Port Online > >> > Pid > >> > > >> > ------------------------------------------------------------ > ------------------ > >> > Brick gs-nfs01:/ftpdata 49154 Y > >> > 2931 > >> > Brick gs-nfs02:/ftpdata 49152 Y > >> > 29875 > >> > Brick gs-nfs03:/ftpdata 49153 Y > >> > 6987 > >> > Brick gs-nfs04:/ftpdata 49153 Y > >> > 24768 > >> > Self-heal Daemon on localhost N/A Y > >> > 2938 > >> > Self-heal Daemon on gs-nfs04 N/A Y > >> > 24788 > >> > Self-heal Daemon on gs-nfs03 N/A Y > >> > 7007 > >> > Self-heal Daemon on <IP> N/A Y 29866 > >> > > >> > Task Status of Volume gsnfs > >> > > >> > ------------------------------------------------------------ > ------------------ > >> > There are no active volume tasks > >> > > >> > > >> > > >> > ===> 'gluster volume status' on upgraded peer > >> > > >> > Gluster process TCP Port RDMA Port > Online > >> > Pid > >> > > >> > ------------------------------------------------------------ > ------------------ > >> > Brick gs-nfs02:/ftpdata 49152 0 Y > >> > 29875 > >> > Self-heal Daemon on localhost N/A N/A Y > >> > 29866 > >> > > >> > Task Status of Volume gsnfs > >> > > >> > ------------------------------------------------------------ > ------------------ > >> > There are no active volume tasks > >> > > >> > > >> > > >> > ===> 'gluster peer status' on non-upgraded peer > >> > > >> > Number of Peers: 3 > >> > > >> > Hostname: gs-nfs03 > >> > Uuid: 4c1544e6-550d-481a-95af-2a1da32d10ad > >> > State: Peer in Cluster (Connected) > >> > > >> > Hostname: <IP> > >> > Uuid: 17d554fd-9181-4b53-9521-55acf69ac35f > >> > State: Peer in Cluster (Connected) > >> > Other names: > >> > gs-nfs02 > >> > > >> > Hostname: gs-nfs04 > >> > Uuid: c6d165e6-d222-414c-b57a-97c64f06c5e9 > >> > State: Peer in Cluster (Connected) > >> > > >> > > >> > > >> > ===> 'gluster peer status' on upgraded peer > >> > > >> > Number of Peers: 3 > >> > > >> > Hostname: gs-nfs03 > >> > Uuid: 4c1544e6-550d-481a-95af-2a1da32d10ad > >> > State: Peer in Cluster (Disconnected) > >> > > >> > Hostname: gs-nfs01 > >> > Uuid: 90d3ed27-61ac-4ad3-93a9-3c2b68f41ecf > >> > State: Peer in Cluster (Disconnected) > >> > Other names: > >> > <IP> > >> > > >> > Hostname: gs-nfs04 > >> > Uuid: c6d165e6-d222-414c-b57a-97c64f06c5e9 > >> > State: Peer in Cluster (Disconnected) > >> > > >> > > >> > -- > >> > - Yong > >> > > >> > _______________________________________________ > >> > Gluster-users mailing list > >> > Gluster-users at gluster.org > >> > http://lists.gluster.org/mailman/listinfo/gluster-users > > > > > > > > > > -- > > - Yong >-- - Yong -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170825/5e7806e2/attachment.html>
Yes, I did an offline upgrade. 1. Stop all clients using gluster servers. 2. Stop glusterfsd and glusterd on both servers. 3. Backed up /var/lib/gluster* in all servers just to be safe. 4. Upgraded all servers from 3.6.x to 3.10.x (I did not have quotas or anything that required special steps) 5. Started gluster daemons again and confirmed everything was fine prior to letting clients connect. 5. Ran 3.10.x with the older op version for a few days to make sure all was OK (Not all was OK for me, but that may be a samba issue as I use it as a file server). 6. Upgraded the op version to maximum available. In my case, I have two servers with bricks and one server that acts as a witness. HTH, Diego On Fri, Aug 25, 2017 at 8:56 AM, Yong Tseng <yongtw123 at gmail.com> wrote:> Hi Diego, > > Just to clarify, so did you do an offline upgrade with an existing cluster > (3.6.x => 3.10.x)? > Thanks. > > On Fri, Aug 25, 2017 at 8:42 PM, Diego Remolina <dijuremo at gmail.com> wrote: >> >> I was never able to go from 3.6.x to 3.7.x without downtime. Then >> 3.7.x did not work well for me, so I stuck with 3.6.x until recently. >> I went from 3.6.x to 3.10.x but downtime was scheduled. >> >> Diego >> >> On Fri, Aug 25, 2017 at 8:25 AM, Yong Tseng <yongtw123 at gmail.com> wrote: >> > Hi Diego, >> > >> > Thanks for the information. I tried only setting 'allow-insecure on' but >> > nada. >> > The sentence "If you are using GlusterFS version 3.4.x or below, you can >> > upgrade it to following" in documentation is surely misleading. >> > So would you suggest creating a new 3.10 cluster from scratch then >> > rsync(?) >> > the data from old cluster to the new? >> > >> > >> > On Fri, Aug 25, 2017 at 7:53 PM, Diego Remolina <dijuremo at gmail.com> >> > wrote: >> >> >> >> You cannot do a rolling upgrade from 3.6.x to 3.10.x You will need >> >> downtime. >> >> >> >> Even 3.6 to 3.7 was not possible... see some references to it below: >> >> >> >> https://marc.info/?l=gluster-users&m=145136214452772&w=2 >> >> https://gluster.readthedocs.io/en/latest/release-notes/3.7.1/ >> >> >> >> # gluster volume set <volname> server.allow-insecure on Edit >> >> /etc/glusterfs/glusterd.vol to contain this line: option >> >> rpc-auth-allow-insecure on >> >> >> >> Post 1, restarting the volume would be necessary: >> >> >> >> # gluster volume stop <volname> >> >> # gluster volume start <volname> >> >> >> >> >> >> HTH, >> >> >> >> Diego >> >> >> >> On Fri, Aug 25, 2017 at 7:46 AM, Yong Tseng <yongtw123 at gmail.com> >> >> wrote: >> >> > Hi all, >> >> > >> >> > I'm currently in process of upgrading a replicated cluster (1 x 4) >> >> > from >> >> > 3.6.3 to 3.10.5. The nodes run CentOS 6. However after upgrading the >> >> > first >> >> > node, the said node fails to connect to other peers (as seen via >> >> > 'gluster >> >> > peer status'), but somehow other non-upgraded peers can still see the >> >> > upgraded peer as connected. >> >> > >> >> > Writes to the Gluster volume via local mounts of non-upgraded peers >> >> > are >> >> > replicated to the upgraded peer, but I can't write via the upgraded >> >> > peer >> >> > as >> >> > the local mount seems forced to read-only. >> >> > >> >> > Launching heal operations from non-upgraded peers will output 'Commit >> >> > failed >> >> > on <upgraded peer IP>. Please check log for details'. >> >> > >> >> > In addition, during upgrade process there were warning messages about >> >> > my >> >> > old >> >> > vol files renamed with .rpmsave extension. I tried starting Gluster >> >> > with >> >> > my >> >> > old vol files but the problem persisted. I tried generating new vol >> >> > files >> >> > with 'glusterd --xlator-option "*.upgrade=on" -N', still no avail. >> >> > >> >> > Also I checked the brick log it had several messages about "failed to >> >> > get >> >> > client opversion". I don't know if this is pertinent. Could it be >> >> > that >> >> > the >> >> > upgraded node cannot connect to older nodes but still can receive >> >> > instructions from them? >> >> > >> >> > Below are command outputs; some data are masked. >> >> > I'd provide more information if required. >> >> > Thanks in advance. >> >> > >> >> > ===> 'gluster volume status' ran on non-upgraded peers >> >> > >> >> > Status of volume: gsnfs >> >> > Gluster process Port >> >> > Online >> >> > Pid >> >> > >> >> > >> >> > ------------------------------------------------------------------------------ >> >> > Brick gs-nfs01:/ftpdata 49154 Y >> >> > 2931 >> >> > Brick gs-nfs02:/ftpdata 49152 Y >> >> > 29875 >> >> > Brick gs-nfs03:/ftpdata 49153 Y >> >> > 6987 >> >> > Brick gs-nfs04:/ftpdata 49153 Y >> >> > 24768 >> >> > Self-heal Daemon on localhost N/A Y >> >> > 2938 >> >> > Self-heal Daemon on gs-nfs04 N/A Y >> >> > 24788 >> >> > Self-heal Daemon on gs-nfs03 N/A Y >> >> > 7007 >> >> > Self-heal Daemon on <IP> N/A Y 29866 >> >> > >> >> > Task Status of Volume gsnfs >> >> > >> >> > >> >> > ------------------------------------------------------------------------------ >> >> > There are no active volume tasks >> >> > >> >> > >> >> > >> >> > ===> 'gluster volume status' on upgraded peer >> >> > >> >> > Gluster process TCP Port RDMA Port >> >> > Online >> >> > Pid >> >> > >> >> > >> >> > ------------------------------------------------------------------------------ >> >> > Brick gs-nfs02:/ftpdata 49152 0 Y >> >> > 29875 >> >> > Self-heal Daemon on localhost N/A N/A Y >> >> > 29866 >> >> > >> >> > Task Status of Volume gsnfs >> >> > >> >> > >> >> > ------------------------------------------------------------------------------ >> >> > There are no active volume tasks >> >> > >> >> > >> >> > >> >> > ===> 'gluster peer status' on non-upgraded peer >> >> > >> >> > Number of Peers: 3 >> >> > >> >> > Hostname: gs-nfs03 >> >> > Uuid: 4c1544e6-550d-481a-95af-2a1da32d10ad >> >> > State: Peer in Cluster (Connected) >> >> > >> >> > Hostname: <IP> >> >> > Uuid: 17d554fd-9181-4b53-9521-55acf69ac35f >> >> > State: Peer in Cluster (Connected) >> >> > Other names: >> >> > gs-nfs02 >> >> > >> >> > Hostname: gs-nfs04 >> >> > Uuid: c6d165e6-d222-414c-b57a-97c64f06c5e9 >> >> > State: Peer in Cluster (Connected) >> >> > >> >> > >> >> > >> >> > ===> 'gluster peer status' on upgraded peer >> >> > >> >> > Number of Peers: 3 >> >> > >> >> > Hostname: gs-nfs03 >> >> > Uuid: 4c1544e6-550d-481a-95af-2a1da32d10ad >> >> > State: Peer in Cluster (Disconnected) >> >> > >> >> > Hostname: gs-nfs01 >> >> > Uuid: 90d3ed27-61ac-4ad3-93a9-3c2b68f41ecf >> >> > State: Peer in Cluster (Disconnected) >> >> > Other names: >> >> > <IP> >> >> > >> >> > Hostname: gs-nfs04 >> >> > Uuid: c6d165e6-d222-414c-b57a-97c64f06c5e9 >> >> > State: Peer in Cluster (Disconnected) >> >> > >> >> > >> >> > -- >> >> > - Yong >> >> > >> >> > _______________________________________________ >> >> > Gluster-users mailing list >> >> > Gluster-users at gluster.org >> >> > http://lists.gluster.org/mailman/listinfo/gluster-users >> > >> > >> > >> > >> > -- >> > - Yong > > > > > -- > - Yong
Hi Diego, That's valuable information to know. Thanks for the input! On Fri, Aug 25, 2017 at 9:08 PM, Diego Remolina <dijuremo at gmail.com> wrote:> Yes, I did an offline upgrade. > > 1. Stop all clients using gluster servers. > 2. Stop glusterfsd and glusterd on both servers. > 3. Backed up /var/lib/gluster* in all servers just to be safe. > 4. Upgraded all servers from 3.6.x to 3.10.x (I did not have quotas or > anything that required special steps) > 5. Started gluster daemons again and confirmed everything was fine > prior to letting clients connect. > 5. Ran 3.10.x with the older op version for a few days to make sure > all was OK (Not all was OK for me, but that may be a samba issue as I > use it as a file server). > 6. Upgraded the op version to maximum available. > > In my case, I have two servers with bricks and one server that acts as > a witness. > > HTH, > > Diego > > On Fri, Aug 25, 2017 at 8:56 AM, Yong Tseng <yongtw123 at gmail.com> wrote: > > Hi Diego, > > > > Just to clarify, so did you do an offline upgrade with an existing > cluster > > (3.6.x => 3.10.x)? > > Thanks. > > > > On Fri, Aug 25, 2017 at 8:42 PM, Diego Remolina <dijuremo at gmail.com> > wrote: > >> > >> I was never able to go from 3.6.x to 3.7.x without downtime. Then > >> 3.7.x did not work well for me, so I stuck with 3.6.x until recently. > >> I went from 3.6.x to 3.10.x but downtime was scheduled. > >> > >> Diego > >> > >> On Fri, Aug 25, 2017 at 8:25 AM, Yong Tseng <yongtw123 at gmail.com> > wrote: > >> > Hi Diego, > >> > > >> > Thanks for the information. I tried only setting 'allow-insecure on' > but > >> > nada. > >> > The sentence "If you are using GlusterFS version 3.4.x or below, you > can > >> > upgrade it to following" in documentation is surely misleading. > >> > So would you suggest creating a new 3.10 cluster from scratch then > >> > rsync(?) > >> > the data from old cluster to the new? > >> > > >> > > >> > On Fri, Aug 25, 2017 at 7:53 PM, Diego Remolina <dijuremo at gmail.com> > >> > wrote: > >> >> > >> >> You cannot do a rolling upgrade from 3.6.x to 3.10.x You will need > >> >> downtime. > >> >> > >> >> Even 3.6 to 3.7 was not possible... see some references to it below: > >> >> > >> >> https://marc.info/?l=gluster-users&m=145136214452772&w=2 > >> >> https://gluster.readthedocs.io/en/latest/release-notes/3.7.1/ > >> >> > >> >> # gluster volume set <volname> server.allow-insecure on Edit > >> >> /etc/glusterfs/glusterd.vol to contain this line: option > >> >> rpc-auth-allow-insecure on > >> >> > >> >> Post 1, restarting the volume would be necessary: > >> >> > >> >> # gluster volume stop <volname> > >> >> # gluster volume start <volname> > >> >> > >> >> > >> >> HTH, > >> >> > >> >> Diego > >> >> > >> >> On Fri, Aug 25, 2017 at 7:46 AM, Yong Tseng <yongtw123 at gmail.com> > >> >> wrote: > >> >> > Hi all, > >> >> > > >> >> > I'm currently in process of upgrading a replicated cluster (1 x 4) > >> >> > from > >> >> > 3.6.3 to 3.10.5. The nodes run CentOS 6. However after upgrading > the > >> >> > first > >> >> > node, the said node fails to connect to other peers (as seen via > >> >> > 'gluster > >> >> > peer status'), but somehow other non-upgraded peers can still see > the > >> >> > upgraded peer as connected. > >> >> > > >> >> > Writes to the Gluster volume via local mounts of non-upgraded peers > >> >> > are > >> >> > replicated to the upgraded peer, but I can't write via the upgraded > >> >> > peer > >> >> > as > >> >> > the local mount seems forced to read-only. > >> >> > > >> >> > Launching heal operations from non-upgraded peers will output > 'Commit > >> >> > failed > >> >> > on <upgraded peer IP>. Please check log for details'. > >> >> > > >> >> > In addition, during upgrade process there were warning messages > about > >> >> > my > >> >> > old > >> >> > vol files renamed with .rpmsave extension. I tried starting Gluster > >> >> > with > >> >> > my > >> >> > old vol files but the problem persisted. I tried generating new vol > >> >> > files > >> >> > with 'glusterd --xlator-option "*.upgrade=on" -N', still no avail. > >> >> > > >> >> > Also I checked the brick log it had several messages about "failed > to > >> >> > get > >> >> > client opversion". I don't know if this is pertinent. Could it be > >> >> > that > >> >> > the > >> >> > upgraded node cannot connect to older nodes but still can receive > >> >> > instructions from them? > >> >> > > >> >> > Below are command outputs; some data are masked. > >> >> > I'd provide more information if required. > >> >> > Thanks in advance. > >> >> > > >> >> > ===> 'gluster volume status' ran on non-upgraded peers > >> >> > > >> >> > Status of volume: gsnfs > >> >> > Gluster process Port > >> >> > Online > >> >> > Pid > >> >> > > >> >> > > >> >> > ------------------------------------------------------------ > ------------------ > >> >> > Brick gs-nfs01:/ftpdata 49154 Y > >> >> > 2931 > >> >> > Brick gs-nfs02:/ftpdata 49152 Y > >> >> > 29875 > >> >> > Brick gs-nfs03:/ftpdata 49153 Y > >> >> > 6987 > >> >> > Brick gs-nfs04:/ftpdata 49153 Y > >> >> > 24768 > >> >> > Self-heal Daemon on localhost N/A Y > >> >> > 2938 > >> >> > Self-heal Daemon on gs-nfs04 N/A Y > >> >> > 24788 > >> >> > Self-heal Daemon on gs-nfs03 N/A Y > >> >> > 7007 > >> >> > Self-heal Daemon on <IP> N/A Y 29866 > >> >> > > >> >> > Task Status of Volume gsnfs > >> >> > > >> >> > > >> >> > ------------------------------------------------------------ > ------------------ > >> >> > There are no active volume tasks > >> >> > > >> >> > > >> >> > > >> >> > ===> 'gluster volume status' on upgraded peer > >> >> > > >> >> > Gluster process TCP Port RDMA Port > >> >> > Online > >> >> > Pid > >> >> > > >> >> > > >> >> > ------------------------------------------------------------ > ------------------ > >> >> > Brick gs-nfs02:/ftpdata 49152 0 Y > >> >> > 29875 > >> >> > Self-heal Daemon on localhost N/A N/A Y > >> >> > 29866 > >> >> > > >> >> > Task Status of Volume gsnfs > >> >> > > >> >> > > >> >> > ------------------------------------------------------------ > ------------------ > >> >> > There are no active volume tasks > >> >> > > >> >> > > >> >> > > >> >> > ===> 'gluster peer status' on non-upgraded peer > >> >> > > >> >> > Number of Peers: 3 > >> >> > > >> >> > Hostname: gs-nfs03 > >> >> > Uuid: 4c1544e6-550d-481a-95af-2a1da32d10ad > >> >> > State: Peer in Cluster (Connected) > >> >> > > >> >> > Hostname: <IP> > >> >> > Uuid: 17d554fd-9181-4b53-9521-55acf69ac35f > >> >> > State: Peer in Cluster (Connected) > >> >> > Other names: > >> >> > gs-nfs02 > >> >> > > >> >> > Hostname: gs-nfs04 > >> >> > Uuid: c6d165e6-d222-414c-b57a-97c64f06c5e9 > >> >> > State: Peer in Cluster (Connected) > >> >> > > >> >> > > >> >> > > >> >> > ===> 'gluster peer status' on upgraded peer > >> >> > > >> >> > Number of Peers: 3 > >> >> > > >> >> > Hostname: gs-nfs03 > >> >> > Uuid: 4c1544e6-550d-481a-95af-2a1da32d10ad > >> >> > State: Peer in Cluster (Disconnected) > >> >> > > >> >> > Hostname: gs-nfs01 > >> >> > Uuid: 90d3ed27-61ac-4ad3-93a9-3c2b68f41ecf > >> >> > State: Peer in Cluster (Disconnected) > >> >> > Other names: > >> >> > <IP> > >> >> > > >> >> > Hostname: gs-nfs04 > >> >> > Uuid: c6d165e6-d222-414c-b57a-97c64f06c5e9 > >> >> > State: Peer in Cluster (Disconnected) > >> >> > > >> >> > > >> >> > -- > >> >> > - Yong > >> >> > > >> >> > _______________________________________________ > >> >> > Gluster-users mailing list > >> >> > Gluster-users at gluster.org > >> >> > http://lists.gluster.org/mailman/listinfo/gluster-users > >> > > >> > > >> > > >> > > >> > -- > >> > - Yong > > > > > > > > > > -- > > - Yong >-- - Yong -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170825/ca6fbe5d/attachment.html>
Reasonably Related Threads
- Rolling upgrade from 3.6.3 to 3.10.5
- Rolling upgrade from 3.6.3 to 3.10.5
- Rolling upgrade from 3.6.3 to 3.10.5
- [SOLVED] [Nfs-ganesha-support] volume start: gv01: failed: Quorum not met. Volume operation not allowed.
- Is there a way to tell the sshd to ignore the security check on t he user's home permissions?