Hu Bert
2019-Mar-18 08:54 UTC
[Gluster-users] Upgrade 5.3 -> 5.4 on debian: public IP is used instead of LAN IP
Good morning :-) for debian the packages are there: https://download.gluster.org/pub/gluster/glusterfs/5/5.5/Debian/stretch/amd64/apt/pool/main/g/glusterfs/ I'll do an upgrade of a test installation 5.3 -> 5.5 and see if there are some errors etc. and report back. btw: no release notes for 5.4 and 5.5 so far? https://docs.gluster.org/en/latest/release-notes/ ? Am Fr., 15. M?rz 2019 um 14:28 Uhr schrieb Shyam Ranganathan <srangana at redhat.com>:> > We created a 5.5 release tag, and it is under packaging now. It should > be packaged and ready for testing early next week and should be released > close to mid-week next week. > > Thanks, > Shyam > On 3/13/19 12:34 PM, Artem Russakovskii wrote: > > Wednesday now with no update :-/ > > > > Sincerely, > > Artem > > > > -- > > Founder, Android Police <http://www.androidpolice.com>, APK Mirror > > <http://www.apkmirror.com/>, Illogical Robot LLC > > beerpla.net <http://beerpla.net/> | +ArtemRussakovskii > > <https://plus.google.com/+ArtemRussakovskii> | @ArtemR > > <http://twitter.com/ArtemR> > > > > > > On Tue, Mar 12, 2019 at 10:28 AM Artem Russakovskii <archon810 at gmail.com > > <mailto:archon810 at gmail.com>> wrote: > > > > Hi Amar, > > > > Any updates on this? I'm still not seeing it in OpenSUSE build > > repos. Maybe later today? > > > > Thanks. > > > > Sincerely, > > Artem > > > > -- > > Founder, Android Police <http://www.androidpolice.com>, APK Mirror > > <http://www.apkmirror.com/>, Illogical Robot LLC > > beerpla.net <http://beerpla.net/> | +ArtemRussakovskii > > <https://plus.google.com/+ArtemRussakovskii> | @ArtemR > > <http://twitter.com/ArtemR> > > > > > > On Wed, Mar 6, 2019 at 10:30 PM Amar Tumballi Suryanarayan > > <atumball at redhat.com <mailto:atumball at redhat.com>> wrote: > > > > We are talking days. Not weeks. Considering already it is > > Thursday here. 1 more day for tagging, and packaging. May be ok > > to expect it on Monday. > > > > -Amar > > > > On Thu, Mar 7, 2019 at 11:54 AM Artem Russakovskii > > <archon810 at gmail.com <mailto:archon810 at gmail.com>> wrote: > > > > Is the next release going to be an imminent hotfix, i.e. > > something like today/tomorrow, or are we talking weeks? > > > > Sincerely, > > Artem > > > > -- > > Founder, Android Police <http://www.androidpolice.com>, APK > > Mirror <http://www.apkmirror.com/>, Illogical Robot LLC > > beerpla.net <http://beerpla.net/> | +ArtemRussakovskii > > <https://plus.google.com/+ArtemRussakovskii> | @ArtemR > > <http://twitter.com/ArtemR> > > > > > > On Tue, Mar 5, 2019 at 11:09 AM Artem Russakovskii > > <archon810 at gmail.com <mailto:archon810 at gmail.com>> wrote: > > > > Ended up downgrading to 5.3 just in case. Peer status > > and volume status are OK now. > > > > zypper install --oldpackage glusterfs-5.3-lp150.100.1 > > Loading repository data... > > Reading installed packages... > > Resolving package dependencies... > > > > Problem: glusterfs-5.3-lp150.100.1.x86_64 requires > > libgfapi0 = 5.3, but this requirement cannot be provided > > not installable providers: > > libgfapi0-5.3-lp150.100.1.x86_64[glusterfs] > > Solution 1: Following actions will be done: > > downgrade of libgfapi0-5.4-lp150.100.1.x86_64 to > > libgfapi0-5.3-lp150.100.1.x86_64 > > downgrade of libgfchangelog0-5.4-lp150.100.1.x86_64 to > > libgfchangelog0-5.3-lp150.100.1.x86_64 > > downgrade of libgfrpc0-5.4-lp150.100.1.x86_64 to > > libgfrpc0-5.3-lp150.100.1.x86_64 > > downgrade of libgfxdr0-5.4-lp150.100.1.x86_64 to > > libgfxdr0-5.3-lp150.100.1.x86_64 > > downgrade of libglusterfs0-5.4-lp150.100.1.x86_64 to > > libglusterfs0-5.3-lp150.100.1.x86_64 > > Solution 2: do not install glusterfs-5.3-lp150.100.1.x86_64 > > Solution 3: break glusterfs-5.3-lp150.100.1.x86_64 by > > ignoring some of its dependencies > > > > Choose from above solutions by number or cancel > > [1/2/3/c] (c): 1 > > Resolving dependencies... > > Resolving package dependencies... > > > > The following 6 packages are going to be downgraded: > > glusterfs libgfapi0 libgfchangelog0 libgfrpc0 > > libgfxdr0 libglusterfs0 > > > > 6 packages to downgrade. > > > > Sincerely, > > Artem > > > > -- > > Founder, Android Police > > <http://www.androidpolice.com>, APK Mirror > > <http://www.apkmirror.com/>, Illogical Robot LLC > > beerpla.net <http://beerpla.net/> | +ArtemRussakovskii > > <https://plus.google.com/+ArtemRussakovskii> | @ArtemR > > <http://twitter.com/ArtemR> > > > > > > On Tue, Mar 5, 2019 at 10:57 AM Artem Russakovskii > > <archon810 at gmail.com <mailto:archon810 at gmail.com>> wrote: > > > > Noticed the same when upgrading from 5.3 to 5.4, as > > mentioned. > > > > I'm confused though. Is actual replication affected, > > because the 5.4 server and the 3x 5.3 servers still > > show heal info as all 4 connected, and the files > > seem to be replicating correctly as well. > > > > So what's actually affected - just the status > > command, or leaving 5.4 on one of the nodes is doing > > some damage to the underlying fs? Is it fixable by > > tweaking transport.socket.ssl-enabled? Does > > upgrading all servers to 5.4 resolve it, or should > > we revert back to 5.3? > > > > Sincerely, > > Artem > > > > -- > > Founder, Android Police > > <http://www.androidpolice.com>, APK Mirror > > <http://www.apkmirror.com/>, Illogical Robot LLC > > beerpla.net <http://beerpla.net/> | > > +ArtemRussakovskii > > <https://plus.google.com/+ArtemRussakovskii> > > | @ArtemR <http://twitter.com/ArtemR> > > > > > > On Tue, Mar 5, 2019 at 2:02 AM Hu Bert > > <revirii at googlemail.com > > <mailto:revirii at googlemail.com>> wrote: > > > > fyi: did a downgrade 5.4 -> 5.3 and it worked. > > all replicas are up and > > running. Awaiting updated v5.4. > > > > thx :-) > > > > Am Di., 5. M?rz 2019 um 09:26 Uhr schrieb Hari > > Gowtham <hgowtham at redhat.com > > <mailto:hgowtham at redhat.com>>: > > > > > > There are plans to revert the patch causing > > this error and rebuilt 5.4. > > > This should happen faster. the rebuilt 5.4 > > should be void of this upgrade issue. > > > > > > In the meantime, you can use 5.3 for this cluster. > > > Downgrading to 5.3 will work if it was just > > one node that was upgrade to 5.4 > > > and the other nodes are still in 5.3. > > > > > > On Tue, Mar 5, 2019 at 1:07 PM Hu Bert > > <revirii at googlemail.com > > <mailto:revirii at googlemail.com>> wrote: > > > > > > > > Hi Hari, > > > > > > > > thx for the hint. Do you know when this will > > be fixed? Is a downgrade > > > > 5.4 -> 5.3 a possibility to fix this? > > > > > > > > Hubert > > > > > > > > Am Di., 5. M?rz 2019 um 08:32 Uhr schrieb > > Hari Gowtham <hgowtham at redhat.com > > <mailto:hgowtham at redhat.com>>: > > > > > > > > > > Hi, > > > > > > > > > > This is a known issue we are working on. > > > > > As the checksum differs between the > > updated and non updated node, the > > > > > peers are getting rejected. > > > > > The bricks aren't coming because of the > > same issue. > > > > > > > > > > More about the issue: > > https://bugzilla.redhat.com/show_bug.cgi?id=1685120 > > > > > > > > > > On Tue, Mar 5, 2019 at 12:56 PM Hu Bert > > <revirii at googlemail.com > > <mailto:revirii at googlemail.com>> wrote: > > > > > > > > > > > > Interestingly: gluster volume status > > misses gluster1, while heal > > > > > > statistics show gluster1: > > > > > > > > > > > > gluster volume status workdata > > > > > > Status of volume: workdata > > > > > > Gluster process > > TCP Port RDMA Port Online Pid > > > > > > > > ------------------------------------------------------------------------------ > > > > > > Brick gluster2:/gluster/md4/workdata > > 49153 0 Y 1723 > > > > > > Brick gluster3:/gluster/md4/workdata > > 49153 0 Y 2068 > > > > > > Self-heal Daemon on localhost > > N/A N/A Y 1732 > > > > > > Self-heal Daemon on gluster3 > > N/A N/A Y 2077 > > > > > > > > > > > > vs. > > > > > > > > > > > > gluster volume heal workdata statistics > > heal-count > > > > > > Gathering count of entries to be healed > > on volume workdata has been successful > > > > > > > > > > > > Brick gluster1:/gluster/md4/workdata > > > > > > Number of entries: 0 > > > > > > > > > > > > Brick gluster2:/gluster/md4/workdata > > > > > > Number of entries: 10745 > > > > > > > > > > > > Brick gluster3:/gluster/md4/workdata > > > > > > Number of entries: 10744 > > > > > > > > > > > > Am Di., 5. M?rz 2019 um 08:18 Uhr > > schrieb Hu Bert <revirii at googlemail.com > > <mailto:revirii at googlemail.com>>: > > > > > > > > > > > > > > Hi Miling, > > > > > > > > > > > > > > well, there are such entries, but > > those haven't been a problem during > > > > > > > install and the last kernel > > update+reboot. The entries look like: > > > > > > > > > > > > > > PUBLIC_IP gluster2.alpserver.de > > <http://gluster2.alpserver.de> gluster2 > > > > > > > > > > > > > > 192.168.0.50 gluster1 > > > > > > > 192.168.0.51 gluster2 > > > > > > > 192.168.0.52 gluster3 > > > > > > > > > > > > > > 'ping gluster2' resolves to LAN IP; I > > removed the last entry in the > > > > > > > 1st line, did a reboot ... no, didn't > > help. From > > > > > > > /var/log/glusterfs/glusterd.log > > > > > > > on gluster 2: > > > > > > > > > > > > > > [2019-03-05 07:04:36.188128] E [MSGID: > > 106010] > > > > > > > > > [glusterd-utils.c:3483:glusterd_compare_friend_volume] > > 0-management: > > > > > > > Version of Cksums persistent differ. > > local cksum = 3950307018, remote > > > > > > > cksum = 455409345 on peer gluster1 > > > > > > > [2019-03-05 07:04:36.188314] I [MSGID: > > 106493] > > > > > > > > > [glusterd-handler.c:3843:glusterd_xfer_friend_add_resp] > > 0-glusterd: > > > > > > > Responded to gluster1 (0), ret: 0, > > op_ret: -1 > > > > > > > > > > > > > > Interestingly there are no entries in > > the brick logs of the rejected > > > > > > > server. Well, not surprising as no > > brick process is running. The > > > > > > > server gluster1 is still in rejected > > state. > > > > > > > > > > > > > > 'gluster volume start workdata force' > > starts the brick process on > > > > > > > gluster1, and some heals are happening > > on gluster2+3, but via 'gluster > > > > > > > volume status workdata' the volumes > > still aren't complete. > > > > > > > > > > > > > > gluster1: > > > > > > > > > ------------------------------------------------------------------------------ > > > > > > > Brick gluster1:/gluster/md4/workdata > > 49152 0 Y 2523 > > > > > > > Self-heal Daemon on localhost > > N/A N/A Y 2549 > > > > > > > > > > > > > > gluster2: > > > > > > > Gluster process > > TCP Port RDMA Port Online Pid > > > > > > > > > ------------------------------------------------------------------------------ > > > > > > > Brick gluster2:/gluster/md4/workdata > > 49153 0 Y 1723 > > > > > > > Brick gluster3:/gluster/md4/workdata > > 49153 0 Y 2068 > > > > > > > Self-heal Daemon on localhost > > N/A N/A Y 1732 > > > > > > > Self-heal Daemon on gluster3 > > N/A N/A Y 2077 > > > > > > > > > > > > > > > > > > > > > Hubert > > > > > > > > > > > > > > Am Di., 5. M?rz 2019 um 07:58 Uhr > > schrieb Milind Changire <mchangir at redhat.com > > <mailto:mchangir at redhat.com>>: > > > > > > > > > > > > > > > > There are probably DNS entries or > > /etc/hosts entries with the public IP Addresses > > that the host names (gluster1, gluster2, > > gluster3) are getting resolved to. > > > > > > > > /etc/resolv.conf would tell which is > > the default domain searched for the node names > > and the DNS servers which respond to the queries. > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Mar 5, 2019 at 12:14 PM Hu > > Bert <revirii at googlemail.com > > <mailto:revirii at googlemail.com>> wrote: > > > > > > > >> > > > > > > > >> Good morning, > > > > > > > >> > > > > > > > >> i have a replicate 3 setup with 2 > > volumes, running on version 5.3 on > > > > > > > >> debian stretch. This morning i > > upgraded one server to version 5.4 and > > > > > > > >> rebooted the machine; after the > > restart i noticed that: > > > > > > > >> > > > > > > > >> - no brick process is running > > > > > > > >> - gluster volume status only shows > > the server itself: > > > > > > > >> gluster volume status workdata > > > > > > > >> Status of volume: workdata > > > > > > > >> Gluster process > > TCP Port RDMA Port Online Pid > > > > > > > >> > > ------------------------------------------------------------------------------ > > > > > > > >> Brick > > gluster1:/gluster/md4/workdata N/A > > N/A N N/A > > > > > > > >> NFS Server on localhost > > N/A N/A N N/A > > > > > > > >> > > > > > > > >> - gluster peer status on the server > > > > > > > >> gluster peer status > > > > > > > >> Number of Peers: 2 > > > > > > > >> > > > > > > > >> Hostname: gluster3 > > > > > > > >> Uuid: > > c7b4a448-ca6a-4051-877f-788f9ee9bc4a > > > > > > > >> State: Peer Rejected (Connected) > > > > > > > >> > > > > > > > >> Hostname: gluster2 > > > > > > > >> Uuid: > > 162fea82-406a-4f51-81a3-e90235d8da27 > > > > > > > >> State: Peer Rejected (Connected) > > > > > > > >> > > > > > > > >> - gluster peer status on the other > > 2 servers: > > > > > > > >> gluster peer status > > > > > > > >> Number of Peers: 2 > > > > > > > >> > > > > > > > >> Hostname: gluster1 > > > > > > > >> Uuid: > > 9a360776-7b58-49ae-831e-a0ce4e4afbef > > > > > > > >> State: Peer Rejected (Connected) > > > > > > > >> > > > > > > > >> Hostname: gluster3 > > > > > > > >> Uuid: > > c7b4a448-ca6a-4051-877f-788f9ee9bc4a > > > > > > > >> State: Peer in Cluster (Connected) > > > > > > > >> > > > > > > > >> I noticed that, in the brick logs, > > i see that the public IP is used > > > > > > > >> instead of the LAN IP. brick logs > > from one of the volumes: > > > > > > > >> > > > > > > > >> rejected node: > > https://pastebin.com/qkpj10Sd > > > > > > > >> connected nodes: > > https://pastebin.com/8SxVVYFV > > > > > > > >> > > > > > > > >> Why is the public IP suddenly used > > instead of the LAN IP? Killing all > > > > > > > >> gluster processes and rebooting > > (again) didn't help. > > > > > > > >> > > > > > > > >> > > > > > > > >> Thx, > > > > > > > >> Hubert > > > > > > > >> > > _______________________________________________ > > > > > > > >> Gluster-users mailing list > > > > > > > >> Gluster-users at gluster.org > > <mailto:Gluster-users at gluster.org> > > > > > > > >> > > https://lists.gluster.org/mailman/listinfo/gluster-users > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > Milind > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > Gluster-users mailing list > > > > > > Gluster-users at gluster.org > > <mailto:Gluster-users at gluster.org> > > > > > > > > https://lists.gluster.org/mailman/listinfo/gluster-users > > > > > > > > > > > > > > > > > > > > -- > > > > > Regards, > > > > > Hari Gowtham. > > > > > > > > > > > > -- > > > Regards, > > > Hari Gowtham. > > _______________________________________________ > > Gluster-users mailing list > > Gluster-users at gluster.org > > <mailto:Gluster-users at gluster.org> > > https://lists.gluster.org/mailman/listinfo/gluster-users > > > > _______________________________________________ > > Gluster-users mailing list > > Gluster-users at gluster.org <mailto:Gluster-users at gluster.org> > > https://lists.gluster.org/mailman/listinfo/gluster-users > > > > > > > > -- > > Amar Tumballi (amarts) > > > > > > _______________________________________________ > > Gluster-users mailing list > > Gluster-users at gluster.org > > https://lists.gluster.org/mailman/listinfo/gluster-users > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users
Hu Bert
2019-Mar-18 10:23 UTC
[Gluster-users] Upgrade 5.3 -> 5.4 on debian: public IP is used instead of LAN IP
update: upgrade from 5.3 -> 5.5 in a replicate 3 test setup with 2 volumes done. In 'gluster peer status' the peers stay connected during the upgrade, no 'peer rejected' messages. No cksum mismatches in the logs. Looks good :-) Am Mo., 18. M?rz 2019 um 09:54 Uhr schrieb Hu Bert <revirii at googlemail.com>:> > Good morning :-) > > for debian the packages are there: > https://download.gluster.org/pub/gluster/glusterfs/5/5.5/Debian/stretch/amd64/apt/pool/main/g/glusterfs/ > > I'll do an upgrade of a test installation 5.3 -> 5.5 and see if there > are some errors etc. and report back. > > btw: no release notes for 5.4 and 5.5 so far? > https://docs.gluster.org/en/latest/release-notes/ ? > > Am Fr., 15. M?rz 2019 um 14:28 Uhr schrieb Shyam Ranganathan > <srangana at redhat.com>: > > > > We created a 5.5 release tag, and it is under packaging now. It should > > be packaged and ready for testing early next week and should be released > > close to mid-week next week. > > > > Thanks, > > Shyam > > On 3/13/19 12:34 PM, Artem Russakovskii wrote: > > > Wednesday now with no update :-/ > > > > > > Sincerely, > > > Artem > > > > > > -- > > > Founder, Android Police <http://www.androidpolice.com>, APK Mirror > > > <http://www.apkmirror.com/>, Illogical Robot LLC > > > beerpla.net <http://beerpla.net/> | +ArtemRussakovskii > > > <https://plus.google.com/+ArtemRussakovskii> | @ArtemR > > > <http://twitter.com/ArtemR> > > > > > > > > > On Tue, Mar 12, 2019 at 10:28 AM Artem Russakovskii <archon810 at gmail.com > > > <mailto:archon810 at gmail.com>> wrote: > > > > > > Hi Amar, > > > > > > Any updates on this? I'm still not seeing it in OpenSUSE build > > > repos. Maybe later today? > > > > > > Thanks. > > > > > > Sincerely, > > > Artem > > > > > > -- > > > Founder, Android Police <http://www.androidpolice.com>, APK Mirror > > > <http://www.apkmirror.com/>, Illogical Robot LLC > > > beerpla.net <http://beerpla.net/> | +ArtemRussakovskii > > > <https://plus.google.com/+ArtemRussakovskii> | @ArtemR > > > <http://twitter.com/ArtemR> > > > > > > > > > On Wed, Mar 6, 2019 at 10:30 PM Amar Tumballi Suryanarayan > > > <atumball at redhat.com <mailto:atumball at redhat.com>> wrote: > > > > > > We are talking days. Not weeks. Considering already it is > > > Thursday here. 1 more day for tagging, and packaging. May be ok > > > to expect it on Monday. > > > > > > -Amar > > > > > > On Thu, Mar 7, 2019 at 11:54 AM Artem Russakovskii > > > <archon810 at gmail.com <mailto:archon810 at gmail.com>> wrote: > > > > > > Is the next release going to be an imminent hotfix, i.e. > > > something like today/tomorrow, or are we talking weeks? > > > > > > Sincerely, > > > Artem > > > > > > -- > > > Founder, Android Police <http://www.androidpolice.com>, APK > > > Mirror <http://www.apkmirror.com/>, Illogical Robot LLC > > > beerpla.net <http://beerpla.net/> | +ArtemRussakovskii > > > <https://plus.google.com/+ArtemRussakovskii> | @ArtemR > > > <http://twitter.com/ArtemR> > > > > > > > > > On Tue, Mar 5, 2019 at 11:09 AM Artem Russakovskii > > > <archon810 at gmail.com <mailto:archon810 at gmail.com>> wrote: > > > > > > Ended up downgrading to 5.3 just in case. Peer status > > > and volume status are OK now. > > > > > > zypper install --oldpackage glusterfs-5.3-lp150.100.1 > > > Loading repository data... > > > Reading installed packages... > > > Resolving package dependencies... > > > > > > Problem: glusterfs-5.3-lp150.100.1.x86_64 requires > > > libgfapi0 = 5.3, but this requirement cannot be provided > > > not installable providers: > > > libgfapi0-5.3-lp150.100.1.x86_64[glusterfs] > > > Solution 1: Following actions will be done: > > > downgrade of libgfapi0-5.4-lp150.100.1.x86_64 to > > > libgfapi0-5.3-lp150.100.1.x86_64 > > > downgrade of libgfchangelog0-5.4-lp150.100.1.x86_64 to > > > libgfchangelog0-5.3-lp150.100.1.x86_64 > > > downgrade of libgfrpc0-5.4-lp150.100.1.x86_64 to > > > libgfrpc0-5.3-lp150.100.1.x86_64 > > > downgrade of libgfxdr0-5.4-lp150.100.1.x86_64 to > > > libgfxdr0-5.3-lp150.100.1.x86_64 > > > downgrade of libglusterfs0-5.4-lp150.100.1.x86_64 to > > > libglusterfs0-5.3-lp150.100.1.x86_64 > > > Solution 2: do not install glusterfs-5.3-lp150.100.1.x86_64 > > > Solution 3: break glusterfs-5.3-lp150.100.1.x86_64 by > > > ignoring some of its dependencies > > > > > > Choose from above solutions by number or cancel > > > [1/2/3/c] (c): 1 > > > Resolving dependencies... > > > Resolving package dependencies... > > > > > > The following 6 packages are going to be downgraded: > > > glusterfs libgfapi0 libgfchangelog0 libgfrpc0 > > > libgfxdr0 libglusterfs0 > > > > > > 6 packages to downgrade. > > > > > > Sincerely, > > > Artem > > > > > > -- > > > Founder, Android Police > > > <http://www.androidpolice.com>, APK Mirror > > > <http://www.apkmirror.com/>, Illogical Robot LLC > > > beerpla.net <http://beerpla.net/> | +ArtemRussakovskii > > > <https://plus.google.com/+ArtemRussakovskii> | @ArtemR > > > <http://twitter.com/ArtemR> > > > > > > > > > On Tue, Mar 5, 2019 at 10:57 AM Artem Russakovskii > > > <archon810 at gmail.com <mailto:archon810 at gmail.com>> wrote: > > > > > > Noticed the same when upgrading from 5.3 to 5.4, as > > > mentioned. > > > > > > I'm confused though. Is actual replication affected, > > > because the 5.4 server and the 3x 5.3 servers still > > > show heal info as all 4 connected, and the files > > > seem to be replicating correctly as well. > > > > > > So what's actually affected - just the status > > > command, or leaving 5.4 on one of the nodes is doing > > > some damage to the underlying fs? Is it fixable by > > > tweaking transport.socket.ssl-enabled? Does > > > upgrading all servers to 5.4 resolve it, or should > > > we revert back to 5.3? > > > > > > Sincerely, > > > Artem > > > > > > -- > > > Founder, Android Police > > > <http://www.androidpolice.com>, APK Mirror > > > <http://www.apkmirror.com/>, Illogical Robot LLC > > > beerpla.net <http://beerpla.net/> | > > > +ArtemRussakovskii > > > <https://plus.google.com/+ArtemRussakovskii> > > > | @ArtemR <http://twitter.com/ArtemR> > > > > > > > > > On Tue, Mar 5, 2019 at 2:02 AM Hu Bert > > > <revirii at googlemail.com > > > <mailto:revirii at googlemail.com>> wrote: > > > > > > fyi: did a downgrade 5.4 -> 5.3 and it worked. > > > all replicas are up and > > > running. Awaiting updated v5.4. > > > > > > thx :-) > > > > > > Am Di., 5. M?rz 2019 um 09:26 Uhr schrieb Hari > > > Gowtham <hgowtham at redhat.com > > > <mailto:hgowtham at redhat.com>>: > > > > > > > > There are plans to revert the patch causing > > > this error and rebuilt 5.4. > > > > This should happen faster. the rebuilt 5.4 > > > should be void of this upgrade issue. > > > > > > > > In the meantime, you can use 5.3 for this cluster. > > > > Downgrading to 5.3 will work if it was just > > > one node that was upgrade to 5.4 > > > > and the other nodes are still in 5.3. > > > > > > > > On Tue, Mar 5, 2019 at 1:07 PM Hu Bert > > > <revirii at googlemail.com > > > <mailto:revirii at googlemail.com>> wrote: > > > > > > > > > > Hi Hari, > > > > > > > > > > thx for the hint. Do you know when this will > > > be fixed? Is a downgrade > > > > > 5.4 -> 5.3 a possibility to fix this? > > > > > > > > > > Hubert > > > > > > > > > > Am Di., 5. M?rz 2019 um 08:32 Uhr schrieb > > > Hari Gowtham <hgowtham at redhat.com > > > <mailto:hgowtham at redhat.com>>: > > > > > > > > > > > > Hi, > > > > > > > > > > > > This is a known issue we are working on. > > > > > > As the checksum differs between the > > > updated and non updated node, the > > > > > > peers are getting rejected. > > > > > > The bricks aren't coming because of the > > > same issue. > > > > > > > > > > > > More about the issue: > > > https://bugzilla.redhat.com/show_bug.cgi?id=1685120 > > > > > > > > > > > > On Tue, Mar 5, 2019 at 12:56 PM Hu Bert > > > <revirii at googlemail.com > > > <mailto:revirii at googlemail.com>> wrote: > > > > > > > > > > > > > > Interestingly: gluster volume status > > > misses gluster1, while heal > > > > > > > statistics show gluster1: > > > > > > > > > > > > > > gluster volume status workdata > > > > > > > Status of volume: workdata > > > > > > > Gluster process > > > TCP Port RDMA Port Online Pid > > > > > > > > > > ------------------------------------------------------------------------------ > > > > > > > Brick gluster2:/gluster/md4/workdata > > > 49153 0 Y 1723 > > > > > > > Brick gluster3:/gluster/md4/workdata > > > 49153 0 Y 2068 > > > > > > > Self-heal Daemon on localhost > > > N/A N/A Y 1732 > > > > > > > Self-heal Daemon on gluster3 > > > N/A N/A Y 2077 > > > > > > > > > > > > > > vs. > > > > > > > > > > > > > > gluster volume heal workdata statistics > > > heal-count > > > > > > > Gathering count of entries to be healed > > > on volume workdata has been successful > > > > > > > > > > > > > > Brick gluster1:/gluster/md4/workdata > > > > > > > Number of entries: 0 > > > > > > > > > > > > > > Brick gluster2:/gluster/md4/workdata > > > > > > > Number of entries: 10745 > > > > > > > > > > > > > > Brick gluster3:/gluster/md4/workdata > > > > > > > Number of entries: 10744 > > > > > > > > > > > > > > Am Di., 5. M?rz 2019 um 08:18 Uhr > > > schrieb Hu Bert <revirii at googlemail.com > > > <mailto:revirii at googlemail.com>>: > > > > > > > > > > > > > > > > Hi Miling, > > > > > > > > > > > > > > > > well, there are such entries, but > > > those haven't been a problem during > > > > > > > > install and the last kernel > > > update+reboot. The entries look like: > > > > > > > > > > > > > > > > PUBLIC_IP gluster2.alpserver.de > > > <http://gluster2.alpserver.de> gluster2 > > > > > > > > > > > > > > > > 192.168.0.50 gluster1 > > > > > > > > 192.168.0.51 gluster2 > > > > > > > > 192.168.0.52 gluster3 > > > > > > > > > > > > > > > > 'ping gluster2' resolves to LAN IP; I > > > removed the last entry in the > > > > > > > > 1st line, did a reboot ... no, didn't > > > help. From > > > > > > > > /var/log/glusterfs/glusterd.log > > > > > > > > on gluster 2: > > > > > > > > > > > > > > > > [2019-03-05 07:04:36.188128] E [MSGID: > > > 106010] > > > > > > > > > > > [glusterd-utils.c:3483:glusterd_compare_friend_volume] > > > 0-management: > > > > > > > > Version of Cksums persistent differ. > > > local cksum = 3950307018, remote > > > > > > > > cksum = 455409345 on peer gluster1 > > > > > > > > [2019-03-05 07:04:36.188314] I [MSGID: > > > 106493] > > > > > > > > > > > [glusterd-handler.c:3843:glusterd_xfer_friend_add_resp] > > > 0-glusterd: > > > > > > > > Responded to gluster1 (0), ret: 0, > > > op_ret: -1 > > > > > > > > > > > > > > > > Interestingly there are no entries in > > > the brick logs of the rejected > > > > > > > > server. Well, not surprising as no > > > brick process is running. The > > > > > > > > server gluster1 is still in rejected > > > state. > > > > > > > > > > > > > > > > 'gluster volume start workdata force' > > > starts the brick process on > > > > > > > > gluster1, and some heals are happening > > > on gluster2+3, but via 'gluster > > > > > > > > volume status workdata' the volumes > > > still aren't complete. > > > > > > > > > > > > > > > > gluster1: > > > > > > > > > > > ------------------------------------------------------------------------------ > > > > > > > > Brick gluster1:/gluster/md4/workdata > > > 49152 0 Y 2523 > > > > > > > > Self-heal Daemon on localhost > > > N/A N/A Y 2549 > > > > > > > > > > > > > > > > gluster2: > > > > > > > > Gluster process > > > TCP Port RDMA Port Online Pid > > > > > > > > > > > ------------------------------------------------------------------------------ > > > > > > > > Brick gluster2:/gluster/md4/workdata > > > 49153 0 Y 1723 > > > > > > > > Brick gluster3:/gluster/md4/workdata > > > 49153 0 Y 2068 > > > > > > > > Self-heal Daemon on localhost > > > N/A N/A Y 1732 > > > > > > > > Self-heal Daemon on gluster3 > > > N/A N/A Y 2077 > > > > > > > > > > > > > > > > > > > > > > > > Hubert > > > > > > > > > > > > > > > > Am Di., 5. M?rz 2019 um 07:58 Uhr > > > schrieb Milind Changire <mchangir at redhat.com > > > <mailto:mchangir at redhat.com>>: > > > > > > > > > > > > > > > > > > There are probably DNS entries or > > > /etc/hosts entries with the public IP Addresses > > > that the host names (gluster1, gluster2, > > > gluster3) are getting resolved to. > > > > > > > > > /etc/resolv.conf would tell which is > > > the default domain searched for the node names > > > and the DNS servers which respond to the queries. > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Mar 5, 2019 at 12:14 PM Hu > > > Bert <revirii at googlemail.com > > > <mailto:revirii at googlemail.com>> wrote: > > > > > > > > >> > > > > > > > > >> Good morning, > > > > > > > > >> > > > > > > > > >> i have a replicate 3 setup with 2 > > > volumes, running on version 5.3 on > > > > > > > > >> debian stretch. This morning i > > > upgraded one server to version 5.4 and > > > > > > > > >> rebooted the machine; after the > > > restart i noticed that: > > > > > > > > >> > > > > > > > > >> - no brick process is running > > > > > > > > >> - gluster volume status only shows > > > the server itself: > > > > > > > > >> gluster volume status workdata > > > > > > > > >> Status of volume: workdata > > > > > > > > >> Gluster process > > > TCP Port RDMA Port Online Pid > > > > > > > > >> > > > ------------------------------------------------------------------------------ > > > > > > > > >> Brick > > > gluster1:/gluster/md4/workdata N/A > > > N/A N N/A > > > > > > > > >> NFS Server on localhost > > > N/A N/A N N/A > > > > > > > > >> > > > > > > > > >> - gluster peer status on the server > > > > > > > > >> gluster peer status > > > > > > > > >> Number of Peers: 2 > > > > > > > > >> > > > > > > > > >> Hostname: gluster3 > > > > > > > > >> Uuid: > > > c7b4a448-ca6a-4051-877f-788f9ee9bc4a > > > > > > > > >> State: Peer Rejected (Connected) > > > > > > > > >> > > > > > > > > >> Hostname: gluster2 > > > > > > > > >> Uuid: > > > 162fea82-406a-4f51-81a3-e90235d8da27 > > > > > > > > >> State: Peer Rejected (Connected) > > > > > > > > >> > > > > > > > > >> - gluster peer status on the other > > > 2 servers: > > > > > > > > >> gluster peer status > > > > > > > > >> Number of Peers: 2 > > > > > > > > >> > > > > > > > > >> Hostname: gluster1 > > > > > > > > >> Uuid: > > > 9a360776-7b58-49ae-831e-a0ce4e4afbef > > > > > > > > >> State: Peer Rejected (Connected) > > > > > > > > >> > > > > > > > > >> Hostname: gluster3 > > > > > > > > >> Uuid: > > > c7b4a448-ca6a-4051-877f-788f9ee9bc4a > > > > > > > > >> State: Peer in Cluster (Connected) > > > > > > > > >> > > > > > > > > >> I noticed that, in the brick logs, > > > i see that the public IP is used > > > > > > > > >> instead of the LAN IP. brick logs > > > from one of the volumes: > > > > > > > > >> > > > > > > > > >> rejected node: > > > https://pastebin.com/qkpj10Sd > > > > > > > > >> connected nodes: > > > https://pastebin.com/8SxVVYFV > > > > > > > > >> > > > > > > > > >> Why is the public IP suddenly used > > > instead of the LAN IP? Killing all > > > > > > > > >> gluster processes and rebooting > > > (again) didn't help. > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> Thx, > > > > > > > > >> Hubert > > > > > > > > >> > > > _______________________________________________ > > > > > > > > >> Gluster-users mailing list > > > > > > > > >> Gluster-users at gluster.org > > > <mailto:Gluster-users at gluster.org> > > > > > > > > >> > > > https://lists.gluster.org/mailman/listinfo/gluster-users > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > Milind > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > > Gluster-users mailing list > > > > > > > Gluster-users at gluster.org > > > <mailto:Gluster-users at gluster.org> > > > > > > > > > > https://lists.gluster.org/mailman/listinfo/gluster-users > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > Regards, > > > > > > Hari Gowtham. > > > > > > > > > > > > > > > > -- > > > > Regards, > > > > Hari Gowtham. > > > _______________________________________________ > > > Gluster-users mailing list > > > Gluster-users at gluster.org > > > <mailto:Gluster-users at gluster.org> > > > https://lists.gluster.org/mailman/listinfo/gluster-users > > > > > > _______________________________________________ > > > Gluster-users mailing list > > > Gluster-users at gluster.org <mailto:Gluster-users at gluster.org> > > > https://lists.gluster.org/mailman/listinfo/gluster-users > > > > > > > > > > > > -- > > > Amar Tumballi (amarts) > > > > > > > > > _______________________________________________ > > > Gluster-users mailing list > > > Gluster-users at gluster.org > > > https://lists.gluster.org/mailman/listinfo/gluster-users > > > > > _______________________________________________ > > Gluster-users mailing list > > Gluster-users at gluster.org > > https://lists.gluster.org/mailman/listinfo/gluster-users