Artem Russakovskii
2019-Mar-13 16:34 UTC
[Gluster-users] Upgrade 5.3 -> 5.4 on debian: public IP is used instead of LAN IP
Wednesday now with no update :-/ Sincerely, Artem -- Founder, Android Police <http://www.androidpolice.com>, APK Mirror <http://www.apkmirror.com/>, Illogical Robot LLC beerpla.net | +ArtemRussakovskii <https://plus.google.com/+ArtemRussakovskii> | @ArtemR <http://twitter.com/ArtemR> On Tue, Mar 12, 2019 at 10:28 AM Artem Russakovskii <archon810 at gmail.com> wrote:> Hi Amar, > > Any updates on this? I'm still not seeing it in OpenSUSE build repos. > Maybe later today? > > Thanks. > > Sincerely, > Artem > > -- > Founder, Android Police <http://www.androidpolice.com>, APK Mirror > <http://www.apkmirror.com/>, Illogical Robot LLC > beerpla.net | +ArtemRussakovskii > <https://plus.google.com/+ArtemRussakovskii> | @ArtemR > <http://twitter.com/ArtemR> > > > On Wed, Mar 6, 2019 at 10:30 PM Amar Tumballi Suryanarayan < > atumball at redhat.com> wrote: > >> We are talking days. Not weeks. Considering already it is Thursday here. >> 1 more day for tagging, and packaging. May be ok to expect it on Monday. >> >> -Amar >> >> On Thu, Mar 7, 2019 at 11:54 AM Artem Russakovskii <archon810 at gmail.com> >> wrote: >> >>> Is the next release going to be an imminent hotfix, i.e. something like >>> today/tomorrow, or are we talking weeks? >>> >>> Sincerely, >>> Artem >>> >>> -- >>> Founder, Android Police <http://www.androidpolice.com>, APK Mirror >>> <http://www.apkmirror.com/>, Illogical Robot LLC >>> beerpla.net | +ArtemRussakovskii >>> <https://plus.google.com/+ArtemRussakovskii> | @ArtemR >>> <http://twitter.com/ArtemR> >>> >>> >>> On Tue, Mar 5, 2019 at 11:09 AM Artem Russakovskii <archon810 at gmail.com> >>> wrote: >>> >>>> Ended up downgrading to 5.3 just in case. Peer status and volume status >>>> are OK now. >>>> >>>> zypper install --oldpackage glusterfs-5.3-lp150.100.1 >>>> Loading repository data... >>>> Reading installed packages... >>>> Resolving package dependencies... >>>> >>>> Problem: glusterfs-5.3-lp150.100.1.x86_64 requires libgfapi0 = 5.3, but >>>> this requirement cannot be provided >>>> not installable providers: libgfapi0-5.3-lp150.100.1.x86_64[glusterfs] >>>> Solution 1: Following actions will be done: >>>> downgrade of libgfapi0-5.4-lp150.100.1.x86_64 to >>>> libgfapi0-5.3-lp150.100.1.x86_64 >>>> downgrade of libgfchangelog0-5.4-lp150.100.1.x86_64 to >>>> libgfchangelog0-5.3-lp150.100.1.x86_64 >>>> downgrade of libgfrpc0-5.4-lp150.100.1.x86_64 to >>>> libgfrpc0-5.3-lp150.100.1.x86_64 >>>> downgrade of libgfxdr0-5.4-lp150.100.1.x86_64 to >>>> libgfxdr0-5.3-lp150.100.1.x86_64 >>>> downgrade of libglusterfs0-5.4-lp150.100.1.x86_64 to >>>> libglusterfs0-5.3-lp150.100.1.x86_64 >>>> Solution 2: do not install glusterfs-5.3-lp150.100.1.x86_64 >>>> Solution 3: break glusterfs-5.3-lp150.100.1.x86_64 by ignoring some of >>>> its dependencies >>>> >>>> Choose from above solutions by number or cancel [1/2/3/c] (c): 1 >>>> Resolving dependencies... >>>> Resolving package dependencies... >>>> >>>> The following 6 packages are going to be downgraded: >>>> glusterfs libgfapi0 libgfchangelog0 libgfrpc0 libgfxdr0 libglusterfs0 >>>> >>>> 6 packages to downgrade. >>>> >>>> Sincerely, >>>> Artem >>>> >>>> -- >>>> Founder, Android Police <http://www.androidpolice.com>, APK Mirror >>>> <http://www.apkmirror.com/>, Illogical Robot LLC >>>> beerpla.net | +ArtemRussakovskii >>>> <https://plus.google.com/+ArtemRussakovskii> | @ArtemR >>>> <http://twitter.com/ArtemR> >>>> >>>> >>>> On Tue, Mar 5, 2019 at 10:57 AM Artem Russakovskii <archon810 at gmail.com> >>>> wrote: >>>> >>>>> Noticed the same when upgrading from 5.3 to 5.4, as mentioned. >>>>> >>>>> I'm confused though. Is actual replication affected, because the 5.4 >>>>> server and the 3x 5.3 servers still show heal info as all 4 connected, and >>>>> the files seem to be replicating correctly as well. >>>>> >>>>> So what's actually affected - just the status command, or leaving 5.4 >>>>> on one of the nodes is doing some damage to the underlying fs? Is it >>>>> fixable by tweaking transport.socket.ssl-enabled? Does upgrading all >>>>> servers to 5.4 resolve it, or should we revert back to 5.3? >>>>> >>>>> Sincerely, >>>>> Artem >>>>> >>>>> -- >>>>> Founder, Android Police <http://www.androidpolice.com>, APK Mirror >>>>> <http://www.apkmirror.com/>, Illogical Robot LLC >>>>> beerpla.net | +ArtemRussakovskii >>>>> <https://plus.google.com/+ArtemRussakovskii> | @ArtemR >>>>> <http://twitter.com/ArtemR> >>>>> >>>>> >>>>> On Tue, Mar 5, 2019 at 2:02 AM Hu Bert <revirii at googlemail.com> wrote: >>>>> >>>>>> fyi: did a downgrade 5.4 -> 5.3 and it worked. all replicas are up and >>>>>> running. Awaiting updated v5.4. >>>>>> >>>>>> thx :-) >>>>>> >>>>>> Am Di., 5. M?rz 2019 um 09:26 Uhr schrieb Hari Gowtham < >>>>>> hgowtham at redhat.com>: >>>>>> > >>>>>> > There are plans to revert the patch causing this error and rebuilt >>>>>> 5.4. >>>>>> > This should happen faster. the rebuilt 5.4 should be void of this >>>>>> upgrade issue. >>>>>> > >>>>>> > In the meantime, you can use 5.3 for this cluster. >>>>>> > Downgrading to 5.3 will work if it was just one node that was >>>>>> upgrade to 5.4 >>>>>> > and the other nodes are still in 5.3. >>>>>> > >>>>>> > On Tue, Mar 5, 2019 at 1:07 PM Hu Bert <revirii at googlemail.com> >>>>>> wrote: >>>>>> > > >>>>>> > > Hi Hari, >>>>>> > > >>>>>> > > thx for the hint. Do you know when this will be fixed? Is a >>>>>> downgrade >>>>>> > > 5.4 -> 5.3 a possibility to fix this? >>>>>> > > >>>>>> > > Hubert >>>>>> > > >>>>>> > > Am Di., 5. M?rz 2019 um 08:32 Uhr schrieb Hari Gowtham < >>>>>> hgowtham at redhat.com>: >>>>>> > > > >>>>>> > > > Hi, >>>>>> > > > >>>>>> > > > This is a known issue we are working on. >>>>>> > > > As the checksum differs between the updated and non updated >>>>>> node, the >>>>>> > > > peers are getting rejected. >>>>>> > > > The bricks aren't coming because of the same issue. >>>>>> > > > >>>>>> > > > More about the issue: >>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1685120 >>>>>> > > > >>>>>> > > > On Tue, Mar 5, 2019 at 12:56 PM Hu Bert <revirii at googlemail.com> >>>>>> wrote: >>>>>> > > > > >>>>>> > > > > Interestingly: gluster volume status misses gluster1, while >>>>>> heal >>>>>> > > > > statistics show gluster1: >>>>>> > > > > >>>>>> > > > > gluster volume status workdata >>>>>> > > > > Status of volume: workdata >>>>>> > > > > Gluster process TCP Port RDMA >>>>>> Port Online Pid >>>>>> > > > > >>>>>> ------------------------------------------------------------------------------ >>>>>> > > > > Brick gluster2:/gluster/md4/workdata 49153 0 >>>>>> Y 1723 >>>>>> > > > > Brick gluster3:/gluster/md4/workdata 49153 0 >>>>>> Y 2068 >>>>>> > > > > Self-heal Daemon on localhost N/A N/A >>>>>> Y 1732 >>>>>> > > > > Self-heal Daemon on gluster3 N/A N/A >>>>>> Y 2077 >>>>>> > > > > >>>>>> > > > > vs. >>>>>> > > > > >>>>>> > > > > gluster volume heal workdata statistics heal-count >>>>>> > > > > Gathering count of entries to be healed on volume workdata >>>>>> has been successful >>>>>> > > > > >>>>>> > > > > Brick gluster1:/gluster/md4/workdata >>>>>> > > > > Number of entries: 0 >>>>>> > > > > >>>>>> > > > > Brick gluster2:/gluster/md4/workdata >>>>>> > > > > Number of entries: 10745 >>>>>> > > > > >>>>>> > > > > Brick gluster3:/gluster/md4/workdata >>>>>> > > > > Number of entries: 10744 >>>>>> > > > > >>>>>> > > > > Am Di., 5. M?rz 2019 um 08:18 Uhr schrieb Hu Bert < >>>>>> revirii at googlemail.com>: >>>>>> > > > > > >>>>>> > > > > > Hi Miling, >>>>>> > > > > > >>>>>> > > > > > well, there are such entries, but those haven't been a >>>>>> problem during >>>>>> > > > > > install and the last kernel update+reboot. The entries look >>>>>> like: >>>>>> > > > > > >>>>>> > > > > > PUBLIC_IP gluster2.alpserver.de gluster2 >>>>>> > > > > > >>>>>> > > > > > 192.168.0.50 gluster1 >>>>>> > > > > > 192.168.0.51 gluster2 >>>>>> > > > > > 192.168.0.52 gluster3 >>>>>> > > > > > >>>>>> > > > > > 'ping gluster2' resolves to LAN IP; I removed the last >>>>>> entry in the >>>>>> > > > > > 1st line, did a reboot ... no, didn't help. From >>>>>> > > > > > /var/log/glusterfs/glusterd.log >>>>>> > > > > > on gluster 2: >>>>>> > > > > > >>>>>> > > > > > [2019-03-05 07:04:36.188128] E [MSGID: 106010] >>>>>> > > > > > [glusterd-utils.c:3483:glusterd_compare_friend_volume] >>>>>> 0-management: >>>>>> > > > > > Version of Cksums persistent differ. local cksum >>>>>> 3950307018, remote >>>>>> > > > > > cksum = 455409345 on peer gluster1 >>>>>> > > > > > [2019-03-05 07:04:36.188314] I [MSGID: 106493] >>>>>> > > > > > [glusterd-handler.c:3843:glusterd_xfer_friend_add_resp] >>>>>> 0-glusterd: >>>>>> > > > > > Responded to gluster1 (0), ret: 0, op_ret: -1 >>>>>> > > > > > >>>>>> > > > > > Interestingly there are no entries in the brick logs of the >>>>>> rejected >>>>>> > > > > > server. Well, not surprising as no brick process is >>>>>> running. The >>>>>> > > > > > server gluster1 is still in rejected state. >>>>>> > > > > > >>>>>> > > > > > 'gluster volume start workdata force' starts the brick >>>>>> process on >>>>>> > > > > > gluster1, and some heals are happening on gluster2+3, but >>>>>> via 'gluster >>>>>> > > > > > volume status workdata' the volumes still aren't complete. >>>>>> > > > > > >>>>>> > > > > > gluster1: >>>>>> > > > > > >>>>>> ------------------------------------------------------------------------------ >>>>>> > > > > > Brick gluster1:/gluster/md4/workdata 49152 0 >>>>>> Y 2523 >>>>>> > > > > > Self-heal Daemon on localhost N/A N/A >>>>>> Y 2549 >>>>>> > > > > > >>>>>> > > > > > gluster2: >>>>>> > > > > > Gluster process TCP Port RDMA >>>>>> Port Online Pid >>>>>> > > > > > >>>>>> ------------------------------------------------------------------------------ >>>>>> > > > > > Brick gluster2:/gluster/md4/workdata 49153 0 >>>>>> Y 1723 >>>>>> > > > > > Brick gluster3:/gluster/md4/workdata 49153 0 >>>>>> Y 2068 >>>>>> > > > > > Self-heal Daemon on localhost N/A N/A >>>>>> Y 1732 >>>>>> > > > > > Self-heal Daemon on gluster3 N/A N/A >>>>>> Y 2077 >>>>>> > > > > > >>>>>> > > > > > >>>>>> > > > > > Hubert >>>>>> > > > > > >>>>>> > > > > > Am Di., 5. M?rz 2019 um 07:58 Uhr schrieb Milind Changire < >>>>>> mchangir at redhat.com>: >>>>>> > > > > > > >>>>>> > > > > > > There are probably DNS entries or /etc/hosts entries with >>>>>> the public IP Addresses that the host names (gluster1, gluster2, gluster3) >>>>>> are getting resolved to. >>>>>> > > > > > > /etc/resolv.conf would tell which is the default domain >>>>>> searched for the node names and the DNS servers which respond to the >>>>>> queries. >>>>>> > > > > > > >>>>>> > > > > > > >>>>>> > > > > > > On Tue, Mar 5, 2019 at 12:14 PM Hu Bert < >>>>>> revirii at googlemail.com> wrote: >>>>>> > > > > > >> >>>>>> > > > > > >> Good morning, >>>>>> > > > > > >> >>>>>> > > > > > >> i have a replicate 3 setup with 2 volumes, running on >>>>>> version 5.3 on >>>>>> > > > > > >> debian stretch. This morning i upgraded one server to >>>>>> version 5.4 and >>>>>> > > > > > >> rebooted the machine; after the restart i noticed that: >>>>>> > > > > > >> >>>>>> > > > > > >> - no brick process is running >>>>>> > > > > > >> - gluster volume status only shows the server itself: >>>>>> > > > > > >> gluster volume status workdata >>>>>> > > > > > >> Status of volume: workdata >>>>>> > > > > > >> Gluster process TCP Port >>>>>> RDMA Port Online Pid >>>>>> > > > > > >> >>>>>> ------------------------------------------------------------------------------ >>>>>> > > > > > >> Brick gluster1:/gluster/md4/workdata N/A >>>>>> N/A N N/A >>>>>> > > > > > >> NFS Server on localhost N/A >>>>>> N/A N N/A >>>>>> > > > > > >> >>>>>> > > > > > >> - gluster peer status on the server >>>>>> > > > > > >> gluster peer status >>>>>> > > > > > >> Number of Peers: 2 >>>>>> > > > > > >> >>>>>> > > > > > >> Hostname: gluster3 >>>>>> > > > > > >> Uuid: c7b4a448-ca6a-4051-877f-788f9ee9bc4a >>>>>> > > > > > >> State: Peer Rejected (Connected) >>>>>> > > > > > >> >>>>>> > > > > > >> Hostname: gluster2 >>>>>> > > > > > >> Uuid: 162fea82-406a-4f51-81a3-e90235d8da27 >>>>>> > > > > > >> State: Peer Rejected (Connected) >>>>>> > > > > > >> >>>>>> > > > > > >> - gluster peer status on the other 2 servers: >>>>>> > > > > > >> gluster peer status >>>>>> > > > > > >> Number of Peers: 2 >>>>>> > > > > > >> >>>>>> > > > > > >> Hostname: gluster1 >>>>>> > > > > > >> Uuid: 9a360776-7b58-49ae-831e-a0ce4e4afbef >>>>>> > > > > > >> State: Peer Rejected (Connected) >>>>>> > > > > > >> >>>>>> > > > > > >> Hostname: gluster3 >>>>>> > > > > > >> Uuid: c7b4a448-ca6a-4051-877f-788f9ee9bc4a >>>>>> > > > > > >> State: Peer in Cluster (Connected) >>>>>> > > > > > >> >>>>>> > > > > > >> I noticed that, in the brick logs, i see that the public >>>>>> IP is used >>>>>> > > > > > >> instead of the LAN IP. brick logs from one of the >>>>>> volumes: >>>>>> > > > > > >> >>>>>> > > > > > >> rejected node: https://pastebin.com/qkpj10Sd >>>>>> > > > > > >> connected nodes: https://pastebin.com/8SxVVYFV >>>>>> > > > > > >> >>>>>> > > > > > >> Why is the public IP suddenly used instead of the LAN >>>>>> IP? Killing all >>>>>> > > > > > >> gluster processes and rebooting (again) didn't help. >>>>>> > > > > > >> >>>>>> > > > > > >> >>>>>> > > > > > >> Thx, >>>>>> > > > > > >> Hubert >>>>>> > > > > > >> _______________________________________________ >>>>>> > > > > > >> Gluster-users mailing list >>>>>> > > > > > >> Gluster-users at gluster.org >>>>>> > > > > > >> https://lists.gluster.org/mailman/listinfo/gluster-users >>>>>> > > > > > > >>>>>> > > > > > > >>>>>> > > > > > > >>>>>> > > > > > > -- >>>>>> > > > > > > Milind >>>>>> > > > > > > >>>>>> > > > > _______________________________________________ >>>>>> > > > > Gluster-users mailing list >>>>>> > > > > Gluster-users at gluster.org >>>>>> > > > > https://lists.gluster.org/mailman/listinfo/gluster-users >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > -- >>>>>> > > > Regards, >>>>>> > > > Hari Gowtham. >>>>>> > >>>>>> > >>>>>> > >>>>>> > -- >>>>>> > Regards, >>>>>> > Hari Gowtham. >>>>>> _______________________________________________ >>>>>> Gluster-users mailing list >>>>>> Gluster-users at gluster.org >>>>>> https://lists.gluster.org/mailman/listinfo/gluster-users >>>>> >>>>> _______________________________________________ >>> Gluster-users mailing list >>> Gluster-users at gluster.org >>> https://lists.gluster.org/mailman/listinfo/gluster-users >> >> >> >> -- >> Amar Tumballi (amarts) >> >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20190313/9b27a4c3/attachment.html>
Shyam Ranganathan
2019-Mar-15 13:28 UTC
[Gluster-users] Upgrade 5.3 -> 5.4 on debian: public IP is used instead of LAN IP
We created a 5.5 release tag, and it is under packaging now. It should be packaged and ready for testing early next week and should be released close to mid-week next week. Thanks, Shyam On 3/13/19 12:34 PM, Artem Russakovskii wrote:> Wednesday now with no update :-/ > > Sincerely, > Artem > > -- > Founder, Android Police <http://www.androidpolice.com>,?APK Mirror > <http://www.apkmirror.com/>, Illogical Robot LLC > beerpla.net <http://beerpla.net/> | +ArtemRussakovskii > <https://plus.google.com/+ArtemRussakovskii> |?@ArtemR > <http://twitter.com/ArtemR> > > > On Tue, Mar 12, 2019 at 10:28 AM Artem Russakovskii <archon810 at gmail.com > <mailto:archon810 at gmail.com>> wrote: > > Hi Amar, > > Any updates on this? I'm still not seeing it in OpenSUSE build > repos. Maybe later today? > > Thanks. > > Sincerely, > Artem > > -- > Founder, Android Police <http://www.androidpolice.com>,?APK Mirror > <http://www.apkmirror.com/>, Illogical Robot LLC > beerpla.net <http://beerpla.net/> | +ArtemRussakovskii > <https://plus.google.com/+ArtemRussakovskii> |?@ArtemR > <http://twitter.com/ArtemR> > > > On Wed, Mar 6, 2019 at 10:30 PM Amar Tumballi Suryanarayan > <atumball at redhat.com <mailto:atumball at redhat.com>> wrote: > > We are talking days. Not weeks. Considering already it is > Thursday here. 1 more day for tagging, and packaging. May be ok > to expect it on Monday. > > -Amar > > On Thu, Mar 7, 2019 at 11:54 AM Artem Russakovskii > <archon810 at gmail.com <mailto:archon810 at gmail.com>> wrote: > > Is the next release going to be an imminent hotfix, i.e. > something like today/tomorrow, or are we talking weeks? > > Sincerely, > Artem > > -- > Founder, Android Police <http://www.androidpolice.com>,?APK > Mirror <http://www.apkmirror.com/>, Illogical Robot LLC > beerpla.net <http://beerpla.net/> | +ArtemRussakovskii > <https://plus.google.com/+ArtemRussakovskii> |?@ArtemR > <http://twitter.com/ArtemR> > > > On Tue, Mar 5, 2019 at 11:09 AM Artem Russakovskii > <archon810 at gmail.com <mailto:archon810 at gmail.com>> wrote: > > Ended up downgrading to 5.3 just in case. Peer status > and volume status are OK now. > > zypper install --oldpackage glusterfs-5.3-lp150.100.1 > Loading repository data... > Reading installed packages... > Resolving package dependencies... > > Problem: glusterfs-5.3-lp150.100.1.x86_64 requires > libgfapi0 = 5.3, but this requirement cannot be provided > ? not installable providers: > libgfapi0-5.3-lp150.100.1.x86_64[glusterfs] > ?Solution 1: Following actions will be done: > ? downgrade of libgfapi0-5.4-lp150.100.1.x86_64 to > libgfapi0-5.3-lp150.100.1.x86_64 > ? downgrade of libgfchangelog0-5.4-lp150.100.1.x86_64 to > libgfchangelog0-5.3-lp150.100.1.x86_64 > ? downgrade of libgfrpc0-5.4-lp150.100.1.x86_64 to > libgfrpc0-5.3-lp150.100.1.x86_64 > ? downgrade of libgfxdr0-5.4-lp150.100.1.x86_64 to > libgfxdr0-5.3-lp150.100.1.x86_64 > ? downgrade of libglusterfs0-5.4-lp150.100.1.x86_64 to > libglusterfs0-5.3-lp150.100.1.x86_64 > ?Solution 2: do not install glusterfs-5.3-lp150.100.1.x86_64 > ?Solution 3: break glusterfs-5.3-lp150.100.1.x86_64 by > ignoring some of its dependencies > > Choose from above solutions by number or cancel > [1/2/3/c] (c): 1 > Resolving dependencies... > Resolving package dependencies... > > The following 6 packages are going to be downgraded: > ? glusterfs libgfapi0 libgfchangelog0 libgfrpc0 > libgfxdr0 libglusterfs0 > > 6 packages to downgrade. > > Sincerely, > Artem > > -- > Founder, Android Police > <http://www.androidpolice.com>,?APK Mirror > <http://www.apkmirror.com/>, Illogical Robot LLC > beerpla.net <http://beerpla.net/> | +ArtemRussakovskii > <https://plus.google.com/+ArtemRussakovskii> |?@ArtemR > <http://twitter.com/ArtemR> > > > On Tue, Mar 5, 2019 at 10:57 AM Artem Russakovskii > <archon810 at gmail.com <mailto:archon810 at gmail.com>> wrote: > > Noticed the same when upgrading from 5.3 to 5.4, as > mentioned. > > I'm confused though. Is actual replication affected, > because the 5.4 server and the 3x 5.3 servers still > show heal info as all 4 connected, and the files > seem to be replicating correctly as well. > > So what's actually affected - just the status > command, or leaving 5.4 on one of the nodes is doing > some damage to the underlying fs? Is it fixable by > tweaking transport.socket.ssl-enabled? Does > upgrading all servers to 5.4 resolve it, or should > we revert back to 5.3? > > Sincerely, > Artem > > -- > Founder, Android Police > <http://www.androidpolice.com>,?APK Mirror > <http://www.apkmirror.com/>, Illogical Robot LLC > beerpla.net <http://beerpla.net/> | > +ArtemRussakovskii > <https://plus.google.com/+ArtemRussakovskii> > |?@ArtemR <http://twitter.com/ArtemR> > > > On Tue, Mar 5, 2019 at 2:02 AM Hu Bert > <revirii at googlemail.com > <mailto:revirii at googlemail.com>> wrote: > > fyi: did a downgrade 5.4 -> 5.3 and it worked. > all replicas are up and > running. Awaiting updated v5.4. > > thx :-) > > Am Di., 5. M?rz 2019 um 09:26 Uhr schrieb Hari > Gowtham <hgowtham at redhat.com > <mailto:hgowtham at redhat.com>>: > > > > There are plans to revert the patch causing > this error and rebuilt 5.4. > > This should happen faster. the rebuilt 5.4 > should be void of this upgrade issue. > > > > In the meantime, you can use 5.3 for this cluster. > > Downgrading to 5.3 will work if it was just > one node that was upgrade to 5.4 > > and the other nodes are still in 5.3. > > > > On Tue, Mar 5, 2019 at 1:07 PM Hu Bert > <revirii at googlemail.com > <mailto:revirii at googlemail.com>> wrote: > > > > > > Hi Hari, > > > > > > thx for the hint. Do you know when this will > be fixed? Is a downgrade > > > 5.4 -> 5.3 a possibility to fix this? > > > > > > Hubert > > > > > > Am Di., 5. M?rz 2019 um 08:32 Uhr schrieb > Hari Gowtham <hgowtham at redhat.com > <mailto:hgowtham at redhat.com>>: > > > > > > > > Hi, > > > > > > > > This is a known issue we are working on. > > > > As the checksum differs between the > updated and non updated node, the > > > > peers are getting rejected. > > > > The bricks aren't coming because of the > same issue. > > > > > > > > More about the issue: > https://bugzilla.redhat.com/show_bug.cgi?id=1685120 > > > > > > > > On Tue, Mar 5, 2019 at 12:56 PM Hu Bert > <revirii at googlemail.com > <mailto:revirii at googlemail.com>> wrote: > > > > > > > > > > Interestingly: gluster volume status > misses gluster1, while heal > > > > > statistics show gluster1: > > > > > > > > > > gluster volume status workdata > > > > > Status of volume: workdata > > > > > Gluster process? ? ? ? ? ? ? ? ? ? ? ? ? > ? ?TCP Port? RDMA Port? Online? Pid > > > > > > ------------------------------------------------------------------------------ > > > > > Brick gluster2:/gluster/md4/workdata? ? > ? ? 49153? ? ?0? ? ? ? ? Y? ? ? ?1723 > > > > > Brick gluster3:/gluster/md4/workdata? ? > ? ? 49153? ? ?0? ? ? ? ? Y? ? ? ?2068 > > > > > Self-heal Daemon on localhost? ? ? ? ? ? > ? ?N/A? ? ? ?N/A? ? ? ? Y? ? ? ?1732 > > > > > Self-heal Daemon on gluster3? ? ? ? ? ? > ? ? N/A? ? ? ?N/A? ? ? ? Y? ? ? ?2077 > > > > > > > > > > vs. > > > > > > > > > > gluster volume heal workdata statistics > heal-count > > > > > Gathering count of entries to be healed > on volume workdata has been successful > > > > > > > > > > Brick gluster1:/gluster/md4/workdata > > > > > Number of entries: 0 > > > > > > > > > > Brick gluster2:/gluster/md4/workdata > > > > > Number of entries: 10745 > > > > > > > > > > Brick gluster3:/gluster/md4/workdata > > > > > Number of entries: 10744 > > > > > > > > > > Am Di., 5. M?rz 2019 um 08:18 Uhr > schrieb Hu Bert <revirii at googlemail.com > <mailto:revirii at googlemail.com>>: > > > > > > > > > > > > Hi Miling, > > > > > > > > > > > > well, there are such entries, but > those haven't been a problem during > > > > > > install and the last kernel > update+reboot. The entries look like: > > > > > > > > > > > > PUBLIC_IP? gluster2.alpserver.de > <http://gluster2.alpserver.de> gluster2 > > > > > > > > > > > > 192.168.0.50 gluster1 > > > > > > 192.168.0.51 gluster2 > > > > > > 192.168.0.52 gluster3 > > > > > > > > > > > > 'ping gluster2' resolves to LAN IP; I > removed the last entry in the > > > > > > 1st line, did a reboot ... no, didn't > help. From > > > > > > /var/log/glusterfs/glusterd.log > > > > > >? on gluster 2: > > > > > > > > > > > > [2019-03-05 07:04:36.188128] E [MSGID: > 106010] > > > > > > > [glusterd-utils.c:3483:glusterd_compare_friend_volume] > 0-management: > > > > > > Version of Cksums persistent differ. > local cksum = 3950307018, remote > > > > > > cksum = 455409345 on peer gluster1 > > > > > > [2019-03-05 07:04:36.188314] I [MSGID: > 106493] > > > > > > > [glusterd-handler.c:3843:glusterd_xfer_friend_add_resp] > 0-glusterd: > > > > > > Responded to gluster1 (0), ret: 0, > op_ret: -1 > > > > > > > > > > > > Interestingly there are no entries in > the brick logs of the rejected > > > > > > server. Well, not surprising as no > brick process is running. The > > > > > > server gluster1 is still in rejected > state. > > > > > > > > > > > > 'gluster volume start workdata force' > starts the brick process on > > > > > > gluster1, and some heals are happening > on gluster2+3, but via 'gluster > > > > > > volume status workdata' the volumes > still aren't complete. > > > > > > > > > > > > gluster1: > > > > > > > ------------------------------------------------------------------------------ > > > > > > Brick gluster1:/gluster/md4/workdata? > ? ? ? 49152? ? ?0? ? ? ? ? Y? ? ? ?2523 > > > > > > Self-heal Daemon on localhost? ? ? ? ? > ? ? ?N/A? ? ? ?N/A? ? ? ? Y? ? ? ?2549 > > > > > > > > > > > > gluster2: > > > > > > Gluster process? ? ? ? ? ? ? ? ? ? ? ? > ? ? ?TCP Port? RDMA Port? Online? Pid > > > > > > > ------------------------------------------------------------------------------ > > > > > > Brick gluster2:/gluster/md4/workdata? > ? ? ? 49153? ? ?0? ? ? ? ? Y? ? ? ?1723 > > > > > > Brick gluster3:/gluster/md4/workdata? > ? ? ? 49153? ? ?0? ? ? ? ? Y? ? ? ?2068 > > > > > > Self-heal Daemon on localhost? ? ? ? ? > ? ? ?N/A? ? ? ?N/A? ? ? ? Y? ? ? ?1732 > > > > > > Self-heal Daemon on gluster3? ? ? ? ? > ? ? ? N/A? ? ? ?N/A? ? ? ? Y? ? ? ?2077 > > > > > > > > > > > > > > > > > > Hubert > > > > > > > > > > > > Am Di., 5. M?rz 2019 um 07:58 Uhr > schrieb Milind Changire <mchangir at redhat.com > <mailto:mchangir at redhat.com>>: > > > > > > > > > > > > > > There are probably DNS entries or > /etc/hosts entries with the public IP Addresses > that the host names (gluster1, gluster2, > gluster3) are getting resolved to. > > > > > > > /etc/resolv.conf would tell which is > the default domain searched for the node names > and the DNS servers which respond to the queries. > > > > > > > > > > > > > > > > > > > > > On Tue, Mar 5, 2019 at 12:14 PM Hu > Bert <revirii at googlemail.com > <mailto:revirii at googlemail.com>> wrote: > > > > > > >> > > > > > > >> Good morning, > > > > > > >> > > > > > > >> i have a replicate 3 setup with 2 > volumes, running on version 5.3 on > > > > > > >> debian stretch. This morning i > upgraded one server to version 5.4 and > > > > > > >> rebooted the machine; after the > restart i noticed that: > > > > > > >> > > > > > > >> - no brick process is running > > > > > > >> - gluster volume status only shows > the server itself: > > > > > > >> gluster volume status workdata > > > > > > >> Status of volume: workdata > > > > > > >> Gluster process? ? ? ? ? ? ? ? ? ? > ? ? ? ? ?TCP Port? RDMA Port? Online? Pid > > > > > > >> > ------------------------------------------------------------------------------ > > > > > > >> Brick > gluster1:/gluster/md4/workdata? ? ? ? N/A? ? ? > ?N/A? ? ? ? N? ? ? ?N/A > > > > > > >> NFS Server on localhost? ? ? ? ? ? > ? ? ? ? ?N/A? ? ? ?N/A? ? ? ? N? ? ? ?N/A > > > > > > >> > > > > > > >> - gluster peer status on the server > > > > > > >> gluster peer status > > > > > > >> Number of Peers: 2 > > > > > > >> > > > > > > >> Hostname: gluster3 > > > > > > >> Uuid: > c7b4a448-ca6a-4051-877f-788f9ee9bc4a > > > > > > >> State: Peer Rejected (Connected) > > > > > > >> > > > > > > >> Hostname: gluster2 > > > > > > >> Uuid: > 162fea82-406a-4f51-81a3-e90235d8da27 > > > > > > >> State: Peer Rejected (Connected) > > > > > > >> > > > > > > >> - gluster peer status on the other > 2 servers: > > > > > > >> gluster peer status > > > > > > >> Number of Peers: 2 > > > > > > >> > > > > > > >> Hostname: gluster1 > > > > > > >> Uuid: > 9a360776-7b58-49ae-831e-a0ce4e4afbef > > > > > > >> State: Peer Rejected (Connected) > > > > > > >> > > > > > > >> Hostname: gluster3 > > > > > > >> Uuid: > c7b4a448-ca6a-4051-877f-788f9ee9bc4a > > > > > > >> State: Peer in Cluster (Connected) > > > > > > >> > > > > > > >> I noticed that, in the brick logs, > i see that the public IP is used > > > > > > >> instead of the LAN IP. brick logs > from one of the volumes: > > > > > > >> > > > > > > >> rejected node: > https://pastebin.com/qkpj10Sd > > > > > > >> connected nodes: > https://pastebin.com/8SxVVYFV > > > > > > >> > > > > > > >> Why is the public IP suddenly used > instead of the LAN IP? Killing all > > > > > > >> gluster processes and rebooting > (again) didn't help. > > > > > > >> > > > > > > >> > > > > > > >> Thx, > > > > > > >> Hubert > > > > > > >> > _______________________________________________ > > > > > > >> Gluster-users mailing list > > > > > > >> Gluster-users at gluster.org > <mailto:Gluster-users at gluster.org> > > > > > > >> > https://lists.gluster.org/mailman/listinfo/gluster-users > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > Milind > > > > > > > > > > > > > _______________________________________________ > > > > > Gluster-users mailing list > > > > > Gluster-users at gluster.org > <mailto:Gluster-users at gluster.org> > > > > > > https://lists.gluster.org/mailman/listinfo/gluster-users > > > > > > > > > > > > > > > > -- > > > > Regards, > > > > Hari Gowtham. > > > > > > > > -- > > Regards, > > Hari Gowtham. > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > <mailto:Gluster-users at gluster.org> > https://lists.gluster.org/mailman/listinfo/gluster-users > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org <mailto:Gluster-users at gluster.org> > https://lists.gluster.org/mailman/listinfo/gluster-users > > > > -- > Amar Tumballi (amarts) > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users >