Artem Russakovskii
2019-Mar-05 19:09 UTC
[Gluster-users] Upgrade 5.3 -> 5.4 on debian: public IP is used instead of LAN IP
Ended up downgrading to 5.3 just in case. Peer status and volume status are OK now. zypper install --oldpackage glusterfs-5.3-lp150.100.1 Loading repository data... Reading installed packages... Resolving package dependencies... Problem: glusterfs-5.3-lp150.100.1.x86_64 requires libgfapi0 = 5.3, but this requirement cannot be provided not installable providers: libgfapi0-5.3-lp150.100.1.x86_64[glusterfs] Solution 1: Following actions will be done: downgrade of libgfapi0-5.4-lp150.100.1.x86_64 to libgfapi0-5.3-lp150.100.1.x86_64 downgrade of libgfchangelog0-5.4-lp150.100.1.x86_64 to libgfchangelog0-5.3-lp150.100.1.x86_64 downgrade of libgfrpc0-5.4-lp150.100.1.x86_64 to libgfrpc0-5.3-lp150.100.1.x86_64 downgrade of libgfxdr0-5.4-lp150.100.1.x86_64 to libgfxdr0-5.3-lp150.100.1.x86_64 downgrade of libglusterfs0-5.4-lp150.100.1.x86_64 to libglusterfs0-5.3-lp150.100.1.x86_64 Solution 2: do not install glusterfs-5.3-lp150.100.1.x86_64 Solution 3: break glusterfs-5.3-lp150.100.1.x86_64 by ignoring some of its dependencies Choose from above solutions by number or cancel [1/2/3/c] (c): 1 Resolving dependencies... Resolving package dependencies... The following 6 packages are going to be downgraded: glusterfs libgfapi0 libgfchangelog0 libgfrpc0 libgfxdr0 libglusterfs0 6 packages to downgrade. Sincerely, Artem -- Founder, Android Police <http://www.androidpolice.com>, APK Mirror <http://www.apkmirror.com/>, Illogical Robot LLC beerpla.net | +ArtemRussakovskii <https://plus.google.com/+ArtemRussakovskii> | @ArtemR <http://twitter.com/ArtemR> On Tue, Mar 5, 2019 at 10:57 AM Artem Russakovskii <archon810 at gmail.com> wrote:> Noticed the same when upgrading from 5.3 to 5.4, as mentioned. > > I'm confused though. Is actual replication affected, because the 5.4 > server and the 3x 5.3 servers still show heal info as all 4 connected, and > the files seem to be replicating correctly as well. > > So what's actually affected - just the status command, or leaving 5.4 on > one of the nodes is doing some damage to the underlying fs? Is it fixable > by tweaking transport.socket.ssl-enabled? Does upgrading all servers to 5.4 > resolve it, or should we revert back to 5.3? > > Sincerely, > Artem > > -- > Founder, Android Police <http://www.androidpolice.com>, APK Mirror > <http://www.apkmirror.com/>, Illogical Robot LLC > beerpla.net | +ArtemRussakovskii > <https://plus.google.com/+ArtemRussakovskii> | @ArtemR > <http://twitter.com/ArtemR> > > > On Tue, Mar 5, 2019 at 2:02 AM Hu Bert <revirii at googlemail.com> wrote: > >> fyi: did a downgrade 5.4 -> 5.3 and it worked. all replicas are up and >> running. Awaiting updated v5.4. >> >> thx :-) >> >> Am Di., 5. M?rz 2019 um 09:26 Uhr schrieb Hari Gowtham < >> hgowtham at redhat.com>: >> > >> > There are plans to revert the patch causing this error and rebuilt 5.4. >> > This should happen faster. the rebuilt 5.4 should be void of this >> upgrade issue. >> > >> > In the meantime, you can use 5.3 for this cluster. >> > Downgrading to 5.3 will work if it was just one node that was upgrade >> to 5.4 >> > and the other nodes are still in 5.3. >> > >> > On Tue, Mar 5, 2019 at 1:07 PM Hu Bert <revirii at googlemail.com> wrote: >> > > >> > > Hi Hari, >> > > >> > > thx for the hint. Do you know when this will be fixed? Is a downgrade >> > > 5.4 -> 5.3 a possibility to fix this? >> > > >> > > Hubert >> > > >> > > Am Di., 5. M?rz 2019 um 08:32 Uhr schrieb Hari Gowtham < >> hgowtham at redhat.com>: >> > > > >> > > > Hi, >> > > > >> > > > This is a known issue we are working on. >> > > > As the checksum differs between the updated and non updated node, >> the >> > > > peers are getting rejected. >> > > > The bricks aren't coming because of the same issue. >> > > > >> > > > More about the issue: >> https://bugzilla.redhat.com/show_bug.cgi?id=1685120 >> > > > >> > > > On Tue, Mar 5, 2019 at 12:56 PM Hu Bert <revirii at googlemail.com> >> wrote: >> > > > > >> > > > > Interestingly: gluster volume status misses gluster1, while heal >> > > > > statistics show gluster1: >> > > > > >> > > > > gluster volume status workdata >> > > > > Status of volume: workdata >> > > > > Gluster process TCP Port RDMA Port >> Online Pid >> > > > > >> ------------------------------------------------------------------------------ >> > > > > Brick gluster2:/gluster/md4/workdata 49153 0 >> Y 1723 >> > > > > Brick gluster3:/gluster/md4/workdata 49153 0 >> Y 2068 >> > > > > Self-heal Daemon on localhost N/A N/A >> Y 1732 >> > > > > Self-heal Daemon on gluster3 N/A N/A >> Y 2077 >> > > > > >> > > > > vs. >> > > > > >> > > > > gluster volume heal workdata statistics heal-count >> > > > > Gathering count of entries to be healed on volume workdata has >> been successful >> > > > > >> > > > > Brick gluster1:/gluster/md4/workdata >> > > > > Number of entries: 0 >> > > > > >> > > > > Brick gluster2:/gluster/md4/workdata >> > > > > Number of entries: 10745 >> > > > > >> > > > > Brick gluster3:/gluster/md4/workdata >> > > > > Number of entries: 10744 >> > > > > >> > > > > Am Di., 5. M?rz 2019 um 08:18 Uhr schrieb Hu Bert < >> revirii at googlemail.com>: >> > > > > > >> > > > > > Hi Miling, >> > > > > > >> > > > > > well, there are such entries, but those haven't been a problem >> during >> > > > > > install and the last kernel update+reboot. The entries look >> like: >> > > > > > >> > > > > > PUBLIC_IP gluster2.alpserver.de gluster2 >> > > > > > >> > > > > > 192.168.0.50 gluster1 >> > > > > > 192.168.0.51 gluster2 >> > > > > > 192.168.0.52 gluster3 >> > > > > > >> > > > > > 'ping gluster2' resolves to LAN IP; I removed the last entry in >> the >> > > > > > 1st line, did a reboot ... no, didn't help. From >> > > > > > /var/log/glusterfs/glusterd.log >> > > > > > on gluster 2: >> > > > > > >> > > > > > [2019-03-05 07:04:36.188128] E [MSGID: 106010] >> > > > > > [glusterd-utils.c:3483:glusterd_compare_friend_volume] >> 0-management: >> > > > > > Version of Cksums persistent differ. local cksum = 3950307018, >> remote >> > > > > > cksum = 455409345 on peer gluster1 >> > > > > > [2019-03-05 07:04:36.188314] I [MSGID: 106493] >> > > > > > [glusterd-handler.c:3843:glusterd_xfer_friend_add_resp] >> 0-glusterd: >> > > > > > Responded to gluster1 (0), ret: 0, op_ret: -1 >> > > > > > >> > > > > > Interestingly there are no entries in the brick logs of the >> rejected >> > > > > > server. Well, not surprising as no brick process is running. The >> > > > > > server gluster1 is still in rejected state. >> > > > > > >> > > > > > 'gluster volume start workdata force' starts the brick process >> on >> > > > > > gluster1, and some heals are happening on gluster2+3, but via >> 'gluster >> > > > > > volume status workdata' the volumes still aren't complete. >> > > > > > >> > > > > > gluster1: >> > > > > > >> ------------------------------------------------------------------------------ >> > > > > > Brick gluster1:/gluster/md4/workdata 49152 0 >> Y 2523 >> > > > > > Self-heal Daemon on localhost N/A N/A >> Y 2549 >> > > > > > >> > > > > > gluster2: >> > > > > > Gluster process TCP Port RDMA >> Port Online Pid >> > > > > > >> ------------------------------------------------------------------------------ >> > > > > > Brick gluster2:/gluster/md4/workdata 49153 0 >> Y 1723 >> > > > > > Brick gluster3:/gluster/md4/workdata 49153 0 >> Y 2068 >> > > > > > Self-heal Daemon on localhost N/A N/A >> Y 1732 >> > > > > > Self-heal Daemon on gluster3 N/A N/A >> Y 2077 >> > > > > > >> > > > > > >> > > > > > Hubert >> > > > > > >> > > > > > Am Di., 5. M?rz 2019 um 07:58 Uhr schrieb Milind Changire < >> mchangir at redhat.com>: >> > > > > > > >> > > > > > > There are probably DNS entries or /etc/hosts entries with the >> public IP Addresses that the host names (gluster1, gluster2, gluster3) are >> getting resolved to. >> > > > > > > /etc/resolv.conf would tell which is the default domain >> searched for the node names and the DNS servers which respond to the >> queries. >> > > > > > > >> > > > > > > >> > > > > > > On Tue, Mar 5, 2019 at 12:14 PM Hu Bert < >> revirii at googlemail.com> wrote: >> > > > > > >> >> > > > > > >> Good morning, >> > > > > > >> >> > > > > > >> i have a replicate 3 setup with 2 volumes, running on >> version 5.3 on >> > > > > > >> debian stretch. This morning i upgraded one server to >> version 5.4 and >> > > > > > >> rebooted the machine; after the restart i noticed that: >> > > > > > >> >> > > > > > >> - no brick process is running >> > > > > > >> - gluster volume status only shows the server itself: >> > > > > > >> gluster volume status workdata >> > > > > > >> Status of volume: workdata >> > > > > > >> Gluster process TCP Port RDMA >> Port Online Pid >> > > > > > >> >> ------------------------------------------------------------------------------ >> > > > > > >> Brick gluster1:/gluster/md4/workdata N/A N/A >> N N/A >> > > > > > >> NFS Server on localhost N/A N/A >> N N/A >> > > > > > >> >> > > > > > >> - gluster peer status on the server >> > > > > > >> gluster peer status >> > > > > > >> Number of Peers: 2 >> > > > > > >> >> > > > > > >> Hostname: gluster3 >> > > > > > >> Uuid: c7b4a448-ca6a-4051-877f-788f9ee9bc4a >> > > > > > >> State: Peer Rejected (Connected) >> > > > > > >> >> > > > > > >> Hostname: gluster2 >> > > > > > >> Uuid: 162fea82-406a-4f51-81a3-e90235d8da27 >> > > > > > >> State: Peer Rejected (Connected) >> > > > > > >> >> > > > > > >> - gluster peer status on the other 2 servers: >> > > > > > >> gluster peer status >> > > > > > >> Number of Peers: 2 >> > > > > > >> >> > > > > > >> Hostname: gluster1 >> > > > > > >> Uuid: 9a360776-7b58-49ae-831e-a0ce4e4afbef >> > > > > > >> State: Peer Rejected (Connected) >> > > > > > >> >> > > > > > >> Hostname: gluster3 >> > > > > > >> Uuid: c7b4a448-ca6a-4051-877f-788f9ee9bc4a >> > > > > > >> State: Peer in Cluster (Connected) >> > > > > > >> >> > > > > > >> I noticed that, in the brick logs, i see that the public IP >> is used >> > > > > > >> instead of the LAN IP. brick logs from one of the volumes: >> > > > > > >> >> > > > > > >> rejected node: https://pastebin.com/qkpj10Sd >> > > > > > >> connected nodes: https://pastebin.com/8SxVVYFV >> > > > > > >> >> > > > > > >> Why is the public IP suddenly used instead of the LAN IP? >> Killing all >> > > > > > >> gluster processes and rebooting (again) didn't help. >> > > > > > >> >> > > > > > >> >> > > > > > >> Thx, >> > > > > > >> Hubert >> > > > > > >> _______________________________________________ >> > > > > > >> Gluster-users mailing list >> > > > > > >> Gluster-users at gluster.org >> > > > > > >> https://lists.gluster.org/mailman/listinfo/gluster-users >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > -- >> > > > > > > Milind >> > > > > > > >> > > > > _______________________________________________ >> > > > > Gluster-users mailing list >> > > > > Gluster-users at gluster.org >> > > > > https://lists.gluster.org/mailman/listinfo/gluster-users >> > > > >> > > > >> > > > >> > > > -- >> > > > Regards, >> > > > Hari Gowtham. >> > >> > >> > >> > -- >> > Regards, >> > Hari Gowtham. >> _______________________________________________ >> Gluster-users mailing list >> Gluster-users at gluster.org >> https://lists.gluster.org/mailman/listinfo/gluster-users > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20190305/5e575562/attachment.html>
Artem Russakovskii
2019-Mar-07 06:23 UTC
[Gluster-users] Upgrade 5.3 -> 5.4 on debian: public IP is used instead of LAN IP
Is the next release going to be an imminent hotfix, i.e. something like today/tomorrow, or are we talking weeks? Sincerely, Artem -- Founder, Android Police <http://www.androidpolice.com>, APK Mirror <http://www.apkmirror.com/>, Illogical Robot LLC beerpla.net | +ArtemRussakovskii <https://plus.google.com/+ArtemRussakovskii> | @ArtemR <http://twitter.com/ArtemR> On Tue, Mar 5, 2019 at 11:09 AM Artem Russakovskii <archon810 at gmail.com> wrote:> Ended up downgrading to 5.3 just in case. Peer status and volume status > are OK now. > > zypper install --oldpackage glusterfs-5.3-lp150.100.1 > Loading repository data... > Reading installed packages... > Resolving package dependencies... > > Problem: glusterfs-5.3-lp150.100.1.x86_64 requires libgfapi0 = 5.3, but > this requirement cannot be provided > not installable providers: libgfapi0-5.3-lp150.100.1.x86_64[glusterfs] > Solution 1: Following actions will be done: > downgrade of libgfapi0-5.4-lp150.100.1.x86_64 to > libgfapi0-5.3-lp150.100.1.x86_64 > downgrade of libgfchangelog0-5.4-lp150.100.1.x86_64 to > libgfchangelog0-5.3-lp150.100.1.x86_64 > downgrade of libgfrpc0-5.4-lp150.100.1.x86_64 to > libgfrpc0-5.3-lp150.100.1.x86_64 > downgrade of libgfxdr0-5.4-lp150.100.1.x86_64 to > libgfxdr0-5.3-lp150.100.1.x86_64 > downgrade of libglusterfs0-5.4-lp150.100.1.x86_64 to > libglusterfs0-5.3-lp150.100.1.x86_64 > Solution 2: do not install glusterfs-5.3-lp150.100.1.x86_64 > Solution 3: break glusterfs-5.3-lp150.100.1.x86_64 by ignoring some of > its dependencies > > Choose from above solutions by number or cancel [1/2/3/c] (c): 1 > Resolving dependencies... > Resolving package dependencies... > > The following 6 packages are going to be downgraded: > glusterfs libgfapi0 libgfchangelog0 libgfrpc0 libgfxdr0 libglusterfs0 > > 6 packages to downgrade. > > Sincerely, > Artem > > -- > Founder, Android Police <http://www.androidpolice.com>, APK Mirror > <http://www.apkmirror.com/>, Illogical Robot LLC > beerpla.net | +ArtemRussakovskii > <https://plus.google.com/+ArtemRussakovskii> | @ArtemR > <http://twitter.com/ArtemR> > > > On Tue, Mar 5, 2019 at 10:57 AM Artem Russakovskii <archon810 at gmail.com> > wrote: > >> Noticed the same when upgrading from 5.3 to 5.4, as mentioned. >> >> I'm confused though. Is actual replication affected, because the 5.4 >> server and the 3x 5.3 servers still show heal info as all 4 connected, and >> the files seem to be replicating correctly as well. >> >> So what's actually affected - just the status command, or leaving 5.4 on >> one of the nodes is doing some damage to the underlying fs? Is it fixable >> by tweaking transport.socket.ssl-enabled? Does upgrading all servers to 5.4 >> resolve it, or should we revert back to 5.3? >> >> Sincerely, >> Artem >> >> -- >> Founder, Android Police <http://www.androidpolice.com>, APK Mirror >> <http://www.apkmirror.com/>, Illogical Robot LLC >> beerpla.net | +ArtemRussakovskii >> <https://plus.google.com/+ArtemRussakovskii> | @ArtemR >> <http://twitter.com/ArtemR> >> >> >> On Tue, Mar 5, 2019 at 2:02 AM Hu Bert <revirii at googlemail.com> wrote: >> >>> fyi: did a downgrade 5.4 -> 5.3 and it worked. all replicas are up and >>> running. Awaiting updated v5.4. >>> >>> thx :-) >>> >>> Am Di., 5. M?rz 2019 um 09:26 Uhr schrieb Hari Gowtham < >>> hgowtham at redhat.com>: >>> > >>> > There are plans to revert the patch causing this error and rebuilt 5.4. >>> > This should happen faster. the rebuilt 5.4 should be void of this >>> upgrade issue. >>> > >>> > In the meantime, you can use 5.3 for this cluster. >>> > Downgrading to 5.3 will work if it was just one node that was upgrade >>> to 5.4 >>> > and the other nodes are still in 5.3. >>> > >>> > On Tue, Mar 5, 2019 at 1:07 PM Hu Bert <revirii at googlemail.com> wrote: >>> > > >>> > > Hi Hari, >>> > > >>> > > thx for the hint. Do you know when this will be fixed? Is a downgrade >>> > > 5.4 -> 5.3 a possibility to fix this? >>> > > >>> > > Hubert >>> > > >>> > > Am Di., 5. M?rz 2019 um 08:32 Uhr schrieb Hari Gowtham < >>> hgowtham at redhat.com>: >>> > > > >>> > > > Hi, >>> > > > >>> > > > This is a known issue we are working on. >>> > > > As the checksum differs between the updated and non updated node, >>> the >>> > > > peers are getting rejected. >>> > > > The bricks aren't coming because of the same issue. >>> > > > >>> > > > More about the issue: >>> https://bugzilla.redhat.com/show_bug.cgi?id=1685120 >>> > > > >>> > > > On Tue, Mar 5, 2019 at 12:56 PM Hu Bert <revirii at googlemail.com> >>> wrote: >>> > > > > >>> > > > > Interestingly: gluster volume status misses gluster1, while heal >>> > > > > statistics show gluster1: >>> > > > > >>> > > > > gluster volume status workdata >>> > > > > Status of volume: workdata >>> > > > > Gluster process TCP Port RDMA Port >>> Online Pid >>> > > > > >>> ------------------------------------------------------------------------------ >>> > > > > Brick gluster2:/gluster/md4/workdata 49153 0 >>> Y 1723 >>> > > > > Brick gluster3:/gluster/md4/workdata 49153 0 >>> Y 2068 >>> > > > > Self-heal Daemon on localhost N/A N/A >>> Y 1732 >>> > > > > Self-heal Daemon on gluster3 N/A N/A >>> Y 2077 >>> > > > > >>> > > > > vs. >>> > > > > >>> > > > > gluster volume heal workdata statistics heal-count >>> > > > > Gathering count of entries to be healed on volume workdata has >>> been successful >>> > > > > >>> > > > > Brick gluster1:/gluster/md4/workdata >>> > > > > Number of entries: 0 >>> > > > > >>> > > > > Brick gluster2:/gluster/md4/workdata >>> > > > > Number of entries: 10745 >>> > > > > >>> > > > > Brick gluster3:/gluster/md4/workdata >>> > > > > Number of entries: 10744 >>> > > > > >>> > > > > Am Di., 5. M?rz 2019 um 08:18 Uhr schrieb Hu Bert < >>> revirii at googlemail.com>: >>> > > > > > >>> > > > > > Hi Miling, >>> > > > > > >>> > > > > > well, there are such entries, but those haven't been a problem >>> during >>> > > > > > install and the last kernel update+reboot. The entries look >>> like: >>> > > > > > >>> > > > > > PUBLIC_IP gluster2.alpserver.de gluster2 >>> > > > > > >>> > > > > > 192.168.0.50 gluster1 >>> > > > > > 192.168.0.51 gluster2 >>> > > > > > 192.168.0.52 gluster3 >>> > > > > > >>> > > > > > 'ping gluster2' resolves to LAN IP; I removed the last entry >>> in the >>> > > > > > 1st line, did a reboot ... no, didn't help. From >>> > > > > > /var/log/glusterfs/glusterd.log >>> > > > > > on gluster 2: >>> > > > > > >>> > > > > > [2019-03-05 07:04:36.188128] E [MSGID: 106010] >>> > > > > > [glusterd-utils.c:3483:glusterd_compare_friend_volume] >>> 0-management: >>> > > > > > Version of Cksums persistent differ. local cksum = 3950307018, >>> remote >>> > > > > > cksum = 455409345 on peer gluster1 >>> > > > > > [2019-03-05 07:04:36.188314] I [MSGID: 106493] >>> > > > > > [glusterd-handler.c:3843:glusterd_xfer_friend_add_resp] >>> 0-glusterd: >>> > > > > > Responded to gluster1 (0), ret: 0, op_ret: -1 >>> > > > > > >>> > > > > > Interestingly there are no entries in the brick logs of the >>> rejected >>> > > > > > server. Well, not surprising as no brick process is running. >>> The >>> > > > > > server gluster1 is still in rejected state. >>> > > > > > >>> > > > > > 'gluster volume start workdata force' starts the brick process >>> on >>> > > > > > gluster1, and some heals are happening on gluster2+3, but via >>> 'gluster >>> > > > > > volume status workdata' the volumes still aren't complete. >>> > > > > > >>> > > > > > gluster1: >>> > > > > > >>> ------------------------------------------------------------------------------ >>> > > > > > Brick gluster1:/gluster/md4/workdata 49152 0 >>> Y 2523 >>> > > > > > Self-heal Daemon on localhost N/A N/A >>> Y 2549 >>> > > > > > >>> > > > > > gluster2: >>> > > > > > Gluster process TCP Port RDMA >>> Port Online Pid >>> > > > > > >>> ------------------------------------------------------------------------------ >>> > > > > > Brick gluster2:/gluster/md4/workdata 49153 0 >>> Y 1723 >>> > > > > > Brick gluster3:/gluster/md4/workdata 49153 0 >>> Y 2068 >>> > > > > > Self-heal Daemon on localhost N/A N/A >>> Y 1732 >>> > > > > > Self-heal Daemon on gluster3 N/A N/A >>> Y 2077 >>> > > > > > >>> > > > > > >>> > > > > > Hubert >>> > > > > > >>> > > > > > Am Di., 5. M?rz 2019 um 07:58 Uhr schrieb Milind Changire < >>> mchangir at redhat.com>: >>> > > > > > > >>> > > > > > > There are probably DNS entries or /etc/hosts entries with >>> the public IP Addresses that the host names (gluster1, gluster2, gluster3) >>> are getting resolved to. >>> > > > > > > /etc/resolv.conf would tell which is the default domain >>> searched for the node names and the DNS servers which respond to the >>> queries. >>> > > > > > > >>> > > > > > > >>> > > > > > > On Tue, Mar 5, 2019 at 12:14 PM Hu Bert < >>> revirii at googlemail.com> wrote: >>> > > > > > >> >>> > > > > > >> Good morning, >>> > > > > > >> >>> > > > > > >> i have a replicate 3 setup with 2 volumes, running on >>> version 5.3 on >>> > > > > > >> debian stretch. This morning i upgraded one server to >>> version 5.4 and >>> > > > > > >> rebooted the machine; after the restart i noticed that: >>> > > > > > >> >>> > > > > > >> - no brick process is running >>> > > > > > >> - gluster volume status only shows the server itself: >>> > > > > > >> gluster volume status workdata >>> > > > > > >> Status of volume: workdata >>> > > > > > >> Gluster process TCP Port RDMA >>> Port Online Pid >>> > > > > > >> >>> ------------------------------------------------------------------------------ >>> > > > > > >> Brick gluster1:/gluster/md4/workdata N/A N/A >>> N N/A >>> > > > > > >> NFS Server on localhost N/A N/A >>> N N/A >>> > > > > > >> >>> > > > > > >> - gluster peer status on the server >>> > > > > > >> gluster peer status >>> > > > > > >> Number of Peers: 2 >>> > > > > > >> >>> > > > > > >> Hostname: gluster3 >>> > > > > > >> Uuid: c7b4a448-ca6a-4051-877f-788f9ee9bc4a >>> > > > > > >> State: Peer Rejected (Connected) >>> > > > > > >> >>> > > > > > >> Hostname: gluster2 >>> > > > > > >> Uuid: 162fea82-406a-4f51-81a3-e90235d8da27 >>> > > > > > >> State: Peer Rejected (Connected) >>> > > > > > >> >>> > > > > > >> - gluster peer status on the other 2 servers: >>> > > > > > >> gluster peer status >>> > > > > > >> Number of Peers: 2 >>> > > > > > >> >>> > > > > > >> Hostname: gluster1 >>> > > > > > >> Uuid: 9a360776-7b58-49ae-831e-a0ce4e4afbef >>> > > > > > >> State: Peer Rejected (Connected) >>> > > > > > >> >>> > > > > > >> Hostname: gluster3 >>> > > > > > >> Uuid: c7b4a448-ca6a-4051-877f-788f9ee9bc4a >>> > > > > > >> State: Peer in Cluster (Connected) >>> > > > > > >> >>> > > > > > >> I noticed that, in the brick logs, i see that the public IP >>> is used >>> > > > > > >> instead of the LAN IP. brick logs from one of the volumes: >>> > > > > > >> >>> > > > > > >> rejected node: https://pastebin.com/qkpj10Sd >>> > > > > > >> connected nodes: https://pastebin.com/8SxVVYFV >>> > > > > > >> >>> > > > > > >> Why is the public IP suddenly used instead of the LAN IP? >>> Killing all >>> > > > > > >> gluster processes and rebooting (again) didn't help. >>> > > > > > >> >>> > > > > > >> >>> > > > > > >> Thx, >>> > > > > > >> Hubert >>> > > > > > >> _______________________________________________ >>> > > > > > >> Gluster-users mailing list >>> > > > > > >> Gluster-users at gluster.org >>> > > > > > >> https://lists.gluster.org/mailman/listinfo/gluster-users >>> > > > > > > >>> > > > > > > >>> > > > > > > >>> > > > > > > -- >>> > > > > > > Milind >>> > > > > > > >>> > > > > _______________________________________________ >>> > > > > Gluster-users mailing list >>> > > > > Gluster-users at gluster.org >>> > > > > https://lists.gluster.org/mailman/listinfo/gluster-users >>> > > > >>> > > > >>> > > > >>> > > > -- >>> > > > Regards, >>> > > > Hari Gowtham. >>> > >>> > >>> > >>> > -- >>> > Regards, >>> > Hari Gowtham. >>> _______________________________________________ >>> Gluster-users mailing list >>> Gluster-users at gluster.org >>> https://lists.gluster.org/mailman/listinfo/gluster-users >> >>-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20190306/02e1210b/attachment.html>