Russell Purinton
2016-Mar-23 02:12 UTC
[Gluster-users] Peer probe succeeded, but "not in 'Peer in Cluster' state"
If the subnet mask is wrong on 122.11 it may forward all traffic to the default gateway. The default gateway may be configured to NAT traffic from the LAN, so the response packet would be seen by .10 as coming from .1. Russ> On Mar 22, 2016, at 10:10 PM, Joshua J. Kugler <joshua at azariah.com> wrote: > > Hmm...I'm wondering if my networking is messed up some how. But why would that > cause host b to see host a as the gateway and not as the proper IP? > > j > > On Tuesday, March 22, 2016 17:33:03 Joshua J. Kugler wrote: >> On Tuesday, March 22, 2016 18:27:46 Atin Mukherjee wrote: >>> This is the problem, peer handshaking hasn't finished yet. To get to >>> know the reason I'd need to get the glusterd log file from >>> 192.168.122.10. >> >> Here's the log from the other machine (.10). >> >>> As a workaround can you do the following? >>> >>> 1. From node 1 open /var/lib/glusterd/peers/<uuid file>, modify state=3 >>> 2. Repeat step 1 for node 2 as well if state is different >>> 3. restart both the glusterd instances. >> >> Hmm, just realized this: >> >> This is "Box A" >> [root at vmserver-a peers]# ls -l >> total 4 >> -rw------- 1 root root 74 Mar 22 17:15 fe722085-ac0f-4449-a43f-2dc9dd1fd8fb >> [root at vmserver-a peers]# cat fe722085-ac0f-4449-a43f-2dc9dd1fd8fb >> uuid=fe722085-ac0f-4449-a43f-2dc9dd1fd8fb >> state=3 >> hostname1=192.168.122.1 >> >> This is "Box B" (from where I'm running the gluster create command: >> [root at vmserver-b peers]# ls -l >> total 4 >> -rw------- 1 root root 75 Mar 22 17:15 d8e1d7a0-077a-4a50-93f6-d3922e3b96b9 >> [root at vmserver-b peers]# cat d8e1d7a0-077a-4a50-93f6-d3922e3b96b9 >> uuid=d8e1d7a0-077a-4a50-93f6-d3922e3b96b9 >> state=3 >> hostname1=192.168.122.10 >> >> Why is the gateway in the peer list? To my knowledge, that's not getting >> added anywhere. This is weird. >> >> Also odd: >> [root at vmserver-a peers]# gluster peer status >> Number of Peers: 1 >> >> Hostname: 192.168.122.1 >> Uuid: fe722085-ac0f-4449-a43f-2dc9dd1fd8fb >> State: Peer in Cluster (Disconnected) >> >> [root at vmserver-b peers]# gluster peer status >> Number of Peers: 1 >> >> Hostname: 192.168.122.10 >> Uuid: d8e1d7a0-077a-4a50-93f6-d3922e3b96b9 >> State: Peer in Cluster (Connected) >> >> So, somehow, on vmserver-a is getting 192.168.122.1 in its peer list instead >> of 192.168.122.11 >> >> Very strange. >> >> j >> >>> ~Atin >>> >>>>> Also send the glusterd log of the node where the commands have failed. >>>> >>>> The two logs are attached. The peer status says connected. The log file >>>> says "FAILED : Host 192.168.122.10 is not in 'Peer in Cluster' state" >>>> I'm confused. :) >>>> >>>> Thanks for your help on this! >>>> >>>> j >>>> >>>> >>>> >>>> _______________________________________________ >>>> Gluster-users mailing list >>>> Gluster-users at gluster.org >>>> http://www.gluster.org/mailman/listinfo/gluster-users > > -- > Joshua J. Kugler - Fairbanks, Alaska > Azariah Enterprises - Programming and Website Design > joshua at azariah.com - Jabber: pedahzur at gmail.com > PGP Key: http://pgp.mit.edu/ ID 0x73B13B6A > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160322/10a456bb/attachment.html>
Joshua J. Kugler
2016-Mar-23 03:07 UTC
[Gluster-users] Peer probe succeeded, but "not in 'Peer in Cluster' state"
On Tuesday, March 22, 2016 22:12:00 Russell Purinton wrote:> If the subnet mask is wrong on 122.11 it may forward all traffic to the > default gateway. The default gateway may be configured to NAT traffic from > the LAN, so the response packet would be seen by .10 as coming from .1.Thanks for the suggestion, but sadly that wasn't it. This is the route table on both hosts: Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 192.168.130.0 0.0.0.0 255.255.255.0 U 0 0 0 br1 192.168.122.0 0.0.0.0 255.255.255.0 U 0 0 0 br0 0.0.0.0 192.168.122.1 0.0.0.0 UG 0 0 0 br0 I haven't ruled it out, though. I am going to do some more log investigation. j -- Joshua J. Kugler - Fairbanks, Alaska Azariah Enterprises - Programming and Website Design joshua at azariah.com - Jabber: pedahzur at gmail.com PGP Key: http://pgp.mit.edu/ ID 0x73B13B6A
Joshua J. Kugler
2016-Mar-23 03:12 UTC
[Gluster-users] Peer probe succeeded, but "not in 'Peer in Cluster' state"
On Tuesday, March 22, 2016 22:12:00 Russell Purinton wrote:> If the subnet mask is wrong on 122.11 it may forward all traffic to the > default gateway. The default gateway may be configured to NAT traffic from > the LAN, so the response packet would be seen by .10 as coming from .1.So, it turns out the subnet isn't wrong, but for some reason, it's still routing through the gateway, and appears to be coming from .1, instead of .11. I'm not sure why. This is a libvirt network, configured thus: <network> <name>default</name> <uuid>f137a5c4-1dd2-453a-a6e6-c161f2918d41</uuid> <forward mode='route'/> <bridge name='virbr0' stp='on' delay='0'/> <mac address='52:54:00:42:87:a9'/> <ip address='192.168.122.1' netmask='255.255.255.0'> <dhcp> <range start='192.168.122.2' end='192.168.122.254'/> </dhcp> </ip> </network> When this was working, I was using forward mode=nat, but then two different libvirt networks couldn't talk to each other. The two machines are on the same segment, on the same virtual switch. I'm not sure why they are getting routing through the gateway. Off to do more troubleshooting! :) j -- Joshua J. Kugler - Fairbanks, Alaska Azariah Enterprises - Programming and Website Design joshua at azariah.com - Jabber: pedahzur at gmail.com PGP Key: http://pgp.mit.edu/ ID 0x73B13B6A