We are testing tinc inside Google Compute within a single region and an external region. Two boxes are created as follows: /etc/tinc/test/tinc_test_1 Subnet = 10.240.0.0/16 Subnet = 10.240.0.4/32 Address = 104.154.59.151 /etc/tinc/test/tinc_test_2 Subnet = 10.240.0.0/16 Subnet = 10.240.0.5/32 Address = 104.197.132.141 /etc/tinc/test/tinc.conf Name = $HOST AddressFamily = ipv4 Interface = tun0 LocalDiscovery = yes Those 2 boxes are in the same subnet and have addresses of 10.240.0.4 and 10.240.0.5, respectively, on their eth0 interface. Port 655 on tcp and udp is open to the world. The tinc_test_2 box has a ConnectTo of tinc_test_1. When tinc_test_2 is started, it prints out: UDP address of tinc_test_1 set to 104.154.59.151 port 655 UDP address of tinc_test_1 set to 10.240.0.4 port 655 UDP address of tinc_test_1 set to 104.154.59.151 port 655 UDP address of tinc_test_1 set to 10.240.0.4 port 655 repeatedly for a minute or so before finally settling on 10.240.0.4. Is there a reason it's flip flopping? Is that expected? Am I doing something wrong? Additionally, we have multiple Google Compute regions with their own subnets and external DCs with their own subnets and we'd like to install tinc on all servers but keep inner-Google traffic to the internal IPs and not over external IPs since it's an order of magnitude cheaper. My first thinking is a hub and spoke model. We have 2 boxes in each subnet that have port 655 open to the world, and all the other servers have 655 open to internal ips only. With LocalDiscovery (as well as IndirectData = yes on "non-public" servers) this works work pretty well, as far as I can tell. But it wouldn't solve the inner-Google traffic between subnets since Google Subnet0 would talk over public to Google Subnet1. What's the best way of doing something like this? I was thinking maybe 2 instances of tinc on the "public" boxes, but Google servers only have a single interface, eth0, that has the internal IP, so I couldn't listen on the external and internal IPs separately. Thanks! -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.tinc-vpn.org/pipermail/tinc/attachments/20170214/e08611e8/attachment.html>
Etienne Dechamps
2017-Feb-14 18:22 UTC
LocalDiscovery flip flopping and network design tips
Can you specify which version of tinc you're using? There are vast differences in the way LocalDiscovery works between 1.0 and 1.1. The former uses broadcast, the latter unicast to explicitly advertised local addresses. You say that tinc_test_1's eth0 interface is configured with 10.240.0.4, and tinc_test_2's eth0 interface is configured with 10.240.0.5. How are the public addresses (104.154.59.151 and 104.197.132.141) configured? Are they simply forwarded by some other router somewhere? Or are they configured on some other network interface on the machine? If you're using tinc 1.1, please provide the output of "tinc dump edges" while your network is up. On 14 February 2017 at 16:21, James Hartig <james at levenlabs.com> wrote:> We are testing tinc inside Google Compute within a single region and an > external region. Two boxes are created as follows: > /etc/tinc/test/tinc_test_1 > Subnet = 10.240.0.0/16 > Subnet = 10.240.0.4/32 > Address = 104.154.59.151 > > /etc/tinc/test/tinc_test_2 > Subnet = 10.240.0.0/16 > Subnet = 10.240.0.5/32 > Address = 104.197.132.141 > > /etc/tinc/test/tinc.conf > Name = $HOST > AddressFamily = ipv4 > Interface = tun0 > LocalDiscovery = yes > > Those 2 boxes are in the same subnet and have addresses of 10.240.0.4 and > 10.240.0.5, respectively, on their eth0 interface. Port 655 on tcp and udp > is open to the world. The tinc_test_2 box has a ConnectTo of tinc_test_1. > When tinc_test_2 is started, it prints out: > UDP address of tinc_test_1 set to 104.154.59.151 port 655 > UDP address of tinc_test_1 set to 10.240.0.4 port 655 > UDP address of tinc_test_1 set to 104.154.59.151 port 655 > UDP address of tinc_test_1 set to 10.240.0.4 port 655 > repeatedly for a minute or so before finally settling on 10.240.0.4. > > Is there a reason it's flip flopping? Is that expected? Am I doing > something wrong? > > Additionally, we have multiple Google Compute regions with their own > subnets and external DCs with their own subnets and we'd like to install > tinc on all servers but keep inner-Google traffic to the internal IPs and > not over external IPs since it's an order of magnitude cheaper. My first > thinking is a hub and spoke model. We have 2 boxes in each subnet that have > port 655 open to the world, and all the other servers have 655 open to > internal ips only. With LocalDiscovery (as well as IndirectData = yes on > "non-public" servers) this works work pretty well, as far as I can tell. > But it wouldn't solve the inner-Google traffic between subnets since Google > Subnet0 would talk over public to Google Subnet1. What's the best way of > doing something like this? I was thinking maybe 2 instances of tinc on the > "public" boxes, but Google servers only have a single interface, eth0, that > has the internal IP, so I couldn't listen on the external and internal IPs > separately. > > Thanks! > > > > > > _______________________________________________ > tinc mailing list > tinc at tinc-vpn.org > https://www.tinc-vpn.org/cgi-bin/mailman/listinfo/tinc > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.tinc-vpn.org/pipermail/tinc/attachments/20170214/2ea67994/attachment.html>
On Tue, Feb 14, 2017 at 11:21:34AM -0500, James Hartig wrote:> Those 2 boxes are in the same subnet and have addresses of 10.240.0.4 and > 10.240.0.5, respectively, on their eth0 interface. Port 655 on tcp and udp > is open to the world. The tinc_test_2 box has a ConnectTo of tinc_test_1. > When tinc_test_2 is started, it prints out: > UDP address of tinc_test_1 set to 104.154.59.151 port 655 > UDP address of tinc_test_1 set to 10.240.0.4 port 655 > UDP address of tinc_test_1 set to 104.154.59.151 port 655 > UDP address of tinc_test_1 set to 10.240.0.4 port 655 > repeatedly for a minute or so before finally settling on 10.240.0.4. > > Is there a reason it's flip flopping? Is that expected? Am I doing > something wrong?No, you are not doing anything wrong. Although I've not seen this kind of flip-flopping behavior myself, it is possible this behaviour occurs, although it should only happen in the first 10 seconds or so. When LocalDiscovery is enabled, tinc tries to send probe packets to both the address it learned from its TCP connections and to the local network. When receiving a valid packet, it notes the source address of that packet. If it is different from the source address of the previous valid UDP packet, a log message is printed about it.> Additionally, we have multiple Google Compute regions with their own > subnets and external DCs with their own subnets and we'd like to install > tinc on all servers but keep inner-Google traffic to the internal IPs and > not over external IPs since it's an order of magnitude cheaper. My first > thinking is a hub and spoke model. We have 2 boxes in each subnet that have > port 655 open to the world, and all the other servers have 655 open to > internal ips only. With LocalDiscovery (as well as IndirectData = yes on > "non-public" servers) this works work pretty well, as far as I can tell. > But it wouldn't solve the inner-Google traffic between subnets since Google > Subnet0 would talk over public to Google Subnet1. What's the best way of > doing something like this? I was thinking maybe 2 instances of tinc on the > "public" boxes, but Google servers only have a single interface, eth0, that > has the internal IP, so I couldn't listen on the external and internal IPs > separately.You can use BindToAddress to have tinc bind to a specific IP address. So you can have two tinc daemons, one binding to the internal IP address, and another to the external IP address, even if they are residing on the same network interface. Would that help? -- Met vriendelijke groet / with kind regards, Guus Sliepen <guus at tinc-vpn.org> -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: Digital signature URL: <http://www.tinc-vpn.org/pipermail/tinc/attachments/20170214/99f64ce3/attachment.sig>
On Tue, Feb 14, 2017 at 1:22 PM, Etienne Dechamps <etienne at edechamps.fr> wrote:> > Can you specify which version of tinc you're using? There are vast differences in the way LocalDiscovery works between 1.0 and 1.1. The former uses broadcast, the latter unicast to explicitly advertised local addresses.I'm using tinc 1.1pre14. I noticed there's an option, LocalDiscoveryAddress, but it was unclear what I should set that to. Should it be the LAN address that I want them to communicate over (i.e. 10.240.0.4 and 10.240.0.5)?> You say that tinc_test_1's eth0 interface is configured with 10.240.0.4, and tinc_test_2's eth0 interface is configured with 10.240.0.5. How are the public addresses (104.154.59.151 and 104.197.132.141) configured? Are they simply forwarded by some other router somewhere? Or are they configured on some other network interface on the machine?Yeah, Google forwards the public-addressed packets to eth0.> If you're using tinc 1.1, please provide the output of "tinc dump edges" while your network is up.The output from "tinc dump edges" from tinc_test_1 (after stabalized) is: tinc_test_1 to tinc_test_2 at 104.197.132.141 port 655 local 10.240.0.4 port 655 options 700000c weight 1 tinc_test_2 to tinc_test_1 at 104.154.59.151 port 655 local 10.240.0.5 port 655 options 700000c weight 1 The output on tinc_test_2 (after stabalized) is: tinc_test_1 to tinc_test_2 at 104.197.132.141 port 655 local 10.240.0.4 port 655 options 700000c weight 1 tinc_test_2 to tinc_test_1 at 104.154.59.151 port 655 local 10.240.0.5 port 655 options 700000c weight 1 I brought down tinc_test_2 and brought it back up and the output from "tinc dump edges" didn't seem to be any different while it was flip flopping.> On 14 February 2017 at 16:21, James Hartig <james at levenlabs.com> wrote: >> >> We are testing tinc inside Google Compute within a single region and an external region. Two boxes are created as follows: >> /etc/tinc/test/tinc_test_1 >> Subnet = 10.240.0.0/16 >> Subnet = 10.240.0.4/32 >> Address = 104.154.59.151 >> >> /etc/tinc/test/tinc_test_2 >> Subnet = 10.240.0.0/16 >> Subnet = 10.240.0.5/32 >> Address = 104.197.132.141 >> >> /etc/tinc/test/tinc.conf >> Name = $HOST >> AddressFamily = ipv4 >> Interface = tun0 >> LocalDiscovery = yes >> >> Those 2 boxes are in the same subnet and have addresses of 10.240.0.4 and 10.240.0.5, respectively, on their eth0 interface. Port 655 on tcp and udp is open to the world. The tinc_test_2 box has a ConnectTo of tinc_test_1. When tinc_test_2 is started, it prints out: >> UDP address of tinc_test_1 set to 104.154.59.151 port 655 >> UDP address of tinc_test_1 set to 10.240.0.4 port 655 >> UDP address of tinc_test_1 set to 104.154.59.151 port 655 >> UDP address of tinc_test_1 set to 10.240.0.4 port 655 >> repeatedly for a minute or so before finally settling on 10.240.0.4. >> >> Is there a reason it's flip flopping? Is that expected? Am I doing something wrong? >> >> Additionally, we have multiple Google Compute regions with their own subnets and external DCs with their own subnets and we'd like to install tinc on all servers but keep inner-Google traffic to the internal IPs and not over external IPs since it's an order of magnitude cheaper. My first thinking is a hub and spoke model. We have 2 boxes in each subnet that have port 655 open to the world, and all the other servers have 655 open to internal ips only. With LocalDiscovery (as well as IndirectData = yes on "non-public" servers) this works work pretty well, as far as I can tell. But it wouldn't solve the inner-Google traffic between subnets since Google Subnet0 would talk over public to Google Subnet1. What's the best way of doing something like this? I was thinking maybe 2 instances of tinc on the "public" boxes, but Google servers only have a single interface, eth0, that has the internal IP, so I couldn't listen on the external and internal IPs separately. >> >> Thanks! >> >> >> >> >> >> _______________________________________________ >> tinc mailing list >> tinc at tinc-vpn.org >> https://www.tinc-vpn.org/cgi-bin/mailman/listinfo/tinc >>
On Tue, Feb 14, 2017 at 1:46 PM, Guus Sliepen <guus at tinc-vpn.org> wrote:> On Tue, Feb 14, 2017 at 11:21:34AM -0500, James Hartig wrote: > >> Those 2 boxes are in the same subnet and have addresses of 10.240.0.4 and >> 10.240.0.5, respectively, on their eth0 interface. Port 655 on tcp and udp >> is open to the world. The tinc_test_2 box has a ConnectTo of tinc_test_1. >> When tinc_test_2 is started, it prints out: >> UDP address of tinc_test_1 set to 104.154.59.151 port 655 >> UDP address of tinc_test_1 set to 10.240.0.4 port 655 >> UDP address of tinc_test_1 set to 104.154.59.151 port 655 >> UDP address of tinc_test_1 set to 10.240.0.4 port 655 >> repeatedly for a minute or so before finally settling on 10.240.0.4. >> >> Is there a reason it's flip flopping? Is that expected? Am I doing >> something wrong? > > No, you are not doing anything wrong. Although I've not seen this kind > of flip-flopping behavior myself, it is possible this behaviour occurs, > although it should only happen in the first 10 seconds or so. When > LocalDiscovery is enabled, tinc tries to send probe packets to both the > address it learned from its TCP connections and to the local network. > When receiving a valid packet, it notes the source address of that > packet. If it is different from the source address of the previous valid > UDP packet, a log message is printed about it.When you say "and to the local network" what IP does it try to send to on the local network? The subnet address? Is this configurable with the LocalDiscoveryAddress? For instance if I have a host: /etc/tinc/test/tinc_test_5 Subnet = 10.80.0.6/32 Address = 104.154.59.152 Will it try to contact 10.80.0.6 over the local network, or if I specify a LocalDiscoveryAddress on tinc_test_5 of 10.240.0.6, will it use 10.240.0.6 instead? Apologizes for my confusion. Since I'm setting Subnet to be the internal IP, is there any difference in that versus LocalDiscoveryAddress?>> Additionally, we have multiple Google Compute regions with their own >> subnets and external DCs with their own subnets and we'd like to install >> tinc on all servers but keep inner-Google traffic to the internal IPs and >> not over external IPs since it's an order of magnitude cheaper. My first >> thinking is a hub and spoke model. We have 2 boxes in each subnet that have >> port 655 open to the world, and all the other servers have 655 open to >> internal ips only. With LocalDiscovery (as well as IndirectData = yes on >> "non-public" servers) this works work pretty well, as far as I can tell. >> But it wouldn't solve the inner-Google traffic between subnets since Google >> Subnet0 would talk over public to Google Subnet1. What's the best way of >> doing something like this? I was thinking maybe 2 instances of tinc on the >> "public" boxes, but Google servers only have a single interface, eth0, that >> has the internal IP, so I couldn't listen on the external and internal IPs >> separately. > > You can use BindToAddress to have tinc bind to a specific IP address. So > you can have two tinc daemons, one binding to the internal IP address, > and another to the external IP address, even if they are residing on the > same network interface. Would that help?I can't use BindToAddress as far as I know because there isn't an interface on the box with the external IP. The servers only have 1 interface, eth0, that has a single IP, eg 10.240.0.4, and Google forwards packets to their public IP, eg 104.154.59.151, to the internal IP, eg 10.240.0.4, so the server itself doesn't actually know its public IP. I think if my understanding above of the local network stuff is correct, and since it's unicast in 1.1, then it should work fine. Google Subnet0 and Google Subnet1 can talk to each other's private IPs, so theoretically discovery should work and it should use their private IPs.> -- > Met vriendelijke groet / with kind regards, > Guus Sliepen <guus at tinc-vpn.org> > > _______________________________________________ > tinc mailing list > tinc at tinc-vpn.org > https://www.tinc-vpn.org/cgi-bin/mailman/listinfo/tinc
Reasonably Related Threads
- LocalDiscovery flip flopping and network design tips
- LocalDiscovery flip flopping and network design tips
- LocalDiscovery flip flopping and network design tips
- LocalDiscovery flip flopping and network design tips
- LocalDiscovery flip flopping and network design tips