Sure, let me reply all here for my finding. @Lars @Guus A’s tinc.conf: Name = bright AddressFamily = ipv4 ConnectTo = aly_hk A’s tinc-up: #!/bin/sh ifconfig $INTERFACE 10.0.0.110 netmask 255.255.255.0 A’s host config: Subnet = 10.0.0.110/32 (VPN address) Subnet = 192.168.31.0/24 (LAN address) IndirectData = yes (enabled for every tinc nodes) The node aly_hk (vpn address 10.0.0.3) connects with other Tinc nodes: B, C, D; and Subnet = 8.8.8.8 been added to aly_hk, B, C, D’s host config: When tinc daemon get up on A(together with some manual scripts for destination to 8.8.8.8), the route table looks like below: 10.0.0.0/24 dev tun0 10.0.0.3 dev tun0 8.8.8.8 via 10.0.0.3 dev tun0 The key point to understand here is how the route happens for traffic head for 8.8.8.8, earlier I thought it can ONLY send to 10.0.0.3 where aly_hk(10.0.0.3) delivery to 8.8.8.8 locally(because aly_hk has the default route for 8.8.8.8, no need to forward to other tinc nodes), but now, I found A will learn multiple 8.8.8.8 subnets from aly_hk, and for example, if I set B’s Subnet =8.8.8.8#9, then A ping 8.8.8.8 will send to aly_hk, and aly_hk will foward to B for final delivery. In summary, the ip route add default via <X> dev $INTERFACE, the X here may or may not be the final exit for VPN, it depends on the Subnet on X is whether preferred(even though the routing table has the entry for exit), it just indicate as the next hop(if only ConnectTo = X and IndirectData =yes). But if I remove the IndirectData, default to no; then I guess for the above use case, maybe A will send traffic directly to B, and B do the final delivery, and in this case, the Tinc packet forwarding totally not follow the route table, where the traffic looks will send to 10.0.0.3 for the next hop, but it isn’t. Is above right?> On 2 May 2017, at 6:33 PM, Lars Kruse <lists at sumpfralle.de> wrote: > > Hello, > > > Am Tue, 2 May 2017 09:16:53 +0800 > schrieb Bright Zhao <startryst at gmail.com>: > >> [..] >> Since A have both ConnectTo to B and C(To D through C), and the default >> gateway set to C, why A chose B instead of D's default route? > > this is starting to become a little bit confusing. Let us clear this up with > facts :) > > Could you please share your routing table with us? > Additionally the subnet definitions of each relevant tinc node would be helpful. > > You surely sync all node files on all hosts, right? > (thus all hosts share the same perspective regarding the other node's subnets) > > Cheers, > Lars > > > PS: I think you are making your life harder by mixing two problems: > 1) "does the traffic reach the intended node?" > 2) "does the traffic to a node pass through the right order of other nodes?" > I think you should solve (1) first and only afterwards approach (2). > _______________________________________________ > tinc mailing list > tinc at tinc-vpn.org > https://www.tinc-vpn.org/cgi-bin/mailman/listinfo/tinc
And BTW, the reason I’m asking those question is by default Tinc will try to automate all the underlying connection and tunnel directly as much as possible, which is fantastic, but in my use case, I need to manipulate and control the traffic to go through certain path (instead of as direct as possible), that’s why I turned IndirectData = yes to only tunnel formed under ConnectTo statement. Earlier, my understanding is everything is follow the route table, but according to the case below, sometime it’s not.> On 2 May 2017, at 9:53 PM, Bright Zhao <startryst at gmail.com> wrote: > > Sure, let me reply all here for my finding. @Lars @Guus > > A’s tinc.conf: > Name = bright > AddressFamily = ipv4 > ConnectTo = aly_hk > > A’s tinc-up: > #!/bin/sh > ifconfig $INTERFACE 10.0.0.110 netmask 255.255.255.0 > > A’s host config: > Subnet = 10.0.0.110/32 (VPN address) > Subnet = 192.168.31.0/24 (LAN address) > IndirectData = yes (enabled for every tinc nodes) > > The node aly_hk (vpn address 10.0.0.3) connects with other Tinc nodes: B, C, D; and Subnet = 8.8.8.8 been added to aly_hk, B, C, D’s host config: > > When tinc daemon get up on A(together with some manual scripts for destination to 8.8.8.8), the route table looks like below: > 10.0.0.0/24 dev tun0 > 10.0.0.3 dev tun0 > 8.8.8.8 via 10.0.0.3 dev tun0 > > The key point to understand here is how the route happens for traffic head for 8.8.8.8, earlier I thought it can ONLY send to 10.0.0.3 where aly_hk(10.0.0.3) delivery to 8.8.8.8 locally(because aly_hk has the default route for 8.8.8.8, no need to forward to other tinc nodes), but now, I found A will learn multiple 8.8.8.8 subnets from aly_hk, and for example, if I set B’s Subnet =8.8.8.8#9, then A ping 8.8.8.8 will send to aly_hk, and aly_hk will foward to B for final delivery. > > In summary, the ip route add default via <X> dev $INTERFACE, the X here may or may not be the final exit for VPN, it depends on the Subnet on X is whether preferred(even though the routing table has the entry for exit), it just indicate as the next hop(if only ConnectTo = X and IndirectData =yes). > > But if I remove the IndirectData, default to no; then I guess for the above use case, maybe A will send traffic directly to B, and B do the final delivery, and in this case, the Tinc packet forwarding totally not follow the route table, where the traffic looks will send to 10.0.0.3 for the next hop, but it isn’t. > > Is above right? > > >> On 2 May 2017, at 6:33 PM, Lars Kruse <lists at sumpfralle.de> wrote: >> >> Hello, >> >> >> Am Tue, 2 May 2017 09:16:53 +0800 >> schrieb Bright Zhao <startryst at gmail.com>: >> >>> [..] >>> Since A have both ConnectTo to B and C(To D through C), and the default >>> gateway set to C, why A chose B instead of D's default route? >> >> this is starting to become a little bit confusing. Let us clear this up with >> facts :) >> >> Could you please share your routing table with us? >> Additionally the subnet definitions of each relevant tinc node would be helpful. >> >> You surely sync all node files on all hosts, right? >> (thus all hosts share the same perspective regarding the other node's subnets) >> >> Cheers, >> Lars >> >> >> PS: I think you are making your life harder by mixing two problems: >> 1) "does the traffic reach the intended node?" >> 2) "does the traffic to a node pass through the right order of other nodes?" >> I think you should solve (1) first and only afterwards approach (2). >> _______________________________________________ >> tinc mailing list >> tinc at tinc-vpn.org >> https://www.tinc-vpn.org/cgi-bin/mailman/listinfo/tinc >
Hi, Am Tue, 2 May 2017 21:53:15 +0800 schrieb Bright Zhao <startryst at gmail.com>:> [..] > The key point to understand here is how the route happens for traffic head > for 8.8.8.8, earlier I thought it can ONLY send to 10.0.0.3 where > aly_hk(10.0.0.3) delivery to 8.8.8.8 locally(because aly_hk has the default > route for 8.8.8.8, no need to forward to other tinc nodes), but now, I found > A will learn multiple 8.8.8.8 subnets from aly_hk, and for example, if I set > B’s Subnet =8.8.8.8#9, then A ping 8.8.8.8 will send to aly_hk, and aly_hk > will foward to B for final delivery.I think, we are getting closer to the source of misunderstanding. Each node (i.e. specifically tinc and the connected scripts - nobody else) learns which subnets are availabe via other nodes. The kernel (or whoever manages routing decisions) is unaware of tinc's concept of subnets. Thus routing tables, handling of conflicting routes (via aly_hk, A or ...) are up to you (handled via subnet-up scripts). Thus if A thinks, it should send packets for 8.8.8.8 to aly_hk, but aly_hk thinks, that B would be a better target, then you will need to adjust your scripts to prevent this situation. Maybe you want to add the "metric" attribute to you routes based on the "weight" of each subnet? Maybe you want to use a routing daemon (e.g. olsrd) for handling different routes (this would be a completely different approach, but still usable with tinc). Or maybe you just ignore subnets from other nodes, if the local nodes announces the same subnets? Think of tinc as something that transports traffic and maybe filters some packets (nodes only accept traffic for IPs within their own subnet definition). Routing is outside of the scope of tinc - this is up to you via your subnet-up scripts. At least this is my understanding. Cheers, Lars
On Tue, May 02, 2017 at 09:53:15PM +0800, Bright Zhao wrote:> When tinc daemon get up on A(together with some manual scripts for destination to 8.8.8.8), the route table looks like below: > 10.0.0.0/24 dev tun0 > 10.0.0.3 dev tun0 > 8.8.8.8 via 10.0.0.3 dev tun0IPv4 packets have only two addresses in their header: the source and the destination address. If you want to send a packet to 8.8.8.8, the destination address of the packet will always be 8.8.8.8, regardless of how it is forwarded via the network. Imagine that your local node would replace the destination address with 10.0.0.3, then the information about the final destination (8.8.8.8) would be lost. The "via 10.0.0.3" only has an effect on Ethernet networks. Ethernet doesn't know about IPv4 or IPv6, it only knows about MAC addresses. What "via 10.0.0.3" does is that when it has to send a packet to 8.8.8.8, it sets the IPv4 header's destination address to 8.8.8.8, but it sets the Ethernet header's destination address to the MAC address of the host which has the address 10.0.0.3. It figured out the last part by doing ARP resolution. It's perhaps a bit confusing that there is no error message when you try to add such a route, even though the "via 10.0.0.3" part doesn't do anything. It's just ignored for non-Ethernet networks. Also, on some (older) BSD platforms, you could not add a route to a "dev tun0", you could only identify interfaces by their IP address. So there the equivalent of "via" would have an effect, but also not the one you intended. -- Met vriendelijke groet / with kind regards, Guus Sliepen <guus at tinc-vpn.org> -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: Digital signature URL: <http://www.tinc-vpn.org/pipermail/tinc/attachments/20170502/1305d04e/attachment.sig>
Hi, Guus I did some test regarding the points you mentioned below, and yes, you’re right, but some of points may need further adjusted: 1. The destination of IPv4 wouldn’t be changed, Yes I agree, that’s the goal and final destination for the communication. But during the path, it may be encapsulated into another packet(tunnel mode), where the outside IP header is the physical address, but the inner destination remain unchanged, and I think Tinc encapsulates into UDP packet, where the outside S/D IP is the physical adapter IP address, but the whole original packet was encapsulated into the UDP content part. Conceptually, it’s the same as IPSec site-to-site VPN: http://www.unixwiz.net/images/IPSec-ESP-Tunnel-Mode.gif <http://www.unixwiz.net/images/IPSec-ESP-Tunnel-Mode.gif> 2. Regarding the “via 10.0.0.3”, you’re right, it’s only for Ethernet ARP resolution for the next hop. And after my test, my understanding is as below for the routing part of Tinc: a. When tinc get up, the tun0 assigned an IP address by tinc-up, let’s say 10.0.0.100/24, this will add an direct attached route into host’s route table which appears as “10.0.0.0 255.255.255.0 tun0”, which is not deletable; this route will enable traffic head to 10.0.0.0/24 will throw into the tun0 interface b. Also, if there’s any other traffic need to convey by tinc VPN, you just add another route, for example: “ip route 8.8.8.8/32 dev tun0”, and actually we don’t need to specify the via, since we only need throw the traffic into tun0 interface. c. No matter for above a or b, when the traffic send to tun0, from the tunnel’s perspective, the source IP will be 10.0.0.100(since routed through this interface), and the destination IP will be the one I pinged, which is 8.8.8.8 d. After above c, then the tinc need to figure out how to encapsulate the original packet (S/10.0.0.100 > D/8.8.8.8) into the tunnel and send to the other Tinc node. Then I guess Tinc will check “ADD_SUBNET” messages it received from the ConnectTo node(learn the whole network), and try to encapsulate the original packet into UDP packets, send to the node where its subnet of 8.8.8.8 is preferred(weight setting on Subnet) e. If Tinc configured by default, then the local host will try to send UDP packet directly to the preferred host which advertise 8.8.8.8 f. If IndirectData = yes configured everywhere, then the local host will always encapsulate the UDP packet to the ConnectTo/aly_hk (No matter 8.8.8.8 is advertised or not), if aly_hk doesn’t advertise 8.8.8.8, it will check it’s Tinc daemon, to see who advertised 8.8.8.8, and then encapsulate UDP to him as the next step to relay Interestingly, earlier I check the below post for my setup, but now I found, the configuration can be simplified into, because tinc-up will make the 172.16.1.0/24 to $INTERFACE, and for those two default routes, the via is not required as well: #!/bin/sh VPN_GATEWAY=172.16.1.1 ORIGINAL_GATEWAY=`ip route show | grep ^default | cut -d ' ' -f 2-5` ip route add $REMOTEADDRESS $ORIGINAL_GATEWAY ip route add $VPN_GATEWAY dev $INTERFACE ip route add 0.0.0.0/1 via $VPN_GATEWAY dev $INTERFACE ip route add 128.0.0.0/1 via $VPN_GATEWAY dev $INTERFACE https://www.tinc-vpn.org/examples/redirect-gateway/ <https://www.tinc-vpn.org/examples/redirect-gateway/>> On 3 May 2017, at 12:36 AM, Guus Sliepen <guus at tinc-vpn.org> wrote: > > On Tue, May 02, 2017 at 09:53:15PM +0800, Bright Zhao wrote: > >> When tinc daemon get up on A(together with some manual scripts for destination to 8.8.8.8), the route table looks like below: >> 10.0.0.0/24 dev tun0 >> 10.0.0.3 dev tun0 >> 8.8.8.8 via 10.0.0.3 dev tun0 > > IPv4 packets have only two addresses in their header: the source and the > destination address. If you want to send a packet to 8.8.8.8, the > destination address of the packet will always be 8.8.8.8, regardless of > how it is forwarded via the network. > > Imagine that your local node would replace the destination address with > 10.0.0.3, then the information about the final destination (8.8.8.8) > would be lost. > > The "via 10.0.0.3" only has an effect on Ethernet networks. Ethernet > doesn't know about IPv4 or IPv6, it only knows about MAC addresses. What > "via 10.0.0.3" does is that when it has to send a packet to 8.8.8.8, it > sets the IPv4 header's destination address to 8.8.8.8, but it sets the > Ethernet header's destination address to the MAC address of the host > which has the address 10.0.0.3. It figured out the last part by doing > ARP resolution. > > It's perhaps a bit confusing that there is no error message when you try > to add such a route, even though the "via 10.0.0.3" part doesn't do > anything. It's just ignored for non-Ethernet networks. Also, on some > (older) BSD platforms, you could not add a route to a "dev tun0", you > could only identify interfaces by their IP address. So there the > equivalent of "via" would have an effect, but also not the one you > intended. > > -- > Met vriendelijke groet / with kind regards, > Guus Sliepen <guus at tinc-vpn.org> > _______________________________________________ > tinc mailing list > tinc at tinc-vpn.org > https://www.tinc-vpn.org/cgi-bin/mailman/listinfo/tinc-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.tinc-vpn.org/pipermail/tinc/attachments/20170503/90521c30/attachment.html>