Sure, I?ll see if we can narrow it down for you.> On Jun 12, 2015, at 2:25 PM, Etienne Dechamps <etienne at edechamps.fr> wrote: > > That's interesting. I'm using a near-HEAD tinc-1.1 myself and haven't > encountered this problem, but I think that's because I'm using it in > router mode, as opposed to switch mode. > > I'm trying to narrow this down to a recent commit, but I can't find > anything obvious. Did you have a working setup with a previous > tinc-1.1 version? Do you have time to try a git bisect? > > On 11 June 2015 at 23:17, Chris Clements <cclements at outlook.com> wrote: >> Have an interesting issue that's cropped up for us. We have a switch based vpn setup with multiple endpoints, some publicly accessible, some behind hide NAT. What we've noticed is that every machine can reach each other's VPN IP's up to a certain packet size. All machines can successfully ping each other up to 297 bytes. At 298 bytes and beyond, only "forwarded" connections succeed. Specifically, a machine behind NAT connected directly to a public machine via UDP cannot ping it with greater than 297 bytes. In this case ICMP pings are an example, but we see the same behavior with other services. >> >> Doing packet dumps on the VPN interfaces, we see that the pings >297 from direct connections do indeed arrive at the destination, but are dropped by the destination tinc interface as "ethertype unknown". Again, forwarded connections show no issues at all. If directly connected public endpoints are reconfigured to instead forward through other endpoints, they exhibit no issues with packet sizes. There are no errors reported related to this we can identify at any debug level. >> >> Examining the differences between the 297 and 298 ping packets: >> >> Server (public) tinc mac addr: 72:49:ef:9b:99:b9 >> vm (natted) tinc mac addr: 26:ae:0d:bb:12:01 >> >> Works: >> server pinging the vm with 'ping -s 297 12.12.0.55' >> -------------------------------------------------------------------- >> vm mac | gw mac | start of ip header >> 26ae 0dbb 1201 | 7249 ef9b 99b9 | 0800 4500 >> >> >> Doesn't Work: >> server pinging the vm with 'ping -s 298 12.12.0.55' >> -------------------------------------------------------------------- >> ? | vm mac | gw mac | start of ip header >> 0024 | 26ae 0dbb 1201 | 7249 ef9b 99b9 | 0800 >> ===================================================================>> >> >> So it appears that extra data "0024" is being pre-pended to packets larger than 297 bytes and this offset is why the kernel drops it as an unknown ethertype. >> >> >> Any thoughts on this? We are using tinc version 1.1pre11-143-gbfe231b at all points. We can share more config info if it would be helpful. We will only have access to this env for another week if there are any troubleshooting suggestions. >> >> >> Chris >> >> _______________________________________________ >> tinc mailing list >> tinc at tinc-vpn.org >> http://www.tinc-vpn.org/cgi-bin/mailman/listinfo/tinc
This looks like the culprit: http://www.tinc-vpn.org/git/browse?p=tinc;a=commit;h=7730d5f3ed9bd7c011dced5808130ffcbd74ea6b <http://www.tinc-vpn.org/git/browse?p=tinc;a=commit;h=7730d5f3ed9bd7c011dced5808130ffcbd74ea6b>> On Jun 12, 2015, at 3:51 PM, Chris Clements <cclements at outlook.com> wrote: > > Sure, I?ll see if we can narrow it down for you. > > >> On Jun 12, 2015, at 2:25 PM, Etienne Dechamps <etienne at edechamps.fr> wrote: >> >> That's interesting. I'm using a near-HEAD tinc-1.1 myself and haven't >> encountered this problem, but I think that's because I'm using it in >> router mode, as opposed to switch mode. >> >> I'm trying to narrow this down to a recent commit, but I can't find >> anything obvious. Did you have a working setup with a previous >> tinc-1.1 version? Do you have time to try a git bisect? >> >> On 11 June 2015 at 23:17, Chris Clements <cclements at outlook.com> wrote: >>> Have an interesting issue that's cropped up for us. We have a switch based vpn setup with multiple endpoints, some publicly accessible, some behind hide NAT. What we've noticed is that every machine can reach each other's VPN IP's up to a certain packet size. All machines can successfully ping each other up to 297 bytes. At 298 bytes and beyond, only "forwarded" connections succeed. Specifically, a machine behind NAT connected directly to a public machine via UDP cannot ping it with greater than 297 bytes. In this case ICMP pings are an example, but we see the same behavior with other services. >>> >>> Doing packet dumps on the VPN interfaces, we see that the pings >297 from direct connections do indeed arrive at the destination, but are dropped by the destination tinc interface as "ethertype unknown". Again, forwarded connections show no issues at all. If directly connected public endpoints are reconfigured to instead forward through other endpoints, they exhibit no issues with packet sizes. There are no errors reported related to this we can identify at any debug level. >>> >>> Examining the differences between the 297 and 298 ping packets: >>> >>> Server (public) tinc mac addr: 72:49:ef:9b:99:b9 >>> vm (natted) tinc mac addr: 26:ae:0d:bb:12:01 >>> >>> Works: >>> server pinging the vm with 'ping -s 297 12.12.0.55' >>> -------------------------------------------------------------------- >>> vm mac | gw mac | start of ip header >>> 26ae 0dbb 1201 | 7249 ef9b 99b9 | 0800 4500 >>> >>> >>> Doesn't Work: >>> server pinging the vm with 'ping -s 298 12.12.0.55' >>> -------------------------------------------------------------------- >>> ? | vm mac | gw mac | start of ip header >>> 0024 | 26ae 0dbb 1201 | 7249 ef9b 99b9 | 0800 >>> ===================================================================>>> >>> >>> So it appears that extra data "0024" is being pre-pended to packets larger than 297 bytes and this offset is why the kernel drops it as an unknown ethertype. >>> >>> >>> Any thoughts on this? We are using tinc version 1.1pre11-143-gbfe231b at all points. We can share more config info if it would be helpful. We will only have access to this env for another week if there are any troubleshooting suggestions. >>> >>> >>> Chris >>> >>> _______________________________________________ >>> tinc mailing list >>> tinc at tinc-vpn.org >>> http://www.tinc-vpn.org/cgi-bin/mailman/listinfo/tinc >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.tinc-vpn.org/pipermail/tinc/attachments/20150612/c30f2c5f/attachment.html>
This makes it work for us:> On Jun 12, 2015, at 5:02 PM, Chris Clements <cclements at outlook.com> wrote: > > This looks like the culprit: > > http://www.tinc-vpn.org/git/browse?p=tinc;a=commit;h=7730d5f3ed9bd7c011dced5808130ffcbd74ea6b <http://www.tinc-vpn.org/git/browse?p=tinc;a=commit;h=7730d5f3ed9bd7c011dced5808130ffcbd74ea6b> > > >> On Jun 12, 2015, at 3:51 PM, Chris Clements <cclements at outlook.com <mailto:cclements at outlook.com>> wrote: >> >> Sure, I?ll see if we can narrow it down for you. >> >> >>> On Jun 12, 2015, at 2:25 PM, Etienne Dechamps <etienne at edechamps.fr <mailto:etienne at edechamps.fr>> wrote: >>> >>> That's interesting. I'm using a near-HEAD tinc-1.1 myself and haven't >>> encountered this problem, but I think that's because I'm using it in >>> router mode, as opposed to switch mode. >>> >>> I'm trying to narrow this down to a recent commit, but I can't find >>> anything obvious. Did you have a working setup with a previous >>> tinc-1.1 version? Do you have time to try a git bisect? >>> >>> On 11 June 2015 at 23:17, Chris Clements <cclements at outlook.com <mailto:cclements at outlook.com>> wrote: >>>> Have an interesting issue that's cropped up for us. We have a switch based vpn setup with multiple endpoints, some publicly accessible, some behind hide NAT. What we've noticed is that every machine can reach each other's VPN IP's up to a certain packet size. All machines can successfully ping each other up to 297 bytes. At 298 bytes and beyond, only "forwarded" connections succeed. Specifically, a machine behind NAT connected directly to a public machine via UDP cannot ping it with greater than 297 bytes. In this case ICMP pings are an example, but we see the same behavior with other services. >>>> >>>> Doing packet dumps on the VPN interfaces, we see that the pings >297 from direct connections do indeed arrive at the destination, but are dropped by the destination tinc interface as "ethertype unknown". Again, forwarded connections show no issues at all. If directly connected public endpoints are reconfigured to instead forward through other endpoints, they exhibit no issues with packet sizes. There are no errors reported related to this we can identify at any debug level. >>>> >>>> Examining the differences between the 297 and 298 ping packets: >>>> >>>> Server (public) tinc mac addr: 72:49:ef:9b:99:b9 >>>> vm (natted) tinc mac addr: 26:ae:0d:bb:12:01 >>>> >>>> Works: >>>> server pinging the vm with 'ping -s 297 12.12.0.55' >>>> -------------------------------------------------------------------- >>>> vm mac | gw mac | start of ip header >>>> 26ae 0dbb 1201 | 7249 ef9b 99b9 | 0800 4500 >>>> >>>> >>>> Doesn't Work: >>>> server pinging the vm with 'ping -s 298 12.12.0.55' >>>> -------------------------------------------------------------------- >>>> ? | vm mac | gw mac | start of ip header >>>> 0024 | 26ae 0dbb 1201 | 7249 ef9b 99b9 | 0800 >>>> ===================================================================>>>> >>>> >>>> So it appears that extra data "0024" is being pre-pended to packets larger than 297 bytes and this offset is why the kernel drops it as an unknown ethertype. >>>> >>>> >>>> Any thoughts on this? We are using tinc version 1.1pre11-143-gbfe231b at all points. We can share more config info if it would be helpful. We will only have access to this env for another week if there are any troubleshooting suggestions. >>>> >>>> >>>> Chris >>>> >>>> _______________________________________________ >>>> tinc mailing list >>>> tinc at tinc-vpn.org <mailto:tinc at tinc-vpn.org> >>>> http://www.tinc-vpn.org/cgi-bin/mailman/listinfo/tinc >> >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.tinc-vpn.org/pipermail/tinc/attachments/20150612/61a261a2/attachment-0002.html> -------------- next part -------------- A non-text attachment was scrubbed... Name: net_packet.c.patch Type: application/octet-stream Size: 768 bytes Desc: not available URL: <http://www.tinc-vpn.org/pipermail/tinc/attachments/20150612/61a261a2/attachment-0001.obj> -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.tinc-vpn.org/pipermail/tinc/attachments/20150612/61a261a2/attachment-0003.html>