Alexandre Beaulieu
2014-Nov-12 19:51 UTC
Connection failing between 2 nodes with dropped packets error
Hi, I'm sometimes getting a failure of connecting 2 nodes when Tinc is started and configured in a LAN. In the logs, there are some unexpected dropped packets with very high or negative seq. I can reproduce this issue ~2% of the time. When this happens, the 2 nodes can no longer ping or ssh each other through the tunnel interface but using eth0 works fine. The connection can recover after at least 20 minutes in some instances but sometimes the connection is still broken after an hour. Here is the tinc log on node3 after I sent the USR1 and USR2 signals: 2014-11-12 18:55:27 tinc.ccvpn[1879]: tincd 1.0.24 (Jul 12 2014 17:00:19) starting, debug level 0 2014-11-12 18:55:27 tinc.ccvpn[1879]: /dev/net/tun is a Linux tun/tap device (tun mode) 2014-11-12 18:55:28 tinc.ccvpn[1879]: Ready 2014-11-12 18:55:34 tinc.ccvpn[1879]: Got HUP signal 2014-11-12 18:56:41 tinc.ccvpn[1879]: Packet from node5 (10.60.19.106 port 655) is 1898092539 seqs in the future, dropped (1) 2014-11-12 19:00:53 tinc.ccvpn[1879]: Packet from node5 (10.60.19.106 port 655) is -1038452044 seqs in the future, dropped (2) 2014-11-12 19:01:29 tinc.ccvpn[1879]: Packet from node5 (10.60.19.106 port 655) is -2035506251 seqs in the future, dropped (3) 2014-11-12 19:05:07 tinc.ccvpn[1879]: Packet from node5 (10.60.19.106 port 655) is 1665101896 seqs in the future, dropped (4) 2014-11-12 19:06:45 tinc.ccvpn[1879]: Connections: 2014-11-12 19:06:45 tinc.ccvpn[1879]: node1 at 10.60.19.107 port 40215 options 8 socket 7 status 00c2 outbuf 525/0/0 2014-11-12 19:06:45 tinc.ccvpn[1879]: node5 at 10.60.19.106 port 53329 options 8 socket 8 status 00c2 outbuf 770/0/0 2014-11-12 19:06:45 tinc.ccvpn[1879]: node4 at 10.60.19.101 port 57404 options 8 socket 16 status 00c2 outbuf 1049/0/0 2014-11-12 19:06:45 tinc.ccvpn[1879]: node2 at 10.60.19.102 port 655 options 8 socket 3 status 01c2 outbuf 1566/0/0 2014-11-12 19:06:45 tinc.ccvpn[1879]: node6 at 10.60.19.111 port 655 options 8 socket 20 status 01c2 outbuf 1630/0/0 2014-11-12 19:06:45 tinc.ccvpn[1879]: End of connections. 2014-11-12 19:06:48 tinc.ccvpn[1879]: Statistics for Linux tun/tap device (tun mode) /dev/net/tun: 2014-11-12 19:06:48 tinc.ccvpn[1879]: total bytes in: 76020994 2014-11-12 19:06:48 tinc.ccvpn[1879]: total bytes out: 75874909 2014-11-12 19:06:48 tinc.ccvpn[1879]: Nodes: 2014-11-12 19:06:48 tinc.ccvpn[1879]: node1 at 10.60.19.107 port 655 cipher 91 digest 0 maclength 4 compression 0 options 8 status 001a nexthop node1 via node1 pmtu 1454 (min 0 max 1518) 2014-11-12 19:06:48 tinc.ccvpn[1879]: node2 at 10.60.19.102 port 655 cipher 91 digest 0 maclength 4 compression 0 options 8 status 001a nexthop node2 via node2 pmtu 1454 (min 0 max 1518) 2014-11-12 19:06:48 tinc.ccvpn[1879]: node3 at MYSELF cipher 0 digest 0 maclength 0 compression 0 options 8 status 0018 nexthop node3 via node3 pmtu 1518 (min 0 max 1518) 2014-11-12 19:06:48 tinc.ccvpn[1879]: node4 at 10.60.19.101 port 655 cipher 91 digest 0 maclength 4 compression 0 options 8 status 001a nexthop node4 via node4 pmtu 1454 (min 0 max 1518) 2014-11-12 19:06:48 tinc.ccvpn[1879]: node5 at 10.60.19.106 port 655 cipher 91 digest 0 maclength 4 compression 0 options 8 status 001a nexthop node5 via node5 pmtu 1454 (min 0 max 1518) 2014-11-12 19:06:48 tinc.ccvpn[1879]: node6 at 10.60.19.111 port 655 cipher 91 digest 0 maclength 4 compression 0 options 8 status 001a nexthop node6 via node6 pmtu 1454 (min 0 max 1518) 2014-11-12 19:06:48 tinc.ccvpn[1879]: End of nodes. 2014-11-12 19:06:48 tinc.ccvpn[1879]: Edges: 2014-11-12 19:06:48 tinc.ccvpn[1879]: node1 to node2 at 10.60.19.102 port 655 options 8 weight 307 2014-11-12 19:06:48 tinc.ccvpn[1879]: node1 to node3 at 10.60.19.108 port 655 options 8 weight 320 2014-11-12 19:06:48 tinc.ccvpn[1879]: node1 to node4 at 10.60.19.101 port 655 options 8 weight 276 2014-11-12 19:06:48 tinc.ccvpn[1879]: node1 to node5 at 10.60.19.106 port 655 options 8 weight 349 2014-11-12 19:06:48 tinc.ccvpn[1879]: node1 to node6 at 10.60.19.111 port 655 options 8 weight 314 2014-11-12 19:06:48 tinc.ccvpn[1879]: node2 to node1 at 10.60.19.107 port 655 options 8 weight 307 2014-11-12 19:06:48 tinc.ccvpn[1879]: node2 to node3 at 10.60.19.108 port 655 options 8 weight 157 2014-11-12 19:06:48 tinc.ccvpn[1879]: node2 to node4 at 10.60.19.101 port 655 options 8 weight 329 2014-11-12 19:06:48 tinc.ccvpn[1879]: node2 to node5 at 10.60.19.106 port 655 options 8 weight 335 2014-11-12 19:06:48 tinc.ccvpn[1879]: node2 to node6 at 10.60.19.111 port 655 options 8 weight 181 2014-11-12 19:06:48 tinc.ccvpn[1879]: node3 to node1 at 10.60.19.107 port 655 options 8 weight 320 2014-11-12 19:06:48 tinc.ccvpn[1879]: node3 to node2 at 10.60.19.102 port 655 options 8 weight 157 2014-11-12 19:06:48 tinc.ccvpn[1879]: node3 to node4 at 10.60.19.101 port 655 options 8 weight 384 2014-11-12 19:06:48 tinc.ccvpn[1879]: node3 to node5 at 10.60.19.106 port 655 options 8 weight 348 2014-11-12 19:06:48 tinc.ccvpn[1879]: node3 to node6 at 10.60.19.111 port 655 options 8 weight 160 2014-11-12 19:06:48 tinc.ccvpn[1879]: node4 to node1 at 10.60.19.107 port 655 options 8 weight 276 2014-11-12 19:06:48 tinc.ccvpn[1879]: node4 to node2 at 10.60.19.102 port 655 options 8 weight 329 2014-11-12 19:06:48 tinc.ccvpn[1879]: node4 to node3 at 10.60.19.108 port 655 options 8 weight 384 2014-11-12 19:06:48 tinc.ccvpn[1879]: node4 to node5 at 10.60.19.106 port 655 options 8 weight 354 2014-11-12 19:06:48 tinc.ccvpn[1879]: node4 to node6 at 10.60.19.111 port 655 options 8 weight 430 2014-11-12 19:06:48 tinc.ccvpn[1879]: node5 to node1 at 10.60.19.107 port 655 options 8 weight 349 2014-11-12 19:06:48 tinc.ccvpn[1879]: node5 to node2 at 10.60.19.102 port 655 options 8 weight 335 2014-11-12 19:06:48 tinc.ccvpn[1879]: node5 to node3 at 10.60.19.108 port 655 options 8 weight 348 2014-11-12 19:06:48 tinc.ccvpn[1879]: node5 to node4 at 10.60.19.101 port 655 options 8 weight 354 2014-11-12 19:06:48 tinc.ccvpn[1879]: node5 to node6 at 10.60.19.111 port 655 options 8 weight 353 2014-11-12 19:06:48 tinc.ccvpn[1879]: node6 to node1 at 10.60.19.107 port 655 options 8 weight 314 2014-11-12 19:06:48 tinc.ccvpn[1879]: node6 to node2 at 10.60.19.102 port 655 options 8 weight 181 2014-11-12 19:06:48 tinc.ccvpn[1879]: node6 to node3 at 10.60.19.108 port 655 options 8 weight 160 2014-11-12 19:06:48 tinc.ccvpn[1879]: node6 to node4 at 10.60.19.101 port 655 options 8 weight 430 2014-11-12 19:06:48 tinc.ccvpn[1879]: node6 to node5 at 10.60.19.106 port 655 options 8 weight 353 2014-11-12 19:06:48 tinc.ccvpn[1879]: End of edges. 2014-11-12 19:06:48 tinc.ccvpn[1879]: Subnet list: 2014-11-12 19:06:48 tinc.ccvpn[1879]: 172.23.1.0/24#10 owner node1 2014-11-12 19:06:48 tinc.ccvpn[1879]: 172.23.2.0/24#10 owner node2 2014-11-12 19:06:48 tinc.ccvpn[1879]: 172.23.3.0/24#10 owner node3 2014-11-12 19:06:48 tinc.ccvpn[1879]: 172.23.4.0/24#10 owner node4 2014-11-12 19:06:48 tinc.ccvpn[1879]: 172.23.5.0/24#10 owner node5 2014-11-12 19:06:48 tinc.ccvpn[1879]: 172.23.6.0/24#10 owner node6 2014-11-12 19:06:48 tinc.ccvpn[1879]: End of subnet list. 2014-11-12 19:08:08 tinc.ccvpn[1879]: Lost 318378807 packets from node5 (10.60.19.106 port 655) 2014-11-12 19:08:41 tinc.ccvpn[1879]: Packet from node5 (10.60.19.106 port 655) is 824328842 seqs in the future, dropped (1) 2014-11-12 19:09:41 tinc.ccvpn[1879]: Packet from node5 (10.60.19.106 port 655) is -673463260 seqs in the future, dropped (2) Thanks, Alexandre Beaulieu -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.tinc-vpn.org/pipermail/tinc/attachments/20141112/809ae97c/attachment.html>
Guus Sliepen
2014-Nov-22 20:19 UTC
Connection failing between 2 nodes with dropped packets error
On Wed, Nov 12, 2014 at 02:51:05PM -0500, Alexandre Beaulieu wrote:> I'm sometimes getting a failure of connecting 2 nodes when Tinc is started > and configured in a LAN. In the logs, there are some unexpected dropped > packets with very high or negative seq. I can reproduce this issue ~2% of > the time.[...]> 2014-11-12 18:56:41 tinc.ccvpn[1879]: Packet from node5 (10.60.19.106 port 655) is 1898092539 seqs in the future, dropped (1) > 2014-11-12 19:00:53 tinc.ccvpn[1879]: Packet from node5 (10.60.19.106 port 655) is -1038452044 seqs in the future, dropped (2) > 2014-11-12 19:01:29 tinc.ccvpn[1879]: Packet from node5 (10.60.19.106 port 655) is -2035506251 seqs in the future, dropped (3) > 2014-11-12 19:05:07 tinc.ccvpn[1879]: Packet from node5 (10.60.19.106 port 655) is 1665101896 seqs in the future, dropped (4)Hm, this should not happen of course. Can you tell me if you have Cipher, Compression, Digest, and/or MACLength set in any of your configuration files? -- Met vriendelijke groet / with kind regards, Guus Sliepen <guus at tinc-vpn.org> -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: Digital signature URL: <http://www.tinc-vpn.org/pipermail/tinc/attachments/20141122/ba98965c/attachment.sig>
Alexandre Beaulieu
2014-Nov-24 02:56 UTC
Connection failing between 2 nodes with dropped packets error
Cipher is enabled with the default setting (Blowfish CBC) Digest is disabled Compression is disabled MACLength is the default (4 if I remember correctly) I suspect that the issue is related to the encryption. I tried disabling encryption and the issue wasn?t reproduced during an automated test run of 400 reconnections. Regards, Alexandre Beaulieu> On Nov 22, 2014, at 3:19 PM, Guus Sliepen <guus at tinc-vpn.org> wrote: > > On Wed, Nov 12, 2014 at 02:51:05PM -0500, Alexandre Beaulieu wrote: > >> I'm sometimes getting a failure of connecting 2 nodes when Tinc is started >> and configured in a LAN. In the logs, there are some unexpected dropped >> packets with very high or negative seq. I can reproduce this issue ~2% of >> the time. > [...] >> 2014-11-12 18:56:41 tinc.ccvpn[1879]: Packet from node5 (10.60.19.106 port 655) is 1898092539 seqs in the future, dropped (1) >> 2014-11-12 19:00:53 tinc.ccvpn[1879]: Packet from node5 (10.60.19.106 port 655) is -1038452044 seqs in the future, dropped (2) >> 2014-11-12 19:01:29 tinc.ccvpn[1879]: Packet from node5 (10.60.19.106 port 655) is -2035506251 seqs in the future, dropped (3) >> 2014-11-12 19:05:07 tinc.ccvpn[1879]: Packet from node5 (10.60.19.106 port 655) is 1665101896 seqs in the future, dropped (4) > > Hm, this should not happen of course. Can you tell me if you have > Cipher, Compression, Digest, and/or MACLength set in any of your > configuration files? > > -- > Met vriendelijke groet / with kind regards, > Guus Sliepen <guus at tinc-vpn.org> > _______________________________________________ > tinc mailing list > tinc at tinc-vpn.org > http://www.tinc-vpn.org/cgi-bin/mailman/listinfo/tinc