tomp at Tomp.co.uk
2013-Nov-28 11:49 UTC
Intermittent TCP connect issues when using tinc 1.0.23 and IPv6
Hi, I am using tinc 1.0.23 in router mode and am having trouble with intermittent TCP connect timeouts that do not occur when routing the TCP over the internet directly. I am using a pure IPv6 setup (i.e direct IPv6 connections, and tunneled IPv6 subnets). My configs are as follows: Site 1: ----------------------- tinc.conf: Name = rsukmhb AddressFamily = ipv6 BindToAddress = 2001:1b40:5000:19::2 ConnectTo = rps IffOneQueue = yes tinc-up: #!/bin/sh logger -t tinc Bringing up interface $INTERFACE ip -6 link set "$INTERFACE" up ip -6 link set "$INTERFACE" txqueuelen 10 ip -6 addr add fdd1:c52a:3c24:3384::2/64 dev "$INTERFACE" subnet-up: #!/bin/sh [ -z "$NAME" ] && exit 0 [ "$NAME" = "$NODE" ] && exit 0 logger -t tinc Adding route to $SUBNET for $NODE on $NAME ip -6 route replace $SUBNET dev $INTERFACE Site 2: ----------------------- tinc.conf: Name = rps AddressFamily = ipv6 BindToAddress = 2001:1b40:5000:9::2 ConnectTo = rsukmhb IffOneQueue = yes tinc-up: #!/bin/sh logger -t tinc Bringing up interface $INTERFACE ip -6 link set "$INTERFACE" up ip -6 link set "$INTERFACE" txqueuelen 10 ip -6 addr add fdd1:c52a:3c24:3384::1/64 dev "$INTERFACE subnet-up: #!/bin/sh [ -z "$NAME" ] && exit 0 [ "$NAME" = "$NODE" ] && exit 0 logger -t tinc Adding route to $SUBNET for $NODE on $NAME ip -6 route replace $SUBNET dev $INTERFACE The VPN tun interfaces use private IPv6 IP addresses in the fdd1:c52a:3c24:3384::/64 range. I am then using the host entries at each site to advertise an IPv6 /64 subnet used internally (although still publically routable address subnet) so that it is routed over the VPN tunnel to the other side. This is working pretty well and I am getting around 80MBits/sec throughput. However we have an cron process that runs every 1 minute and makes several HTTP requests over the VPN every 10s. About 10-12 times a day we are getting these errors after the specified 5 second connect timeout set in the app. Request failed: connect() timed out! If we disable the VPN and just route the requests over the internet (as I said they are publically routable addresses) then we do not see these errors, suggesting this is not an application problem. We are also running Cacti and Nagios over the VPN for monitoring the remote site. The traffic level is low, averaging about 20 kbps, but quiet bursty, as every few minutes nagios/cacti runs, and the rest of the time the tunnel is quiet except for this cron app that makes a very small HTTP request every 10s. Do you know what I could be looking at to fix this? I have found these posts that suggest a similar connect() timeout issue: Page 12 of this doc: http://tinc-vpn.org/presentations/fosdem-2011/ec2_vpn_fosdem2011.pdf http://thr3ads.net/tinc-devel/2010/10/1964008-Tweaks-for-high-bandwidth-tinc I have tried disabled replay protection, setting IffOneQueue, lowering and increasing the txqueuelen, setting priority of the tinc process. None of these settings have improved things though. Any thoughts would be much appreciated. Thanks Tom
tomp at tomp.co.uk
2013-Nov-28 12:12 UTC
Intermittent TCP connect issues when using tinc 1.0.23 and IPv6
Here's some more info, as a timeout just occurred using rsync from rsukmhb to rps sites: Nov 28 11:59:01 rsuk-mhb-pubweb101 php: ict-pubweb [SyncSites] (21294) Info: Plugin started Nov 28 11:59:01 rsuk-mhb-pubweb101 php: ict-pubweb [SyncSites] (21294) Info: Last sync build time: 2013-11-27 09:50:40 Nov 28 11:59:16 rsuk-mhb-pubweb101 php: ict-pubweb [SyncSites] (21294) Error: Rsync failed: rsync error: timeout waiting for daemon connection (code 35) at socket.c(279) [receiver=3.0.6] Nov 28 11:59:16 rsuk-mhb-pubweb101 php: ict-pubweb [SyncSites] (21294) Info: Plugin stopped Rsync is configured with a connection timeout of 15s, so the connection was started at 11:59:01 and timed out at 11:59:16. At the same time our import cron process ran from rps to the same server that was running the rsync client: Nov 28 11:59:01 rsuk-mhb-pubweb101 httpd: pubweb.infinity.local rsuk-mhb-pubweb101.ictnw.net 2001:1b40:5600:1::76 - ict-pubweb [28/Nov/2013:11:59:01 +0000] "GET /v1/cmsEvents?limit=1000&sort%5B0%5D=rowId-asc&filter%5B0%5D=siteId-in-value-2&filter%5B1%5D=rowId-gt-value-933&filter%5B2%5D=eventType-eq-value-lead HTTP/1.0" 200 - "-" "Icc HTTP Client" I am running tinc with debug mode 5 on both firewalls so here is a sample of the logs from both at that time: rsukmhb site (where rsync client is): ----------------------------------------- [thomas.parrott at rsuk-mhb-fw01 ~]$ grep '11:59:' /var/log/messages Nov 28 11:59:01 rsuk-mhb-fw01 tinc.dcvpn[24911]: Clamping MSS of packet from rsukmhb to rps to 1369 Nov 28 11:59:01 rsuk-mhb-fw01 tinc.dcvpn[24911]: Sending packet of 86 bytes to rps (2001:1b40:5000:9::2 port 655) Nov 28 11:59:01 rsuk-mhb-fw01 tinc.dcvpn[24911]: Clamping MSS of packet from rsukmhb to rps to 1369 Nov 28 11:59:01 rsuk-mhb-fw01 tinc.dcvpn[24911]: Sending packet of 86 bytes to rps (2001:1b40:5000:9::2 port 655) Nov 28 11:59:01 rsuk-mhb-fw01 tinc.dcvpn[24911]: Sending packet of 74 bytes to rps (2001:1b40:5000:9::2 port 655) Nov 28 11:59:01 rsuk-mhb-fw01 tinc.dcvpn[24911]: Sending packet of 224 bytes to rps (2001:1b40:5000:9::2 port 655) Nov 28 11:59:01 rsuk-mhb-fw01 tinc.dcvpn[24911]: Sending packet of 74 bytes to rps (2001:1b40:5000:9::2 port 655) Nov 28 11:59:01 rsuk-mhb-fw01 tinc.dcvpn[24911]: Sending packet of 74 bytes to rps (2001:1b40:5000:9::2 port 655) Nov 28 11:59:01 rsuk-mhb-fw01 tinc.dcvpn[24911]: Clamping MSS of packet from rsukmhb to rps to 1369 Nov 28 11:59:01 rsuk-mhb-fw01 tinc.dcvpn[24911]: Sending packet of 86 bytes to rps (2001:1b40:5000:9::2 port 655) Nov 28 11:59:01 rsuk-mhb-fw01 tinc.dcvpn[24911]: Sending packet of 74 bytes to rps (2001:1b40:5000:9::2 port 655) Nov 28 11:59:01 rsuk-mhb-fw01 tinc.dcvpn[24911]: Sending packet of 88 bytes to rps (2001:1b40:5000:9::2 port 655) Nov 28 11:59:01 rsuk-mhb-fw01 tinc.dcvpn[24911]: Sending packet of 74 bytes to rps (2001:1b40:5000:9::2 port 655) Nov 28 11:59:01 rsuk-mhb-fw01 tinc.dcvpn[24911]: Sending packet of 87 bytes to rps (2001:1b40:5000:9::2 port 655) Nov 28 11:59:01 rsuk-mhb-fw01 tinc.dcvpn[24911]: Sending packet of 108 bytes to rps (2001:1b40:5000:9::2 port 655) Nov 28 11:59:01 rsuk-mhb-fw01 tinc.dcvpn[24911]: Sending packet of 82 bytes to rps (2001:1b40:5000:9::2 port 655) Nov 28 11:59:01 rsuk-mhb-fw01 tinc.dcvpn[24911]: Sending packet of 136 bytes to rps (2001:1b40:5000:9::2 port 655) Nov 28 11:59:02 rsuk-mhb-fw01 tinc.dcvpn[24911]: Sending packet of 74 bytes to rps (2001:1b40:5000:9::2 port 655) Nov 28 11:59:02 rsuk-mhb-fw01 tinc.dcvpn[24911]: Sending packet of 74 bytes to rps (2001:1b40:5000:9::2 port 655) Nov 28 11:59:02 rsuk-mhb-fw01 tinc.dcvpn[24911]: Sending packet of 82 bytes to rps (2001:1b40:5000:9::2 port 655) Nov 28 11:59:02 rsuk-mhb-fw01 tinc.dcvpn[24911]: Sending packet of 74 bytes to rps (2001:1b40:5000:9::2 port 655) Nov 28 11:59:02 rsuk-mhb-fw01 tinc.dcvpn[24911]: Sending packet of 74 bytes to rps (2001:1b40:5000:9::2 port 655) Nov 28 11:59:02 rsuk-mhb-fw01 tinc.dcvpn[24911]: Sending packet of 74 bytes to rps (2001:1b40:5000:9::2 port 655) Nov 28 11:59:02 rsuk-mhb-fw01 tinc.dcvpn[24911]: Sending packet of 79 bytes to rps (2001:1b40:5000:9::2 port 655) Nov 28 11:59:02 rsuk-mhb-fw01 tinc.dcvpn[24911]: Sending packet of 74 bytes to rps (2001:1b40:5000:9::2 port 655) Nov 28 11:59:02 rsuk-mhb-fw01 tinc.dcvpn[24911]: Sending packet of 74 bytes to rps (2001:1b40:5000:9::2 port 655) Nov 28 11:59:02 rsuk-mhb-fw01 tinc.dcvpn[24911]: Sending packet of 74 bytes to rps (2001:1b40:5000:9::2 port 655) Nov 28 11:59:02 rsuk-mhb-fw01 tinc.dcvpn[24911]: Sending packet of 89 bytes to rps (2001:1b40:5000:9::2 port 655) Nov 28 11:59:02 rsuk-mhb-fw01 tinc.dcvpn[24911]: Sending packet of 89 bytes to rps (2001:1b40:5000:9::2 port 655) Nov 28 11:59:02 rsuk-mhb-fw01 tinc.dcvpn[24911]: Sending packet of 74 bytes to rps (2001:1b40:5000:9::2 port 655) Nov 28 11:59:02 rsuk-mhb-fw01 tinc.dcvpn[24911]: Sending packet of 74 bytes to rps (2001:1b40:5000:9::2 port 655) Nov 28 11:59:02 rsuk-mhb-fw01 tinc.dcvpn[24911]: Sending packet of 80 bytes to rps (2001:1b40:5000:9::2 port 655) Nov 28 11:59:02 rsuk-mhb-fw01 tinc.dcvpn[24911]: Clamping MSS of packet from rsukmhb to rps to 1369 Nov 28 11:59:02 rsuk-mhb-fw01 tinc.dcvpn[24911]: Sending packet of 86 bytes to rps (2001:1b40:5000:9::2 port 655) Nov 28 11:59:02 rsuk-mhb-fw01 tinc.dcvpn[24911]: Sending packet of 74 bytes to rps (2001:1b40:5000:9::2 port 655) Nov 28 11:59:02 rsuk-mhb-fw01 tinc.dcvpn[24911]: Sending packet of 74 bytes to rps (2001:1b40:5000:9::2 port 655) Nov 28 11:59:02 rsuk-mhb-fw01 tinc.dcvpn[24911]: Sending packet of 79 bytes to rps (2001:1b40:5000:9::2 port 655) Nov 28 11:59:02 rsuk-mhb-fw01 tinc.dcvpn[24911]: Sending packet of 74 bytes to rps (2001:1b40:5000:9::2 port 655) Nov 28 11:59:03 rsuk-mhb-fw01 tinc.dcvpn[24911]: Sending packet of 139 bytes to rps (2001:1b40:5000:9::2 port 655) Nov 28 11:59:04 rsuk-mhb-fw01 tinc.dcvpn[24911]: Clamping MSS of packet from rsukmhb to rps to 1369 Nov 28 11:59:04 rsuk-mhb-fw01 tinc.dcvpn[24911]: Sending packet of 86 bytes to rps (2001:1b40:5000:9::2 port 655) Nov 28 11:59:08 rsuk-mhb-fw01 tinc.dcvpn[24911]: Clamping MSS of packet from rsukmhb to rps to 1369 Nov 28 11:59:08 rsuk-mhb-fw01 tinc.dcvpn[24911]: Sending packet of 86 bytes to rps (2001:1b40:5000:9::2 port 655) Nov 28 11:59:10 rsuk-mhb-fw01 tinc.dcvpn[24911]: Sending packet of 118 bytes to rps (2001:1b40:5000:9::2 port 655) Nov 28 11:59:11 rsuk-mhb-fw01 tinc.dcvpn[24911]: Sending packet of 118 bytes to rps (2001:1b40:5000:9::2 port 655) Nov 28 11:59:11 rsuk-mhb-fw01 tinc.dcvpn[24911]: Clamping MSS of packet from rsukmhb to rps to 1369 Nov 28 11:59:11 rsuk-mhb-fw01 tinc.dcvpn[24911]: Sending packet of 86 bytes to rps (2001:1b40:5000:9::2 port 655) Nov 28 11:59:11 rsuk-mhb-fw01 tinc.dcvpn[24911]: Sending packet of 74 bytes to rps (2001:1b40:5000:9::2 port 655) Nov 28 11:59:11 rsuk-mhb-fw01 tinc.dcvpn[24911]: Sending packet of 224 bytes to rps (2001:1b40:5000:9::2 port 655) Nov 28 11:59:11 rsuk-mhb-fw01 tinc.dcvpn[24911]: Sending packet of 74 bytes to rps (2001:1b40:5000:9::2 port 655) Nov 28 11:59:11 rsuk-mhb-fw01 tinc.dcvpn[24911]: Sending packet of 74 bytes to rps (2001:1b40:5000:9::2 port 655) Nov 28 11:59:12 rsuk-mhb-fw01 tinc.dcvpn[24911]: Sending packet of 118 bytes to rps (2001:1b40:5000:9::2 port 655) Nov 28 11:59:13 rsuk-mhb-fw01 tinc.dcvpn[24911]: Sending packet of 118 bytes to rps (2001:1b40:5000:9::2 port 655) Nov 28 11:59:14 rsuk-mhb-fw01 tinc.dcvpn[24911]: Sending packet of 118 bytes to rps (2001:1b40:5000:9::2 port 655) Nov 28 11:59:16 rsuk-mhb-fw01 tinc.dcvpn[24911]: Clamping MSS of packet from rsukmhb to rps to 1369 Nov 28 11:59:16 rsuk-mhb-fw01 tinc.dcvpn[24911]: Sending packet of 86 bytes to rps (2001:1b40:5000:9::2 port 655) Nov 28 11:59:21 rsuk-mhb-fw01 tinc.dcvpn[24911]: Clamping MSS of packet from rsukmhb to rps to 1369 Nov 28 11:59:21 rsuk-mhb-fw01 tinc.dcvpn[24911]: Sending packet of 86 bytes to rps (2001:1b40:5000:9::2 port 655) Nov 28 11:59:21 rsuk-mhb-fw01 tinc.dcvpn[24911]: Sending packet of 74 bytes to rps (2001:1b40:5000:9::2 port 655) Nov 28 11:59:21 rsuk-mhb-fw01 tinc.dcvpn[24911]: Sending packet of 224 bytes to rps (2001:1b40:5000:9::2 port 655) Nov 28 11:59:21 rsuk-mhb-fw01 tinc.dcvpn[24911]: Sending packet of 74 bytes to rps (2001:1b40:5000:9::2 port 655) Nov 28 11:59:21 rsuk-mhb-fw01 tinc.dcvpn[24911]: Sending packet of 74 bytes to rps (2001:1b40:5000:9::2 port 655) Nov 28 11:59:29 rsuk-mhb-fw01 tinc.dcvpn[24911]: Clamping MSS of packet from rsukmhb to rps to 1369 Nov 28 11:59:29 rsuk-mhb-fw01 tinc.dcvpn[24911]: Sending packet of 86 bytes to rps (2001:1b40:5000:9::2 port 655) Nov 28 11:59:29 rsuk-mhb-fw01 tinc.dcvpn[24911]: Sending packet of 74 bytes to rps (2001:1b40:5000:9::2 port 655) Nov 28 11:59:29 rsuk-mhb-fw01 tinc.dcvpn[24911]: Sending packet of 224 bytes to rps (2001:1b40:5000:9::2 port 655) Nov 28 11:59:29 rsuk-mhb-fw01 tinc.dcvpn[24911]: Sending packet of 74 bytes to rps (2001:1b40:5000:9::2 port 655) Nov 28 11:59:29 rsuk-mhb-fw01 tinc.dcvpn[24911]: Sending packet of 74 bytes to rps (2001:1b40:5000:9::2 port 655) Nov 28 11:59:34 rsuk-mhb-fw01 tinc.dcvpn[24911]: Sending packet of 129 bytes to rps (2001:1b40:5000:9::2 port 655) Nov 28 11:59:39 rsuk-mhb-fw01 tinc.dcvpn[24911]: Clamping MSS of packet from rsukmhb to rps to 1369 Nov 28 11:59:39 rsuk-mhb-fw01 tinc.dcvpn[24911]: Sending packet of 86 bytes to rps (2001:1b40:5000:9::2 port 655) Nov 28 11:59:39 rsuk-mhb-fw01 tinc.dcvpn[24911]: Sending packet of 74 bytes to rps (2001:1b40:5000:9::2 port 655) Nov 28 11:59:39 rsuk-mhb-fw01 tinc.dcvpn[24911]: Sending packet of 224 bytes to rps (2001:1b40:5000:9::2 port 655) Nov 28 11:59:39 rsuk-mhb-fw01 tinc.dcvpn[24911]: Sending packet of 74 bytes to rps (2001:1b40:5000:9::2 port 655) Nov 28 11:59:39 rsuk-mhb-fw01 tinc.dcvpn[24911]: Sending packet of 74 bytes to rps (2001:1b40:5000:9::2 port 655) Nov 28 11:59:42 rsuk-mhb-fw01 tinc.dcvpn[24911]: Got MTU probe length 1443 from rps (2001:1b40:5000:9::2 port 655) Nov 28 11:59:42 rsuk-mhb-fw01 tinc.dcvpn[24911]: Got MTU probe length 1443 from rps (2001:1b40:5000:9::2 port 655) Nov 28 11:59:42 rsuk-mhb-fw01 tinc.dcvpn[24911]: Got MTU probe length 1443 from rps (2001:1b40:5000:9::2 port 655) Nov 28 11:59:43 rsuk-mhb-fw01 tinc.dcvpn[24911]: Sending MTU probe length 1451 to rps (2001:1b40:5000:9::2 port 655) Nov 28 11:59:43 rsuk-mhb-fw01 tinc.dcvpn[24911]: Sending MTU probe length 1443 to rps (2001:1b40:5000:9::2 port 655) Nov 28 11:59:43 rsuk-mhb-fw01 tinc.dcvpn[24911]: Sending MTU probe length 1443 to rps (2001:1b40:5000:9::2 port 655) Nov 28 11:59:43 rsuk-mhb-fw01 tinc.dcvpn[24911]: Sending MTU probe length 1443 to rps (2001:1b40:5000:9::2 port 655) Nov 28 11:59:43 rsuk-mhb-fw01 tinc.dcvpn[24911]: Got MTU probe length 1443 from rps (2001:1b40:5000:9::2 port 655) Nov 28 11:59:43 rsuk-mhb-fw01 tinc.dcvpn[24911]: Got MTU probe length 1443 from rps (2001:1b40:5000:9::2 port 655) Nov 28 11:59:43 rsuk-mhb-fw01 tinc.dcvpn[24911]: Got MTU probe length 1443 from rps (2001:1b40:5000:9::2 port 655) Nov 28 11:59:43 rsuk-mhb-fw01 tinc.dcvpn[24911]: Sending packet of 118 bytes to rps (2001:1b40:5000:9::2 port 655) Nov 28 11:59:44 rsuk-mhb-fw01 tinc.dcvpn[24911]: Sending packet of 118 bytes to rps (2001:1b40:5000:9::2 port 655) Nov 28 11:59:45 rsuk-mhb-fw01 tinc.dcvpn[24911]: Sending packet of 118 bytes to rps (2001:1b40:5000:9::2 port 655) Nov 28 11:59:46 rsuk-mhb-fw01 tinc.dcvpn[24911]: Sending packet of 118 bytes to rps (2001:1b40:5000:9::2 port 655) Nov 28 11:59:47 rsuk-mhb-fw01 tinc.dcvpn[24911]: Sending packet of 118 bytes to rps (2001:1b40:5000:9::2 port 655) Nov 28 11:59:49 rsuk-mhb-fw01 tinc.dcvpn[24911]: Clamping MSS of packet from rsukmhb to rps to 1369 Nov 28 11:59:49 rsuk-mhb-fw01 tinc.dcvpn[24911]: Sending packet of 86 bytes to rps (2001:1b40:5000:9::2 port 655) Nov 28 11:59:49 rsuk-mhb-fw01 tinc.dcvpn[24911]: Sending packet of 74 bytes to rps (2001:1b40:5000:9::2 port 655) Nov 28 11:59:49 rsuk-mhb-fw01 tinc.dcvpn[24911]: Sending packet of 224 bytes to rps (2001:1b40:5000:9::2 port 655) Nov 28 11:59:49 rsuk-mhb-fw01 tinc.dcvpn[24911]: Sending packet of 74 bytes to rps (2001:1b40:5000:9::2 port 655) Nov 28 11:59:49 rsuk-mhb-fw01 tinc.dcvpn[24911]: Sending packet of 74 bytes to rps (2001:1b40:5000:9::2 port 655) Nov 28 11:59:55 rsuk-mhb-fw01 tinc.dcvpn[24911]: Clamping MSS of packet from rsukmhb to rps to 1369 Nov 28 11:59:55 rsuk-mhb-fw01 tinc.dcvpn[24911]: Sending packet of 86 bytes to rps (2001:1b40:5000:9::2 port 655) Nov 28 11:59:55 rsuk-mhb-fw01 tinc.dcvpn[24911]: Sending packet of 112 bytes to rps (2001:1b40:5000:9::2 port 655) Nov 28 11:59:55 rsuk-mhb-fw01 tinc.dcvpn[24911]: Sending packet of 74 bytes to rps (2001:1b40:5000:9::2 port 655) Nov 28 11:59:55 rsuk-mhb-fw01 tinc.dcvpn[24911]: Sending packet of 106 bytes to rps (2001:1b40:5000:9::2 port 655) Nov 28 11:59:55 rsuk-mhb-fw01 tinc.dcvpn[24911]: Sending packet of 89 bytes to rps (2001:1b40:5000:9::2 port 655) Nov 28 11:59:55 rsuk-mhb-fw01 tinc.dcvpn[24911]: Sending packet of 74 bytes to rps (2001:1b40:5000:9::2 port 655) Nov 28 11:59:55 rsuk-mhb-fw01 tinc.dcvpn[24911]: Sending packet of 74 bytes to rps (2001:1b40:5000:9::2 port 655) rps site (where rsync server is): ----------------------------------------- [thomas.parrott at rps-fw03 ~]$ grep '11:59:' /var/log/messages Nov 28 11:59:01 rps-fw03 tinc.dcvpn[17321]: Clamping MSS of packet from rps to rsukmhb to 1369 Nov 28 11:59:01 rps-fw03 tinc.dcvpn[17321]: Sending packet of 86 bytes to rsukmhb (2001:1b40:5000:19::2 port 655) Nov 28 11:59:01 rps-fw03 tinc.dcvpn[17321]: Sending packet of 74 bytes to rsukmhb (2001:1b40:5000:19::2 port 655) Nov 28 11:59:01 rps-fw03 tinc.dcvpn[17321]: Sending packet of 433 bytes to rsukmhb (2001:1b40:5000:19::2 port 655) Nov 28 11:59:01 rps-fw03 tinc.dcvpn[17321]: Sending packet of 74 bytes to rsukmhb (2001:1b40:5000:19::2 port 655) Nov 28 11:59:01 rps-fw03 tinc.dcvpn[17321]: Sending packet of 74 bytes to rsukmhb (2001:1b40:5000:19::2 port 655) Nov 28 11:59:01 rps-fw03 tinc.dcvpn[17321]: Clamping MSS of packet from rps to rsukmhb to 1369 Nov 28 11:59:01 rps-fw03 tinc.dcvpn[17321]: Sending packet of 86 bytes to rsukmhb (2001:1b40:5000:19::2 port 655) Nov 28 11:59:01 rps-fw03 tinc.dcvpn[17321]: Sending packet of 74 bytes to rsukmhb (2001:1b40:5000:19::2 port 655) Nov 28 11:59:01 rps-fw03 tinc.dcvpn[17321]: Sending packet of 88 bytes to rsukmhb (2001:1b40:5000:19::2 port 655) Nov 28 11:59:01 rps-fw03 tinc.dcvpn[17321]: Sending packet of 115 bytes to rsukmhb (2001:1b40:5000:19::2 port 655) Nov 28 11:59:01 rps-fw03 tinc.dcvpn[17321]: Sending packet of 86 bytes to rsukmhb (2001:1b40:5000:19::2 port 655) Nov 28 11:59:01 rps-fw03 tinc.dcvpn[17321]: Sending packet of 74 bytes to rsukmhb (2001:1b40:5000:19::2 port 655) Nov 28 11:59:01 rps-fw03 tinc.dcvpn[17321]: Sending packet of 74 bytes to rsukmhb (2001:1b40:5000:19::2 port 655) Nov 28 11:59:01 rps-fw03 tinc.dcvpn[17321]: Sending packet of 75 bytes to rsukmhb (2001:1b40:5000:19::2 port 655) Nov 28 11:59:02 rps-fw03 tinc.dcvpn[17321]: Sending packet of 78 bytes to rsukmhb (2001:1b40:5000:19::2 port 655) Nov 28 11:59:02 rps-fw03 tinc.dcvpn[17321]: Sending packet of 78 bytes to rsukmhb (2001:1b40:5000:19::2 port 655) Nov 28 11:59:02 rps-fw03 tinc.dcvpn[17321]: Sending packet of 1443 bytes to rsukmhb (2001:1b40:5000:19::2 port 655) Nov 28 11:59:02 rps-fw03 tinc.dcvpn[17321]: Sending packet of 1443 bytes to rsukmhb (2001:1b40:5000:19::2 port 655) Nov 28 11:59:02 rps-fw03 tinc.dcvpn[17321]: Sending packet of 1443 bytes to rsukmhb (2001:1b40:5000:19::2 port 655) Nov 28 11:59:02 rps-fw03 tinc.dcvpn[17321]: Sending packet of 1443 bytes to rsukmhb (2001:1b40:5000:19::2 port 655) Nov 28 11:59:02 rps-fw03 tinc.dcvpn[17321]: Sending packet of 1443 bytes to rsukmhb (2001:1b40:5000:19::2 port 655) Nov 28 11:59:02 rps-fw03 tinc.dcvpn[17321]: Sending packet of 1417 bytes to rsukmhb (2001:1b40:5000:19::2 port 655) Nov 28 11:59:02 rps-fw03 tinc.dcvpn[17321]: Sending packet of 1443 bytes to rsukmhb (2001:1b40:5000:19::2 port 655) Nov 28 11:59:02 rps-fw03 tinc.dcvpn[17321]: Sending packet of 1443 bytes to rsukmhb (2001:1b40:5000:19::2 port 655) Nov 28 11:59:02 rps-fw03 tinc.dcvpn[17321]: Sending packet of 1443 bytes to rsukmhb (2001:1b40:5000:19::2 port 655) Nov 28 11:59:02 rps-fw03 tinc.dcvpn[17321]: Sending packet of 1443 bytes to rsukmhb (2001:1b40:5000:19::2 port 655) Nov 28 11:59:02 rps-fw03 tinc.dcvpn[17321]: Sending packet of 1443 bytes to rsukmhb (2001:1b40:5000:19::2 port 655) Nov 28 11:59:02 rps-fw03 tinc.dcvpn[17321]: Sending packet of 1443 bytes to rsukmhb (2001:1b40:5000:19::2 port 655) Nov 28 11:59:02 rps-fw03 tinc.dcvpn[17321]: Sending packet of 1035 bytes to rsukmhb (2001:1b40:5000:19::2 port 655) Nov 28 11:59:02 rps-fw03 tinc.dcvpn[17321]: Sending packet of 74 bytes to rsukmhb (2001:1b40:5000:19::2 port 655) Nov 28 11:59:02 rps-fw03 tinc.dcvpn[17321]: Sending packet of 89 bytes to rsukmhb (2001:1b40:5000:19::2 port 655) Nov 28 11:59:02 rps-fw03 tinc.dcvpn[17321]: Sending packet of 89 bytes to rsukmhb (2001:1b40:5000:19::2 port 655) Nov 28 11:59:02 rps-fw03 tinc.dcvpn[17321]: Sending packet of 79 bytes to rsukmhb (2001:1b40:5000:19::2 port 655) Nov 28 11:59:02 rps-fw03 tinc.dcvpn[17321]: Sending packet of 98 bytes to rsukmhb (2001:1b40:5000:19::2 port 655) Nov 28 11:59:02 rps-fw03 tinc.dcvpn[17321]: Sending packet of 74 bytes to rsukmhb (2001:1b40:5000:19::2 port 655) Nov 28 11:59:02 rps-fw03 tinc.dcvpn[17321]: Sending packet of 74 bytes to rsukmhb (2001:1b40:5000:19::2 port 655) Nov 28 11:59:03 rps-fw03 tinc.dcvpn[17321]: Sending packet of 107 bytes to rsukmhb (2001:1b40:5000:19::2 port 655) Nov 28 11:59:10 rps-fw03 tinc.dcvpn[17321]: Sending packet of 118 bytes to rsukmhb (2001:1b40:5000:19::2 port 655) Nov 28 11:59:11 rps-fw03 tinc.dcvpn[17321]: Sending packet of 118 bytes to rsukmhb (2001:1b40:5000:19::2 port 655) Nov 28 11:59:11 rps-fw03 tinc.dcvpn[17321]: Clamping MSS of packet from rps to rsukmhb to 1369 Nov 28 11:59:11 rps-fw03 tinc.dcvpn[17321]: Sending packet of 86 bytes to rsukmhb (2001:1b40:5000:19::2 port 655) Nov 28 11:59:11 rps-fw03 tinc.dcvpn[17321]: Sending packet of 74 bytes to rsukmhb (2001:1b40:5000:19::2 port 655) Nov 28 11:59:11 rps-fw03 tinc.dcvpn[17321]: Sending packet of 433 bytes to rsukmhb (2001:1b40:5000:19::2 port 655) Nov 28 11:59:11 rps-fw03 tinc.dcvpn[17321]: Sending packet of 74 bytes to rsukmhb (2001:1b40:5000:19::2 port 655) Nov 28 11:59:11 rps-fw03 tinc.dcvpn[17321]: Sending packet of 74 bytes to rsukmhb (2001:1b40:5000:19::2 port 655) Nov 28 11:59:12 rps-fw03 tinc.dcvpn[17321]: Sending packet of 118 bytes to rsukmhb (2001:1b40:5000:19::2 port 655) Nov 28 11:59:13 rps-fw03 tinc.dcvpn[17321]: Sending packet of 118 bytes to rsukmhb (2001:1b40:5000:19::2 port 655) Nov 28 11:59:14 rps-fw03 tinc.dcvpn[17321]: Sending packet of 118 bytes to rsukmhb (2001:1b40:5000:19::2 port 655) Nov 28 11:59:21 rps-fw03 tinc.dcvpn[17321]: Clamping MSS of packet from rps to rsukmhb to 1369 Nov 28 11:59:21 rps-fw03 tinc.dcvpn[17321]: Sending packet of 86 bytes to rsukmhb (2001:1b40:5000:19::2 port 655) Nov 28 11:59:21 rps-fw03 tinc.dcvpn[17321]: Sending packet of 74 bytes to rsukmhb (2001:1b40:5000:19::2 port 655) Nov 28 11:59:21 rps-fw03 tinc.dcvpn[17321]: Sending packet of 433 bytes to rsukmhb (2001:1b40:5000:19::2 port 655) Nov 28 11:59:21 rps-fw03 tinc.dcvpn[17321]: Sending packet of 74 bytes to rsukmhb (2001:1b40:5000:19::2 port 655) Nov 28 11:59:21 rps-fw03 tinc.dcvpn[17321]: Sending packet of 74 bytes to rsukmhb (2001:1b40:5000:19::2 port 655) Nov 28 11:59:29 rps-fw03 tinc.dcvpn[17321]: Clamping MSS of packet from rps to rsukmhb to 1369 Nov 28 11:59:29 rps-fw03 tinc.dcvpn[17321]: Sending packet of 86 bytes to rsukmhb (2001:1b40:5000:19::2 port 655) Nov 28 11:59:29 rps-fw03 tinc.dcvpn[17321]: Sending packet of 74 bytes to rsukmhb (2001:1b40:5000:19::2 port 655) Nov 28 11:59:29 rps-fw03 tinc.dcvpn[17321]: Sending packet of 433 bytes to rsukmhb (2001:1b40:5000:19::2 port 655) Nov 28 11:59:29 rps-fw03 tinc.dcvpn[17321]: Sending packet of 74 bytes to rsukmhb (2001:1b40:5000:19::2 port 655) Nov 28 11:59:29 rps-fw03 tinc.dcvpn[17321]: Sending packet of 74 bytes to rsukmhb (2001:1b40:5000:19::2 port 655) Nov 28 11:59:34 rps-fw03 tinc.dcvpn[17321]: Sending packet of 122 bytes to rsukmhb (2001:1b40:5000:19::2 port 655) Nov 28 11:59:39 rps-fw03 tinc.dcvpn[17321]: Clamping MSS of packet from rps to rsukmhb to 1369 Nov 28 11:59:39 rps-fw03 tinc.dcvpn[17321]: Sending packet of 86 bytes to rsukmhb (2001:1b40:5000:19::2 port 655) Nov 28 11:59:39 rps-fw03 tinc.dcvpn[17321]: Sending packet of 74 bytes to rsukmhb (2001:1b40:5000:19::2 port 655) Nov 28 11:59:39 rps-fw03 tinc.dcvpn[17321]: Sending packet of 433 bytes to rsukmhb (2001:1b40:5000:19::2 port 655) Nov 28 11:59:39 rps-fw03 tinc.dcvpn[17321]: Sending packet of 74 bytes to rsukmhb (2001:1b40:5000:19::2 port 655) Nov 28 11:59:39 rps-fw03 tinc.dcvpn[17321]: Sending packet of 74 bytes to rsukmhb (2001:1b40:5000:19::2 port 655) Nov 28 11:59:42 rps-fw03 tinc.dcvpn[17321]: Sending MTU probe length 1451 to rsukmhb (2001:1b40:5000:19::2 port 655) Nov 28 11:59:42 rps-fw03 tinc.dcvpn[17321]: Sending MTU probe length 1443 to rsukmhb (2001:1b40:5000:19::2 port 655) Nov 28 11:59:42 rps-fw03 tinc.dcvpn[17321]: Sending MTU probe length 1443 to rsukmhb (2001:1b40:5000:19::2 port 655) Nov 28 11:59:42 rps-fw03 tinc.dcvpn[17321]: Sending MTU probe length 1443 to rsukmhb (2001:1b40:5000:19::2 port 655) Nov 28 11:59:42 rps-fw03 tinc.dcvpn[17321]: Got MTU probe length 1443 from rsukmhb (2001:1b40:5000:19::2 port 655) Nov 28 11:59:42 rps-fw03 tinc.dcvpn[17321]: Got MTU probe length 1443 from rsukmhb (2001:1b40:5000:19::2 port 655) Nov 28 11:59:42 rps-fw03 tinc.dcvpn[17321]: Got MTU probe length 1443 from rsukmhb (2001:1b40:5000:19::2 port 655) Nov 28 11:59:43 rps-fw03 tinc.dcvpn[17321]: Got MTU probe length 1443 from rsukmhb (2001:1b40:5000:19::2 port 655) Nov 28 11:59:43 rps-fw03 tinc.dcvpn[17321]: Got MTU probe length 1443 from rsukmhb (2001:1b40:5000:19::2 port 655) Nov 28 11:59:43 rps-fw03 tinc.dcvpn[17321]: Got MTU probe length 1443 from rsukmhb (2001:1b40:5000:19::2 port 655) Nov 28 11:59:43 rps-fw03 tinc.dcvpn[17321]: Sending packet of 118 bytes to rsukmhb (2001:1b40:5000:19::2 port 655) Nov 28 11:59:44 rps-fw03 tinc.dcvpn[17321]: Sending packet of 118 bytes to rsukmhb (2001:1b40:5000:19::2 port 655) Nov 28 11:59:45 rps-fw03 tinc.dcvpn[17321]: Sending packet of 118 bytes to rsukmhb (2001:1b40:5000:19::2 port 655) Nov 28 11:59:46 rps-fw03 tinc.dcvpn[17321]: Sending packet of 118 bytes to rsukmhb (2001:1b40:5000:19::2 port 655) Nov 28 11:59:47 rps-fw03 tinc.dcvpn[17321]: Sending packet of 118 bytes to rsukmhb (2001:1b40:5000:19::2 port 655) Nov 28 11:59:49 rps-fw03 tinc.dcvpn[17321]: Clamping MSS of packet from rps to rsukmhb to 1369 Nov 28 11:59:49 rps-fw03 tinc.dcvpn[17321]: Sending packet of 86 bytes to rsukmhb (2001:1b40:5000:19::2 port 655) Nov 28 11:59:49 rps-fw03 tinc.dcvpn[17321]: Sending packet of 74 bytes to rsukmhb (2001:1b40:5000:19::2 port 655) Nov 28 11:59:49 rps-fw03 tinc.dcvpn[17321]: Sending packet of 433 bytes to rsukmhb (2001:1b40:5000:19::2 port 655) Nov 28 11:59:49 rps-fw03 tinc.dcvpn[17321]: Sending packet of 74 bytes to rsukmhb (2001:1b40:5000:19::2 port 655) Nov 28 11:59:49 rps-fw03 tinc.dcvpn[17321]: Sending packet of 74 bytes to rsukmhb (2001:1b40:5000:19::2 port 655) Nov 28 11:59:55 rps-fw03 tinc.dcvpn[17321]: Clamping MSS of packet from rps to rsukmhb to 1369 Nov 28 11:59:55 rps-fw03 tinc.dcvpn[17321]: Sending packet of 86 bytes to rsukmhb (2001:1b40:5000:19::2 port 655) Nov 28 11:59:55 rps-fw03 tinc.dcvpn[17321]: Sending packet of 74 bytes to rsukmhb (2001:1b40:5000:19::2 port 655) Nov 28 11:59:55 rps-fw03 tinc.dcvpn[17321]: Sending packet of 74 bytes to rsukmhb (2001:1b40:5000:19::2 port 655) Nov 28 11:59:55 rps-fw03 tinc.dcvpn[17321]: Sending packet of 105 bytes to rsukmhb (2001:1b40:5000:19::2 port 655) Nov 28 11:59:55 rps-fw03 tinc.dcvpn[17321]: Sending packet of 80 bytes to rsukmhb (2001:1b40:5000:19::2 port 655) Nov 28 11:59:55 rps-fw03 tinc.dcvpn[17321]: Sending packet of 74 bytes to rsukmhb (2001:1b40:5000:19::2 port 655) Nov 28 11:59:55 rps-fw03 tinc.dcvpn[17321]: Sending packet of 74 bytes to rsukmhb (2001:1b40:5000:19::2 port 655) Neither firewalls are heaving loaded. Thanks Tom On 2013-11-28 11:49, tomp at Tomp.co.uk wrote:> Hi, > > I am using tinc 1.0.23 in router mode and am having trouble with > intermittent TCP connect timeouts that do not occur when routing the > TCP over the internet directly. > > I am using a pure IPv6 setup (i.e direct IPv6 connections, and > tunneled IPv6 subnets). > > My configs are as follows: > > Site 1: > ----------------------- > > tinc.conf: > > Name = rsukmhb > AddressFamily = ipv6 > BindToAddress = 2001:1b40:5000:19::2 > ConnectTo = rps > IffOneQueue = yes > > tinc-up: > > #!/bin/sh > logger -t tinc Bringing up interface $INTERFACE > ip -6 link set "$INTERFACE" up > ip -6 link set "$INTERFACE" txqueuelen 10 > ip -6 addr add fdd1:c52a:3c24:3384::2/64 dev "$INTERFACE" > > subnet-up: > > #!/bin/sh > [ -z "$NAME" ] && exit 0 > [ "$NAME" = "$NODE" ] && exit 0 > logger -t tinc Adding route to $SUBNET for $NODE on $NAME > ip -6 route replace $SUBNET dev $INTERFACE > > Site 2: > ----------------------- > > tinc.conf: > > Name = rps > AddressFamily = ipv6 > BindToAddress = 2001:1b40:5000:9::2 > ConnectTo = rsukmhb > IffOneQueue = yes > > tinc-up: > > #!/bin/sh > logger -t tinc Bringing up interface $INTERFACE > ip -6 link set "$INTERFACE" up > ip -6 link set "$INTERFACE" txqueuelen 10 > ip -6 addr add fdd1:c52a:3c24:3384::1/64 dev "$INTERFACE > > subnet-up: > > #!/bin/sh > [ -z "$NAME" ] && exit 0 > [ "$NAME" = "$NODE" ] && exit 0 > logger -t tinc Adding route to $SUBNET for $NODE on $NAME > ip -6 route replace $SUBNET dev $INTERFACE > > The VPN tun interfaces use private IPv6 IP addresses in the > fdd1:c52a:3c24:3384::/64 range. > > I am then using the host entries at each site to advertise an IPv6 > /64 subnet used internally (although still publically routable address > subnet) so that it is routed over the VPN tunnel to the other side. > > > This is working pretty well and I am getting around 80MBits/sec > throughput. > > However we have an cron process that runs every 1 minute and makes > several HTTP requests over the VPN every 10s. > > About 10-12 times a day we are getting these errors after the > specified 5 second connect timeout set in the app. > > Request failed: connect() timed out! > > > If we disable the VPN and just route the requests over the internet > (as I said they are publically routable addresses) then we do not see > these errors, suggesting this is not an application problem. > > We are also running Cacti and Nagios over the VPN for monitoring the > remote site. > > The traffic level is low, averaging about 20 kbps, but quiet bursty, > as every few minutes nagios/cacti runs, and the rest of the time the > tunnel is quiet except for this cron app that makes a very small HTTP > request every 10s. > > > Do you know what I could be looking at to fix this? > > I have found these posts that suggest a similar connect() timeout > issue: > > Page 12 of this doc: > http://tinc-vpn.org/presentations/fosdem-2011/ec2_vpn_fosdem2011.pdf > > http://thr3ads.net/tinc-devel/2010/10/1964008-Tweaks-for-high-bandwidth-tinc > > I have tried disabled replay protection, setting IffOneQueue, > lowering and increasing the txqueuelen, setting priority of the tinc > process. > > None of these settings have improved things though. > > Any thoughts would be much appreciated. > > Thanks > Tom
Guus Sliepen
2013-Nov-28 12:14 UTC
Intermittent TCP connect issues when using tinc 1.0.23 and IPv6
On Thu, Nov 28, 2013 at 11:49:50AM +0000, tomp at Tomp.co.uk wrote: [...]> However we have an cron process that runs every 1 minute and makes > several HTTP requests over the VPN every 10s. > > About 10-12 times a day we are getting these errors after the > specified 5 second connect timeout set in the app. > > Request failed: connect() timed out! > > If we disable the VPN and just route the requests over the internet > (as I said they are publically routable addresses) then we do not > see these errors, suggesting this is not an application problem.Tinc itself changes its symmetric encryption keys every hour. This could cause a few packets to be dropped. If that happens with the HTTP connection you make every 10 seconds, and the HTTP connection's TCP stream doesn't resend the missed packets within your 5 second timeout, then this might explain the dropped connections. You can try setting KeyExpire to a larger value (86400 for example) and see if that reduces the frequency of those timeouts. Another issue might be the fact that tinc encapsulates the TCP packets from the HTTP request inside UDP packets, and some router along the way might be configured to give lower priority to UDP packets, causing them to drop more often, and hence causing the HTTP request to time out. You can try to run tinc with with a high debugging level and let it log to a file, so you can check afterwards if tinc itself reported anything strange happening: tincd -n <netname> -d5 --logfile=/var/log/tinc.log Beware that at -d5, it could grow quite large. -- Met vriendelijke groet / with kind regards, Guus Sliepen <guus at tinc-vpn.org> -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: Digital signature URL: <http://www.tinc-vpn.org/pipermail/tinc/attachments/20131128/a6c8a3a2/attachment.sig>
Maybe Matching Threads
- tinc Digest, Vol 109, Issue 7
- ntlm_auth from pppd help
- [Bridge] Neighbour table overflow
- [PATCH net-next v2 4/4] virtio-net: initial debugfs support, export mergeable rx buffer size
- [PATCH net-next v2 4/4] virtio-net: initial debugfs support, export mergeable rx buffer size