tinc - Sep 2017 - Packet capture to analysis the tinc connection close

Hi, All

Recently, one of my tinc client always suffer connection drop, I was suspect the
connection was not stable to cause this issue, and BTW, I’ve set the PingTimeout
to 10 seconds already, but this situation still happens a lot sometimes, but
when the connection drop happens, the connection recovery pretty fast, normally
in a minutes.

In order to deep dive into the cause, or proven the network quality problem, I
capture the tcpdump from client to server to see what’s going on.

Client side configure:

tinc.conf:
AddressFamily = ipv4
Name = box2
ProcessPriority = high
PingTimeout = 10
TunnelServer = yes
ConnectTo = abc

box2:
Subnet = 10.0.0.102/32
IndirectData = yes
Port = 8102


Server side configure:

tinc.conf:
Name = abc
AddressFamily = ipv4
PingTimeout = 10

abc:
Address = 47.152.x.x(public address, 172.31.x.x as the private real NIC address)
Subnet = 10.0.0.16/32
Port = 443
IndirectData = yes


As you saw from https://ibb.co/mRyG3a <https://ibb.co/mRyG3a>, the
connection get drop and re-establish very frequently, and the one highlighted as
yellow, it’s the connection we’ll go into deep dive, which happens on 07:41:35,
when you cross check this with https://ibb.co/b740UF
<https://ibb.co/b740UF>, you’ll find that event match the packet of
485/486 which is the server side RST packet to close the connection, but let’s
move our focus to the packet of 481, which is the packet server(47.52.x.x) send
to tinc client to close the connection, which happens at 07:41:27.

Then the below logs are captured from tinc server side that you can see at
07:41:27, the server report it didn’t receive any Ping respond from the client,
so that it close the connection, but let’s take a look for the tcpdump from
server side, which is the server_tcpdump(in attachment). No.463 packet is the
one where server send FIN to tinc client to close the connection at 07:41:27,
but the assumption is server didn’t receive any Ping response from client so
that the server initiate the closure, but as we see from the screenshot that,
the server(172.31.x.x) received and sent couple of packets with tinc
client(123.151.x.x) before 07:41:27, which is https://ibb.co/isECbv
<https://ibb.co/isECbv>, for example packet of 457~462. So why tinc server
believe it doesn’t receive any response from tinc client, but the packet capture
shows it had regular communication with tinc client with 10 second. I would like
to get to know the cause for this tinc frequent drop issue.

Sep  5 07:41:27 abc tinc.myvpn[8510]: box2 (123.151.x.x port 51402) didn't
respond to PING in 10 seconds
Sep  5 07:41:27 abc tinc.myvpn[8510]: Closing connection with box2 (123.151.x.x
port 51402)
Sep  5 07:41:40 abc tinc.myvpn[8510]: Connection with box2 (123.151.x.x port
51418) activated


Best Regards

Bright
✉
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://www.tinc-vpn.org/pipermail/tinc/attachments/20170905/85ed6504/attachment-0001.html>

On Tue, Sep 05, 2017 at 12:27:59PM +0800, Bright Zhao wrote:
> Recently, one of my tinc client always suffer connection drop, I was
suspect the connection was not stable to cause this issue, and BTW, I’ve set the
PingTimeout to 10 seconds already, but this situation still happens a lot
sometimes, but when the connection drop happens, the connection recovery pretty
fast, normally in a minutes.
[...]
> As you saw from https://ibb.co/mRyG3a <https://ibb.co/mRyG3a>, the
connection get drop and re-establish very frequently,
It looks like it happens almost exactly 10 minutes after a connection
was established. I suspect that there is a firewall or NAT device in
front of the client that drops any TCP connection after 10 minutes,
regardless of whether there was any recent traffic over that connection
or not. There is not much tinc can do about that, besides reconnecting
when it happens.

-- 
Met vriendelijke groet / with kind regards,
     Guus Sliepen <guus at tinc-vpn.org>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL:
<http://www.tinc-vpn.org/pipermail/tinc/attachments/20170907/c1120cf3/attachment.sig>

Thanks, Guus, but unfortunately, both are public IP routable on Internet
without firewall/NAT.

Guus Sliepen <guus at tinc-vpn.org>于2017年9月8日 周五上午1:18写道：
> On Tue, Sep 05, 2017 at 12:27:59PM +0800, Bright Zhao wrote:
>
> > Recently, one of my tinc client always suffer connection drop, I was
> suspect the connection was not stable to cause this issue, and BTW, I’ve
> set the PingTimeout to 10 seconds already, but this situation still happens
> a lot sometimes, but when the connection drop happens, the connection
> recovery pretty fast, normally in a minutes.
> [...]
> > As you saw from https://ibb.co/mRyG3a <https://ibb.co/mRyG3a>,
the
> connection get drop and re-establish very frequently,
>
> It looks like it happens almost exactly 10 minutes after a connection
> was established. I suspect that there is a firewall or NAT device in
> front of the client that drops any TCP connection after 10 minutes,
> regardless of whether there was any recent traffic over that connection
> or not. There is not much tinc can do about that, besides reconnecting
> when it happens.
>
> --
> Met vriendelijke groet / with kind regards,
>      Guus Sliepen <guus at tinc-vpn.org>
> _______________________________________________
> tinc mailing list
> tinc at tinc-vpn.org
> https://www.tinc-vpn.org/cgi-bin/mailman/listinfo/tinc
>
-- 
Sent from iPhone
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://www.tinc-vpn.org/pipermail/tinc/attachments/20170907/d68e3640/attachment.html>

And, I ran a constant ping from the tinc client’s IP to the tinc server’s IP, it
shows, the pings are all successfully back and forth, no any packet loss during
the connection drop happens, so will this help to exclude any NAT/firewall cause
the connection drop?

And as you saw from the earlier screen shot, when it happens, it drop all tinc
connections, and those connections are for different direction of tinc servesr,
and the packet capture is just show one of the connections.

I do realize the connection drop in the picture happens 10 mins, but the tinc
client is direct connect to SP with public routable IP, without any
firewall/NAT. Just trying to know if there’s any other reason might cause this?

And BTW, this type of connection drop is not regularly with pattern, sometimes
it happens, sometimes it doesn’t.

> On 8 Sep 2017, at 1:18 AM, Guus Sliepen <guus at tinc-vpn.org> wrote:
> 
> On Tue, Sep 05, 2017 at 12:27:59PM +0800, Bright Zhao wrote:
> 
>> Recently, one of my tinc client always suffer connection drop, I was
suspect the connection was not stable to cause this issue, and BTW, I’ve set the
PingTimeout to 10 seconds already, but this situation still happens a lot
sometimes, but when the connection drop happens, the connection recovery pretty
fast, normally in a minutes.
> [...]
>> As you saw from https://ibb.co/mRyG3a <https://ibb.co/mRyG3a>,
the connection get drop and re-establish very frequently,
> 
> It looks like it happens almost exactly 10 minutes after a connection
> was established. I suspect that there is a firewall or NAT device in
> front of the client that drops any TCP connection after 10 minutes,
> regardless of whether there was any recent traffic over that connection
> or not. There is not much tinc can do about that, besides reconnecting
> when it happens.
> 
> -- 
> Met vriendelijke groet / with kind regards,
>     Guus Sliepen <guus at tinc-vpn.org>
> _______________________________________________
> tinc mailing list
> tinc at tinc-vpn.org
> https://www.tinc-vpn.org/cgi-bin/mailman/listinfo/tinc