thr3ads.net - tinc - Aw: Re: very high traffic without any load [May 2019]

If this information is useful, please help other people find it:
Share via:

Christopher Klinge

2019-May-03 18:06 UTC

Aw: Re: very high traffic without any load

<html><head></head><body><div
style="font-family: Verdana;font-size: 12.0px;"><div>
<div>Hi again,</div>

<div> </div>

<div>I did some digging, and thus far I could not find any other culprit
other than tinc itself. The packages that are being sent are addressed directly
to the other tinc hosts on their vpn addresses. During my latest tests, within
about 12 seconds 100MB of data were transmitted this way. I captured this test
in wireshark. Sadly, I lack the expertise of understanding what is
happening.</div>

<div> </div>

<div>At the very beginning, normal connections are being set up and a few
ICMP neighbor advertisements/solicitations are being exchanged. Next a short TCP
session was created between the public IP addresses of two of my hosts, through
the VPN. This is something that I would like to support and theoretically it
should be possible. However, to me it looks like the connection could not be
established. Right afterwards, one of the nodes involved in the TCP connection
and the third node I used for testing started exchaning the weird packages that
I am complaining about.</div>

<div> </div>

<div>They are UDP packets varying in size and as far as I can tell
unrelated to any outside application. All of them belong to one connection. If
anyone would like to take a look at the dump itself, I'll provide it
directly, since I don't want to make all of my servers' addresses
public.</div>

<div> </div>

<div>Warning, wall of text incoming:</div>

<div>
<div>
<div>Source                Destination           Protocol Length
Info<br/>
node01-public         node04-public         TCP      929    tinc(655) → 40690
[PSH, ACK] Seq=1 Ack=1 Win=240 Len=843 TSval=66121145 TSecr=65947641<br/>
node01-public         node04-public         TCP      1294   tinc(655) → 40690
[ACK] Seq=844 Ack=1 Win=240 Len=1208 TSval=66121145 TSecr=65947641<br/>
node01-public         node04-public         TCP      110    tinc(655) → 40690
[PSH, ACK] Seq=2052 Ack=1 Win=240 Len=24 TSval=66121145
TSecr=65947641<br/>
node01-public         node04-public         TCP      1294   tinc(655) → 40690
[ACK] Seq=2076 Ack=1 Win=240 Len=1208 TSval=66121145 TSecr=65947641<br/>
node01-public         node04-public         TCP      373    tinc(655) → 40690
[PSH, ACK] Seq=3284 Ack=1 Win=240 Len=287 TSval=66121145
TSecr=65947641<br/>
node01-public         node04-public         TCP      1294   tinc(655) → 40690
[ACK] Seq=3571 Ack=1 Win=240 Len=1208 TSval=66121145 TSecr=65947641<br/>
node01-public         node04-public         TCP      636    tinc(655) → 40690
[PSH, ACK] Seq=4779 Ack=1 Win=240 Len=550 TSval=66121145
TSecr=65947641<br/>
node01-public         node04-public         TCP      1294   tinc(655) → 40690
[ACK] Seq=5329 Ack=1 Win=240 Len=1208 TSval=66121149 TSecr=65947641<br/>
node01-public         node04-public         TCP      899    tinc(655) → 40690
[PSH, ACK] Seq=6537 Ack=1 Win=240 Len=813 TSval=66121149
TSecr=65947641<br/>
node01-public         node04-public         TCP      1294   tinc(655) → 40690
[ACK] Seq=7350 Ack=1 Win=240 Len=1196 TSval=66121196 TSecr=65947693
SLE=4294967052 SRE=1<br/>
node01-public         node04-public         TCP      1294   tinc(655) → 40690
[ACK] Seq=8546 Ack=1 Win=240 Len=1208 TSval=66121200 TSecr=65947693<br/>
node01-public         node04-public         TCP      98     [TCP Dup ACK 63#1]
tinc(655) → 40690 [ACK] Seq=9754 Ack=1 Win=240 Len=0 TSval=66121248
TSecr=65947745 SLE=4294967052 SRE=1<br/>
node01-public         node04-public         TCP      1294   tinc(655) → 40690
[ACK] Seq=9754 Ack=1 Win=240 Len=1208 TSval=66121252 TSecr=65947745<br/>
node01-public         node04-public         TCP      929    [TCP Retransmission]
tinc(655) → 40690 [PSH, ACK] Seq=1 Ack=1 Win=240 Len=843 TSval=66121304
TSecr=65947745<br/>
node01-public         node04-public         TCP      98     [TCP Dup ACK 63#2]
tinc(655) → 40690 [ACK] Seq=10962 Ack=1 Win=240 Len=0 TSval=66121351
TSecr=65947848 SLE=4294967052 SRE=1<br/>
node01-public         node04-public         TCP      929    [TCP Retransmission]
tinc(655) → 40690 [PSH, ACK] Seq=1 Ack=1 Win=240 Len=843 TSval=66121408
TSecr=65947848<br/>
node01-public         node04-public         TCP      98     [TCP Dup ACK 63#3]
tinc(655) → 40690 [ACK] Seq=10962 Ack=1 Win=240 Len=0 TSval=66121559
TSecr=65948056 SLE=4294967052 SRE=1<br/>
node01-public         node04-public         TCP      929    [TCP Retransmission]
tinc(655) → 40690 [PSH, ACK] Seq=1 Ack=1 Win=240 Len=843 TSval=66121616
TSecr=65948056<br/>
node02-vpn            ff02::1:ff00:4        ICMPv6   86     Neighbor
Solicitation for 1111:1::4 from 96:6a:04:92:56:4e<br/>
fe80::946a:4ff:fe92:564e ff02::1:ff00:4        ICMPv6   86     Neighbor
Solicitation for 1111:1::4 from 96:6a:04:92:56:4e<br/>
node01-public         node04-public         TCP      98     [TCP Dup ACK 63#4]
tinc(655) → 40690 [ACK] Seq=10962 Ack=1 Win=240 Len=0 TSval=66121975
TSecr=65948472 SLE=4294967052 SRE=1<br/>
node01-public         node04-public         TCP      929    [TCP Retransmission]
tinc(655) → 40690 [PSH, ACK] Seq=1 Ack=1 Win=240 Len=843 TSval=66122032
TSecr=65948472<br/>
node02-vpn            ff02::1:ff00:4        ICMPv6   86     Neighbor
Solicitation for 1111:1::4 from 96:6a:04:92:56:4e<br/>
node02-vpn            ff02::1:ff00:4        ICMPv6   86     Neighbor
Solicitation for 1111:1::4 from 96:6a:04:92:56:4e<br/>
fe80::1471:c8ff:fe7b:1003 1111:1::4             ICMPv6   86     Neighbor
Solicitation for 1111:1::4 from 16:71:c8:7b:10:03<br/>
fe80::1471:c8ff:fe7b:1003 1111:1::4             ICMPv6   86     Neighbor
Solicitation for 1111:1::4 from 16:71:c8:7b:10:03<br/>
node01-public         node04-public         TCP      98     [TCP Dup ACK 63#5]
tinc(655) → 40690 [ACK] Seq=10962 Ack=1 Win=240 Len=0 TSval=66122815
TSecr=65949312 SLE=4294967052 SRE=1<br/>
node01-public         node04-public         TCP      929    [TCP Retransmission]
tinc(655) → 40690 [PSH, ACK] Seq=1 Ack=1 Win=240 Len=843 TSval=66122880
TSecr=65949312<br/>
fe80::1471:c8ff:fe7b:1003 1111:1::4             ICMPv6   86     Neighbor
Solicitation for 1111:1::4 from 16:71:c8:7b:10:03<br/>
fe80::1471:c8ff:fe7b:1003 ff02::1:ff00:1        ICMPv6   86     Neighbor
Solicitation for 1111:1::1 from 16:71:c8:7b:10:03<br/>
node02-vpn            fe80::1471:c8ff:fe7b:1003 ICMPv6   86     Neighbor
Advertisement 1111:1::1 (sol, ovr) is at 96:6a:04:92:56:4e<br/>
node01-vpn            node01-public         UDP      113    tinc(655) →
tinc(655) Len=51<br/>
node01-vpn            node01-public         UDP      1294   tinc(655) →
tinc(655) Len=1232<br/>
node01-vpn            node01-public         UDP      113    tinc(655) →
tinc(655) Len=51<br/>
node02-vpn            ff02::1:ff00:3        ICMPv6   86     Neighbor
Solicitation for 1111:1::3 from 96:6a:04:92:56:4e<br/>
node01-vpn            node02-vpn            ICMPv6   86     Neighbor
Advertisement 1111:1::3 (sol, ovr) is at 16:71:c8:7b:10:03<br/>
node01-vpn            node01-public         UDP      113    tinc(655) →
tinc(655) Len=51<br/>
node02-vpn            node01-vpn            UDP      113    tinc(655) →
tinc(655) Len=51<br/>
node02-vpn            node01-vpn            UDP      113    tinc(655) →
tinc(655) Len=51<br/>
node01-vpn            node02-vpn            UDP      113    tinc(655) →
tinc(655) Len=51<br/>
node01-vpn            node02-vpn            UDP      113    tinc(655) →
tinc(655) Len=51<br/>
node01-vpn            node01-public         UDP      113    tinc(655) →
tinc(655) Len=51<br/>
node02-vpn            node01-vpn            UDP      1294   tinc(655) →
tinc(655) Len=1232<br/>
node02-vpn            node01-vpn            UDP      113    tinc(655) →
tinc(655) Len=51<br/>
node01-vpn            node02-vpn            UDP      113    tinc(655) →
tinc(655) Len=51<br/>
node01-vpn            node02-vpn            UDP      113    tinc(655) →
tinc(655) Len=51<br/>
node01-vpn            node02-vpn            UDP      1294   tinc(655) →
tinc(655) Len=1232<br/>
node02-vpn            node01-vpn            UDP      113    tinc(655) →
tinc(655) Len=51<br/>
node02-vpn            node01-vpn            UDP      113    tinc(655) →
tinc(655) Len=51<br/>
node01-vpn            node02-vpn            UDP      113    tinc(655) →
tinc(655) Len=51<br/>
node01-vpn            node02-vpn            UDP      113    tinc(655) →
tinc(655) Len=51<br/>
node02-vpn            node01-vpn            UDP      1253   tinc(655) →
tinc(655) Len=1191<br/>
node02-vpn            node01-vpn            UDP      1158   tinc(655) →
tinc(655) Len=1096<br/>
node02-vpn            node01-vpn            UDP      1063   tinc(655) →
tinc(655) Len=1001<br/>
node02-vpn            node01-vpn            UDP      968    tinc(655) →
tinc(655) Len=906<br/>
node02-vpn            node01-vpn            UDP      873    tinc(655) →
tinc(655) Len=811<br/>
node02-vpn            node01-vpn            UDP      778    tinc(655) →
tinc(655) Len=716<br/>
node02-vpn            node01-vpn            UDP      683    tinc(655) →
tinc(655) Len=621<br/>
node02-vpn            node01-vpn            UDP      1253   tinc(655) →
tinc(655) Len=1191<br/>
node02-vpn            node01-vpn            UDP      588    tinc(655) →
tinc(655) Len=526<br/>
node02-vpn            node01-vpn            UDP      1158   tinc(655) →
tinc(655) Len=1096<br/>
node02-vpn            node01-vpn            UDP      493    tinc(655) →
tinc(655) Len=431<br/>
node02-vpn            node01-vpn            UDP      1063   tinc(655) →
tinc(655) Len=1001<br/>
node02-vpn            node01-vpn            UDP      398    tinc(655) →
tinc(655) Len=336<br/>
node02-vpn            node01-vpn            UDP      968    tinc(655) →
tinc(655) Len=906<br/>
node02-vpn            node01-vpn            UDP      303    tinc(655) →
tinc(655) Len=241<br/>
node02-vpn            node01-vpn            UDP      873    tinc(655) →
tinc(655) Len=811<br/>
node02-vpn            node01-vpn            UDP      208    tinc(655) →
tinc(655) Len=146<br/>
node02-vpn            node01-vpn            UDP      778    tinc(655) →
tinc(655) Len=716<br/>
node02-vpn            node01-vpn            UDP      113    tinc(655) →
tinc(655) Len=51<br/>
node02-vpn            node01-vpn            UDP      683    tinc(655) →
tinc(655) Len=621</div>

<div>...</div>
</div>
</div>

<div> </div>

<div> </div>

<div>Kind regards</div>

<div>Christopher</div>

<div> </div>

<div>
<div name="quote" style="margin:10px 5px 5px 10px; padding:
10px 0 10px 10px; border-left:2px solid #C3D9E5; word-wrap: break-word;
-webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
<div style="margin:0 0 10px 0;"><b>Gesendet:</b>
Donnerstag, 02. Mai 2019 um 23:00 Uhr<br/>
<b>Von:</b> "Lars Kruse"
<lists@sumpfralle.de><br/>
<b>An:</b> tinc@tinc-vpn.org<br/>
<b>Betreff:</b> Re: very high traffic without any load</div>

<div name="quoted-content">Hello Christoph,<br/>
<br/>
<br/>
Am Thu, 2 May 2019 19:42:25 +0200<br/>
schrieb "Christopher Klinge" <Christ.Klinge@web.de>:<br/>
<br/>> all of my servers where set up fresh with no other applications
running<br/>
> besides tinc and my ssh sessions.<br/><br/>
Did you try something as simple as "tcpdump -npi
TINC_INTERFACE"?<br/>
This should give you a good impression of the traffic flowing through the
VPN.<br/>
<br/>
Cheers,<br/>
Lars<br/>
_______________________________________________<br/>
tinc mailing list<br/>
tinc@tinc-vpn.org<br/>
<a href="https://www.tinc-vpn.org/cgi-bin/mailman/listinfo/tinc"
target="_blank">https://www.tinc-vpn.org/cgi-bin/mailman/listinfo/tinc</a></div>
</div>
</div>
</div></div></body></html>

Ann Brown

2019-May-04 00:40 UTC

head link

Aw: Re: very high traffic without any load

On Friday, May 3, 2019 6:06 PM, Christopher Klinge <Christ.Klinge at
web.de> wrote:
> Hi again,
>
> I did some digging, and thus far I could not find any other culprit other
than tinc itself. The packages that are being sent are addressed directly to the
other tinc hosts on their vpn addresses. During my latest tests, within about 12
seconds 100MB of data were transmitted this way. I captured this test in
wireshark. Sadly, I lack the expertise of understanding what is happening.
>
> At the very beginning, normal connections are being set up and a few ICMP
neighbor advertisements/solicitations are being exchanged. Next a short TCP
session was created between the public IP addresses of two of my hosts, through
the VPN. This is something that I would like to support and theoretically it
should be possible. However, to me it looks like the connection could not be
established. Right afterwards, one of the nodes involved in the TCP connection
and the third node I used for testing started exchaning the weird packages that
I am complaining about.
>
> They are UDP packets varying in size and as far as I can tell unrelated to
any outside application. All of them belong to one connection. If anyone would
like to take a look at the dump itself, I'll provide it directly, since I
don't want to make all of my servers' addresses public.
<snip>

Could this be tinc connecting via tinc?

Perhaps try ip6tables rules.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://www.tinc-vpn.org/pipermail/tinc/attachments/20190504/46ff9758/attachment-0001.html>

Lars Kruse

2019-May-04 00:45 UTC

head link

very high traffic without any load

Hello Christopher,


Am Fri, 3 May 2019 20:06:54 +0200
schrieb "Christopher Klinge" <Christ.Klinge at web.de>:
> I did some digging, and thus far I could not find any other culprit other
> than tinc itself. The packages that are being sent are addressed directly
to
> the other tinc hosts on their vpn addresses. During my latest tests, within
> about 12 seconds 100MB of data were transmitted this way.
Just in order to avoid any misunderstandings:
* you took a look at the traffic *through* the tinc network interface
  (this should be the payload that you expect to see floating through your VPN)
* this traffic uses the internal VPN addresses of your VPN
  (we expect this)
* you are surprised by the amount of traffic

This sounds like a routing issue.
(traffic passing through the VPN that should take a different path)

> At the very beginning, normal connections are being set up and a few ICMP
> neighbor advertisements/solicitations are being exchanged. Next a short TCP
> session was created between the public IP addresses of two of my hosts,
> through the VPN.
What do you mean with "session"? Some http-requests that you are
sending
through the VPN? Or something special?

> Warning, wall of text incoming:
> Source                Destination           Protocol Length Info
> node01-public         node04-public         TCP      929    tinc(655) →
40690 [PSH, ACK] Seq=1 Ack=1 Win=240 Len=843 TSval=66121145 TSecr=65947641
> node01-public         node04-public         TCP      1294   tinc(655) →
40690 [ACK] Seq=844 Ack=1 Win=240 Len=1208 TSval=66121145 TSecr=65947641
> [..]
The packets above belong to the tinc connection. They should be routed through
your uplink network interfaces (or whatever is between your tinc peers).

If you really see these packets *within* the tinc network, then it is very
likely that you were adding some routes after establishing the tinc VPN. Maybe
these routes changed the path of the connection between the tinc peers.
Obviously the tinc traffic between the peers may *never* go through the
VPN itself.
Thus you may want to verify, that the routes on the tinc peers (while the
VPN is established) meet your expectations. Maybe you want to share these
(obfuscated) routes with us?
(just run "ip route" on both hosts)

Cheers,
Lars

Christopher Klinge

2019-May-04 06:30 UTC

head link

Aw: Re: very high traffic without any load

<html><head></head><body><div
style="font-family: Verdana;font-size: 12.0px;"><div>
<div>Hi everyone,</div>

<div> </div>

<div> </div>

<div>>> I did some digging, and thus far I could not find any other
culprit other<br/>>> than tinc itself. The packages that are being sent are addressed
directly to<br/>
>> the other tinc hosts on their vpn addresses. During my latest tests,
within<br/>
>> about 12 seconds 100MB of data were transmitted this way.<br/>
><br/>
> Just in order to avoid any misunderstandings:<br/>
> * you took a look at the traffic *through* the tinc network
interface<br/>
> (this should be the payload that you expect to see floating through your
VPN)<br/>
> * this traffic uses the internal VPN addresses of your VPN<br/>
> (we expect this)<br/>
> * you are surprised by the amount of traffic<br/>
><br/>
> This sounds like a routing issue.<br/>
> (traffic passing through the VPN that should take a different
path)</div>
<div> </div>

<div>1. Yes, I ran wireshark on the interface vpn0, which is the set up by
tinc.</div>

<div> </div>

<div>2. and 3. Yes.</div>

<div> </div>

<div> </div>

<div>>> At the very beginning, normal connections are being set up
and a few ICMP<br/>>> neighbor advertisements/solicitations are being exchanged. Next a short
TCP<br/>
>> session was created between the public IP addresses of two of my
hosts,<br/>
>> through the VPN.<br/>
><br/>
> What do you mean with "session"? Some http-requests that you are
sending<br/>
> through the VPN? Or something special?</div>
<div>
<div> </div>

<div>Just like before, there should not have been any payload traffic
involved. I was referring to the TCP connection you attribute to tinc
itself.</div>

<div> </div>

<div> </div>

<div>>> Thus you may want to verify, that the routes on the tinc
peers (while the<br/>>> VPN is established) meet your expectations. Maybe you want to share
these<br/>
>> (obfuscated) routes with us?</div>
<div> </div>

<div>I think I understand the problem now. My node-up files look like
this:</div>

<div> </div>

<div># node01-up</div>

<div>
<div>ip -6 route del 1111:1:1::/64 via 1111:1::1<br/>
ip -6 route del <node01's public ipv6>/64 via 1111:1::1</div>

<div> </div>

<div>I just tested whether the second rule causes the issue and it does.
When I remove this line from all node-ups on all but one of my hosts, everything
is fine as long as that "faulty" host is not up. As soon as I run tinc
on the host with the additional configuration line, things start going haywire.
<div>My intention was to make my nodes route traffic to their respective
public interfaces through the VPN as well. The goal was that all connections
between my nodes, regardless of whether they are destined for internal or
external ipv6 addresses, end up using the VPN.</div>

<div> </div>
</div>

<div>Kind regards and thank you very much for your help</div>

<div>Christopher</div>
</div>
</div>
</div></div></body></html>

Reasonably Related Threads

Search for more maybe matching threads

tinc - May 2019 - Aw: Re: very high traffic without any load

Aw: Re: very high traffic without any load

Aw: Re: very high traffic without any load

very high traffic without any load

Aw: Re: very high traffic without any load

Reasonably Related Threads