openssh unix dev - Nov 2018 - openssh-based file transfers (e.g. rsync, scp, ...) are running 40 (!!) times faster via IPv4 than IPv6

openssh 7.6p1-lp150.7.4 on OpenSuse Leap 15 (both server and client)



Hi all,


first post to list, hopefully on-topic. Haven't found anything on the net,
tried to ask at first in OpenSuse forums a
while ago
(https://forums.opensuse.org/showthread.php/533588-rsnapshot-rsync-massive-performance-decrease)
and today
opened a bug in OpenSuse's Bugzilla
(https://bugzilla.opensuse.org/show_bug.cgi?id=1115075)

As written in bug report, as general network issues could be excluded, any hints
where to start, how to narrow down if
this is a specific openssh issue or something else?


Thanks in advance,
Michael

Il giorno mer 7 nov 2018 alle ore 17:58 michaelof at rocketmail.com
<michaelof at rocketmail.com> ha scritto:
>
> openssh 7.6p1-lp150.7.4 on OpenSuse Leap 15 (both server and client)
>
>
>
> Hi all,
>
>
> first post to list, hopefully on-topic. Haven't found anything on the
net, tried to ask at first in OpenSuse forums a
> while ago
(https://forums.opensuse.org/showthread.php/533588-rsnapshot-rsync-massive-performance-decrease)
and today
> opened a bug in OpenSuse's Bugzilla
(https://bugzilla.opensuse.org/show_bug.cgi?id=1115075)
>
> As written in bug report, as general network issues could be excluded, any
hints where to start, how to narrow down if
> this is a specific openssh issue or something else?
>
>
> Thanks in advance,
> Michael

Can you try the same transfer with a (totally) different protocol,
like FTP, NFS, CIFS?
This is to exclude any network-related slowdown.

-- 
Vincenzo Romano - NotOrAnd.IT
Information Technologies
--
NON QVIETIS MARIBVS NAVTA PERITVS

Vincenzo,


thanks for answering !!!

As I found out before that this slow down only happens for encrypted
transmissions, I've followed your suggestion and
tested with a https-based download, from my Nextcloud instance on my VPS.

Same slow speed when connecting via IPv6, and as fast as expected when
connecting via IPv4.

Just to confirm: There's NO dependency/relation between openssh and https,
on Linux, correct? (Web Server is Apache)
If NO, sorry for wasting your time, and again THANK YOU for leading me into
another direction!


Regards,
Michael


Am 07.11.18 um 19:05 schrieb Vincenzo Romano:
> Il giorno mer 7 nov 2018 alle ore 17:58 michaelof at rocketmail.com
> <michaelof at rocketmail.com> ha scritto:
>>
>> openssh 7.6p1-lp150.7.4 on OpenSuse Leap 15 (both server and client)
>>
>>
>>
>> Hi all,
>>
>>
>> first post to list, hopefully on-topic. Haven't found anything on
the net, tried to ask at first in OpenSuse forums a
>> while ago
(https://forums.opensuse.org/showthread.php/533588-rsnapshot-rsync-massive-performance-decrease)
and today
>> opened a bug in OpenSuse's Bugzilla
(https://bugzilla.opensuse.org/show_bug.cgi?id=1115075)
>>
>> As written in bug report, as general network issues could be excluded,
any hints where to start, how to narrow down if
>> this is a specific openssh issue or something else?
>>
>>
>> Thanks in advance,
>> Michael
> 
> 
> Can you try the same transfer with a (totally) different protocol,
> like FTP, NFS, CIFS?
> This is to exclude any network-related slowdown.
>

> first post to list, hopefully on-topic. Haven't found anything on the
> net, tried to ask at first in OpenSuse forums a
> while ago
>
(https://forums.opensuse.org/showthread.php/533588-rsnapshot-rsync-massive-performance-decrease)
> and today
> opened a bug in OpenSuse's Bugzilla
> (https://bugzilla.opensuse.org/show_bug.cgi?id=1115075)
> 
> As written in bug report, as general network issues could be excluded,
> any hints where to start, how to narrow down if
> this is a specific openssh issue or something else?
Your IPv6 _route_ has 3 times the latency:

   $ time /usr/sbin/traceroute vserver.domain.tld
   ...
    7  domain.tld (IPv4)  26.002 ms !X  24.859 ms !X  25.230 ms !X

   $ time /usr/sbin/traceroute6 vserver.domain.tld
   ...
    8  domain.tld (IPv6)  80.430 ms !X  79.301 ms !X  80.444 ms !X

The big time waster seems to be this link:
    6  nug-d-i40-v6.telia.net (2001:2000:3018:8d::1)  32.046 ms  23.850 
ms  24.097 ms
    7  contabo-ic-305268-ffm-b11.c.telia.net (2001:2000:3080:953::2)  
74.829 ms  77.704 ms  78.066 ms

I'll now speculate that you might as well have some MTU mismatches,
meaning that packages have to get split up along the way, further
slowing down your connection.

Do you have native IPv6 all the way, or is there an 6-in-4 tunnel
inbetween (from the fritzbox on)?

Please try
    # ping -M do -s <size> <ip>
for sizes between 1300 and 1500 and find the largest (even) packet size
that still works, and then do the same with ping6 as well.

Also note whether some packetsize works, but just larger ones vanish
along the way.

Servus Philipp,


Unfortunately the traceroute(6) results are both more or less random. Sometimes
traceroute "hangs" a while, wherever,
sometimes traceroute6. Sometimes traceroute is faster, sometimes traceroute6.
Not reliable.

Your MTU question, tried as adviced:

Maximum size for IPv4 is 1466, and max size for IPv6 is 1444. Exceeding these
values leads to a "ping: local error:
Message too long, mtu=1492" in both cases.

My Fritzbox says "Native IPv6", no tunnel, no underlying "IPv4
via DS-Lite". These are the default settings for ISP
"Deutsche Telekom", VDSL-50.


VINCENZO: Tried as adviced to test with FTP, installed pure-ftpd and opened
VPS's ftp port in firewall. Maximum IPv4
speed was much slower (1.5-2 MB/sec) than other tests, maybe temporary issue or
virtualization. But again: IPv6 was MUCH
slower than IPv4.





Am 07.11.18 um 20:57 schrieb Philipp Marek:
> 
>> first post to list, hopefully on-topic. Haven't found anything on
the
>> net, tried to ask at first in OpenSuse forums a
>> while ago
>>
(https://forums.opensuse.org/showthread.php/533588-rsnapshot-rsync-massive-performance-decrease)
>> and today
>> opened a bug in OpenSuse's Bugzilla
>> (https://bugzilla.opensuse.org/show_bug.cgi?id=1115075)
>>
>> As written in bug report, as general network issues could be excluded,
>> any hints where to start, how to narrow down if
>> this is a specific openssh issue or something else?
> 
> Your IPv6 _route_ has 3 times the latency:
> 
> ? $ time /usr/sbin/traceroute vserver.domain.tld
> ? ...
> ?? 7? domain.tld (IPv4)? 26.002 ms !X? 24.859 ms !X? 25.230 ms !X
> 
> ? $ time /usr/sbin/traceroute6 vserver.domain.tld
> ? ...
> ?? 8? domain.tld (IPv6)? 80.430 ms !X? 79.301 ms !X? 80.444 ms !X
> 
> The big time waster seems to be this link:
> ?? 6? nug-d-i40-v6.telia.net (2001:2000:3018:8d::1)? 32.046 ms? 23.850 ms?
24.097 ms
> ?? 7? contabo-ic-305268-ffm-b11.c.telia.net (2001:2000:3080:953::2)? 74.829
ms? 77.704 ms? 78.066 ms
> 
> I'll now speculate that you might as well have some MTU mismatches,
> meaning that packages have to get split up along the way, further
> slowing down your connection.
> 
> Do you have native IPv6 all the way, or is there an 6-in-4 tunnel
> inbetween (from the fritzbox on)?
> 
> Please try
> ?? # ping -M do -s <size> <ip>
> for sizes between 1300 and 1500 and find the largest (even) packet size
> that still works, and then do the same with ping6 as well.
> 
> Also note whether some packetsize works, but just larger ones vanish
> along the way.