openssh bugs - Jun 2017 - [Bug 2726] New: Uploading of large files (1GB+) fails when using SFTP in chrooted configuration

https://bugzilla.mindrot.org/show_bug.cgi?id=2726

            Bug ID: 2726
           Summary: Uploading of large files (1GB+) fails when using SFTP
                    in chrooted configuration
           Product: Portable OpenSSH
           Version: 7.3p1
          Hardware: amd64
                OS: Linux
            Status: NEW
          Severity: normal
          Priority: P5
         Component: sftp
          Assignee: unassigned-bugs at mindrot.org
          Reporter: artur.maj at gmail.com

Created attachment 2986
  --> https://bugzilla.mindrot.org/attachment.cgi?id=2986&action=edit
SSHD config used in a test chrooted environment

I am unable unable to upload large files (1GB or greater) via SFTP when
using SFTP in a chrooted configuration (using ChrootDirectory and
"ForceCommand internal-sftp" in SSHD config). The symptomps are that
WinSCP (or other SFTP clients) interrupts the connection with SFTP
server once number of bytes transferred exceeds 1GB. Further diagnosis
showed that this behaviour is related to the value of Rekeylimit - when
reaching the Rekeylimit (as set explicitly in config file) the
connection is interrupted immediately. Unfortunately, setting it's
value to "none" does not solve the problem - probably SSHD takes the
default value (1GB?) and the connection is interrupted when number of
transferred bytes exceeds this value. SSHD ignores values larger than
1G so the one cannot set it over 1GB. 

I did not experience this bug when using SFTP without chrooting it - in
a "regular" SSHD/SFTP configuration session key is renegotiated
properly and large files can be transferred without problems.

Verified on the latest Gentoo and Ubuntu 16.04 LTS operating systems.

Attached you can find reference SSHD config that can be used to
replicate the issue. Rekeylimit can be set, for instance, to 10M to
experience the issue with smaller files.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2726

Damien Miller <djm at mindrot.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |djm at mindrot.org

--- Comment #1 from Damien Miller <djm at mindrot.org> ---
To diagnose this, we really need a debug log from the server. You can
get one by running "ssh -ddd -p 2222" and connecting to port 2222 with
your sftp client. Please attach the resulting debug trace.

Are you using sshd that you compiled from source or your OS vendors'
packages?

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2726

Darren Tucker <dtucker at zip.com.au> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |dtucker at zip.com.au

--- Comment #2 from Darren Tucker <dtucker at zip.com.au> ---
(In reply to Artur from comment #0)
> SSHD ignores values larger than 1G so the one cannot set
> it over 1GB. 
Note that as per RFC4344 that depends on the cipher.

"""
   Let L be the block length (in bits) of an SSH encryption method's
   block cipher (e.g., 128 for AES).  If L is at least 128, then, after
   rekeying, an SSH implementation SHOULD NOT encrypt more than
2**(L/4)
   blocks before rekeying again.
[...]
   If L is less than 128 (which is the case
   for older ciphers such as 3DES, Blowfish, CAST-128, and IDEA), then,
   [...] rekey at least once for every gigabyte
"""

sshd follows this, so you'll see more frequent rekeying for ciphers
like 3des.  For AES ciphers it should be 4GB, so if you're using 3des
you might be able to work around the problem by changing ciphers.

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2726

--- Comment #3 from Artur <artur.maj at gmail.com> ---
Created attachment 2992
  --> https://bugzilla.mindrot.org/attachment.cgi?id=2992&action=edit
Debug log file

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2726

--- Comment #4 from Artur <artur.maj at gmail.com> ---
Here you go, you can find the debug log from sample test server as
attachment.

The SSHD was started, then I tried to upload 43.5M file via WinSCP,
while RekeyLimit was set to "5M none" (just to speed up occurance of
rekey issue, without this it would happen at 1GB). As expected,
transfer broke at 5M with WinSCP reporting an issue "Server
unexpectedly closed network connection".

The example comes from the Gentoo OS, so it is compiled from the source
(this is a default installation approach on Gentoo) using the below
parameters:

net-misc/openssh-7.3_p1-r7::gentoo  USE="bindist hpn pam (pie)
(selinux) ssl -X -X509 -debug -kerberos -ldap -ldns -libedit
(-libressl) -livecd -sctp -skey -ssh1 -static {-test}" 0 KiB

Regards,
Artur

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2726

--- Comment #5 from Artur <artur.maj at gmail.com> ---
Created attachment 2993
  --> https://bugzilla.mindrot.org/attachment.cgi?id=2993&action=edit
WinSCP debug log

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2726

--- Comment #6 from Damien Miller <djm at mindrot.org> ---
(In reply to Artur from comment #4)
> net-misc/openssh-7.3_p1-r7::gentoo  USE="bindist hpn pam (pie)
> (selinux) ssl -X -X509 -debug -kerberos -ldap -ldns -libedit
> (-libressl) -livecd -sctp -skey -ssh1 -static {-test}" 0 KiB
That is OpenSSH plus a number of third-party patches that we neither
support nor maintain. Please try to reproduce against OpenSSH that
you've built from source yourself, or take this bug up with your OS
vendor.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2726

--- Comment #7 from Artur <artur.maj at gmail.com> ---
The same behavior I experienced on default Ubuntu 16.04 LTS distro
(installed from the package) - from my point of view it seems the bug
does not depend on a particular Linux distro.

As suggested ealier, you can easily replicate it on other Linux
variants by using the attached sshd config.

Please let me know if the logs attached are sufficient or you still
need logs from eg. Ubuntu, to re-assure this bug is OS independent.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2726

--- Comment #8 from Darren Tucker <dtucker at zip.com.au> ---
(In reply to Artur from comment #7)
> The same behavior I experienced on default Ubuntu 16.04 LTS distro
Ubuntu 16.04 ships an older version of OpenSSH than this bug report is
against.
> (installed from the package) - from my point of view it seems the
> bug does not depend on a particular Linux distro.
Please consider it from our point of view: you are asking us to debug a
system we have never seen, cannot interact with and, in the case of
Gentoo, may well be unique.  There are any number of differences
between that and source code we can debug, and we do not know what they
are.  If we cannot replicate locally (and I've tried your config on
-current on OpenBSD and 7.3p1 on Fedora and failed, despite those
sessions rekeying multiple times) then we do not know out what the
issue is.

So, there are a number of differences between the system that you're
experiencing the problem on and the ones we can observe.

If you wish to make progress with this bug then your choices are:
a) help us reduce the variables by testing the *most recent version of*
code we actually ship
b) report you problem to the people who ship the thing you are having
the problem with (and they can report it to us if it turns out to be in
the code we ship).
> Please let me know if the logs attached are sufficient or you still
> need logs from eg. Ubuntu, to re-assure this bug is OS independent.
No, we need logs from the code we ship, not from some other random
binary.

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2726

Darren Tucker <dtucker at zip.com.au> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         Resolution|---                         |WORKSFORME
             Status|NEW                         |RESOLVED

--- Comment #9 from Darren Tucker <dtucker at zip.com.au> ---
If you can't provide logs from the code we ship then there's not much
more we can do to help you.  Please feel free to reopen if you are able
to provide them.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2726

Artur <artur.maj at gmail.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|RESOLVED                    |VERIFIED

--- Comment #10 from Artur <artur.maj at gmail.com> ---
I verified OpenSSH behaviour once again, based on the most recent
stable version (7.5p1) compiled from the sources. This version turn to
work properly.

To summary, the reported issue affects default Ubuntu 16.04LTS and
Gentoo distributions. As a workaround users of these systems may need
to download, compile and use OpenSSH from the sources.

Therefore the bug can be closed.

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2726

Damien Miller <djm at mindrot.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|VERIFIED                    |CLOSED

--- Comment #11 from Damien Miller <djm at mindrot.org> ---
Close all resolved bugs after release of OpenSSH 7.7.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.