openssh bugs - Jun 2006 - [Bug 1195] sftp corrupts large files on new macbook pro

http://bugzilla.mindrot.org/show_bug.cgi?id=1195

           Summary: sftp corrupts large files on new macbook pro
           Product: Portable OpenSSH
           Version: 4.3p2
          Platform: ix86
        OS/Version: Mac OS X
            Status: NEW
          Severity: normal
          Priority: P2
         Component: sftp
        AssignedTo: bitbucket at mindrot.org
        ReportedBy: ski at allafrica.com


(originally reported to apple bug reporter, bug # 4529709, but they
punted)

Bug ID #: 4529709
Bug Title: sftp corrupts large files on new macbook pro
-------------------------------------------------------
<GMT27-Apr-2006 20:27:29GMT> Darrin Bodner:
Summary:
Large files (on the order of 1GB) are sometimes (~50%) corrupted when
transferred by using sftp on a new macbookpro (mac os x/intel arch)

Steps to Reproduce:
Enable sshd.
$ sftp localhost
> > get largefile largefile.copy
$ md5 largefile
$ md5 largefile.copy

The results usually differ. Using diff(1) confirms this is not a bug
with md5. Subsequent transfers can have the correct checksum or
different incorrect checksums, suggesting the corruption does not
always happen in the same place. sftp'ing a file from a remote server
running freebsd 6.1 - OpenSSH_4.2p1 FreeBSD-20050903, OpenSSL 0.9.7e-p1
25 Oct 2004 - also gives this problem.

Expected Results:
MD5 sums match, resulting file is not corrupt.

Actual Results:
File is corrupted ~50% of time for a 1GB file. Corruption rate seems to
increase with size.

Regression:
Tested on a new macbook pro with mac os x 10.4.6 with all patches
applied as of 4/27/2006. Transfers from two different freebsd machines
have similar results (different md5 sum, corrupted data).

Notes:
openssh from darwinports's sftp client has the same problem, so there
is likely an upstream bug. strangely, scp does not exhibit this bug
with either the darwinport or apple's built in command.




------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

http://bugzilla.mindrot.org/show_bug.cgi?id=1195


ski at allafrica.com changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |ski at allafrica.com






------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

http://bugzilla.mindrot.org/show_bug.cgi?id=1195





------- Comment #1 from ski at allafrica.com  2006-06-09 07:24 -------
Bug ID #: 4529709
Bug Title: sftp corrupts large files on new macbook pro
-------------------------------------------------------
<GMT27-Apr-2006 20:27:29GMT> Darrin Bodner:
Summary:
Large files (on the order of 1GB are sometimes (~50%) corrupted when
transferred by using sftp).

Steps to Reproduce:
Enable sshd.
$ sftp localhost
> > get largefile largefile.copy
$ md5 largefile
$ md5 largefile.copy

The results usually differ. Using diff(1) confirms this is not a bug
with md5. Subsequent transfers can have the correct checksum or
different incorrect checksums, suggesting the corruption does not
always happen in the same place. Grabbing the file from a freebsd 6.1
server (with OpenSSH_4.2p1 FreeBSD-20050903, OpenSSL 0.9.7e-p1 25 Oct
2004) shows the same problem, indicating it is probably not a problem
with the server, but in the client instead.

Expected Results:
MD5 sums match, resulting file is not corrupt.

Actual Results:
File is corrupted ~50% of time for a 1GB file. Corruption rate seems to
increase with size.

Regression:
Tested on a new macbook pro with mac os x 10.4.6 with all patches
applied as of 4/27/2006. Transfers from two different freebsd machines
have similar results (different md5 sum, corrupted data).

Notes:
openssh from darwinports's sftp client has the same problem, so there
is likely an upstream bug. strangely, scp does not exhibit this bug
with either the darwinport or apple's built in command.




------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

http://bugzilla.mindrot.org/show_bug.cgi?id=1195





------- Comment #2 from dtucker at zip.com.au  2006-06-09 20:15 -------
Does OpenSSL's self-test ("make tests") pass on that platform? 
Also,
does enabling/disabling compression make any difference?  Do the files
all end up exactly the same size?

The thing that occurs to me as a possible cause is pipes/socketpairs:
both sftp and scp use them to talk to the underlying ssh process, but
scp uses a small fixed buffer (2kb) and sftp uses a variable sized
buffer that defaults to 32kb.  If you repeat your test forcing the sftp
buffer to 2k ("sftp -B 2048") does the corruption still occur?




------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.