openssh bugs - Jan 2010 - [Bug 1697] New: scp transfers from remote cygwin machine fail with ssh versions >= 4.6

https://bugzilla.mindrot.org/show_bug.cgi?id=1697

           Summary: scp transfers from remote cygwin machine fail with ssh
                    versions >= 4.6
           Product: Portable OpenSSH
           Version: 4.6p1
          Platform: ix86
        OS/Version: Linux
            Status: NEW
          Severity: major
          Priority: P2
         Component: scp
        AssignedTo: unassigned-bugs at mindrot.org
        ReportedBy: cdavis at ingenuous.net


I am in France and try to load a 20 MB file from a server in
California. Thus from California to France.  In California, the server
is a Dell 2650, running WIndows 2000 with cygwin installed. The ssh
version on that machine is 4.7. The server sits on a DSL line with a
maximum up speed of 250 kbits/s. Down is 1.5 mbit.

In France, I run various versions of Ububtu and Redhat, or windows XP
sp3. In france I am also on a DSL line that is about 4 times faster
than the California one.

For several scenarios, the file transfer stalls transferring a large
file. It stalls after around 50 kBytes. For some other scenarios, the
file transfer succeeds.

the specific command used is this: scp
permsession at xxx.xxx.xxx.xxx:/cygdrive/m/mm/mmdev/* ./

For these setups in France, the transfer fails:

Centos 5.4   ssh version not known.
Ubuntu 9.10  ssh version 5.1
Ubuntu 7.10  ssh version 4.6

In these setups, the transfer succeeds:
Ubuntu 6.10 ssh version 4.3
Windows cygwin ssh version 4.7

I have tried this several times with several i86 computers in France
and three total WIn2k server computers in California and the problem is
100% reproducible.

Here may be a valuable clue:  I did the same experiment, but instead of
loading a big file from my server in California that sits on a DSL
line, I tried to download a big file from one of my nearly identical
servers at a data center in Chicago, again toward France.  In this
scenario, the transfer never fails. All setups work.

-- 
Configure bugmail: https://bugzilla.mindrot.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=1697

Darren Tucker <dtucker at zip.com.au> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |dtucker at zip.com.au

--- Comment #1 from Darren Tucker <dtucker at zip.com.au> 2010-01-13
22:25:50 EST ---
I suspect you have a device in your network path that cannot handle
large TCP windows, probably window scaling.

One of the changes in the 4.7 release was:

 * The SSH channel window size has been increased, and both ssh(1)
   sshd(8) now send window updates more aggressively. These improves
   performance on high-BDP (Bandwidth Delay Product) networks.

Previously this was limited to 64k, which is not enough to exercise TCP
window scaling.  You can confirm this by looking at the "SendQ" field
in the connection in the netstat output on the sending side (this is on
Linux, I don't know how to get the equivalent data on Windows).  If the
number is non-zero and doesn't decrease this is almost certainly the
problem.

You can also try disabling window scaling.  On Linux, that's "echo
0
>/proc/sys/net/ipv4/tcp_window_scaling".  I don't know about Windows
but presumably it's in the registry somewhere.   You'll only need to
disable it on one end and retry the copy.

-- 
Configure bugmail: https://bugzilla.mindrot.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=1697

Charles Davis <cdavis at ingenuous.net> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|                            |FIXED

--- Comment #2 from Charles Davis <cdavis at ingenuous.net> 2010-01-14
01:23:56 EST ---
(In reply to comment #1)
> I suspect you have a device in your network path that cannot handle
> large TCP windows, probably window scaling.
> 
> One of the changes in the 4.7 release was:
> 
>  * The SSH channel window size has been increased, and both ssh(1)
>    sshd(8) now send window updates more aggressively. These improves
>    performance on high-BDP (Bandwidth Delay Product) networks.
> 
> Previously this was limited to 64k, which is not enough to exercise TCP
> window scaling.  You can confirm this by looking at the "SendQ"
field
> in the connection in the netstat output on the sending side (this is on
> Linux, I don't know how to get the equivalent data on Windows).  If the
> number is non-zero and doesn't decrease this is almost certainly the
> problem.
> 
> You can also try disabling window scaling.  On Linux, that's "echo
0
> >/proc/sys/net/ipv4/tcp_window_scaling".  I don't know about
Windows
> but presumably it's in the registry somewhere.   You'll only need
to
> disable it on one end and retry the copy.
This:
 "echo 0 >/proc/sys/net/ipv4/tcp_window_scaling"
completely fixes the problem. Thanks loads. 

As far as I am concerned it is fixed. So I am changing the status.

However, before I came to you fellows, I searched extensively and there
seems to ba a lot of instances of this problem. Is there somewhere I
can add a FAQ or soemthing like that??
Charles Davis

-- 
Configure bugmail: https://bugzilla.mindrot.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=1697

--- Comment #3 from Darren Tucker <dtucker at zip.com.au> 2010-01-14
09:51:37 EST ---
(In reply to comment #2)
> This:
>  "echo 0 >/proc/sys/net/ipv4/tcp_window_scaling"
> completely fixes the problem. Thanks loads. 
You're welcome.

What type of ADSL device is it, so that we can add it to the mental
list of broken ones?

I suspect it's creating a firewall state only in the ethernet -> ADSL
direction, and since the window scale option is only on the first
packet, when you connect from outside the state doesn't know the
connection has window scaling and thus gets the TCP windows wrong.  If
this is the case, you would see that scping out would work (ie the same
data transfer direction but initiated from the other end).
> As far as I am concerned it is fixed. So I am changing the status.
> 
> However, before I came to you fellows, I searched extensively and there
> seems to ba a lot of instances of this problem. Is there somewhere I
> can add a FAQ or soemthing like that??
There's http://www.openssh.com/faq.html.  If you want to write
something I'll be happy to review and/or add it, but it's not a very
common problem (or at least, it's likely that the network brokenness
would be noticed long before anyone starts trying to scp files around).

-- 
Configure bugmail: https://bugzilla.mindrot.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=1697

--- Comment #4 from Charles Davis <cdavis at ingenuous.net> 2010-01-14
17:39:36 EST ---
(In reply to comment #3)
In reply to comment #2)
> > This:
> >  "echo 0 >/proc/sys/net/ipv4/tcp_window_scaling"
> > completely fixes the problem. Thanks loads. 
> 
> You're welcome.
> 
> What type of ADSL device is it, so that we can add it to the mental
> list of broken ones?
> 
Its some sort of Cisco router. When I disabled its firewall, the
problem went away. Further screwing around with the firewall has caused
me to get myself locked out from the installation, hence the lack of
detail. When I get back in I'll provide specifics. THanks again.

> I suspect it's creating a firewall state only in the ethernet ->
ADSL
> direction, and since the window scale option is only on the first
> packet, when you connect from outside the state doesn't know the
> connection has window scaling and thus gets the TCP windows wrong.  If
> this is the case, you would see that scping out would work (ie the same
> data transfer direction but initiated from the other end).
> 
> > As far as I am concerned it is fixed. So I am changing the status.
> > 
> > However, before I came to you fellows, I searched extensively and
there
> > seems to ba a lot of instances of this problem. Is there somewhere I
> > can add a FAQ or soemthing like that??
> 
> There's http://www.openssh.com/faq.html.  If you want to write
> something I'll be happy to review and/or add it, but it's not a
very
> common problem (or at least, it's likely that the network brokenness
> would be noticed long before anyone starts trying to scp files around).
-- 
Configure bugmail: https://bugzilla.mindrot.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=1697

--- Comment #5 from Charles Davis <cdavis at ingenuous.net> 2010-01-15
21:06:32 EST ---
Offending router is Linksys RV082. Seems to work fine if the firewall
is disabled.

-- 
Configure bugmail: https://bugzilla.mindrot.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=1697

--- Comment #6 from Charles Davis <cdavis at ingenuous.net> 2010-01-27
09:29:26 EST ---
RV082 router no longer exhibits the problem is the latest (2.0.0.19)
firmware is used.

-- 
Configure bugmail: https://bugzilla.mindrot.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=1697

Damien Miller <djm at mindrot.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|RESOLVED                    |CLOSED

--- Comment #7 from Damien Miller <djm at mindrot.org> 2010-04-16 15:51:04
EST ---
Mass move of bugs RESOLVED->CLOSED following the release of
openssh-5.5p1

-- 
Configure bugmail: https://bugzilla.mindrot.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.