openssh bugs - Aug 2013 - [Bug 2147] New: OpenSSH remote forwarding of dynamic ports doesn't work when you create more than one

https://bugzilla.mindrot.org/show_bug.cgi?id=2147

            Bug ID: 2147
           Summary: OpenSSH remote forwarding of dynamic ports doesn't
                    work when you create more than one
           Product: Portable OpenSSH
           Version: -current
          Hardware: All
                OS: All
            Status: NEW
          Severity: normal
          Priority: P5
         Component: sshd
          Assignee: unassigned-bugs at mindrot.org
          Reporter: ronf at timeheart.net

Created attachment 2330
  --> https://bugzilla.mindrot.org/attachment.cgi?id=2330&action=edit
Patch for remote forwarding of dynamic ports

I recently ran across a problem with remote port forwarding in OpenSSH
when trying to use dynamic ports. While it is possible to use OpenSSH
to request a dynamic port and the OpenSSH sshd handles it just fine,
the OpenSSH client gets confused when multiple ports are opened this
way, due to the information passed in the "forwarded-tcpip"
SSH_MSG_CHANNEL_OPEN message which is sent back to the client when
connections are opened. To illustrate this problem, I tried the
following with OpenSSH 6.2p1:

ssh -v -R 0:localhost:80 -R 0:localhost:81 localhost

In the debug output, I saw the lines:

debug1: remote forward success for: listen 0, connect localhost:80
Allocated port 60013 for remote forward to localhost:80
debug1: Updating allowed port 60013 for forwarding to host localhost
port 80
debug1: remote forward success for: listen 0, connect localhost:81
Allocated port 60014 for remote forward to localhost:81
debug1: Updating allowed port 60014 for forwarding to host localhost
port 81

So far, so good!

Connecting to port 60013 worked fine, causing the client to connect to
localhost:80 as requested, with the following debug output:

debug1: client_input_channel_open: ctype forwarded-tcpip rchan 5 win
2097152 max 32768
debug1: client_request_forwarded_tcpip: listen localhost port 0,
originator ::1 port 60153
debug1: connect_next: host localhost ([::1]:80) in progress, fd=9
debug1: channel 1: new [::1]
debug1: confirm forwarded-tcpip
debug1: channel 1: connected to localhost port 80

However, connecting to port 60014 did not work as expected. In that
case, I saw:

debug1: client_input_channel_open: ctype forwarded-tcpip rchan 5 win
2097152 max 32768
debug1: client_request_forwarded_tcpip: listen localhost port 0,
originator ::1 port 60182
debug1: connect_next: host localhost ([::1]:80) in progress, fd=9
debug1: channel 1: new [::1]
debug1: confirm forwarded-tcpip
debug1: channel 1: connected to localhost port 80

Note that even though this was to the second listening port I set up,
the connection was locally forwarded to port 80. The reason is that the
SSH_MSG_CHANNEL_OPEN of type "forwarded-tcpip" from sshd reported the
destination host & port as "localhost" port 0 for both the 60013
and
60014 connections, instead of reporting the actual listening port. The
client seems to expect the 0 value here and does the right thing when
you only have one dynamic listening port, but this breaks in the case
where there are multiple, since there's nothing in the
SSH_MSG_CHANNEL_OPEN which distinguishes between the two dynamic
listeners. RFC 4254 is not completely clear on what is expected in the
message. It says:

  When a connection comes to a port for which remote forwarding has
  been requested, a channel is opened to forward the port to the other
  side.

     byte      SSH_MSG_CHANNEL_OPEN
     string    "forwarded-tcpip"
     uint32    sender channel
     uint32    initial window size
     uint32    maximum packet size
     string    address that was connected
     uint32    port that was connected
     string    originator IP address
     uint32    originator port

I was expecting "port that was connected" in this message to be the
dynamically allocated port so that it would always be a unique value,
but this is not the case (at least with OpenSSH's sshd). Instead, it
always seems to be the "port number to bind" value passed in the
original SSH_MSG_GLOBAL_REQUEST "tcpip-forward" message (which is 0
since we're asking for a dynamic port).

I did an experiment and was pleasantly surprised to find that the
OpenSSH client does work correctly when the real port number is
returned by the server. So, it won't break existing clients out there
if this is fixed. Both the "address that was connected" and the
"port
that was connected" can be the real address and port rather than the
values passed in the original SSH_MSG_GLOBAL_REQUEST and everything
works fine.

I have attached a patch which can be applied to OpenSSH 6.2p2 which
fixes this problem. I would appreciate it if you would consider
including it in a future release.

One minor ugliness here is that I had to use "get_sock_port()" instead
of get_local_port(), as get_local_port() doesn't take a socket as an
argument the way get_local_ipaddr() does and there's no equivalent
function under another name. Cleaning this up would impact other files,
though, and I wanted to keep the diff as clean as possible.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2147

Ron Frederick <ronf at timeheart.net> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Attachment #2330|0                           |1
        is obsolete|                            |

--- Comment #1 from Ron Frederick <ronf at timeheart.net> ---
Created attachment 2334
  --> https://bugzilla.mindrot.org/attachment.cgi?id=2334&action=edit
Updated patch for OpenSSH 6.3p1

Now that OpenSSH 6.3p1 is released, I've updated my patch for the
problem described here. Enclosed is a version that should apply cleanly
against the latest release.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2147

Darren Tucker <dtucker at zip.com.au> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |dtucker at zip.com.au
             Blocks|                            |2130

--- Comment #2 from Darren Tucker <dtucker at zip.com.au> ---
look at this for the next release

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2147

--- Comment #3 from Ron Frederick <ronf at timeheart.net> ---
I appreciate you considering this for the next release.

I've been doing some more testing on this, and I think it may have been
a mistake for me to replace the hostname in the channel open message
with the actual listening address. By doing that, it makes it more
difficult for the client to match up against the right listener.

That said, the reason I didn't discover this in my testing is that the
current openssh client code doesn't actually try to match on the
hostname at all when matching which remote listener it is -- it only
matches on port! So, if you try to set up multiple listeners at once
from a client on the same server-side port (by setting
GatewayPorts=clientspecified in the server config and binding to
different addresses), the client always matches on the first listener
with that port, as seen in the following code snippet:

Channel *
channel_connect_by_listen_address(u_short listen_port, char *ctype,
char *rname)
{           
        int i;

        for (i = 0; i < num_permitted_opens; i++) {
                if (permitted_opens[i].host_to_connect != NULL &&
                    port_match(permitted_opens[i].listen_port,
listen_port)) {
                        return connect_to(
                            permitted_opens[i].host_to_connect,
                            permitted_opens[i].port_to_connect, ctype,
rname);
                }
        }
        error("WARNING: Server requests forwarding for unknown
listen_port %d",
            listen_port);
        return NULL;
}

Note that it checks here that a host_to_connect is set, but it doesn't
even pass in the listen_address to match against when this is called
from client_request_forwarded_tcpip() in clientloop.c. This means you
can't actually have two different listeners on the server side with the
same port listening on different addresses and forwarding to different
destinations.

My suggestion is to keep the portion of my change which fills in the
proper listening port instead of 0, but leaves the reported listening
address as the hostname/address which was requested. That way, a client
which does proper matching of both the listening host & port would
always be able to do so (since it got the actual licensing port back in
the global request's response when dynamic ports were requested). To
make this work right with OpenSSH as a client, though, additional
changes would be needed on the client side to extend the listener
matching to include both the host & port.

If you'd like me to submit a new patch for the server change here, let
me know. The client change is something we might want to handle in a
separate bug.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2147

--- Comment #4 from Ron Frederick <ronf at timeheart.net> ---
Created attachment 2344
  --> https://bugzilla.mindrot.org/attachment.cgi?id=2344&action=edit
Simplified patch for 6.3p1 which only replaces port and not host

Here's the simpler patch I mentioned for the server side - it was a
straightforward change from my previous one.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2147

--- Comment #5 from Ron Frederick <ronf at timeheart.net> ---
I just tested the most recent patch here against the 6.4p1 release that
came out last night, and it still applies cleanly to that version.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2147

Ron Frederick <ronf at timeheart.net> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Attachment #2334|0                           |1
        is obsolete|                            |

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2147

Damien Miller <djm at mindrot.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
                 CC|                            |djm at mindrot.org
         Resolution|---                         |FIXED

--- Comment #6 from Damien Miller <djm at mindrot.org> ---
Great diagnosis - thanks. Patch applied and will be in OpenSSH 6.5.

I've opened bug 2189 for the multiple listen address case.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2147

Damien Miller <djm at mindrot.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|RESOLVED                    |CLOSED

--- Comment #7 from Damien Miller <djm at mindrot.org> ---
Set all RESOLVED bugs to CLOSED with release of OpenSSH 7.1

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2147

debacle at debian.org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |debacle at debian.org

--- Comment #8 from debacle at debian.org ---
It looks like this change leads to a problem. See Debian bug
https://bugs.debian.org/846379 from 2016-11-30:

I have an OpenSSH client 1:5.5p1-6+squeeze3 on Debian 6 (Squeeze)
building a reverse tunnel with dynamic port to a Jessie server:

$ /usr/bin/ssh -t -t -R *:0:localhost:443 user at server somecommand

If the server runs OpenSSH up to version 6.4p1-2 ("forwardports" from
snaphot.d.o), everything is fine. With any OpenSSH server from 6.5p1-6
to 7.3p1-3, the tunnel gets established, but no data can be
transmitted over it. On any activity, the client side shows:
WARNING: Server requests forwarding for unknown listen_port 54321
debug1: failure forwarded-tcpip
When the client specifies the port (= no dynamic port):

$ /usr/bin/ssh -t -t -R *:54321:localhost:443 user at server somecommand

everything works fine, independent of the OpenSSH server version.

Reverting part of the change above solves the problem:

--- openssh-7.3p1.orig/channels.c
+++ openssh-7.3p1/channels.c
@@ -1458,7 +1458,7 @@ port_open_helper(Channel *c, char *rtype
                 } else {
                         /* listen address, port */
                         packet_put_cstring(c->path);
-                        packet_put_int(local_port);
+                        packet_put_int(c->listening_port);
                 }
                 if (strcmp(rtype,  
"forwarded-streamlocal at openssh.com") == 0) {
                         /* reserved for future owner/mode info */

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2147

--- Comment #9 from Damien Miller <djm at mindrot.org> ---
Not sure why the two numbers should be different. In 7.6p1 they are
definitely the same (I just checked), so there might be another bug in
the older versions. 

I just used:

+error("XXXXXXXXXXXXXXX listen %d lport %d", c->listening_port,
local_port);

above the "/* listen address, port */" line to see what was going on.

In any case, 7.6p1 seems to be sending the correct thing on the wire.

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2147

--- Comment #10 from debacle at debian.org ---
Damian, many thanks for checking this!

Unfortunately, I can still confirm the error with both OpenSSH server
7.4p1 and 7.6p1. With 7.4p1 and 7.6p1 my patch still helps, but for
7.6p1 it must be changed a little bit to:

@@ -1643,7 +1643,7 @@
        } else {
                /* listen address, port */
                if ((r = sshpkt_put_cstring(ssh, c->path)) != 0 ||
-                   (r = sshpkt_put_u32(ssh, local_port)) != 0) {
+                   (r = sshpkt_put_u32(ssh, c->listening_port)) != 0)
{
                        fatal("%s: channel %i: reply %s", __func__,
                            c->self, ssh_err(r));
                }

Did you use OpenSSH client version 5.5p1 as described in my comment and
the Debian bug report? With newer clients the problem is not present,
so one really needs to use this old version. (Unfortunately, the client
in my setup cannot be upgraded.)

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.