openssh unix dev - Jan 2017 - OpenSSH Hangs After Successful Authentication

Greetings...

** Short Description / Abstract **

Using my _internal_ WiFi card, OpenSSH succeeds to local (internal) LAN
hosts, but hangs after authentication to external LAN hosts; however
PuTTY works for all hosts.

Using an _external_ WiFi card, OpenSSH does succeed to all LAN hosts (as
did PuTTY, still, as well).

** Longer Description and Details **

*** HW Description ***

I have a laptop with Debian 8 (Jessie) installed from the 8.6 XFCE .iso.

Native WiFi Device is:
  - Broadcom BCM43602 (PCI ID 14e4:43ba) using brcmfmac43602-pcie.bin
    drivers (see https://wiki.debian.org/brcmfmac)

  - To get the Native WiFi device to work, I needed to install a
    backported kernel (from "http.debian.net/debian"), specifically,
    /uname -a/ returns:

       Linux myhost 4.8.0-0.bpo.2-amd64 #1 SMP Debian 4.8.11-1-bpo8+1 \
         (2016-12-14) x86_64 GNU/Linux

External WiFi Device is:
  - Penguin Wireless N USB Adapter (TPE-N150USB; Atheros AR9271) using
    Atheros drivers (see https://wiki.debian.org/ath9k)

All packages (except for Wifi drivers) were installed from repos
(including PuTTY)...I've not compiled anything.

*** Problem Description ***

While using the Broadcom WiFi, PuTTY connects fine to any host.
However, OpenSSH only succeeds connection for hosts on the local LAN
(same subnet, 192.168.1.x).  "Success" meaning I ultimately get a
shell
on the remote system.  (Note: no firewall is currently enabled; all
tables are currently default ACCEPT)

When I disable the Broadcom device (remove firmware driver files and
reboot) and use the TPE-N150SUSB Device, both OpenSSH and PuTTY work fine.

When hanging, OpenSSH hangs after successful authentication.  Here is a
snippet where OpenSSH hangs after issuing "ssh -vvv uname at host.com"

```
{cut for brevity...}
debug3: sign_and_send_pubkey: XXXXXX \
   xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx
debug1: Authentication succeeded (publickey).
Authenticated to somehost.com ([nnn.nnn.nnn.nnn]:22).
debug1: channel 0: new [client-session]
debug3: ssh_session2_open: channel_new: 0
debug2: channel 0: send open
debug1: Requesting no-more-sessions at openssh.com
debug1: Entering interactive session.
debug2: callback start
debug2: fd 3 setting TCP_NODELAY
debug3: packet_set_tos: set IP_TOS 0x10
debug2: client_session2_setup: id 0
debug2: channel 0: request pty-req confirm 1
debug1: Sending environment.
debug3: Ignored env XDG_VTNR
debug3: Ignored env SSH_AGENT_PID
debug3: Ignored env XDG_SESSION_ID
debug3: Ignored env XDG_GREETER_DATA_DIR
debug3: Ignored env GPG_AGENT_INFO
debug3: Ignored env GLADE_PIXMAP_PATH
debug3: Ignored env TERM
debug3: Ignored env SHELL
debug3: Ignored env XDG_MENU_PREFIX
debug3: Ignored env WINDOWID
debug3: Ignored env USER
debug3: Ignored env LS_COLORS
debug3: Ignored env XDG_SESSION_PATH
debug3: Ignored env GLADE_MODULE_PATH
debug3: Ignored env XDG_SEAT_PATH
debug3: Ignored env SSH_AUTH_SOCK
debug3: Ignored env SESSION_MANAGER
debug3: Ignored env XDG_CONFIG_DIRS
debug3: Ignored env PATH
debug3: Ignored env DESKTOP_SESSION
debug3: Ignored env PWD
debug3: Ignored env EDITOR
debug1: Sending env LANG = en_US.utf8
debug2: channel 0: request env confirm 0
debug3: Ignored env GDMSESSION
debug3: Ignored env SHLVL
debug3: Ignored env XDG_SEAT
debug3: Ignored env HOME
debug3: Ignored env LOGNAME
debug3: Ignored env XDG_DATA_DIRS
debug3: Ignored env DBUS_SESSION_BUS_ADDRESS
debug3: Ignored env XDG_RUNTIME_DIR
debug3: Ignored env DISPLAY
debug3: Ignored env GLADE_CATALOG_PATH
debug3: Ignored env XDG_CURRENT_DESKTOP
debug3: Ignored env COLORTERM
debug3: Ignored env XAUTHORITY
debug3: Ignored env _
debug2: channel 0: request shell confirm 1
debug2: callback done
debug2: channel 0: open confirm rwindow 0 rmax 32768
```

...and then the system hangs.  I can *not* C-c out or use OpenSSH's
"<ENTER>~." sequence to kill the session.  The entire terminal
is
unresponsive until 15-16 minutes later when I get "write error: broken
pipe" and the terminal comes back.  During that time, both client and
server /netstat -ant/ show the connection as "established".

Also, side note, no apparent issues occur with web surfing.  Email is
not setup so I've not tested it.

Based on research already done and some various testing, I can not
conclusively determine if the problem is with the Broadcom drivers with
PuTTY somehow compensating for an issue, or the problem is with OpenSSH
(perhaps OpenSSH just not "compensating" for a driver issue).  I'm
leaning toward the drivers since my other systems w/Debian 8 all work
just fine; they all have the same versions of "openssh-client", but
only
the problematic laptop has a backported kernel.

I'm sending this inquiry to both the OpenSSH mailing list and the
Linux-Wireless list (in separate emails) in hopes I can sort out which
side is having the issue.  Any ideas on how to further test/investigate
or, hopefully rectify, this matter would be greatly appreciated!

Thank you!

David A. Gershman
gershman at dagertech.net

On Mon, Jan 9, 2017 at 8:54 AM, David A. Gershman
<dagershman at dagertech.net> wrote:
> Using my _internal_ WiFi card, OpenSSH succeeds to local (internal) LAN
> hosts, but hangs after authentication to external LAN hosts; however
> PuTTY works for all hosts.
Two possibilities I can think of:

1) MTU black hole.  Check the "send-q" column on both client and
server in netstat when it's in the hung state, compare MTUs between
working and not working interfaces and try "ifconfig wlan0 mtu 576"
before starting the ssh connection.

2) ssh(1) sets the IP type of service bits around the time of your
observed hang.  In the past we have had reports of stateful devices
not coping with the QoS of an established connection changing.  Try
"ssh -o 'IPQos lowdelay lowdelay' yourserver".

My bet is on #1, and my guess is the default MTU is different between
your interfaces.

-- 
Darren Tucker (dtucker at zip.com.au)
GPG key 11EAA6FA / A86E 3E07 5B19 5880 E860  37F4 9357 ECEF 11EA A6FA (new)
    Good judgement comes with experience. Unfortunately, the experience
usually comes from bad judgement.

On 01/08/2017 02:20 PM, Darren Tucker wrote:
> On Mon, Jan 9, 2017 at 8:54 AM, David A. Gershman
> <dagershman at dagertech.net> wrote:
>> Using my _internal_ WiFi card, OpenSSH succeeds to local (internal) LAN
>> hosts, but hangs after authentication to external LAN hosts; however
>> PuTTY works for all hosts.
> 
> Two possibilities I can think of:
> 
> 1) MTU black hole.  Check the "send-q" column on both client and
> server in netstat when it's in the hung state, compare MTUs between
> working and not working interfaces and try "ifconfig wlan0 mtu
576"
> before starting the ssh connection.
I've tried setting the MTU as stated (found that hint on another email
online)...no luck. With MTU @ either setting, the Send-Q column on the
server indicated 40 while the client is at 424.
> 
> 2) ssh(1) sets the IP type of service bits around the time of your
> observed hang.  In the past we have had reports of stateful devices
> not coping with the QoS of an established connection changing.  Try
> "ssh -o 'IPQos lowdelay lowdelay' yourserver".
Also no luck.  After a check on whether 'lowdelay' or a value was needed
(wasn't sure, so I checked), neither:

     ssh -o IPQoS="lowdelay lowdelay" myserver
or
     ssh -o "IPQoS lowdelay lowdelay" myserver

worked.
> 
> My bet is on #1, and my guess is the default MTU is different between
> your interfaces.
The default MTU on both my native Wifi and TPE device are both 1500 as
is the server and my other systems.  Since my other systems don't have
any problem, I'm presuming the access point isn't the issue either
(w.r.t. MTU).  So far, MTU doesn't seem to have an effect.

--dag

So it may help others, I found a forum post 
<http://askubuntu.com/questions/344863/ssh-new-connection-begins-to-hang-not-reject-or-terminate-after-a-day-or-so-on>
with the following note:

    /A problem can arise when you are trying to connect from behind a
    NAT router using OpenSSH. During session setup, after the password
    has been given, OpenSSH sets the TOS (type of service) field in the
    IP datagram. Some routers are known to choke on this. The effect is
    that your session hangs indefinitely after you gave your password./

As such, the post suggests using NetCat as a proxy:

    ssh -o "ProxyCommand nc %h %p" {user-name}@host.com


which worked for me.

I don't see this as a fix as, again, PuTTY and OpenSSH on the MacOS side 
of my laptop work fine as does OpenSSH with my TP-LINK USB device.

Good Luck!


On 01/08/2017 02:20 PM, Darren Tucker wrote:
> On Mon, Jan 9, 2017 at 8:54 AM, David A. Gershman
> <dagershman at dagertech.net> wrote:
>> Using my _internal_ WiFi card, OpenSSH succeeds to local (internal) LAN
>> hosts, but hangs after authentication to external LAN hosts; however
>> PuTTY works for all hosts.
> Two possibilities I can think of:
>
> 1) MTU black hole.  Check the "send-q" column on both client and
> server in netstat when it's in the hung state, compare MTUs between
> working and not working interfaces and try "ifconfig wlan0 mtu
576"
> before starting the ssh connection.
>
> 2) ssh(1) sets the IP type of service bits around the time of your
> observed hang.  In the past we have had reports of stateful devices
> not coping with the QoS of an established connection changing.  Try
> "ssh -o 'IPQos lowdelay lowdelay' yourserver".
>
> My bet is on #1, and my guess is the default MTU is different between
> your interfaces.
>