openssh unix dev - Nov 2008 - ssh weirdness - hanging sessions intermittently with no connectivity after for an hour or so...

I just finished rebuilding my server after a penetration last week
which left my filesystem in shambles.  I've gotten most everything
running again better than before with one exception.

sshd doesn't work as well as it used to and I'm not sure why, I'm
including a sanitized log snippet, hopefully someone can point out my
stupidity for me...

If I open a putty session from another network it's 50/50 whether or
not I even get a response, and if I do, then it usually hangs after I
enter my password and then times out - OR - it will connect, negotiate
a session, and then die after about 20 minutes - in any case once dead
it takes at least 20 - 30 minutes before it will allow me to connect
again...  Any ideas?


Nov 10 13:32:53 localhost sshd[21009]: debug1: Forked child 26900.
Nov 10 13:32:53 localhost sshd[26900]: debug1: rexec start in 4 out 4
newsock 4 pipe 6 sock 7
Nov 10 13:32:53 localhost sshd[26900]: debug1: inetd sockets after dupping: 3, 3
Nov 10 13:32:53 localhost sshd[26900]: Connection from xxx.xxx.xxx.xxx port 1554
Nov 10 13:32:53 localhost sshd[26900]: debug1: Client protocol version
2.0; client software version PuTTY_Release_0.60
Nov 10 13:32:53 localhost sshd[26900]: debug1: no match: PuTTY_Release_0.60
Nov 10 13:32:53 localhost sshd[26900]: debug1: Enabling compatibility
mode for protocol 2.0
Nov 10 13:32:53 localhost sshd[26900]: debug1: Local version string
SSH-2.0-OpenSSH_4.3p2 Debian-9etch3
Nov 10 13:32:54 localhost sshd[26900]: debug1: PAM: initializing for
"user"
Nov 10 13:32:54 localhost sshd[26900]: reverse mapping checking
getaddrinfo for dhcpxxx-xx-xxx-xxx.xxxx.xx.xx.xxxx.xxxxxxx.xxx failed
- POSSIBLE BREAK-IN ATTEMPT!
Nov 10 13:32:54 localhost sshd[26900]: debug1: PAM: setting PAM_RHOST
to "xxx.xx.xxx.xxx"
Nov 10 13:32:54 localhost sshd[26900]: debug1: PAM: setting PAM_TTY to
"ssh"
Nov 10 13:32:54 localhost sshd[26900]: Failed none for tony from
xxx.xx.xxx.xxx port 1554 ssh2
Nov 10 13:32:57 localhost sshd[26900]: debug1: PAM: password
authentication accepted for user
Nov 10 13:32:57 localhost sshd[26900]: debug1: do_pam_account: called
Nov 10 13:32:57 localhost sshd[26900]: Accepted password for user from
xxx.xx.xxx.xxx port 1554 ssh2
Nov 10 13:32:57 localhost sshd[26900]: debug1: monitor_child_preauth:
user has been authenticated by privileged process
Nov 10 13:32:57 localhost sshd[26902]: (pam_unix) session opened for
user user by (uid=0)
Nov 10 13:32:57 localhost sshd[26902]: debug1: PAM: reinitializing credentials
Nov 10 13:32:57 localhost sshd[26902]: debug1: permanently_set_uid: 1000/1000
Nov 10 13:32:57 localhost sshd[26902]: debug1: Entering interactive
session for SSH2.
Nov 10 13:32:57 localhost sshd[26902]: debug1: server_init_dispatch_20
Nov 10 13:32:57 localhost sshd[26902]: debug1:
server_input_channel_open: ctype session rchan 256 win 16384 max 16384
Nov 10 13:32:57 localhost sshd[26902]: debug1: input_session_request
Nov 10 13:32:57 localhost sshd[26902]: debug1: channel 0: new [server-session]
Nov 10 13:32:57 localhost sshd[26902]: debug1: session_new: init
Nov 10 13:32:57 localhost sshd[26902]: debug1: session_new: session 0
Nov 10 13:32:57 localhost sshd[26902]: debug1: session_open: channel 0
Nov 10 13:32:57 localhost sshd[26902]: debug1: session_open: session
0: link with channel 0
Nov 10 13:32:57 localhost sshd[26902]: debug1:
server_input_channel_open: confirm session
Nov 10 13:32:57 localhost sshd[26902]: debug1:
server_input_channel_req: channel 0 request pty-req reply 1
Nov 10 13:32:57 localhost sshd[26902]: debug1: session_by_channel:
session 0 channel 0
Nov 10 13:32:57 localhost sshd[26902]: debug1:
session_input_channel_req: session 0 req pty-req
Nov 10 13:32:57 localhost sshd[26902]: debug1: Allocating pty.
Nov 10 13:32:57 localhost sshd[26900]: debug1: session_new: init
Nov 10 13:32:57 localhost sshd[26900]: debug1: session_new: session 0
Nov 10 13:32:57 localhost sshd[26902]: debug1: session_pty_req:
session 0 alloc /dev/pts/2
Nov 10 13:32:57 localhost sshd[26902]: debug1:
server_input_channel_req: channel 0 request shell reply 1
Nov 10 13:32:57 localhost sshd[26902]: debug1: session_by_channel:
session 0 channel 0
Nov 10 13:32:57 localhost sshd[26902]: debug1:
session_input_channel_req: session 0 req shell
Nov 10 13:32:57 localhost sshd[26902]: debug1: PAM: setting PAM_TTY to
"/dev/pts/2"
Nov 10 13:32:57 localhost sshd[26903]: debug1: Setting controlling tty
using TIOCSCTTY.
Nov 10 13:33:13 localhost sshd[26874]: debug1: Got 100/81 for keepalive
Nov 10 13:33:13 localhost sshd[26851]: Disconnecting: Timeout, your
session not responding.
Nov 10 13:33:13 localhost sshd[26851]: debug1: do_cleanup
Nov 10 13:33:13 localhost sshd[26851]: debug1: PAM: cleanup
Nov 10 13:33:13 localhost sshd[26851]: (pam_unix) session closed for user user
Nov 10 13:33:13 localhost sshd[26848]: debug1: do_cleanup
Nov 10 13:33:13 localhost sshd[26848]: debug1: PAM: cleanup
Nov 10 13:33:13 localhost sshd[26848]: debug1: session_pty_cleanup:
session 0 release /dev/pts/0

On Monday 2008-11-10 20:53, Tony Alfaro wrote:
>sshd doesn't work as well as it used to and I'm not sure why,
I'm
>including a sanitized log snippet, hopefully someone can point out my
>stupidity for me...
>
>If I open a putty session from another network it's 50/50 whether or
>not I even get a response, and if I do, then it usually hangs after I
>enter my password and then times out - OR - it will connect, negotiate
>a session, and then die after about 20 minutes - in any case once dead
>it takes at least 20 - 30 minutes before it will allow me to connect
>again...  Any ideas?
It might be that there is a shitty router along the path that chokes
on TCP packets with SACK/DSACK. Been there - any packet bursts
(connection setup, or just loads of output from `ls -Rl`, or a bulk
transfer with rsync/scp/sftp) hung it.
Suggestion to try to disable SACK/DSACK.
In case your sshd is on Linux, you can use iptables's TCPOPTSTRIP to
get rid of the SACK pieces on the TCP SYN for SSH connections.