openssh bugs - Sep 2016 - [Bug 2615] New: LoginGraceTime bypass (DoS)

https://bugzilla.mindrot.org/show_bug.cgi?id=2615

            Bug ID: 2615
           Summary: LoginGraceTime bypass (DoS)
           Product: Portable OpenSSH
           Version: 7.3p1
          Hardware: Sparc
                OS: Solaris
            Status: NEW
          Severity: normal
          Priority: P5
         Component: sshd
          Assignee: unassigned-bugs at mindrot.org
          Reporter: tomas.kuthan at oracle.com

On one of our internal systems we have encountered an issue, which we
believe could be exploited to mount a deny-of-service attack on sshd.

It was reported, that ssh service on that host was occasionally
refusing new connections:

$ ssh triassic date
ssh_exchange_identification: Connection closed by remote host

It turned out, that there were multiple processes hanging in the kernel
trying to access user's authorized_keys file on an NFS home directory.

The reason of the NFS hang is not that important (user's home dir has
been moved to another server, but the directory was never unmounted, so
NFS client was still trying to access the old server, where nfs/server
service has already been disabled.)

The monitor processes are blocked in open(), called from
auth_openfile(), called from user_key_allowed():

  core 'core.sshd.699975' of 699975:      /usr/lib/ssh/sshd -R 
     00007ff5c3658fbe __systemcall6 () + 1e 
     00007ff5c3622d4a __open () + 1a 
     00007ff5c363dbee open () + 12e 
     000000000045a20d auth_openfile () + 3d 
     0000000000465ccc user_key_allowed () + 3fc 
     000000000046999b mm_answer_keyallowed () + 45b 
     000000000046bf08 monitor_read () + 118 
     000000000046c2f8 monitor_child_preauth () + 308 
     000000000044cba0 main () + 1eb0 
     00000000004492d3 _start () + 43

NFS blocks most signals for the duration of the over-the-wire call,
including SIGALRM. The alarm implementing login_grace_time was queued,
but never delivered to the process. As a result, sshd process stayed
unauthenticated much longer than LoginGraceTime seconds. The user tried
ssh-ing in multiple times, eventually wasting up soft limit of
MaxStartups connections. After that, sshd started probabilistically
dropping connections of other users.

In this case this has happened by an accident.
But an attacker, who controls their NFS home directory, could use this
to mount a DoS attacke on sshd. All they needed to do is stop
nfs/service on their home dir server and try ssh-in using public key
auth in a loop. Eventually it would hoard hard limit of MaxStartup
connections and all successive ssh attempts would be dropped.

To reproduce:

----------------------------------
NFS server (servera)
----
zfs create rpool/nora
zfs set share.nfs=on rpool/nora
useradd -u 6378 -d /rpool/nora nora
chown -R nora /rpool/nora
# disable NFS to create hang:
svcadm disable nfs/server

SSH server (serverb)
----
useradd -u 6378 -d /home/nora nora
echo 'nora servera:/rpool/nora' >>/etc/auto_home

SSH client (can be the same host as NFS server)
----
# create key for pubkey auth:
ssh-keygen -t rsa
ssh nora at serverb

Result
----
ssh command hangs indefinitely.
There will be an sshd process on SSH server hanging in NFS.
----------------------------------

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2615

--- Comment #1 from Tomas Kuthan <tomas.kuthan at oracle.com> ---
Created attachment 2875
  --> https://bugzilla.mindrot.org/attachment.cgi?id=2875&action=edit
watchdog process backing-up login_grace_time alarm

I have implemented and successfully tested a candidate fix - a single
purpose watchdog process backing up login_grace_time alarm in the main
process. If the main process doesn't authenticate or exit in
login_grace_time seconds, the watchdog kills it by SIGTERM (or
eventually SIGKILL). Patch attached.

I have rejected several other fix ideas:
- threads - unlikely to be accepted upstream
- main sshd process keeping track of unauthenticated children
    - too much logic in process listening for new connection
- allow preauth child to send signal to the monitor
    - too much privs to unprivileged process
    - wouldn't work w/o privilege separation

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2615

Darren Tucker <dtucker at zip.com.au> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |dtucker at zip.com.au

--- Comment #2 from Darren Tucker <dtucker at zip.com.au> ---
The down side is that the extra process makes it a bit easier to DoS it
by pid exhaustion, although admittedly LoginGraceTime is much more
likely to be the limiting factor.  I kinda hope there's another way to
do this but I can't think of one offhand :-(

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2615

Damien Miller <djm at mindrot.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|---                         |WONTFIX
                 CC|                            |djm at mindrot.org

--- Comment #3 from Damien Miller <djm at mindrot.org> ---
I don't think this is sshd's responsibility to solve. If you have home
directories on NFS then you should match your NFS and LoginGraceTime
timeouts to ensure that sshd has a chance to proceed.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.