openssh bugs - Aug 2024 - [Bug 3723] New: sshd failed to close session when client specifies no remote command

https://bugzilla.mindrot.org/show_bug.cgi?id=3723

            Bug ID: 3723
           Summary: sshd failed to close session when client specifies no
                    remote command
           Product: Portable OpenSSH
           Version: 8.0p1
          Hardware: amd64
                OS: Linux
            Status: NEW
          Severity: minor
          Priority: P5
         Component: PAM support
          Assignee: unassigned-bugs at mindrot.org
          Reporter: szhang at gen-info.osaka-u.ac.jp

We limit the number of sessions a user can close using pam_limits.so by
setting maxlogins in /etc/security/limits.conf

This works well when user tries to start a normal ssh session or sftp
session. They would be told about Too many logins, or Received message
too long and the session with the client would close.

However if the user specified -N, things go differently. No error
message would be shown even though the user failed pam_limits.so on
session creation. The connection persists, although port forwarding
would fail for being administratively prohibited.

To make matters worse, systemd would consider such failed session "a
session", and create a scope for it. It would show up on the loginctl
output, but not on the "w".

systemd has a limit on all sessions in the system it could handle, and
such "failed session" counted against such limit. Thus, a user with
ssh
access could potentially create 8192 sessions through ssh -T to such
machine, disregarding any limitation set on pam_limits regarding the
maxlogins, flooding the systemd, and denying any other legit user from
creating a systemd-managed session to launch gnome and other software.

I wonder if such behavior is by design, or if there is anything I could
do to make sshd kill such session if it failed pam_limits.so on session
creation, even if no subprocess would be launched.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3723

Damien Miller <djm at mindrot.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |djm at mindrot.org

--- Comment #1 from Damien Miller <djm at mindrot.org> ---
What operating system are you using? Did you get your openssh packages
from your OS vendor? I ask because OpenSSH doesn't have any integration
with systemd that would support tracking of connections without
shell/login sessions. It's possible your OS vendor has added this, but
done it in a manner that causes the problem you mention.

sshd doesn't set any limit on concurrent active connections nor does it
terminate connections without shell/login sessions automatically by
default.

OpenSSH 9.2 added an UnusedConnectionTimeout option that you might find
useful here. It allows connections without shell/login sessions to be
automatically terminated for inactivity after a configurable interval.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3723

--- Comment #2 from S Zhang <szhang at gen-info.osaka-u.ac.jp> ---
I was on RHEL 8.10 and Oracle 8.10. I am currently testing it on other
distros.

I believe that the systemd session was created via PAM, but somehow
even if the sshd failed to create the remote session, as no remote
executable was specified by -N, the client side did not know that the
session failed, thus the connection remains. Maybe systemd sees that
the connection survived and did not close their session/scope as well.

Are there any conditions for the server to initiate close connection,
and which side initiated the connection close sequence when the server
side failed to create a session due to pam_limits.so?

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3723

--- Comment #3 from S Zhang <szhang at gen-info.osaka-u.ac.jp> ---
I contacted systemd side and they said that they created a session for
such pam_limits.so rejected ssh session because the sshd_session
process spawned by the sshd process survived.

Maybe I need to take a look at how sshd_session handle the situation
where the PAM session failed? It appears that as long as PAM auth
passed, everything was left to the client side to decide to keep or
kill the sshd_session.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3723

--- Comment #4 from S Zhang <szhang at gen-info.osaka-u.ac.jp> ---
I took a look at the openssh-portable part that handles the session.

It appears that when the PAM session failed, we still create a
sshd_session, but everything in sshauthopt unset meaning nothing
including port forwarding can be done.

https://github.com/openssh/openssh-portable/blob/10ccf611ab8ecba9ce6b0548c5ccd8c1220baf92/auth-pam.c#L1220-L1225

https://github.com/openssh/openssh-portable/blob/10ccf611ab8ecba9ce6b0548c5ccd8c1220baf92/auth.c#L763-L780

However the sshd_session that failed PAM for session keeps running with
restricted sshauthopt. Systemd saw the running sshd_session owned by
the user and created a session under such user for it. While the
sshd_session and the ssh client cannot do anything, the sshd_session
wait for the user to close the connection to stop the local process.
Normally the ssh client attempted to run something and got rejected to
initiate the connection close process, but when the ssh client was
running with -N set, such thing did not happen, and the sshd_session
process persists with the systemd-logind session.

I wonder why would such restricted session remain, and why would the
sshd side not initiate the close session part but wait for the client
side to close the session? Are there any use case where such restricted
session useful?

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3723

--- Comment #5 from Damien Miller <djm at mindrot.org> ---
Ah, pam_limits.so is session module? That's very late in the stack, and
session modules AFAIK are mostly intended for configuring the session
with things like environment variables and aren't really supposed to
block access.

In particular, a failure of a session module (e.g. pam_limits) is not
specified to be fatal to the session. This is the reason that sshd
doesn't terminate the session when the PAM session initialisation
fails.

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3723

S Zhang <szhang at gen-info.osaka-u.ac.jp> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|---                         |INVALID

--- Comment #6 from S Zhang <szhang at gen-info.osaka-u.ac.jp> ---
A quick recap on what the three components I suspected were up to:

linux-pam side provided the pam_limit.so that offers a session module
only to limit how many concurrent login a user can have. They based
their session count on the utmp version of session, and correctly
failed the session on the pam stack.

openssh side considered that the login succeeded the auth stack, and
accordingly spawned a sshd_session process under the user. It respected
the failed session and restricted such session to not able to do
anything including port forwarding.

systemd side created a session as there is a process of sshd_session,
even though that it failed the PAM session part. That at least
guaranteed the process being put under the user's cgroup before it got
overwhelmed.

It seems that everyone in openssh, systemd, and linux-pam is doing the
right thing. It is just unfortunate that this combination had a
discrepancy on how utmp count differs than systemd-logind count, the
pam_limits.so failed session is not considered a failed auth, and I
lacked the functionality of stopping a user from creating too many
systemd-logind session that resulted in a DoS in my server.

I would close this issue right now. Now I think I had a better
understanding on what happened to all this session mess that I caught
into.

Thanks for pointing out that UnusedConnectionTimeout option. It would
correctly close such useless but troubling session in my test server,
though I probably need to wait a few more years before the production
server currently running RHEL 8/9 can benefit from it.

Maybe I should also consider modifying the pam_limits.so to provide an
auth module to disallow logins exceeding the session limit.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.