openssh unix dev - Mar 2022 - Does a known security issue allow ssh login via system accounts?

On 2022-02-28T22:16, Whit Blauvelt <whit at transpect.com>
wrote:
> Hi Alexander,
>
> Much appreciate the quick response and advice. May I ask for just a bit
> more? It would be useful to find out more precisely how the intrusion
worked
> to know where its method should be appropriately reported.
>
> > There are a few more mechanisms that may allow password logins despite
> > the measures you named: When using PAM authentication (which I would
> > assume, since you are talking about Ubuntu) all the configuration in
> > /etc/pam.d regarding ssh logins, so /etc/pam.d/ssh and
> > /etc/pam.d/common* will be in effect. If e.g. some module there
> > specifies a different password and shell for backup, e.g. from LDAP,
> > NIS, ActiveDirectory or different passwd/shadow files, those might
> > apply. You said below that pam.d is "standard", but
installed packages
> > might modify configuration there, so I'd suggest having another
look
> > there.
>
> Checking the files in /etc/pam.d/ and /lib/x86_64-linux-gnu/security/, they
> exactly match the same sets of files on another Ubuntu system of the same
> vintage with no public exposure, both in files there and each file's
> particulars. There is no LDAP, NIS or AD use by the system. The string
> "backup" is not in any file in /etc/pam.d/; the two places it
obviously
> shows up in /lib/x86_64-linux-gnu/security/*.so files are in message
> strings, not references to the userid.
The string 'backup' wouldn't have to necessarily appear in the
config.
Also, the .so files should be irrelevant, those are binaries, not config.

To check, I'd look at: PAM-Config (/etc/pam.d/*). That that config is
the same as another machine doesn't mean that there isn't a problem in
there. It just might mean that an attacker didn't introduce the problem
(or the problem was introduced in all machines). Then I'd look at
/etc/nsswitch.conf to check for other passwd/group/shadow mechanisms
than 'file'. Then I'd look at the relevant configs for those
mechanisms.
Then I'd compare "getent passwd" to /etc/passwd, same for group
and
shadow. There might be sssd in use, check config and cache.  And check
/etc/shells, this should only contain proper shells, not anything like
/usr/bin/nologin, /bin/false, ...

The unfortunate problem is that Linux AAA has become a messy stack of
things that might cause problems. In the extreme case Authentication
might e.g. call into:
SSH -> /etc/pam.d/ssh -> /etc/pam.d/common* -> pam_unix.so ->
/etc/nsswitch.conf -> sssd (maybe cached) -> LDAP -> Kerberos

And there can be side-branches, e.g. SSH will also check for SSH-Keys,
PAM may try multiple modules one after another and can have
conditionals, same for nsswitch (but not as complex), same for sssd (but
even more complex).

In the sshd_config, there might also be Match-blocks or other
configuration that makes certain users be treated specially.

It is hard to say what your exact problem is withouth having seen all of
the above. There are just too many things that can go wrong and a lot of
"traditional nuances" in the configs one has to watch out for.
> > > Adding this to sshd_config was effective:
> > >
> > >   DenyUsers backup
> > >
> > > If that's still not the default for system-level users like
"backup", would
> > > adding it be a reasonble feature request? Or is that on the
distros to
> > > define their default sshd_config settings?
> >
> > I'd consider it good practice to selectively 'AllowGroup
ssh-login' or
> > something to that effect for any internet-exposed system. But common
> > distros are far more open than that.  I don't think OpenSSH is the
right
> > place for a policy change, since as mentioned before, there is a lot
> > going on in PAM that will affect what OpenSSH does here, without being
> > part of OpenSSH.
>
> Is there a valid use case where standard systems users (sync, backup, etc.)
> should have ssh access? If not, blocking by default should be prudent. On
> the other hand, if there are valid, standard uses for ssh for such users,
> what are normal ways to enable them? I should see if any of them were used
> here.
I don't see any common use cases for system users to be accessible via
SSH directly. Usually one uses /bin/false as a shell to disable login
for those users. /usr/bin/nologin does the same, but with a message.
This is what most distros do.  I've seen problems with a homegrown
version of /usr/bin/nologin that displayed a multipage message with
'press any key to continue' and timeouts, which caused the exact
portforwarding problem you might have.

Another approach would be AllowGroups in the sshd_config, or appropriate
PAM configuration (there are tons of possibilities here).


Ciao,

Alexander W?rstlein.

On Mon, 02/28/22, 2022 at 11:17:12PM +0100, Alexander Wuerstlein wrote:
> To check, I'd look at: PAM-Config (/etc/pam.d/*). That that config is
> the same as another machine doesn't mean that there isn't a problem
in
> there. 
Right. Just saying if the problem's there, it's endemic to Ubuntu 16, as
those files don't differ from the same on other Ubuntu 16 systems
> It just might mean that an attacker didn't introduce the problem
> (or the problem was introduced in all machines). Then I'd look at
> /etc/nsswitch.conf to check for other passwd/group/shadow mechanisms
> than 'file'. 
The nsswitch.conf is consistent even with Ubuntu 20 systems, except that the
latter have added systemd on the passwd and group lines. 
> Then I'd look at the relevant configs for those mechanisms.
All I've found are standard. 
> Then I'd compare "getent passwd" to /etc/passwd, same for
group and
> shadow. 
They each produce dumps identical to the files. However the VM has been
rebooted since the compromise, so this doesn't rule out prior subverion
there.
> There might be sssd in use, check config and cache.  
I'm not familiar with sssd. Certainly never configured it. There's no
"sssd"
process running.
> And check etc/shells, this should only contain proper shells, not anything
> /like usr/bin/nologin, /bin/false, ...
Just the standard list there. 
> ... In the sshd_config, there might also be Match-blocks or other
> configuration that makes certain users be treated specially.
The match-blocks in sshd_config are only of the restrictive kind for certain
groups, not such as to add any extra permissions.
> It is hard to say what your exact problem is withouth having seen all of
> the above. There are just too many things that can go wrong and a lot of
> "traditional nuances" in the configs one has to watch out for.
> > Is there a valid use case where standard systems users (sync, backup,
etc.)
> > should have ssh access? If not, blocking by default should be prudent.
On
> > the other hand, if there are valid, standard uses for ssh for such
users,
> > what are normal ways to enable them? I should see if any of them were
used
> > here.
> 
> I don't see any common use cases for system users to be accessible via
> SSH directly. Usually one uses /bin/false as a shell to disable login
> for those users. /usr/bin/nologin does the same, but with a message.
> This is what most distros do.  I've seen problems with a homegrown
> version of /usr/bin/nologin that displayed a multipage message with
> 'press any key to continue' and timeouts, which caused the exact
> portforwarding problem you might have.
It's the standard Ubuntu 16 version of that file.
> Another approach would be AllowGroups in the sshd_config, or appropriate
> PAM configuration (there are tons of possibilities here).
What _is_ different here -- the only thing distinguishing this system from
stock Ubuntu 16.04.7 -- is the custom compile of a later (but now far from
latest) version of OpenSSH. The vulnerabilty looks to be either in Ubuntu
16.04.7, now beyond EOL, or OpenSSH_8.6p1 using OpenSSL 1.0.2g, also now
antiquated -- or their combination. Adding a DenyUsers line in sshd_config
listing all the system user accounts works to block this intrusion, and will
be my standard practice now. I found some discussion from about 8 years ago
claiming older versions of OpenSSH checked for the "nologin" shell (at
least
in some configurations) and themselves blocked accordingly, which if true
would be equivalent to what adding that DenyUsers now can do.

Much appreciate all your help. Most generous! I only wish I'd have found
some additional clue by now.

Best,
Whit