openssh unix dev - Feb 2022 - Does a known security issue allow ssh login via system accounts?

Hi Alexander,

Much appreciate the quick response and advice. May I ask for just a bit
more? It would be useful to find out more precisely how the intrusion worked
to know where its method should be appropriately reported.
> There are a few more mechanisms that may allow password logins despite
> the measures you named: When using PAM authentication (which I would
> assume, since you are talking about Ubuntu) all the configuration in
> /etc/pam.d regarding ssh logins, so /etc/pam.d/ssh and
> /etc/pam.d/common* will be in effect. If e.g. some module there
> specifies a different password and shell for backup, e.g. from LDAP,
> NIS, ActiveDirectory or different passwd/shadow files, those might
> apply. You said below that pam.d is "standard", but installed
packages
> might modify configuration there, so I'd suggest having another look
> there.
Checking the files in /etc/pam.d/ and /lib/x86_64-linux-gnu/security/, they
exactly match the same sets of files on another Ubuntu system of the same
vintage with no public exposure, both in files there and each file's
particulars. There is no LDAP, NIS or AD use by the system. The string
"backup" is not in any file in /etc/pam.d/; the two places it
obviously
shows up in /lib/x86_64-linux-gnu/security/*.so files are in message
strings, not references to the userid.
> Also, depending on your configuration, /usr/sbin/nologin may not be
> sufficient to prevent port-forwarding without an open shell, thus
> enabling a spammer to forward some SMTP port for sending spam without
> authentication by "looking like" coming from an internal server. 
This
> is also hard to see and distinguish from an attacker having a shell in
> the logs.
Likely right about port forwarding. When I caught the intrusion in
action "backup" did have an open ssh connection; unsure if
"backup" had a
shell. Postfix was seeing the email origin as local.
> > Adding this to sshd_config was effective:
> > 
> >   DenyUsers backup
> > 
> > If that's still not the default for system-level users like
"backup", would
> > adding it be a reasonble feature request? Or is that on the distros to
> > define their default sshd_config settings?
> 
> I'd consider it good practice to selectively 'AllowGroup
ssh-login' or
> something to that effect for any internet-exposed system. But common
> distros are far more open than that.  I don't think OpenSSH is the
right
> place for a policy change, since as mentioned before, there is a lot
> going on in PAM that will affect what OpenSSH does here, without being
> part of OpenSSH.
Is there a valid use case where standard systems users (sync, backup, etc.)
should have ssh access? If not, blocking by default should be prudent. On
the other hand, if there are valid, standard uses for ssh for such users,
what are normal ways to enable them? I should see if any of them were used
here. 

Highjacking a server for spam is a pedestrian goal; this evidences something
the hands of the script kiddies which requires better defense. Neither OSSEC
(Wazuh) on the system, nor regular scans with Nessus and OpenVAS, gave
warning. It would be good to find a signature for it; a start would be
adding a watch in OSSEC for system-level userids showing up in auth.log, but
there's some additional trick enabling those to bypass standard security.

Thanks again,

Whit

On 2022-02-28T22:16, Whit Blauvelt <whit at transpect.com>
wrote:
> Hi Alexander,
>
> Much appreciate the quick response and advice. May I ask for just a bit
> more? It would be useful to find out more precisely how the intrusion
worked
> to know where its method should be appropriately reported.
>
> > There are a few more mechanisms that may allow password logins despite
> > the measures you named: When using PAM authentication (which I would
> > assume, since you are talking about Ubuntu) all the configuration in
> > /etc/pam.d regarding ssh logins, so /etc/pam.d/ssh and
> > /etc/pam.d/common* will be in effect. If e.g. some module there
> > specifies a different password and shell for backup, e.g. from LDAP,
> > NIS, ActiveDirectory or different passwd/shadow files, those might
> > apply. You said below that pam.d is "standard", but
installed packages
> > might modify configuration there, so I'd suggest having another
look
> > there.
>
> Checking the files in /etc/pam.d/ and /lib/x86_64-linux-gnu/security/, they
> exactly match the same sets of files on another Ubuntu system of the same
> vintage with no public exposure, both in files there and each file's
> particulars. There is no LDAP, NIS or AD use by the system. The string
> "backup" is not in any file in /etc/pam.d/; the two places it
obviously
> shows up in /lib/x86_64-linux-gnu/security/*.so files are in message
> strings, not references to the userid.
The string 'backup' wouldn't have to necessarily appear in the
config.
Also, the .so files should be irrelevant, those are binaries, not config.

To check, I'd look at: PAM-Config (/etc/pam.d/*). That that config is
the same as another machine doesn't mean that there isn't a problem in
there. It just might mean that an attacker didn't introduce the problem
(or the problem was introduced in all machines). Then I'd look at
/etc/nsswitch.conf to check for other passwd/group/shadow mechanisms
than 'file'. Then I'd look at the relevant configs for those
mechanisms.
Then I'd compare "getent passwd" to /etc/passwd, same for group
and
shadow. There might be sssd in use, check config and cache.  And check
/etc/shells, this should only contain proper shells, not anything like
/usr/bin/nologin, /bin/false, ...

The unfortunate problem is that Linux AAA has become a messy stack of
things that might cause problems. In the extreme case Authentication
might e.g. call into:
SSH -> /etc/pam.d/ssh -> /etc/pam.d/common* -> pam_unix.so ->
/etc/nsswitch.conf -> sssd (maybe cached) -> LDAP -> Kerberos

And there can be side-branches, e.g. SSH will also check for SSH-Keys,
PAM may try multiple modules one after another and can have
conditionals, same for nsswitch (but not as complex), same for sssd (but
even more complex).

In the sshd_config, there might also be Match-blocks or other
configuration that makes certain users be treated specially.

It is hard to say what your exact problem is withouth having seen all of
the above. There are just too many things that can go wrong and a lot of
"traditional nuances" in the configs one has to watch out for.
> > > Adding this to sshd_config was effective:
> > >
> > >   DenyUsers backup
> > >
> > > If that's still not the default for system-level users like
"backup", would
> > > adding it be a reasonble feature request? Or is that on the
distros to
> > > define their default sshd_config settings?
> >
> > I'd consider it good practice to selectively 'AllowGroup
ssh-login' or
> > something to that effect for any internet-exposed system. But common
> > distros are far more open than that.  I don't think OpenSSH is the
right
> > place for a policy change, since as mentioned before, there is a lot
> > going on in PAM that will affect what OpenSSH does here, without being
> > part of OpenSSH.
>
> Is there a valid use case where standard systems users (sync, backup, etc.)
> should have ssh access? If not, blocking by default should be prudent. On
> the other hand, if there are valid, standard uses for ssh for such users,
> what are normal ways to enable them? I should see if any of them were used
> here.
I don't see any common use cases for system users to be accessible via
SSH directly. Usually one uses /bin/false as a shell to disable login
for those users. /usr/bin/nologin does the same, but with a message.
This is what most distros do.  I've seen problems with a homegrown
version of /usr/bin/nologin that displayed a multipage message with
'press any key to continue' and timeouts, which caused the exact
portforwarding problem you might have.

Another approach would be AllowGroups in the sshd_config, or appropriate
PAM configuration (there are tons of possibilities here).


Ciao,

Alexander W?rstlein.

Hi Whit,

Did you solve the riddle already? When not, maybe use another
approach? My favorite is:

"HAL: I would recommend that we put the unit back in operation
and let it fail." (http://miscel.wikidot.com/2001-transcript)

Could you enable the old state with the spurious login being
possible again (maybe blocking external SSH access to the machine
for a moment), attach to the SSH root process by calling something
like

strace -o x.dump -s1024 -vvv -f -tt -p [pid]

and then try yourself to perform a password login as backup
to the machine?

Then hook SSH on another machine in the similar way and login
there too. Then compare the two dumps. If you you do not easily
see the difference try focusing only on the output of the PID
running the login process. Maybe also create to text files
with only the syscall names from both attempts and "diff" them
to see when things start getting weird.

If the syscall sequence is identical, the mmaps might be interesting
to compare. With same software version, the PROT_x flags and
also the file offsets should be identical. Differences indicate
backdoored libraries, sometimes even unmasking kernel root kits
hiding that changes with normal command line ls/stat - what might
be unlikely on a host just used for spamming. For that kind of
analysis grep "mmap" and sed works best (get rid of ASLR entropy
with sed).

Kind regards,
hd