openssh unix dev - Feb 2022 - Does a known security issue allow ssh login via system accounts?

Hi,

If this is not the right place to ask this, please redirect me. Hopefully it
is a known vulnerability, due to out of date software. We had a server
running OpenSSH_8.6p1 compiled on Ubuntu 16.04.7 which was compromised last
week. The intruder managed to achieve this:

  Feb 24 08:13:52 localhost sshd[32276]: Accepted password for backup from
5.161.47.185 port 37962 ssh2

This despite that /etc/passwd has:

  backup:x:34:34:backup:/var/backups:/usr/sbin/nologin

And /etc/shadow has:

  backup:*:16359:0:99999:7:::

Either the /usr/bin/nologin or the "*" in the second field of
/etc/shadow
should have been enough to prevent "Accepted password for backup." The
/usr/sbin/nologin is the standard version for that Ubuntu generation, byte
identical.

Adding this to sshd_config was effective:

  DenyUsers backup

If that's still not the default for system-level users like
"backup", would
adding it be a reasonble feature request? Or is that on the distros to
define their default sshd_config settings?

The files in pam.d on the compromised system are standard. There's no public
key for "backup", and no ".ssh" folder in /var/backups. The
intruder managed
to send out spam via the local postfix service, which is what made the
intrusion obvious. OSSEC (Wazuh) didn't spot anything. We've of course
taken
the system offline. But we'd like to understand how that login by
"backup"
was possible.

Thanks for any pointers. It's hard to google for this, due to
"backup" being
such a generic term.  

Whit

On Mon, Feb 28, 2022 at 12:48:31PM -0500, Whit Blauvelt
wrote:
> Hi,
> 
> If this is not the right place to ask this, please redirect me. Hopefully
it
> is a known vulnerability, due to out of date software. We had a server
> running OpenSSH_8.6p1 compiled on Ubuntu 16.04.7 which was compromised last
> week. The intruder managed to achieve this:
> 
>   Feb 24 08:13:52 localhost sshd[32276]: Accepted password for backup from
5.161.47.185 port 37962 ssh2
> 
> This despite that /etc/passwd has:
> 
>   backup:x:34:34:backup:/var/backups:/usr/sbin/nologin
> 
> And /etc/shadow has:
> 
>   backup:*:16359:0:99999:7:::
> 
> Either the /usr/bin/nologin or the "*" in the second field of
/etc/shadow
> should have been enough to prevent "Accepted password for
backup." The
> /usr/sbin/nologin is the standard version for that Ubuntu generation, byte
> identical.
> 
> Adding this to sshd_config was effective:
> 
>   DenyUsers backup
> ...
Possibly OT, but have you verified the system's PAM modules/configuration as
well? They may
have used the backup account post-exploitation if everything else checked out.

-c

On 2022-02-28T18:55, Whit Blauvelt <whit at transpect.com>
wrote:
> Hi,
> 
> If this is not the right place to ask this, please redirect me. Hopefully
it
> is a known vulnerability, due to out of date software. We had a server
> running OpenSSH_8.6p1 compiled on Ubuntu 16.04.7 which was compromised last
> week. The intruder managed to achieve this:
> 
>   Feb 24 08:13:52 localhost sshd[32276]: Accepted password for backup from
5.161.47.185 port 37962 ssh2
> 
> This despite that /etc/passwd has:
> 
>   backup:x:34:34:backup:/var/backups:/usr/sbin/nologin
> 
> And /etc/shadow has:
> 
>   backup:*:16359:0:99999:7:::
> 
> Either the /usr/bin/nologin or the "*" in the second field of
/etc/shadow
> should have been enough to prevent "Accepted password for
backup." The
> /usr/sbin/nologin is the standard version for that Ubuntu generation, byte
> identical.
There are a few more mechanisms that may allow password logins despite
the measures you named: When using PAM authentication (which I would
assume, since you are talking about Ubuntu) all the configuration in
/etc/pam.d regarding ssh logins, so /etc/pam.d/ssh and
/etc/pam.d/common* will be in effect. If e.g. some module there
specifies a different password and shell for backup, e.g. from LDAP,
NIS, ActiveDirectory or different passwd/shadow files, those might
apply. You said below that pam.d is "standard", but installed packages
might modify configuration there, so I'd suggest having another look
there.

Also, depending on your configuration, /usr/sbin/nologin may not be
sufficient to prevent port-forwarding without an open shell, thus
enabling a spammer to forward some SMTP port for sending spam without
authentication by "looking like" coming from an internal server.  This
is also hard to see and distinguish from an attacker having a shell in
the logs.
> Adding this to sshd_config was effective:
> 
>   DenyUsers backup
> 
> If that's still not the default for system-level users like
"backup", would
> adding it be a reasonble feature request? Or is that on the distros to
> define their default sshd_config settings?
I'd consider it good practice to selectively 'AllowGroup ssh-login'
or
something to that effect for any internet-exposed system. But common
distros are far more open than that.  I don't think OpenSSH is the right
place for a policy change, since as mentioned before, there is a lot
going on in PAM that will affect what OpenSSH does here, without being
part of OpenSSH.


Ciao,

Alexander W?rstlein.

On Tue, 1 Mar 2022 at 04:52, Whit Blauvelt <whit at transpect.com>
wrote:
> If this is not the right place to ask this, please redirect me. Hopefully
it
> is a known vulnerability, due to out of date software. We had a server
> running OpenSSH_8.6p1 compiled on Ubuntu 16.04.7
What options did you configure it with?  In particular, did you enable PAM?

[...]
> Either the /usr/bin/nologin or the "*" in the second field of
/etc/shadow
> should have been enough to prevent "Accepted password for
backup."
If you enabled PAM then that's a function of the PAM stack and its config.
> The /usr/sbin/nologin is the standard version for that Ubuntu generation,
> byte identical.
Have you verified that the sshd has not been tampered with?
> Adding this to sshd_config was effective:
>   DenyUsers backup
> If that's still not the default for system-level users like
"backup", would
> adding it be a reasonble feature request? Or is that on the distros to
> define their default sshd_config settings?
That would be up to the distros.
> The files in pam.d on the compromised system are standard.
"standard" as in "vendor-supplied" or "as we normally
set them"?
> There's no public
> key for "backup", and no ".ssh" folder in /var/backups.
The intruder managed
> to send out spam via the local postfix service, which is what made the
> intrusion obvious. OSSEC (Wazuh) didn't spot anything. We've of
course taken
> the system offline. But we'd like to understand how that login by
"backup"
> was possible.
I'd be having a very close look at the PAM config.  I've seen multiple
instances where a misconfigured PAM stack failed open and accepted
either an empty password or any password.  One instance also ended up
being used for spam as you describe.

You can use pam-test-harness.c (https://www.dtucker.net/patches/) to
test your config.

-- 
Darren Tucker (dtucker at dtucker.net)
GPG key 11EAA6FA / A86E 3E07 5B19 5880 E860  37F4 9357 ECEF 11EA A6FA (new)
    Good judgement comes with experience. Unfortunately, the experience
usually comes from bad judgement.

On Mon, 28 Feb 2022, Whit Blauvelt wrote:
> Hi,
> 
> If this is not the right place to ask this, please redirect me. Hopefully
it
> is a known vulnerability, due to out of date software. We had a server
> running OpenSSH_8.6p1 compiled on Ubuntu 16.04.7 which was compromised last
> week. The intruder managed to achieve this:
> 
>   Feb 24 08:13:52 localhost sshd[32276]: Accepted password for backup from
5.161.47.185 port 37962 ssh2
> 
> This despite that /etc/passwd has:
> 
>   backup:x:34:34:backup:/var/backups:/usr/sbin/nologin
> 
> And /etc/shadow has:
> 
>   backup:*:16359:0:99999:7:::
We're not aware of any security problems in OpenSSH 8.6 that could yield
access to a locked account like this.

On Ubuntu, OpenSSH delegates to PAM for password authentication and
verification of the user's shell against /etc/shells. You can check
whether this actually enabled using 'sudo sshd -T | grep ^usepam'

It sounds like you have already verified that your PAM configuration was
not tampered with, so that removes one possibility. Reviewing the Ubuntu
PAM configurations and the patches they apply to sshd seem to be prudent
next steps.

It's possible that the above log message was not from the initial
compromise, but rather the attacker running their own sshd to act as a
more reliable foothold.

Hopefully you're able to preseve and perform forensics on the host.
If there is evidence of a bug in sshd, then please send it to the
security bug reporting contact: openssh at openssh.com

Thanks,
Damien Miller