openssh unix dev - Mar 2023 - Minimize sshd log clutter/spam from unauthenticated connections

Dear OpenSSH developers,

a publicly accessible sshd on port 22 generates a lot of log clutter 
from unauthenticated connections. For an exemplary host on a university 
network, sshd accumulates 5~20k log lines on a single day (more than 90% 
of the total amount of syslog lines). That is despite the host having a 
restricted configuration (no SSH password authentication, firewall rate 
limit for new SSH connections on /24 subnets permitting a few 
connections per hour, however with a shorter timeout). I'd expect even 
more log messages for a default configuration (password auth enabled and 
no firewall rate limit).

Would you be open to introducing a new config option to suppress any log 
messages from yet unauthenticated connections? If such a suggestion has 
been discussed before, please direct me to it. I haven't found anything 
in the archives.

Any log messages including successful authentication and afterwards are 
still desired, so changing the log level to above INFO will not help. 
Additionally, even unauthenticated connections cause messages with 
levels ERROR ("kex_exchange_identification: Connection closed by remote 
host") or even CRITICAL ("Timeout before authentication"). As I 
periodically scan my hosts' syslogs for messages with level WARNING or 
above, I currently have to filter these messages to keep my inbox from 
overflowing.

Thanks and best regards,
Carsten

I guess you might find fail2ban useful.

It scans logfiles (like /var/log/sshd.log), and when it sees too many
authentication failures from an IP address (or network range) it can issue
commands to drop any further attempts via a firewall.

By having it read its own logfile it's possible to have repeated offenders
be cut out for longer and longer time spans.

https://www.fail2ban.org/wiki/index.php/Main_Page
https://supine.com/posts/2012/08/fail2ban-monitoring-itself-recursively/

modern syslog daemons (including rsyslog, which is default on just about every 
linux system) allow you to filter efficiently on the message contents, not just 
the severity, so you can opt to throw out the messages you don't want.

I advocate for a slightly different way of dealing with it, filter these 
messages from your main logstream, but put them into either a script directly, 
or a separate file and have a script run against it. Have the script report the 
number of these messgaes that you get in a time period (minute, hour, whatever 
you want) and log that count back into your log stream

as Marcus Ranum said in his Artificial Ignorance writeup, the number of times 
that an uninteresting thing happens can be interesting.

If you see a big spike (or drop) is these attempts, it can indicate cause for 
concern.

David Lang

  On Sat, 
18 Mar 2023, Carsten Andrich wrote:
> Date: Sat, 18 Mar 2023 13:15:29 +0100
> From: Carsten Andrich <carsten.andrich at tu-ilmenau.de>
> To: openssh-unix-dev at mindrot.org
> Subject: Minimize sshd log clutter/spam from unauthenticated connections
> 
> Dear OpenSSH developers,
>
> a publicly accessible sshd on port 22 generates a lot of log clutter from 
> unauthenticated connections. For an exemplary host on a university network,
> sshd accumulates 5~20k log lines on a single day (more than 90% of the
total
> amount of syslog lines). That is despite the host having a restricted 
> configuration (no SSH password authentication, firewall rate limit for new 
> SSH connections on /24 subnets permitting a few connections per hour,
however
> with a shorter timeout). I'd expect even more log messages for a
default
> configuration (password auth enabled and no firewall rate limit).
>
> Would you be open to introducing a new config option to suppress any log 
> messages from yet unauthenticated connections? If such a suggestion has
been
> discussed before, please direct me to it. I haven't found anything in
the
> archives.
>
> Any log messages including successful authentication and afterwards are
still
> desired, so changing the log level to above INFO will not help.
Additionally,
> even unauthenticated connections cause messages with levels ERROR 
> ("kex_exchange_identification: Connection closed by remote host")
or even
> CRITICAL ("Timeout before authentication"). As I periodically
scan my hosts'
> syslogs for messages with level WARNING or above, I currently have to
filter
> these messages to keep my inbox from overflowing.
>
> Thanks and best regards,
> Carsten
>
> _______________________________________________
> openssh-unix-dev mailing list
> openssh-unix-dev at mindrot.org
> https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
>

Dropping logs/audits and possible indications-of-attack is NOT cyber security
best practices.


> On Mar 18, 2023, at 6:15 AM, Carsten Andrich <carsten.andrich at
tu-ilmenau.de> wrote:
> 
> Dear OpenSSH developers,
> 
> a publicly accessible sshd on port 22 generates a lot of log clutter from
unauthenticated connections. For an exemplary host on a university network, sshd
accumulates 5~20k log lines on a single day (more than 90% of the total amount
of syslog lines). That is despite the host having a restricted configuration (no
SSH password authentication, firewall rate limit for new SSH connections on /24
subnets permitting a few connections per hour, however with a shorter timeout).
I'd expect even more log messages for a default configuration (password auth
enabled and no firewall rate limit).
> 
> Would you be open to introducing a new config option to suppress any log
messages from yet unauthenticated connections? If such a suggestion has been
discussed before, please direct me to it. I haven't found anything in the
archives.
> 
> Any log messages including successful authentication and afterwards are
still desired, so changing the log level to above INFO will not help.
Additionally, even unauthenticated connections cause messages with levels ERROR
("kex_exchange_identification: Connection closed by remote host") or
even CRITICAL ("Timeout before authentication"). As I periodically
scan my hosts' syslogs for messages with level WARNING or above, I currently
have to filter these messages to keep my inbox from overflowing.
> 
> Thanks and best regards,
> Carsten

1On Sat, Mar 18, 2023 at 8:23?AM Carsten Andrich
<carsten.andrich at tu-ilmenau.de> wrote:
> a publicly accessible sshd on port 22 generates a lot of log clutter
> from unauthenticated connections.
This is not unique to OpenSSH. Any well-known service on any
publicly-accessible host will be probed incessantly.

In addition to source-blocking solutions (portknocking, fail2ban, et.
al.) and log filtering solutions, something that can be surprisingly
effective is to simply configure sshd to listen on a TCP port other
than 22. Virtually no attackers perform full port scans looking for
hidden ssh daemons, because there?s so much low-hanging fruit on port
22.

On my home network, I have a publicly-accessible sshd instance that is
listening on a TCP port <1024 that is not port 22. For more than 10
years (until a botnet finally found it), I was the only person who
ever attempted to connect to it. (In response to its discovery, I
could have simply moved the sshd instance to a different port, but
instead I used that as an excuse to experiment with a portknocker
configuration.)

In InfoSec, obfuscation has a bad rap, because it is frequently
deployed in an attempt to hide a security vulnerability instead of
applying remediation. But there are a few instances where obfuscation
can be a valuable and appropriate tool in one?s toolbox. And reducing
voluminous log noise by evading 99.999% of ankle-biting script kiddies
is one of those instances.