openssh unix dev - Mar 2023 - Minimize sshd log clutter/spam from unauthenticated connections

modern syslog daemons (including rsyslog, which is default on just about every 
linux system) allow you to filter efficiently on the message contents, not just 
the severity, so you can opt to throw out the messages you don't want.

I advocate for a slightly different way of dealing with it, filter these 
messages from your main logstream, but put them into either a script directly, 
or a separate file and have a script run against it. Have the script report the 
number of these messgaes that you get in a time period (minute, hour, whatever 
you want) and log that count back into your log stream

as Marcus Ranum said in his Artificial Ignorance writeup, the number of times 
that an uninteresting thing happens can be interesting.

If you see a big spike (or drop) is these attempts, it can indicate cause for 
concern.

David Lang

  On Sat, 
18 Mar 2023, Carsten Andrich wrote:
> Date: Sat, 18 Mar 2023 13:15:29 +0100
> From: Carsten Andrich <carsten.andrich at tu-ilmenau.de>
> To: openssh-unix-dev at mindrot.org
> Subject: Minimize sshd log clutter/spam from unauthenticated connections
> 
> Dear OpenSSH developers,
>
> a publicly accessible sshd on port 22 generates a lot of log clutter from 
> unauthenticated connections. For an exemplary host on a university network,
> sshd accumulates 5~20k log lines on a single day (more than 90% of the
total
> amount of syslog lines). That is despite the host having a restricted 
> configuration (no SSH password authentication, firewall rate limit for new 
> SSH connections on /24 subnets permitting a few connections per hour,
however
> with a shorter timeout). I'd expect even more log messages for a
default
> configuration (password auth enabled and no firewall rate limit).
>
> Would you be open to introducing a new config option to suppress any log 
> messages from yet unauthenticated connections? If such a suggestion has
been
> discussed before, please direct me to it. I haven't found anything in
the
> archives.
>
> Any log messages including successful authentication and afterwards are
still
> desired, so changing the log level to above INFO will not help.
Additionally,
> even unauthenticated connections cause messages with levels ERROR 
> ("kex_exchange_identification: Connection closed by remote host")
or even
> CRITICAL ("Timeout before authentication"). As I periodically
scan my hosts'
> syslogs for messages with level WARNING or above, I currently have to
filter
> these messages to keep my inbox from overflowing.
>
> Thanks and best regards,
> Carsten
>
> _______________________________________________
> openssh-unix-dev mailing list
> openssh-unix-dev at mindrot.org
> https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev
>

On 18.03.23 14:34, David Lang wrote:
> modern syslog daemons (including rsyslog, which is default on just 
> about every linux system) allow you to filter efficiently on the 
> message contents, not just the severity, so you can opt to throw out 
> the messages you don't want.
>
> I advocate for a slightly different way of dealing with it, filter 
> these messages from your main logstream, but put them into either a 
> script directly, or a separate file and have a script run against it. 
> Have the script report the number of these messgaes that you get in a 
> time period (minute, hour, whatever you want) and log that count back 
> into your log stream
>
> as Marcus Ranum said in his Artificial Ignorance writeup, the number 
> of times that an uninteresting thing happens can be interesting.
>
> If you see a big spike (or drop) is these attempts, it can indicate 
> cause for concern.
I run Debian with systemd-journald instead of rsyslog. AFAIK journald 
does not support filtering of its ingress log messages. Only the output 
can be filtered with journalctl, but by then it's already too late in 
terms of log spam on disk.

Regardless of which syslog solution is being used, I find it 
inconvenient to manually assemble and maintain a filter list. I believe 
an sshd configuration option, which of course defaults to false, would 
be the best solution. Currently, the sheer amount of messages from 
unauthenticated connections drowns out anything else. I took this 
opportunity to sift through sshd's messages of almost 2 years via:

journalctl -t sshd -o cat \
     | grep -v '^Accepted ' \
     | sed -E 's/[Uu]ser \S+/user .../' \
     | sed -E 's/[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+/0.0.0.0/' \
     | sed -E 's/port\s+[0-9]+/port 0/' \
     | sed -E 's/".*"/"..."/' \
     | sed -E 's/Change of username or service not allowed: .+/Change of
username or service not allowed: .../' \
     | sed -E 's/Their offer: .+/Their offer: .../' \
     | sort -u

I found a select few attempts to mess with, identify, or exploit log 
parsing IDS/IPS software like fail2ban (and feel confirmed in my choice 
of an alternative solution with far less attack surface, see my other mail):

Invalid user $(ping -c 1 16e939dc.ad.xspzo.com) from ...
Invalid user ' $(ping -c 1 16e939dc.ad.xspzo.com) from ...
Invalid user ' or '1'='1' - from 176.100.42.41

Only two concerning messages came up:

error: beginning MaxStartups throttling
fatal: ssh_sandbox_violation: unexpected system call (arch:0xc000003e,syscall:20
@ 0x7f1d8469e1f5) [preauth]

I definitely wouldn't want to miss the above among an almost endless 
list of (invalid) usernames, (dis-)connecting IP addresses, invalid 
version strings, etc. that various script kiddies are playing with. 
While I do get your point, I doubt analyzing all of these individual 
messages has a reasonable cost-benefit ratio. I occasionally look at the 
amount of rejected connections (my nftables rules keep track of that) to 
see any unusual uptick of interest, upon which I investigate.

Best regards,
Carsten