openssh unix dev - Sep 2025 - (PerSource)Penalties default perhaps too aggressive?

On Wed, 10 Sep 2025, Colin Watson wrote:
> On Wed, Sep 10, 2025 at 09:13:54PM +0200, hvjunk wrote:
> > Busy with my first deployment/lab test of PVE9/Debian13 that uses
OpenSSH
> 10.0-p1 (1:10.0p1-7 Deb package version) and my normal ssh-copy-id triggers
> the penalty and then doesn?t install the keys. In *my* case I have like 4x
> keys to load, so ssh-copy-id tries them all, and then get the penalty
triggers
> and the keys aren?t loaded.
> 
> I cherry-picked a post-10.0 upstream fix in this area into Debian 
> testing/unstable, but haven't yet issued a stable update for it.  Does 
> https://bugs.debian.org/1080350 sound as though it matches your 
> symptoms?
AFAIK that's a different condition: that would cause MaxStartups to
"latch" on under some situations.

I think what is happening here is that ssh-copy-id is making a separate
ssh connection for each key it tries. If these connections fail because
the key is not accepted, then each failed connection will add the
"authfail" penalty to the running total for your source address.

In the default configuration, a single authfail penalty (5s) is below
the enforcement threshold (15s), so it take at least three attempts to
reach the threshold. Trying four keys this way that fail authentication
would certainly hit the penalty.

Once the penalty thresold is hit, connections are blocked until it
expires.

Maybe ssh-copy-id's behaviour of filtering keys by attempting a
connection with each one could be modified. Perhaps instead it could
attempt authentication normally and filter keys by grepping in
authorized_keys. On the other hand there's probably a reason for why it
works the way it does currently.

Another, more annoying thing ssh-copy-id could do is ratelimit
connections to ensure that the the penalty threshold is never it. If it
assumed the default PerSourcePenalties, then it would only need to do
this if when it is asked to copy three or more keys and even then would
only need to throttle after the second key is tried.

If you're in control of the server, you can also avoid this condition
by reducing the authfail penalty and/or increasing the "min" penalty
threshold. You want authfail * nkeys < min

Alternately, as others have suggested, you can allow-list your client
via PerSourcePenaltyExemptList.

Finally, the PerSourcePenalties defaults are little more than my guesses
at values that will usefully throttle attacks without significantly
affecting legitimate traffic. Feedback is always welcome.

-d

Hi Damien,

 Let me first state by saying: Thank you for adding this ?feature?, and though
it is adding value and I?ll use and tune/tweak it in future as need be, I do
believe you?ve chosen too aggressive defaults
> On 11 Sep 2025, at 01:36, Damien Miller <djm at mindrot.org> wrote:
> 
> 
> Maybe ssh-copy-id's behaviour of filtering keys by attempting a
> connection with each one could be modified. Perhaps instead it could
> attempt authentication normally and filter keys by grepping in
> authorized_keys. On the other hand there's probably a reason for why it
> works the way it does currently.
Well, that is not always that easy to do:
What about keys in LDAP/etc.?
What about keys in a different directories like
/etc/ssh/authorized_keys.d/${user}/ ?

How do you propose it would check for those other sources, other than
reimplementing a openssh server config parser? What about the cases where it
hits a chrooted environment after login that it can?t check those other sources?
or the system is hardened and the user can?t read the /etc/ssh/sshd_config*
files ?

ssh-copy-id, from going through the script, is doing a K.I.S.S. operation and
that is what makes it so easy to use and useful IMHO.
> Another, more annoying thing ssh-copy-id could do is ratelimit
> connections to ensure that the the penalty threshold is never it. If it
> assumed the default PerSourcePenalties, then it would only need to do
> this if when it is asked to copy three or more keys and even then would
> only need to throttle after the second key is tried.
Yes, as I mentioned in another reply, I patched it to get passed this problem?
but suddenly I?m waiting like 15seconds for it now? I have to do a bug request,
and have user complaints coming about it :(=)
> If you're in control of the server,
Yes, *IF*
> you can also avoid this condition
> by reducing the authfail penalty and/or increasing the "min"
penalty
> threshold. You want authfail * nkeys < min
This is going to be the normal for me, and to be honest, was a very UNEXPECTED
problem, and the documentation are scarce, with more false positives coming my
way than I?d say it is solving (in my cases and users at least)
 
> Alternately, as others have suggested, you can allow-list your client
> via PerSourcePenaltyExemptList.
What about the users from different places/mobiles?
 It?ll amount to 2000::/3, again, not adding the intended value then :(
> Finally, the PerSourcePenalties defaults are little more than my guesses
> at values that will usefully throttle attacks without significantly
> affecting legitimate traffic. Feedback is always welcome.
And *my* humble feedback:

 Yes, it is TOO AGGRESSIVE, perhaps take a leaf from sshguard?s defaults as I?ve
found it to be blocking a lot of incoming ?attacks? without getting in me and my
user?s way (yes, when I do trigger it, it was like when I su?d or used wrong
destination user multiple times, or automation systems with wrong keys/etc or
where the remote implemented password expiries)

If you ask me, I?d say to make the defaults to only fail/disconnect after the
10th attempt like this in a 5 second window, as anything more is a much better
indicator and much less false positive of a key-check-attack. Real key check
attacks I?ve not seen doing more than a single key to a single host/user, they
typically from what it appears, tried it on multiple hosts in sequence, thus
this setting doesn?t help that much, it will only really help when they have a
corpus of keys they want to check, and then they?ll check rather a bunch in the
same session as session setup and teardown takes time.

The attacks I *DO* see constantly in my ssh logs (that sshguard are blocking) is
multiple users and then for each user multiple passwords in the SAME SESSION.
The [preauth] ?attacks? I?ve seen was remote automation systems that the remote
sysadmins have made some user/password/key errors (and a connection
drop/disconnect in those cases had them confused and reported firewall issues?)
and they retry ?quickly? on first failure

Thank you for the feature, but making it more permissive and letting the
sysadmin(s) tweak would be my advice.

HEndrik