openssh bugs - Apr 2020 - [Bug 3153] New: Prefer user specified keys to avoid the agent overloading MaxAuthTries before even trying the key that was specified

https://bugzilla.mindrot.org/show_bug.cgi?id=3153

            Bug ID: 3153
           Summary: Prefer user specified keys to avoid the agent
                    overloading MaxAuthTries before even trying the key
                    that was specified
           Product: Portable OpenSSH
           Version: 8.2p1
          Hardware: Other
                OS: Linux
            Status: NEW
          Severity: enhancement
          Priority: P5
         Component: ssh
          Assignee: unassigned-bugs at mindrot.org
          Reporter: christian.ehrhardt at canonical.com

Created attachment 3385
  --> https://bugzilla.mindrot.org/attachment.cgi?id=3385&action=edit
RFC patch to start the discussion

Hi,
in a recent bug report to Ubuntu [1] I was wondering that this issue
seems to exist for so long and affect so many people
[2][3][4][5][6][7][8].
It was also discussed within openssh in [9][10].
But in all of those cases all that was done was to suggest workarounds
like "PubkeyAuthentication=no" or "IdentitiesOnly=yes", but
IMHO those
are all just exactly that - workarounds.

Also there are many "related but not entirely the same" upstream bugs
like [11][12], but it seems no one has yet discussed the approach we
had in mind.

If a usual user calls ssh like
  ssh -i <mykey> ...
And gets:
  "Too many authentication failures"
He'd not even think about <mykey> not even being tried.

The problem is that the current order in pubkey_prepare will order
those directly specified keys too late.

 * try keys in the following order:
 *     1. certificates listed in the config file
 *     2. other input certificates
 *     3. agent keys that are found in the config file
 *     4. other agent keys
 *     5. keys that are only listed in the config file

Yes, if <mykey> is in the agent it will come at #3 which usually is
early enough.
But if <mykey> isn't in the agent which is usual for any new key that
a
user created/tries it will come at #5.
If now the number of keys in #4 exceed the servers "i" the user gets
rejected without even trying <mykey> that he has specified.

Config may be config, but if a user specifies something directly on
invocation I'd expect it to supersede the configuration.

I'm clear that there might be more needed:
- has it to be ordered behind certs?
- do user-specified certs also need to get up in the list?
- does a test need to be added?
- discussion here about potential drawbacks
- ...

But I wanted to get things started and therefore I have attached a
sugegsted initial patch as well as a Ubuntu PPA [13] with it applied.

[1]: https://bugs.launchpad.net/debian/+source/openssh/+bug/1872145
[2]:
https://serverfault.com/questions/36291/how-to-recover-from-too-many-authentication-failures-for-user-root
[3]:
https://www.tecmint.com/fix-ssh-too-many-authentication-failures-error/
[4]:
https://superuser.com/questions/187779/too-many-authentication-failures-for-username
[5]:
https://blog.jasonmeridth.com/posts/ssh-too-many-authentication-failures/
[6]:
https://security.stackexchange.com/questions/65120/ssh-always-too-many-authentication-failures
[7]: https://www.unixtutorial.org/ssh-too-many-authentication-failures
[8]: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=203700
[9]:
https://lists.mindrot.org/pipermail/openssh-unix-dev/2003-December/019931.html
[10]:
https://lists.mindrot.org/pipermail/openssh-unix-dev/2016-April/035029.html
[11]: https://bugzilla.mindrot.org/show_bug.cgi?id=1499
[12]: https://bugzilla.mindrot.org/show_bug.cgi?id=1937
[13]:
https://launchpad.net/~paelzer/+archive/ubuntu/bug-872145-ssh-prefer-user-configured-key

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3153

--- Comment #1 from Christian Ehrhardt <christian.ehrhardt at
canonical.com> ---
Example effect of the patch:

Former behavior on a server with MaxAuthTries 4 the explicitly
specified key would not have been tried:

$ ssh -i /tmp/testkey -v horsea "echo 1" |& grep "Will
attempt"
debug1: Will attempt key: /home/paelzer/.ssh/id_rsa RSA ... agent
debug1: Will attempt key: ubuntu at cpaelzer-bastion RSA ... agent
debug1: Will attempt key: paelzer at lap RSA ... agent
debug1: Will attempt key: paelzer at swarm.n RSA ... agent
debug1: Will attempt key: /tmp/testkey RSA ... explicit

With the change becomes  this and works:

$ ssh -i /tmp/testkey -v horsea "echo 1" |& grep "Will
attempt"
debug1: Will attempt key: /tmp/testkey RSA ... explicit
debug1: Will attempt key: /home/paelzer/.ssh/id_rsa RSA ... agent
debug1: Will attempt key: ubuntu at cpaelzer-bastion RSA ... agent
debug1: Will attempt key: paelzer at lap RSA ... agent
debug1: Will attempt key: paelzer at swarm.n RSA ... agent

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3153

Damien Miller <djm at mindrot.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |djm at mindrot.org

--- Comment #2 from Damien Miller <djm at mindrot.org> ---
Created attachment 3387
  --> https://bugzilla.mindrot.org/attachment.cgi?id=3387&action=edit
identitiesOnly=explicit

maybe we could do something like this: allow IdentitiesOnly=explicit to
disable adding agent keys that aren't explicitly listed as
IdentityFiles/-i

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3153

--- Comment #3 from Christian Ehrhardt <christian.ehrhardt at
canonical.com> ---
Hi Damien,
the suggested IdentitiesOnly=explicit is interesting, but it won't
cover the part of the users that need the fix the most.

I was mostly thinking about the less experienced users - those who'd
not understand why things are failing and not know how to hunt for the
existing workarounds.
The IdentitiesOnly=explicit option would only fix it for those people
that know what is going on (as they need to set it) unless if it would
be the default config value. But as default it would break plenty of
other use cases.

But thinking about configs, maybe we'd want/need to go a step further.
Today the preference order is in the code, maybe we'd want to expose
that as a config. With my patch applied we have 6 classes of Auth to
offer.

We might apply my patch, but then revamp it completely to have the
order configurable. The following would represent the order with my
patch applied:

IdentitiesOrder=key-explicit,cert-configured,cert-other,key-agent-configured,key-agent,key-other

Everyone is welcome to bikeshed on the terms, but it actually was more
about discussing the idea :-)

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3153

Roumen Petrov <bugtrack at roumenpetrov.info> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |bugtrack at roumenpetrov.info

--- Comment #4 from Roumen Petrov <bugtrack at roumenpetrov.info> ---
I cannot understand what is issue with agent keys.

User start agent and adds some keys(identities). It is expected those
keys to take precedence over all other keys as they are loaded first!
Then when is started client it could add other identities.

Directive IdentitiesOnly set to yes is intended to minimize used agent
keys.


Sample:
agent with keys
  agent1
  agent2
  agent3

To simplify let assume that configuration does no add other identities.

a) client .. -i no_agent -i agent2 .. 

If IdentitiesOnly is set to yes client should try "agent2" and
"no_agent". 

b) client .. -i no_agent .. 
If IdentitiesOnly is set to yes client should try only "no_agent".


So I cannot see why IdentitiesOnly=yes is not solution.



Reading OpenSSH manual page I partially agree with first report:
----
-i identity_file
  Selects a file from which the identity (private key) for public key
authentication is read.  The default is .... Identity files may also be
specified on a per-host basis in the configuration file.  It is
possible to have multiple -i options (and multiple identities specified
in configuration files).
----

The only things missing is that ssh(1) does not suggest for more
details user to see directive IdentityFile ssh_config(5) where:
----
IdentityFile
...
 Additionally, any identities represented by the authentication agent
will be used for authentication unless IdentitiesOnly is set.
...
----

"Additionally" is not appropriate word as agent keys are loaded first
and is expected to be used first.


It seems to me this report is just documentation issue.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3153

--- Comment #5 from Christian Ehrhardt <christian.ehrhardt at
canonical.com> ---
Hi Roumen,
I can absolutely see your POV that I'd like to summarize "if you
read/know all of the documentation you see what happens". And I can
follow your argument that from there the obvious improvement would be
to enhance the docs to be more obvious.

But if I turn it around to the users perspective I'd rather convinced
of the proposed behavior:

user-Example A)
If we describe 100 admins the following scenario:
1. ssh agent has 5 keys loaded
2. you run ssh -i ExplicitKey foo at bar
And we then ask them "Do you expect that ExplicitKey will be tried?"
I'm pretty sure the majority will answer "yes it will try
ExplicitKey".

And even if you then hint at MaxAuthTries limiting the amount that can
be tried I assume that most would expect "what I specified explicitly
would go first, since after all I specified it explicitly".

user-Example B)
What currently happens to users is something like:
1. `ssh -i ExplicitKey foo at bar` works fine
2. .. N. some other actions which eventually make ssh-agent hold
>MaxAuthTries other keys
3. `ssh -i ExplicitKey foo at bar` suddenly fails now
4. Puzzled ?!?, after a long time finding the subtle details of
Agent/MaxAuthTries and wishes that at least what he specified
explicitly would have been tried.

Improved-Messaging example C)
Turning the case around again (no offense please, this example is
phrased slightly provocative to show my point). If the behavior isn't
changed, then I'd suggest instead of a doc change that people first
have to fail, then find the doc then understand it all ...
Instead if ssh gives up failing before the key on the commandline was
even tried ssh could emit a slightly different error.
Instead of
  "Too many authentication failures"
It could say:
  "Too many authentication failures, But just so you know, the key you
thought you use wasn't even tried"

I hope that helps to clarify why I think IdentitiesOnly and/or the
documentation thereof isn't enough.
Thanks in advance,
Christian

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3153

Ben <ben at smokingkangaroo.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |ben at smokingkangaroo.com

--- Comment #6 from Ben <ben at smokingkangaroo.com> ---
The problem that arises isn't just overloading MaxAuthTries. Very
popular services like GitHub and BitBucket use the SSH key that is
presented to identify what the user has access to. So when you have
multiple accounts, you NEED to be able to control which exact identity
is used, rather than merely adding an identity to the set and letting
SSH try them in whatever order it decides.

I know the order IS determined, but in some very common setups (like
using AddKeysToAgent=yes and an agent with that times out keys) the set
of keys that are in the agent at any moment in time is essentially
"random", which means it becomes highly unstable which key SSH will
decide to present to a given host.

The core of the issue is: for the majority of options, setting them on
the command line takes precedence over those in the config files, but
there's no way to do that with identities. Explicitly passing an
identity on the command line *is* treated as "read first", so it's
at
the head of the list and will be tried first by default. If that was
the end of the story it would be close enough to allowing ssh -i <key>
to work as an "override", as most other command line settings would
do.

But SSH then re-orders the list of identities based on what's currently
in the agent, so if one of the keys you're trying to override is in the
agent and the one you passed on the command line is not, then it lets
the one you were trying to override take precedence. IdentitiesOnly is
completely unrelated to this aspect of the problem; it stops SSH from
trying ADDITIONAL keys it finds in the agent, but it doesn't give you
any way to express on the command line "definitely use THIS key".

This re-ordering only really makes sense if you assume that if multiple
keys could authenticate then any of them is equally good AND that you
can try as many keys as are configured until one succeeds. For a lot of
common use cases these assumptions don't seem to hold up.

We can work around when the "there's no limit on the number of keys you
can try" assumption is false (which was the problem in all of those
reports Christian linked to) by using IdentitiesOnly and carefully
crafting our configuration so that there only are a small enough number
of keys that SSH will try. But this at least you can configure
"privately" for yourself, and once you've set it up there's no
manual
steps you need to make on each SSH call.

When the "all keys that work are equally good" assumption is false
(e.g. multiple GitHub accounts), the only workarounds I know are to set
up Host aliases (each with one key configured, and use IdentitiesOnly)
and use those instead of the real host, or else to configure multiple
IdentityFiles (or no IdentityFiles and use IdentitiesOnly=no) and
carefully manage which key is loaded in the agent. The host alias
workaround then can't be private if the SSH command is scripted and
shared with other developers (e.g. everyone working at Company123 needs
to know that Company123's private GitHub repositories should be
accessed as if they were at company123-github, not github.com), and
managing what keys are in the agent requires extra manual steps before
each SSH call. Neither option is something that can just be set on the
command line when a tool is calling SSH; they both require knowing
what's in the user's ssh config.

The change Christian originally proposed (where keys explicitly passed
on the command line keys are tried first EVEN if some other keys are
already loaded in the agent) would make it possible to use multiple
GitHub accounts nicely. Git can change the SSH command line, so you
tell it to add -i <the-right-key-for-this-repository> in the config for
certain repositories. If SSH would be guaranteed to try that key first
it would "just work" as an override of a default key specified in
~/.ssh/config. At present it can only be made to work if you don't have
a default set, and need to override the SSH command for every
repository. Alternatively, a setting that said "ignore any other keys
found in configuration read after this point" would also make this
usable (you could pass that setting on the command line along with -i
<key>).

Or alternatively alternatively, a setting to simply disable the
re-ordering based on the agent state would also work fine! Is there any
reason to do the reordering that other than maybe saving entering a
passphrase, if one of the already unlocked keys works? It seems to be
optimising for the wrong thing, in my opinion; I would rather be able
to control the order in which keys are tried, and I'm not sure I've
ever had a setup where I have multiple equally-good keys and it's
helpful for them to be chosen based on which happens to be unlocked.

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.