openssh bugs - Nov 2022 - [Bug 3490] Inconsistent behaviour when using -i and -J options

https://bugzilla.mindrot.org/show_bug.cgi?id=3490

            Bug ID: 3490
           Summary: Inconsistent behaviour when using -i and -J options
           Product: Portable OpenSSH
           Version: 8.7p1
          Hardware: All
                OS: Linux
            Status: NEW
          Severity: normal
          Priority: P5
         Component: ssh
          Assignee: unassigned-bugs at mindrot.org
          Reporter: wonczak at uni-koeln.de

I already reported this in bugzilla.redhat.com
(https://bugzilla.redhat.com/show_bug.cgi?id=2127970), but  Dmitry
Belyavskiy told me to raise the issue here as well. 
For completeness sake: I'm on CentOS 9 Stream, fully updated.

I'll copy my bug report from the other bugzilla:

Description of problem:
Keys explicitly provided by "-i /path/to/keyfile" is ignored when -J
<jumphost> is on the command line

Version-Release number of selected component (if applicable):

openssh-8.7p1-22.el9.x86_64

How reproducible:
always

Steps to Reproduce:
1. ssh -i /path/to/ssh-key -J user at jumphost user at targethost

Actual results:
ssk-key is ignored, login fails

Expected results:
ssh-key should be used, login succeeds

Additional info:
Detailed description and observations. 

In the process of setting up a jumphost I noticed what I would classify
as a bug in ssh.

I set up a jumphost which knows userA, which then passes the connection
though to userA at targethost. The jumphost does not allow password-login
of userA, just key-login. A special key was generated for this
connection chain, which is stored in /path/to/ssh-key. The public key
was put into the authorized_keys-file of userA on both jumphost and
targethost.

I can login into the jumphost by

ssh -i /path/to/ssh-key userA at jumphost

I can also directly login to the targethost:

ssh -i /path/to/ssh-key userA at jumphost

This will be disallowed later, hence the requirement for a jumphost.
The jumphost will also disallow direct logins of users later; direct
login is only allowed for debugging right now. Now for the weird part:

ssh -i /path/to/ssh-key -J userA at jumphost userA at targethost

fails with 

userA at jumphost: Permission denied
(publickey,gssapi-keyex,gssapi-with-mic).
kex_exchange_identification: Connection closed by remote host
Connection closed by UNKNOWN port 65535

checking with "ssh -v", the key provided by the -i option is not even
read (keys replaced by #### for privacy):

(...)
debug1: Will attempt key: /home/localuser/.ssh/id_rsa RSA SHA256:#### 
agent
debug1: Will attempt key: user,11.11.2008,linux mgr user RSA
SHA256:#### agent
debug1: Will attempt key: /home/localuser/.ssh/id_dsa 
debug1: Will attempt key: /home/localuser/.ssh/id_ecdsa 
(...)
debug1: Next authentication method: publickey
debug1: Offering public key: /home/localuser/.ssh/id_rsa RSA
SHA256:#### agent
debug1: Authentications that can continue:
publickey,gssapi-keyex,gssapi-with-mic
debug1: Offering public key: user,11.11.2008,linux mgr user RSA
SHA256:#### agent
d
(...)

If I place /path/to/ssh-key into the .ssh-folder of localuser, the key
-is- offered automatically (no -i option this time), and login to
targethost works

(...)
debug1: Will attempt key: /home/localuser/.ssh/id_rsa RSA SHA256:#### 
agent
debug1: Will attempt key: user ECDSA SHA256:#### agent
debug1: Will attempt key: user,11.11.2008,linux mgr user RSA
SHA256:#### agent
debug1: Will attempt key: /home/localuser/.ssh/id_dsa 
debug1: Will attempt key: /home/localuser/.ssh/id_ecdsa 
(...)
debug1: Next authentication method: publickey
debug1: Offering public key: /home/localuser/.ssh/id_rsa RSA
SHA256:#### agent
debug1: Authentications that can continue:
publickey,gssapi-keyex,gssapi-with-mic
debug1: Offering public key: user ECDSA SHA256:#### agent
debug1: Server accepts key: user ECDSA SHA256:#### agent
debug1: Offering public key: user,11.11.2008,linux mgr user RSA
SHA256:#### agent
d
(...)

If I remove the key from localuser's .ssh-folder again, but add it to
ssh-agent: (ssh-add /path/to/ssh-key), login again works (no -i
option):

(...)
debug1: Will attempt key: /home/localuser/.ssh/id_rsa RSA SHA256:#### 
agent
debug1: Will attempt key: user ECDSA SHA256:#### agent
debug1: Will attempt key: user,11.11.2008,linux mgr user RSA
SHA256:#### agent
debug1: Will attempt key: /home/localuser/.ssh/id_dsa 
debug1: Will attempt key: /home/localuser/.ssh/id_ecdsa 
(...)
debug1: Next authentication method: publickey
debug1: Offering public key: /home/localuser/.ssh/id_rsa RSA
SHA256:#### agent
debug1: Authentications that can continue:
publickey,gssapi-keyex,gssapi-with-mic
debug1: Offering public key: /path/to/ssh-key ECDSA SHA256:#### agent
debug1: Server accepts key: /path/to/ssh-key ECDSA SHA256:#### agent
(...)

At one point I had forgotten to remove the -i from the command line
after adding the key to ssh-agent, and the last to lines changed to:

(...)
debug1: Offering public key: /path/to/ssh-kedebug1: Offering public
key: /path/to/ssh-key ECDSA SHA256:#### agent
debug1: Server accepts key: /path/to/ssh-key ECDSA SHA256:#### agent
y ECDSA SHA256:#### explicit agent
debug1: Server accepts key: /path/to/ssh-key ECDSA SHA256:#### explicit
agent
(...)

So it seems that the -i option -is- evaluated, but not used in all
cases. Very strange, and IMHO inconsistent behaviour indeed.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3490

Damien Miller <djm at mindrot.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         Resolution|---                         |WONTFIX
                 CC|                            |djm at mindrot.org
             Status|NEW                         |RESOLVED

--- Comment #1 from Damien Miller <djm at mindrot.org> ---
This is intentional - most command-line arguments are applied only to
the destination (targethost in your example) and not the connection to
the jumphost. This is noted in the manual page description for -J:
> Note that configuration directives supplied on the command-line
> generally apply to the destination host and not any specified
> jump hosts.  Use ~/.ssh/config to specify configuration for jump
> hosts.
-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3490

--- Comment #2 from Stephan Wonczak <wonczak at uni-koeln.de> ---
I still think this is inconsistent an unexpected behaviour. For one
thing, the documentation in "man" for -J is far from clear, expecially
in respect to the interactions with keys. 
If an explicit key is supplied by "-i" it should at least be offered
to
the jump host - as a user this is what I would expect. And in a way,
this is the behaviour if I use ssh-agent to have a key in my local key
purse. I see no reason why this should differ from supplying a key on
the command line, especially when it is possible to use multiple "-i".
This would also cover the case when the required key for the jump host
differs from the key for the target host - simply supply both keys via
-i.
On the other hand it really is unexpected when an explicitly given
(correct!) key is not even offered to the jump host.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3490

--- Comment #3 from Damien Miller <djm at mindrot.org> ---
Well, it's definitely not inconsistent. Inconsistency would be some
options being passed and others not - and we definitely don't want to
unconditionally pass commandline options through. E.g. I'd almost never
want -A passed through to the jump host.

So we have to make a decision: pass no options and be consistent, or
pass some options and be inconsistent. We chose consistency, because
it's easier to reason about deterministically.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3490

Damien Miller <djm at mindrot.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|RESOLVED                    |CLOSED

--- Comment #4 from Damien Miller <djm at mindrot.org> ---
OpenSSH 9.3 has been released. Close resolved bugs

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.