openssh unix dev - May 2022 - Azure DevOps hosted git, and Bastion, failing to negotiate encryption?

I'm dealing with Azure DevOps git services with which recent, security
hardened SSH clients on a RHEL 8 variant cannot stablish public-key
based  SSH links to the Azure Devops. Other RHEL 8 based systems work
well.

Conversely, Azure Bastion cannot use Azure key vault stored private
SSH keys to access the same RHEL 8 based servers, though they can
reach other hosts with the same private key and the same public key.
I've also used the private SSH key, locally, to connect to the various
servers.

Has anyone else seen this issue? Have there been incremental updates
in RHEL 8 published OpenSSH, or does someone else here have a handle
on public key incompatibilities with Microsoft's hosted services. I've
gotten past the "key vault stored keys must be large enough, and in
PEM format, and can't be cut & pasted into the Azure Portal"
issues.

Nico Kadel-Garcia wrote:
> I'm dealing with Azure DevOps git services with which recent, security
> hardened SSH clients on a RHEL 8 variant cannot stablish public-key
> based  SSH links to the Azure Devops.
How was that security hardning done?

And if it's something RHEL-specific then what about upstream OpenSSH?

> Conversely, Azure Bastion cannot use Azure key vault stored private
> SSH keys to access the same RHEL 8 based servers,
Try running sshd with -ddd and see what it says?

Although this seems to be a RHEL issue maybe you can debug it yourself..


//Peter

On 21.05.22 01:41, Nico Kadel-Garcia wrote:
> I'm dealing with Azure DevOps git services with which recent, security
> hardened SSH clients on a RHEL 8 variant cannot stablish public-key
> based  SSH links to the Azure Devops. Other RHEL 8 based systems work
> well.
> 
> Conversely, Azure Bastion cannot use Azure key vault stored private
> SSH keys to access the same RHEL 8 based servers, though they can
> reach other hosts with the same private key and the same public key.
> I've also used the private SSH key, locally, to connect to the various
> servers.
In my experience, if you can run a plain "ssh -v" as the client, the 
output *does* include details about the problem *somewhere*. To wit:
> $ ssh -v root at SomeOldBox
[...]
> Unable to negotiate with SomeOldBox port 22: no matching key exchange > 
method found. Their offer: diffie-hellman-group-exchange-sha1,>
     diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
> $ ssh -v -o "KexAlgorithms +diffie-hellman-group14-sha1" root at
SomeOldBox
[...]
> Unable to negotiate with SomeOldBox port 22: no matching
>    host key type found. Their offer: ssh-rsa,ssh-dss
> $ ssh -v -o "KexAlgorithms +diffie-hellman-group14-sha1"
>    -o "HostKeyAlgorithms +ssh-rsa" root at SomeOldBox
[...]
> Unable to negotiate with SomeOldBox port 22: no matching MAC found.
>    Their offer: hmac-md5,hmac-sha1,hmac-ripemd160,
>       hmac-ripemd160 at openssh.com,hmac-sha1-96,hmac-md5-96
etc. etc.. If I had to *guess*, I'd say that the dissent is about the 
key vault somehow offering (only) ssh-rsa while the hardened RHEL now 
insists on rsa-sha2-* for a key algorithm ...

Regards,
-- 
Jochen Bern
Systemingenieur

Binect GmbH
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3449 bytes
Desc: S/MIME Cryptographic Signature
URL:
<http://lists.mindrot.org/pipermail/openssh-unix-dev/attachments/20220523/6523fc4f/attachment-0001.p7s>

On Mon, May 23, 2022 at 5:07 AM Jochen Bern <Jochen.Bern at binect.de>
wrote:
>
> On 21.05.22 01:41, Nico Kadel-Garcia wrote:
> > I'm dealing with Azure DevOps git services with which recent,
security
> > hardened SSH clients on a RHEL 8 variant cannot stablish public-key
> > based  SSH links to the Azure Devops. Other RHEL 8 based systems work
> > well.
> >
> > Conversely, Azure Bastion cannot use Azure key vault stored private
> > SSH keys to access the same RHEL 8 based servers, though they can
> > reach other hosts with the same private key and the same public key.
> > I've also used the private SSH key, locally, to connect to the
various
> > servers.
>
> In my experience, if you can run a plain "ssh -v" as the client,
the
> output *does* include details about the problem *somewhere*. To wit:
>
> > $ ssh -v root at SomeOldBox
> [...]
> > Unable to negotiate with SomeOldBox port 22: no matching key exchange
>    method found. Their offer: diffie-hellman-group-exchange-sha1,>
>      diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
>
> > $ ssh -v -o "KexAlgorithms +diffie-hellman-group14-sha1"
root at SomeOldBox
> [...]
> > Unable to negotiate with SomeOldBox port 22: no matching
> >    host key type found. Their offer: ssh-rsa,ssh-dss
>
> > $ ssh -v -o "KexAlgorithms +diffie-hellman-group14-sha1"
> >    -o "HostKeyAlgorithms +ssh-rsa" root at SomeOldBox
> [...]
> > Unable to negotiate with SomeOldBox port 22: no matching MAC found.
> >    Their offer: hmac-md5,hmac-sha1,hmac-ripemd160,
> >       hmac-ripemd160 at openssh.com,hmac-sha1-96,hmac-md5-96
> etc. etc.. If I had to *guess*, I'd say that the dissent is about the
> key vault somehow offering (only) ssh-rsa while the hardened RHEL now
> insists on rsa-sha2-* for a key algorithm ...
The key vault stores whatever key I put in it. The issue is that on
the CIS published OpenSSH client setup, they no longer support plain
"ssh-rsa" protocol anymore. The trick is to enable it as needed:

    Host azure-git-server.domain
        PuubKeyAcceptedKeyTypes +ssh-rsa

The safest place to put it for all clients on the server is:

    /etc/ssh/ssh_config.d/10-azure-git-server

I'm unsure of the change is built into the most recent versions of
OpenSSH on RHEL 8, or in another package like crypto-policies that was
hand-modified by the authors of the particular VM. But it's working
well now.