openssh bugs - Aug 2023 - [Bug 3603] New: ssh clients can't communicate with server with default cipher when fips is enabled at server end

https://bugzilla.mindrot.org/show_bug.cgi?id=3603

            Bug ID: 3603
           Summary: ssh clients can't communicate with server with default
                    cipher when fips is enabled at server end
           Product: Portable OpenSSH
           Version: 9.4p1
          Hardware: All
                OS: Linux
            Status: NEW
          Severity: critical
          Priority: P5
         Component: sshd
          Assignee: unassigned-bugs at mindrot.org
          Reporter: sshedi at vmware.com

Hi,

This seems like a regression at first but there is a way to work around
it.

When fips is enabled at server end and server has the following cipher
set,

```
root at phdev:~ $ sshd -T | grep ciphers
ciphers
chacha20-poly1305 at openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm at
openssh.com,aes256-gcm at openssh.com

root at phdev:~ $ rpm -q openssh
openssh-9.1p1-10.ph5.x86_64 (this happens with 9.4p1 as well)
```

The handshake with client starts with chacha20-poly1305 and this cipher
is not fips complaint.

I'm not sure what the intention was but in this commit:
https://github.com/openssh/openssh-portable/commit/a22b9ef21285e81775732436f7c84a27bd3f71e0

chacha20 cipher was promoted.

At the client end, "ssh user at IP" doesn't work and aborts almost
immediately.

To workaround this issue, we need to do:

"ssh -c aes128-ctr user at IP"

In place of aes128-ctr, we can use any other algo which is fips
complaint (aes256-ctr, aes192-ctr etc).

Expected result:
ssh server should handle this gracefully.

Possible solutions:

1. Change the cipher order in KEX_SERVER_ENCRYPT (myproposal.h)

2. Use the same order but tweak the cipher list at run time based on
fips status in the system.

We did something like in PhotonOS 3.0:
https://github.com/vmware/photon/blob/3.0/SPECS/openssh/openssh-7.8p1-fips.patch

But we are unsure about the issue this might cause.

3. Server should send a proper error message to client in this case
showing some details on what went wrong; currently client simply aborts
with zero info.

4. If fips is enabled and sshd_config has ciphers which are
incompatible in fips mode, sshd should throw a warning and use the next
available fips complaint cipher from the list.

Even now, we can do the following in sshd_config,

cipherlist
aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm at openssh.com,aes256-gcm at
openssh.com,chacha20-poly1305 at openssh.com

But we have to do it in all the server instances. I think this should
be handled by server considering fips scenario.

Please feel free to correct me if I'm wrong here.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3603

Shreenidhi Shedi <sshedi at vmware.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |djm at mindrot.org,
                   |                            |dtucker at dtucker.net

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3603

--- Comment #1 from Darren Tucker <dtucker at dtucker.net> ---
(In reply to Shreenidhi Shedi from comment #0)
[...]
> When fips is enabled at server end and server has the following
> cipher set,
> 
> ```
> root at phdev:~ $ sshd -T | grep ciphers
> ciphers
> chacha20-poly1305 at openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,
> aes128-gcm at openssh.com,aes256-gcm at openssh.com
This indicates your FIPS modifications to the server are incorrect.

The server seems to be advertising the chacha20 cipher but seems unable
or unwilling to actually use it.  If it's not going to support them it
should not offer them, and if the server's config explicitly includes
it then it should either refuse to start or remove them and log a
warning.
> root at phdev:~ $ rpm -q openssh
> openssh-9.1p1-10.ph5.x86_64 (this happens with 9.4p1 as well)
> ```
> 
> The handshake with client starts with chacha20-poly1305 and this
> cipher is not fips complaint.
> 
> I'm not sure what the intention was but in this commit:
> https://github.com/openssh/openssh-portable/commit/
> a22b9ef21285e81775732436f7c84a27bd3f71e0
> 
> chacha20 cipher was promoted.
That was the intent, and it's fine.  Simplifying a bit, the way it's
supposed to work is that the client picks the cipher from the list that
the server offers that it likes most.

Your server is lying about what ciphers it supports, and when the
client takes it up on its offer for chacha20, it doesn't work (from
your description, probably the server is aborting, but you'd have to
check your server logs).

Fix your server and it should work fine.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3603

--- Comment #2 from Shreenidhi Shedi <sshedi at vmware.com>
---
> Your server is lying about what ciphers it supports
This is the concern I have here. We are not explicitly setting these in
sshd_config and using defaults. Why does default cipher list show
chacha20 when it is not supporting it?

Or is it the suggested method from upstream that all downstream
instances should set cipherlist explicitly omitting chacha20 when fips
is enabled?

This list is coming from here:
https://github.com/openssh/openssh-portable/blob/master/myproposal.h#L59

I'm certain that we are not adding any cipher list explicitly.

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3603

--- Comment #3 from Darren Tucker <dtucker at dtucker.net> ---
(In reply to Shreenidhi Shedi from comment #2)
> > Your server is lying about what ciphers it supports
> 
> This is the concern I have here. We are not explicitly setting these
> in sshd_config and using defaults. Why does default cipher list show
> chacha20 when it is not supporting it?
Because your modifications to the server are insufficient.  They should
remove it but don't.

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3603

--- Comment #4 from Shreenidhi Shedi <sshedi at vmware.com> ---
(In reply to Darren Tucker from comment #3)
> (In reply to Shreenidhi Shedi from comment #2)
> > > Your server is lying about what ciphers it supports
> > 
> > This is the concern I have here. We are not explicitly setting these
> > in sshd_config and using defaults. Why does default cipher list show
> > chacha20 when it is not supporting it?
> 
> Because your modifications to the server are insufficient.  They
> should remove it but don't.
Can you please point out what we are missing? here is our spec file.

https://github.com/vmware/photon/blob/5.0/SPECS/openssh/openssh.spec

Thanks a lot for your quick response and inputs. Like I mentioned
earlier, we are not modifying things much; do we need to enable any
config during configure stage?

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3603

--- Comment #5 from Darren Tucker <dtucker at dtucker.net> ---
(In reply to Shreenidhi Shedi from comment #4)
> Can you please point out what we are missing? here is our spec file.
> 
> https://github.com/vmware/photon/blob/5.0/SPECS/openssh/openssh.spec
> 
> Thanks a lot for your quick response and inputs. Like I mentioned
> earlier, we are not modifying things much; do we need to enable any
> config during configure stage?
In comment#1 you said:
"We did something like in PhotonOS 3.0:
https://github.com/vmware/photon/blob/3.0/SPECS/openssh/openssh-7.8p1-fips.patch"

but now you're pointing to a repo whos SPEC file doesn't contain
anything like that patch.  It also specifies an OpenSSH version that
corresponds to *neither* of the versions you mentioned in comment#1.

You are applying a FIPS patch to OpenSSH or are you not?  Are you
building a stock(ish) OpenSSH against a FIPS OpenSSL?   Exactly which
versions are you talking about and exactly which modifications are you
making to it?

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3603

--- Comment #6 from Shreenidhi Shedi <sshedi at vmware.com> ---
Sorry for the confusion.

The patch I mentioned #1 is in Photon 3.0. Ignore the patch related
info I provided, that's something we are doing in 3.0 version only.

Now I'm hitting this issue in Photon 5.0 where we are using 9.3p2.

In this we just use what's given by upstream with mostly default
settings.

In this our cipher list is, exactly this
https://github.com/openssh/openssh-portable/blob/master/myproposal.h#L59

And we are hitting this issue.

You can try this in your setup as well, enable openssl fips in server,
use latest openssh server and try connecting from a client with no
ciphers mentioned.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3603

Damien Miller <djm at mindrot.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         Resolution|---                         |INVALID
             Status|NEW                         |RESOLVED

--- Comment #7 from Damien Miller <djm at mindrot.org> ---
Darren has stated several times what the problem is: your FIPS
modifications are simply broken. Refer to his answer in comment 1.

We're not in a position to fix this because we didn't write the FIPS
patch that you're using. You need to contact its authors if you want
support - we cannot help beyond pointing out how it is broken, which
Darren has already done.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3603

--- Comment #8 from Shreenidhi Shedi <sshedi at vmware.com> ---
Created attachment 3718
  --> https://bugzilla.mindrot.org/attachment.cgi?id=3718&action=edit
sshd-config and sshd_config

Adding sshd config & client log both.

Doing ssh to localhost after enabling fips.

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3603

--- Comment #9 from Shreenidhi Shedi <sshedi at vmware.com> ---
Hi Damien, thanks for your response.

Please ignore the details on the patch I provided.

I'm currently talking about newer openssh with no patch included, check
this spec.

https://github.com/vmware/photon/blob/5.0/SPECS/openssh/openssh.spec

I'm using 9.3p2 here and no patch and don't intend to add it as well.

Now question is, if I take latest openssh and deploy it on both client
and server and enable fips in server, will client be able to
communicate with the server?

I am attaching the log for this scenario but with a slightly lower
version of server and client. PTAL.

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3603

Shreenidhi Shedi <sshedi at vmware.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         Resolution|INVALID                     |---
             Status|RESOLVED                    |REOPENED

--- Comment #10 from Shreenidhi Shedi <sshedi at vmware.com> ---
I have to reopen this. I think the ticket is closed with
misunderstanding.

Please check the logs I attached and try it once in your machine.

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3603

--- Comment #11 from Shreenidhi Shedi <sshedi at vmware.com> ---
Created attachment 3719
  --> https://bugzilla.mindrot.org/attachment.cgi?id=3719&action=edit
client connection log

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3603

--- Comment #12 from Damien Miller <djm at mindrot.org> ---
Created attachment 3720
  --> https://bugzilla.mindrot.org/attachment.cgi?id=3720&action=edit
runtime probing of ciphers
> You can try this in your setup as well, enable openssl fips in server, use 
> latest openssh server and try connecting from a client with no ciphers 
> mentioned.
ok, you're putting OpenSSL in FIPS mode and not patching OpenSSH.
You've then created a situation where the OpenSSL you're using is
behaving differently to the OpenSSL that OpenSSH was compiled with, and
currently OpenSSH is not in a position to detect this.

Changing this basically requires that OpenSSH do runtime probing of all
cryptography to see whether something has changed underneath it.

This patch is an example of how we might approach this. Maybe it helps
your case? It certainly isn't complete - we'd need to do effectively
the same thing for MACs, public key algorithms and key agreement
algorithms too since I bet some of those could be disabled by OpenSSL's
FIPS support too. The patch could probably be simplified if there's a
simpler way to query whether OpenSSL supports a particular algorithm.

In the meantime, if you're changing your crypto library to disable
particular algorithms, then you need to *manually* change your
ssh_config and sshd_config to match.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3603

--- Comment #13 from Shreenidhi Shedi <sshedi at vmware.com> ---
Hi Damien,

Thanks for being patient and understanding the problem.

So if I have understood it right, upstream is intending to fix this
issue and we can expect a fix in the future releases of openssh.

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3603

--- Comment #14 from Damien Miller <djm at mindrot.org> ---
No, I don't think we've decided that yet.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3603

Dmitry Belyavskiy <dbelyavs at redhat.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |dbelyavs at redhat.com

--- Comment #15 from Dmitry Belyavskiy <dbelyavs at redhat.com> ---
In case of OpenSSL 3.0+ EVP_fetch* methods can be used for this
purpose.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3603

--- Comment #16 from Damien Miller <djm at mindrot.org> ---
I don't think that does what we want - that function seems to allow
going from a string name to an EVP but we already have the EVP and just
want to check that it is enabled by OpenSSL without having to arrange
everything EVP_*_init() wants.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3603

Damien Miller <djm at mindrot.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Attachment #3720|0                           |1
        is obsolete|                            |

--- Comment #17 from Damien Miller <djm at mindrot.org> ---
Created attachment 3721
  --> https://bugzilla.mindrot.org/attachment.cgi?id=3721&action=edit
probe ciphers, MACs and KEX

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3603

Roumen Petrov <bugtrack at roumenpetrov.info> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |bugtrack at roumenpetrov.info

--- Comment #18 from Roumen Petrov <bugtrack at roumenpetrov.info> ---
So, world knows hot to detect what is available in FIPS mode.
More then 15 years OpenBSD ream refuse to do simple detection. It is
expected to continue in next decades.

Shreenidhi Shedi, 
Just use PKIX-SSH instead - could be used with OpenSSL, RedHad and
Solaris FIPS validated OpenSSL libraries.

Regards,
Roumen

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3603

--- Comment #19 from Shreenidhi Shedi <sshedi at vmware.com>
---
>Shreenidhi Shedi, 
> Just use PKIX-SSH instead - could be used with OpenSSL, RedHad and Solaris
FIPS validated OpenSSL libraries.
Hi Roumen,

Thanks for the input.
I'm not sure what pkix-ssh is, can you share some more info please?

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3603

--- Comment #20 from Dmitry Belyavskiy <dbelyavs at redhat.com> ---
I see several problems with the proposed patch. It resolves the case
when the run-time and build-time OpenSSL version differs in
capabilities. The problem is it relies on legacy OpenSSL API that
contradicts the initial request (FIPS compatibility). Also EC curve
detection uses the API OpenSSL considers legacy (and so not
FIPS-compliant). And from the FIPS perspective, all NIST curves
supported by OpenSSH are allowed.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.