openssh bugs - Nov 2020 - [Bug 3233] New: Error loading key ".ssh/id_rsa": error in libcrypto

https://bugzilla.mindrot.org/show_bug.cgi?id=3233

            Bug ID: 3233
           Summary: Error loading key ".ssh/id_rsa": error in
libcrypto
           Product: Portable OpenSSH
           Version: 8.4p1
          Hardware: Other
                OS: Windows 10
            Status: NEW
          Severity: enhancement
          Priority: P5
         Component: ssh-add
          Assignee: unassigned-bugs at mindrot.org
          Reporter: tpgxyz at gmail.com

Hi,

i have an issue with latest openssh-8.4p1.
My RSA ssh key which is quite old and can't be handled with ssh tools.

[tpg at tpg-virtualbox .ssh]$ ssh-add
Error loading key "/home/tpg/.ssh/id_rsa": error in libcrypto

At first i thought culprit may be OpenSSL3 so i filled issue upstream
https://github.com/openssl/openssl/issues/13443, during the discussion
it looks like it may be on OpenSSH side. I was trying to track some
potential commits but did not found anything relevant.

Ofcourse ssh-keygen -t rsa works an with that new key ssh-add also
works.

My system information:
Distribution: OpenMandriva Cooker
kernel-5.9.8
glibc-2.32
libxcrypt-4.4.17
LLVM-11.x
By default in this distro everything is compiled with LLVM/clang and
linke with LLVM/lld

[tpg at tpg-virtualbox .ssh]$ rpm -qa | grep openssl
openssl-3.0.0-0.alpha8.1.x86_64
[tpg at tpg-virtualbox .ssh]$ rpm -qa | grep openssh
openssh-8.4p1-2.x86_64

RPM Sources:
https://github.com/OpenMandrivaAssociation/openssh
https://github.com/OpenMandrivaAssociation/openssl

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3233

Tomasz Pawe? Gajc <tpgxyz at gmail.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 OS|Windows 10                  |Linux
           Hardware|Other                       |ix86
           Severity|enhancement                 |normal

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3233

Damien Miller <djm at mindrot.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |djm at mindrot.org

--- Comment #1 from Damien Miller <djm at mindrot.org>
---
>From the github issue it looks like your key is in the OpenSSH format.
If you have an older OpenSSH around, then could you try converting your
key to the OpenSSL PEM format and see if that works?

cp id_rsa id_rsa_pem
ssh-keygen -p -m PEM -N '' -f id_rsa_pem

You'll also be able to check the key using OpenSSL's tools, e.g.
"openssl rsa -check -noout -in id_rsa_pem"

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3233

--- Comment #2 from Tomasz Pawe? Gajc <tpgxyz at gmail.com> ---
(In reply to Damien Miller from comment #1)
> From the github issue it looks like your key is in the OpenSSH
> format.
> 
> If you have an older OpenSSH around, then could you try converting
> your key to the OpenSSL PEM format and see if that works?
> 
Yes, my ssh key works with older openssh/openssl


[live at localhost .ssh]$ ssh-keygen -p -m PEM -N ''  -f id_rsa_pem 
Enter old passphrase: 
Your identification has been saved with the new passphrase.
[live at localhost .ssh]$ openssl rsa -check -noout -in id
id_rsa      id_rsa_pem  id_rsa.pub  
[live at localhost .ssh]$ openssl rsa -check -noout -in id_rsa_pem 
RSA key ok

[live at localhost .ssh]$ rpm -qa | grep openssl
openssl-1.1.1c-1.x86_64
lib64qca2-plugin-openssl-2.1.4-2.x86_64
[live at localhost .ssh]$ rpm -qa | grep openssh
openssh-8.1p1-1.x86_64
openssh-server-8.1p1-1.x86_64
openssh-askpass-common-8.1p1-1.x86_64
openssh-clients-8.1p1-1.x86_64

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3233

--- Comment #3 from Damien Miller <djm at mindrot.org> ---
Created attachment 3457
  --> https://bugzilla.mindrot.org/attachment.cgi?id=3457&action=edit
print libcrypto errors in ssh-add

okay, can you load the PEM key on the openssl-3.x host?

Also, you could try building OpenSSH from git with the attached patch
applied to see why openssl-3.x is refusing the key.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3233

--- Comment #4 from Tomasz Pawe? Gajc <tpgxyz at gmail.com> ---
Sorry for long delay in my response.

I've applied your patch, recompiled opnssh, installed it and i executed
ssh-add on my old key, and here is the output:

[root at tpg-virtualbox home]# ssh-add -vvv
Error loading key "/root/.ssh/id_rsa": error in libcrypt

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3233

Damien Miller <djm at mindrot.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Attachment #3457|0                           |1
        is obsolete|                            |

--- Comment #5 from Damien Miller <djm at mindrot.org> ---
Created attachment 3469
  --> https://bugzilla.mindrot.org/attachment.cgi?id=3469&action=edit
more debugging

(In reply to Tomasz Pawe? Gajc from comment #4)
> Sorry for long delay in my response.
No problem - thanks for helping debug this problem!
> I've applied your patch, recompiled opnssh, installed it and i
> executed ssh-add on my old key, and here is the output:
> 
> [root at tpg-virtualbox home]# ssh-add -vvv
> Error loading key "/root/.ssh/id_rsa": error in libcrypt
I don't understand how the patch has failed to print more information.
I have attached a patch that should show a little more - can you please
try that?

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3233

--- Comment #6 from Tomasz Pawe? Gajc <tpgxyz at gmail.com> ---
Hi, i did apply your second patch
(https://github.com/OpenMandrivaAssociation/openssh/blob/errors/openssh-8.4p1-catch-openssl-errors.patch),
here is the output:

[root at tpg-virtualbox home]# ssh-add -vvv
306: Error loading key "/root/.ssh/id_rsa": error in libcrypto


Here are the build logs
https://abf.openmandriva.org/build_lists/981796

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3233

--- Comment #7 from Tomasz Pawe? Gajc <tpgxyz at gmail.com> ---
Output of a strace ssh-add -vvv :

newfstatat(AT_FDCWD, "/foo.ssh/id_rsa", {st_mode=S_IFREG|0600,
st_size=3326, ...}, 0) = 0
openat(AT_FDCWD, "/foo/.ssh/id_rsa", O_RDONLY) = 4
newfstatat(4, "", {st_mode=S_IFREG|0600, st_size=3326, ...},
AT_EMPTY_PATH) = 0
getuid()                                = 0
newfstatat(4, "", {st_mode=S_IFREG|0600, st_size=3326, ...},
AT_EMPTY_PATH) = 0
read(4, "-----BEGIN RSA PRIVATE KEY-----\n"..., 4096) = 3326
read(4, "", 770)                        = 0
read(4, "", 4096)                       = 0
close(4)                                = 0
futex(0x7f29f67cf20c, FUTEX_WAKE_PRIVATE, 2147483647) = 0
futex(0x7f29f67cde50, FUTEX_WAKE_PRIVATE, 2147483647) = 0
futex(0x7f29f67cde58, FUTEX_WAKE_PRIVATE, 2147483647) = 0
write(2, "306: Error loading key \"/foo/.s"..., 63306: Error
loading
key "/foo/.ssh/id_rsa": error in libcrypto
) = 63

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3233

--- Comment #8 from Tomasz Pawe? Gajc <tpgxyz at gmail.com> ---
I am more and more convinced this is openssl3 internal issue with
reading keys.

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3233

gl041188 at gmail.com changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |gl041188 at gmail.com
            Summary|Error loading key           |Error loading key
                   |".ssh/id_rsa": error in    
|".ssh/guanlong_huang_rsa":
                   |libcrypto                   |error in libcrypto

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3233

Tomasz Pawe? Gajc <tpgxyz at gmail.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|---                         |FIXED

--- Comment #9 from Tomasz Pawe? Gajc <tpgxyz at gmail.com> ---
Issue has been fixed with openssl-3.0.0-alpha16

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3233

Darren Tucker <dtucker at dtucker.net> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
            Summary|Error loading key           |Error loading key
                   |".ssh/guanlong_huang_rsa": 
|".ssh/id_rsa": error in
                   |error in libcrypto          |libcrypto
                 CC|                            |dtucker at dtucker.net

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3233

Damien Miller <djm at mindrot.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|RESOLVED                    |CLOSED

--- Comment #10 from Damien Miller <djm at mindrot.org> ---
closing bugs resolved before openssh-8.9

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.