openssh bugs - Jun 2023 - [Bug 3581] New: ssh-keyscan fails with `fdlim_get: bad value` with large file descriptor limit due to type confusion

https://bugzilla.mindrot.org/show_bug.cgi?id=3581

            Bug ID: 3581
           Summary: ssh-keyscan fails with `fdlim_get: bad value` with
                    large file descriptor limit due to type confusion
           Product: Portable OpenSSH
           Version: 9.3p1
          Hardware: ARM64
                OS: Mac OS X
            Status: NEW
          Severity: enhancement
          Priority: P5
         Component: ssh-keyscan
          Assignee: unassigned-bugs at mindrot.org
          Reporter: janerik at fnordig.de

ssh-keyscan fails with an obscure `fdlim_get: bad value` error when the
values returned from `sysconf(_SC_OPEN_MAX)` does not fit into the 32
bits of an `int`.

This happens on my M1 MacBook (with the system-provided `ssh-keyscan`)
where I somehow have an `unlimited` file descriptor limit (which really
is `INT64_MAX` or `9223372036854775807`, note that this seems
non-standard).

Changing the limit to something below `INT_MAX` works.
And then `ssh-keyscan` picks `maxfd=256` anyway.

See this shell session:

```
$ ulimit -n
unlimited
$ ssh-keyscan -t ed25519 github.com
ssh-keyscan: fdlim_get: bad value
$ ulimit -n 4294967295
$ ssh-keyscan -t ed25519 github.com
ssh-keyscan: fdlim_get: bad value
$ ulimit -n 2147483648
$ ssh-keyscan -t ed25519 github.com
ssh-keyscan: fdlim_get: bad value
$ ulimit -n 2147483647
$ ssh-keyscan -t ed25519 github.com
# github.com:22 SSH-2.0-babeld-6732d282
github.com ssh-ed25519
AAAAC3NzaC1lZDI1NTE5AAAAIOMqqnkVzrm0SdG6UOoqKLsabgH5C9okWi0dh2l9GKJl
```

Looking at the code of openssh-portable here[1] shows that `fdlim_get`
returns an `int`, but the fields of `struct rlimit` and the return
value of `sysconf`[2] are `long`, thus 64-bit.

Given that large values like on my machine probably are not set too
often this probably won't affect many people. And a simple workaround
exists.
So maybe this is a wontfix, but I figured I'd still report it and let
you decide.

[1]:
https://github.com/openssh/openssh-portable/blob/b4ac435b4e67f8eb5932d8f59eb5b3cf7dc38df0/ssh-keyscan.c#L129-L130
[2]: `SSH_SYSFDMAX` is defined as `sysconf(_SC_OPEN_MAX)`

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3581

Damien Miller <djm at mindrot.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |djm at mindrot.org,
                   |                            |dtucker at dtucker.net
   Attachment #3702|                            |ok?(dtucker at dtucker.net)
              Flags|                            |

--- Comment #1 from Damien Miller <djm at mindrot.org> ---
Created attachment 3702
  --> https://bugzilla.mindrot.org/attachment.cgi?id=3702&action=edit
avoid values >INT_MAX

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3581

Darren Tucker <dtucker at dtucker.net> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Attachment #3702|ok?(dtucker at dtucker.net)    |ok+
              Flags|                            |

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3581

Damien Miller <djm at mindrot.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|---                         |FIXED
             Blocks|                            |3162

--- Comment #2 from Damien Miller <djm at mindrot.org> ---
thanks for the report - this has been committed and will be in OpenSSH
8.4, due in a few months.


Referenced Bugs:

https://bugzilla.mindrot.org/show_bug.cgi?id=3162
[Bug 3162] Tracking bug for 8.4 release
-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3581

jan-erik <janerik at fnordig.de> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         Resolution|FIXED                       |---
             Status|RESOLVED                    |REOPENED

--- Comment #3 from jan-erik <janerik at fnordig.de> ---
That patch does not fix the issue though, because it still returns
`SSH_SYSFDMAX`, which is the huge value that wraps over to -1.

And `rlfd.rlim_max == RLIM_INFINITY` was already true, because they are
the same value regardless of what type it wraps to.

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3581

Darren Tucker <dtucker at dtucker.net> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Blocks|3162                        |3549


Referenced Bugs:

https://bugzilla.mindrot.org/show_bug.cgi?id=3162
[Bug 3162] Tracking bug for 8.4 release
https://bugzilla.mindrot.org/show_bug.cgi?id=3549
[Bug 3549] Tracking bug for OpenSSH 9.4
-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3581

Damien Miller <djm at mindrot.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Attachment #3703|                            |ok?(dtucker at dtucker.net)
              Flags|                            |

--- Comment #4 from Damien Miller <djm at mindrot.org> ---
Created attachment 3703
  --> https://bugzilla.mindrot.org/attachment.cgi?id=3703&action=edit
handle overlong sysconf() returns too

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3581

Darren Tucker <dtucker at dtucker.net> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Attachment #3703|ok?(dtucker at dtucker.net)    |ok+
              Flags|                            |

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3581

Damien Miller <djm at mindrot.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         Resolution|---                         |FIXED
             Status|REOPENED                    |RESOLVED

--- Comment #5 from Damien Miller <djm at mindrot.org> ---
commit cfca6f17e64 should fix the sysconf(SC_OPEN_MAX) case too

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.