openssh bugs - Sep 2024 - [Bug 3729] New: the new sshd does not work under the supervision of inetd

https://bugzilla.mindrot.org/show_bug.cgi?id=3729

            Bug ID: 3729
           Summary: the new sshd does not work under the supervision of
                    inetd
           Product: Portable OpenSSH
           Version: 9.8p1
          Hardware: ARM
                OS: Linux
            Status: NEW
          Severity: major
          Priority: P5
         Component: sshd
          Assignee: unassigned-bugs at mindrot.org
          Reporter: khalid.outaali at nav-timing.safrangroup.com

**Bug Description:**

When attempting to connect via SSH, I consistently receive the
following error message:

```
kex_exchange_identification: read: Connection reset by peer
```

This issue occurs **only when running `sshd` in inetd mode using the
`-i` option** (either under `xinetd` or similar super-server
configurations). However, when running the `sshd` service independently
(via `sshd.service` under systemd or manually), connections are
established successfully without errors.

---

**Steps to Reproduce:**

1. Configure `sshd` to run in inetd mode using the `-i` option (e.g.,
in `xinetd.conf` or triggered via systemd socket activation).
2. Attempt to connect to the SSH server using an SSH client.
3. Observe that the connection fails with the error
`kex_exchange_identification: read: Connection reset by peer`.

---

**Expected Behavior:**

SSH connections should be successfully established in inetd mode, just
as they are when `sshd` is run independently without the `-i` option.

---

**Actual Behavior:**

SSH connections fail with the error:

```
kex_exchange_identification: read: Connection reset by peer
```

This only happens when `sshd` is run in inetd mode using the `-i`
option. Running the `sshd` service outside of inetd (e.g.,
systemd-based `sshd.service`) works as expected without any connection
issues.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3729

Darren Tucker <dtucker at dtucker.net> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |dtucker at dtucker.net

--- Comment #1 from Darren Tucker <dtucker at dtucker.net> ---
It's probably not inetd-mode in general since a lot of our tests use
that (they use ssh to invoke sshd -i as a proxycommand) and these are
regularly invoked hundreds of time across many configurations,
including some ARM and ARM64.

Given this report is for ARM, I suspect this may be some variation on
the seccomp filter issues (eg bug#3702) which are dependent on kernel,
glibc and header versions present at compile time.  Can you describe
the host system some more?  Distro, kernel, glibc version, hardware?

Can you attach the logs from the server during a failed connection? 
Either from syslog or use sshd -E to write the log someplace.

If the problem is related to the seccomp sandbox, one other thing that
might shed some light is to build with sandbox debugging enabled (but
note that this is not safe for production use):

./configure --with-cflags=-DSANDBOX_SECCOMP_FILTER_DEBUG

then build, install and generate a new log.

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3729

--- Comment #2 from khalid outaali <khalid.outaali at
nav-timing.safrangroup.com> ---
Thanks for your feedback, Darren. Here are the details you requested:

**Host System Details:**

- **Distro:** None, custom build with buildroot
- **Kernel Version:** Linux 6.1.32 #1 SMP PREEMPT
- **glibc Version:** GNU C Library (Buildroot) stable release version
2.38, Compiled by GNU CC version 12.3.0.
- **Hardware Architecture:** armv7l (Altera Cyclone V)

**Server Logs:**

I?ve attached the logs from the xined and sshd during a failed
connection. You can find the relevant logs below:

```
spfactory at securesync-0e0000:~$ sudo /usr/sbin/xinetd -dontfork -d &
[1] 3195
spfactory at securesync-0e0000:~$ 24/9/10 at 07:54:57: DEBUG: 3198
{handle_includedir} Reading included configuration file:
/etc/xinetd.d/daytime [file=/etc/xinetd.conf] [line=15]
24/9/10 at 07:54:57: DEBUG: 3198 {handle_includedir} Reading included
configuration file: /etc/xinetd.d/ssh [file=/etc/xinetd.d/ssh]
[line=27]
24/9/10 at 07:54:57: DEBUG: 3198 {handle_includedir} Reading included
configuration file: /etc/xinetd.d/telnet [file=/etc/xinetd.d/telnet]
[line=20]
24/9/10 at 07:54:57: DEBUG: 3198 {handle_includedir} Reading included
configuration file: /etc/xinetd.d/time [file=/etc/xinetd.d/time]
[line=17]
24/9/10 at 07:54:57: DEBUG: 3198 {remove_disabled_services} removing
daytime
24/9/10 at 07:54:57: DEBUG: 3198 {remove_disabled_services} removing
daytime
24/9/10 at 07:54:57: DEBUG: 3198 {remove_disabled_services} removing
telnet
24/9/10 at 07:54:57: DEBUG: 3198 {remove_disabled_services} removing time
24/9/10 at 07:54:57: DEBUG: 3198 {remove_disabled_services} removing time
Service defaults
        Instances = 60
        CPS = max conn:25 wait:30
        Bind = All addresses.
        Only from:  0.0.0.0(NUMERIC) ::/::(NET)
        No access: No blocked sites
        Logging to syslog. Facility = authpriv, level = info
        Log_on_success flags = HOST PID
        Log_on_failure flags = HOST

Service configuration: ssh
        id = ssh
        flags = REUSE IPv6
        socket_type = stream
        Protocol (name,number) = (tcp,6)
        port = 22
        Instances = 9
        wait = no
        user = 0
        Groups = yes
        PER_SOURCE = -1
        Bind = All addresses.
        Server = /usr/sbin/sshd
        Server argv = sshd -i -e -ddd
        Only from:  0.0.0.0(NUMERIC) ::/::(NET)
        No access: No blocked sites
        Logging to file: /tmp/sshd.log (no limits)
        Log_on_success flags = HOST DURATION EXIT PID
        Log_on_failure flags = HOST USERID

24/9/10 at 07:54:57: DEBUG: 3198 {cnf_start_services} Started service: ssh
24/9/10 at 07:54:57: DEBUG: 3198 {cnf_start_services} mask_max = 6,
services_started = 1
24/9/10 at 07:54:57: NOTICE: 3198 {main} xinetd Version 2.3.15 started
with no options compiled in.
24/9/10 at 07:54:57: NOTICE: 3198 {main} Started working: 1 available
service
24/9/10 at 07:54:57: DEBUG: 3198 {main_loop} active_services = 1

spfactory at securesync-0e0000:~$ ssh spfactory at localhost
24/9/10 at 07:55:16: DEBUG: 3198 {main_loop} select returned 1
24/9/10 at 07:55:16: DEBUG: 3198 {server_start} Starting service ssh
24/9/10 at 07:55:16: DEBUG: 3198 {main_loop} active_services = 1
24/9/10 at 07:55:16: DEBUG: 3299 {exec_server} duping 8
kex_exchange_identification: read: Connection reset by peer
Connection reset by 127.0.0.1 port 22
24/9/10 at 07:55:16: DEBUG: 3198 {main_loop} active_services = 1
24/9/10 at 07:55:16: DEBUG: 3198 {main_loop} select returned 1
24/9/10 at 07:55:16: DEBUG: 3198 {check_pipe} Got signal 17 (Child exited)
24/9/10 at 07:55:16: DEBUG: 3198 {child_exit} waitpid returned = 3299
24/9/10 at 07:55:16: DEBUG: 3198 {server_end} ssh server 3299 exited
24/9/10 at 07:55:16: INFO: 3198 {conn_free} freeing connection
24/9/10 at 07:55:16: DEBUG: 3198 {child_exit} waitpid returned = -1
24/9/10 at 07:55:16: DEBUG: 3198 {main_loop} active_services = 1
spfactory at securesync-0e0000:~$ cat /tmp/sshd.log 
24/9/10 at 07:51:21: START: ssh pid=1993 from=::ffff:127.0.0.1
24/9/10 at 07:51:21: EXIT: ssh status=1 pid=1993 duration=0(sec)
24/9/10 at 07:55:16: START: ssh pid=3299 from=::ffff:127.0.0.1
24/9/10 at 07:55:16: EXIT: ssh status=255 pid=3299 duration=0(sec)
```

---

I'll also try rebuilding `sshd` with sandbox debugging enabled to
gather more information. Once built with:

```
./configure --with-cflags=-DSANDBOX_SECCOMP_FILTER_DEBUG
```

with this nothing has changed.

I doubted the issue has something to do with the absolute paths, or the
re-exec from sshd-session. So I was testing and I found out that this
small change somehow fixes the issue (as a workaround):

```
khalid at work:~/openssh-portable$ gdiff
diff --git a/misc.c b/misc.c
index afdf5142e..a29981586 100644
--- a/misc.c
+++ b/misc.c
@@ -2558,7 +2558,7 @@ parse_pattern_interval(const char *s, char
**typep, int *secsp)
 int
 path_absolute(const char *path)
 {
-       return (*path == '/') ? 1 : 0;
+       return 1;
 }

 void
```

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3729

--- Comment #3 from Darren Tucker <dtucker at dtucker.net> ---
Unfortunately none of those logs are the sshd debug logs.  They should
look like this:

debug2: load_server_config: filename /etc/ssh/sshd_config
debug2: load_server_config: done config len = 2395
[...]
debug1: rexec_argv[1]='-ddd'
debug1: rexec_argv[2]='-p'
debug3: using /usr/libexec/sshd-session for re-exec
debug2: fd 7 setting O_NONBLOCK
debug1: Bind to port 2022 on 0.0.0.0.
Server listening on 0.0.0.0 port 2022.
debug2: fd 8 setting O_NONBLOCK
debug1: Bind to port 2022 on ::.
Server listening on :: port 2022.
>Server argv = sshd -i -e -ddd
Changing -e to -E/path/to/logfile should get that.
> ./configure --with-cflags=-DSANDBOX_SECCOMP_FILTER_DEBUG
> with this nothing has changed.
The extra messages we want will the last entries in the sshd debug
logs.

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3729

--- Comment #4 from khalid outaali <khalid.outaali at
nav-timing.safrangroup.com> ---
Yes, I tried but somehow whenever `-i` is present in argv in the
context of xinetd, sshd does not log anything, I tried with:
* `/sbin/sshd -i -e -ddd`
* `/sbin/sshd -i -ddd -E /tmp/sshd.log`
* '/sbin/sshd -D -e -o LogLevel=debug`
* `/bin/sh -c "/sbin/sshd -i -ddd -e >> /var/log/sshd.log
2>&1"`

but all those command outside of xinetd log to stdout or file:

```
spfactory at securesync-0e0000:~$ sudo /usr/sbin/sshd -i -d -E
/tmp/sshd.log 
SSH-2.0-OpenSSH_9.8
^C
spfactory at securesync-0e0000:~$ sudo cat /tmp/sshd.log 
debug1: sshd version OpenSSH_9.8, OpenSSL 3.0.13+SL_CCS-2.1.1 30 Jan
2024
debug1: private host key #0: ssh-rsa
SHA256:/GfNXD7wb4t4vv1ZOA2sHgjjQR3slQ/5X2ikgr+dhDE
debug1: private host key #1: ecdsa-sha2-nistp256
SHA256:LDTO37zRXVJG8ixaoq8PfTrjccLiUxAmDzNAbZ/HXUU
debug1: private host key #2: ssh-ed25519
SHA256:oZb1dPHiGlCN9Y27KDM5/F9yEKe2xd5+41vF6glHcaw
debug1: rexec_argv[1]='-i'
debug1: rexec_argv[2]='-d'
debug1: rexec_argv[3]='-E'
debug1: rexec_argv[4]='/tmp/sshd.log'
debug1: rexec start in -1 out -1 newsock -1 pipe -1 sock 8/9
debug1: sshd version OpenSSH_9.8, OpenSSL 3.0.13+SL_CCS-2.1.1 30 Jan
2024
debug1: network sockets: 5, 6
Connection from UNKNOWN port 65535 on UNKNOWN port 65535
debug1: Local version string SSH-2.0-OpenSSH_9.8

```

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3729

Damien Miller <djm at mindrot.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |djm at mindrot.org
             Status|NEW                         |RESOLVED
         Resolution|---                         |DUPLICATE

--- Comment #5 from Damien Miller <djm at mindrot.org> ---
Could this be bug 3717?b

*** This bug has been marked as a duplicate of bug 3717 ***

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3729

Damien Miller <djm at mindrot.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         Resolution|DUPLICATE                   |---
             Status|RESOLVED                    |REOPENED

--- Comment #6 from Damien Miller <djm at mindrot.org> ---
oops, I didn't mean to make it as duplicate; just ask the question

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3729

--- Comment #7 from Darren Tucker <dtucker at dtucker.net>
---
> * `/sbin/sshd -i -ddd -E /tmp/sshd.log`
-d will put the server into debug mode which implies logging to stderr.
 -e does the same thing.  I think you want:


/sbin/sshd -i -oLogLevel=debug3 -E /tmp/sshd.log

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3729

--- Comment #8 from Darren Tucker <dtucker at dtucker.net>
---
> Could this be bug 3717?
oh, yeah that could be it.  If it is, specifying a full path in the
argv portion of the (x)inetd config might help, ie:

        Server = /usr/sbin/sshd
        Server argv = /usr/sbin/sshd -i -e -ddd

(but you probably don't want the -ddd for normal operation).

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.