openssh bugs - Jan 2024 - [Bug 3649] New: Control sockets do not connect anymore (after updating OpenSSH)

https://bugzilla.mindrot.org/show_bug.cgi?id=3649

            Bug ID: 3649
           Summary: Control sockets do not connect anymore (after updating
                    OpenSSH)
           Product: Portable OpenSSH
           Version: 9.6p1
          Hardware: ARM
                OS: Mac OS X
            Status: NEW
          Severity: enhancement
          Priority: P5
         Component: ssh
          Assignee: unassigned-bugs at mindrot.org
          Reporter: wolfgang.liessmann at web.de

After updating OpenSSH, control sockets do not connect anymore:

```
Control socket
connect(/Users/wolfgang/sockets/2024_01_05_151948-c5fbd266): Connection
refused
```

It worked well previously.

Current version:

```
# ssh -V
OpenSSH_9.6p1, OpenSSL 3.2.0 23 Nov 2023

# openssl version
OpenSSL 1.1.1t  7 Feb 2023
```

Installed with homebrew on a current Mac.


Login:

```
ssh -MNf -o ControlPersist=$(SSH_ACCESS_SECONDS) -S
$(SSH_MASTER_SOCKET) host
```

Access examples:

```
scp -o ControlPath=$(SSH_MASTER_SOCKET) -p file
host:/targetdir/targetfilename
ssh -S $(SSH_MASTER_SOCKET) host "cmd1; cmd2"
```


Original post: https://github.com/openssl/openssl/issues/23215

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3649

Damien Miller <djm at mindrot.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |djm at mindrot.org

--- Comment #1 from Damien Miller <djm at mindrot.org> ---
Please attach debug logs (ssh -ddd ...) from both the multiplexing main
and client processes

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3649

--- Comment #2 from wolfgang.liessmann at web.de ---
There is no option '-d', see output below.
Do you mean '-vvv' for increasing the verbosity level?


```
$ ssh -ddd [...]
ssh: illegal option -- d
usage: ssh [-46AaCfGgKkMNnqsTtVvXxYy] [-B bind_interface] [-b
bind_address]
           [-c cipher_spec] [-D [bind_address:]port] [-E log_file]
           [-e escape_char] [-F configfile] [-I pkcs11] [-i
identity_file]
           [-J destination] [-L address] [-l login_name] [-m mac_spec]
           [-O ctl_cmd] [-o option] [-P tag] [-p port] [-R address]
           [-S ctl_path] [-W host:port] [-w local_tun[:remote_tun]]
           destination [command [argument ...]]
       ssh [-Q query_option]


$ ssh -d
ssh: illegal option -- d
usage: ssh [-46AaCfGgKkMNnqsTtVvXxYy] [-B bind_interface] [-b
bind_address]
           [-c cipher_spec] [-D [bind_address:]port] [-E log_file]
           [-e escape_char] [-F configfile] [-I pkcs11] [-i
identity_file]
           [-J destination] [-L address] [-l login_name] [-m mac_spec]
           [-O ctl_cmd] [-o option] [-P tag] [-p port] [-R address]
           [-S ctl_path] [-W host:port] [-w local_tun[:remote_tun]]
           destination [command [argument ...]]
       ssh [-Q query_option]


$ ssh -V      
OpenSSH_9.6p1, OpenSSL 3.2.0 23 Nov 2023
```

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3649

--- Comment #3 from Damien Miller <djm at mindrot.org> ---
yes sorry - "ssh -vvv ..." (-d is for the server)

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3649

--- Comment #4 from wolfgang.liessmann at web.de ---
Created attachment 3785
  --> https://bugzilla.mindrot.org/attachment.cgi?id=3785&action=edit
SSH log (master and client)

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3649

--- Comment #5 from wolfgang.liessmann at web.de ---
Anonymized logs attached (for both master and client).

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3649

--- Comment #6 from Damien Miller <djm at mindrot.org> ---
Comment on attachment 3785
  --> https://bugzilla.mindrot.org/attachment.cgi?id=3785
SSH log (master and client)
> debug1: forking to background
unfortunately logs from the mux master process are lost after this
point. Can you reproduce the problem without -f on its commandline? If
so, please try to catch debug logs from it in that state.

I assume both commands are being run in the same directory? Does "ls -l
socket.tmp" show a socket on the filesystem when you run the client?

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3649

--- Comment #7 from wolfgang.liessmann at web.de ---
Even if I leave away '-f' the master process forks into background.
A file 'socket.tmp' is created (see below), with both processes running
in the same directory.


Master process:
> $ ssh -vvv -MN -o ControlPersist=20 -S socket.tmp host1
> [...]
> debug1: forking to background
> $

Client process:
> $ ls -l socket.tmp
> srw-------  1 wolf  staff  0 18 Jan 01:43 socket.tmp
> $ ssh -vvv -S socket.tmp host1 /some_path/some_script.sh
> [...]
> Control socket connect(socket.tmp): Connection refused
> [We then proceed with manual authentication.]
> [...]
-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3649

--- Comment #8 from Damien Miller <djm at mindrot.org> ---
(In reply to wolfgang.liessmann from comment #7)
> Even if I leave away '-f' the master process forks into background.
> A file 'socket.tmp' is created (see below), with both processes
> running in the same directory.
right but:
> Master process:
> 
> > $ ssh -vvv -MN -o ControlPersist=20 -S socket.tmp host1
> > [...]
> > debug1: forking to background
this still shows a mux master that is forking as this is implied by
ControlPersist - sorry for not mentioning is previously. We need the
logs after this point to figure out what has gone wrong, so if you can
recreate without -f and ControlPersist then we might see better what is
happening.

What version of OpenSSH were you using previously?

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3649

--- Comment #9 from wolfgang.liessmann at web.de ---
Running the master process in the foreground with only 'ssh -vvv -MN -S
socket.tmp host1' (and then the client process in a different terminal)
works well.

But any background execution of the master process (with '-f' and
either '-o ControlPersist=20' or 'ControlPersist=yes') seems to
fail
and the process disappears.
There is no ssh process shown in the output of 'ps -ax' (Mac), and
trying to bring it into the foreground with 'fg' fails ('fg: no
current
job').

I don't know the previous openssh version. But on the server side it
still says 'OpenSSH_9.2p1, OpenSSL 1.1.1s  1 Nov 2022', maybe this one
worked.

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3649

--- Comment #10 from Damien Miller <djm at mindrot.org> ---
Does the main process disappear when using -f but not ControlPersist?

If so, then we can use that to get logs. Something like:


ssh -oControlPersist=no -nNf -vvv -oControlMaster=yes yourhost


will background the main mux process but it will still write debug logs
to the terminal. Hopefully one of these will shed light on what is
happening here.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3649

--- Comment #11 from wolfgang.liessmann at web.de ---
Yes, it's still forking into background, even when starting the master
process with -f and ControlPersist=no:

$ rm -f socket.tmp
$ ssh -vvv -MNf -o ControlPersist=no -S socket.tmp host1
[...]
debug1: forking to background
$ ls -l socket.tmp
[...]
Control socket connect(socket.tmp): Connection refused

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3649

--- Comment #12 from wolfgang.liessmann at web.de ---
I had a look at the issue again and was able to track it down to the
OpenSSH client, not the sshd service at the server side.

When using the SSH client shipped with current macOS, it works, but the
SSH client shipped with Homebrew fails.
So the problem is caused by the client, not the server.

The server sshd log contains:
2024-09-06 22:14:37.[...]+[...] [...]    sshd: Read error from remote
host 192.168.1.[...] port [...]: Connection reset by peer

As a preliminary solution, I will stick with the SSH client shipped
with macOS.


Step to reproduce


Create Makefile:

#SSH := /usr/bin/ssh # Shipped with macOS
SSH := /opt/homebrew/bin/ssh # Shipped with Homebrew
SSH_MASTER_SOCKET := ~/.ssh/socket
SSH_ACCESS_SECONDS := 20

test_ssh:
$(SSH) -MNf -o ControlPersist=$(SSH_ACCESS_SECONDS) -S
$(SSH_MASTER_SOCKET) root at server
$(SSH) -S $(SSH_MASTER_SOCKET) root at server ls


Run:

# make test_ssh


Error output at client side:

Control socket connect(/Users/wolfgang/.ssh/socket): Connection refused


Server sshd log:

2024-09-06 22:14:37.[...]+[...] [...]    sshd: Read error from remote
host 192.168.1.[...] port [...]: Connection reset by peer

(Internet search on this error message didn't produce any helpful
results.)


To view the sshd log on the server (also a Mac), the following steps
were necessary:

1. Adding the following line to /etc/ssh/sshd_config:
LogLevel DEBUG3

2. Restarting the sshd service:
sudo launchctl stop system/com.openssh.sshd
sudo launchctl start system/com.openssh.sshd

3. Accessing the logs as described at:
https://stackoverflow.com/a/54148031


Enabling/disabling the firewalls produces no different result.


Versions (all software updated):

# /usr/bin/ssh -V
OpenSSH_9.7p1, LibreSSL 3.3.6
# /opt/homebrew/bin/ssh -V
OpenSSH_9.8p1, OpenSSL 3.3.1 4 Jun 2024

# sw_vers
ProductName:            macOS
ProductVersion:         14.6.1
BuildVersion:           23G93

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3649

--- Comment #13 from Damien Miller <djm at mindrot.org> ---
Can you run the multiplexing process without -f? At the moment the logs
are lost after it forks.

Could you try specifying an absolute path to the socket in both cases?
Even -S $PWD/socket.tmp could work.

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3649

--- Comment #14 from wolfgang.liessmann at web.de ---
Removing '-f' makes no difference, as reported earlier at
https://bugzilla.mindrot.org/show_bug.cgi?id=3649#c7 (I have tested
again).

Master process:
> debug1: forking to background
The client process:
> Control socket connect(/Users/wolfgang/.ssh/socket): Connection refused

Neither absolute path nor current working directory work, in either
combination (with and without '-f').

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.