openssh bugs - Jan 2024 - [Bug 3649] New: Control sockets do not connect anymore (after updating OpenSSH)

https://bugzilla.mindrot.org/show_bug.cgi?id=3649

            Bug ID: 3649
           Summary: Control sockets do not connect anymore (after updating
                    OpenSSH)
           Product: Portable OpenSSH
           Version: 9.6p1
          Hardware: ARM
                OS: Mac OS X
            Status: NEW
          Severity: enhancement
          Priority: P5
         Component: ssh
          Assignee: unassigned-bugs at mindrot.org
          Reporter: wolfgang.liessmann at web.de

After updating OpenSSH, control sockets do not connect anymore:

```
Control socket
connect(/Users/wolfgang/sockets/2024_01_05_151948-c5fbd266): Connection
refused
```

It worked well previously.

Current version:

```
# ssh -V
OpenSSH_9.6p1, OpenSSL 3.2.0 23 Nov 2023

# openssl version
OpenSSL 1.1.1t  7 Feb 2023
```

Installed with homebrew on a current Mac.


Login:

```
ssh -MNf -o ControlPersist=$(SSH_ACCESS_SECONDS) -S
$(SSH_MASTER_SOCKET) host
```

Access examples:

```
scp -o ControlPath=$(SSH_MASTER_SOCKET) -p file
host:/targetdir/targetfilename
ssh -S $(SSH_MASTER_SOCKET) host "cmd1; cmd2"
```


Original post: https://github.com/openssl/openssl/issues/23215

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3649

Damien Miller <djm at mindrot.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |djm at mindrot.org

--- Comment #1 from Damien Miller <djm at mindrot.org> ---
Please attach debug logs (ssh -ddd ...) from both the multiplexing main
and client processes

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3649

--- Comment #2 from wolfgang.liessmann at web.de ---
There is no option '-d', see output below.
Do you mean '-vvv' for increasing the verbosity level?


```
$ ssh -ddd [...]
ssh: illegal option -- d
usage: ssh [-46AaCfGgKkMNnqsTtVvXxYy] [-B bind_interface] [-b
bind_address]
           [-c cipher_spec] [-D [bind_address:]port] [-E log_file]
           [-e escape_char] [-F configfile] [-I pkcs11] [-i
identity_file]
           [-J destination] [-L address] [-l login_name] [-m mac_spec]
           [-O ctl_cmd] [-o option] [-P tag] [-p port] [-R address]
           [-S ctl_path] [-W host:port] [-w local_tun[:remote_tun]]
           destination [command [argument ...]]
       ssh [-Q query_option]


$ ssh -d
ssh: illegal option -- d
usage: ssh [-46AaCfGgKkMNnqsTtVvXxYy] [-B bind_interface] [-b
bind_address]
           [-c cipher_spec] [-D [bind_address:]port] [-E log_file]
           [-e escape_char] [-F configfile] [-I pkcs11] [-i
identity_file]
           [-J destination] [-L address] [-l login_name] [-m mac_spec]
           [-O ctl_cmd] [-o option] [-P tag] [-p port] [-R address]
           [-S ctl_path] [-W host:port] [-w local_tun[:remote_tun]]
           destination [command [argument ...]]
       ssh [-Q query_option]


$ ssh -V      
OpenSSH_9.6p1, OpenSSL 3.2.0 23 Nov 2023
```

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3649

--- Comment #3 from Damien Miller <djm at mindrot.org> ---
yes sorry - "ssh -vvv ..." (-d is for the server)

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3649

--- Comment #4 from wolfgang.liessmann at web.de ---
Created attachment 3785
  --> https://bugzilla.mindrot.org/attachment.cgi?id=3785&action=edit
SSH log (master and client)

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3649

--- Comment #5 from wolfgang.liessmann at web.de ---
Anonymized logs attached (for both master and client).

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3649

--- Comment #6 from Damien Miller <djm at mindrot.org> ---
Comment on attachment 3785
  --> https://bugzilla.mindrot.org/attachment.cgi?id=3785
SSH log (master and client)
> debug1: forking to background
unfortunately logs from the mux master process are lost after this
point. Can you reproduce the problem without -f on its commandline? If
so, please try to catch debug logs from it in that state.

I assume both commands are being run in the same directory? Does "ls -l
socket.tmp" show a socket on the filesystem when you run the client?

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3649

--- Comment #7 from wolfgang.liessmann at web.de ---
Even if I leave away '-f' the master process forks into background.
A file 'socket.tmp' is created (see below), with both processes running
in the same directory.


Master process:
> $ ssh -vvv -MN -o ControlPersist=20 -S socket.tmp host1
> [...]
> debug1: forking to background
> $

Client process:
> $ ls -l socket.tmp
> srw-------  1 wolf  staff  0 18 Jan 01:43 socket.tmp
> $ ssh -vvv -S socket.tmp host1 /some_path/some_script.sh
> [...]
> Control socket connect(socket.tmp): Connection refused
> [We then proceed with manual authentication.]
> [...]
-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3649

--- Comment #8 from Damien Miller <djm at mindrot.org> ---
(In reply to wolfgang.liessmann from comment #7)
> Even if I leave away '-f' the master process forks into background.
> A file 'socket.tmp' is created (see below), with both processes
> running in the same directory.
right but:
> Master process:
> 
> > $ ssh -vvv -MN -o ControlPersist=20 -S socket.tmp host1
> > [...]
> > debug1: forking to background
this still shows a mux master that is forking as this is implied by
ControlPersist - sorry for not mentioning is previously. We need the
logs after this point to figure out what has gone wrong, so if you can
recreate without -f and ControlPersist then we might see better what is
happening.

What version of OpenSSH were you using previously?

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3649

--- Comment #9 from wolfgang.liessmann at web.de ---
Running the master process in the foreground with only 'ssh -vvv -MN -S
socket.tmp host1' (and then the client process in a different terminal)
works well.

But any background execution of the master process (with '-f' and
either '-o ControlPersist=20' or 'ControlPersist=yes') seems to
fail
and the process disappears.
There is no ssh process shown in the output of 'ps -ax' (Mac), and
trying to bring it into the foreground with 'fg' fails ('fg: no
current
job').

I don't know the previous openssh version. But on the server side it
still says 'OpenSSH_9.2p1, OpenSSL 1.1.1s  1 Nov 2022', maybe this one
worked.

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3649

--- Comment #10 from Damien Miller <djm at mindrot.org> ---
Does the main process disappear when using -f but not ControlPersist?

If so, then we can use that to get logs. Something like:


ssh -oControlPersist=no -nNf -vvv -oControlMaster=yes yourhost


will background the main mux process but it will still write debug logs
to the terminal. Hopefully one of these will shed light on what is
happening here.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3649

--- Comment #11 from wolfgang.liessmann at web.de ---
Yes, it's still forking into background, even when starting the master
process with -f and ControlPersist=no:

$ rm -f socket.tmp
$ ssh -vvv -MNf -o ControlPersist=no -S socket.tmp host1
[...]
debug1: forking to background
$ ls -l socket.tmp
[...]
Control socket connect(socket.tmp): Connection refused

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.