openssh unix dev - Jun 2022 - Problems using RemoteForward for gpg-agent with multiple sessions

Hello!

Many of my colleages at work are running into an issue with the
RemoteForward feature on unix domain sockets. I believe eliminating the
specific problem would require OpenSSH Portable source code changes, and
I'm hoping to start a discussion on what sort of patch would be
acceptable before submitting one.

We're using the RemoteForward config as suggested by the gnupg.org wiki
to forward the gpg-agent process's unix socket.

https://wiki.gnupg.org/AgentForwarding#OpenSSH_.3E.3D_6.7

Following the above, ~/.ssh/config file typically looks like this:

 Host example
   RemoteForward /run/user/1000/gnupg/S.gpg-agent
/home/local/.gnupg/S.gpg-agent.extra

## Problem Details

This works well, but intermittently the remote forward is unexpectedly
destroyed. We've narrowed down the problem to the RemoteForward config
not interacting well with multiple SSH clients. Specifically, each
subsequent client attempts to initiate a new RemoteForward, destroying
any previous forwards when doing so. When the most recently connected SSH
client disconnects, the forward is left in an unbound state for all other
existing clients. The typical workflow here is a single user opening
multiple terminal tabs and running "ssh example". A simple
reproduction
would be:

 1. Run "ssh example" in terminal tab A.
 2. Leaving tab A open, run "ssh example" in terminal tab B.
 3. Close terminal tab B.
 4. Switch back to tab A.
 5. Observe that S.gpg-agent is now a dead link.

In retrospect, this is probably obvious behavior and expected. For
RemoteForward sockets to stay alive and avoid multiple connections
fighting over managing it, clients would have to be aware of each other.
Such a design would add significant complexity. (We're not using
ControlMaster due to security concerns.)

I think there's a few alternatives that might be reasonable
based on how the ForwardAgent feature works.

## Possibility 1: New RemoteForward syntax

As opposed to gpg-agent forwarding, ssh-agent forwarding does not have
this problem. I suspect avoiding this problem was actually an intentional
part of ssh-agent forwarding's design. Instead of a hard-coded file
system path, the ForwardAgent config instructs the server to create a new
socket bind in a temporary location unique to that SSH client connection.
The SSH_AUTH_SOCK environment variable is initialized to this temporary
path so processes in the remote host know how to communicate with the
forwarded ssh-agent.

 $ echo "$SSH_AUTH_SOCK"
 /tmp/ssh-KbsgWY3jcN/agent.340671

My first proposal would be to extend the RemoteForward ssh_config to
allow similar behavior.

 Host
   RemoteForward env:GPG_AGENT_SOCK /home/local/.gnupg/S.gpg-agent.extra

In this proposal, the local path would be forwarded to a random path on
the remote and set in $GPG_AGENT_SOCK.

 $ echo "$GPG_AGENT_SOCK"
 /tmp/ssh-UIzrILLqkZ/sock.340851

This design enhances unix socket forwarding in general and is therefore
agnostic to gpg-agent, but to complete the example workflow we'd then set
the S.gpg-agent file to:

 $ cat /run/user/1000/gnupg/S.gpg-agent
 %Assuan%
 socket=${GPG_AGENT_SOCK}

Note: The gpg command has socket redirection builtin through this special
%Assuan% directive and is documented in the assuan_sock_set_sockaddr_un
%function.

For security, the server config should set AcceptEnv.

 AcceptEnv GPG_AGENT_SOCK

## Possiblity 2: New ForwardGpgAgent config

Another option is to build gpg-agent forwarding directly in OpenSSH.

 Host
   ForwardGpgAgent /home/local/.gnupg/S.gpg-agent.extra

I'm not particularly excited about this option since it's unnecessarily
specific, but seems to follow the pattern of the exiting ForwardAgent and
ForwardX11 ssh_config options.

## Possibility 3: Allow client-side TCP port to be used in bind path

Looking over the % token expansions available to RemoteForward, none of
the existing tokens would allow a different remote path to be used for
different SSH client connections. Since the overall problem is a
hard-coded static path, anything dynamic would fix the problem.

Experimenting a bit, I was able to add a percent encoding for the
client-side TCP port (%d in the example below), which would be specific
to each connection.

 Host
   RemoteForward /tmp/%d-gpg.sock /home/local/.gnupg/S.gpg-agent.extra

This was a clever workaround I got working, but it turns out the client
TCP port is undefined when using the ProxyCommand option. So this
unfortunately wouldn't solve the problem for my team at work.

---

I'd appreciate any and all thoughts. I mentioned a few things that I
think are obvious to OpenSSH developers, but elaborated regardless so
anyone could contribute thoughts on the problem.

Thanks everyone,

 Brandon

On Tue, 7 Jun 2022, Brandon Cheng wrote:
> not interacting well with multiple SSH clients. Specifically, each
> subsequent client attempts to initiate a new RemoteForward, destroying
This is the same for TCP port forwarding, except the next attempts
say they cannot bind.

Canonical solution for that is to use a muxmaster, which then does
the forwards, and subsequent connections just pick up the mux.

bye,
//mirabilos
-- 
Infrastrukturexperte ? tarent solutions GmbH
Am Dickobskreuz 10, D-53121 Bonn ? http://www.tarent.de/
Telephon +49 228 54881-393 ? Fax: +49 228 54881-235
HRB AG Bonn 5168 ? USt-ID (VAT): DE122264941
Gesch?ftsf?hrer: Dr. Stefan Barth, Kai Ebenrett, Boris Esser, Alexander Steeg

                        ****************************************************
/?\ The UTF-8 Ribbon
??? Campaign against      Mit dem tarent-Newsletter nichts mehr verpassen:
??? HTML eMail! Also,     https://www.tarent.de/newsletter
??? header encryption!
                        ****************************************************

On Tue, 7 Jun 2022, Brandon Cheng wrote:
> This works well, but intermittently the remote forward is unexpectedly
> destroyed. We've narrowed down the problem to the RemoteForward config
> not interacting well with multiple SSH clients. Specifically, each
> subsequent client attempts to initiate a new RemoteForward, destroying
> any previous forwards when doing so. When the most recently connected SSH
> client disconnects, the forward is left in an unbound state for all other
> existing clients.
[...]
> ## Possibility 1: New RemoteForward syntax
[...]
> ## Possiblity 2: New ForwardGpgAgent config
[...]
> ## Possibility 3: Allow client-side TCP port to be used in bind path
Another possibility would be to have some %-expansion that expands to
a random value that is long enough to be safely used as a temporary
path.

E.g. %R expanding to 24 base64 characters. You could use this to obtain
effectively unique paths.

-d