openssh unix dev - Jul 2018 - SSH Agent Forwarding Not Working

Hi,

I know this might be the most asked question, so I've done anything
possible to troubleshoot the problem myself, but still, my SSH Agent
Forwarding is not working for me.

The best troubleshooting guide that I found, and also the one I've
been using, is the ssh forwarding guide on github -
https://help.github.com/articles/using-ssh-agent-forwarding

I've checked all things there, yet it is just still not working. E.g.,

echo "$SSH_AUTH_SOCK" shows correctly on the server and my local
machine, and I have checked both .ssh/config and /etc/ssh_config and
/etc/ssh/sshd_config on the server (machine B) - they're all set up
correctly.

However, the forwarding doesn't seem to be actually working - "ssh -T
git at github.com" works locally, but not if I run it on the server
(machine B). `ssh-add -l` prints fine locally, but prints "The agent
has no identities" on the server.

I've tried openssh-server with both Ubuntu & Debian,

$ apt-cache policy openssh-server
openssh-server:
  Installed: 1:7.6p1-4
  Candidate: 1:7.6p1-4
  Version table:
 *** 1:7.6p1-4 500
        500 http://archive.ubuntu.com/ubuntu bionic/main amd64 Packages

$ apt-cache policy openssh-server
openssh-server:
  Installed: 1:7.4p1-10+deb9u3
  Candidate: 1:7.4p1-10+deb9u3
  Version table:
     1:7.7p1-2 50
         50 http://cdn-fastly.deb.debian.org/debian unstable/main amd64 Packages
 *** 1:7.4p1-10+deb9u3 500

and both are not working.

In the end, this is what I get to ssh from A to C:

$ ssh -T git at gitlab.com
Welcome to GitLab, @myid!

this is what I get to ssh from A to B, then B to C, with my local ~/.ssh folder

$ ssh -T git at gitee.com
Enter passphrase for key ...

the full `ssh -vvvT git at gitlab.com` is available at
https://pastebin.com/did1EsPe (via Debian as B, without my local ~/.ssh folder)
or,
https://superuser.com/questions/840340/ssh-agent-forwarding-not-working-even-when-using-ssh-a
(via Ubuntu as B)

How to troubleshoot the problem from here?
Please help. thx.

> However, the forwarding doesn't seem to be actually working - "ssh
-T
> git at github.com" works locally, but not if I run it on the server
> (machine B). `ssh-add -l` prints fine locally, but prints "The agent
> has no identities" on the server.
it sounds like something on B is running a new ssh-agent process when
you log in.

when you ssh to B, is your $SSH_AUTH_SOCK owned by 'sshd' or
'ssh-agent'? if you've forwarded your agent, it should be owned by
sshd on the remote host.

this is good:

$ sudo lsof -n | grep $SSH_AUTH_SOCK
sshd       71286            pmoody   10u     unix 0xffff8802058a9400
     0t0  741054208 /tmp/ssh-9bmDCfYasv/agent.71286

vs:

$ eval `ssh-agent`
Agent pid 72317
$ sudo lsof -n | grep $SSH_AUTH_SOCK
ssh-agent  72317            pmoody    3u     unix 0xffff880031322800
     0t0  741071876 /tmp/ssh-tc6wDsKNfgFi/agent.72316

On Wed, Jul 11, 2018 at 11:53 PM Peter Moody wrote:
>
> > However, the forwarding doesn't seem to be actually working -
"ssh -T
> > git at github.com" works locally, but not if I run it on the
server
> > (machine B). `ssh-add -l` prints fine locally, but prints "The
agent
> > has no identities" on the server.
>
> it sounds like something on B is running a new ssh-agent process when
> you log in.
>
> when you ssh to B, is your $SSH_AUTH_SOCK owned by 'sshd' or
> 'ssh-agent'? if you've forwarded your agent, it should be owned
by
> sshd on the remote host.
BINGO!!!

OMG! I've been trying to troubleshoot this problem for nearly **7**
years, and it now get solved just like that!

I launch keychain in my ~/.profile, which starts its own 'ssh-agent'.
Guess I'll remove it and start it manually only from machine A from
now on.

Thanks again Peter, I can't express how appreciate I am enough!