Hello, I'm tasked with establishing a persistent SSH connection across a very unreliable link, for a remote port forward (always port 2217). I figured I'd use ServerAliveInterval to make sure that the ssh(1) process dies when the connection appears down, and I use systemd to restart it in this case. This works fine. What does not work fine, however, is the server-side. If the connection goes down, sshd(8) lingers on the remote, bound to the port that I need to forward back across the link. As a result, new connections can't bind the port. The sshd(8) process stays around for an unspecified time (>90min) after the connection went down. I realise I could use ClientAliveInterval on the server-side, but there seems to be no way to specify this per-connection, and there simply won't be a server-wide policy change in this case. The ssh_config(5) manpage *does* state: The server alive mechanism is valuable when the client or server depend on knowing when a connection has become inactive. but this promise currently isn't kept, because the server actually doesn't find out about the "server alive" mechanism. Would it be too far-fetched for ssh(1) to communicate to sshd(8) on the remote that it intends to send pings and that the connection will be torn down if those pings remain unanswered? Then the sshd(8) process would know and could react accordingly. Or is there another way to do this per-process/-connection? Short of running a separate sshd(8) on another port? It's probably a big change, as it'd require the protocol to be touched, but maybe provisions for such extensions already exist? Do you have any thoughts, gauges of success of such a wishlist, or short-term solutions? Thanks, -- @martinkrafft | http://madduck.net/ | http://two.sentenc.es/ who's general failure, and why's he reading my disk? spamtraps: madduck.bogus at madduck.net -------------- next part -------------- A non-text attachment was scrubbed... Name: digital_signature_gpg.asc Type: application/pgp-signature Size: 1118 bytes Desc: Digital GPG signature (see http://martin-krafft.net/gpg/sig-policy/999bbcc4/current) URL: <http://lists.mindrot.org/pipermail/openssh-unix-dev/attachments/20171115/4097d571/attachment.asc>
On Wed, Nov 15, 2017 at 8:08 AM, martin f krafft <madduck at madduck.net> wrote:> Hello, > > I'm tasked with establishing a persistent SSH connection across > a very unreliable link, for a remote port forward (always port > 2217). I figured I'd use ServerAliveInterval to make sure that the > ssh(1) process dies when the connection appears down, and I use > systemd to restart it in this case. This works fine.I use "autossh" for precisely this reason, partly because I've found it very effective at re-establishing the link, partly for the logging, and partly for the daemonization of what can be used to keep an SSH tunnel alive consistently.
also sprach martin f krafft <madduck at madduck.net> [2017-11-15 14:08 +0100]:> The sshd(8) process stays around for an unspecified time (>90min) > after the connection went down.For the record, I've attached strace and tcpdump to the various processes to see what happens server-side after the connection goes down, but there is *nothing*: 1. strace on the privileged sshd process has it sitting at restart_syscall(<? resuming interrupted poll ?>) = 1 until at some indefinite point, the unprivileged child dies, at which point it tears down after catching SIGCHLD. 2. strace on the unprivileged sshd process, the one that bound the remote port just stays at select(10, [3 6 8 9], [], NULL, NULL? indefinitely. Those FDs are (3) main SSH connection, (6) a pipe, (8) IPv6 listening socket for port 2217, (9) IPv4 listening socket for port 2217. 3. tcpdump on the main SSH connection has not seen a single packet in the last 20 minutes. Curiously, netstat/lsof seem to think that the connection is ESTABLISHED. This isn't good, but maybe it's also not a problem of SSH, but the OS/kernel (4.9.0). Best regards, -- @martinkrafft | http://madduck.net/ | http://two.sentenc.es/ "in just seven days, i can make you a man!" -- the rocky horror picture show spamtraps: madduck.bogus at madduck.net -------------- next part -------------- A non-text attachment was scrubbed... Name: digital_signature_gpg.asc Type: application/pgp-signature Size: 1118 bytes Desc: Digital GPG signature (see http://martin-krafft.net/gpg/sig-policy/999bbcc4/current) URL: <http://lists.mindrot.org/pipermail/openssh-unix-dev/attachments/20171115/32e31873/attachment.asc>
also sprach Nico Kadel-Garcia <nkadel at gmail.com> [2017-11-15 14:42 +0100]:> > I'm tasked with establishing a persistent SSH connection across > > a very unreliable link, for a remote port forward (always port > > 2217). I figured I'd use ServerAliveInterval to make sure that the > > ssh(1) process dies when the connection appears down, and I use > > systemd to restart it in this case. This works fine. > > I use "autossh" for precisely this reason, partly because I've found > it very effective at re-establishing the link, partly for the logging, > and partly for the daemonization of what can be used to keep an SSH > tunnel alive consistently.Yes, autossh is effectively the same as ServerAliveInterval + systemd. I've used it for many many years, but with SAI and systemd, it's become obsolete I think. However, I can't see how autossh would solve the lingering sshd(8) process problem on the server. -- @martinkrafft | http://madduck.net/ | http://two.sentenc.es/ "i worked myself up from nothing to a state of extreme poverty." -- groucho marx spamtraps: madduck.bogus at madduck.net -------------- next part -------------- A non-text attachment was scrubbed... Name: digital_signature_gpg.asc Type: application/pgp-signature Size: 1118 bytes Desc: Digital GPG signature (see http://martin-krafft.net/gpg/sig-policy/999bbcc4/current) URL: <http://lists.mindrot.org/pipermail/openssh-unix-dev/attachments/20171115/fc9ba588/attachment.asc>
On 2017-11-15 at 14:08 +0100, martin f krafft wrote:> What does not work fine, however, is the server-side. If the > connection goes down, sshd(8) lingers on the remote, bound to the > port that I need to forward back across the link. As a result, new > connections can't bind the port. > > The sshd(8) process stays around for an unspecified time (>90min) > after the connection went down. > > I realise I could use ClientAliveInterval on the server-side, but > there seems to be no way to specify this per-connection, and > there simply won't be a server-wide policy change in this case.So, instead of running no command server side, would a suitable workaround be to run 'while sleep 1; do echo .; done' as the server command and discard stdout from ssh on the client side? That way, the server side should detect the dropped link sooner, leading to sshd exit and port release. -Phil
On 11/15/2017 6:08 AM, martin f krafft wrote:> I'm tasked with establishing a persistent SSH connection across > a very unreliable link, for a remote port forward (always port > 2217). I figured I'd use ServerAliveInterval to make sure that the > ssh(1) process dies when the connection appears down, and I use > systemd to restart it in this case. This works fine.Have you looked at using mosh? It's very good with unreliable connections. -Sam George
also sprach Phil Pennock <phil.pennock at globnix.org> [2017-11-15 19:41 +0100]:> So, instead of running no command server side, would a suitable > workaround be to run 'while sleep 1; do echo .; done' as the server > command and discard stdout from ssh on the client side? That way, the > server side should detect the dropped link sooner, leading to sshd exit > and port release.First, we need to debate to death whether it should be while sleep 1; do echo .; done or while echo .; do sleep 1; done ;) Nevermind! What an excellent idea, it's close to client-specified ClientAliveInterval! I can't exactly quantify the effects yet, but a first test (I used `cat /etc/fstab` instead of `echo .` just to fill the pipe a bit faster) shows that the sshd(8) process (a) still sticks around longer than it should but (b) nowhere near as long as it does when using the ssh(1) -N option instead (i.e. run no command on the remote). Thank you! -- @martinkrafft | http://madduck.net/ | http://two.sentenc.es/ #include <signature.h> spamtraps: madduck.bogus at madduck.net -------------- next part -------------- A non-text attachment was scrubbed... Name: digital_signature_gpg.asc Type: application/pgp-signature Size: 1118 bytes Desc: Digital GPG signature (see http://martin-krafft.net/gpg/sig-policy/999bbcc4/current) URL: <http://lists.mindrot.org/pipermail/openssh-unix-dev/attachments/20171115/c52cdcdc/attachment-0001.asc>
also sprach Sam George <ciradrak at centurylink.net> [2017-11-15 22:37 +0100]:> Have you looked at using mosh? It's very good with unreliable > connections.mosh can't do port forwardings. For interactive sessions, I wholeheartedly agree with you though. ? mosh. -- @martinkrafft | http://madduck.net/ | http://two.sentenc.es/ "'oh, that was easy,' says Man, and for an encore goes on to prove that black is white and gets himself killed on the next zebra crossing." -- douglas adams, "the hitchhiker's guide to the galaxy" spamtraps: madduck.bogus at madduck.net -------------- next part -------------- A non-text attachment was scrubbed... Name: digital_signature_gpg.asc Type: application/pgp-signature Size: 1118 bytes Desc: Digital GPG signature (see http://martin-krafft.net/gpg/sig-policy/999bbcc4/current) URL: <http://lists.mindrot.org/pipermail/openssh-unix-dev/attachments/20171115/19cfacb1/attachment.asc>