The attached patch adds an option (off by default to preserve current behavior) to set a timeout on the select() statement that waits for input in clientloop.c. This fixes a timeout issue for me (explained below) and probably also fixes the timeouts mentioned in last month's thread "Idle time out". The patch is also available by http from: http://www.chaos2.org/~jacob/code/patch-openssh-1.2.2-trans_inter I am ssh-ing from a machine on my home network to one on the internet. This goes out over a Linux ip_masquerade firewall. When I wrote the attached patch, I thought it was the firewall that was killing the connection by timing out on the redirected port due to lack of traffic. But after reading some similar posts on this list, I think there might be problems even if a firewall isn't involved. Also note that in the tcpdump below, I did have KeepAlive turned on (both server and client) and yet I don't see any traffic being generated due to this, which seems to render KeepAlive pretty useless... When ssh dies on me (when no max idle time is set) it gives me the error below: " velius:~% Read from remote host velius.chaos2.org: Connection reset by peer Connection to velius.chaos2.org closed. jacob:~# " From the tcpdump below, we see that the firewall has assigned a new ip_masq port. This shows all the packets; specifically, none are generated in the interim. " 00:59:19.987703 velius.chaos2.org.ssh > c392100-a.crvlls1.or.home.com.64579: P 1:21(20) ack 20 win 32120 <nop,nop,timestamp 46926353 47417028> (DF) 00:59:19.998389 c392100-a.crvlls1.or.home.com.64579 > velius.chaos2.org.ssh: . ack 21 win 32120 <nop,nop,timestamp 47417072 46926353> (DF) [tos 0x10] ... time passes here but no traffic to velius ... 01:20:37.477884 c392100-a.crvlls1.or.home.com.64687 > velius.chaos2.org.ssh: P 2954940853:2954940873(20) ack 2970631452 win 32120 <nop,nop,timestamp 47544804 46926353> (DF) [tos 0x10] 01:20:37.583097 velius.chaos2.org.ssh > c392100-a.crvlls1.or.home.com.64687: R 2970631452:2970631452(0) win 0 [tos 0x10] " The attached patch allows the user to put a TransmitInterlude option in their ssh_config file that gives how many seconds are allowed to pass without generating traffic. A value of 300 completely solves the timeouts for me and I haven't observed any stability issues. Please cc me with comments as I am not subscribed to the list. Jacob Lundberg jacob at chaos2.org -- "Heh. You mean this is Stef's source code?" -User Friendly -------------- next part -------------- diff -ur openssh-1.2.2/clientloop.c openssh-1.2.2-trans_inter/clientloop.c --- openssh-1.2.2/clientloop.c Mon Dec 6 20:38:32 1999 +++ openssh-1.2.2-trans_inter/clientloop.c Fri Mar 3 11:21:12 2000 @@ -396,8 +396,10 @@ */ void -client_wait_until_can_do_something(fd_set * readset, fd_set * writeset) +client_wait_until_can_do_something(fd_set * readset, fd_set * writeset, int trans_inter) { + int select_return; + /* Initialize select masks. */ FD_ZERO(readset); @@ -436,15 +438,32 @@ max_fd = channel_max_fd(); /* - * Wait for something to happen. This will suspend the process until - * some selected descriptor can be read, written, or has some other - * event pending. Note: if you want to implement SSH_MSG_IGNORE - * messages to fool traffic analysis, this might be the place to do - * it: just have a random timeout for the select, and send a random - * SSH_MSG_IGNORE packet when the timeout expires. + * Wait for something to happen. This will suspend the process + * until some selected descriptor can be read, written, or has some + * other event pending. + * Implemented timeout SSH_MSG_NONE packets to keep a minimum + * frequency of traffic present on a connection. This can be used to + * prevent a firewall (ip_masq f.e.) from timing out and causing a new + * port to be allocated which effectively kills the connection. + * To fool traffic analysis, use SSH_MSG_IGNORE packets and set + * the timeout randomly. Fill the packets with some random traffic. + * But NOTE that this packet type seems to cause some ssh servers to + * close the connection when it arrives and they are expecting data. */ - if (select(max_fd + 1, readset, writeset, NULL, NULL) < 0) { + if( trans_inter > 0 ) { + struct timeval timeout; + timeout.tv_sec = trans_inter; + timeout.tv_usec = 0; + select_return = select(max_fd + 1, readset, writeset, NULL, &timeout); + if(select_return == 0) { + packet_start(SSH_MSG_NONE); + packet_send(); + } + } else + select_return = select(max_fd + 1, readset, writeset, NULL, NULL); + + if( select_return < 0 ) { char buf[100]; /* Some systems fail to clear these automatically. */ FD_ZERO(readset); @@ -863,7 +882,7 @@ * Wait until we have something to do (something becomes * available on one of the descriptors). */ - client_wait_until_can_do_something(&readset, &writeset); + client_wait_until_can_do_something(&readset, &writeset, options.trans_inter); if (quit_pending) break; diff -ur openssh-1.2.2/readconf.c openssh-1.2.2-trans_inter/readconf.c --- openssh-1.2.2/readconf.c Sun Dec 5 16:47:29 1999 +++ openssh-1.2.2-trans_inter/readconf.c Fri Mar 3 11:21:12 2000 @@ -78,6 +78,7 @@ UseRsh no StrictHostKeyChecking yes KeepAlives no + TransmitInterlude 0 IdentityFile ~/.ssh/identity Port 22 EscapeChar ~ @@ -101,8 +102,8 @@ oUser, oHost, oEscapeChar, oRhostsRSAAuthentication, oProxyCommand, oGlobalKnownHostsFile, oUserKnownHostsFile, oConnectionAttempts, oBatchMode, oCheckHostIP, oStrictHostKeyChecking, oCompression, - oCompressionLevel, oKeepAlives, oNumberOfPasswordPrompts, oTISAuthentication, - oUsePrivilegedPort, oLogLevel + oCompressionLevel, oKeepAlives, oTransmitInterlude, oNumberOfPasswordPrompts, + oTISAuthentication, oUsePrivilegedPort, oLogLevel } OpCodes; /* Textual representations of the tokens. */ @@ -148,6 +149,7 @@ { "compression", oCompression }, { "compressionlevel", oCompressionLevel }, { "keepalive", oKeepAlives }, + { "transmitinterlude", oTransmitInterlude }, { "numberofpasswordprompts", oNumberOfPasswordPrompts }, { "tisauthentication", oTISAuthentication }, { "loglevel", oLogLevel }, @@ -355,6 +357,10 @@ intptr = &options->keepalives; goto parse_flag; + case oTransmitInterlude: + intptr = &options->trans_inter; + goto parse_int; + case oNumberOfPasswordPrompts: intptr = &options->number_of_password_prompts; goto parse_int; @@ -610,6 +616,7 @@ options->strict_host_key_checking = -1; options->compression = -1; options->keepalives = -1; + options->trans_inter = -1; options->compression_level = -1; options->port = -1; options->connection_attempts = -1; @@ -677,6 +684,8 @@ options->compression = 0; if (options->keepalives == -1) options->keepalives = 1; + if (options->trans_inter == -1) + options->trans_inter = 0; if (options->compression_level == -1) options->compression_level = 6; if (options->port == -1) diff -ur openssh-1.2.2/readconf.h openssh-1.2.2-trans_inter/readconf.h --- openssh-1.2.2/readconf.h Sun Dec 5 16:47:29 1999 +++ openssh-1.2.2-trans_inter/readconf.h Fri Mar 3 11:21:12 2000 @@ -56,6 +56,7 @@ int compression_level; /* Compression level 1 (fast) to 9 * (best). */ int keepalives; /* Set SO_KEEPALIVE. */ + int trans_inter; /* Guarantee transmit every n seconds. */ LogLevel log_level; /* Level for logging. */ int port; /* Port to connect. */ diff -ur openssh-1.2.2/ssh.0 openssh-1.2.2-trans_inter/ssh.0 --- openssh-1.2.2/ssh.0 Wed Jan 26 19:17:09 2000 +++ openssh-1.2.2-trans_inter/ssh.0 Fri Mar 3 11:21:12 2000 @@ -486,6 +486,19 @@ be verified automatically in either case. The argument must be ``yes'' or ``no''. + TransmitInterlude + Specifies a maximum time to allow between transmitting packets, + in seconds. If this amount of time passes and the client has no + data to send, it will send an empty packet to the server. One + example where this is useful is when using ssh from behind a Lin- + ux ip_masquerade firewall. If packets aren't sent through such a + firewall periodically, the firewall may forget about the connec- + tion. Then when a packet finally is sent, the firewall will as- + sign a new port, which will cause the remote server to disconnect + the session. This option defaults to ``0'', which means not + sending periodic packets. A setting of a few hundred seconds + should be about right if this is needed. + UsePrivilegedPort Specifies whether to use a privileged port for outgoing connec- tions. The argument must be ``yes'' or ``no''. The default is diff -ur openssh-1.2.2/ssh.1 openssh-1.2.2-trans_inter/ssh.1 --- openssh-1.2.2/ssh.1 Sat Jan 22 00:57:40 2000 +++ openssh-1.2.2-trans_inter/ssh.1 Fri Mar 3 11:21:12 2000 @@ -720,6 +720,19 @@ .Dq yes or .Dq no . +.It Cm TransmitInterlude +Specifies a maximum time to allow between transmitting packets, +in seconds. If this amount of time passes and the client has +no data to send, it will send an empty packet to the server. +One example where this is useful is when using ssh from behind +a Linux ip_masquerade firewall. If packets aren't sent through +such a firewall periodically, the firewall may forget about the +connection. Then when a packet finally is sent, the firewall +will assign a new port, which will cause the remote server to +disconnect the session. This option defaults to +.Dq 0 , +which means not sending periodic packets. A setting of a few +hundred seconds should be about right if this is needed. .It Cm UsePrivilegedPort Specifies whether to use a privileged port for outgoing connections. The argument must be
Hi, I have applied this patch, and it works pretty well. Well, I agree with you about the KeepAlive thing, it seems not functioning, at least on Linux machine. I just glaced at your patch, seems you send a "NOP" every TransInterclude time. While this solves the problem while using the openssh client, but people who use different ssh client may be unlucky. Maybe the best choice is to implement it on server side?? Anyway, my problem is gone and thank you very much! On Fri, 3 Mar 2000, Jacob Luna Lundberg wrote:> > The attached patch adds an option (off by default to preserve current > behavior) to set a timeout on the select() statement that waits for input > in clientloop.c. This fixes a timeout issue for me (explained below) and > probably also fixes the timeouts mentioned in last month's thread "Idle > time out". The patch is also available by http from: > http://www.chaos2.org/~jacob/code/patch-openssh-1.2.2-trans_inter > > I am ssh-ing from a machine on my home network to one on the > internet. This goes out over a Linux ip_masquerade firewall. When I > wrote the attached patch, I thought it was the firewall that was killing > the connection by timing out on the redirected port due to lack of > traffic. But after reading some similar posts on this list, I think there > might be problems even if a firewall isn't involved. Also note that in > the tcpdump below, I did have KeepAlive turned on (both server and client) > and yet I don't see any traffic being generated due to this, which seems > to render KeepAlive pretty useless... > > When ssh dies on me (when no max idle time is set) it gives me the > error below: > > " > velius:~% Read from remote host velius.chaos2.org: Connection reset by peer > Connection to velius.chaos2.org closed. > jacob:~# > " > > From the tcpdump below, we see that the firewall has assigned a new > ip_masq port. This shows all the packets; specifically, none are > generated in the interim. > > " > 00:59:19.987703 velius.chaos2.org.ssh > c392100-a.crvlls1.or.home.com.64579: P 1:21(20) ack 20 win 32120 > <nop,nop,timestamp 46926353 47417028> (DF) > 00:59:19.998389 c392100-a.crvlls1.or.home.com.64579 > velius.chaos2.org.ssh: . ack 21 win 32120 > <nop,nop,timestamp 47417072 46926353> (DF) [tos 0x10] > ... time passes here but no traffic to velius ... > 01:20:37.477884 c392100-a.crvlls1.or.home.com.64687 > velius.chaos2.org.ssh: P 2954940853:2954940873(20) ack > 2970631452 win 32120 <nop,nop,timestamp 47544804 46926353> (DF) [tos 0x10] > 01:20:37.583097 velius.chaos2.org.ssh > c392100-a.crvlls1.or.home.com.64687: R 2970631452:2970631452(0) win 0 > [tos 0x10] > " > > The attached patch allows the user to put a TransmitInterlude option > in their ssh_config file that gives how many seconds are allowed to pass > without generating traffic. A value of 300 completely solves the timeouts > for me and I haven't observed any stability issues. > > Please cc me with comments as I am not subscribed to the list. > > Jacob Lundberg > jacob at chaos2.org > > -- > > "Heh. You mean this is Stef's source code?" > -User Friendly >-- "My grandpa told me to remember two things in life. Look out for Number One, and remember your number" - Orville Cogswell -- http://members.xoom.com/_XOOM/dizhao/index.html
On Fri, 3 Mar 2000, Jacob Luna Lundberg wrote:> The attached patch allows the user to put a TransmitInterlude > option in their ssh_config file that gives how many seconds are > allowed to pass without generating traffic. A value of 300 > completely solves the timeouts for me and I haven't observed any > stability issues.I would first rather get to the bottom of figuring out why keepalives aren't working. Is "KeepAlive yes" set for both client and server? Is /proc/sys/net/ipv4/tcp_keepalive_time set to less than the masquerading timeouts? -d -- | "Bombay is 250ms from New York in the new world order" - Alan Cox | Damien Miller - http://www.mindrot.org/ | Email: djm at mindrot.org (home) -or- djm at ibs.com.au (work)
On Fri, 3 Mar 2000 13:33:40 -0800 (PST), you wrote:> When ssh dies on me (when no max idle time is set) it gives me the >error below: > >" >velius:~% Read from remote host velius.chaos2.org: Connection reset by peer >Connection to velius.chaos2.org closed. >jacob:~# >"I see the very same problem in my setup. Do I have to apply that patch to both client and server? Greetings Marc -- -------------------------------------- !! No courtesy copies, please !! ----- Marc Haber | " Questions are the | Mailadresse im Header Karlsruhe, Germany | Beginning of Wisdom " | Fon: *49 721 966 32 15 Nordisch by Nature | Lt. Worf, TNG "Rightful Heir" | Fax: *49 721 966 31 29
> I see the very same problem in my setup. Do I have to apply that patch > to both client and server?It only patches the client, so you can get away with just putting it on your client (which won't help people ssh-ing _from_ the server). If you're running Linux on the client and can become root, you could also try (with KeepAlive yes set on both server and client): /bin/echo "300\c" > /proc/sys/net/ipv4/tcp_keepalive_time> !! No courtesy copies, please !!Fair enough, but please do cc me; I don't usually read the list. -Jacob -- "Heh. You mean this is Stef's source code?" -User Friendly
the patch looks reasonable, but SSH_MSG_NONE type packets must not travel over the wire. this violates the protocol spec. On Fri, Mar 03, 2000 at 01:33:40PM -0800, Jacob Luna Lundberg wrote:> + packet_start(SSH_MSG_NONE); > + packet_send();SSH_MSG_IGNORE should be used, e.g.: packet_start(SSH_MSG_IGNORE); packet_put_string("bla", 3); packet_send(); -markus
Marc Haber <openssh-unix-dev.mindrot.org at marc-haber.de> wrote:> >If you can get root on the box that houses the server, try changing > >_its_ keepalive timeout to 300 seconds. That might do it for you. > > Which config setting is that? sshd_config does only seem to have > keepalive =3D yes./proc/sys/net/ipv4/tcp_keepalive_time Which defaults to 7200 on recent Linux kernel. So try: /bin/echo "300\c" > /proc/sys/net/ipv4/tcp_keepalive_time Which will give you a 5 min timeout instead of 2 hours...> And I still don't understand why this problem only shows when the TCP > connection is NATed.A good question. The ssh connection seems to expect the keepalives more frequently than every two hours. So without keepalives (or other packets) happening however frequently it is ssh wants them to, it terminates the connection (or perhaps linux reclaims the socket, I don't know). -Jacob -- "Heh. You mean this is Stef's source code?" -User Friendly
Updated my last patch to 1.2.2p1. http://www.chaos2.org/~jacob/code/patch-openssh-1.2.2p1-trans_inter-r2 I presume ;) that you guys will let me know if there is a preferred way to generate a random string rather than using the function that just moved into random.c... -Jacob -- "Heh. You mean this is Stef's source code?" -User Friendly -------------- next part -------------- diff -ur openssh-1.2.2p1/clientloop.c openssh-1.2.2p1-trans_inter-r2/clientloop.c --- openssh-1.2.2p1/clientloop.c Mon Dec 6 20:38:32 1999 +++ openssh-1.2.2p1-trans_inter-r2/clientloop.c Tue Mar 7 13:23:29 2000 @@ -23,6 +23,7 @@ #include "buffer.h" #include "authfd.h" #include "readconf.h" +#include "random.h" /* Flag indicating that stdin should be redirected from /dev/null. */ extern int stdin_null_flag; @@ -396,8 +397,10 @@ */ void -client_wait_until_can_do_something(fd_set * readset, fd_set * writeset) +client_wait_until_can_do_something(fd_set * readset, fd_set * writeset, int trans_inter) { + int select_return; + /* Initialize select masks. */ FD_ZERO(readset); @@ -436,15 +439,35 @@ max_fd = channel_max_fd(); /* - * Wait for something to happen. This will suspend the process until - * some selected descriptor can be read, written, or has some other - * event pending. Note: if you want to implement SSH_MSG_IGNORE - * messages to fool traffic analysis, this might be the place to do - * it: just have a random timeout for the select, and send a random - * SSH_MSG_IGNORE packet when the timeout expires. + * Wait for something to happen. This will suspend the process + * until some selected descriptor can be read, written, or has some + * other event pending. + * Implemented timeout SSH_MSG_IGNORE packets to keep a minimum + * frequency of traffic present on a connection. This can be used to + * prevent a firewall (ip_masq f.e.) from timing out and causing a new + * port to be allocated which effectively kills the connection. + * To fool traffic analysis, set the timeout on the SSH_MSG_IGNORE + * packets randomly instead of periodically. */ - if (select(max_fd + 1, readset, writeset, NULL, NULL) < 0) { + if( trans_inter > 0 ) { + struct timeval timeout; + timeout.tv_sec = trans_inter; + timeout.tv_usec = 0; + select_return = select(max_fd + 1, readset, writeset, NULL, &timeout); + if(select_return == 0) { + int random_length = arc4random() & 0xff; + char *random_string = (char *)xmalloc(random_length); + get_random_bytes(random_string, random_length); + packet_start(SSH_MSG_IGNORE); + packet_put_string(random_string, random_length); + packet_send(); + xfree(random_string); + } + } else + select_return = select(max_fd + 1, readset, writeset, NULL, NULL); + + if( select_return < 0 ) { char buf[100]; /* Some systems fail to clear these automatically. */ FD_ZERO(readset); @@ -863,7 +886,7 @@ * Wait until we have something to do (something becomes * available on one of the descriptors). */ - client_wait_until_can_do_something(&readset, &writeset); + client_wait_until_can_do_something(&readset, &writeset, options.trans_inter); if (quit_pending) break; diff -ur openssh-1.2.2p1/readconf.c openssh-1.2.2p1-trans_inter-r2/readconf.c --- openssh-1.2.2p1/readconf.c Sun Dec 5 16:47:29 1999 +++ openssh-1.2.2p1-trans_inter-r2/readconf.c Tue Mar 7 13:19:29 2000 @@ -78,6 +78,7 @@ UseRsh no StrictHostKeyChecking yes KeepAlives no + TransmitInterlude 0 IdentityFile ~/.ssh/identity Port 22 EscapeChar ~ @@ -101,8 +102,8 @@ oUser, oHost, oEscapeChar, oRhostsRSAAuthentication, oProxyCommand, oGlobalKnownHostsFile, oUserKnownHostsFile, oConnectionAttempts, oBatchMode, oCheckHostIP, oStrictHostKeyChecking, oCompression, - oCompressionLevel, oKeepAlives, oNumberOfPasswordPrompts, oTISAuthentication, - oUsePrivilegedPort, oLogLevel + oCompressionLevel, oKeepAlives, oTransmitInterlude, oNumberOfPasswordPrompts, + oTISAuthentication, oUsePrivilegedPort, oLogLevel } OpCodes; /* Textual representations of the tokens. */ @@ -148,6 +149,7 @@ { "compression", oCompression }, { "compressionlevel", oCompressionLevel }, { "keepalive", oKeepAlives }, + { "transmitinterlude", oTransmitInterlude }, { "numberofpasswordprompts", oNumberOfPasswordPrompts }, { "tisauthentication", oTISAuthentication }, { "loglevel", oLogLevel }, @@ -355,6 +357,10 @@ intptr = &options->keepalives; goto parse_flag; + case oTransmitInterlude: + intptr = &options->trans_inter; + goto parse_int; + case oNumberOfPasswordPrompts: intptr = &options->number_of_password_prompts; goto parse_int; @@ -610,6 +616,7 @@ options->strict_host_key_checking = -1; options->compression = -1; options->keepalives = -1; + options->trans_inter = -1; options->compression_level = -1; options->port = -1; options->connection_attempts = -1; @@ -677,6 +684,8 @@ options->compression = 0; if (options->keepalives == -1) options->keepalives = 1; + if (options->trans_inter == -1) + options->trans_inter = 0; if (options->compression_level == -1) options->compression_level = 6; if (options->port == -1) diff -ur openssh-1.2.2p1/readconf.h openssh-1.2.2p1-trans_inter-r2/readconf.h --- openssh-1.2.2p1/readconf.h Sun Dec 5 16:47:29 1999 +++ openssh-1.2.2p1-trans_inter-r2/readconf.h Tue Mar 7 13:19:29 2000 @@ -56,6 +56,7 @@ int compression_level; /* Compression level 1 (fast) to 9 * (best). */ int keepalives; /* Set SO_KEEPALIVE. */ + int trans_inter; /* Guarantee transmit every n seconds. */ LogLevel log_level; /* Level for logging. */ int port; /* Port to connect. */ diff -ur openssh-1.2.2p1/ssh.0 openssh-1.2.2p1-trans_inter-r2/ssh.0 --- openssh-1.2.2p1/ssh.0 Tue Mar 7 03:06:05 2000 +++ openssh-1.2.2p1-trans_inter-r2/ssh.0 Tue Mar 7 13:19:29 2000 @@ -486,6 +486,21 @@ be verified automatically in either case. The argument must be ``yes'' or ``no''. + TransmitInterlude + Specifies a maximum time to allow between transmitting packets, + in seconds. If this amount of time passes and the client has no + data to send, it will send an ignore packet to the server. One + example where this is useful is when using ssh from behind a Lin- + ux ip_masquerade firewall. If packets aren't sent through such a + firewall periodically, the firewall may forget about the connec- + tion. Then when a packet finally is sent, the firewall will as- + sign a new port, which will cause the remote server to disconnect + the session. This option defaults to ``0'', which means not + sending periodic packets. A setting of a few hundred seconds + should be about right if this is needed. You should probably try + setting KeepAlive to ``yes'' in your conf files on both the serv- + er and the client first. + UsePrivilegedPort Specifies whether to use a privileged port for outgoing connec- tions. The argument must be ``yes'' or ``no''. The default is diff -ur openssh-1.2.2p1/ssh.1 openssh-1.2.2p1-trans_inter-r2/ssh.1 --- openssh-1.2.2p1/ssh.1 Fri Mar 3 03:48:49 2000 +++ openssh-1.2.2p1-trans_inter-r2/ssh.1 Tue Mar 7 13:19:29 2000 @@ -720,6 +720,22 @@ .Dq yes or .Dq no . +.It Cm TransmitInterlude +Specifies a maximum time to allow between transmitting packets, +in seconds. If this amount of time passes and the client has +no data to send, it will send an ignore packet to the server. +One example where this is useful is when using ssh from behind +a Linux ip_masquerade firewall. If packets aren't sent through +such a firewall periodically, the firewall may forget about the +connection. Then when a packet finally is sent, the firewall +will assign a new port, which will cause the remote server to +disconnect the session. This option defaults to +.Dq 0 , +which means not sending periodic packets. A setting of a few +hundred seconds should be about right if this is needed. You +should probably try setting KeepAlive to +.Dq yes +in your conf files on both the server and the client first. .It Cm UsePrivilegedPort Specifies whether to use a privileged port for outgoing connections. The argument must be