At my company we use port forwarding as an alternative to VPN. In previous releases of openssh (pre 6.0) we could run a script and fetch the thousands of forwards to our local machine to connect to remote machines. Since openssh 6.x, whenever we run the same script we get a bufferoverflow error. ---- debug1: channel 4577: new [port listener] debug1: channel 4578: new [client-session] debug1: Requesting no-more-sessions at openssh.com debug1: Entering interactive session. *** buffer overflow detected ***: ssh terminated ======= Backtrace: ========/lib64/libc.so.6(__fortify_fail+0x37)[0x7f3516baf8f7] /lib64/libc.so.6(+0x10bac0)[0x7f3516badac0] /lib64/libc.so.6(+0x10d867)[0x7f3516baf867] ssh(+0x26825)[0x7f3519304825] ssh(+0x2aece)[0x7f3519308ece] ssh(+0x12d05)[0x7f35192f0d05] ssh(+0xb3ee)[0x7f35192e93ee] /lib64/libc.so.6(__libc_start_main+0xf5)[0x7f3516ac3af5] ssh(+0xc289)[0x7f35192ea289] ---- I assume this is a security measure to thwart overflow attacks. I've tried utilizing multiplexing, but received the same type of buffer overflow. Are there any settings that I can change to allow for more forwards? At the moment we have a work around to only pull back ports in batches but that isn't a long term solution. Any advice would be appreciated. Thanks -Todd Morgan
Todd Morgan <bamamorgans at gmail.com> on Fri, 2014/09/26 15:01:> At my company we use port forwarding as an alternative to VPN. In previous > releases of openssh (pre 6.0) we could run a script and fetch the thousands > of forwards to our local machine to connect to remote machines. Since > openssh 6.x, whenever we run the same script we get a bufferoverflow error.Did not take a look at you issue, but (if I understand your needs correctly) using sshuttle [0] may be an option. Buffer overflow should not occur, though... [0] https://github.com/apenwarr/sshuttle -- Schoene Gruesse Chris O< ascii ribbon campaign stop html mail - www.asciiribbon.org -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: not available URL: <http://lists.mindrot.org/pipermail/openssh-unix-dev/attachments/20140926/42e16b3e/attachment-0001.bin>
On Fri, 26 Sep 2014, Todd Morgan wrote:> At my company we use port forwarding as an alternative to VPN. In previous > releases of openssh (pre 6.0) we could run a script and fetch the thousands > of forwards to our local machine to connect to remote machines. Since > openssh 6.x, whenever we run the same script we get a bufferoverflow error. > > ---- > debug1: channel 4577: new [port listener] > debug1: channel 4578: new [client-session] > debug1: Requesting no-more-sessions at openssh.com > debug1: Entering interactive session. > *** buffer overflow detected ***: ssh terminated > ======= Backtrace: ========> /lib64/libc.so.6(__fortify_fail+0x37)[0x7f3516baf8f7] > /lib64/libc.so.6(+0x10bac0)[0x7f3516badac0] > /lib64/libc.so.6(+0x10d867)[0x7f3516baf867] > ssh(+0x26825)[0x7f3519304825] > ssh(+0x2aece)[0x7f3519308ece] > ssh(+0x12d05)[0x7f35192f0d05] > ssh(+0xb3ee)[0x7f35192e93ee] > /lib64/libc.so.6(__libc_start_main+0xf5)[0x7f3516ac3af5] > ssh(+0xc289)[0x7f35192ea289] > ---- > > I assume this is a security measure to thwart overflow attacks. I've tried > utilizing multiplexing, but received the same type of buffer overflow. > > Are there any settings that I can change to allow for more forwards? At the > moment we have a work around to only pull back ports in batches but that > isn't a long term solution. > > Any advice would be appreciated. ThanksIt looks like it is crashing inside libc, but it is hard to tell. Did you build OpenSSH yourself? If so, could you try compiling with debugging symbols enabled (ensure -g is in Makefile CFLAGS, "make clean; make"). Then try to run ssh under a debugger, e.g. from your build directory gdb --args ./ssh user at host When it crashes, enter "bt" to get a backtrace. That should give us a good start to figure out what is going wrong. -d
On Fri, 26 Sep 2014, Todd Morgan wrote:> At my company we use port forwarding as an alternative to VPN. In previous > releases of openssh (pre 6.0) we could run a script and fetch the thousands > of forwards to our local machine to connect to remote machines. Since > openssh 6.x, whenever we run the same script we get a bufferoverflow error. > > ---- > debug1: channel 4577: new [port listener] > debug1: channel 4578: new [client-session] > debug1: Requesting no-more-sessions at openssh.com > debug1: Entering interactive session. > *** buffer overflow detected ***: ssh terminatedI figured this out - there is no actual buffer overflow, but a misguided FD_SET check misfiring. Some background. OpenSSH uses select(2) for fd monitoring. Yes, this is a crappy interface but it is the most portable way to do it and for the vast majority of use absolutely fine. One of the biggest problems with select(2) is that the POSIX-specified API limits the number of file descriptors that can be monitored to FD_SETSIZE (typically 1024). Many Unix variants, including Linux, several commercial Unix and all current BSD allow exceeding this limit by manually allocating the fd_set. OpenSSH has used this for ~15 years. Recently, we enabled FORTIFY_SOURCE to get some libc/toolchain hardening checks. It turns out that one of these is a dumb check in the FD_SET macro that fd < 1024. The check isn't aware of allocated fd_sets or that no overflow actually happens. https://sourceware.org/bugzilla/show_bug.cgi?id=10352 I'm not sure how to disable this check (which is broken) without turning off the rest of FORTIFY_SOURCE (which gives some good hardening). Suggestions welcome. In the meantime, you can disable this check by editing your Makefile (as generated by configure) and removing the -DFORTIFY_SOURCE option from CFLAGS. -d