Laurence Marks
2013-Jul-18 13:22 UTC
Any way to over-ride the "-q" option to debug a possible race condition?
I have a slightly complicated question. I am using an mpi application (Intel impi) which uses ssh to launch applications. For about 0.1-0.2% of the ssh connections I end up with a zombie, i.e. ps will show [ssh] <defunct>. Something is going wrong with the ssh when it starts, perhaps a race condition, it is not disconnecting after some time, that I have been able to verify. The most obvious thing I can think of to do is include LogLevel DEBUG in my .ssh/config to try and trace what is going on, but the ssh process is being launched with a "-q" embedded in it which is over-riding this. I am hopeful that there is something I can set to short-circuit the "-q", although I realise that the chance of their being some undocumented option is small (but finite). For reference ssh -V OpenSSH_5.3p1, OpenSSL 1.0.0-fips 29 Mar 2010 impi 4.1 The structure is that mpirun starts mpiexec.hydra which is turn launches a series of mpi tasks via a ssh connection. N.B., if there are some known issues with that ssh version that may be relevant, information would be useful as I could push on the sys_admin. -- Professor Laurence Marks Department of Materials Science and Engineering Northwestern University www.numis.northwestern.edu 1-847-491-3996 "Research is to see what everybody else has seen, and to think what nobody else has thought" Albert Szent-Gyorgi
Flavien
2013-Jul-18 14:39 UTC
Any way to over-ride the "-q" option to debug a possible race condition?
Laurence Marks ecrivait :> [...] > > The most obvious thing I can think of to do is include LogLevel DEBUG > in my .ssh/config to try and trace what is going on, but the ssh > process is being launched with a "-q" embedded in it which is > over-riding this. I am hopeful that there is something I can set to > short-circuit the "-q", although I realise that the chance of their > being some undocumented option is small (but finite). > [....]If you cannot change the command line that calls ssh with "-q", a warkaround can be to change the ssh binary itself to a wrapper that calls ssh without the "-q". Just move the binary to ssh.bin, and have a small shell script named ssh that removes the "-q" option before "exec"-ing ssh.bin. Hope this helps, Flavien.
Laurence Marks
2013-Jul-18 15:05 UTC
Any way to over-ride the "-q" option to debug a possible race condition?
Thanks, very nice and it even works #!/bin/bash a=`echo $@ | sed -e 's/-q//'` ssh $a On Thu, Jul 18, 2013 at 9:39 AM, Flavien <flavien-ssh at lebarbe.net> wrote:> Laurence Marks ecrivait : >> [...] >> >> The most obvious thing I can think of to do is include LogLevel DEBUG >> in my .ssh/config to try and trace what is going on, but the ssh >> process is being launched with a "-q" embedded in it which is >> over-riding this. I am hopeful that there is something I can set to >> short-circuit the "-q", although I realise that the chance of their >> being some undocumented option is small (but finite). >> [....] > > If you cannot change the command line that calls ssh with "-q", a > warkaround can be to change the ssh binary itself to a wrapper that > calls ssh without the "-q". > Just move the binary to ssh.bin, and have a small shell script named > ssh that removes the "-q" option before "exec"-ing ssh.bin. > > Hope this helps, > > Flavien.-- Professor Laurence Marks Department of Materials Science and Engineering Northwestern University www.numis.northwestern.edu 1-847-491-3996 "Research is to see what everybody else has seen, and to think what nobody else has thought" Albert Szent-Gyorgi