openssh bugs - Aug 2008 - [Bug 1517] New: ssh ControlMaster process is crashing frequently when multiplexing ssh and scp connections with error 'select: Invalid argument'

https://bugzilla.mindrot.org/show_bug.cgi?id=1517

           Summary: ssh ControlMaster process is crashing frequently when
                    multiplexing ssh and scp connections with error
                    'select: Invalid argument'
           Product: Portable OpenSSH
           Version: 5.1p1
          Platform: Sparc
        OS/Version: Solaris
            Status: NEW
          Severity: normal
          Priority: P2
         Component: ssh
        AssignedTo: unassigned-bugs at mindrot.org
        ReportedBy: james at bluecat.demon.co.uk


ssh ControlMaster process is crashing frequently when multiplexing ssh
and scp connections with error 'select: Invalid argument'.

We have an application which is constantly making two to three hundred
concurrent multiplexed scp and ssh connections between two Solaris
platforms.  The scripts are transferring 3KB files and then performing
checkum and mv commands.  We are using ssh multiplexing to reduce CPU
and network load.  We have a script in cron to restart the
ControlMaster process; it is restarting approx 280 times per day. 

The ssh ControlMaster syntax:
/usr/local/bin/ssh -oControlMaster=yes
-oControlPath="/cdrc/.ssh/mp_sock-scholzite" -fnN -c blowfish-cbc
transfer at scholzite-be

We are using Openssh-5.1p1 compiled from source (and Openssl-0.9.8h). 
This OpenSSH version is used on the client and server, both servers are
Solaris 9 Sparc platforms.  The max file descriptors kernel parameter
on both systems is 8192.

The following parameters are set in the sshd_config (these values are
overkill, but we were clutching at straws!), if we set values below 300
for MaxSessions we get 'Administratively Prohibited' authentication
failures on the server side. 
MaxSessions 8192
MaxStartups 8192

I have attached level 9 debug.  The debug always ends with 'select:
Invalid argument' when the process fails.

Last few lines of debug:
debug3: channel 296: close_fds r 1023 w 1024 e 1025 c 1022
debug3: fd 0 is not O_NONBLOCK
debug3: fd 1 is not O_NONBLOCK
debug3: fd 2 is not O_NONBLOCK
select: Invalid argument
Transferred: sent 12310448, received 106560 bytes, in 88.1 seconds
Bytes per second: sent 139799.0, received 1210.1
debug1: Exit status -1

-- 
Configure bugmail: https://bugzilla.mindrot.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=1517





--- Comment #1 from James Morrish <james at bluecat.demon.co.uk> 
2008-08-30 03:47:08 ---
Created an attachment (id=1564)
 --> (http://bugzilla.mindrot.org/attachment.cgi?id=1564)
ssh Debug Level 9 from Multiplex Control Master

-- 
Configure bugmail: https://bugzilla.mindrot.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=1517





--- Comment #2 from Damien Miller <djm at mindrot.org>  2008-12-08
10:55:28 ---
Created an attachment (id=1587)
 --> (http://bugzilla.mindrot.org/attachment.cgi?id=1587)
debug wrapper for select()

You will run into out-of-file-descriptor conditions with
MaxSessions=8192 and a fd limit of 8192 - each session may consume (in
the worst case) five file descriptors.

That being said, it should not crash the client. Unfortunately,
debugging these things is tricky. Could you see if you can catch the
crash with the attached diff applied? It is a debugging wrapper for
select() that logs the arguments and should catch invalid fds that are
being passed.

be warned: it will cause a lot of log spam, especially since you have
so many open connections.

-- 
Configure bugmail: https://bugzilla.mindrot.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=1517

Damien Miller <djm at mindrot.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |djm at mindrot.org
             Status|NEW                         |RESOLVED
         Resolution|                            |WORKSFORME

--- Comment #3 from Damien Miller <djm at mindrot.org> 2010-04-23 11:45:58
EST ---
1 year with no followup + rewrite of mux code in the meantime => bug
closed

-- 
Configure bugmail: https://bugzilla.mindrot.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=1517

Damien Miller <djm at mindrot.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|RESOLVED                    |CLOSED

--- Comment #4 from Damien Miller <djm at mindrot.org> 2011-01-24 12:34:04
EST ---
Move resolved bugs to CLOSED after 5.7 release

-- 
Configure bugmail: https://bugzilla.mindrot.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.