openssh bugs - Aug 2009 - [Bug 1633] New: Race condition in ssh-agent AUTH_CONNECTION

https://bugzilla.mindrot.org/show_bug.cgi?id=1633

           Summary: Race condition in ssh-agent AUTH_CONNECTION
           Product: Portable OpenSSH
           Version: 5.2p1
          Platform: ix86
        OS/Version: Linux
            Status: NEW
          Keywords: patch
          Severity: normal
          Priority: P2
         Component: ssh-agent
        AssignedTo: unassigned-bugs at mindrot.org
        ReportedBy: noodle10000 at googlemail.com
                CC: djm at mindrot.org, ohannet at allez-oop.net
        Depends on: 1254


--- Comment #0 from noodle10000 at googlemail.com 2009-08-19 06:27:26 EST ---
I have the same issue as encountered in bug 1254.  When launching
thousands of SSH connections via a script (the open source
taktuk/kanif) using ssh-agent to forward keys, occasionally I will see
ssh-agent hang and consume 100% of one CPU.  This does not happen every
time, but around 1 out of every 3 runs.

I have compiled 5.2p1 which also exhibits the same issue.   strace at
the time of the hang reports an EAGAIN error on a read call.  A few
printfs isolated the code in question to be the same as mentioned in
bug 1254, but the suggested workaround (add a usleep before trying the
read again) does not work in any case.

This issue is also reported at
http://www.plug.org/pipermail/plug/2009-April/033800.html



+++ This bug was initially created as a clone of Bug #1254 +++

In function after_select(), case AUTH_CONNECTION, the do-loop which
handles socket reads will peg my CPU at close to 100% when errno is
EAGAIN.

I'm running FreeBSD 6.2 pre-release, with OpenSSH built from the ports
collection (security/openssh-portable).

The problem only occurs for me while running an automation script that
sends commands through ssh to about a hundred servers at at time, and I
have not been successful in identifying which server causes the
problem.  But the bottom line is that the read fails with errno EAGAIN,
and continues to fail in a very tight loop until a timeout occurs at
some point.

My work-around was to introduce a tiny sleep before the continue
statement in that loop, which is apparently enough to allow some data
to become available for reading, and makes the problem go away.

I will attach my work-around as a patch, realizing that usleep() is
probably not available on all platforms.

-- 
Configure bugmail: https://bugzilla.mindrot.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=1633



--- Comment #1 from Damien Miller <djm at mindrot.org> 2009-08-19 06:37:05
EST ---
Created an attachment (id=1670)
fall back to select() on read/write interruptions

Could you try to reproduce the problem with this patch applied?

-- 
Configure bugmail: https://bugzilla.mindrot.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=1633



--- Comment #2 from Damien Miller <djm at mindrot.org> 2009-08-19 06:46:27
EST ---
... and here is a theory on how it occurs:

on a heavily loaded ssh-agent, we can create a new socket in the
ssh-agent.c:after_select() loop, via the AUTH_SOCKET case calling
new_socket(). This might increase sockets_alloc past the value it had
when execution enters after_select().

The for() loop in after_select() can therefore progress into sockets
that did not exist when select() and, critically, prepare_select() was
called. prepare_select() sizes and clears the fd_sets that select()
subsequently populates and after_select() tests.

So a new AUTH_CONNECTION socket whose creation increments sockets_alloc
can cause after_select to test past the end of the allocated fd_sets
and might (depending on what it finds) treat them as ready for reading.

-- 
Configure bugmail: https://bugzilla.mindrot.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=1633


Damien Miller <djm at mindrot.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Attachment #1670|0                           |1
        is obsolete|                            |


--- Comment #3 from Damien Miller <djm at mindrot.org> 2009-08-19 06:50:43
EST ---
Created an attachment (id=1671)
fix the root cause of the problem too

-- 
Configure bugmail: https://bugzilla.mindrot.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=1633


Damien Miller <djm at mindrot.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Blocks|                            |1626


-- 
Configure bugmail: https://bugzilla.mindrot.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=1633



--- Comment #4 from noodle10000 at googlemail.com 2009-08-19 07:14:46 EST ---
Patch applied to the ssh-agent.c in openssh-5.2p1 (RCS revision 1.159).
 I have now successfully run our scripts against 6000 hosts for the
first time, so it appears to have solved the issue.  

I will be soak-testing over the next 48 hours and will update after
that. 

(and thanks for the very quick response!)

-- 
Configure bugmail: https://bugzilla.mindrot.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=1633



--- Comment #5 from Damien Miller <djm at mindrot.org> 2009-08-27 03:30:12
EST ---
Have you been able to reproduce the problem with patch #1671 applied?

-- 
Configure bugmail: https://bugzilla.mindrot.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=1633



--- Comment #6 from noodle10000 at googlemail.com 2009-08-27 19:38:03 EST ---
(In reply to comment #5)
> Have you been able to reproduce the problem with patch #1671 applied?
We've not had any further problems with ssh-agent since applying #1671
- looks like it's fixed.  Thanks!

-- 
Configure bugmail: https://bugzilla.mindrot.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=1633


Damien Miller <djm at mindrot.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|                            |FIXED


--- Comment #7 from Damien Miller <djm at mindrot.org> 2009-09-02 00:43:47
EST ---
patch applied. This will be in openssh-5.4.

-- 
Configure bugmail: https://bugzilla.mindrot.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=1633

Damien Miller <djm at mindrot.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|RESOLVED                    |CLOSED

--- Comment #8 from Damien Miller <djm at mindrot.org> 2009-10-06 15:02:33
EST ---
Mass move of RESOLVED bugs to CLOSED now that 5.3 is out.

-- 
Configure bugmail: https://bugzilla.mindrot.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=1633

Damien Miller <djm at mindrot.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |focht at gmx.net

--- Comment #9 from Damien Miller <djm at mindrot.org> 2009-11-20 10:41:44
EST ---
*** Bug 1135 has been marked as a duplicate of this bug. ***

-- 
Configure bugmail: https://bugzilla.mindrot.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=1633

--- Comment #10 from Darren Tucker <dtucker at zip.com.au> 2010-03-26
10:51:18 EST ---
With the release of 5.4p1, this bug is now considered closed.

-- 
Configure bugmail: https://bugzilla.mindrot.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.