bugzilla-daemon at bugzilla.mindrot.org
2009-Aug-18 20:27 UTC
[Bug 1633] New: Race condition in ssh-agent AUTH_CONNECTION
https://bugzilla.mindrot.org/show_bug.cgi?id=1633
Summary: Race condition in ssh-agent AUTH_CONNECTION
Product: Portable OpenSSH
Version: 5.2p1
Platform: ix86
OS/Version: Linux
Status: NEW
Keywords: patch
Severity: normal
Priority: P2
Component: ssh-agent
AssignedTo: unassigned-bugs at mindrot.org
ReportedBy: noodle10000 at googlemail.com
CC: djm at mindrot.org, ohannet at allez-oop.net
Depends on: 1254
--- Comment #0 from noodle10000 at googlemail.com 2009-08-19 06:27:26 EST ---
I have the same issue as encountered in bug 1254. When launching
thousands of SSH connections via a script (the open source
taktuk/kanif) using ssh-agent to forward keys, occasionally I will see
ssh-agent hang and consume 100% of one CPU. This does not happen every
time, but around 1 out of every 3 runs.
I have compiled 5.2p1 which also exhibits the same issue. strace at
the time of the hang reports an EAGAIN error on a read call. A few
printfs isolated the code in question to be the same as mentioned in
bug 1254, but the suggested workaround (add a usleep before trying the
read again) does not work in any case.
This issue is also reported at
http://www.plug.org/pipermail/plug/2009-April/033800.html
+++ This bug was initially created as a clone of Bug #1254 +++
In function after_select(), case AUTH_CONNECTION, the do-loop which
handles socket reads will peg my CPU at close to 100% when errno is
EAGAIN.
I'm running FreeBSD 6.2 pre-release, with OpenSSH built from the ports
collection (security/openssh-portable).
The problem only occurs for me while running an automation script that
sends commands through ssh to about a hundred servers at at time, and I
have not been successful in identifying which server causes the
problem. But the bottom line is that the read fails with errno EAGAIN,
and continues to fail in a very tight loop until a timeout occurs at
some point.
My work-around was to introduce a tiny sleep before the continue
statement in that loop, which is apparently enough to allow some data
to become available for reading, and makes the problem go away.
I will attach my work-around as a patch, realizing that usleep() is
probably not available on all platforms.
--
Configure bugmail: https://bugzilla.mindrot.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.
bugzilla-daemon at bugzilla.mindrot.org
2009-Aug-18 20:37 UTC
[Bug 1633] Race condition in ssh-agent AUTH_CONNECTION
https://bugzilla.mindrot.org/show_bug.cgi?id=1633 --- Comment #1 from Damien Miller <djm at mindrot.org> 2009-08-19 06:37:05 EST --- Created an attachment (id=1670) fall back to select() on read/write interruptions Could you try to reproduce the problem with this patch applied? -- Configure bugmail: https://bugzilla.mindrot.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching the assignee of the bug. You are watching someone on the CC list of the bug.
bugzilla-daemon at bugzilla.mindrot.org
2009-Aug-18 20:46 UTC
[Bug 1633] Race condition in ssh-agent AUTH_CONNECTION
https://bugzilla.mindrot.org/show_bug.cgi?id=1633 --- Comment #2 from Damien Miller <djm at mindrot.org> 2009-08-19 06:46:27 EST --- ... and here is a theory on how it occurs: on a heavily loaded ssh-agent, we can create a new socket in the ssh-agent.c:after_select() loop, via the AUTH_SOCKET case calling new_socket(). This might increase sockets_alloc past the value it had when execution enters after_select(). The for() loop in after_select() can therefore progress into sockets that did not exist when select() and, critically, prepare_select() was called. prepare_select() sizes and clears the fd_sets that select() subsequently populates and after_select() tests. So a new AUTH_CONNECTION socket whose creation increments sockets_alloc can cause after_select to test past the end of the allocated fd_sets and might (depending on what it finds) treat them as ready for reading. -- Configure bugmail: https://bugzilla.mindrot.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching the assignee of the bug. You are watching someone on the CC list of the bug.
bugzilla-daemon at bugzilla.mindrot.org
2009-Aug-18 20:50 UTC
[Bug 1633] Race condition in ssh-agent AUTH_CONNECTION
https://bugzilla.mindrot.org/show_bug.cgi?id=1633
Damien Miller <djm at mindrot.org> changed:
What |Removed |Added
----------------------------------------------------------------------------
Attachment #1670|0 |1
is obsolete| |
--- Comment #3 from Damien Miller <djm at mindrot.org> 2009-08-19 06:50:43
EST ---
Created an attachment (id=1671)
fix the root cause of the problem too
--
Configure bugmail: https://bugzilla.mindrot.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.
bugzilla-daemon at bugzilla.mindrot.org
2009-Aug-18 20:51 UTC
[Bug 1633] Race condition in ssh-agent AUTH_CONNECTION
https://bugzilla.mindrot.org/show_bug.cgi?id=1633
Damien Miller <djm at mindrot.org> changed:
What |Removed |Added
----------------------------------------------------------------------------
Blocks| |1626
--
Configure bugmail: https://bugzilla.mindrot.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.
bugzilla-daemon at bugzilla.mindrot.org
2009-Aug-18 21:14 UTC
[Bug 1633] Race condition in ssh-agent AUTH_CONNECTION
https://bugzilla.mindrot.org/show_bug.cgi?id=1633 --- Comment #4 from noodle10000 at googlemail.com 2009-08-19 07:14:46 EST --- Patch applied to the ssh-agent.c in openssh-5.2p1 (RCS revision 1.159). I have now successfully run our scripts against 6000 hosts for the first time, so it appears to have solved the issue. I will be soak-testing over the next 48 hours and will update after that. (and thanks for the very quick response!) -- Configure bugmail: https://bugzilla.mindrot.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching the assignee of the bug. You are watching someone on the CC list of the bug.
bugzilla-daemon at bugzilla.mindrot.org
2009-Aug-26 17:30 UTC
[Bug 1633] Race condition in ssh-agent AUTH_CONNECTION
https://bugzilla.mindrot.org/show_bug.cgi?id=1633 --- Comment #5 from Damien Miller <djm at mindrot.org> 2009-08-27 03:30:12 EST --- Have you been able to reproduce the problem with patch #1671 applied? -- Configure bugmail: https://bugzilla.mindrot.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching the assignee of the bug. You are watching someone on the CC list of the bug.
bugzilla-daemon at bugzilla.mindrot.org
2009-Aug-27 09:38 UTC
[Bug 1633] Race condition in ssh-agent AUTH_CONNECTION
https://bugzilla.mindrot.org/show_bug.cgi?id=1633 --- Comment #6 from noodle10000 at googlemail.com 2009-08-27 19:38:03 EST --- (In reply to comment #5)> Have you been able to reproduce the problem with patch #1671 applied?We've not had any further problems with ssh-agent since applying #1671 - looks like it's fixed. Thanks! -- Configure bugmail: https://bugzilla.mindrot.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching the assignee of the bug. You are watching someone on the CC list of the bug.
bugzilla-daemon at bugzilla.mindrot.org
2009-Sep-01 14:43 UTC
[Bug 1633] Race condition in ssh-agent AUTH_CONNECTION
https://bugzilla.mindrot.org/show_bug.cgi?id=1633
Damien Miller <djm at mindrot.org> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |RESOLVED
Resolution| |FIXED
--- Comment #7 from Damien Miller <djm at mindrot.org> 2009-09-02 00:43:47
EST ---
patch applied. This will be in openssh-5.4.
--
Configure bugmail: https://bugzilla.mindrot.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.
bugzilla-daemon at bugzilla.mindrot.org
2009-Oct-06 04:02 UTC
[Bug 1633] Race condition in ssh-agent AUTH_CONNECTION
https://bugzilla.mindrot.org/show_bug.cgi?id=1633
Damien Miller <djm at mindrot.org> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|RESOLVED |CLOSED
--- Comment #8 from Damien Miller <djm at mindrot.org> 2009-10-06 15:02:33
EST ---
Mass move of RESOLVED bugs to CLOSED now that 5.3 is out.
--
Configure bugmail: https://bugzilla.mindrot.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.
bugzilla-daemon at bugzilla.mindrot.org
2009-Nov-19 23:41 UTC
[Bug 1633] Race condition in ssh-agent AUTH_CONNECTION
https://bugzilla.mindrot.org/show_bug.cgi?id=1633
Damien Miller <djm at mindrot.org> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |focht at gmx.net
--- Comment #9 from Damien Miller <djm at mindrot.org> 2009-11-20 10:41:44
EST ---
*** Bug 1135 has been marked as a duplicate of this bug. ***
--
Configure bugmail: https://bugzilla.mindrot.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.
bugzilla-daemon at bugzilla.mindrot.org
2010-Mar-25 23:51 UTC
[Bug 1633] Race condition in ssh-agent AUTH_CONNECTION
https://bugzilla.mindrot.org/show_bug.cgi?id=1633 --- Comment #10 from Darren Tucker <dtucker at zip.com.au> 2010-03-26 10:51:18 EST --- With the release of 5.4p1, this bug is now considered closed. -- Configure bugmail: https://bugzilla.mindrot.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching the assignee of the bug. You are watching someone on the CC list of the bug.