openssh bugs - Feb 2015 - [Bug 2355] New: general protection / segfaults when PermitOpen=none

https://bugzilla.mindrot.org/show_bug.cgi?id=2355

            Bug ID: 2355
           Summary: general protection / segfaults when PermitOpen=none
           Product: Portable OpenSSH
           Version: 6.7p1
          Hardware: amd64
                OS: Linux
            Status: NEW
          Severity: major
          Priority: P5
         Component: sshd
          Assignee: unassigned-bugs at mindrot.org
          Reporter: calestyo at scientia.net

Hey.

I found a "special" situation in which ssh connections crash every few
tries and sometimes (but not always) one get's any of these along:
[527879.021049] traps: sshd[14583] general protection ip:7fbc7f04a664
sp:7fff3939fe58 error:0 in libc-2.19.so[7fbc7efce000+19f000Hey.

I found a special situation in which ssh connections crash every few
tries and sometimes (but not always) one get's any of these along:
[527879.021049] traps: sshd[14583] general protection ip:7fbc7f04a664
sp:7fff3939fe58 error:0 in libc-2.19.so[7fbc7efce000+19f000]
[527945.727953] traps: sshd[14660] general protection ip:7f069558d664
sp:7fffc4223c88 error:0 in libc-2.19.so[7f0695511000+19f000]
[528046.264330] traps: sshd[14826] general protection ip:7f1b26eed664
sp:7fff521d7178 error:0 in libc-2.19.so[7f1b26e71000+19f000]
[536582.887955] traps: sshd[26078] general protection ip:7f96158b4664
sp:7fff2fef4a08 error:0 in libc-2.19.so[7f9615838000+19f000]
[536628.489940] traps: sshd[26206] general protection ip:7f9cc14a9664
sp:7fffdacfb478 error:0 in libc-2.19.so[7f9cc142d000+19f000]
[536734.550558] traps: sshd[26320] general protection ip:7f260fc18664
sp:7ffffb25be88 error:0 in libc-2.19.so[7f260fb9c000+19f000]
[536841.887230] traps: sshd[26513] general protection ip:7f168b350664
sp:7fff8a85a2c8 error:0 in libc-2.19.so[7f168b2d4000+19f000]
[536860.256030] traps: sshd[26572] general protection ip:7fba93937664
sp:7ffffcf18928 error:0 in libc-2.19.so[7fba938bb000+19f000]
[536949.787928] sshd[27137]: segfault at 8100000038 ip 00007f84523e666
sp 00007fff2cc1d908 error 4 in libc-2.19.so[7f845236a000+19f000]
[537088.405962] traps: sshd[27582] general protection ip:7f349cde6664
sp:7fffaf183ee8 error:0 in libc-2.19.so[7f349cd6a000+19f000]

What I do is basically the following:
Having sshd running (my sshd_config is attached), and gitolite3
(from sid) installed.

Gitolite (which I use with the "git" username) in turn has entries
like these:
command="/usr/share/gitolite3/gitolite-shell
admin",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty
ssh-ed25519 ...
in its authorized_key files


Then I repeatedly do:
$ ssh git at myserver info

Sometimes this works and I get:
> hello someName, this is git at myserver running gitolite3 3.6.1-3 (Debian)
on git 2.1.4
But more than every 2nd time it fails and I get
> Write failed: Broken pipe
Sometimes (not always) with a general protection or segfault.

>From my sshd_config, which uses a Match block for the git
user (for reasons of hardening), I found that the
> PermitOpen none
line is the cause of the problem
When I comment it, then the connections *always* succeed (well at least
from about ~20 successive tries).
]
[527945.727953] traps: sshd[14660] general protection ip:7f069558d664
sp:7fffc4223c88 error:0 in libc-2.19.so[7f0695511000+19f000]
[528046.264330] traps: sshd[14826] general protection ip:7f1b26eed664
sp:7fff521d7178 error:0 in libc-2.19.so[7f1b26e71000+19f000]
[536582.887955] traps: sshd[26078] general protection ip:7f96158b4664
sp:7fff2fef4a08 error:0 in libc-2.19.so[7f9615838000+19f000]
[536628.489940] traps: sshd[26206] general protection ip:7f9cc14a9664
sp:7fffdacfb478 error:0 in libc-2.19.so[7f9cc142d000+19f000]
[536734.550558] traps: sshd[26320] general protection ip:7f260fc18664
sp:7ffffb25be88 error:0 in libc-2.19.so[7f260fb9c000+19f000]
[536841.887230] traps: sshd[26513] general protection ip:7f168b350664
sp:7fff8a85a2c8 error:0 in libc-2.19.so[7f168b2d4000+19f000]
[536860.256030] traps: sshd[26572] general protection ip:7fba93937664
sp:7ffffcf18928 error:0 in libc-2.19.so[7fba938bb000+19f000]
[536949.787928] sshd[27137]: segfault at 8100000038 ip 00007f84523e666
sp 00007fff2cc1d908 error 4 in libc-2.19.so[7f845236a000+19f000]
[537088.405962] traps: sshd[27582] general protection ip:7f349cde6664
sp:7fffaf183ee8 error:0 in libc-2.19.so[7f349cd6a000+19f000]

What I do is basically the following:
Having sshd running (my sshd_config is attached), and gitolite3
(from sid) installed.

Gitolite (which I use with the "git" username) in turn has entries
like these:
command="/usr/share/gitolite3/gitolite-shell
admin",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty
ssh-ed25519 ...
in its authorized_key files


Then I repeatedly do:
$ ssh git at myserver info

Sometimes this works and I get:
> hello someName, this is git at myserver running gitolite3 3.6.1-3 (Debian)
on git 2.1.4
But more than every 2nd time it fails and I get
> Write failed: Broken pipe
Sometimes (not always) with a general protection or segfault.

>From my sshd_config, which uses a Match block for the git
user (for reasons of hardening), I found that the
> PermitOpen none
line is the cause of the problem
When I comment it, then the connections *always* succeed (well at least
from about ~20 successive tries).

I should probably further notice: systemd/logind/PAM is used (not sure
if this could somehow interfere).
Also, I'm a bit unsure whether the "main" sshd is crashing or
whethr
it's just the processes of the sessions.
I didn't manually restart sshd, but it might be that systemd does that
automatically? How would I find out?


So some bug is hidden there...

Cheers,
Chris

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2355

--- Comment #1 from Christoph Anton Mitterer <calestyo at scientia.net>
---
Created attachment 2550
  --> https://bugzilla.mindrot.org/attachment.cgi?id=2550&action=edit
sshd_config

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2355

Christoph Anton Mitterer <calestyo at scientia.net> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           See Also|                            |http://bugs.debian.org/7788
                   |                            |07

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2355

Damien Miller <djm at mindrot.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |djm at mindrot.org

--- Comment #2 from Damien Miller <djm at mindrot.org> ---
Could you try running a sshd in debugging mode on a different port
(i.e. "sshd -dddp 2222") and catching it in the act of crashing?
Seeing
where it fails would be a great help.

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2355

--- Comment #3 from Damien Miller <djm at mindrot.org> ---
BTW, I can't replicate this with HEAD

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2355

--- Comment #4 from Christoph Anton Mitterer <calestyo at scientia.net>
---
Created attachment 2551
  --> https://bugzilla.mindrot.org/attachment.cgi?id=2551&action=edit
ssh.log

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2355

--- Comment #5 from Christoph Anton Mitterer <calestyo at scientia.net>
---
Created attachment 2552
  --> https://bugzilla.mindrot.org/attachment.cgi?id=2552&action=edit
sshd.log

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2355

--- Comment #6 from Christoph Anton Mitterer <calestyo at scientia.net>
---
(In reply to Damien Miller from comment #2)
> Could you try running a sshd in debugging mode on a different port
> (i.e. "sshd -dddp 2222") and catching it in the act of crashing?
> Seeing where it fails would be a great help.
Sure, see attached files: sshd and ssh output, from the later you see
which tries failed (with which error) and which worked.

Interestingly, the sshd quite *every time* after the end of the
connection... is this because of -D?


(In reply to Damien Miller from comment #3)
> BTW, I can't replicate this with HEAD
Mhh and have you tried with an older tag as well (i.e. 6.7p1?) and
could replicate it there?

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2355

Darren Tucker <dtucker at zip.com.au> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |dtucker at zip.com.au

--- Comment #7 from Darren Tucker <dtucker at zip.com.au> ---
(In reply to Christoph Anton Mitterer from comment #6)
> (In reply to Damien Miller from comment #2)
> > Could you try running a sshd in debugging mode on a different port
> > (i.e. "sshd -dddp 2222") and catching it in the act of
crashing?
> > Seeing where it fails would be a great help.
> Sure, see attached files: sshd and ssh output, from the later you
> see which tries failed (with which error) and which worked.
> 
> Interestingly, the sshd quite *every time* after the end of the
> connection... is this because of -D?
-D and -d mean different things, but yes -d means "run in debug mode
once then exit".  If you want to leave it up you can use this instead
of -d:

/path/to/sshd -De -o LogLevel=debug3
> (In reply to Damien Miller from comment #3)
> > BTW, I can't replicate this with HEAD
> Mhh and have you tried with an older tag as well (i.e. 6.7p1?) and
> could replicate it there?
Note that you are running a vendor-modified version of OpenSSH:
Local version string SSH-2.0-OpenSSH_6.7p1 Debian-3

It's not all that easy for us to know exactly what each vendor has
changed.  Even things like which compiler flags they use can make a
difference.  Have you reported this to Debian?  They're in a much
better position to reproduce a problem.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2355

Darren Tucker <dtucker at zip.com.au> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Attachment #2550|application/octet-stream    |text/plain
          mime type|                            |

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2355

Darren Tucker <dtucker at zip.com.au> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Attachment #2551|text/x-log                  |text/plain
          mime type|                            |

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2355

Darren Tucker <dtucker at zip.com.au> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Attachment #2552|text/x-log                  |text/plain
          mime type|                            |

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2355

--- Comment #8 from Darren Tucker <dtucker at zip.com.au> ---
Damien and I spent a couple of hours with a VM trying to figure this
out and we now think we know what the cause is.  I'll update this bug
again once we're sure we are on the right track.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2355

--- Comment #9 from Christoph Anton Mitterer <calestyo at scientia.net>
---
Hey.

Sorry for not having answered earlier... this got somehow hidden under
a huge pile of mails the last days =)

Yes I did report it in Debian, see the URL set here in the bug report's
"See Also section" (http://bugs.debian.org/778807)
But nothing has happened there so far.

When I reported this in the beginning, I had a short glance whether any
of Debian's patches obviously touches something in this area,...
nothing I'd have seen (OTOH I'm not an OpenSSH code expert).



It's great to hear that you possibly found the issue :-)


Thanks,
Chris.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2355

Darren Tucker <dtucker at zip.com.au> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Blocks|                            |2360

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2355

Darren Tucker <dtucker at zip.com.au> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |ASSIGNED
           Assignee|unassigned-bugs at mindrot.org |dtucker at zip.com.au

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2355

--- Comment #10 from Darren Tucker <dtucker at zip.com.au> ---
Created attachment 2617
  --> https://bugzilla.mindrot.org/attachment.cgi?id=2617&action=edit
calloc permitted_adm_opens instead of malloc to ensure it's zeroed

Here's the fix.  I'll be commit this shortly.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2355

Darren Tucker <dtucker at zip.com.au> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|ASSIGNED                    |RESOLVED
         Resolution|---                         |FIXED

--- Comment #11 from Darren Tucker <dtucker at zip.com.au> ---
The patch has been applied and will be in the 6.9 release.  Thanks for
the report!

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2355

Damien Miller <djm at mindrot.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|RESOLVED                    |CLOSED

--- Comment #12 from Damien Miller <djm at mindrot.org> ---
Set all RESOLVED bugs to CLOSED with release of OpenSSH 7.1

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.