openssh bugs - Apr 2013 - [Bug 2090] New: SSH/SSHD hang with a Match User setting in sshd_config .

https://bugzilla.mindrot.org/show_bug.cgi?id=2090

            Bug ID: 2090
           Summary: SSH/SSHD hang with a Match User setting in sshd_config
                    .
    Classification: Unclassified
           Product: Portable OpenSSH
           Version: 6.1p1
          Hardware: Other
                OS: AIX
            Status: NEW
          Severity: critical
          Priority: P5
         Component: sshd
          Assignee: unassigned-bugs at mindrot.org
          Reporter: vikas.uit10 at gmail.com

Created attachment 2243
  --> https://bugzilla.mindrot.org/attachment.cgi?id=2243&action=edit
Detailed Debug info.

Problem Description - 
=====================
SSH/SSHD hang when SSHD is configured to allow only sftp connection
requests from a particular user 
( Match User setting in sshd_config ).


Steps to reproduce - 
====================NOTE- These steps are for recreating the scenario only to a
localhost(
which acts as both Client & Server) and similar steps can be used 
to recreate it for a Client & Server ( different machines ).

1. Create a local user say "test" .Also,define a password for user
"test" .

2. Set-up the password less( public key ) authentication for user
"test" ,
   i.e Copy /home/test/.ssh/id_rsa.pub to 
/home/test/.ssh/authorized_keys ( So, I don't need to type the test's
password every time).

3. Edit the sshd_config file to include the Match User setting, so that
user "test"  can do only sftp to this server ( i.e localhost )- 

   # Example of overriding settings on a per-user basis
   #Match User anoncvs
   #       X11Forwarding no
   #       AllowTcpForwarding no
   #       ForceCommand cvs server
   Match User test
        ForceCommand internal-sftp

4. Restart SSHD .

5. Try to login using sftp (should be OK as
   expected):                                


   $ sftp test at localhost                                                

   Connecting to localhost...                                           

   test at localhost's password:                                           

   sftp>  

6. Try to ssh to the localhost (Here is the issue . It will hang.):  

   SSH hangs after random number of repeated login requests ( see the
infinite while loop used to send repeated login
   requests . Also ,Since Public key authentication has been configured
for user "test" , it will not ask for its password ).

(0) root @ ic15: 7.1.0.0: /
# su test

(0) test @ ic15: 7.1.0.0: /
# while
> true
> do
> ssh localhost
> done
This service allows sftp connections only.
Connection to localhost closed.
This service allows sftp connections only.
Connection to localhost closed.
This service allows sftp connections only.
Connection to localhost closed.
This service allows sftp connections only.
Connection to localhost closed.
This service allows sftp connections only.
Connection to localhost closed.
This service allows sftp connections only.
Connection to localhost closed.               <== It hangs here after a
random number of login requests.


The below SSHD processes are active at the same time - 

(0) root @ ic15: 7.1.0.0: /
# ps -ef | grep sshd
    root 5111844 9502940   0 15:24:49      -  0:00 sshd: test [priv]
    test 9175114 5111844   0 15:24:50      -  0:00 sshd: test at pts/7
    root 9502940 3604646   0 15:17:24      -  0:00 /usr/sbin/sshd

I believe, there is some Communication problem between the monitor sshd
process (5111844) and its child 9175114.
(Some signal is being missed from either side ).

Looks like, both Client (ssh) & Server (sshd) are waiting for each
other and a deadlock kind of scenario is being created.
Please check the attached .txt file for detailed debugging information.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2090

Darren Tucker <dtucker at zip.com.au> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |dtucker at zip.com.au
            Version|6.1p1                       |6.0p1

--- Comment #1 from Darren Tucker <dtucker at zip.com.au> ---
the debug logs say 6.0p1 not 6.1p1.  other than that, they're not all
that useful.  They just show the monitor sitting there waiting to be
asked to do something (which is normal) and and on the slave side just
shows that it's waiting for input.

Which version (and ML) of AIX is this?

Can you get the debug output from ssh and sshd and attach it to this
bug?  For the client it should be simple:

$ while true; do ssh -vvv localhost; done

for the server, you can try this assuming you have sudo available:

$ while true; do ssh -o 'proxycommand sudo /usr/sbin/sshd -e -i -o
loglevel=debug3' localhost; done

it's possible that won't trigger the problem, in that case you'll
need
to set loglevel debug3 in your sshd_config and grep the debug logs out
of syslog or something.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2090

--- Comment #2 from Vikas K Vicky <vikas.uit10 at gmail.com> ---
Actually I have tested with OpenSSH 5.8p1,6.0p1,6.1p1 & 6.2p1 and this
problem is present in all these versions.

The problem can be reproduced in AIX version 6.1 & 7.1 both with any of
the above mentioned OpenSSH versions.

I am attaching here the debug logs from both Client & Server.

NOTE- One Debug log (Debug-Output-Without-Hang) is for successful
scenario , i.e No SSH/SSHD doesn't hang with the configuration as
mentioned in the recreation steps.

Other Debug log(Debug-Output-With-Hang) is for the Hang scenario , i.e
SSH/SSHD hangs with the configuration as mentioned in the recreation
steps ( you need to try ssh login manually for random number of times
to recreate it).

One thing I noticed that for SSH/SSHD hang to happen - ssh login
requests should be very frequent from Client side. If there is
considerable amount of time gap between two consecutive ssh login
requests , then chances for Hang is very rare.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2090

--- Comment #3 from Vikas K Vicky <vikas.uit10 at gmail.com> ---
Created attachment 2246
  --> https://bugzilla.mindrot.org/attachment.cgi?id=2246&action=edit
Debug-Output-Without-Hang

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2090

--- Comment #4 from Vikas K Vicky <vikas.uit10 at gmail.com> ---
Created attachment 2247
  --> https://bugzilla.mindrot.org/attachment.cgi?id=2247&action=edit
Debug-Output-With-Hang

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2090

--- Comment #5 from Darren Tucker <dtucker at zip.com.au> ---
is this a stock version of openssh compiled from the source at
openssh.com?  I see entries in the logs which don't seem to belong
("debug3: ACCESS KEY", ".  If so, can you reproduce the problem
with a
stock version?  If so, please attach those logs.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2090

rafaelantonioli at bb.com.br changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |rafaelantonioli at bb.com.br

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2090

Damien Miller <djm at mindrot.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |djm at mindrot.org
           Severity|critical                    |normal

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2090

--- Comment #6 from Vikas K Vicky <vikas.uit10 at gmail.com> ---
Ok.. I will upload the logs from stock version very soon.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2090

--- Comment #7 from Vikas K Vicky <vikas.uit10 at gmail.com> ---
Created attachment 2343
  --> https://bugzilla.mindrot.org/attachment.cgi?id=2343&action=edit
Debug-Output-with-Hang-NEW

Attaching "Debug-Output-with-Hang-NEW.txt" which is the debug log from
stock OpenSSH version 6.0p1.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2090

--- Comment #8 from Darren Tucker <dtucker at zip.com.au> ---
Comment on attachment 2343
  --> https://bugzilla.mindrot.org/attachment.cgi?id=2343
Debug-Output-with-Hang-NEW
>debug1: init_func_ptrs success
that is not from the stock openssh available from openssh.com
>debug1: Value of krb5 is 1
or that
>debug1: Value of gssapi is 1
or that
>From Server side - 
[...]
>debug3: Value for authType is STD_AUTH
or that
>debug1: init_func_ptrs passed
or that
>debug1: value of krb5 is : 1
or that
>debug1: value of gssapi is : 1
or that
>debug1: Failed to collect Cookie from Keystore
or that
>debug1: Keystore Opening wil be failed after login
or that
>debug1: Cookie received :
or that
>debug1: Failed to collect Cookie from Keystore
or that
>debug1: Keystore Opening wil be failed after login
or that.  and it's not even spelt correctly.

You are using a (heavily!) modified version of openssh.  We have no
idea what's in the software you're running.  You need to seek
assistance from whoever supplied this modified version of openssh.

If you can download the tarball from openssh.com, compile, install and
reproduce the problem then we may be able to help you. If you can do
that, please reopen this bug and attach the output of ./configure, the
config.h file and the debug output from the server.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2090

Darren Tucker <dtucker at zip.com.au> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|---                         |INVALID

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2090

Damien Miller <djm at mindrot.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|RESOLVED                    |CLOSED

--- Comment #9 from Damien Miller <djm at mindrot.org> ---
closing resolved bugs as of 8.6p1 release

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.