openssh bugs - May 2012 - [Bug 2008] New: IPV6 Bind to port 22 failed

https://bugzilla.mindrot.org/show_bug.cgi?id=2008

             Bug #: 2008
           Summary: IPV6 Bind to port 22 failed
    Classification: Unclassified
           Product: Portable OpenSSH
           Version: 5.9p1
          Platform: ix86
        OS/Version: Linux
            Status: NEW
          Severity: critical
          Priority: P2
         Component: sshd
        AssignedTo: unassigned-bugs at mindrot.org
        ReportedBy: kchen001 at alcatel-lucent.com


After upgraded to openssh 5.9p1, we cannot ssh to this node from other,
and find that port22 didn't bind ipv6 interface, after restart sshd it
works well.

We found this issue is resolved in official redhat version4.3p2, and in
the comments#18, it reused the source code of OpenSSH.

https://bugzilla.redhat.com/show_bug.cgi?id=640857

Here's the secure log:

grep error secure* secure:May  7 15:07:25 wc3il01pdanic1v1 sshd[27451]:
error: Bind to port 22 on 0.0.0.0 failed: Address already in use.


It's our ssh version:
]# rpm -qa|grep openssh
openssh-5.9p1-1
openssh-clients-5.9p1-1
openssh-server-5.9p1-1

In /var/log/secure we found the following statement:
May  7 18:02:00 wc3il01pddnic2v1 sshd[6124]: Received signal 15;
terminating.
May  7 18:02:00 wc3il01pddnic2v1 sshd[3919]: error: Bind to port 22 on
:: failed: Address already in use.
May  7 18:02:01 wc3il01pddnic2v1 sshd[3919]: Server listening on
0.0.0.0 port 22.

-- 
Configure bugmail: https://bugzilla.mindrot.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2008

Darren Tucker <dtucker at zip.com.au> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |dtucker at zip.com.au
           Severity|critical                    |trivial

--- Comment #1 from Darren Tucker <dtucker at zip.com.au> 2012-05-14
18:37:50 EST ---
What do you get when you run sshd from openssh-5.9p1 in debug mode (ie
"/path/to/your/sshd -ddd -p 2022")?  You won't even need to
connect to
it.

On fedora, I get:


debug2: load_server_config: filename /usr/local/etc/sshd_config
debug2: load_server_config: done config len = 435
debug2: parse_server_config: config /usr/local/etc/sshd_config len 435
debug3: /usr/local/etc/sshd_config:13 setting Port 22
debug3: /usr/local/etc/sshd_config:58 setting
ChallengeResponseAuthentication yes
debug3: /usr/local/etc/sshd_config:60 setting Ciphers
aes128-ctr,aes256-ctr,arcfour256,arcfour128,arcfour
debug3: cipher ok: aes128-ctr
[aes128-ctr,aes256-ctr,arcfour256,arcfour128,arcfour]
debug3: cipher ok: aes256-ctr
[aes128-ctr,aes256-ctr,arcfour256,arcfour128,arcfour]
debug3: cipher ok: arcfour256
[aes128-ctr,aes256-ctr,arcfour256,arcfour128,arcfour]
debug3: cipher ok: arcfour128
[aes128-ctr,aes256-ctr,arcfour256,arcfour128,arcfour]
debug3: cipher ok: arcfour
[aes128-ctr,aes256-ctr,arcfour256,arcfour128,arcfour]
debug3: ciphers ok:
[aes128-ctr,aes256-ctr,arcfour256,arcfour128,arcfour]
debug3: /usr/local/etc/sshd_config:76 setting AllowTcpForwarding yes
debug3: /usr/local/etc/sshd_config:77 setting GatewayPorts
clientspecified
debug3: /usr/local/etc/sshd_config:78 setting X11Forwarding yes
debug3: /usr/local/etc/sshd_config:79 setting XAuthLocation
/usr/bin/xauth
debug3: /usr/local/etc/sshd_config:81 setting X11UseLocalhost yes
debug3: /usr/local/etc/sshd_config:82 setting PrintMotd no
debug3: /usr/local/etc/sshd_config:89 setting ClientAliveInterval 300
debug3: /usr/local/etc/sshd_config:90 setting ClientAliveCountMax 3
debug3: /usr/local/etc/sshd_config:100 setting Subsystem sftp   
/usr/local/libexec/sftp-server -f LOCAL7 -l INFO
debug1: sshd version OpenSSH_5.9p1
debug3: Incorrect RSA1 identifier
debug1: read PEM private key done: type RSA
debug1: private host key: #0 type 1 RSA
debug3: Incorrect RSA1 identifier
debug1: read PEM private key done: type DSA
debug1: private host key: #1 type 2 DSA
debug1:
rexec_argv[0]='/home/dtucker/openssh/portable/openssh-5.9p1/sshd'
debug1: rexec_argv[1]='-ddd'
debug1: rexec_argv[2]='-p'
debug1: rexec_argv[3]='2022'
debug3: oom_adjust_setup
Set /proc/self/oom_score_adj from 0 to -1000
debug2: fd 3 setting O_NONBLOCK
debug1: Bind to port 2022 on 0.0.0.0.
Server listening on 0.0.0.0 port 2022.
debug2: fd 4 setting O_NONBLOCK
debug3: sock_set_v6only: set socket 4 IPV6_V6ONLY
debug1: Bind to port 2022 on ::.
Server listening on :: port 2022.

which indicates it's working as expected here.

Are you sure you don't still have the old sshd or something else
listening on port 22?  immediately before starting the sshd, do you get
any "LISTEN" states from "lsof -i :22" ?

-- 
Configure bugmail: https://bugzilla.mindrot.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2008

--- Comment #2 from kchen001 at alcatel-lucent.com 2012-05-16 18:52:03 EST ---
Hi, 

Here's what we got, while currently the node's ssh function works well,
is that log helpful? Or do I need to reproduce it?

debug2: load_server_config: filename /etc/ssh/sshd_config
debug2: load_server_config: done config len = 808
debug2: parse_server_config: config /etc/ssh/sshd_config len 808
debug3: /etc/ssh/sshd_config:14 setting Protocol 2
debug3: /etc/ssh/sshd_config:32 setting SyslogFacility AUTHPRIV
debug3: /etc/ssh/sshd_config:33 setting LogLevel INFO
debug3: /etc/ssh/sshd_config:38 setting PermitRootLogin no
debug3: /etc/ssh/sshd_config:39 setting StrictModes yes
debug3: /etc/ssh/sshd_config:40 setting MaxAuthTries 6
debug3: /etc/ssh/sshd_config:43 setting PubkeyAuthentication yes
debug3: /etc/ssh/sshd_config:47 setting RhostsRSAAuthentication no
debug3: /etc/ssh/sshd_config:49 setting HostbasedAuthentication no
debug3: /etc/ssh/sshd_config:54 setting IgnoreRhosts yes
debug3: /etc/ssh/sshd_config:58 setting PermitEmptyPasswords no
debug3: /etc/ssh/sshd_config:59 setting PasswordAuthentication yes
debug3: /etc/ssh/sshd_config:63 setting ChallengeResponseAuthentication
no
debug3: /etc/ssh/sshd_config:73 setting GSSAPIAuthentication yes
debug3: /etc/ssh/sshd_config:75 setting GSSAPICleanupCredentials yes
debug3: /etc/ssh/sshd_config:86 setting UsePAM yes
debug3: /etc/ssh/sshd_config:89 setting AcceptEnv LANG LC_CTYPE
LC_NUMERIC LC_TIME LC_COLLATE LC_MONETARY LC_MESSAGES
debug3: /etc/ssh/sshd_config:90 setting AcceptEnv LC_PAPER LC_NAME
LC_ADDRESS LC_TELEPHONE LC_MEASUREMENT
debug3: /etc/ssh/sshd_config:91 setting AcceptEnv LC_IDENTIFICATION
LC_ALL
debug3: /etc/ssh/sshd_config:93 setting GatewayPorts no
debug3: /etc/ssh/sshd_config:94 setting X11Forwarding no
debug3: /etc/ssh/sshd_config:97 setting PrintMotd yes
debug3: /etc/ssh/sshd_config:98 setting PrintLastLog no
debug3: /etc/ssh/sshd_config:104 setting ClientAliveInterval 900
debug3: /etc/ssh/sshd_config:105 setting ClientAliveCountMax 0
debug3: /etc/ssh/sshd_config:114 setting Banner
/etc/ssh/sshd_banner_pcm
debug3: /etc/ssh/sshd_config:117 setting Subsystem sftp
/usr/libexec/openssh/sftp-server
debug1: sshd version OpenSSH_5.9p1
debug3: Incorrect RSA1 identifier
debug1: read PEM private key done: type RSA
debug1: private host key: #0 type 1 RSA
debug3: Incorrect RSA1 identifier
debug1: read PEM private key done: type DSA
debug1: private host key: #1 type 2 DSA
debug1: rexec_argv[0]='/usr/sbin/sshd'
debug1: rexec_argv[1]='-ddd'
debug1: rexec_argv[2]='-p'
debug1: rexec_argv[3]='2022'
debug3: oom_adjust_setup
Set /proc/self/oom_adj from 0 to -17
debug2: fd 3 setting O_NONBLOCK
debug3: sock_set_v6only: set socket 3 IPV6_V6ONLY
debug1: Bind to port 2022 on ::.
Server listening on :: port 2022.
debug2: fd 4 setting O_NONBLOCK
debug1: Bind to port 2022 on 0.0.0.0.
Server listening on 0.0.0.0 port 2022.

Regards,
Carol

-- 
Configure bugmail: https://bugzilla.mindrot.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2008

--- Comment #3 from Darren Tucker <dtucker at zip.com.au> 2012-05-19
15:00:47 EST ---
The log does not show any problems binding to IPv6 or IPv4, so the
problem is either a) you have something else listening on port 22 other
than the newly-build sshd, or the log messages in your syslog are not
from the newly-built sshd (possibly the original vendor-supplied one?)

-- 
Configure bugmail: https://bugzilla.mindrot.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2008

--- Comment #4 from Darren Tucker <dtucker at zip.com.au> 2012-06-01
10:20:25 EST ---
As I asked before: "Are you sure you don't still have the old sshd or
something else listening on port 22?  immediately before starting the
sshd, do you get any "LISTEN" states from "lsof -i :22"
?"

-- 
Configure bugmail: https://bugzilla.mindrot.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2008

--- Comment #5 from kchen001 at alcatel-lucent.com 2012-06-04 17:55:17 EST ---
Hi,

1. How do I know whether there're other sshd or something else listten
ning on port 22?
2. About the suggestion "starting the sshd, do you get any
"LISTEN"
states from "lsof -i :22" ?"", would you please clarify how
shall I do
to check the states? If you provide detailed steps would be
appreciated.

Regards,
Carol

-- 
Configure bugmail: https://bugzilla.mindrot.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2008

kchen001 at alcatel-lucent.com changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |kchen001 at alcatel-lucent.com

-- 
Configure bugmail: https://bugzilla.mindrot.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2008

--- Comment #6 from kchen001 at alcatel-lucent.com 2012-06-11 17:17:36 EST ---
Hi,

Any update?

Regards,
Carol

-- 
Configure bugmail: https://bugzilla.mindrot.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2008

--- Comment #7 from Darren Tucker <dtucker at zip.com.au> 2012-06-11
20:50:05 EST ---
did you try running the lsof command I asked for earlier (twice)?

-- 
Configure bugmail: https://bugzilla.mindrot.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2008

--- Comment #8 from kchen001 at alcatel-lucent.com 2012-06-12 11:20:39 EST ---
(In reply to comment #7)
> did you try running the lsof command I asked for earlier (twice)?
Sorry, I didn't quit understand your earlier request. Let me clarify
it:
Shall I do like this:
1. stop sshd
2. run 'lsof -i :22' and send you the result.

Please let me know whether this is the correct procedure you need.

Regards,
Carol

-- 
Configure bugmail: https://bugzilla.mindrot.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2008

--- Comment #9 from Darren Tucker <dtucker at zip.com.au> 2012-06-12
17:24:22 EST ---
lsof is "list open files".  The "-i" means "show me
what's listening on
Internet sockets" (see the lsof man page for mor info).  In this case,
we're interested in what's listening on port 22 other than the sshd you
just installed.

So, yes, stop the sshd you just install, then run "lsof -i :22"
(you'll
probably need to run this as root).  Depending on who or what is
connected to ssh on the machine you may get a bunch of output. but the
only lines we're interested in will have "LISTEN" in them and look
something like this:

$ sudo lsof -i :22
COMMAND  PID    USER   FD   TYPE  DEVICE SIZE/OFF NODE NAME
sshd    2044    root    3u  IPv4   21174      0t0  TCP *:ssh (LISTEN)
sshd    2044    root    4u  IPv6   21176      0t0  TCP *:ssh (LISTEN)

In this example, we have one sshd (pid 2044) listening on IPv4 and IPv6
sockets.  I suspect what you'll see is one line with sshd listening
even after you stop the new sshd you just installed.

In your case, you can just run "sudo lsof -i :22 | grep LISTEN" once
you've stopped sshd and that will show you whether or not there's
something else listening on port 22, and if so what the name of the
binary is.

-- 
Configure bugmail: https://bugzilla.mindrot.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2008

--- Comment #10 from kchen001 at alcatel-lucent.com 2012-06-12 18:12:00 EST ---
Just found a server which had bind error before. And stop sshd then run
the lsof command, there's no listen in them, but there're some
"ESTABLISHED" in them. 

Actually I cannot reproduce this issue, is it helpful? Or would it be
helpful to run lsof when it's reproduced?

Regards,
Carol

-- 
Configure bugmail: https://bugzilla.mindrot.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2008

--- Comment #11 from kchen001 at alcatel-lucent.com 2012-06-14 18:16:25 EST ---
Hi,

If it's hard to analyze this issue, I will keep an eye on it and when
it's reproduced, I'll try to run the "lsof -i :22" command
before start
the sshd service. Is there anything I can do about it?

Regards,
Carol

-- 
Configure bugmail: https://bugzilla.mindrot.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2008

Darren Tucker <dtucker at zip.com.au> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|                            |WORKSFORME

--- Comment #12 from Darren Tucker <dtucker at zip.com.au> 2012-06-17
13:43:50 EST ---
Actually, I don't think there is any issue.  The debug output from your
system shows that it's happily binding correctly, which means that the
log messages you observed are either due to another process binding to
port 22, or that the log messages are in fact from the *old* version of
sshd.

If you do see this again, check that the message is really from the new
sshd and that you don't have another process listening on port 22
(using the lsof command I gave earlier).

-- 
Configure bugmail: https://bugzilla.mindrot.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2008

--- Comment #13 from kchen001 at alcatel-lucent.com 2012-06-20 11:27:15 EST ---
Thanks for your help. Then I'll keep an eye on it. When it's
reproduced, I will use this command "lsof -i :22" to check whether
there's another process bind on it.

Thanks,
Carol

-- 
Configure bugmail: https://bugzilla.mindrot.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.