openssh bugs - Apr 2019 - [Bug 2993] New: core dump during reboot, if sshd 8.0p1 was started.

https://bugzilla.mindrot.org/show_bug.cgi?id=2993

            Bug ID: 2993
           Summary: core dump during reboot, if sshd 8.0p1 was started.
           Product: Portable OpenSSH
           Version: 8.0p1
          Hardware: amd64
                OS: Linux
            Status: NEW
          Severity: minor
          Priority: P5
         Component: sshd
          Assignee: unassigned-bugs at mindrot.org
          Reporter: aleksey.moroz at gmail.com

linux mint 18.2

$ uname -mvr
4.15.0-47-generic #50~16.04.1-Ubuntu SMP Fri Mar 15 16:06:21 UTC 2019
x86_64 (but all the same on the other versions of the kernel, i'd used
for simplicity the latest stable one)

$ openssl version
LibreSSL 2.9.1

i'd tried three variants of starting sshd: with systemd (just copied
proper (and edited in accordance with my system) files from
openssh-server-7.2p2.deb /etc & /lib), from rc.local
("/usr/local/sbin/sshd -4 -f /etc/ssh/sshd_config &") and from
command
line with sudo - results are the same (and it does not depend was sshd
killed after start manually or not).

reboot goes almost to the end, as i may see, all partitions are
unmounted before the last screen.
this "screen of the death" is not always the same, sometimes before
register's values there are the last lines with the list of the network
modules of the kernel.

if sshd is not started, reboot proceed seamlessly.

occasional observation: if sshd starts as service via systemd, it could
not be managed neither from systemd-manager-pkexec, nor from command
line (i.e kill & pkill does not work in this case).

to be totally sure, i'd restored tonight a backup from two months ago,
installed libressl 2.9.1, ldns 1.7, opensshd 8.0p1 and got the same sad
result.

donno how to save core dump to file.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2993

admzzz <aleksey.moroz at gmail.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         Resolution|---                         |WONTFIX
             Status|NEW                         |RESOLVED

--- Comment #1 from admzzz <aleksey.moroz at gmail.com> ---
i'd extracted binary sshd 7.2p2 from the deb, put it in the /opt and
during the start of it got an error message about the absence of
/var/run/sshd.
after required mkdir sshd 7.2p2 started, worked and reboot fine,
moreover, 8.0p1 began permit reboot without core dump. ;-)

i'd installed 8.0p1 via checkinstall, maybe it would make sense to add
the creation of /var/run/sshd to installation's script? ;-)

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2993

--- Comment #2 from admzzz <aleksey.moroz at gmail.com> ---
Created attachment 3263
  --> https://bugzilla.mindrot.org/attachment.cgi?id=3263&action=edit
photo of kernel panic

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2993

admzzz <aleksey.moroz at gmail.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         Resolution|WONTFIX                     |---
             Status|RESOLVED                    |REOPENED

--- Comment #3 from admzzz <aleksey.moroz at gmail.com> ---
i'd closed bug too early: 8.0p1 goes to the same kernel panic, if it
was started from rc.local and was not killed manually before reboot.


clear reboot was only once - when i'd started and killed it from the
command line (check of successful start|kill is making only with
netstat -tunaple)

...guess that the same mistake will occur with the start via systemd...

any ideas/suggestions?

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2993

Darren Tucker <dtucker at dtucker.net> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |dtucker at dtucker.net

--- Comment #4 from Darren Tucker <dtucker at dtucker.net> ---
As a general rule, if a userspace program like sshd can cause a kernel
panic that indicates a problem with the kernel.

Anyway, I do have a proposed experiment for you: run the stock sshd
*and* your new sshd on a different port (eg /usr/local/sbin/sshd -d -p
222) then reboot and see if it panics.  I'm wondering if systemd is
somehow getting out of sync and waiting for sshd to stop even though it
already has and the kernel eventually shoots pid 1.

BTW:
> ("/usr/local/sbin/sshd -4 -f /etc/ssh/sshd_config &")
The & is not needed, sshd will daemonize itself after startup.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2993

--- Comment #5 from admzzz <aleksey.moroz at gmail.com> ---
as to the kernel, 4.15.0-47 is the latest stock's one (i.e. from linux
mint repository) now, but the problem does not depend from the kernel
version, it occurs on the old 4.4 and on experimental 5.0.5 kernels as
well.

excuse me, i didn't catch your idea - you suggest to run both sshd
simultaneously or separately, one by one?

and i didn't mention that i'm using 443 port in all my experiments:

$ cat /etc/ssh/sshd_config|grep Port
Port 443
#GatewayPorts no

i'd totally cleared systemd related staff, and right now both sshd
reboots clear if they were started from command line (after "mkdir
/var/run/sshd" in case of 7.2p2). but 8.0p1 causes panic if it was
started from rc.local.

may i ask for a proper/clear examples of systemd related files for
8.0p1?

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2993

--- Comment #6 from Darren Tucker <dtucker at dtucker.net> ---
(In reply to admzzz from comment #5)
[...]
> excuse me, i didn't catch your idea - you suggest to run both sshd
> simultaneously or separately, one by one?
1) Install the stock sshd without running the new sshd and make sure it
reboots OK.

2) Start new locally built sshd in addition to the stock sshd and
reboot.

3) Replace the stock sshd with the locally built one and reboot.

Basically we're trying to tell whether the difference is the absence of
the stock sshd rather than the presence of the locally built one.
> and i didn't mention that i'm using 443 port in all my experiments:
that shouldn't matter as long as the 2nd sshd uses a different port.
> may i ask for a proper/clear examples of systemd related files for
> 8.0p1?
Sorry, I don't know a great deal about systemd configs.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2993

admzzz <aleksey.moroz at gmail.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|REOPENED                    |RESOLVED
         Resolution|---                         |FIXED

--- Comment #7 from admzzz <aleksey.moroz at gmail.com> ---
your points 1) and  2) works fine, 3) caused panic: 8.0p1 does not
reboot gracefully, if it was started from the systemd's 7.2p2
environment. though if both sshd was started simultaneously from
command line, reboot went clearly.

so, the problem is in the systemd's 7.2p2 files only.

i'd deleted all systemd's files except ssh.service and ssh.socket in
/lib/systemd/system, and reboot works fine at last.

thank you for your help and answers.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=2993

Damien Miller <djm at mindrot.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|RESOLVED                    |CLOSED

--- Comment #8 from Damien Miller <djm at mindrot.org> ---
close bugs that were resolved in OpenSSH 8.5 release cycle

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.