openssh bugs - May 2022 - [Bug 3430] New: 64 bit time and seccomp conflict

https://bugzilla.mindrot.org/show_bug.cgi?id=3430

            Bug ID: 3430
           Summary: 64 bit time and seccomp conflict
           Product: Portable OpenSSH
           Version: 8.9p1
          Hardware: ARM
                OS: Linux
            Status: NEW
          Severity: major
          Priority: P5
         Component: sshd
          Assignee: unassigned-bugs at mindrot.org
          Reporter: alacki93 at gmail.com

I found that glibc with support for 64-bit time could crash sshd
process with enabled seccomp seccomp. Test environment:
-Kernel 4.14.x
-32 bit ARM CPU
-glibc 2.34
-OpenSSH 8.9p1
-Toolchain: GCC 10

Syscall wrappers like a clock_gettime
(https://github.com/bminor/glibc/blob/glibc-2.34/sysdeps/unix/sysv/linux/clock_gettime.c)
first run syscall dedicated for kernels that support 64 bit time. If
this syscall failed with ENOSYS error, then glibc calls traditional
variant for 32 bit time.

OpenSSH in source code tries to figure which syscalls are supported by
kernel and only supported syscalls are added as an allowed in seccomp
(https://github.com/openssh/openssh-portable/blob/master/sandbox-seccomp-filter.c).
Because kernel 4.14 doesn?t support syscalls for 64 bit time,
__NR_clock_gettime64 is not added as a trusted syscall to seccomp. As a
result OpenSSH process is killed by seccomp every time when it tries to
use clock_gettime from glibc.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3430

Darren Tucker <dtucker at dtucker.net> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |dtucker at dtucker.net

--- Comment #1 from Darren Tucker <dtucker at dtucker.net> ---
I don't follow your description of what's happening.  The sandbox code
has this (and has had since about 8.6p1, see bz#3093):

#ifdef __NR_clock_gettime64
        SC_ALLOW(__NR_clock_gettime64),
#endif

(In reply to Lacky from comment #0)
[...]
> OpenSSH in source code tries to figure which syscalls are supported
> by kernel and only supported syscalls are added as an allowed in
> seccomp
That's not an accurate description.  It allows any of the syscalls in
its list for which there is a definition, regardless of whether or not
it's supported by the currently running kernel.

If you build an sshd against and older set of headers that does not
define __NR_clock_gettime64 then it will not be included, but that's
because sshd has no way of knowing about it or what the syscall number
is at compile time.  I could imagine this biting you if you installed a
new libc.so without recompiling sshd with the new headers.

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3430

--- Comment #2 from Lacky <alacki93 at gmail.com> ---
Created attachment 3592
  --> https://bugzilla.mindrot.org/attachment.cgi?id=3592&action=edit
Log from sshd -D -ddd

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3430

--- Comment #3 from Lacky <alacki93 at gmail.com> ---
Created attachment 3593
  --> https://bugzilla.mindrot.org/attachment.cgi?id=3593&action=edit
Log from strace -f

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3430

--- Comment #4 from Lacky <alacki93 at gmail.com> ---
(In reply to Darren Tucker from comment #1)
> 
> If you build an sshd against and older set of headers that does not
> define __NR_clock_gettime64 then it will not be included, but that's
> because sshd has no way of knowing about it or what the syscall
> number is at compile time.  I could imagine this biting you if you
> installed a new libc.so without recompiling sshd with the new
> headers.
Yes, but glibc doesn?t check it. They define syscalls in source code
(https://github.com/bminor/glibc/blob/glibc-2.34/sysdeps/unix/sysv/linux/arm/arch-syscall.h).

Now what happens on described test environment. Every time when sshd
tries to use clock_gettime from glibc you can find in strace something
like this: 

4628  clock_gettime64(CLOCK_BOOTTIME, 0xbe8c1488) = -1 ENOSYS (Function
not implemented) 
4628  clock_gettime(CLOCK_BOOTTIME, {tv_sec=247, tv_nsec=653301168}) 0 

Glibc first tries to use clock_gettime for 64 bit time. This fails
because kernel 4.14 doesn?t have support for this syscall (ENOSYS).
Then glibc tries to use classic clock_gettime. Usually this is not a
problem, but sshd uses seccomp. As you said sshd doesn?t add
clock_gettime64 as an allowed syscall to seccomp because it cannot find
it in kernel headers. Now let?s look at strace log. This entry means
that seccomp was initialized successfully for process with PID 4637: 

4637  prctl(PR_SET_SECCOMP, SECCOMP_MODE_FILTER, {len=107,
filter=0x4ee324} <unfinished ...> 
4637  <... prctl resumed>)              = 0 

After initialization of seccomp, process 4637 is killed when it tries
use unallowed clock_gettime64: 

4637  clock_gettime64(CLOCK_BOOTTIME,  <unfinished ...> 
4637  <... clock_gettime64 resumed> <unfinished ...>) = ? 
4637  +++ killed by SIGSYS +++ 
4628  --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_KILLED, si_pid=4637,
si_uid=101, si_status=SIGSYS, si_utime=4, si_stime=0} --- 
4628  kill(4637, SIGKILL) 
>From user perspective it looks like connection with server was closed
before any password prompt. 

In attachments you can find log from sshd (run with ?-D ?ddd") and
strace (run with ?-f?). 

Temporary solution to fix this bugs are: 
-Building OpenSSH without seccomp
-Building kernel without seccomp 
-Patching glibc to remove clock_gettime64 syscall. 

Disabling seccomp is the easiest workaround, but it decreases security
of sshd.

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3430

--- Comment #5 from Darren Tucker <dtucker at dtucker.net> ---
(In reply to Lacky from comment #4)
> As you said sshd doesn?t add
> clock_gettime64 as an allowed syscall to seccomp because it cannot
> find it in kernel headers.
This is the root of the problem.  Why isn't __NR_clock_gettime64
defined?

I have a Debian 11 32bit ARM system here with glibc 2.31 and it has it:

$ uname -a
Linux tock 5.16.10-bone14 #1bullseye PREEMPT Tue Feb 22 00:07:39 UTC
2022 armv7l GNU/Linux

$ sudo dpkg -l | grep libc-bin
ii  libc-bin    2.31-13+deb11u3     armhf     GNU C Library: Binaries

$ grep -r __NR_clock_gettime64 /usr/include/*
[...]
/usr/include/asm-generic/unistd.h:#define __NR_clock_gettime64 403
/usr/include/asm-generic/unistd.h:__SYSCALL(__NR_clock_gettime64,
sys_clock_gettime)

I confirmed it's set in sshd (to 403) by adding a debug call.

What distro are you using?

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3430

--- Comment #6 from Lacky <alacki93 at gmail.com> ---
(In reply to Darren Tucker from comment #5)
> This is the root of the problem.  Why isn't __NR_clock_gettime64
> defined?
> 
> I have a Debian 11 32bit ARM system here with glibc 2.31 and it has
> it:
> 
> $ uname -a
> Linux tock 5.16.10-bone14 #1bullseye PREEMPT Tue Feb 22 00:07:39 UTC
> 2022 armv7l GNU/Linux
> 
Support for 64-bit time in 32 bit platforms has been added in kernel
5.1
(https://www.phoronix.com/scan.php?page=news_item&px=Linux-5.1-Year-2038-Syscalls).
Older kernels (like 4.14.x) that doesn't have this feature are still
supported by kernel developers.
> 
> What distro are you using?
This is Buildroot-based Linux (https://buildroot.org/).

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3430

--- Comment #7 from Darren Tucker <dtucker at dtucker.net> ---
(In reply to Lacky from comment #6)
[...]
> Support for 64-bit time in 32 bit platforms has been added in kernel
> 5.1
The kernel version you're running should be irrelevant.  If you were
running the same binary on a kernel that had 64bit time you'd have the
same problem because the headers don't have __NR_clock_gettime64 and
thus sshd didn't add it to the filter.
> > What distro are you using?
> 
> This is Buildroot-based Linux (https://buildroot.org/).
Sorry, I don't think there's anything OpenSSH could do differently. 
You've got a glibc that's making system calls that aren't in the
exported list of system calls.  I think your buildroot should be
defining __NR_clock_gettime64 if its glibc is going to call it.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3430

Tabor Kelly <taborkelly at gmail.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |taborkelly at gmail.com

--- Comment #8 from Tabor Kelly <taborkelly at gmail.com> ---
I ran into this on openssh 8.9p1 with glibc 2.35 on kernel 4.14 running
on 32 bit arm. I "fixed" it with a stupid patch in
sandbox-seccomp-filter.c, but I think this is actually a glibc bug.

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3430

--- Comment #9 from Tabor Kelly <taborkelly at gmail.com> ---
Created attachment 3836
  --> https://bugzilla.mindrot.org/attachment.cgi?id=3836&action=edit
My quick hack to get around what is probably a glibc bug

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3430

Rustam Abdullaev <rustamabd at gmail.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |rustamabd at gmail.com

--- Comment #10 from Rustam Abdullaev <rustamabd at gmail.com> ---
I've run into this issue on a 32-bit ARM platform with Ubuntu 22.04 and
kernel 6.1.36 (via buildroot).

Only the sshd seccomp fails with "SIGSYS, Bad system call". All other
invocations of clock_gettime work fine.

Core was generated by `sshd-session: [net]'.
Program terminated with signal SIGSYS, Bad system call.
#0  0xb6b349a6 in ?? () from /usr/lib/libc.so.6
#1  0xb6b95892 in __clock_gettime64 () from /usr/lib/libc.so.6
#2  0xb6b9593c in clock_gettime () from /usr/lib/libc.so.6
#3  0x00496b1c in ?? ()

Building with BR2_PACKAGE_OPENSSH_SANDBOX=n "fixes" the issue. So
there's definitely something wrong with seccomp on a 32-bit ARM
platform.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3430

--- Comment #11 from Darren Tucker <dtucker at dtucker.net> ---
(In reply to Rustam Abdullaev from comment #10)
[...]
> Building with BR2_PACKAGE_OPENSSH_SANDBOX=n "fixes" the issue. So
> there's definitely something wrong with seccomp on a 32-bit ARM
> platform.
As I said in comment#7, one possible reason for this is if the headers
do not define __NR_clock_gettime64 at compile time. 
sandbox-seccomp-filter.c has this:

#ifdef __NR_clock_gettime64
        SC_ALLOW(__NR_clock_gettime64),
#endif

so if the headers don't define __NR_clock_gettime64 the sandbox won't
know to allow it.  This is independent of whether or not clock_gettime
works.  If you run ./configure --with-cflags=-D__NR_clock_gettime64=403
&& make and it works then that's likely the problem.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3430

--- Comment #12 from Rustam Abdullaev <rustamabd at gmail.com> ---
We're using the official ARM toolchain 11.2 from
https://developer.arm.com/downloads/-/arm-gnu-toolchain-downloads

Even the latest one,
arm-gnu-toolchain-14.3.rel1-x86_64-arm-none-linux-gnueabihf doesn't
have __NR_clock_gettime64 defined in unistd.h.

The ARM toolchain seems to be including kernel headers 4.20.13 ?

$ cat arm-none-linux-gnueabihf/libc/usr/include/linux/version.h
#define LINUX_VERSION_CODE 267277
#define KERNEL_VERSION(a,b,c) (((a) << 16) + ((b) << 8) + (c))

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.