openssh bugs - Jun 2024 - [Bug 3702] New: sshd fork crashed when compiled with seccomp

https://bugzilla.mindrot.org/show_bug.cgi?id=3702

            Bug ID: 3702
           Summary: sshd fork crashed when compiled with seccomp
           Product: Portable OpenSSH
           Version: 9.7p1
          Hardware: ARM64
                OS: Linux
            Status: NEW
          Severity: minor
          Priority: P5
         Component: sshd
          Assignee: unassigned-bugs at mindrot.org
          Reporter: root at nixsum.net

Created attachment 3819
  --> https://bugzilla.mindrot.org/attachment.cgi?id=3819&action=edit
strace

Hello,

After compiling on raspbian OS, with the seccomp sandbox enabled by
default the server program does not accept connections. When compiled
without the sandbox all is good.
When inspecting closer it looks like the fork of sshd is killed by
SIGSYS signal due to an access violation.

Kernel log at the time of crash:

[147024.127628] audit: type=1326 audit(1718443919.577:9): auid=1000
uid=103 gid=65534 ses=298 pid=17516 comm="sshd"
exe="/home/pi/openssh-9.7p1/sshd" sig=31 arch=40000028 syscall=384
compat=1 ip=0xf798d330 code=0x0

I am also attaching the strace of sshd and its children.

I tested this on x86 debian with the same setup, same seccomp kernel
parameters as far as I know them, and the issue does not occur.

This looks like something specific to the arm kernel (I tested both the
64bit and 32bit kernels), but I cannot tell what exactly.

And one last thing, I found the "EPERM (Operation not permitted)"
events in the strace output to be misleading, as they appeared when I
straced on x86 debian as well.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3702

Darren Tucker <dtucker at dtucker.net> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |dtucker at dtucker.net

--- Comment #1 from Darren Tucker <dtucker at dtucker.net> ---
Can you test this with -DSANDBOX_SECCOMP_FILTER_DEBUG (eg ./configure
--with-cflags=-DSANDBOX_SECCOMP_FILTER_DEBUG" or uncomment it in
sandbox-seccomp-filter.c)?  That'll log the syscall it was trying to do
when it failed.   (Note that this is not safe for use in production).

The kernel log you supplied says "syscall=384", but
/usr/include/asm-generic/unistd.h on an armbian system here lists that
range as "/* 295 through 402 are unassigned to sync up with generic
numbers, don't use */".

The only hit I get for 384 is on x86-64:

$ grep 384 -r /usr/include/ | grep _NR_
/usr/include/asm/unistd_32.h:#define __NR_arch_prctl 384

but armbian has prctl as 167:
/usr/include/asm-generic/unistd.h:#define __NR_prctl 167

so I'm not sure what to make of your kernel log.

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3702

--- Comment #2 from Nikola <root at nixsum.net> ---
Yes, here are the results from what I understand.

sshd logs:

debug1: inetd sockets after dupping: 4, 4
debug3: process_channel_timeouts: setting 0 timeouts
debug3: channel_clear_timeouts: clearing
Connection from ::1 port 46784 on ::1 port 222 rdomain ""
debug1: Local version string SSH-2.0-OpenSSH_9.7
debug1: Remote protocol version 2.0, remote software version
OpenSSH_9.2p1 Raspbian-2+deb12u2
debug1: compat_banner: match: OpenSSH_9.2p1 Raspbian-2+deb12u2 pat
OpenSSH* compat 0x04000000
debug2: fd 4 setting O_NONBLOCK
debug3: ssh_sandbox_init: preparing seccomp filter sandbox
debug2: Network child is on pid 13980
debug3: preauth child monitor started
debug3: privsep user:group 103:65534 [preauth]
debug1: permanently_set_uid: 103/65534 [preauth]
debug3: ssh_sandbox_child_debugging: installing SIGSYS handler
[preauth]
debug3: ssh_sandbox_child: setting PR_SET_NO_NEW_PRIVS [preauth]
debug3: ssh_sandbox_child: attaching seccomp filter program [preauth]
debug1: monitor_read_log: child log fd closed
debug3: mm_request_receive: entering
debug1: do_cleanup
debug1: Killing privsep child 13980
debug3: oom_adjust_setup
debug1: Set /proc/self/oom_score_adj from 0 to -1000
debug2: fd 4 setting O_NONBLOCK
debug1: Bind to port 222 on 0.0.0.0.
Server listening on 0.0.0.0 port 222.
debug2: fd 5 setting O_NONBLOCK
debug3: sock_set_v6only: set socket 5 IPV6_V6ONLY
debug1: Bind to port 222 on ::.
Server listening on :: port 222.
debug3: fd 6 is not O_NONBLOCK
debug1: Forked child 13999.
debug3: send_rexec_state: entering fd = 9 config len 3153
debug3: ssh_msg_send: type 0
debug3: send_rexec_state: done
debug3: oom_adjust_restore
debug1: Set /proc/self/oom_score_adj to 0
debug1: rexec start in 6 out 6 newsock 6 pipe 8 sock 9
debug1: inetd sockets after dupping: 4, 4
debug3: process_channel_timeouts: setting 0 timeouts
debug3: channel_clear_timeouts: clearing
Connection from ::1 port 37764 on ::1 port 222 rdomain ""
debug1: Local version string SSH-2.0-OpenSSH_9.7
debug1: Remote protocol version 2.0, remote software version
OpenSSH_9.2p1 Raspbian-2+deb12u2
debug1: compat_banner: match: OpenSSH_9.2p1 Raspbian-2+deb12u2 pat
OpenSSH* compat 0x04000000
debug2: fd 4 setting O_NONBLOCK
debug3: ssh_sandbox_init: preparing seccomp filter sandbox
debug2: Network child is on pid 14000
debug3: preauth child monitor started
debug3: privsep user:group 103:65534 [preauth]
debug1: permanently_set_uid: 103/65534 [preauth]
debug3: ssh_sandbox_child_debugging: installing SIGSYS handler
[preauth]
debug3: ssh_sandbox_child: setting PR_SET_NO_NEW_PRIVS [preauth]
debug3: ssh_sandbox_child: attaching seccomp filter program [preauth]
debug1: monitor_read_log: child log fd closed
debug3: mm_request_receive: entering
debug1: do_cleanup
debug1: Killing privsep child 14000

After more careful inspection, the kernel log seems to sometimes
indicate syscall "384" and at other times "20" when
initiating an ssh
connection:

[ 5106.975827] audit: type=1326 audit(1718651226.051:2): auid=1000
uid=103 gid=65534 ses=8 pid=8620 comm="sshd"
exe="/home/pi/openssh-9.7p1/sshd" sig=31 arch=40000028 syscall=384
compat=1 ip=0xf7a4d330 code=0x0
[ 5159.653285] audit: type=1326 audit(1718651278.730:3): auid=1000
uid=103 gid=65534 ses=8 pid=8651 comm="sshd"
exe="/home/pi/openssh-9.7p1/sshd" sig=31 arch=40000028 syscall=384
compat=1 ip=0xf74bd330 code=0x0
[ 5332.463045] audit: type=1326 audit(1718651451.531:4): auid=1000
uid=103 gid=65534 ses=8 pid=8711 comm="sshd"
exe="/home/pi/openssh-9.7p1/sshd" sig=31 arch=40000028 syscall=384
compat=1 ip=0xf744d330 code=0x0
[ 5435.206654] audit: type=1326 audit(1718651554.277:5): auid=1000
uid=103 gid=65534 ses=8 pid=8746 comm="sshd"
exe="/home/pi/openssh-9.7p1/sshd" sig=31 arch=40000028 syscall=20
compat=1 ip=0xf779252c code=0x0
[ 5483.156487] audit: type=1326 audit(1718651602.225:6): auid=1000
uid=103 gid=65534 ses=8 pid=8780 comm="sshd"
exe="/home/pi/openssh-9.7p1/sshd" sig=31 arch=40000028 syscall=20
compat=1 ip=0xf7b0252c code=0x0
[ 5930.773091] audit: type=1326 audit(1718652049.838:7): auid=1000
uid=103 gid=65534 ses=8 pid=8842 comm="sshd"
exe="/home/pi/openssh-9.7p1/sshd" sig=31 arch=40000028 syscall=384
compat=1 ip=0xf749d330 code=0x0
[ 6229.482998] audit: type=1326 audit(1718652348.541:8): auid=1000
uid=103 gid=65534 ses=8 pid=8934 comm="sshd"
exe="/home/pi/openssh-9.7p1/sshd" sig=31 arch=40000028 syscall=20
compat=1 ip=0xf743252c code=0x0
[ 6585.188845] audit: type=1326 audit(1718652704.242:9): auid=1000
uid=103 gid=65534 ses=8 pid=16018 comm="sshd"
exe="/home/pi/openssh-9.7p1/sshd" sig=31 arch=40000028 syscall=20
compat=1 ip=0xf7a0252c code=0x0
[ 6796.206919] audit: type=1326 audit(1718652915.250:10): auid=1000
uid=103 gid=65534 ses=8 pid=23058 comm="sshd"
exe="/home/pi/openssh-9.7p1/sshd" sig=31 arch=40000028 syscall=20
compat=1 ip=0xf7a8252c code=0x0

Although I'm not sure whether to trust that either.

Another thing I've noticed is that procfs shows processes performing a
syscall that is not implemented according to the header file and
multiple other sources I found online.

$ sudo cat /proc/1/syscall
252 0x4 0x18f69a8 0x8e 0xffffffff 0xffffffff 0x8e 0xffcce428 0xf75e7158
$ 
$ grep 252 /usr/include/asm-generic/unistd.h 
$ 
$ sudo cat /proc/13994/syscall
336 0x2425a08 0x2 0x0 0xffa76818 0x8 0x2 0xffa76528 0xf742297c
$ 
$ grep 336 /usr/include/asm-generic/unistd.h 
$

I don't know if this is due to some sort of incorrect mapping, I will
investigate further and see what I find.

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3702

Damien Miller <djm at mindrot.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |djm at mindrot.org

--- Comment #3 from Damien Miller <djm at mindrot.org> ---
What is you compilation environment? Did you cross-compile? Are you
using the standard toolchain and headers that ships with Rasbian?

I have a couple of raspberry pi around, if you can provide version and
software information then I can replicate your build environment.

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3702

--- Comment #4 from Darren Tucker <dtucker at dtucker.net> ---
If you've built with -DSANDBOX_SECCOMP_FILTER_DEBUG you should get a
debug line like this:

    ssh_sandbox_violation: unexpected system call
(arch:0xc000003e,syscall:271 @ 0x7f84be11dc3b) [preauth]

If you're building the current devel version you'll need to either
install sshd-session where sshd expects it or specify the one in your
build directory:

    $ make && sudo `pwd`/sshd -De -ologlevel=debug3 -p 2022
-osshdsessionpath=`pwd`/sshd-session

One other thing: you can ask the compiler to dump the syscall numbers
for you:

    $ cc -dD -E -I. sandbox-seccomp-filter.c | grep '#define __NR'

(As an aside, tt'd be nice if the debugging code could do this for us.)
 On my Armbian system this says 20 is epoll:

#define __NR_epoll_create1 20
#define __NR_epoll_ctl 21
#define __NR_epoll_pwait 22

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3702

--- Comment #5 from Damien Miller <djm at mindrot.org> ---
Created attachment 3820
  --> https://bugzilla.mindrot.org/attachment.cgi?id=3820&action=edit
convert syscall names that we know

(In reply to Darren Tucker from comment #4)
> (As an aside, tt'd be nice if the debugging code could do this for
> us.)  On my Armbian system this says 20 is epoll:
We could convert the syscall numbers that we know (see attached), but
after writing that I realised that it isn't so useful - it's the
syscall numbers that we *don't* know that are typically the problem.

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3702

Darren Tucker <dtucker at dtucker.net> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Attachment #3820|0                           |1
        is obsolete|                            |

--- Comment #6 from Darren Tucker <dtucker at dtucker.net> ---
Created attachment 3821
  --> https://bugzilla.mindrot.org/attachment.cgi?id=3821&action=edit
Provide names for syscalls

(In reply to Damien Miller from comment #5)
[...]
> We could convert the syscall numbers that we know (see attached),
> but after writing that I realised that it isn't so useful - it's
the
> syscall numbers that we *don't* know that are typically the problem.
Yeah I started there too and quickly gave that up for the same reason. 
I ended up extracting the syscalls from part-preprocessed source.  I
chose to use the seccomp file itself because that guarantees it picks
up the same headers, and put it into another compilation unit because
otherwise you get problems due to overwriting the generated file.  That
would be resolvable, but keeping it separate seemed cleaner (well,
slightly less icky).

It looks like:

ssh_sandbox_violation: unexpected system call
(arch:0xc000003e,syscall:271(__NR_ppoll) @ 0x7f5cccd1dc3b) [preauth]

Probably a bit close to release for something this invasive (at least
as far as the build system) but maybe immediately after.

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3702

Darren Tucker <dtucker at dtucker.net> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
   Attachment #3821|0                           |1
        is obsolete|                            |

--- Comment #7 from Darren Tucker <dtucker at dtucker.net> ---
Created attachment 3822
  --> https://bugzilla.mindrot.org/attachment.cgi?id=3822&action=edit
Provide names for syscalls
> That would be resolvable, but keeping it separate seemed cleaner (well,
slightly less icky)
Actually that version might be less icky since it doesn't need a new
source file.

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3702

--- Comment #8 from Nikola <root at nixsum.net> ---
I have realized that the problem lies within my shallow understanding
and I am in deep waters here.

When installing 64bit Raspbian I never knew that only the kernel is
64bit, whereas all the binaries, including the compiler are actually
32bit ARM.

When i run the configure script the below line is seen:

    checking for seccomp architecture... "AUDIT_ARCH_AARCH64"

So in the end it produces a 32bit ARM binary with the 64bit ARM seccomp
sandbox, resulting in isolating the wrong syscalls due to the major
differences between them (or at least that's to what conclusions I
came).

Trying to run ./configure --host=arm reports that seccomp is not
supported.

    checking kernel for seccomp_filter support... no

I was able to fool it by hard coding it, but that's probably considered
a crime and I'm not aware if there's another way to override it:

'''
printf %s "checking for seccomp architecture... " >&6; }
        seccomp_audit_arch        case "$host" in
        x86_64-*)
                seccomp_audit_arch=AUDIT_ARCH_X86_64
                ;;
        i*86-*)
                seccomp_audit_arch=AUDIT_ARCH_I386
                ;;
        arm*-*)
                seccomp_audit_arch=AUDIT_ARCH_ARM
                ;;
        aarch64*-*)
#               seccomp_audit_arch=AUDIT_ARCH_AARCH64
                seccomp_audit_arch=AUDIT_ARCH_ARM
                ;;
'''

With the above "fix" the resulting binary works as intended.

It also works well when I boot the 32bit kernel and then compile, which
is something I did not figure out initially.

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3702

--- Comment #9 from Darren Tucker <dtucker at dtucker.net> ---
(In reply to Nikola from comment #8)
> I have realized that the problem lies within my shallow
> understanding and I am in deep waters here.
> 
> When installing 64bit Raspbian I never knew that only the kernel is
> 64bit, whereas all the binaries, including the compiler are actually
> 32bit ARM.
That's.. unfortunate.  What does ./config.guess and gcc -dumpmachine
report?

[...]
> Trying to run ./configure --host=arm reports that seccomp is not
> supported.
You need to provide a full "triplet", or at least enough of one to
match the case statement, eg:

$ ./config.guess 
aarch64-unknown-linux-gnu

$ ./configure --host=arm-linux-gnueabihf | grep AUDIT_ARCH
checking for seccomp architecture... "AUDIT_ARCH_ARM"
> I was able to fool it by hard coding it, but that's probably
> considered a crime and I'm not aware if there's another way to
> override it:
The other alternative is you could explicitly disable the sandbox
(--without-sandbox) or specify the fallback rlimit sandbox
(--with-sandbox=rlimit).
> It also works well when I boot the 32bit kernel and then compile,
> which is something I did not figure out initially.
Maybe we could add a check to make sure sizeof(long)==8 as a sanity
check, either in configure or at runtime.

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3702

--- Comment #10 from Damien Miller <djm at mindrot.org> ---
Repeating this so it doesn't lost: I'd really like to reproduce this
myself, as Rasbian is a popular platform. What hardware an OS version
are you using?

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3702

--- Comment #11 from Nikola <root at nixsum.net> ---
Yes, apologies for not providing this initially.

It's a Raspberry pi 4 model B 4gb.

OS is the official 64bit raspbian image.

https://downloads.raspberrypi.com/raspios_lite_arm64/images/raspios_lite_arm64-2024-03-15/2024-03-15-raspios-bookworm-arm64-lite.img.xz

You can also find it at
https://www.raspberrypi.com/software/operating-systems/
it's listed as 
"Raspberry Pi OS Lite
Release date: March 15th 2024
System: 64-bit"

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.