openssh bugs - Oct 2023 - [Bug 3629] New: Building with Clang-17 fails due to -fzero-call-used-regs

https://bugzilla.mindrot.org/show_bug.cgi?id=3629

            Bug ID: 3629
           Summary: Building with Clang-17 fails due to
                    -fzero-call-used-regs
           Product: Portable OpenSSH
           Version: 9.5p1
          Hardware: amd64
                OS: Mac OS X
            Status: NEW
          Severity: critical
          Priority: P5
         Component: Build system
          Assignee: unassigned-bugs at mindrot.org
          Reporter: debohman at gmail.com

I am using the public LLVM / Clang 17 release as my toolchain. I
noticed https://bugzilla.mindrot.org/show_bug.cgi?id=3475, where there
was a problem with the use of -fzero-call-used-regs, which was worked
around for Clang 15 and the Apple Clang.

This problem is with Clang 17, and to resolve it, I had to completely
remove the usage of -fzero-call-used-regs.

The issue occurs when compiling ssh-ecdsa-sk.c. It results in:

cc: error: clang frontend command failed with exit code 139

clang version 17.0.2
Target: x86_64-apple-darwin16.7.0
Thread model: posix
InstalledDir: /usr/local/bin

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3629

Damien Miller <djm at mindrot.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |djm at mindrot.org

--- Comment #1 from Damien Miller <djm at mindrot.org> ---
That looks like a compiler ICE. Please file a bug with the clang/llvm
project - I'm sure they would like to know that their compiler is
crashing.

Could you try editing the Makefile and changing
-fzero-call-used-regs=all to -fzero-call-used-regs=used and see if that
helper?

(maybe, having been burned by this before twice, we should make this
the default for this option?)

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3629

Damien Miller <djm at mindrot.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |morbo at google.com

--- Comment #2 from Damien Miller <djm at mindrot.org> ---
+Bill Wendling who looked at the last instance of this AFAIK

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3629

--- Comment #3 from David Bohman <debohman at gmail.com> ---
(In reply to Damien Miller from comment #1)
> That looks like a compiler ICE. Please file a bug with the
> clang/llvm project - I'm sure they would like to know that their
> compiler is crashing.
> 
> Could you try editing the Makefile and changing
> -fzero-call-used-regs=all to -fzero-call-used-regs=used and see if
> that helper?
> 
> (maybe, having been burned by this before twice, we should make this
> the default for this option?)
Yes, I tried using the fix for Clang-15, but the problem persisted. I
modified the configure.ac to not use -fzero-call-used-regs at all for
Clang-17, and the project built successfully.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3629

--- Comment #4 from David Bohman <debohman at gmail.com> ---
Filed bug against clang
https://github.com/llvm/llvm-project/issues/69794.

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3629

Darren Tucker <dtucker at dtucker.net> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |dtucker at dtucker.net

--- Comment #5 from Darren Tucker <dtucker at dtucker.net> ---
(In reply to David Bohman from comment #3)
> Yes, I tried using the fix for Clang-15, but the problem persisted.
> I modified the configure.ac to not use -fzero-call-used-regs at all
> for Clang-17, and the project built successfully.
does -fzero-call-used-regs=used work?  (as opposed to
-fzero-call-used-regs=all)

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3629

--- Comment #6 from David Bohman <debohman at gmail.com> ---
(In reply to Darren Tucker from comment #5)
> (In reply to David Bohman from comment #3)
> > Yes, I tried using the fix for Clang-15, but the problem persisted.
> > I modified the configure.ac to not use -fzero-call-used-regs at all
> > for Clang-17, and the project built successfully.
> 
> does -fzero-call-used-regs=used work?  (as opposed to
> -fzero-call-used-regs=all)
No, I tried that before completely removing -fzero-call-used-regs.

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3629

Darren Tucker <dtucker at dtucker.net> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         Resolution|---                         |FIXED
             Status|NEW                         |RESOLVED
             Blocks|                            |3628

--- Comment #7 from Darren Tucker <dtucker at dtucker.net> ---
OK,
https://github.com/openssh/openssh-portable/commit/ff220d4010717f7bfbbc02a2400666fb9d24f250
stops using -fzero-call-used-regs on all Apple clang versions.  Please
reopen if this does not solve your problem.

We can allowlist specific versions that are know to work if people
report those.


Referenced Bugs:

https://bugzilla.mindrot.org/show_bug.cgi?id=3628
[Bug 3628] tracking bug for openssh-9.6
-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3629

--- Comment #8 from Darren Tucker <dtucker at dtucker.net> ---
Sigh.  Looks like clang is not the only compiler to have problems with
this flag: gcc 11 on mips and mipsel (at least on OpenWRT, not sure if
it's specific to that or not) also does:

cc -g -O2 -pipe -Wno-error=format-truncation -Wall -Wpointer-arith
-Wuninitialized -Wsign-compare -Wformat-security
-Wsizeof-pointer-memaccess -Wno-pointer-sign -Wno-unused-result
-Wimplicit-fallthrough -Wmisleading-indentation -fno-strict-aliasing
-D_FORTIFY_SOURCE=2 -ftrapv -fzero-call-used-regs=used
-fno-builtin-memset -fstack-protector-strong   -fPIC -I. -I.. -I.
-I./..  -D_XOPEN_SOURCE=600 -D_BSD_SOURCE -D_DEFAULT_SOURCE
-D_GNU_SOURCE -DHAVE_CONFIG_H -c arc4random_uniform.c
during RTL pass: final
arc4random_uniform.c: In function 'arc4random_uniform':
arc4random_uniform.c:63:1: internal compiler error: in
mips_output_move, at config/mips/mips.c:5327
   63 | }
      | ^
Please submit a full bug report,
with preprocessed source if appropriate.
See <https://dev.openwrt.org/> for instructions.
{standard input}: Assembler messages:
{standard input}: Warning: missing .end at end of assembly
{standard input}: Error: open CFI at the end of file; missing
.cfi_endproc directive

$ cc -v
Using built-in specs.
COLLECT_GCC=cc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/mipsel-openwrt-linux-musl/11.2.0/lto-wrapper
Target: mipsel-openwrt-linux-musl
Configured with:
/builder/shared-workdir/build/sdk/build_dir/target-mipsel_24kc_musl/gcc-11.2.0/configure
--target=mipsel-openwrt-linux --host=mipsel-openwrt-linux
--build=x86_64-pc-linux-gnu --program-prefix= --program-suffix--prefix=/usr
--exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin
--libexecdir=/usr/lib --sysconfdir=/etc --datadir=/usr/share
--localstatedir=/var --mandir=/usr/man --infodir=/usr/info
--disable-nls 'CXXFLAGS_FOR_TARGET=-g -O2
-D_GLIBCXX_INCLUDE_NEXT_C_HEADERS' --build=x86_64-pc-linux-gnu
--host=mipsel-openwrt-linux-musl --target=mipsel-openwrt-linux-musl
--enable-languages=c,c++ --with-bugurl=https://dev.openwrt.org/
--with-pkgversion='OpenWrt GCC 11.2.0' --enable-shared
--disable-__cxa_atexit --with-default-libstdcxx-abi=gcc4-compatible
--enable-target-optspace --with-gnu-ld --disable-nls
--disable-libsanitizer --disable-libvtv --disable-libcilkrts
--disable-libmudflap --disable-libmpx --disable-multilib
--disable-libgomp --disable-libquadmath --disable-libssp
--disable-decimal-float --disable-libstdcxx-pch
--with-host-libstdcxx=-lstdc++ --prefix=/usr --libexecdir=/usr/lib
--with-local-prefix=/usr --with-stage1-ldflags=-lstdc++
--with-float=soft
Thread model: posix
Supported LTO compression algorithms: zlib zstd
gcc version 11.2.0 (OpenWrt GCC 11.2.0) 

We can reduce this to a fairly minimal testcase:

$ cat conftest.c 
unsigned int
arc4random_uniform(unsigned int upper_bound)
{
    return arc4random() % upper_bound;
}

$ cc -O -fzero-call-used-regs=used -c conftest.c 
conftest.c: In function 'arc4random_uniform':
conftest.c:4:16: warning: implicit declaration of function 'arc4random'
[-Wimplicit-function-declaration]
    4 |         return arc4random() % upper_bound;
      |                ^~~~~~~~~~
during RTL pass: final
conftest.c:5:1: internal compiler error: in mips_output_move, at
config/mips/mips.c:5327
    5 | }
      | ^

Prior to my change, the AC_COMPILE_IFELSE test in
OSSH_CHECK_CFLAG_COMPILE was sufficient to catch the problem with
-fzero-call-used-regs=all, but does not catch it with
-fzero-call-used-regs=used.  Expanding the testcase  in
OSSH_CHECK_CFLAG_COMPILE does seem to help:

-void f(int n) {}
+int f(int n) {return rand() % n;}

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3629

--- Comment #9 from David Bohman <debohman at gmail.com> ---
I cherry-picked 2a19e02 ff220d4 99a2df5 on top of V_9_5_P1, and it does
build successfully on my system.

Are there security implications associated with not using this flag?

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3629

--- Comment #10 from Darren Tucker <dtucker at dtucker.net> ---
(In reply to David Bohman from comment #9)
> I cherry-picked 2a19e02 ff220d4 99a2df5 on top of V_9_5_P1, and it
> does build successfully on my system.
Thanks, good to hear!
> Are there security implications associated with not using this flag?
Not immediately.  If there is a security bug in future this flag (and
similar hardening flags) may prevent a given exploit from working, or
make it less likely to work, but the exact effect would depend on the
details of the bug and exploit in question.

This flag makes it harder to write ROP exploits, where an attacker
chains together little fragments of existing code that end in a
"return" (called "gadgets") in a binary.  The attacker finds
gadgets
that together do what they want and fakes up a call stack that returns
to each gadget in turn.  The flag zeros (some) registers before
functions return, which may disrupt the intended behaviour.

https://www.jerkeby.se/newsletter/posts/rop-reduction-zero-call-user-regs/
seems like a reasonable explanation.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3629

--- Comment #11 from David Bohman <debohman at gmail.com> ---
Thank you for the detailed note, I am reading the article now. It is
about what I expected. Unfortunately, the llvm / clang folks were not
able to reproduce the problem with this flag on Linux.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
You are watching someone on the CC list of the bug.

https://bugzilla.mindrot.org/show_bug.cgi?id=3629

--- Comment #12 from Bill Wendling <morbo at google.com> ---
(In reply to David Bohman from comment #11)
> Thank you for the detailed note, I am reading the article now. It is
> about what I expected. Unfortunately, the llvm / clang folks were
> not able to reproduce the problem with this flag on Linux.
I'm going to try again on my MacBook Pro. It'll take a bit though as
it's slooowww.

-- 
You are receiving this mail because:
You are watching someone on the CC list of the bug.
You are watching the assignee of the bug.