llvm dev - Aug 2015 - Analysis of a curious ABI bug

Hi,

This email mainly serves as light entertainment. Some expert opinion
on the resolution of the problem will be appreciated.

The MATLAB max function returns the maximum of two positive integers,
and returns the negative integer, given a positive and negative
integer. Classic signed wrapping, you'd think. It's not so simple, as
the problem reproduces only under the following circumstances:
1. LLVM code is calling the max function (muIntScalarMax_sint16 to be precise).
2. The mathutil library is compiled without debugging information.
3. This happens when the entire world has been built with XCode 6+, on a Mac.

More information: The LLVM IR is exactly the same between an XCode 5
sandbox, an XCode 6 sandbox, and a GNU/Linux sandbox. The
corresponding assembly diff is also clean.

The corresponding C++ code (is actually specialized with short) is:

  template <typename T>
  struct IntMaxFtor {
    FORCEINLINE T operator()(T a, T b) const
    {
      return (a >= b ? a : b);
    }
    FORCEINLINE T operator()(T a, double b) const
    {
      return ((b > a) ?
          IntRoundAndSaturateNoNaNCheck<T>(b) : a);
    }
    FORCEINLINE T operator()(double a, T b) const
    {
      return ((a >= b) ?
          IntRoundAndSaturateNoNaNCheck<T>(a) : b);
    }
    typedef T result_type;
  };


While in lldb, the assembly instructions look like:

--------------- [XCode 5]
    frame #0: 0x0000000108af1930 libmwmathutil.dylib`muIntScalarMax_sint16
libmwmathutil.dylib`muIntScalarMax_sint16:
-> 0x108af1930:  pushq  %rbp
   0x108af1931:  movq   %rsp, %rbp
   0x108af1934:  cmpw   %si, %di
   0x108af1937:  cmovgew %di, %si
   0x108af193b:  movswl %si, %eax
   0x108af193e:  popq   %rbp
   0x108af193f:  retq

--------------- [XCode 6]
    frame #0: 0x0000000108d1ac10 libmwmathutil.dylib`muIntScalarMax_sint16
libmwmathutil.dylib`muIntScalarMax_sint16:
-> 0x108d1ac10:  pushq  %rbp
   0x108d1ac11:  movq   %rsp, %rbp
   0x108d1ac14:  cmpl   %esi, %edi
   0x108d1ac16:  cmovgew %di, %si
   0x108d1ac1a:  movswl %si, %eax
   0x108d1ac1d:  popq   %rbp
   0x108d1ac1e:  retq

---------------
So, it's comparing the extended registers, but cmpl/cmpw difference is
messing up somehow.

After execution of the cmpl/cmpw:

--------------- [XCode 5]
(lldb) register read --format int16_t[] di
      di = {-48}
(lldb) register read --format int16_t[] si
      si = {34}
(lldb) register read --format int16_t[] edi
     edi = {-48 7103}
(lldb) register read --format int16_t[] esi
     esi = {34 7103}
(lldb) register read --format b rflags
  rflags = 0b0000000000000000000000000000000000000000000000000000001010010010

--------------- [XCode 6]
(lldb) register read --format int16_t[] di
      di = {-48}
(lldb) register read --format int16_t[] si
      si = {34}
(lldb) register read --format int16_t[] edi
     edi = {-48 0}
(lldb) register read --format int16_t[] esi
     esi = {34 0}
(lldb) register read --format b rflags
  rflags = 0b0000000000000000000000000000000000000000000000000000001000010010

(sign flag is different)

--------------- [XCode 6, with C code calling max]
(lldb) register read --format int16_t[] di
      di = {-48}
(lldb) register read --format int16_t[] si
      si = {34}
(lldb) register read --format int16_t[] edi
     edi = {-48 -1}
                    ^
         In the LLVM call, this is zero

(lldb) register read --format int16_t[] esi
     esi = {34 0}

In conclusion, this is a Clang/LLVM version mismatch. It's compiling
muIntScalarMax_sint16 to emit a compl instead of a compw. The fact
that the bug only reproduces with LLVM is because: LLVM doesn't set up
the extended form of the register correctly for negative numbers, when
calling a function with a word-sized register.

The SysV ABI does not guarantee that the i16 will be sext'ed to i32.
Clang ABI is probably an enhancement over the SysV ABI, and LLVM 3.5
doesn't know about this.

Some pointers on how to patch LLVM 3.5 for this would be appreciated.

Thanks.

Ram