flac dev - Mar 2015 - crash on lpc_restore_signal_16_intrin_sse2

On Mon, Mar 9, 2015 at 5:15 PM, lvqcl <lvqcl.mail at gmail.com>
wrote:
> Janne Hyv?rinen wrote:
>
>> I compiled flac.exe without NASM and verified that
>> FLAC__lpc_restore_signal_16_intrin_sse2 is used. Decoding the same file
>> that made VLC 2.2 crash decoded without issues. Would be nice if VLC
was
>> compilable with MSVC and we could use its debugger.
>>
>
> VLC 2.2.0 crashes here inside WinXP VM on instruction:
>
>      movaps  xmmword ptr [esp+10h], xmm7
>
> And ESP is equal to 0x02A8FCC8. The stack is not properly aligned
> for some reason, yet GCC uses movaps instruction.
Sounds a bit like:
http://www.peterstock.co.uk/games/mingw_sse/

Also worth noting, VLC uses yasm but not nasm, so we may be able to
sidestep this problem by adding nasm to our toolchain. That said, it
still seems like other people using MinGW and libFLAC will get bit by
this so I'll keep digging.

Tristan Matthews wrote:
> Sounds a bit like:
> http://www.peterstock.co.uk/games/mingw_sse/
>
> Also worth noting, VLC uses yasm but not nasm, so we may be able to
> sidestep this problem by adding nasm to our toolchain.
Then you're probably not guaranteed from crashes inside
FLAC__lpc_restore_signal_wide_intrin_sse41() function. Looks like it
also requires aligned stack when compiled with GCC.
(libFLAC uses it to decode 24-bit flac files)

> That said, it
> still seems like other people using MinGW and libFLAC will get bit by
> this so I'll keep digging.
Great.

Unable to test on win32 at the moment, please give this a try. Feedback welcome.

Avoids crashes due to unaligned ops when built with mingw.
---
 src/libFLAC/include/private/cpu.h | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/src/libFLAC/include/private/cpu.h
b/src/libFLAC/include/private/cpu.h
index 8927897..bd40012 100644
--- a/src/libFLAC/include/private/cpu.h
+++ b/src/libFLAC/include/private/cpu.h
@@ -72,8 +72,13 @@
     #define FLAC__FMA_SUPPORTED 1
   #endif
 #elif defined __GNUC__
+  #ifdef __MINGW32__
+    #define FLAC__STACK_ALIGN __attribute__((force_align_arg_pointer))
+  #else
+    #define FLAC__STACK_ALIGN
+  #endif /* SSE stack alignment fixups */
   #if (__GNUC__ > 4 || (__GNUC__ == 4 && __GNUC_MINOR__ >= 9)) /*
since GCC 4.9 -msse.. compiler options aren't necessary */
-    #define FLAC__SSE_TARGET(x) __attribute__ ((__target__ (x)))
+    #define FLAC__SSE_TARGET(x) __attribute__ ((__target__ (x)))
FLAC__STACK_ALIGN
     #define FLAC__SSE_SUPPORTED 1
     #define FLAC__SSE2_SUPPORTED 1
     #define FLAC__SSSE3_SUPPORTED 1
@@ -82,7 +87,7 @@
     #define FLAC__AVX2_SUPPORTED 1
     #define FLAC__FMA_SUPPORTED 1
   #else /* for GCC older than 4.9 */
-    #define FLAC__SSE_TARGET(x)
+    #define FLAC__SSE_TARGET(x) FLAC__STACK_ALIGN
     #ifdef __SSE__
       #define FLAC__SSE_SUPPORTED 1
     #endif
-- 
2.1.0