thr3ads.net - llvm dev - [LLVMdev] Non-deterministic builds [Nov 2010]

If this information is useful, please help other people find it:
Share via:

Erik Cederstrand

2010-Nov-12 14:26 UTC

[LLVMdev] Non-deterministic builds

Hello LLVM'ers

I have noticed that two consecutive builds of clang, clang++ and tblgen
don't produce identical binaries (as in md5 sums) on identical source code
(I'm on FreeBSD). I ran strings(1) on the two clang binaries, and I get the
following:

248400,248403c248400,248403
<
N128_GLOBAL__N__usr_home_erik_freebsd_head_src_lib_clang_libllvmcore_.._.._.._contrib_llvm_lib_VMCore_Verifier.cpp_00000000_36FAEB5D11PreVerifierE
<
N128_GLOBAL__N__usr_home_erik_freebsd_head_src_lib_clang_libllvmcore_.._.._.._contrib_llvm_lib_VMCore_Verifier.cpp_00000000_36FAEB5D8VerifierE
<
N4llvm11InstVisitorIN128_GLOBAL__N__usr_home_erik_freebsd_head_src_lib_clang_libllvmcore_.._.._.._contrib_llvm_lib_VMCore_Verifier.cpp_00000000_36FAEB5D8VerifierEvEE
<
N128_GLOBAL__N__usr_home_erik_freebsd_head_src_lib_clang_libllvmcore_.._.._.._contrib_llvm_lib_VMCore_Verifier.cpp_00000000_36FAEB5D7TypeSetE
--->
N128_GLOBAL__N__usr_home_erik_freebsd_head_src_lib_clang_libllvmcore_.._.._.._contrib_llvm_lib_VMCore_Verifier.cpp_00000000_95F4F51B11PreVerifierE
>
N128_GLOBAL__N__usr_home_erik_freebsd_head_src_lib_clang_libllvmcore_.._.._.._contrib_llvm_lib_VMCore_Verifier.cpp_00000000_95F4F51B8VerifierE
>
N4llvm11InstVisitorIN128_GLOBAL__N__usr_home_erik_freebsd_head_src_lib_clang_libllvmcore_.._.._.._contrib_llvm_lib_VMCore_Verifier.cpp_00000000_95F4F51B8VerifierEvEE
>
N128_GLOBAL__N__usr_home_erik_freebsd_head_src_lib_clang_libllvmcore_.._.._.._contrib_llvm_lib_VMCore_Verifier.cpp_00000000_95F4F51B7TypeSetE248735,248736c248735,248736
<
N135_GLOBAL__N__usr_home_erik_freebsd_head_src_lib_clang_libllvmcore_.._.._.._contrib_llvm_lib_VMCore_PrintModulePass.cpp_00000000_E8B12D4D15PrintModulePassE
<
N135_GLOBAL__N__usr_home_erik_freebsd_head_src_lib_clang_libllvmcore_.._.._.._contrib_llvm_lib_VMCore_PrintModulePass.cpp_00000000_E8B12D4D17PrintFunctionPassE
--->
N135_GLOBAL__N__usr_home_erik_freebsd_head_src_lib_clang_libllvmcore_.._.._.._contrib_llvm_lib_VMCore_PrintModulePass.cpp_00000000_BDCFB9C615PrintModulePassE
>
N135_GLOBAL__N__usr_home_erik_freebsd_head_src_lib_clang_libllvmcore_.._.._.._contrib_llvm_lib_VMCore_PrintModulePass.cpp_00000000_BDCFB9C617PrintFunctionPassE248855c248855
<
N131_GLOBAL__N__usr_home_erik_freebsd_head_src_lib_clang_libllvmcore_.._.._.._contrib_llvm_lib_VMCore_PassManager.cpp_00000000_B138034413BBPassManagerE
--->
N131_GLOBAL__N__usr_home_erik_freebsd_head_src_lib_clang_libllvmcore_.._.._.._contrib_llvm_lib_VMCore_PassManager.cpp_00000000_9C5AE4C313BBPassManagerE249398c249398
<
N124_GLOBAL__N__usr_home_erik_freebsd_head_src_lib_clang_libllvmcore_.._.._.._contrib_llvm_lib_VMCore_Pass.cpp_00000000_16144DA716GetCFGOnlyPassesE
--->
N124_GLOBAL__N__usr_home_erik_freebsd_head_src_lib_clang_libllvmcore_.._.._.._contrib_llvm_lib_VMCore_Pass.cpp_00000000_6C67211416GetCFGOnlyPassesE251862c251862
<
N4llvm3sys11ThreadLocalIKN144_GLOBAL__N__usr_home_erik_freebsd_head_src_lib_clang_libllvmsupport_.._.._.._contrib_llvm_lib_Support_CrashRecoveryContext.cpp_00000000_BFDB511124CrashRecoveryContextImplEEE
--->
N4llvm3sys11ThreadLocalIKN144_GLOBAL__N__usr_home_erik_freebsd_head_src_lib_clang_libllvmsupport_.._.._.._contrib_llvm_lib_Support_CrashRecoveryContext.cpp_00000000_A7307A7424CrashRecoveryContextImplEEE

clang++ and tblgen produce similar diffs.

I'm trying to build a system to detect the actual binary impact on specific
commits to FreeBSD, so things like this is noise for me. Also, things like
running ccache on top of clang is a lot easier if ccache can just use the
checksum of /usr/bin/clang to verify that it can use the cache.

Would it be possible to unify the above output as to create deterministic builds
of clang, clang++ and tblgen?

I believe the binaries are created with binutils-as and binutils-ld, if it
matters.

Thanks,
Erik Cederstrand
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 1928 bytes
Desc: not available
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20101112/7378e939/attachment.bin>

Sebastian Redl

2010-Nov-12 15:10 UTC

head link

[LLVMdev] Non-deterministic builds

On 12.11.2010 15:26, Erik Cederstrand wrote:> I have noticed that two consecutive builds of clang, clang++ and tblgen
don't produce identical binaries (as in md5 sums) on identical source code
(I'm on FreeBSD). I ran strings(1) on the two clang binaries, and I get the
following:
> 251862c251862
> < 
N4llvm3sys11ThreadLocalIKN144_GLOBAL__N__usr_home_erik_freebsd_head_src_lib_clang_libllvmsupport_.._.._.._contrib_llvm_lib_Support_CrashRecoveryContext.cpp_00000000_BFDB511124CrashRecoveryContextImplEEE
> ---
>>
N4llvm3sys11ThreadLocalIKN144_GLOBAL__N__usr_home_erik_freebsd_head_src_lib_clang_libllvmsupport_.._.._.._contrib_llvm_lib_Support_CrashRecoveryContext.cpp_00000000_A7307A7424CrashRecoveryContextImplEEE
>Urgh, are we using random elements in the unnamed namespace symbol 
generation?

Sebastian

John Regehr

2010-Nov-12 16:46 UTC

head link

[LLVMdev] Non-deterministic builds

I've seen compiler bugs + ASLR resulting in non-deterministic builds, 
could that be the culprit?

John Regehr

Chris Lattner

2010-Nov-12 17:19 UTC

head link

[LLVMdev] Non-deterministic builds

Yes, this is controlled by the -frandom-seed flag.  Sadly, it is expected
behavior.

-Chris

On Nov 12, 2010, at 7:10 AM, Sebastian Redl <sebastian.redl at
getdesigned.at> wrote:
> On 12.11.2010 15:26, Erik Cederstrand wrote:
>> I have noticed that two consecutive builds of clang, clang++ and tblgen
don't produce identical binaries (as in md5 sums) on identical source code
(I'm on FreeBSD). I ran strings(1) on the two clang binaries, and I get the
following:
>> 251862c251862
>> < 
N4llvm3sys11ThreadLocalIKN144_GLOBAL__N__usr_home_erik_freebsd_head_src_lib_clang_libllvmsupport_.._.._.._contrib_llvm_lib_Support_CrashRecoveryContext.cpp_00000000_BFDB511124CrashRecoveryContextImplEEE
>> ---
>>>
N4llvm3sys11ThreadLocalIKN144_GLOBAL__N__usr_home_erik_freebsd_head_src_lib_clang_libllvmsupport_.._.._.._contrib_llvm_lib_Support_CrashRecoveryContext.cpp_00000000_A7307A7424CrashRecoveryContextImplEEE
>> 
> Urgh, are we using random elements in the unnamed namespace symbol 
> generation?
> 
> Sebastian
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev

John McCall

2010-Nov-12 18:38 UTC

head link

[LLVMdev] Non-deterministic builds

On Nov 12, 2010, at 7:10 AM, Sebastian Redl wrote:
> On 12.11.2010 15:26, Erik Cederstrand wrote:
>> I have noticed that two consecutive builds of clang, clang++ and tblgen
don't produce identical binaries (as in md5 sums) on identical source code
(I'm on FreeBSD). I ran strings(1) on the two clang binaries, and I get the
following:
>> 251862c251862
>> < 
N4llvm3sys11ThreadLocalIKN144_GLOBAL__N__usr_home_erik_freebsd_head_src_lib_clang_libllvmsupport_.._.._.._contrib_llvm_lib_Support_CrashRecoveryContext.cpp_00000000_BFDB511124CrashRecoveryContextImplEEE
>> ---
>>>
N4llvm3sys11ThreadLocalIKN144_GLOBAL__N__usr_home_erik_freebsd_head_src_lib_clang_libllvmsupport_.._.._.._contrib_llvm_lib_Support_CrashRecoveryContext.cpp_00000000_A7307A7424CrashRecoveryContextImplEEE
>> 
> Urgh, are we using random elements in the unnamed namespace symbol 
> generation?

This is definitely not the direct output of clang.  We mangle anonymous
namespaces the same way that GCC does, i.e. 12_GLOBAL__N_1.  If there's a
compiler that mangles filenames into anonymous namespace names — or anything
later in the toolchain which does it — I'm not aware of it.

Erik, what compiler are you using?

John.

Reasonably Related Threads

Search for more apparently analagous threads

llvm dev - Nov 2010 - [LLVMdev] Non-deterministic builds

[LLVMdev] Non-deterministic builds

[LLVMdev] Non-deterministic builds

[LLVMdev] Non-deterministic builds

[LLVMdev] Non-deterministic builds

[LLVMdev] Non-deterministic builds

Reasonably Related Threads