llvm dev - Feb 2016 - [FYI] CMake's Ninja generator is non-deterministic

It is with great sadness that I must tell everyone CMake’s Ninja generator is
non-deterministic (https://public.kitware.com/Bug/view.php?id=15968
<https://public.kitware.com/Bug/view.php?id=15968>).

I’m not sure if this impacts all versions of CMake, but it certainly impacts all
the recent releases. You might ask why this matters? Sadly the non-determinism
*does* impact determinism in the final builds. I haven’t fully dug into the
extent of the differences, but I was seeing non-reproducibility in LTO builds.
Whatever Ninja is doing differently causes LTO to make different optimization
decisions. I have verified that this does not impact CMake’s makefile generator.

The important takeaway from this is that if you are building for a purpose where
non-determinism can impact your results, don’t use Ninja. For most engineering
tasks building and testing locally Ninja, as always, works great. For
incremental bots testing correctness, no problem.

Any bots testing performance need to move to a deterministic generator (i.e.
Unix Makefiles). Anyone building or testing a release of LLVM or clang for
packaging purposes should also use a deterministic generator.

Sorry to be the bearer of bad news…

-Chris
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160210/43c0a7fb/attachment.html>

From what I see reproducing offline, it is the order of dependencies that is
changing (I bet they iterate over an unordered container):

Example:

< build cmake_order_depends_target_llvm-profdata: phony ||
lib/libLLVMSupport.a lib/libLLVMCore.a lib/libLLVMBitReader.a lib/libLLVMMC.a
lib/libLLVMMCParser.a lib/libLLVMObject.a lib/libLLVMProfileData.a
---
> build cmake_order_depends_target_llvm-profdata: phony ||
lib/libLLVMProfileData.a lib/libLLVMSupport.a lib/libLLVMCore.a
lib/libLLVMBitReader.a lib/libLLVMMC.a lib/libLLVMMCParser.a lib/libLLVMObject.a

The consequence is the order of the libraries is not the same on the command
line.
Have you seen other differences?

You may want to update the bug report with these information if you think it
could be useful.


-- 
Mehdi


> On Feb 10, 2016, at 10:01 AM, Chris Bieneman via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> 
> It is with great sadness that I must tell everyone CMake’s Ninja generator
is non-deterministic (https://public.kitware.com/Bug/view.php?id=15968
<https://public.kitware.com/Bug/view.php?id=15968>).
> 
> I’m not sure if this impacts all versions of CMake, but it certainly
impacts all the recent releases. You might ask why this matters? Sadly the
non-determinism *does* impact determinism in the final builds. I haven’t fully
dug into the extent of the differences, but I was seeing non-reproducibility in
LTO builds. Whatever Ninja is doing differently causes LTO to make different
optimization decisions. I have verified that this does not impact CMake’s
makefile generator.
> 
> The important takeaway from this is that if you are building for a purpose
where non-determinism can impact your results, don’t use Ninja. For most
engineering tasks building and testing locally Ninja, as always, works great.
For incremental bots testing correctness, no problem.
> 
> Any bots testing performance need to move to a deterministic generator
(i.e. Unix Makefiles). Anyone building or testing a release of LLVM or clang for
packaging purposes should also use a deterministic generator.
> 
> Sorry to be the bearer of bad news…
> 
> -Chris
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160210/bb0b679d/attachment.html>

Brad King has already replied with a patch that removes many of the differences
(but not all). It was iterating over a std::map keyed on pointers.

-Chris
> On Feb 10, 2016, at 2:12 PM, Mehdi Amini <mehdi.amini at apple.com>
wrote:
> 
> From what I see reproducing offline, it is the order of dependencies that
is changing (I bet they iterate over an unordered container):
> 
> Example:
> 
> < build cmake_order_depends_target_llvm-profdata: phony ||
lib/libLLVMSupport.a lib/libLLVMCore.a lib/libLLVMBitReader.a lib/libLLVMMC.a
lib/libLLVMMCParser.a lib/libLLVMObject.a lib/libLLVMProfileData.a
> ---
> > build cmake_order_depends_target_llvm-profdata: phony ||
lib/libLLVMProfileData.a lib/libLLVMSupport.a lib/libLLVMCore.a
lib/libLLVMBitReader.a lib/libLLVMMC.a lib/libLLVMMCParser.a lib/libLLVMObject.a
> 
> 
> The consequence is the order of the libraries is not the same on the
command line.
> Have you seen other differences?
> 
> You may want to update the bug report with these information if you think
it could be useful.
> 
> 
> -- 
> Mehdi
> 
> 
> 
>> On Feb 10, 2016, at 10:01 AM, Chris Bieneman via llvm-dev <llvm-dev
at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote:
>> 
>> It is with great sadness that I must tell everyone CMake’s Ninja
generator is non-deterministic (https://public.kitware.com/Bug/view.php?id=15968
<https://public.kitware.com/Bug/view.php?id=15968>).
>> 
>> I’m not sure if this impacts all versions of CMake, but it certainly
impacts all the recent releases. You might ask why this matters? Sadly the
non-determinism *does* impact determinism in the final builds. I haven’t fully
dug into the extent of the differences, but I was seeing non-reproducibility in
LTO builds. Whatever Ninja is doing differently causes LTO to make different
optimization decisions. I have verified that this does not impact CMake’s
makefile generator.
>> 
>> The important takeaway from this is that if you are building for a
purpose where non-determinism can impact your results, don’t use Ninja. For most
engineering tasks building and testing locally Ninja, as always, works great.
For incremental bots testing correctness, no problem.
>> 
>> Any bots testing performance need to move to a deterministic generator
(i.e. Unix Makefiles). Anyone building or testing a release of LLVM or clang for
packaging purposes should also use a deterministic generator.
>> 
>> Sorry to be the bearer of bad news…
>> 
>> -Chris
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
<http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160210/5692f48b/attachment.html>