llvm dev - Apr 2021 - Incrementally compiling LLVM

This flow is being used successfully by both npcomp and circt, which are
llvm incubator projects.  You should be able to leverage/build off of the
github actions flows that they define.  see https://github.com/llvm/circt
and https://github.com/llvm/mlir-npcomp

Steve

On Wed, Apr 14, 2021 at 12:59 PM Francesco Bertolaccini via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> On 14/04/2021 21:18, David Blaikie wrote:
> > Sounds like something's going wrong for sure. How are you
measuring
> > the time? (how much time is it taking) and what are you testing by
> > "partial recompilation"? Touching an ADT .h file is likely
to
> > recompile nearly everything so might not be much better than a clean
> > build - but touching a .cpp file (especially a .cpp file for a single
> > tool, rather than a library) might be quite quick. So try timing a
> > full clean build (ninja -t clean, ninja clang) and then incrementally
> > touching just a .cpp file (eg: clang/tools/driver/driver.cpp  ) and
> > see how they compare
>
> I currently do not have sufficient resources to compile LLVM on my
> personal machine, so I am (ab?)using GitHub Actions. My workflow is
> setup such that the build artifacts are cached and restored on every
> worflow run. The time is measured automatically by GitHub itself, and is
> around ~2h for each run, whether from scratch or starting from the cache.
>
> I don't generally edit .h files directly, unless they are generated by
> modifying TableGen files.
>
> The fact that someone else has successfully been able to do incremental
> builds _does_ make it seem like it's a configuration issue on my part.
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20210414/e0edd5b7/attachment.html>

On 15/04/2021 03:36, Stephen Neuendorffer wrote:
> This flow is being used successfully by both npcomp and circt, which are
> llvm incubator projects.  You should be able to leverage/build off of
> the github actions flows that they define. 
> see https://github.com/llvm/circt <https://github.com/llvm/circt>
> and https://github.com/llvm/mlir-npcomp
> <https://github.com/llvm/mlir-npcomp>
> 
> Steve
> 
I have not yet tried the ccache solution mentioned before, but I tried
changing my flow to use the Visual Studio generator instead of Ninja,
like the linked projects do, and it still takes a long while to do a
recompilation, even though I just re-run the job
(https://github.com/frabert/llvm-project/runs/2354303561)

I am now wondering: does the fact that I am *not* enabling the host
target affect something? I am only interested in the Mips target so it's
the only one I enable, thinking it was going to speed up compilation,
but maybe I am not doing myself any favors?

Thanks to everyone so far, I am going to try ccache next

Francesco

On Wed, Apr 14, 2021 at 6:36 PM Stephen Neuendorffer via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> This flow is being used successfully by both npcomp and circt, which are
> llvm incubator projects.  You should be able to leverage/build off of the
> github actions flows that they define.  see https://github.com/llvm/circt
> and https://github.com/llvm/mlir-npcomp
>
Is it the same flow?
As far as I can tell you're not getting a build directory in order to do an
incremental rebuild with modified sources, but instead getting the build
artifacts in order to build a downstream project entirely.

The only way I can see incremental compilation to work would be to get the
entire source tree from the cache alongside with the build tree, and then
run `git fetch && git checkout <rev>`. This would mark only the
source file
changed by git during the checkout.
Otherwise a fresh `git clone` will have the source files modified data as
the time of the checkout, so more recent than the cached build dir.

-- 
Mehdi


>
> Steve
>
> On Wed, Apr 14, 2021 at 12:59 PM Francesco Bertolaccini via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>> On 14/04/2021 21:18, David Blaikie wrote:
>> > Sounds like something's going wrong for sure. How are you
measuring
>> > the time? (how much time is it taking) and what are you testing by
>> > "partial recompilation"? Touching an ADT .h file is
likely to
>> > recompile nearly everything so might not be much better than a
clean
>> > build - but touching a .cpp file (especially a .cpp file for a
single
>> > tool, rather than a library) might be quite quick. So try timing a
>> > full clean build (ninja -t clean, ninja clang) and then
incrementally
>> > touching just a .cpp file (eg: clang/tools/driver/driver.cpp  )
and
>> > see how they compare
>>
>> I currently do not have sufficient resources to compile LLVM on my
>> personal machine, so I am (ab?)using GitHub Actions. My workflow is
>> setup such that the build artifacts are cached and restored on every
>> worflow run. The time is measured automatically by GitHub itself, and
is
>> around ~2h for each run, whether from scratch or starting from the
cache.
>>
>> I don't generally edit .h files directly, unless they are generated
by
>> modifying TableGen files.
>>
>> The fact that someone else has successfully been able to do incremental
>> builds _does_ make it seem like it's a configuration issue on my
part.
>>
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20210415/00114bf3/attachment.html>