llvm dev - Apr 2021 - Incrementally compiling LLVM

Good point.  I don't think it's the same flow....sorry for the
confusion.

Steve


On Thu, Apr 15, 2021, 11:36 AM Mehdi AMINI <joker.eph at gmail.com> wrote:
>
>
> On Wed, Apr 14, 2021 at 6:36 PM Stephen Neuendorffer via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>> This flow is being used successfully by both npcomp and circt, which
are
>> llvm incubator projects.  You should be able to leverage/build off of
the
>> github actions flows that they define.  see
https://github.com/llvm/circt
>> and https://github.com/llvm/mlir-npcomp
>>
>
> Is it the same flow?
> As far as I can tell you're not getting a build directory in order to
do
> an incremental rebuild with modified sources, but instead getting the build
> artifacts in order to build a downstream project entirely.
>
> The only way I can see incremental compilation to work would be to get the
> entire source tree from the cache alongside with the build tree, and then
> run `git fetch && git checkout <rev>`. This would mark only
the source file
> changed by git during the checkout.
> Otherwise a fresh `git clone` will have the source files modified data as
> the time of the checkout, so more recent than the cached build dir.
>
> --
> Mehdi
>
>
>
>>
>> Steve
>>
>> On Wed, Apr 14, 2021 at 12:59 PM Francesco Bertolaccini via llvm-dev
<
>> llvm-dev at lists.llvm.org> wrote:
>>
>>> On 14/04/2021 21:18, David Blaikie wrote:
>>> > Sounds like something's going wrong for sure. How are you
measuring
>>> > the time? (how much time is it taking) and what are you
testing by
>>> > "partial recompilation"? Touching an ADT .h file is
likely to
>>> > recompile nearly everything so might not be much better than a
clean
>>> > build - but touching a .cpp file (especially a .cpp file for a
single
>>> > tool, rather than a library) might be quite quick. So try
timing a
>>> > full clean build (ninja -t clean, ninja clang) and then
incrementally
>>> > touching just a .cpp file (eg: clang/tools/driver/driver.cpp 
) and
>>> > see how they compare
>>>
>>> I currently do not have sufficient resources to compile LLVM on my
>>> personal machine, so I am (ab?)using GitHub Actions. My workflow is
>>> setup such that the build artifacts are cached and restored on
every
>>> worflow run. The time is measured automatically by GitHub itself,
and is
>>> around ~2h for each run, whether from scratch or starting from the
cache.
>>>
>>> I don't generally edit .h files directly, unless they are
generated by
>>> modifying TableGen files.
>>>
>>> The fact that someone else has successfully been able to do
incremental
>>> builds _does_ make it seem like it's a configuration issue on
my part.
>>>
>>> _______________________________________________
>>> LLVM Developers mailing list
>>> llvm-dev at lists.llvm.org
>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20210416/1a1f0ab5/attachment-0001.html>

On 16/04/2021 16:57, Stephen Neuendorffer wrote:
> Good point.  I don't think it's the same flow....sorry for the
confusion.
> 
> Steve
> 
> 
> On Thu, Apr 15, 2021, 11:36 AM Mehdi AMINI <joker.eph at gmail.com
> <mailto:joker.eph at gmail.com>> wrote:
> 
>     Is it the same flow?
>     As far as I can tell you're not getting a build directory in order
>     to do an incremental rebuild with modified sources, but instead
>     getting the build artifacts in order to build a downstream project
>     entirely.
> 
>     The only way I can see incremental compilation to work would be to
>     get the entire source tree from the cache alongside with the build
>     tree, and then run `git fetch && git checkout <rev>`.
This would
>     mark only the source file changed by git during the checkout.
>     Otherwise a fresh `git clone` will have the source files modified
>     data as the time of the checkout, so more recent than the cached
>     build dir.
> 
>     -- 
>     Mehdi
> 
Thanks guys, I finally figured it out. Caching the whole git repo +
build artifacts was the missing piece.

Cheers!

Francesco