llvm dev - Apr 2021 - Incrementally compiling LLVM

On Wed, Apr 14, 2021 at 6:36 PM Stephen Neuendorffer via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> This flow is being used successfully by both npcomp and circt, which are
> llvm incubator projects.  You should be able to leverage/build off of the
> github actions flows that they define.  see https://github.com/llvm/circt
> and https://github.com/llvm/mlir-npcomp
>
Is it the same flow?
As far as I can tell you're not getting a build directory in order to do an
incremental rebuild with modified sources, but instead getting the build
artifacts in order to build a downstream project entirely.

The only way I can see incremental compilation to work would be to get the
entire source tree from the cache alongside with the build tree, and then
run `git fetch && git checkout <rev>`. This would mark only the
source file
changed by git during the checkout.
Otherwise a fresh `git clone` will have the source files modified data as
the time of the checkout, so more recent than the cached build dir.

-- 
Mehdi


>
> Steve
>
> On Wed, Apr 14, 2021 at 12:59 PM Francesco Bertolaccini via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>> On 14/04/2021 21:18, David Blaikie wrote:
>> > Sounds like something's going wrong for sure. How are you
measuring
>> > the time? (how much time is it taking) and what are you testing by
>> > "partial recompilation"? Touching an ADT .h file is
likely to
>> > recompile nearly everything so might not be much better than a
clean
>> > build - but touching a .cpp file (especially a .cpp file for a
single
>> > tool, rather than a library) might be quite quick. So try timing a
>> > full clean build (ninja -t clean, ninja clang) and then
incrementally
>> > touching just a .cpp file (eg: clang/tools/driver/driver.cpp  )
and
>> > see how they compare
>>
>> I currently do not have sufficient resources to compile LLVM on my
>> personal machine, so I am (ab?)using GitHub Actions. My workflow is
>> setup such that the build artifacts are cached and restored on every
>> worflow run. The time is measured automatically by GitHub itself, and
is
>> around ~2h for each run, whether from scratch or starting from the
cache.
>>
>> I don't generally edit .h files directly, unless they are generated
by
>> modifying TableGen files.
>>
>> The fact that someone else has successfully been able to do incremental
>> builds _does_ make it seem like it's a configuration issue on my
part.
>>
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20210415/00114bf3/attachment.html>

On 15/04/2021 20:36, Mehdi AMINI wrote:
> The only way I can see incremental compilation to work would be to get
> the entire source tree from the cache alongside with the build tree, and
> then run `git fetch && git checkout <rev>`. This would mark
only the
> source file changed by git during the checkout.
> Otherwise a fresh `git clone` will have the source files modified data
> as the time of the checkout, so more recent than the cached build dir.
> 
Thanks, this seems to be the crucial piece of info I was missing. I was
under the impression that something like the hashes of the files were
taken into account, not the last edit time. I'll experiment with this.

Francesco

Good point.  I don't think it's the same flow....sorry for the
confusion.

Steve


On Thu, Apr 15, 2021, 11:36 AM Mehdi AMINI <joker.eph at gmail.com> wrote:
>
>
> On Wed, Apr 14, 2021 at 6:36 PM Stephen Neuendorffer via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>> This flow is being used successfully by both npcomp and circt, which
are
>> llvm incubator projects.  You should be able to leverage/build off of
the
>> github actions flows that they define.  see
https://github.com/llvm/circt
>> and https://github.com/llvm/mlir-npcomp
>>
>
> Is it the same flow?
> As far as I can tell you're not getting a build directory in order to
do
> an incremental rebuild with modified sources, but instead getting the build
> artifacts in order to build a downstream project entirely.
>
> The only way I can see incremental compilation to work would be to get the
> entire source tree from the cache alongside with the build tree, and then
> run `git fetch && git checkout <rev>`. This would mark only
the source file
> changed by git during the checkout.
> Otherwise a fresh `git clone` will have the source files modified data as
> the time of the checkout, so more recent than the cached build dir.
>
> --
> Mehdi
>
>
>
>>
>> Steve
>>
>> On Wed, Apr 14, 2021 at 12:59 PM Francesco Bertolaccini via llvm-dev
<
>> llvm-dev at lists.llvm.org> wrote:
>>
>>> On 14/04/2021 21:18, David Blaikie wrote:
>>> > Sounds like something's going wrong for sure. How are you
measuring
>>> > the time? (how much time is it taking) and what are you
testing by
>>> > "partial recompilation"? Touching an ADT .h file is
likely to
>>> > recompile nearly everything so might not be much better than a
clean
>>> > build - but touching a .cpp file (especially a .cpp file for a
single
>>> > tool, rather than a library) might be quite quick. So try
timing a
>>> > full clean build (ninja -t clean, ninja clang) and then
incrementally
>>> > touching just a .cpp file (eg: clang/tools/driver/driver.cpp 
) and
>>> > see how they compare
>>>
>>> I currently do not have sufficient resources to compile LLVM on my
>>> personal machine, so I am (ab?)using GitHub Actions. My workflow is
>>> setup such that the build artifacts are cached and restored on
every
>>> worflow run. The time is measured automatically by GitHub itself,
and is
>>> around ~2h for each run, whether from scratch or starting from the
cache.
>>>
>>> I don't generally edit .h files directly, unless they are
generated by
>>> modifying TableGen files.
>>>
>>> The fact that someone else has successfully been able to do
incremental
>>> builds _does_ make it seem like it's a configuration issue on
my part.
>>>
>>> _______________________________________________
>>> LLVM Developers mailing list
>>> llvm-dev at lists.llvm.org
>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20210416/1a1f0ab5/attachment-0001.html>