llvm dev - Aug 2018 - [cfe-dev] Proposal: pull benchmark library to the LLVM main repository

Thank you very much for the feedback!

What Chandler said about test-suite totally makes sense to me since it's
also excluded from LLVM git monorepo. I will try to land benchmark library
to LLVM core repo and update it to the latest version.

I have not been doing much CMake/project structure before, but I'll start
looking into that next week. I'll reach out to Dominic if anything goes
wrong, thank you for offering assistance!

Kind regards,
Kirill Bobyrev

On Mon, Jul 30, 2018, 9:40 AM Dominic Hamon via cfe-dev <
cfe-dev at lists.llvm.org> wrote:
> If you need any help integrating with lit, or any changes made to
> benchmark, please let me know.
>
> On Sun, 29 Jul 2018, 00:34 Chandler Carruth via cfe-dev, <
> cfe-dev at lists.llvm.org> wrote:
>
>> I'm happy to have this in the main LLVM repositiory.
>>
>> The version in the test suite should likely stay there because the test
>> suite should be buildable w/o LLVM itself -- it is largely a distinct
>> thing. We re-use lit, but not much else from LLVM, and we wouldn't
want to
>> install the benchmark library the way we do lit.
>>
>> One interesting point: we should have some way of running the in-tree
>> benchmarks, likely with lit, much like we currently allow running
unittests
>> with lit. May be something you want to think about.
>>
>> On Sat, Jul 28, 2018 at 4:04 PM Dean Michael Berris via cfe-dev <
>> cfe-dev at lists.llvm.org> wrote:
>>
>>> I’m a huge fan of having more benchmarks, and support this
proposal.
>>>
>>> On Sat, 28 Jul 2018 at 2:16 am, Kirill Bobyrev via llvm-dev <
>>> llvm-dev at lists.llvm.org> wrote:
>>>
>>>> As a part of upcoming new Clangd symbol index implementation,
we would
>>>> like to start support benchmarks of different Clangd pieces,
such as index
>>>> queries and code completion.
>>>>
>>>> There are already two projects in the LLVM tree using
google/benchmark
>>>> library while keeping its source code in-tree: libcxx
>>>> (libcxx/utils/google-benchmark) and test-suite
>>>> (test-suite/MicroBenchmarks/libs/benchmark-1.3.0). Storing
another copy of
>>>> benchmark library sources in clang-tools-extra would be
unreasonable. We
>>>> already have google test library in LLVM tree
>>>> (llvm/utils/unittest/googletest) and it is used across all
other
>>>> subprojects, which looks to be very similar to the benchmark
library in
>>>> terms of reusing it across the projects. I would like to know
if putting
>>>> benchmark library along with googletest would be the best
option. At the
>>>> same time, benchmark library could be updated to the newer
version (1.4.1)
>>>> in the process of pulling it to the main LLVM repository.
>>>>
>>>> It would be great to get feedback on whether this proposal
looks
>>>> reasonable to the LLVM Community and having benchmark in the
llvm/
>>>> repository would be the best solution to the described problem.
>>>>
>>>> Kind regards,
>>>> Kirill Bobyrev
>>>> _______________________________________________
>>>> LLVM Developers mailing list
>>>> llvm-dev at lists.llvm.org
>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>
>>> --
>>> Dean
>>> _______________________________________________
>>> cfe-dev mailing list
>>> cfe-dev at lists.llvm.org
>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>>>
>> _______________________________________________
>> cfe-dev mailing list
>> cfe-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>>
> _______________________________________________
> cfe-dev mailing list
> cfe-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20180802/c949753a/attachment-0001.html>

Hi everyone,

Over the last few days I've been looking into pulling the benchmark library
to LLVM main repository as proposed before. I submitted a patch for
suppressing warnings in benchmark library itself which prevented it from
compiling in LLVM tree (due to enabled -Werror) and managed to build it
in-tree.

However, I have found that cxx-benchmarks target (containing all libc++
benchmarks) is not buildable for a while. That fact and my experience of
build failures when latest Clang is used to compile the project give me an
impression that libc++ benchmarks are not frequently used (e.g.
continuously run by buildbots). There are multiple commits in which
cxx-benchmarks are not buildable, the one that is most likely causing the
issue would be https://reviews.llvm.org/rL338103. Another complication is
that libc++ can be built out-of-tree and it's possible to pull gtest (with
no fixed version) from GitHub using ExternalProject_Add in one of the
configurations. Also, libc++ seems to be capable of building benchmark
library in two different configurations within the same build which
probably means that some libc++-specific code would still end up in libc++
even if google/benchmark is pulled out of that repository (and also we
would have to treat benchmark the same way gtest is handled for out-of-tree
builds).

I was willing to reach out to the community and ask for more feedback on
the following actions. The possible solutions are:

* Pull benchmark library into LLVM core repository without affecting libc++
at all (and introducing another copy of the library). We already have two
distinct benchmark copies as mentioned before and I personally am not happy
with introducing another one, but libc++ out-of-tree builds and specific
requirements complicate things. libc++ also seems to have buildbots for
out-of-tree builds which might be different from the "standard" LLVM
workflow.
* See if the benchmark build can be fixed and handle benchmark similarly to
gtest for out-of-tree libc++ builds. While this approach looks *correct* to
me, I don't know if many users rely on building libc++ out-of-tree *and*
testing it (and also potentially running benchmarks), e.g. the version of
gtest downloaded from Github is pulled from master branch, which gives me
an impression that this option is not frequently used (otherwise the
version would probably be fixed for stability). It would also be harder to
execute.
* Execute the first plan by introducing another copy in the LLVM core
repository, then migrate libc++ to the LLVM's benchmark copy if the
maintainers of libc++ are in favor. This seems both easier and still
"correct" to me.

Since libc++ is currently the only user of benchmark library and I might
miss some details, I wanted to see if anyone actively developing it could
comment on that. I would also like to hear what the others think should be
done to introduce benchmark library to the whole LLVM source tree, because
I can't figure out what would be the best series of actions.

Also, is running benchmarks continuously something the community would be
happy about in the long run?

Kind regards,
Kirill Bobyrev

On Thu, Aug 2, 2018 at 10:22 AM Kirill Bobyrev <kbobyrev.lists at
gmail.com>
wrote:
> Thank you very much for the feedback!
>
> What Chandler said about test-suite totally makes sense to me since
it's
> also excluded from LLVM git monorepo. I will try to land benchmark library
> to LLVM core repo and update it to the latest version.
>
> I have not been doing much CMake/project structure before, but I'll
start
> looking into that next week. I'll reach out to Dominic if anything goes
> wrong, thank you for offering assistance!
>
> Kind regards,
> Kirill Bobyrev
>
> On Mon, Jul 30, 2018, 9:40 AM Dominic Hamon via cfe-dev <
> cfe-dev at lists.llvm.org> wrote:
>
>> If you need any help integrating with lit, or any changes made to
>> benchmark, please let me know.
>>
>> On Sun, 29 Jul 2018, 00:34 Chandler Carruth via cfe-dev, <
>> cfe-dev at lists.llvm.org> wrote:
>>
>>> I'm happy to have this in the main LLVM repositiory.
>>>
>>> The version in the test suite should likely stay there because the
test
>>> suite should be buildable w/o LLVM itself -- it is largely a
distinct
>>> thing. We re-use lit, but not much else from LLVM, and we
wouldn't want to
>>> install the benchmark library the way we do lit.
>>>
>>> One interesting point: we should have some way of running the
in-tree
>>> benchmarks, likely with lit, much like we currently allow running
unittests
>>> with lit. May be something you want to think about.
>>>
>>> On Sat, Jul 28, 2018 at 4:04 PM Dean Michael Berris via cfe-dev
<
>>> cfe-dev at lists.llvm.org> wrote:
>>>
>>>> I’m a huge fan of having more benchmarks, and support this
proposal.
>>>>
>>>> On Sat, 28 Jul 2018 at 2:16 am, Kirill Bobyrev via llvm-dev
<
>>>> llvm-dev at lists.llvm.org> wrote:
>>>>
>>>>> As a part of upcoming new Clangd symbol index
implementation, we would
>>>>> like to start support benchmarks of different Clangd
pieces, such as index
>>>>> queries and code completion.
>>>>>
>>>>> There are already two projects in the LLVM tree using
google/benchmark
>>>>> library while keeping its source code in-tree: libcxx
>>>>> (libcxx/utils/google-benchmark) and test-suite
>>>>> (test-suite/MicroBenchmarks/libs/benchmark-1.3.0). Storing
another copy of
>>>>> benchmark library sources in clang-tools-extra would be
unreasonable. We
>>>>> already have google test library in LLVM tree
>>>>> (llvm/utils/unittest/googletest) and it is used across all
other
>>>>> subprojects, which looks to be very similar to the
benchmark library in
>>>>> terms of reusing it across the projects. I would like to
know if putting
>>>>> benchmark library along with googletest would be the best
option. At the
>>>>> same time, benchmark library could be updated to the newer
version (1.4.1)
>>>>> in the process of pulling it to the main LLVM repository.
>>>>>
>>>>> It would be great to get feedback on whether this proposal
looks
>>>>> reasonable to the LLVM Community and having benchmark in
the llvm/
>>>>> repository would be the best solution to the described
problem.
>>>>>
>>>>> Kind regards,
>>>>> Kirill Bobyrev
>>>>> _______________________________________________
>>>>> LLVM Developers mailing list
>>>>> llvm-dev at lists.llvm.org
>>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>>
>>>> --
>>>> Dean
>>>> _______________________________________________
>>>> cfe-dev mailing list
>>>> cfe-dev at lists.llvm.org
>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>>>>
>>> _______________________________________________
>>> cfe-dev mailing list
>>> cfe-dev at lists.llvm.org
>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>>>
>> _______________________________________________
>> cfe-dev mailing list
>> cfe-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20180808/dbe4e241/attachment.html>

> On 9 Aug 2018, at 00:59, Kirill Bobyrev via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> 
> Hi everyone,
> 
> Over the last few days I've been looking into pulling the benchmark
library to LLVM main repository as proposed before. I submitted a patch for
suppressing warnings in benchmark library itself which prevented it from
compiling in LLVM tree (due to enabled -Werror) and managed to build it in-tree.
> 
> However, I have found that cxx-benchmarks target (containing all libc++
benchmarks) is not buildable for a while. That fact and my experience of build
failures when latest Clang is used to compile the project give me an impression
that libc++ benchmarks are not frequently used (e.g. continuously run by
buildbots). There are multiple commits in which cxx-benchmarks are not
buildable, the one that is most likely causing the issue would be
https://reviews.llvm.org/rL338103. Another complication is that libc++ can be
built out-of-tree and it's possible to pull gtest (with no fixed version)
from GitHub using ExternalProject_Add in one of the configurations. Also, libc++
seems to be capable of building benchmark library in two different
configurations within the same build which probably means that some
libc++-specific code would still end up in libc++ even if google/benchmark is
pulled out of that repository (and also we would have to treat benchmark the
same way gtest is handled for out-of-tree builds).
> 
> I was willing to reach out to the community and ask for more feedback on
the following actions. The possible solutions are:
> 
> * Pull benchmark library into LLVM core repository without affecting libc++
at all (and introducing another copy of the library). We already have two
distinct benchmark copies as mentioned before and I personally am not happy with
introducing another one, but libc++ out-of-tree builds and specific requirements
complicate things. libc++ also seems to have buildbots for out-of-tree builds
which might be different from the "standard" LLVM workflow.
> * See if the benchmark build can be fixed and handle benchmark similarly to
gtest for out-of-tree libc++ builds. While this approach looks *correct* to me,
I don't know if many users rely on building libc++ out-of-tree *and* testing
it (and also potentially running benchmarks), e.g. the version of gtest
downloaded from Github is pulled from master branch, which gives me an
impression that this option is not frequently used (otherwise the version would
probably be fixed for stability). It would also be harder to execute.
> * Execute the first plan by introducing another copy in the LLVM core
repository, then migrate libc++ to the LLVM's benchmark copy if the
maintainers of libc++ are in favor. This seems both easier and still
"correct" to me.
> 
This last option seems like a good plan to follow.
> Since libc++ is currently the only user of benchmark library and I might
miss some details, I wanted to see if anyone actively developing it could
comment on that. I would also like to hear what the others think should be done
to introduce benchmark library to the whole LLVM source tree, because I
can't figure out what would be the best series of actions.
> 
Eizan (CC’ed) worked on getting the benchmark library included and upgraded in
the test-suite. Maybe he can answer some of the more specific questions on the
process he came up with.
> Also, is running benchmarks continuously something the community would be
happy about in the long run?
> 
Personally, I’d settle for a pseudo-target that isolates microbenchmarks as a
whole and some pseudo-targets by project or group. This would be a bare minimum
for example of we want to benchmark individual support libraries, data
structures, utilities, parsers/etc. — I am actually looking forward to putting
more microbenchmarks on some parts of the compiler-rt project that live beside
the implementation to allow for benchmark-qualified (performance driven?)
testing/improvements.

Continuous running is a nice to have, but it’s really hard to get statistically
stable measurements if they’re running in non-fully-controlled environments.
Even in full-control environments, there’s quite a bit of infrastructure
required to ensure that the numbers are repeatable and variance is
well-accounted for. That’s a lot of effort for arguably little gain.

My gut tells me that having the benchmarks in-tree and making it easy to run as
part of the normal development process is more valuable than having it
continuously run. As we grow the quantity and improve quality of benchmarks
in-tree we can make a determination of how to collect data from multiple
environments and maybe generating reports or dashboards.

Needless to say, I’m looking forward to having more and easier access to
microbenchmarks in-tree! :)

Cheers

-- Dean