llvm dev - Oct 2021 - Proposal: introduce dependency on abseil when building benchmarks

On Thu, 7 Oct 2021 at 17:46, Mircea Trofin <mtrofin at google.com> wrote:
> Making sure we're talking about the same thing: this is strictly about
> benchmarks that require flipping LLVM_BUILD_BENCHMARKS
> or LIBCXX_INCLUDE_BENCHMARKS; and it's about exploring how those
specific
> benchmarks relate to the 'benchmark' project which we currently
have as a
> source copy in the llvm tree (i.e. we're sometimes manually
cherry-picking
> changes over, but we're not updating it through any automated means)
>
Honestly, I didn't understand what this means... :)

But I mostly run the benchmarks via CMake from a recently built LLVM copy,
and I don't do that "professionally", so I'm not the right
person to such
things.

I looked under llvm-zorg, and can't find any bot configuration that
flips
> those flags in the first place, so perhaps these specific benchmarks are
> run elsewhere (or my search was naive - I just grep-ed for
'_BENCHMARKS' -
> and only got one entry turning off TEST_SUITE_RUN_BENCHMARKS, which appears
> to be unrelated)?
>
AFAIK, most of the benchmarks using our test-suite are not done via
buildbots. I've set up one in the long past but even then it didn't seem
to
be the norm.

Back then, I remember every single major target was running the test-suite
in benchmark mode in an internal infrastructure, because they were also
running private benchmarks and had their own local reporting system that
allowed them to track regressions.

Looking for bot-owners to add to the thread only gets you as far as knowing
who they think should be copied, or if you're luck, they're the same
people. :)
>From the OS x CPU matrix above, it seems the main gap is in the CPU column
> - since abseil supports Linux, MacOS, and Windows, but doesn't appear
to
> specifically support mips, powerpc, systemz, or riscv. So if anyone runs
> these kinds of benchmarks (i.e. the ones that require flipping those cmake
> flags) on those CPUs, *and* we wanted, in llvm, to periodically auto-update
> the source copy of 'benchmark' project, only then we would have a
breaking
> change.
>
There's also the fact that abseil only supports "Linux with Arm64"
and
"ChromeOS on Armv7", so it's not really a matrix, just a list of
combinations.

- I think the main one is: was there ever any intent to
automatically
> update the 2 copies of 'benchmarks' in the llvm tree? what about
the one in
> llvm-test-suite?
>
Right, and only the people who actually run benchmarks can answer that.

- the second is getting a better understanding of where folks build and
run
> those specific benchmarks (it'd help the upstream 'benchmark'
project with
> data points).
>
Yes, unfortunately, because many of those are internal, people don't
communicate what they do too well. And if some of the additional benchmarks
are secret, they may not even be able to communicate.

Renato, would you mind detailing your scenario (i.e. which of those 2
types
> of benchmarks, on which OSxCPU)
>
I'm afraid I cannot, as I haven't done any benchmarking for a long time.
Hopefully Tom may be able to help you more than I can.

cheers,
--renato
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20211007/e75a8a37/attachment.html>

On Thu, Oct 7, 2021 at 10:00 AM Renato Golin <rengolin at gmail.com>
wrote:
> On Thu, 7 Oct 2021 at 17:46, Mircea Trofin <mtrofin at google.com>
wrote:
>
>> Making sure we're talking about the same thing: this is strictly
about
>> benchmarks that require flipping LLVM_BUILD_BENCHMARKS
>> or LIBCXX_INCLUDE_BENCHMARKS; and it's about exploring how those
specific
>> benchmarks relate to the 'benchmark' project which we currently
have as a
>> source copy in the llvm tree (i.e. we're sometimes manually
cherry-picking
>> changes over, but we're not updating it through any automated
means)
>>
>
> Honestly, I didn't understand what this means... :)
>
> But I mostly run the benchmarks via CMake from a recently built LLVM copy,
> and I don't do that "professionally", so I'm not the
right person to such
> things.
>
Oh, OK - so what's your cmake  invocation like - i.e. if you don't flip
those 2 flags, then you probably don't build those benchmarks.

Another way to put it: do you build anything under:

llvm-test-suite/MicroBenchmarks/libs/benchmark/
llvm/utils/benchmark
libcxx/utils/google-benchmark


> I looked under llvm-zorg, and can't find any bot configuration that
flips
>> those flags in the first place, so perhaps these specific benchmarks
are
>> run elsewhere (or my search was naive - I just grep-ed for
'_BENCHMARKS' -
>> and only got one entry turning off TEST_SUITE_RUN_BENCHMARKS, which
appears
>> to be unrelated)?
>>
>
> AFAIK, most of the benchmarks using our test-suite are not done via
> buildbots. I've set up one in the long past but even then it didn't
seem to
> be the norm.
>
> Back then, I remember every single major target was running the test-suite
> in benchmark mode in an internal infrastructure, because they were also
> running private benchmarks and had their own local reporting system that
> allowed them to track regressions.
>
> Looking for bot-owners to add to the thread only gets you as far as
> knowing who they think should be copied, or if you're luck, they're
the
> same people. :)
>
> From the OS x CPU matrix above, it seems the main gap is in the CPU column
>> - since abseil supports Linux, MacOS, and Windows, but doesn't
appear to
>> specifically support mips, powerpc, systemz, or riscv. So if anyone
runs
>> these kinds of benchmarks (i.e. the ones that require flipping those
cmake
>> flags) on those CPUs, *and* we wanted, in llvm, to periodically
auto-update
>> the source copy of 'benchmark' project, only then we would have
a breaking
>> change.
>>
>
> There's also the fact that abseil only supports "Linux with
Arm64" and
> "ChromeOS on Armv7", so it's not really a matrix, just a list
of
> combinations.
>
> - I think the main one is: was there ever any intent to automatically
>> update the 2 copies of 'benchmarks' in the llvm tree? what
about the one in
>> llvm-test-suite?
>>
>
> Right, and only the people who actually run benchmarks can answer that.
>
> - the second is getting a better understanding of where folks build and
>> run those specific benchmarks (it'd help the upstream
'benchmark' project
>> with data points).
>>
>
> Yes, unfortunately, because many of those are internal, people don't
> communicate what they do too well. And if some of the additional benchmarks
> are secret, they may not even be able to communicate.
>
> Renato, would you mind detailing your scenario (i.e. which of those 2
>> types of benchmarks, on which OSxCPU)
>>
>
> I'm afraid I cannot, as I haven't done any benchmarking for a long
time.
> Hopefully Tom may be able to help you more than I can.
>
> cheers,
> --renato
>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20211007/7c373b02/attachment.html>