llvm dev - Oct 2021 - Proposal: introduce dependency on abseil when building benchmarks

On Thu, Oct 7, 2021 at 8:40 AM Renato Golin <rengolin at gmail.com> wrote:
> On Thu, 30 Sept 2021 at 18:08, Mircea Trofin via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>> Abseil has some requirements
>>
<https://abseil.io/docs/cpp/platforms/platforms#:~:text=Abseil%20requires%20a%20code%20base,14%20through%20C%2B%2B20).>
>> that may not perfectly match those of the impacted projects. For
example,
>> abseil stopped supporting Ubuntu 14.04 before its TLS.
>>
>
> The list of supported platforms is *definitely* too small for LLVM users.
> Half of their support is "best effort", which really isn't
going to cut it
> once we forcefully depend on it.
>
> We definitely run benchmarks on X86_64, Arm32/64, MIPS, PowerPC (Linux,
> Mac and Windows on a mix of those), and there are probably people running
> on SystemZ, RISCV and other less known architectures.
>
Making sure we're talking about the same thing: this is strictly about
benchmarks that require flipping LLVM_BUILD_BENCHMARKS
or LIBCXX_INCLUDE_BENCHMARKS; and it's about exploring how those specific
benchmarks relate to the 'benchmark' project which we currently have as
a
source copy in the llvm tree (i.e. we're sometimes manually cherry-picking
changes over, but we're not updating it through any automated means)

I looked under llvm-zorg, and can't find any bot configuration that flips
those flags in the first place, so perhaps these specific benchmarks are
run elsewhere (or my search was naive - I just grep-ed for '_BENCHMARKS'
-
and only got one entry turning off TEST_SUITE_RUN_BENCHMARKS, which appears
to be unrelated)?
>From the OS x CPU matrix above, it seems the main gap is in the CPU column
- since abseil supports Linux, MacOS, and Windows, but doesn't appear to
specifically support mips, powerpc, systemz, or riscv. So if anyone runs
these kinds of benchmarks (i.e. the ones that require flipping those cmake
flags) on those CPUs, *and* we wanted, in llvm, to periodically auto-update
the source copy of 'benchmark' project, only then we would have a
breaking
change.

I think there are 2 topics:
- I think the main one is: was there ever any intent to automatically
update the 2 copies of 'benchmarks' in the llvm tree? what about the one
in
llvm-test-suite?

- the second is getting a better understanding of where folks build and run
those specific benchmarks (it'd help the upstream 'benchmark'
project with
data points).

Renato, would you mind detailing your scenario (i.e. which of those 2 types
of benchmarks, on which OSxCPU)

If there are other folks that can chime in with their scenario, it'd be
much appreciated.

Thanks!


> What is your plan for all the other platforms where abseil isn't
supported?
>
> cheers,
> --renato
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20211007/1e31f93b/attachment.html>

On Thu, 7 Oct 2021 at 17:46, Mircea Trofin <mtrofin at google.com> wrote:
> Making sure we're talking about the same thing: this is strictly about
> benchmarks that require flipping LLVM_BUILD_BENCHMARKS
> or LIBCXX_INCLUDE_BENCHMARKS; and it's about exploring how those
specific
> benchmarks relate to the 'benchmark' project which we currently
have as a
> source copy in the llvm tree (i.e. we're sometimes manually
cherry-picking
> changes over, but we're not updating it through any automated means)
>
Honestly, I didn't understand what this means... :)

But I mostly run the benchmarks via CMake from a recently built LLVM copy,
and I don't do that "professionally", so I'm not the right
person to such
things.

I looked under llvm-zorg, and can't find any bot configuration that
flips
> those flags in the first place, so perhaps these specific benchmarks are
> run elsewhere (or my search was naive - I just grep-ed for
'_BENCHMARKS' -
> and only got one entry turning off TEST_SUITE_RUN_BENCHMARKS, which appears
> to be unrelated)?
>
AFAIK, most of the benchmarks using our test-suite are not done via
buildbots. I've set up one in the long past but even then it didn't seem
to
be the norm.

Back then, I remember every single major target was running the test-suite
in benchmark mode in an internal infrastructure, because they were also
running private benchmarks and had their own local reporting system that
allowed them to track regressions.

Looking for bot-owners to add to the thread only gets you as far as knowing
who they think should be copied, or if you're luck, they're the same
people. :)
>From the OS x CPU matrix above, it seems the main gap is in the CPU column
> - since abseil supports Linux, MacOS, and Windows, but doesn't appear
to
> specifically support mips, powerpc, systemz, or riscv. So if anyone runs
> these kinds of benchmarks (i.e. the ones that require flipping those cmake
> flags) on those CPUs, *and* we wanted, in llvm, to periodically auto-update
> the source copy of 'benchmark' project, only then we would have a
breaking
> change.
>
There's also the fact that abseil only supports "Linux with Arm64"
and
"ChromeOS on Armv7", so it's not really a matrix, just a list of
combinations.

- I think the main one is: was there ever any intent to
automatically
> update the 2 copies of 'benchmarks' in the llvm tree? what about
the one in
> llvm-test-suite?
>
Right, and only the people who actually run benchmarks can answer that.

- the second is getting a better understanding of where folks build and
run
> those specific benchmarks (it'd help the upstream 'benchmark'
project with
> data points).
>
Yes, unfortunately, because many of those are internal, people don't
communicate what they do too well. And if some of the additional benchmarks
are secret, they may not even be able to communicate.

Renato, would you mind detailing your scenario (i.e. which of those 2
types
> of benchmarks, on which OSxCPU)
>
I'm afraid I cannot, as I haven't done any benchmarking for a long time.
Hopefully Tom may be able to help you more than I can.

cheers,
--renato
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20211007/e75a8a37/attachment.html>

Am Do., 7. Okt. 2021 um 11:47 Uhr schrieb Mircea Trofin via llvm-dev
<llvm-dev at lists.llvm.org>:
> I looked under llvm-zorg, and can't find any bot configuration that
flips those flags in the first place, so perhaps these specific benchmarks are
run elsewhere (or my search was naive - I just grep-ed for '_BENCHMARKS'
- and only got one entry turning off TEST_SUITE_RUN_BENCHMARKS, which appears to
be unrelated)?
Google Benchmark (from MicroBenchmarks/libs/benchmark) automatically
compiles/runs if TES_SUITE_SUBDIRS includes MicroBenchmarks, which is
included by default.

TEST_SUITE_RUN_BENCHMARKS is unrelated

Michael