[llvm-dev] Separate test harness for LLVM test-suite

Matthias Braun via llvm-dev
Mon, 09 Jul 2018 18:46:28 -0000

> On Jul 9, 2018, at 10:55 AM, Roman Lebedev via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
>
> On Mon, Jul 9, 2018 at 10:26 AM, Elena Lepilkina via llvm-dev
> <llvm-dev at lists.llvm.org <mailto:llvm-dev at
lists.llvm.org>> wrote:
>> Dear community,
>>
>>
>>
>> LLVM test-suite includes both default applications (benchmarks, tests)
and
>> test harness (CMake modules, lit-support, some scripts, etc.). Test
suite
>> provides extension interface to add “external” tests suite and folders.
This
>> worked well, we enhanced basic harness and integrated large set of
internal
>> benchmarks and applications.
>>
>>
>>
>> However, we have users interested in LLVM test harness to build,
profile,
>> test applications and experiment with tools/compilers without using
>> applications from full LLVM test-suite (~2,5 Gb).
>>
>>
>>
>> Are there reasons why separation of test-suite into 1) basic
test-harness
>> and 2) applications is undesirable or impossible?
> I would like to cautiously +1 this question.
>
> I've been personally eyeing using LNT for personal benchmarking needs,
> but so far did not get to it. There seems to be no good docs (with
examples!)
> on how to actually write an external lnt test producer, which is
understandable,
> and the test-suite itself is rather tightly coupled with the LLVM.

Note that you do not need LNT to run benchmarks. LNT is good for storing and
tracking benchmark results over time. For simple A/B tests you are usually
better of just running llvm-testsuite directly via cmake.

Thanks to Chris there is some introduction about it here:
http://llvm.org/docs/TestSuiteMakefileGuide.html#running-the-test-suite-via-cmake
<http://llvm.org/docs/TestSuiteMakefileGuide.html#running-the-test-suite-via-cmake>

You can conveniently view the results via test-suite/results/compare.py

llvm-testsuites cmake files are also designed to be extensible. Unfortunately
this is not separately documented yet (the 1 sentence explanation is: Create a
folder and write your own CMakeLists.txt and point TEST_SUITE_SUBDIRS to it at
cmake time).

I'd be happy to answer any questions in the area; I'd also highly
apreciate it if someone wanted to contribute to LLVM by writing the missing
documentation ;-)

>
> Thus i too think that such a split into the supporting framework,
> and all the actual benchmarks *may* be a good thing.
> Or at least i would love to see some good docs.

The driving code is already modular and you can mix and match testsuite (the
included ones or by writing external ones) by setting TEST_SUITE_SUBDIRS. We
successfully use this for some additional test-suite modes like CTMark or the
new MicroBenchmarks; we also have some internal test-suite here that now plug
into this system. This is good software design and we should make sure it stays
the way it is.

Regardless I would vote to not split things up: The situation actually feels
similar to the mono-repository vs. multi-repository debate we have on the
clang/llvm/compiler-rt/... side. My take here would be that things that are
usually shipped or used as a unit should stay in the same repository. If you end
up in a situation where you often have to synchronize commits across
repositories things become more complicated and development slows down. My
impression is that the test-suite driver and the core test-suite is in the
majority of cases used together, indicating to me that it is benefitial to keep
them in the same repository.

- Matthias

>
>> We will be glad to hear opinions of community.
>>
>>
>>
>> Best regards,
>>
>> Elena Lepilkina,
> Roman.
>
>> Synopsys Corp, ARC processors
>>
>>
>>
>>
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
<http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
>>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
<http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20180709/78f96624/attachment.html>