[llvm-dev] Benchmarks for LLVM-generated Binaries

Matthias Braun via llvm-dev
Mon, 12 Sep 2016 22:31:54 -0000

I added support for extensions to the test-suite cmake/lit a while ago: In a
default cmake run it will look for */CMakeLists.txt and compiles/runs the
benchmarks there. Usually that will be External/SingleSource/MultiSource, but if
you drop additional directories in your test-suite checkout it should pick those
up. You can also manually specify the TEST_SUITE_SUBDIRS variable instead if you
do not want to run everything or want it to pick up directories outside the llvm
test-suite.

That should be enough to move specialized tests into independent repositories
while re-using the running/cmake infrastructure. It would probably be a good
idea to this with the External and Bitcode directories today, we just have to
follow through and figure out which buildbots need to run what and which
additional repositories they need to check out.

- Matthias

> On Sep 7, 2016, at 10:40 AM, Eric Christopher via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
>
> That's correct. It seems like a fairly well contained local patch for
your local testsuite, otherwise you could probably change test-suite to try to
understand arbitrary directories.
>
> On Wed, Sep 7, 2016 at 10:31 AM Michael Zolotukhin <mzolotukhin at
apple.com <mailto:mzolotukhin at apple.com>> wrote:
> Hi Eric,
>
> Yeah, I know about Externals and SPEC specifically. But as far as I
understand, you have to have kind of description of the tests in test-suite even
if you don’t provide the source codes - that’s what I would like to avoid. I.e.
you have to have CMakeLists.txt and other files in place all the time, open to
everyone.
>
> Now, imagine I have a small testsuite, which probably is not very
interesting to anyone else, but extremely interesting for me. AFAIU, to make it
a part of LNT now, I have to modify ‘test-suite’ repo so that it’s aware of this
new test-suite. I will probably not be able to upstream these changes (as they
are only interesting to me), and that’s the source of some inconvenience. I’d be
happy to be mistaken here, but that how I understand the current infrastructure,
and that where my question came from.
>
> Thanks,
> Michael
>
>> On Sep 7, 2016, at 10:13 AM, Eric Christopher <echristo at gmail.com
<mailto:echristo at gmail.com>> wrote:
>>
>> Hi Michael,
>>
>> You'll want to look into the externals part of the test suite :)
It's how things like SPEC etc are run.
>>
>> -eric
>>
>> On Tue, Sep 6, 2016 at 4:08 PM Michael Zolotukhin via llvm-dev
<llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>>
wrote:
>>
>> > On Sep 1, 2016, at 8:14 AM, Renato Golin via llvm-dev <llvm-dev
at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote:
>> >
>> > On 1 September 2016 at 07:45, Dean Michael Berris via llvm-dev
>> > <llvm-dev at lists.llvm.org <mailto:llvm-dev at
lists.llvm.org>> wrote:
>> >> I've lately been wondering where benchmarks for
LLVM-generated binaries are hosted, and whether they're tracked over time.
>> >
>> > Hi Dean,
>> >
>> > Do you mean Perf?
>> >
>> > http://llvm.org/perf/ <http://llvm.org/perf/>
>> >
>> > Example, ARM and AArch64 tracking performance at:
>> >
>> > http://llvm.org/perf/db_default/v4/nts/machine/41
<http://llvm.org/perf/db_default/v4/nts/machine/41>
>> > http://llvm.org/perf/db_default/v4/nts/machine/46
<http://llvm.org/perf/db_default/v4/nts/machine/46>
>> >
>> >
>> >> - Is the test-suite repository the right place to put these
generated-code benchmarks?
>> >
>> > I believe there would be the best place, yes.
>> >
>> >
>> >> - Are there any objections to using a later version of the
Google benchmarking library [0] in the test-suite?
>> >
>> > While this looks like a very nice tool set, I wonder how we're
going
>> > to integrate it.
>> >
>> > Checking it out in the test-suite wouldn't be the best option
(version
>> > rot), but neither would be requiring people to install it before
>> > running the test-suite, especially if the installation process
isn't
>> > as easy as "apt-get install", like all the other
dependencies.
>> >
>> >
>> >> - Are the docs on the Testing Infrastructure Guide still
relevant and up-to-date, and is that a good starting point for exploration here?
>> >
>> > Unfortunately, that's mostly for the "make check"
tests, not for the
>> > test-suite. The test-suite execution is covered by LNT's doc
>> > (http://llvm.org/docs/lnt <http://llvm.org/docs/lnt>), but
it's mostly about LNT internals and
>> > not the test-suite itself.
>> >
>> > However, it's not that hard to understand the test-suite
structure. To
>> > add new tests, you just need to find a suitable place {
(SingleSource
>> > / MultiSource) / Benchmarks / YourBench } and copy (
CMakeFiles.txt,
>> > Makefile, lit.local.cfg ), change to your needs, and it should be
>> > done.
>> Hi Renato and others,
>>
>> Is it possible/how hard will it be to make the testsuite kind of
extendable? Like, I copy a folder with some tests that comply to some rules
(e.g. have CMakeLists.txt, lit.local.cfg, etc.) and run them via the standard
infrastructure without changing anything in test-suite files?
>>
>> The motivation of this question is the following: we (and probably many
other companies too) have internal tests that we can’t share, but still want to
track. Currently, the process of adding them to the existing test-suite is not
clear to me (or at least not very well documented), and while I can figure it
out, it would be cool if we can streamline this process.
>>
>> Ideally, I’d like to see this process like this:
>> 1) Add following files to your benchmark suite:
>>   1.a) CMakeLists.txt having this and that target doing this and that.
>>   1.b) lit.local.cfg script having this and that.
>>   …
>> 2) Make sure the test report results in the following format/provide a
wrapper script to convert results to the specified form. /* TODO: Results format
is specified here */
>> 3) Run your tests using the standard LNT command, like  "lnt
runtest … --only-test=External/MyTestSuite/TestA”
>>
>> If that’s already implemented, then I’ll be glad to help with
documentation, and if not, I can try implementing it. What do you think?
>>
>> Thanks,
>> Michael
>>
>> >
>> > cheers,
>> > --renato
>> > _______________________________________________
>> > LLVM Developers mailing list
>> > llvm-dev at lists.llvm.org <mailto:llvm-dev at
lists.llvm.org>
>> > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
<http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
>>
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
<http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160912/2fdea9c8/attachment.html>