llvm dev - May 2016 - RFC: LNT/Test-suite support for custom metrics and test parameterization

Hi Matthias,

Thank you for your answer.
But can you answer for some more questions?
First of all, now LNT uses make-style of running tests and parse results from
result csv file. Are there any plans to go to cmake?
As I understood lit will run and collect all metrics, but there is no
opportunity to make any settings for choosing what metrics I would like to
collect. Test reports files allow to choose what report I would like. One time I
can use one, second time I can use another. I can do this with cmake only by
changing test_modules in file.
So I can’t group some metrics and give them some name.

Am I right?

Thanks,
Elena.


From: Matthias Braun [mailto:matze at braunis.de]
Sent: Friday, May 20, 2016 8:03 PM
To: Elena Lepilkina <Elena.Lepilkina at synopsys.com>
Cc: Daniel Dunbar <daniel at zuster.org>; llvm-dev <llvm-dev at
lists.llvm.org>; nd <nd at arm.com>
Subject: Re: [llvm-dev] RFC: LNT/Test-suite support for custom metrics and test
parameterization

On May 12, 2016, at 11:21 PM, Elena Lepilkina via llvm-dev <llvm-dev at
lists.llvm.org<mailto:llvm-dev at lists.llvm.org>> wrote:

Hi all,

As we understood great changes will be done in LNT, so we are waiting to new LNT
version and stopped our work in LNT.

One more question about using test-suite separately with cmake. Cmake can only
build all tests and generate lit tests. After that we can run LIT and get report
which is not equal with report (simple) got with make. Cmake test-suite version
has no features to run custom metrics and generate other report type, right?

Are these features of make-version of test-suite planned to be added?
The lit test-suite runner supports arbitrary metrics, it already features
codesizes for different segments, compiletime, linktime, executable hash,
execution time. I designed it be easily extensible with further metrics. Not all
of these metrics are understood by LNT yet so they may get lost after submission
to an LNT database.

We do not use GenerateReport.pl and friends in the cmake/lit version anymore, as
the main data collection and reporting mechanism in llvm has mostly shifted to
LNT these days. In any case it should be easy to use “lit -o result.json” and
process the resulting json file in your favorite scripting language to generate
arbitrary reports.

As far as the TEST.*.Makefile things go: Many of those variants have no
equivalent in cmake/lit. But I have the strong feeling that the majority of
those is not really used or even broken so there are no plans to adding them to.
On the other hand I would love to hear from people that actually use any of
those besides TEST.simple.Makefile, it should be possible to transition most of
them but I’d first like to  hear what they are used for.

- Matthias


-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160525/f444ed02/attachment.html>

Hi Elena,
> First of all, now LNT uses make-style of running tests and parse results
from result csv file. Are there any plans to go to cmake?


There are two test drivers in LNT. "lnt runtests nt" uses the old
Makefile-based system and "lnt runtests test-suite" uses the new
cmake-based system. It sounds like you are using the former and should
switch to the latter.

Cheers,

James

On Wed, 25 May 2016 at 10:04 Elena Lepilkina via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> Hi Matthias,
>
>
>
> Thank you for your answer.
>
> But can you answer for some more questions?
>
> First of all, now LNT uses make-style of running tests and parse results
> from result csv file. Are there any plans to go to cmake?
>
> As I understood lit will run and collect all metrics, but there is no
> opportunity to make any settings for choosing what metrics I would like to
> collect. Test reports files allow to choose what report I would like. One
> time I can use one, second time I can use another. I can do this with cmake
> only by changing test_modules in file.
>
> So I can’t group some metrics and give them some name.
>
>
>
> Am I right?
>
>
>
> Thanks,
>
> Elena.
>
>
>
>
>
> *From:* Matthias Braun [mailto:matze at braunis.de]
> *Sent:* Friday, May 20, 2016 8:03 PM
> *To:* Elena Lepilkina <Elena.Lepilkina at synopsys.com>
> *Cc:* Daniel Dunbar <daniel at zuster.org>; llvm-dev <llvm-dev at
lists.llvm.org>;
> nd <nd at arm.com>
>
>
> *Subject:* Re: [llvm-dev] RFC: LNT/Test-suite support for custom metrics
> and test parameterization
>
>
>
> On May 12, 2016, at 11:21 PM, Elena Lepilkina via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>
>
> Hi all,
>
>
>
> As we understood great changes will be done in LNT, so we are waiting to
> new LNT version and stopped our work in LNT.
>
>
>
> One more question about using test-suite separately with cmake. Cmake can
> only build all tests and generate lit tests. After that we can run LIT and
> get report which is not equal with report (simple) got with make. Cmake
> test-suite version has no features to run custom metrics and generate other
> report type, right?
>
>
>
> Are these features of make-version of test-suite planned to be added?
>
> The lit test-suite runner supports arbitrary metrics, it already features
> codesizes for different segments, compiletime, linktime, executable hash,
> execution time. I designed it be easily extensible with further metrics.
> Not all of these metrics are understood by LNT yet so they may get lost
> after submission to an LNT database.
>
>
>
> We do not use GenerateReport.pl and friends in the cmake/lit version
> anymore, as the main data collection and reporting mechanism in llvm has
> mostly shifted to LNT these days. In any case it should be easy to use “lit
> -o result.json” and process the resulting json file in your favorite
> scripting language to generate arbitrary reports.
>
>
>
> As far as the TEST.*.Makefile things go: Many of those variants have no
> equivalent in cmake/lit. But I have the strong feeling that the majority of
> those is not really used or even broken so there are no plans to adding
> them to. On the other hand I would love to hear from people that actually
> use any of those besides TEST.simple.Makefile, it should be possible to
> transition most of them but I’d first like to  hear what they are used for.
>
>
>
> - Matthias
>
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160525/027c8574/attachment.html>

Hi James,

Thank you. We use last one, but I don’t know about this option, because I found
no information about this in documentation. It may be very useful to add more
information about modes of running tests in documentation.
Now I saw source code for test-suite mode.

Thanks,
Elena.

From: James Molloy [mailto:james at jamesmolloy.co.uk]
Sent: Wednesday, May 25, 2016 1:43 PM
To: Elena Lepilkina <Elena.Lepilkina at synopsys.com>; Matthias Braun
<matze at braunis.de>
Cc: llvm-dev <llvm-dev at lists.llvm.org>; nd <nd at arm.com>
Subject: Re: [llvm-dev] RFC: LNT/Test-suite support for custom metrics and test
parameterization

Hi Elena,
> First of all, now LNT uses make-style of running tests and parse results
from result csv file. Are there any plans to go to cmake?

There are two test drivers in LNT. "lnt runtests nt" uses the old
Makefile-based system and "lnt runtests test-suite" uses the new
cmake-based system. It sounds like you are using the former and should switch to
the latter.

Cheers,

James

On Wed, 25 May 2016 at 10:04 Elena Lepilkina via llvm-dev <llvm-dev at
lists.llvm.org<mailto:llvm-dev at lists.llvm.org>> wrote:
Hi Matthias,

Thank you for your answer.
But can you answer for some more questions?
First of all, now LNT uses make-style of running tests and parse results from
result csv file. Are there any plans to go to cmake?
As I understood lit will run and collect all metrics, but there is no
opportunity to make any settings for choosing what metrics I would like to
collect. Test reports files allow to choose what report I would like. One time I
can use one, second time I can use another. I can do this with cmake only by
changing test_modules in file.
So I can’t group some metrics and give them some name.

Am I right?

Thanks,
Elena.


From: Matthias Braun [mailto:matze at braunis.de<mailto:matze at
braunis.de>]
Sent: Friday, May 20, 2016 8:03 PM
To: Elena Lepilkina <Elena.Lepilkina at
synopsys.com<mailto:Elena.Lepilkina at synopsys.com>>
Cc: Daniel Dunbar <daniel at zuster.org<mailto:daniel at
zuster.org>>; llvm-dev <llvm-dev at lists.llvm.org<mailto:llvm-dev
at lists.llvm.org>>; nd <nd at arm.com<mailto:nd at arm.com>>

Subject: Re: [llvm-dev] RFC: LNT/Test-suite support for custom metrics and test
parameterization

On May 12, 2016, at 11:21 PM, Elena Lepilkina via llvm-dev <llvm-dev at
lists.llvm.org<mailto:llvm-dev at lists.llvm.org>> wrote:

Hi all,

As we understood great changes will be done in LNT, so we are waiting to new LNT
version and stopped our work in LNT.

One more question about using test-suite separately with cmake. Cmake can only
build all tests and generate lit tests. After that we can run LIT and get report
which is not equal with report (simple) got with make. Cmake test-suite version
has no features to run custom metrics and generate other report type, right?

Are these features of make-version of test-suite planned to be added?
The lit test-suite runner supports arbitrary metrics, it already features
codesizes for different segments, compiletime, linktime, executable hash,
execution time. I designed it be easily extensible with further metrics. Not all
of these metrics are understood by LNT yet so they may get lost after submission
to an LNT database.

We do not use GenerateReport.pl and friends in the cmake/lit version anymore, as
the main data collection and reporting mechanism in llvm has mostly shifted to
LNT these days. In any case it should be easy to use “lit -o result.json” and
process the resulting json file in your favorite scripting language to generate
arbitrary reports.

As far as the TEST.*.Makefile things go: Many of those variants have no
equivalent in cmake/lit. But I have the strong feeling that the majority of
those is not really used or even broken so there are no plans to adding them to.
On the other hand I would love to hear from people that actually use any of
those besides TEST.simple.Makefile, it should be possible to transition most of
them but I’d first like to  hear what they are used for.

- Matthias

_______________________________________________
LLVM Developers mailing list
llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org>
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160525/cd624627/attachment.html>

> On May 25, 2016, at 1:54 AM, Elena Lepilkina via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> 
> Hi Matthias,
>  
> Thank you for your answer.
> But can you answer for some more questions?
> First of all, now LNT uses make-style of running tests and parse results
from result csv file. Are there any plans to go to cmake?
As James already said "lnt runtest test-suite".
> As I understood lit will run and collect all metrics, but there is no
opportunity to make any settings for choosing what metrics I would like to
collect. Test reports files allow to choose what report I would like. One time I
can use one, second time I can use another. I can do this with cmake only by
changing test_modules in file.
> So I can’t group some metrics and give them some name.
The test_modules flag does indeed allow some customisation in what metrics are
collected, my main usage for this has been do setup things to just collect
codesize but not run the benchmark, a setup in which benchmarks are cross
compiled and then run on a remote device through ssh and the "default"
mode which just runs all benchmarks and collects as much metrics as possible.
However lit is a tool to run the benchmarks, you still need other tools to
filter analyze and present the data afterwards: The typical process we have
today is to run the tests with lnt, store the results in lnt and filter/analyze
with the lnt webpage. If you need custom reports then you can either add new
views to lnt or write some scripts that parse the json output produced by lit. I
have some personal scripts I use to compare the results of two test-suite runs
because I find that easier to do ad-hoc from the commandline rather than getting
lnt running just to compare two runs, I can send you those scripts as an
example, however if you want something better looking than commandline output or
need to process data from a lot of runs then I recommend extending lnt for your
reporting needs.

- Matthias
>  
> Am I right?
>  
> Thanks,
> Elena.
>  
>  
> From: Matthias Braun [mailto:matze at braunis.de] 
> Sent: Friday, May 20, 2016 8:03 PM
> To: Elena Lepilkina <Elena.Lepilkina at synopsys.com>
> Cc: Daniel Dunbar <daniel at zuster.org>; llvm-dev <llvm-dev at
lists.llvm.org>; nd <nd at arm.com>
> Subject: Re: [llvm-dev] RFC: LNT/Test-suite support for custom metrics and
test parameterization
>  
> On May 12, 2016, at 11:21 PM, Elena Lepilkina via llvm-dev <llvm-dev at
lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote:
>  
> Hi all,
>  
> As we understood great changes will be done in LNT, so we are waiting to
new LNT version and stopped our work in LNT.
>  
> One more question about using test-suite separately with cmake. Cmake can
only build all tests and generate lit tests. After that we can run LIT and get
report which is not equal with report (simple) got with make. Cmake test-suite
version has no features to run custom metrics and generate other report type,
right?
>  
> Are these features of make-version of test-suite planned to be added?
> The lit test-suite runner supports arbitrary metrics, it already features
codesizes for different segments, compiletime, linktime, executable hash,
execution time. I designed it be easily extensible with further metrics. Not all
of these metrics are understood by LNT yet so they may get lost after submission
to an LNT database.
>  
> We do not use GenerateReport.pl and friends in the cmake/lit version
anymore, as the main data collection and reporting mechanism in llvm has mostly
shifted to LNT these days. In any case it should be easy to use “lit -o
result.json” and process the resulting json file in your favorite scripting
language to generate arbitrary reports.
>  
> As far as the TEST.*.Makefile things go: Many of those variants have no
equivalent in cmake/lit. But I have the strong feeling that the majority of
those is not really used or even broken so there are no plans to adding them to.
On the other hand I would love to hear from people that actually use any of
those besides TEST.simple.Makefile, it should be possible to transition most of
them but I’d first like to  hear what they are used for.
>  
> - Matthias
>  
>  
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
<http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160525/82ecb437/attachment-0001.html>

I understood your modules and I see as them can be used in LNT. But there are
some question about old features.

1.       With Makefiles we can write pass and collect some statistics. There was
an example of branch pass. Metrics can be collected by

@-$(LOPT) -load dcc888$(SHLIBEXT) -branch-counter -stats \ -time-passes
-disable-output $< 2>>$@

in makefile. In report file we write how data should be parsed. Can we do same
things now with cmake+lit?

2.       As I saw in LNT code there is no opportunity to compile but not execute
tests. Some metrics cab be collected without execution, than run can be faster.

3.       Before each group of metrics has name. Now if we would like to
integrate cmake with custom metrics in LNT, we should compare all metrics of
each run to make sure if this is the same test-suite or not. And we should give
some random names to each group.

These moments are unclear for us now. These question are connected with first
described RFC where shouldn’t be some reports which are known before, but there
should be flexible system when we don’t know what metrics user would loke to
collect. I will be very grateful for your answers.

Thanks,
Elena.

From: Matthias Braun [mailto:matze at braunis.de]
Sent: Wednesday, May 25, 2016 9:23 PM
To: Elena Lepilkina <Elena.Lepilkina at synopsys.com>
Cc: llvm-dev <llvm-dev at lists.llvm.org>; nd <nd at arm.com>
Subject: Re: [llvm-dev] RFC: LNT/Test-suite support for custom metrics and test
parameterization


On May 25, 2016, at 1:54 AM, Elena Lepilkina via llvm-dev <llvm-dev at
lists.llvm.org<mailto:llvm-dev at lists.llvm.org>> wrote:

Hi Matthias,

Thank you for your answer.
But can you answer for some more questions?
First of all, now LNT uses make-style of running tests and parse results from
result csv file. Are there any plans to go to cmake?
As James already said "lnt runtest test-suite".


As I understood lit will run and collect all metrics, but there is no
opportunity to make any settings for choosing what metrics I would like to
collect. Test reports files allow to choose what report I would like. One time I
can use one, second time I can use another. I can do this with cmake only by
changing test_modules in file.
So I can’t group some metrics and give them some name.
The test_modules flag does indeed allow some customisation in what metrics are
collected, my main usage for this has been do setup things to just collect
codesize but not run the benchmark, a setup in which benchmarks are cross
compiled and then run on a remote device through ssh and the "default"
mode which just runs all benchmarks and collects as much metrics as possible.
However lit is a tool to run the benchmarks, you still need other tools to
filter analyze and present the data afterwards: The typical process we have
today is to run the tests with lnt, store the results in lnt and filter/analyze
with the lnt webpage. If you need custom reports then you can either add new
views to lnt or write some scripts that parse the json output produced by lit. I
have some personal scripts I use to compare the results of two test-suite runs
because I find that easier to do ad-hoc from the commandline rather than getting
lnt running just to compare two runs, I can send you those scripts as an
example, however if you want something better looking than commandline output or
need to process data from a lot of runs then I recommend extending lnt for your
reporting needs.

- Matthias


Am I right?

Thanks,
Elena.


From: Matthias Braun [mailto:matze at braunis.de]
Sent: Friday, May 20, 2016 8:03 PM
To: Elena Lepilkina <Elena.Lepilkina at
synopsys.com<mailto:Elena.Lepilkina at synopsys.com>>
Cc: Daniel Dunbar <daniel at zuster.org<mailto:daniel at
zuster.org>>; llvm-dev <llvm-dev at lists.llvm.org<mailto:llvm-dev
at lists.llvm.org>>; nd <nd at arm.com<mailto:nd at arm.com>>
Subject: Re: [llvm-dev] RFC: LNT/Test-suite support for custom metrics and test
parameterization

On May 12, 2016, at 11:21 PM, Elena Lepilkina via llvm-dev <llvm-dev at
lists.llvm.org<mailto:llvm-dev at lists.llvm.org>> wrote:

Hi all,

As we understood great changes will be done in LNT, so we are waiting to new LNT
version and stopped our work in LNT.

One more question about using test-suite separately with cmake. Cmake can only
build all tests and generate lit tests. After that we can run LIT and get report
which is not equal with report (simple) got with make. Cmake test-suite version
has no features to run custom metrics and generate other report type, right?

Are these features of make-version of test-suite planned to be added?
The lit test-suite runner supports arbitrary metrics, it already features
codesizes for different segments, compiletime, linktime, executable hash,
execution time. I designed it be easily extensible with further metrics. Not all
of these metrics are understood by LNT yet so they may get lost after submission
to an LNT database.

We do not use GenerateReport.pl and friends in the cmake/lit version anymore, as
the main data collection and reporting mechanism in llvm has mostly shifted to
LNT these days. In any case it should be easy to use “lit -o result.json” and
process the resulting json file in your favorite scripting language to generate
arbitrary reports.

As far as the TEST.*.Makefile things go: Many of those variants have no
equivalent in cmake/lit. But I have the strong feeling that the majority of
those is not really used or even broken so there are no plans to adding them to.
On the other hand I would love to hear from people that actually use any of
those besides TEST.simple.Makefile, it should be possible to transition most of
them but I’d first like to  hear what they are used for.

- Matthias


_______________________________________________
LLVM Developers mailing list
llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org>
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160526/9d4f9cf0/attachment.html>