Tobias Grosser via llvm-dev
2017-Jul-04 06:27 UTC
[llvm-dev] Performance metrics with LLVM
Dear all, I wanted to gather LLVM statistics with lnt and found a nice flag, but am unsure how such statistics can be made available in the LNT web interface. --cmake-define=TEST_SUITE_COLLECT_STATS=ON which allows me to gather all the LLVM "-stats" output. On top of this I see that the LNT cmake test-suite also dumps code-size statistics when running, that look as follows: size: 10848 size..bss: 48 size..comment: 218 size..ctors: 16 size..data: 4 size..dtors: 16 size..dynamic: 416 size..dynsym: 168 size..eh_frame: 172 size..eh_frame_hdr: 44 I can find all these statistics in a file called: /scratch/leone/grosser/base/sandbox/test-2017-07-04_06-14-43/outputTd2xPU.json but they do not appear in: /scratch/leone/grosser/base/sandbox/test-2017-07-04_06-14-43/report.json and in fact do not seem to be submitted to the LNT server. Matthias added support for TEST_SUITE_COLLECT_STATS a while ago, but I am unsure how it is expected to be used. A google search did not find any relevant documentation. Is anybody using this feature today? Best, Tobias
Hi Tobias, The metrics that you can collect in LNT are fixed per "test suite". There are 2 such "test suite"s defined in LNT at the moment: nts and compile. For more details on this, see http://llvm.org/docs/lnt/concepts.html#test-suites. AFAIK, If you need to collect different metrics, you'll need to define a new "test suite". I'm afraid I don't really know what is needed for that. I'm guessing you may need to write some LNT code to do so, but I'm not sure. Hopefully Matthias or Chris will be able to explain how to do that. We probably should investigate how to make it easier to define new "test-suite"s more easily. Or at least make it easier to record different sets of metrics more easily, without having to change the LNT code or a running LNT server instance. The question on recording a different set of metrics has come up on this list before, so it seems like it's an issue people do run into from time to time. Thanks, Kristof On 4 Jul 2017, at 08:27, Tobias Grosser <tobias.grosser at inf.ethz.ch<mailto:tobias.grosser at inf.ethz.ch>> wrote: Dear all, I wanted to gather LLVM statistics with lnt and found a nice flag, but am unsure how such statistics can be made available in the LNT web interface. --cmake-define=TEST_SUITE_COLLECT_STATS=ON which allows me to gather all the LLVM "-stats" output. On top of this I see that the LNT cmake test-suite also dumps code-size statistics when running, that look as follows: size: 10848 size..bss: 48 size..comment: 218 size..ctors: 16 size..data: 4 size..dtors: 16 size..dynamic: 416 size..dynsym: 168 size..eh_frame: 172 size..eh_frame_hdr: 44 I can find all these statistics in a file called: /scratch/leone/grosser/base/sandbox/test-2017-07-04_06-14-43/outputTd2xPU.json but they do not appear in: /scratch/leone/grosser/base/sandbox/test-2017-07-04_06-14-43/report.json and in fact do not seem to be submitted to the LNT server. Matthias added support for TEST_SUITE_COLLECT_STATS a while ago, but I am unsure how it is expected to be used. A google search did not find any relevant documentation. Is anybody using this feature today? Best, Tobias -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20170704/99ba4270/attachment-0001.html>
Tobias Grosser via llvm-dev
2017-Jul-04 09:02 UTC
[llvm-dev] Performance metrics with LLVM
On Tue, Jul 4, 2017, at 09:48 AM, Kristof Beyls wrote:> Hi Tobias, > > The metrics that you can collect in LNT are fixed per "test suite". > There are 2 such "test suite"s defined in LNT at the moment: nts and > compile. > For more details on this, see > http://llvm.org/docs/lnt/concepts.html#test-suites. > > AFAIK, If you need to collect different metrics, you'll need to define a > new "test suite". I'm afraid I don't really know what is needed for that. > I'm guessing you may need to write some LNT code to do so, but I'm not > sure. Hopefully Matthias or Chris will be able to explain how to do that. > > We probably should investigate how to make it easier to define new > "test-suite"s more easily. Or at least make it easier to record different > sets of metrics more easily, without having to change the LNT code or a > running LNT server instance. > The question on recording a different set of metrics has come up on this > list before, so it seems like it's an issue people do run into from time > to time.Hi Kristof, thanks for your fast reply. This is a very helpful summary that confirms my current understanding in parts. I never run the "compile" test suite, so I am not sure how much of the statistics interface is used by it (if at all). I somehow had the feeling something else might exist, as the cmake test-suite runner dumps some of the statistics to stdout. Would be interested to read if Chris or Matthias have more insights. Best, Tobias> Thanks, > > Kristof > > > On 4 Jul 2017, at 08:27, Tobias Grosser > <tobias.grosser at inf.ethz.ch<mailto:tobias.grosser at inf.ethz.ch>> wrote: > > Dear all, > > I wanted to gather LLVM statistics with lnt and found a nice flag, but > am unsure how such statistics can be made available in the LNT web > interface. > > --cmake-define=TEST_SUITE_COLLECT_STATS=ON > > which allows me to gather all the LLVM "-stats" output. On top of this I > see that the LNT cmake test-suite also dumps code-size statistics when > running, that look as follows: > > size: 10848 > size..bss: 48 > size..comment: 218 > size..ctors: 16 > size..data: 4 > size..dtors: 16 > size..dynamic: 416 > size..dynsym: 168 > size..eh_frame: 172 > size..eh_frame_hdr: 44 > > I can find all these statistics in a file called: > > /scratch/leone/grosser/base/sandbox/test-2017-07-04_06-14-43/outputTd2xPU.json > > but they do not appear in: > > /scratch/leone/grosser/base/sandbox/test-2017-07-04_06-14-43/report.json > > and in fact do not seem to be submitted to the LNT server. > > Matthias added support for TEST_SUITE_COLLECT_STATS a while ago, but I > am unsure how it is expected to be used. A google search did not find > any relevant documentation. Is anybody using this feature today? > > Best, > Tobias >