similar to: Performance metrics with LLVM

Hi Tobias, The metrics that you can collect in LNT are fixed per "test suite". There are 2 such "test suite"s defined in LNT at the moment: nts and compile. For more details on this, see http://llvm.org/docs/lnt/concepts.html#test-suites. AFAIK, If you need to collect different metrics, you'll need to define a new "test suite". I'm afraid I don't really

> On Jul 4, 2017, at 2:02 AM, Tobias Grosser <tobias.grosser at inf.ethz.ch> wrote: > >> On Tue, Jul 4, 2017, at 09:48 AM, Kristof Beyls wrote: >> Hi Tobias, >> >> The metrics that you can collect in LNT are fixed per "test suite". >> There are 2 such "test suite"s defined in LNT at the moment: nts and >> compile. >> For

The test-suite schema is defined in the DB. It is not hard to extend if you have server access. The docs detail the steps to add a new metric: http://llvm.org/docs/lnt/importing_data.html#custom-test-suites I have setup several custom suites. It works. Matthias is working on something to make that even easier by having the test suites self describe their schemas. This will still require

Greetings everyone, We would like to improve LNT. The following RFC describes two LNT enhancements: * Custom (extensible) metrics * Test parameterization The main idea is in document https://docs.google.com/document/d/1zWWfu_iBQhFaHo73mhqqcL6Z82thHNAoCxaY7BveSf4/edit?usp=sharing. Thanks, Elena. -------------- next part -------------- An HTML attachment was scrubbed... URL:

Hi Elena, This is great, I would love to see extensible support for arbitrary metrics. On Tue, Apr 19, 2016 at 12:39 AM, Kristof Beyls <Kristof.Beyls at arm.com> wrote: > Hi Elena, > > Many thanks for working on this! > > May I first suggest to convert the google document to email content? That > may make it a little bit easier for more people to review. It also makes

On 22 Apr 2016, at 11:14, Mehdi Amini <mehdi.amini at apple.com<mailto:mehdi.amini at apple.com>> wrote: On Apr 22, 2016, at 12:45 AM, Kristof Beyls via llvm-dev <llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org>> wrote: On 21 Apr 2016, at 17:44, Sergey Yakoushkin <sergey.yakoushkin at gmail.com<mailto:sergey.yakoushkin at gmail.com>> wrote: Hi

On 21 Apr 2016, at 15:00, Elena Lepilkina <Elena.Lepilkina at synopsys.com<mailto:Elena.Lepilkina at synopsys.com>> wrote: Hi Kristof and Daniel, Thanks for your answers. Unfortunately I haven’t tried scaling up to a large data set before. Today I tried and results are quite bad. So database scheme should be rebuild. Now I thought about creating one sample table for each test-suite,

On 21 Apr 2016, at 17:44, Sergey Yakoushkin <sergey.yakoushkin at gmail.com<mailto:sergey.yakoushkin at gmail.com>> wrote: Hi Kristof, The way we use LNT, we would run different configuration (e.g. -O3 vs -Os) as different "machines" in LNT's model. O2/O3 is indeed bad example. We're also using different machines for Os/O3 - such parameters apply to all

Hi, Chris. Thank you for your answer about compile tests. As I understood during looking through code of compile tests they don’t use test suite at all. Am I right? There is lack of information and examples of running compile tests in LNT documentation. We understood that there are two groups of users: users using servers and collecting a lot of data and SQLite users, but these users as I think

From: Elena Lepilkina Sent: Monday, April 25, 2016 2:33 PM To: 'James Molloy' <james at jamesmolloy.co.uk>; Kristof Beyls <Kristof.Beyls at arm.com>; Mehdi Amini <mehdi.amini at apple.com> Cc: nd <nd at arm.com>; Matthias Braun <matze at braunis.de> Subject: RE: [llvm-dev] RFC: LNT/Test-suite support for custom metrics and test parameterization Hi everyone,

> On May 12, 2016, at 11:21 PM, Elena Lepilkina via llvm-dev <llvm-dev at lists.llvm.org> wrote: > > Hi all, > > As we understood great changes will be done in LNT, so we are waiting to new LNT version and stopped our work in LNT. > > One more question about using test-suite separately with cmake. Cmake can only build all tests and generate lit tests. After that we

> On May 25, 2016, at 1:54 AM, Elena Lepilkina via llvm-dev <llvm-dev at lists.llvm.org> wrote: > > Hi Matthias, > > Thank you for your answer. > But can you answer for some more questions? > First of all, now LNT uses make-style of running tests and parse results from result csv file. Are there any plans to go to cmake? As James already said "lnt runtest

Hi Elena, > First of all, now LNT uses make-style of running tests and parse results from result csv file. Are there any plans to go to cmake? There are two test drivers in LNT. "lnt runtests nt" uses the old Makefile-based system and "lnt runtests test-suite" uses the new cmake-based system. It sounds like you are using the former and should switch to the latter.

Hi Elena, Thanks for pushing forward with this. I like the idea of using a NoSQL solution. My primary reservation is about adding the new NoSQL stuff as an extra backend. I think the current database backend and its use of SQLAlchemy is extremely complex and is in fact the most complex part of LNT. Adding something more (as opposed to *replacing* it) would just make this worse and make it more

> On May 26, 2016, at 7:08 AM, Elena Lepilkina via llvm-dev <llvm-dev at lists.llvm.org> wrote: > > I understood your modules and I see as them can be used in LNT. But there are some question about old features. > 1. With Makefiles we can write pass and collect some statistics. There was an example of branch pass. Metrics can be collected by > @-$(LOPT) -load

Hi, So, I owe you all an apology. I was totally unaware that the llvm.org server, running at UIUC, was still in operation and hosting an active LNT instance. I was under the assumption the LNT server ran elsewhere. As a result, the ability for the LNT bots to submit run information to the "old" LNT server is now gone. I am very sorry for this oversight. In speaking with Chris

Hi Matthias, Thank you for your answer. But can you answer for some more questions? First of all, now LNT uses make-style of running tests and parse results from result csv file. Are there any plans to go to cmake? As I understood lit will run and collect all metrics, but there is no opportunity to make any settings for choosing what metrics I would like to collect. Test reports files allow to

Hi all, As we understood great changes will be done in LNT, so we are waiting to new LNT version and stopped our work in LNT. One more question about using test-suite separately with cmake. Cmake can only build all tests and generate lit tests. After that we can run LIT and get report which is not equal with report (simple) got with make. Cmake test-suite version has no features to run custom

I understood your modules and I see as them can be used in LNT. But there are some question about old features. 1. With Makefiles we can write pass and collect some statistics. There was an example of branch pass. Metrics can be collected by @-$(LOPT) -load dcc888$(SHLIBEXT) -branch-counter -stats \ -time-passes -disable-output $< 2>>$@ in makefile. In report file we write how

Of course it'll be running as now. But user will need have installed MongoDB. Installation on linux with support of .deb packages is quite easy. sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv EA312927 echo "deb http://repo.mongodb.org/apt/debian wheezy/mongodb-org/3.2 main" | sudo tee /etc/apt/sources.list.d/mongodb-org-3.2.list sudo apt-get update sudo apt-get