llvm dev - Apr 2016 - RFC: LNT/Test-suite support for custom metrics and test parameterization

Greetings everyone,

We would like to improve LNT.
The following RFC describes two LNT enhancements:

  *   Custom (extensible) metrics
  *   Test parameterization

The main idea is in document
https://docs.google.com/document/d/1zWWfu_iBQhFaHo73mhqqcL6Z82thHNAoCxaY7BveSf4/edit?usp=sharing.

Thanks,

Elena.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160418/bddb9b99/attachment.html>

Hi Elena,

Many thanks for working on this!

May I first suggest to convert the google document to email content? That may
make it a little bit easier for more people to review. It also makes sure the
content is archived on the llvm's mail servers.
I'll refrain from making detailed comments until the text is in email, so
that comments remain close to the text they make a comment on.
>From a high-level point-of-view, a few thoughts I had on the custom metrics
proposal:
* My understanding is that you suggest, to be able to add custom metrics, to
change the database schema to something that resembles a key-value pair way of
storing data more. Often, storing data in key-value pairs in a relational
databases can slow down queries a lot, depending on how data typically gets
queried. I think that for LNT usage, the schema you suggest may work well in
practice. But I think you'll need to do query time measurements and web page
load time measurements to compare the speed before and after your suggested
schema change, on a database with as much real-world data as you can lay your
hands on. Ideally, both for the sqlite and the postgres database engines.
* Quite a few users of LNT only use the server and webui, and use a different
system to produce the test data in the json file format that can be submitted to
the LNT server. Therefore, I think it's useful if you'd also describe
how the JSON file structure would change for this proposal.

For the proposal to add test-suite parameters: I'm not sure I've
understood the problem you're trying to solve well. Maybe an example of a
more concrete use case could help demonstrate what the value is of having
multiple sets of CFLAGS per test program in a single run?
It seems that you're working on a patch that adapts the Makefile structures
in test-suite. Maybe it would be better to switch to using the new cmake+lit
system to build and run the programs in the test-suite and fix the problem
there?

Thanks,

Kristof


On 18 Apr 2016, at 17:16, Elena Lepilkina <Elena.Lepilkina at
synopsys.com<mailto:Elena.Lepilkina at synopsys.com>> wrote:

Greetings everyone,

We would like to improve LNT.
The following RFC describes two LNT enhancements:

  *   Custom (extensible) metrics
  *   Test parameterization


The main idea is in document
https://docs.google.com/document/d/1zWWfu_iBQhFaHo73mhqqcL6Z82thHNAoCxaY7BveSf4/edit?usp=sharing.

Thanks,

Elena.

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160419/51ed05c6/attachment.html>

Hi Elena,

This is great, I would love to see extensible support for arbitrary metrics.

On Tue, Apr 19, 2016 at 12:39 AM, Kristof Beyls <Kristof.Beyls at arm.com>
wrote:
> Hi Elena,
>
> Many thanks for working on this!
>
> May I first suggest to convert the google document to email content? That
> may make it a little bit easier for more people to review. It also makes
> sure the content is archived on the llvm's mail servers.
> I'll refrain from making detailed comments until the text is in email,
so
> that comments remain close to the text they make a comment on.
>
Unfortunately now Kristof started the ball. :)

>
> From a high-level point-of-view, a few thoughts I had on the custom
> metrics proposal:
> * My understanding is that you suggest, to be able to add custom metrics,
> to change the database schema to something that resembles a key-value pair
> way of storing data more. Often, storing data in key-value pairs in a
> relational databases can slow down queries a lot, depending on how data
> typically gets queried. I think that for LNT usage, the schema you suggest
> may work well in practice. But I think you'll need to do query time
> measurements and web page load time measurements to compare the speed
> before and after your suggested schema change, on a database with as much
> real-world data as you can lay your hands on. Ideally, both for the sqlite
> and the postgres database engines.
>
I agree with Kristof here.

LNT actually did previously use a very key-value centric model which was
extensible, but was untenable as we scaled our server up to 10s of million
records (for reference, the llvm.org server currently has over 100 million
samples). It was also very cumbersome to program reports against, given
that we have little-to-no infrastructure for doing background processing of
the data to transform it into a more efficient query representation. Given
that it sounds like you have already implemented this (partially?) have you
tried scaling up to a large data set?

One of the reasons for the current LNT design (where we instantiate tables
in response to a configuration) was so that extensible metrics could be
added directly to the underlying SQL schema. The idea was that a
user/client would configure a particular set of metrics to be tracked when
they create the server (hopefully with features to track new metrics
later), and then we would create a corresponding schema that tracked each
metric separately.

For true "arbitrarily extensible" data where it doesn't make sense
to be
present in all samples for a schema, then I envisioned that users would
attach additional samples into a BLOB sample field (which could be JSON or
MsgPack, etc.) and then build up infrastructure for doing richer (but slow)
queries over that data.

 - Daniel

* Quite a few users of LNT only use the server and webui, and use
a
> different system to produce the test data in the json file format that can
> be submitted to the LNT server. Therefore, I think it's useful if
you'd
> also describe how the JSON file structure would change for this proposal.
>
> For the proposal to add test-suite parameters: I'm not sure I've
> understood the problem you're trying to solve well. Maybe an example of
a
> more concrete use case could help demonstrate what the value is of having
> multiple sets of CFLAGS per test program in a single run?
> It seems that you're working on a patch that adapts the Makefile
> structures in test-suite. Maybe it would be better to switch to using the
> new cmake+lit system to build and run the programs in the test-suite and
> fix the problem there?
>
> Thanks,
>
> Kristof
>
>
> On 18 Apr 2016, at 17:16, Elena Lepilkina <Elena.Lepilkina at
synopsys.com>
> wrote:
>
> Greetings everyone,
>
> We would like to improve LNT.
> The following RFC describes two LNT enhancements:
>
>    - Custom (extensible) metrics
>    - Test parameterization
>
>
> The main idea is in document
>
https://docs.google.com/document/d/1zWWfu_iBQhFaHo73mhqqcL6Z82thHNAoCxaY7BveSf4/edit?usp=sharing
> .
>
> Thanks,
>
> Elena.
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160419/4385fd3a/attachment-0001.html>