llvm dev - May 2016 - RFC: LNT/Test-suite support for custom metrics and test parameterization

Of course it'll be running as now. But user will need have installed
MongoDB.

Installation on linux with support of .deb packages is quite easy.
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv EA312927
echo "deb http://repo.mongodb.org/apt/debian wheezy/mongodb-org/3.2
main" | sudo tee /etc/apt/sources.list.d/mongodb-org-3.2.list
sudo apt-get update
sudo apt-get install -y mongodb-org
sudo service mongod start

After that mongo will be running service in localhost:27017. After that user
should create database
mongo
use <db name>

And set database name in config file. As additional fields will be host and port
for users who will do settings for their server, which will have default
values(localhost:27017).

After that old steps
~/mysandbox/bin/python ~/lnt/setup.py develop 
lnt create ~/myperfdb
lnt runserver ~/myperfdb

MongoDB has detailed instructions for installing for all operating systems. 

Extra 6-7 commands for install MongoDB, which takes about 2 minutes and should
be executed once shouldn't be a great problem for new users, who would like
to try LNT.

Thanks,

Elena.

-----Original Message-----
From: Kristof Beyls [mailto:Kristof.Beyls at arm.com] 
Sent: Tuesday, April 26, 2016 5:15 PM
To: James Molloy <james at jamesmolloy.co.uk>
Cc: Elena Lepilkina <Elena.Lepilkina at synopsys.com>; Chris Matthews
<chris.matthews at apple.com>; llvm-dev <llvm-dev at
lists.llvm.org>; nd <nd at arm.com>
Subject: Re: [llvm-dev] RFC: LNT/Test-suite support for custom metrics and test
parameterization

I think it's important that it remains simple to get a simple local instance
up and running.
That will make it easier for people to give LNT a try, and also makes it easier
for LNT developers to have everything they need for developing locally.
I have no experience with NoSQL database engines. Would it be possible, assuming
you have the MongoDB/other packages installed for your system, to just run

$ ~/mysandbox/bin/python ~/lnt/setup.py develop $ lnt create ~/myperfdb $ lnt
runserver ~/myperfdb

and be up and running (which is roughly what is required at the moment)?
Of course good documentation could go a long way if a few extra steps would be
needed.

I do agree with James that if there are no major concerns for using a NoSQL
database, it would be easiest if we only supported one database engine.
For example, I had to do quite a bit of LNT regression test mangling to make
them work on both sqlite and postgres, and it remains harder than it should be
to write regression tests that test the database interface.

Thanks,

Kristof

> On 26 Apr 2016, at 11:37, James Molloy via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> 
> Hi Elena,
> 
> Thanks for pushing forward with this. I like the idea of using a NoSQL
solution.
> 
> My primary reservation is about adding the new NoSQL stuff as an extra
backend. I think the current database backend and its use of SQLAlchemy is
extremely complex and is in fact the most complex part of LNT. Adding something
more (as opposed to *replacing* it) would just make this worse and make it more
likely that contributors wouldn't be able to test LNT very well (having
three engines to test: SQLite, PostgreSQL and MongoDB).
> 
> I think it'd be far better all around, if we decide to go with the
NoSQL solution, to just bite the bullet and force users who want to run a server
to install MongoDB.
> 
> In my experience most of the teams I've seen using LNT have a single
LNT server instance and submit results to that, rather than launching small
instances to run "lnt viewcomparison".
> 
> Cheers,
> 
> James
> 
> On Tue, 26 Apr 2016 at 09:15 Elena Lepilkina via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> Hi everyone.
> 
>  
> 
> Thanks to everyone who took participant in discussion of this proposal.
> 
> After discussion we understood how other users use LNT and how great
datasets may be.
> 
>  
> 
> So there is new updated proposal.
> 
> (Google docs version with some images - 
> https://docs.google.com/document/d/11qHNWRIQ2gc2aWH3gNBeeCUB3-JPe7AoMt
> n7n9JoCeY/edit?usp=sharing)
> 
>  
> 
> Goal is the same.
> 
> Enable LNT support of custom metrics such as: user-defined run-time and
static metrics (power, etc.) and LLVM pass statistic counters. Provide
integration with LLVM testsuite to automatically collect LLVM statistic counters
or custom metrics.
> 
>  
> 
> Analysis of current Database
> 
>  
> 
> Limitations
> 
> 1.      This structure isn’t flexible.
> 
> There is no opportunity to run another test-suite except simple one.
> 
> 2.      Performance is quite bad when database has a lot of records.
> 
> For example, rendering graph is too slow. On
green-dragon-07-x86_64-O3-flto:42 SingleSource/Benchmarks/Shootout/objinst  
compile_time need for rendering 191.8 seconds.
> 
> 3.       It’s difficult to add new features which need queries to sample
table in database(if we use BLOB field for custom metrics).
> 
> Queries will be needed for more complex analysis. For example, if we would
like to add some additional check for tests which compile time is too long, we
should get result of query where this metric is  greater than some constant.
> 
> Or we would like to compare tests with different run options, so we should
get only some tests but not all.
> 
> BLOB field will help to save current structure and make system a bit more
flexible. But in the nearest future it will be not enough.
> 
> Getting all metrics of all tests will make work slow on great datasets. And
this way isn’t enough optimal.
> 
> So we wouldn’t like to do BLOB field, which wouldn’t help to add new
features and have flexible system in future.
> 
>  
> 
> Proposal
> 
>  
> 
> We suggest to do third part of LNT (as Chris Matthews suggested). This part
will be used for getting custom metrics and running any test-suite.
> 
> We suggest to use NoSQL database (for example, MongoDB or JSON/JSONB
extension of PostgresSQL, which let use PostgresSQL as NoSQL database) for this
part. This part will be enable if there is path to NoSQL database in config
file.
> 
> It helps to have one Sample table(collection in NoSQL). If we use
schemaless feature in MongoDB, for example, then it’s possible to add new fields
when new testsuite is running.  Then there would be one table with a lot of
fields, some of which are empty. At any moment of time it will be possible to
change schema of table(document).
> 
> A small prototype was made with MongoDB and ORM MongoEngine. This ORM was
choosen because MongoAlchemy doesn’t support schemaless features and last
MongoKit version has error with last pymongo release.
> 
> I try it on virtual machine and get next results on 5 000 000 records.
> 
> Current scheme - 13.72 seconds
> 
> MongoDB – 1.35 seconds.
> 
> Results of course will be better on real server machine .
> 
>  
> 
> For use some test-suite user should describe fields in file with format
.fields such way:
> 
> {
> 
>  "Fields" : [{
> 
>    "TestSuiteName" : "Bytecode",
> 
>    "Type" : "integer",
> 
>    "BiggerIsBetter" : 0,
> 
>     "Show" : true
> 
>  },
> 
>  {
> 
>    "TestSuiteName" : "GCC",
> 
>    "Type" : "real",
> 
>    "BiggerIsBetter" : 0,
> 
>    "Name" : "GCC time"
> 
>  },
> 
>  {
> 
>    "TestSuiteName" : "JIT",
> 
>    "Type" : "real",
> 
>    "BiggerIsBetter" : 0,
> 
>    "Name" : "JIT Compile time",
> 
>    "Show" : true
> 
>  },
> 
>  {
> 
>    "TestSuiteName" : "GCC/LLC",
> 
>    "Type" : "string",
> 
>    "BiggerIsBetter" : 0
> 
>  }]
> 
> }
> 
>  
> 
> There was added one field “Show” for describing if this metric should be
shown by default on web page (as James Molloy suggested). Other metrics would be
added to page if user chooses them in view options.
> 
>  
> 
> Conclusion
> 
>  
> 
> This change will let user to choose if he wants to use flexible powerful
system or use limited version with SQLite database.
> 
> If user chooses NoSQL version his data can be copied from its old database
to new one. This will help to use new features without losing old data.
> 
>  
> 
> The actual question is which NoSQL database will be better for LNT. We are
interested in opinions of people, who know features of LNT better.
> 
>  
> 
> Thanks,
> 
>  
> 
> Elena.
> 
>  
> 
> From: llvm-dev [mailto:llvm-dev-bounces at lists.llvm.org] On Behalf Of 
> Elena Lepilkina via llvm-dev
> Sent: Tuesday, April 26, 2016 9:07 AM
> To: chris.matthews at apple.com
> 
> 
> Cc: llvm-dev <llvm-dev at lists.llvm.org>
> Subject: Re: [llvm-dev] RFC: LNT/Test-suite support for custom metrics 
> and test parameterization
> 
>  
> 
> Hi, Chris.
> 
>  
> 
> Thank you for your answer about compile tests. As I understood during
looking through code of compile tests they don’t use test suite at all. Am I
right? There is lack of information and examples of running compile tests in LNT
documentation.
> 
> We understood that there are two groups of users: users using servers and
collecting a lot of data and SQLite users, but these users as I think wouldn’t
have about millions of sample records.
> 
> I think that it’s obvious that there is no universal solution for simple
installing process and flexible high-loaded system.
> 
> I will update proposal and take into consideration your suggestion about
third part of test-suite.
> 
>  
> 
> Thanks
> 
>  
> 
> Elena.
> 
>  
> 
> From: chris.matthews at apple.com [mailto:chris.matthews at apple.com]
> Sent: Monday, April 25, 2016 8:06 PM
> To: Elena Lepilkina <Elena.Lepilkina at synopsys.com>
> Cc: llvm-dev <llvm-dev at lists.llvm.org>
> Subject: Re: [llvm-dev] RFC: LNT/Test-suite support for custom metrics 
> and test parameterization
> 
>  
> 
> I am really torn about this.
> 
>  
> 
> When I implemented the regression tracking stuff recently, it really showed
me how badly we are scaling.  On our production server, the run ingestion can
take well over 100s.  Time is mostly spent in FieldChange generation and
regression grouping. Both have to access a lot of recent samples. This is not
the end of the world, because it runs in a background process.  Where this
really sucks is when a regression has a lot indicators. The web interface
renders these in a graph, and just trying to pull down 100 graphs worth of data
kills the server.  I ended up limiting those to a max of 10 datasets, and even
that takes 30s to load.
> 
>  
> 
> So I do think we need some improvements to the scalability.
> 
>  
> 
> LNT usage is spread between two groups. Users who setup big servers, with
Postgres and apache/Gunicorn. For those uses I think a NoSQL is the way to go.  
However, our second (and probably more common) user, is the people running
little instance on their own machine to do some local compiler benchmarking. 
Their setup process needs to be dead simple, and I think requiring a NoSQL
database to be setup on their machine first is a no starter.  Like we do with
sqlite, I think we need a transparent fall back for people who don’t have a
NoSQL database.
> 
>  
> 
> Would it be helpful to anyone if I got a dump of the llvm.org LNT Postgres
database?  It is a good dataset big dataset to test with, and I assume everyone
is okay with it being public, since the LNT server already is.
> 
>  
> 
>  
> 
> On Apr 25, 2016, at 4:33 AM, Elena Lepilkina via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> 
>  
> 
>  
> 
>  
> 
> From: Elena Lepilkina
> Sent: Monday, April 25, 2016 2:33 PM
> To: 'James Molloy' <james at jamesmolloy.co.uk>; Kristof
Beyls
> <Kristof.Beyls at arm.com>; Mehdi Amini <mehdi.amini at
apple.com>
> Cc: nd <nd at arm.com>; Matthias Braun <matze at braunis.de>
> Subject: RE: [llvm-dev] RFC: LNT/Test-suite support for custom metrics 
> and test parameterization
> 
>  
> 
> Hi everyone,
> 
>  
> 
> Thank you for your answer. BLOB format adds some more actions for working
with metrics. We know that ComparisonResult class makes analysis work. But it
gets all metrics by request from database, we will need additional time for work
with fields during analysis in ComparisonResult class. May be it will be better
to do one Sample table for each testsuite, as it was suggested before. It should
be more quickly, shouldn’t it? Moreover, next wished LNT changes will need
getting some metrics separately and BLOB format will add some delay in time for
queries.
> 
>  
> 
> As we see now problem of performance is actual, because time for rendering
graph page is about 3 minutes.
> 
> <image001.png>
> 
> So maybe it will be better to start working with NoSql databases? I made a
small prototype with TestSuite, TestSuiteFields, Test, Run and Sample tables for
getting time metrics. It works quickly. And using NoSQL helps solve problems
with  different fields for samples metrics fields. Then it will be possible to
store different metrics for different testsuites in one table.
> 
> What do you think about this proposal?
> 
> I used MongoDB, but I know that there is NoSQL extension for 
> PostgresSQL with JSONB fields which are more
> 
> effective than JSON-encoded BLOB, because it can be included in queries
very simply and let use indexes.
> 
>  
> 
> About proposal that not all metrics should be shown. It can be added as a
field in JSON in .fields file, which describes fields getted from test-suite. To
see other metrics user should choose them with checkboxes in view options. Will
be this solution suitable?
> 
> We can make as you wrote
> 
> “I'd also suggest that if we're adding many more metrics to a test,
we should create a "test sample information" page that the test link
goes to instead of just the graph. This page could contain all counter/metric
data, historic sparklines, the full graph and profiling links.
> 
> ”
> 
> But the render time of this page will be too great because of graph render
time. In my opinion, some users wouldn’t like to wait so long for see some
additional metrics.
> 
>  
> 
> Thanks for your suggestions,
> 
>  
> 
> Elena.
> 
> From: llvm-dev [mailto:llvm-dev-bounces at lists.llvm.org] On Behalf Of 
> James Molloy via llvm-dev
> Sent: Monday, April 25, 2016 12:43 PM
> To: Kristof Beyls <Kristof.Beyls at arm.com>; Mehdi Amini 
> <mehdi.amini at apple.com>
> Cc: llvm-dev <llvm-dev at lists.llvm.org>; nd <nd at arm.com>;
Matthias
> Braun <matze at braunis.de>
> Subject: Re: [llvm-dev] RFC: LNT/Test-suite support for custom metrics 
> and test parameterization
> 
>  
> 
> Hi Sergey, Elena,
> 
>  
> 
> Firstly, thanks for this RFC. It's great to see more people actively
using and modifying LNT and the test metrics support in general is rather weak
currently.
> 
>  
> 
> Metrics
> 
> -------
> 
>  
> 
> I agree with Daniel and Kristof that your proposed schema changes have the
potential to make many queries extremely slow. Certainly for the metrics
enhancements, I don't see a reason why we need such a radical change in
schema.
> 
>  
> 
> To add custom metrics on the fly, we need to change the schema for the
Sample table. Currently this consists of a column for each metric, but actually
we never ever query those metric values. We never query for example for
"all failing tests in a run" - when we do analyses we use the
ComparisonResult class which reads *all* samples from the database for a run and
performs the analysis entirely in Python.
> 
>  
> 
> Therefore, having a semi-structured format where some fields are
first-class columns and the rest are in a JSON-encoded BLOB (as Daniel suggests)
seems totally acceptable. There is certainly an argument now that we're
using the wrong backend storage solution and that a key-value store might be
more suitable, but that's a very invasive change and I don't think
we've reached the point where we need to force a move from the simplicity of
SQLite.
> 
>  
> 
> Adding an extra BLOB column would be easy - there would just need to be
logic in testsuitedb.py for reading and writing it - the Sample model class
would expose the JSON-encoded fields as normal python fields so the rest of LNT
would be isolated from this change.
> 
>  
> 
> But I think this is a small detail compared to the bigger problem of how to
effectively *display* all this new data. Currently every new metric gets its own
separate table in the report/run views, and this does not scale well at all.
> 
>  
> 
> I think we need some more concepts in the metric system to make it
scaleable:
> 
>  
> 
>   * What "attribute" of the test is this metric measuring? For
example, both "exec_time" and "score" measure the same
attribute; performance of the generated code. It's superfluous to have them
displayed in separate tables. However mem_size and compile_time both measure
completely different aspects of the test.
> 
>   * Is this metric useful to display at the top level? or should it only be
exposed when more data about a test result is requested?
> 
>     * An example of this is the pass statistics. I don't want my daily
report view cluttered by the time spent in register allocation for every test!
OK, this is useful information when debugging a problem, but it should be
available when requested rather than by default.
> 
>  
> 
> An example of why we need the above is your screenshots in your google doc.
I'm looking at the last screenshot, and it's incredibly difficult to
read and get useful information out of.
> 
>  
> 
> I'd also suggest that if we're adding many more metrics to a test,
we should create a "test sample information" page that the test link
goes to instead of just the graph. This page could contain all counter/metric
data, historic sparklines, the full graph and profiling links.
> 
>  
> 
> Cheers,
> 
>  
> 
> James
> 
>  
> 
> On Fri, 22 Apr 2016 at 10:17 Kristof Beyls via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> 
>  
> 
> On 22 Apr 2016, at 11:14, Mehdi Amini <mehdi.amini at apple.com>
wrote:
> 
>  
> 
> 
> On Apr 22, 2016, at 12:45 AM, Kristof Beyls via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> 
>  
> 
> 
> On 21 Apr 2016, at 17:44, Sergey Yakoushkin <sergey.yakoushkin at
gmail.com> wrote:
> 
>  
> 
> Hi Kristof,
> 
>  
> 
>        The way we use LNT, we would run different configuration (e.g. -O3
vs -Os) as different "machines" in LNT's model.
> 
>  
> 
> O2/O3 is indeed bad example. We're also using different machines for
Os/O3 - such parameters apply to all tests and we don't propose major
changes.
> 
> Elena was only extending LNT interface a bit to ease LLVM-testsuite
execution with different compiler or HW flags.
> 
>  
> 
> Oh I see, this boils down to extending the lnt runtest interface to be 
> able to specify a set of configurations, rather than a single 
> configuration and making
> 
> sure configurations get submitted under different machine names? We 
> kick off the different configuration runs through a script invoking 
> lnt runtest multiple
> 
> times. I don't see a big problem with extending the lnt runtest 
> interface to do this, assuming it doesn't break the underlying 
> concepts assumed throughout
> 
> LNT. Maybe the only downside is that this will add even more command 
> line options to lnt runtest, which already has a lot (too many?) 
> command line
> 
> options.
> 
>  
> 
> Maybe some changes are required to analyze and compare metrics between
"machines": e.g. code size/performance between Os/O2/O3.
> 
> Do you perform such comparisons?
> 
>  
> 
> We typically do these kinds of comparisons when we test our patches
pre-commit, i.e. comparing for example '-O3' with '-O3 'mllvm
-enable-my-new-pass'.
> 
> To stick with the LNT concepts, tests enabling new passes are stored as a
different "machine".
> 
> The only way I know to be able to do a comparison between runs on 2 
> different "machine"s is to manually edit the URL for run vs run 
> comparison
> 
> and fill in the runids of the 2 runs you want to compare.
> 
> For example, the following URL is a comparison of
green-dragon-07-x86_64-O3-flto vs green-dragon-06-x86_64-O0-g on the public
llvm.org/perf server:
> 
> http://llvm.org/perf/db_default/v4/nts/70644?compare_to=70634
> 
> I had to manually look up and fill in the run ids 70644 and 70634.
> 
> It would be great if there was a better way to be able to do these kind of
comparisons - i.e. not having to manually fill in run ids, but having a webui to
easily find and pick the runs you want to compare.
> 
> (As an aside: I find it intriguing that the URL above suggests that there
are quite a few cases where "-O0 -g" produces faster code than
"-O3 -flto").
> 
>  
> 
> Can you be more explicit which ones? I don't see any regression (other
than compared to the baseline, or on the compile time).
> 
>  
> 
> --
> 
> Mehdi
> 
>  
> 
> D'Oh! I was misinterpreting the compile time differences as execution
time differences. Indeed, there is no unexpected result in there.
> 
> Sorry for the noise!
> 
>  
> 
> Kristof
> 
>  
> 
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> 
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> 
>  
> 
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

Hi all,

First off, let me ask one question about the use case you intend to support:

Is your expectation for LLVM statistics style data that this would be
present for almost all runs in a database, or that it would only be present
for a small subset of runs?


Second, here is my general perspective on the LNT system architecture:

1. It is very important to me that LNT continue to support an
"turn-key"
experience. In fact, I *wish* that experience would get easier, not harder,
for example by moving all interactions with the server to a true admin
interface. I would prefer not to introduce additional dependencies in that
case.

I will add that I am not completely opposed to moving to a "turn-key"
architecture which requires substantially more components (e.g.,
PostgreSQL, memcached, etc.) as long as it was packaged in such a way that
it could offer a nice turn key experience. For example, if someone was
willing to implement & document a Vagrant or Docker based development
model, that would be ok with me, as long as there was still the option to
do fully native deployments on OS X with real system administrator support.

2. Our internal application architecture is severely lacking. I believe the
standard architecture these days would be to have (load-balancer +
front-end + [load-balancer] + back-end + database), and I think partly we
are suffering from missing most of that infrastructure. In particular, we
are missing two important components:
- There should be a separate backend, which would allow us to implement
improved caching, and a clear model for long-lived or persistent state. I
filed this as: https://llvm.org/bugs/show_bug.cgi?id=27534
- This would give us a place to manage background processing tasks, for
example jobs that reprocess the raw sample data for efficient queries.
If we had such a thing, we could consider using something like memcached
for improving caching (particularly in a larger deployment).

3. My intention was always to use JSON blobs for situations where a small %
of samples want to include arbitrary blobs of extra data. I think
standardizing on PostgreSQL/JSON is the right approach here, given the
standard PaaS support for PostgreSQL and its existing use within our
deployments. SQLAlchemy has native support for this.

4. I don't actually think that a NoSQL database buys us much if anything
(particularly when we have PostgreSQL/JSON available). We are not in a
situation where we would need something like eventual consistency around
writes, which leaves us wanting raw query performance over large sets of
relatively well-structured data. I believe that is a situation which is
best served by SQL with properly designed tables. This is also an area
where having an infrastructure that could handle background processing to
load query-optimized tables & indices would be valuable.

I think it is a mistake to believe that using NoSQL and a schema-less model
without also introducing substantial caching will somehow give better
performance than a real SQL database, and would be very interested to see
performance data showing otherwise (e.g., compare your MongoDB
implementation to one in which all data was in a full-schematized table
with proper indices).

5. The database schema used by LNT (which is dynamically instantiated in
response to test suite configurations) is admittedly unorthodox. However, I
still believe this is the right answer to support a turn-key performance
tracking solution that can be customized by the user based on the fields
they wish to track (in fact, if Elena's data is present for almost all
samples then it is exactly the designed use case). I'm not yet convinced
that the right answer here isn't to improve the actual implementation of
this approach; for example, we could require the definition to be created
when the LNT instance is set up, which might solve some of the issues Chris
mentioned. Or, we could expose improved tools or a UI for interacting with
the schema via an admin interface, for example to promote individual from a
JSON blob to being first class for improved performance/reporting.

6. To echo one of James's points, I also do think that ultimately the most
complicated part of adding support for arbitrary metrics is not the
database implementation, but managing the complexity in the reporting and
graphing logic (which is already cumbersome). I am fine seeing a layered
implementation approach where we first focus on how just to get the data
in, but I do think it is worth spending some time looking at how to manage
the visualizations of that data.

 - Daniel


On Tue, Apr 26, 2016 at 11:17 PM, Elena Lepilkina via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> Of course it'll be running as now. But user will need have installed
> MongoDB.
>
> Installation on linux with support of .deb packages is quite easy.
> sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv EA312927
> echo "deb http://repo.mongodb.org/apt/debian wheezy/mongodb-org/3.2
main"
> | sudo tee /etc/apt/sources.list.d/mongodb-org-3.2.list
> sudo apt-get update
> sudo apt-get install -y mongodb-org
> sudo service mongod start
>
> After that mongo will be running service in localhost:27017. After that
> user should create database
> mongo
> use <db name>
>
> And set database name in config file. As additional fields will be host
> and port for users who will do settings for their server, which will have
> default values(localhost:27017).
>
> After that old steps
> ~/mysandbox/bin/python ~/lnt/setup.py develop
> lnt create ~/myperfdb
> lnt runserver ~/myperfdb
>
> MongoDB has detailed instructions for installing for all operating systems.
>
> Extra 6-7 commands for install MongoDB, which takes about 2 minutes and
> should be executed once shouldn't be a great problem for new users, who
> would like to try LNT.
>
> Thanks,
>
> Elena.
>
> -----Original Message-----
> From: Kristof Beyls [mailto:Kristof.Beyls at arm.com]
> Sent: Tuesday, April 26, 2016 5:15 PM
> To: James Molloy <james at jamesmolloy.co.uk>
> Cc: Elena Lepilkina <Elena.Lepilkina at synopsys.com>; Chris Matthews
<
> chris.matthews at apple.com>; llvm-dev <llvm-dev at
lists.llvm.org>; nd <
> nd at arm.com>
> Subject: Re: [llvm-dev] RFC: LNT/Test-suite support for custom metrics and
> test parameterization
>
> I think it's important that it remains simple to get a simple local
> instance up and running.
> That will make it easier for people to give LNT a try, and also makes it
> easier for LNT developers to have everything they need for developing
> locally.
> I have no experience with NoSQL database engines. Would it be possible,
> assuming you have the MongoDB/other packages installed for your system, to
> just run
>
> $ ~/mysandbox/bin/python ~/lnt/setup.py develop $ lnt create ~/myperfdb $
> lnt runserver ~/myperfdb
>
> and be up and running (which is roughly what is required at the moment)?
> Of course good documentation could go a long way if a few extra steps
> would be needed.
>
> I do agree with James that if there are no major concerns for using a
> NoSQL database, it would be easiest if we only supported one database
> engine.
> For example, I had to do quite a bit of LNT regression test mangling to
> make them work on both sqlite and postgres, and it remains harder than it
> should be to write regression tests that test the database interface.
>
> Thanks,
>
> Kristof
>
>
> > On 26 Apr 2016, at 11:37, James Molloy via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
> >
> > Hi Elena,
> >
> > Thanks for pushing forward with this. I like the idea of using a NoSQL
> solution.
> >
> > My primary reservation is about adding the new NoSQL stuff as an extra
> backend. I think the current database backend and its use of SQLAlchemy is
> extremely complex and is in fact the most complex part of LNT. Adding
> something more (as opposed to *replacing* it) would just make this worse
> and make it more likely that contributors wouldn't be able to test LNT
very
> well (having three engines to test: SQLite, PostgreSQL and MongoDB).
> >
> > I think it'd be far better all around, if we decide to go with the
NoSQL
> solution, to just bite the bullet and force users who want to run a server
> to install MongoDB.
> >
> > In my experience most of the teams I've seen using LNT have a
single LNT
> server instance and submit results to that, rather than launching small
> instances to run "lnt viewcomparison".
> >
> > Cheers,
> >
> > James
> >
> > On Tue, 26 Apr 2016 at 09:15 Elena Lepilkina via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
> > Hi everyone.
> >
> >
> >
> > Thanks to everyone who took participant in discussion of this
proposal.
> >
> > After discussion we understood how other users use LNT and how great
> datasets may be.
> >
> >
> >
> > So there is new updated proposal.
> >
> > (Google docs version with some images -
> > https://docs.google.com/document/d/11qHNWRIQ2gc2aWH3gNBeeCUB3-JPe7AoMt
> > n7n9JoCeY/edit?usp=sharing)
> >
> >
> >
> > Goal is the same.
> >
> > Enable LNT support of custom metrics such as: user-defined run-time
and
> static metrics (power, etc.) and LLVM pass statistic counters. Provide
> integration with LLVM testsuite to automatically collect LLVM statistic
> counters or custom metrics.
> >
> >
> >
> > Analysis of current Database
> >
> >
> >
> > Limitations
> >
> > 1.      This structure isn’t flexible.
> >
> > There is no opportunity to run another test-suite except simple one.
> >
> > 2.      Performance is quite bad when database has a lot of records.
> >
> > For example, rendering graph is too slow. On
> green-dragon-07-x86_64-O3-flto:42 SingleSource/Benchmarks/Shootout/objinst
>  compile_time need for rendering 191.8 seconds.
> >
> > 3.       It’s difficult to add new features which need queries to
sample
> table in database(if we use BLOB field for custom metrics).
> >
> > Queries will be needed for more complex analysis. For example, if we
> would like to add some additional check for tests which compile time is too
> long, we should get result of query where this metric is  greater than some
> constant.
> >
> > Or we would like to compare tests with different run options, so we
> should get only some tests but not all.
> >
> > BLOB field will help to save current structure and make system a bit
> more flexible. But in the nearest future it will be not enough.
> >
> > Getting all metrics of all tests will make work slow on great
datasets.
> And this way isn’t enough optimal.
> >
> > So we wouldn’t like to do BLOB field, which wouldn’t help to add new
> features and have flexible system in future.
> >
> >
> >
> > Proposal
> >
> >
> >
> > We suggest to do third part of LNT (as Chris Matthews suggested). This
> part will be used for getting custom metrics and running any test-suite.
> >
> > We suggest to use NoSQL database (for example, MongoDB or JSON/JSONB
> extension of PostgresSQL, which let use PostgresSQL as NoSQL database) for
> this part. This part will be enable if there is path to NoSQL database in
> config file.
> >
> > It helps to have one Sample table(collection in NoSQL). If we use
> schemaless feature in MongoDB, for example, then it’s possible to add new
> fields when new testsuite is running.  Then there would be one table with a
> lot of fields, some of which are empty. At any moment of time it will be
> possible to change schema of table(document).
> >
> > A small prototype was made with MongoDB and ORM MongoEngine. This ORM
> was choosen because MongoAlchemy doesn’t support schemaless features and
> last MongoKit version has error with last pymongo release.
> >
> > I try it on virtual machine and get next results on 5 000 000 records.
> >
> > Current scheme - 13.72 seconds
> >
> > MongoDB – 1.35 seconds.
> >
> > Results of course will be better on real server machine .
> >
> >
> >
> > For use some test-suite user should describe fields in file with
format
> .fields such way:
> >
> > {
> >
> >  "Fields" : [{
> >
> >    "TestSuiteName" : "Bytecode",
> >
> >    "Type" : "integer",
> >
> >    "BiggerIsBetter" : 0,
> >
> >     "Show" : true
> >
> >  },
> >
> >  {
> >
> >    "TestSuiteName" : "GCC",
> >
> >    "Type" : "real",
> >
> >    "BiggerIsBetter" : 0,
> >
> >    "Name" : "GCC time"
> >
> >  },
> >
> >  {
> >
> >    "TestSuiteName" : "JIT",
> >
> >    "Type" : "real",
> >
> >    "BiggerIsBetter" : 0,
> >
> >    "Name" : "JIT Compile time",
> >
> >    "Show" : true
> >
> >  },
> >
> >  {
> >
> >    "TestSuiteName" : "GCC/LLC",
> >
> >    "Type" : "string",
> >
> >    "BiggerIsBetter" : 0
> >
> >  }]
> >
> > }
> >
> >
> >
> > There was added one field “Show” for describing if this metric should
be
> shown by default on web page (as James Molloy suggested). Other metrics
> would be added to page if user chooses them in view options.
> >
> >
> >
> > Conclusion
> >
> >
> >
> > This change will let user to choose if he wants to use flexible
powerful
> system or use limited version with SQLite database.
> >
> > If user chooses NoSQL version his data can be copied from its old
> database to new one. This will help to use new features without losing old
> data.
> >
> >
> >
> > The actual question is which NoSQL database will be better for LNT. We
> are interested in opinions of people, who know features of LNT better.
> >
> >
> >
> > Thanks,
> >
> >
> >
> > Elena.
> >
> >
> >
> > From: llvm-dev [mailto:llvm-dev-bounces at lists.llvm.org] On Behalf
Of
> > Elena Lepilkina via llvm-dev
> > Sent: Tuesday, April 26, 2016 9:07 AM
> > To: chris.matthews at apple.com
> >
> >
> > Cc: llvm-dev <llvm-dev at lists.llvm.org>
> > Subject: Re: [llvm-dev] RFC: LNT/Test-suite support for custom metrics
> > and test parameterization
> >
> >
> >
> > Hi, Chris.
> >
> >
> >
> > Thank you for your answer about compile tests. As I understood during
> looking through code of compile tests they don’t use test suite at all. Am
> I right? There is lack of information and examples of running compile tests
> in LNT documentation.
> >
> > We understood that there are two groups of users: users using servers
> and collecting a lot of data and SQLite users, but these users as I think
> wouldn’t have about millions of sample records.
> >
> > I think that it’s obvious that there is no universal solution for
simple
> installing process and flexible high-loaded system.
> >
> > I will update proposal and take into consideration your suggestion
about
> third part of test-suite.
> >
> >
> >
> > Thanks
> >
> >
> >
> > Elena.
> >
> >
> >
> > From: chris.matthews at apple.com [mailto:chris.matthews at apple.com]
> > Sent: Monday, April 25, 2016 8:06 PM
> > To: Elena Lepilkina <Elena.Lepilkina at synopsys.com>
> > Cc: llvm-dev <llvm-dev at lists.llvm.org>
> > Subject: Re: [llvm-dev] RFC: LNT/Test-suite support for custom metrics
> > and test parameterization
> >
> >
> >
> > I am really torn about this.
> >
> >
> >
> > When I implemented the regression tracking stuff recently, it really
> showed me how badly we are scaling.  On our production server, the run
> ingestion can take well over 100s.  Time is mostly spent in FieldChange
> generation and regression grouping. Both have to access a lot of recent
> samples. This is not the end of the world, because it runs in a background
> process.  Where this really sucks is when a regression has a lot
> indicators. The web interface renders these in a graph, and just trying to
> pull down 100 graphs worth of data kills the server.  I ended up limiting
> those to a max of 10 datasets, and even that takes 30s to load.
> >
> >
> >
> > So I do think we need some improvements to the scalability.
> >
> >
> >
> > LNT usage is spread between two groups. Users who setup big servers,
> with Postgres and apache/Gunicorn. For those uses I think a NoSQL is the
> way to go.   However, our second (and probably more common) user, is the
> people running little instance on their own machine to do some local
> compiler benchmarking.  Their setup process needs to be dead simple, and I
> think requiring a NoSQL database to be setup on their machine first is a no
> starter.  Like we do with sqlite, I think we need a transparent fall back
> for people who don’t have a NoSQL database.
> >
> >
> >
> > Would it be helpful to anyone if I got a dump of the llvm.org LNT
> Postgres database?  It is a good dataset big dataset to test with, and I
> assume everyone is okay with it being public, since the LNT server already
> is.
> >
> >
> >
> >
> >
> > On Apr 25, 2016, at 4:33 AM, Elena Lepilkina via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
> >
> >
> >
> >
> >
> >
> >
> > From: Elena Lepilkina
> > Sent: Monday, April 25, 2016 2:33 PM
> > To: 'James Molloy' <james at jamesmolloy.co.uk>; Kristof
Beyls
> > <Kristof.Beyls at arm.com>; Mehdi Amini <mehdi.amini at
apple.com>
> > Cc: nd <nd at arm.com>; Matthias Braun <matze at
braunis.de>
> > Subject: RE: [llvm-dev] RFC: LNT/Test-suite support for custom metrics
> > and test parameterization
> >
> >
> >
> > Hi everyone,
> >
> >
> >
> > Thank you for your answer. BLOB format adds some more actions for
> working with metrics. We know that ComparisonResult class makes analysis
> work. But it gets all metrics by request from database, we will need
> additional time for work with fields during analysis in ComparisonResult
> class. May be it will be better to do one Sample table for each testsuite,
> as it was suggested before. It should be more quickly, shouldn’t it?
> Moreover, next wished LNT changes will need getting some metrics separately
> and BLOB format will add some delay in time for queries.
> >
> >
> >
> > As we see now problem of performance is actual, because time for
> rendering graph page is about 3 minutes.
> >
> > <image001.png>
> >
> > So maybe it will be better to start working with NoSql databases? I
made
> a small prototype with TestSuite, TestSuiteFields, Test, Run and Sample
> tables for getting time metrics. It works quickly. And using NoSQL helps
> solve problems with  different fields for samples metrics fields. Then it
> will be possible to store different metrics for different testsuites in one
> table.
> >
> > What do you think about this proposal?
> >
> > I used MongoDB, but I know that there is NoSQL extension for
> > PostgresSQL with JSONB fields which are more
> >
> > effective than JSON-encoded BLOB, because it can be included in
queries
> very simply and let use indexes.
> >
> >
> >
> > About proposal that not all metrics should be shown. It can be added
as
> a field in JSON in .fields file, which describes fields getted from
> test-suite. To see other metrics user should choose them with checkboxes in
> view options. Will be this solution suitable?
> >
> > We can make as you wrote
> >
> > “I'd also suggest that if we're adding many more metrics to a
test, we
> should create a "test sample information" page that the test link
goes to
> instead of just the graph. This page could contain all counter/metric data,
> historic sparklines, the full graph and profiling links.
> >
> > ”
> >
> > But the render time of this page will be too great because of graph
> render time. In my opinion, some users wouldn’t like to wait so long for
> see some additional metrics.
> >
> >
> >
> > Thanks for your suggestions,
> >
> >
> >
> > Elena.
> >
> > From: llvm-dev [mailto:llvm-dev-bounces at lists.llvm.org] On Behalf
Of
> > James Molloy via llvm-dev
> > Sent: Monday, April 25, 2016 12:43 PM
> > To: Kristof Beyls <Kristof.Beyls at arm.com>; Mehdi Amini
> > <mehdi.amini at apple.com>
> > Cc: llvm-dev <llvm-dev at lists.llvm.org>; nd <nd at
arm.com>; Matthias
> > Braun <matze at braunis.de>
> > Subject: Re: [llvm-dev] RFC: LNT/Test-suite support for custom metrics
> > and test parameterization
> >
> >
> >
> > Hi Sergey, Elena,
> >
> >
> >
> > Firstly, thanks for this RFC. It's great to see more people
actively
> using and modifying LNT and the test metrics support in general is rather
> weak currently.
> >
> >
> >
> > Metrics
> >
> > -------
> >
> >
> >
> > I agree with Daniel and Kristof that your proposed schema changes have
> the potential to make many queries extremely slow. Certainly for the
> metrics enhancements, I don't see a reason why we need such a radical
> change in schema.
> >
> >
> >
> > To add custom metrics on the fly, we need to change the schema for the
> Sample table. Currently this consists of a column for each metric, but
> actually we never ever query those metric values. We never query for
> example for "all failing tests in a run" - when we do analyses we
use the
> ComparisonResult class which reads *all* samples from the database for a
> run and performs the analysis entirely in Python.
> >
> >
> >
> > Therefore, having a semi-structured format where some fields are
> first-class columns and the rest are in a JSON-encoded BLOB (as Daniel
> suggests) seems totally acceptable. There is certainly an argument now that
> we're using the wrong backend storage solution and that a key-value
store
> might be more suitable, but that's a very invasive change and I
don't think
> we've reached the point where we need to force a move from the
simplicity
> of SQLite.
> >
> >
> >
> > Adding an extra BLOB column would be easy - there would just need to
be
> logic in testsuitedb.py for reading and writing it - the Sample model class
> would expose the JSON-encoded fields as normal python fields so the rest of
> LNT would be isolated from this change.
> >
> >
> >
> > But I think this is a small detail compared to the bigger problem of
how
> to effectively *display* all this new data. Currently every new metric gets
> its own separate table in the report/run views, and this does not scale
> well at all.
> >
> >
> >
> > I think we need some more concepts in the metric system to make it
> scaleable:
> >
> >
> >
> >   * What "attribute" of the test is this metric measuring?
For example,
> both "exec_time" and "score" measure the same
attribute; performance of the
> generated code. It's superfluous to have them displayed in separate
tables.
> However mem_size and compile_time both measure completely different aspects
> of the test.
> >
> >   * Is this metric useful to display at the top level? or should it
only
> be exposed when more data about a test result is requested?
> >
> >     * An example of this is the pass statistics. I don't want my
daily
> report view cluttered by the time spent in register allocation for every
> test! OK, this is useful information when debugging a problem, but it
> should be available when requested rather than by default.
> >
> >
> >
> > An example of why we need the above is your screenshots in your google
> doc. I'm looking at the last screenshot, and it's incredibly
difficult to
> read and get useful information out of.
> >
> >
> >
> > I'd also suggest that if we're adding many more metrics to a
test, we
> should create a "test sample information" page that the test link
goes to
> instead of just the graph. This page could contain all counter/metric data,
> historic sparklines, the full graph and profiling links.
> >
> >
> >
> > Cheers,
> >
> >
> >
> > James
> >
> >
> >
> > On Fri, 22 Apr 2016 at 10:17 Kristof Beyls via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
> >
> >
> >
> > On 22 Apr 2016, at 11:14, Mehdi Amini <mehdi.amini at apple.com>
wrote:
> >
> >
> >
> >
> > On Apr 22, 2016, at 12:45 AM, Kristof Beyls via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
> >
> >
> >
> >
> > On 21 Apr 2016, at 17:44, Sergey Yakoushkin <sergey.yakoushkin at
gmail.com>
> wrote:
> >
> >
> >
> > Hi Kristof,
> >
> >
> >
> >        The way we use LNT, we would run different configuration (e.g.
> -O3 vs -Os) as different "machines" in LNT's model.
> >
> >
> >
> > O2/O3 is indeed bad example. We're also using different machines
for
> Os/O3 - such parameters apply to all tests and we don't propose major
> changes.
> >
> > Elena was only extending LNT interface a bit to ease LLVM-testsuite
> execution with different compiler or HW flags.
> >
> >
> >
> > Oh I see, this boils down to extending the lnt runtest interface to be
> > able to specify a set of configurations, rather than a single
> > configuration and making
> >
> > sure configurations get submitted under different machine names? We
> > kick off the different configuration runs through a script invoking
> > lnt runtest multiple
> >
> > times. I don't see a big problem with extending the lnt runtest
> > interface to do this, assuming it doesn't break the underlying
> > concepts assumed throughout
> >
> > LNT. Maybe the only downside is that this will add even more command
> > line options to lnt runtest, which already has a lot (too many?)
> > command line
> >
> > options.
> >
> >
> >
> > Maybe some changes are required to analyze and compare metrics between
> "machines": e.g. code size/performance between Os/O2/O3.
> >
> > Do you perform such comparisons?
> >
> >
> >
> > We typically do these kinds of comparisons when we test our patches
> pre-commit, i.e. comparing for example '-O3' with '-O3
'mllvm
> -enable-my-new-pass'.
> >
> > To stick with the LNT concepts, tests enabling new passes are stored
as
> a different "machine".
> >
> > The only way I know to be able to do a comparison between runs on 2
> > different "machine"s is to manually edit the URL for run vs
run
> > comparison
> >
> > and fill in the runids of the 2 runs you want to compare.
> >
> > For example, the following URL is a comparison of
> green-dragon-07-x86_64-O3-flto vs green-dragon-06-x86_64-O0-g on the public
> llvm.org/perf server:
> >
> > http://llvm.org/perf/db_default/v4/nts/70644?compare_to=70634
> >
> > I had to manually look up and fill in the run ids 70644 and 70634.
> >
> > It would be great if there was a better way to be able to do these
kind
> of comparisons - i.e. not having to manually fill in run ids, but having a
> webui to easily find and pick the runs you want to compare.
> >
> > (As an aside: I find it intriguing that the URL above suggests that
> there are quite a few cases where "-O0 -g" produces faster code
than "-O3
> -flto").
> >
> >
> >
> > Can you be more explicit which ones? I don't see any regression
(other
> than compared to the baseline, or on the compile time).
> >
> >
> >
> > --
> >
> > Mehdi
> >
> >
> >
> > D'Oh! I was misinterpreting the compile time differences as
execution
> time differences. Indeed, there is no unexpected result in there.
> >
> > Sorry for the noise!
> >
> >
> >
> > Kristof
> >
> >
> >
> > _______________________________________________
> > LLVM Developers mailing list
> > llvm-dev at lists.llvm.org
> > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> >
> > _______________________________________________
> > LLVM Developers mailing list
> > llvm-dev at lists.llvm.org
> > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> >
> >
> >
> > _______________________________________________
> > LLVM Developers mailing list
> > llvm-dev at lists.llvm.org
> > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> > _______________________________________________
> > LLVM Developers mailing list
> > llvm-dev at lists.llvm.org
> > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160427/3b01d9b1/attachment-0001.html>

Hi Daniel,

About your suggestion with BLOB. As I understood, we need change Sample table to
do custom metrics next way: remove compile_time, execution_time and etc columns,
and make one column BLOB with JSON {“compile_time”: <value>, “exec_time”:
<value>}. We can do only such way, because other testsuites don’t have
these fields, but have others.
For example, user has 1 milion samples in table and 4 testsuite: simple, nightly
and 2 custom testsuite in proportion 25%(250 000 samples for each). Now I would
like to get for all runs of testsuite simple tests, which have compile_time >
constant and make some analysis and check if such big compile_time reasonable or
there is an error. I would take all 250 000 records and check field value by
myself. I think that it will be quite slow.
Even for showing data decoding from BLOB for great number of records will affect
perfomance badly.

Thanks,

Elena.
From: daniel.dunbar at gmail.com [mailto:daniel.dunbar at gmail.com] On Behalf
Of Daniel Dunbar
Sent: Wednesday, April 27, 2016 10:15 AM
To: Elena Lepilkina <Elena.Lepilkina at synopsys.com>
Cc: Kristof Beyls <Kristof.Beyls at arm.com>; James Molloy <james at
jamesmolloy.co.uk>; llvm-dev <llvm-dev at lists.llvm.org>; nd <nd at
arm.com>
Subject: Re: [llvm-dev] RFC: LNT/Test-suite support for custom metrics and test
parameterization

Hi all,

First off, let me ask one question about the use case you intend to support:

Is your expectation for LLVM statistics style data that this would be present
for almost all runs in a database, or that it would only be present for a small
subset of runs?


Second, here is my general perspective on the LNT system architecture:

1. It is very important to me that LNT continue to support an
"turn-key" experience. In fact, I *wish* that experience would get
easier, not harder, for example by moving all interactions with the server to a
true admin interface. I would prefer not to introduce additional dependencies in
that case.

I will add that I am not completely opposed to moving to a "turn-key"
architecture which requires substantially more components (e.g., PostgreSQL,
memcached, etc.) as long as it was packaged in such a way that it could offer a
nice turn key experience. For example, if someone was willing to implement &
document a Vagrant or Docker based development model, that would be ok with me,
as long as there was still the option to do fully native deployments on OS X
with real system administrator support.

2. Our internal application architecture is severely lacking. I believe the
standard architecture these days would be to have (load-balancer + front-end +
[load-balancer] + back-end + database), and I think partly we are suffering from
missing most of that infrastructure. In particular, we are missing two important
components:
- There should be a separate backend, which would allow us to implement improved
caching, and a clear model for long-lived or persistent state. I filed this as:
https://llvm.org/bugs/show_bug.cgi?id=27534
- This would give us a place to manage background processing tasks, for example
jobs that reprocess the raw sample data for efficient queries.
If we had such a thing, we could consider using something like memcached for
improving caching (particularly in a larger deployment).

3. My intention was always to use JSON blobs for situations where a small % of
samples want to include arbitrary blobs of extra data. I think standardizing on
PostgreSQL/JSON is the right approach here, given the standard PaaS support for
PostgreSQL and its existing use within our deployments. SQLAlchemy has native
support for this.

4. I don't actually think that a NoSQL database buys us much if anything
(particularly when we have PostgreSQL/JSON available). We are not in a situation
where we would need something like eventual consistency around writes, which
leaves us wanting raw query performance over large sets of relatively
well-structured data. I believe that is a situation which is best served by SQL
with properly designed tables. This is also an area where having an
infrastructure that could handle background processing to load query-optimized
tables & indices would be valuable.

I think it is a mistake to believe that using NoSQL and a schema-less model
without also introducing substantial caching will somehow give better
performance than a real SQL database, and would be very interested to see
performance data showing otherwise (e.g., compare your MongoDB implementation to
one in which all data was in a full-schematized table with proper indices).

5. The database schema used by LNT (which is dynamically instantiated in
response to test suite configurations) is admittedly unorthodox. However, I
still believe this is the right answer to support a turn-key performance
tracking solution that can be customized by the user based on the fields they
wish to track (in fact, if Elena's data is present for almost all samples
then it is exactly the designed use case). I'm not yet convinced that the
right answer here isn't to improve the actual implementation of this
approach; for example, we could require the definition to be created when the
LNT instance is set up, which might solve some of the issues Chris mentioned.
Or, we could expose improved tools or a UI for interacting with the schema via
an admin interface, for example to promote individual from a JSON blob to being
first class for improved performance/reporting.

6. To echo one of James's points, I also do think that ultimately the most
complicated part of adding support for arbitrary metrics is not the database
implementation, but managing the complexity in the reporting and graphing logic
(which is already cumbersome). I am fine seeing a layered implementation
approach where we first focus on how just to get the data in, but I do think it
is worth spending some time looking at how to manage the visualizations of that
data.

 - Daniel


On Tue, Apr 26, 2016 at 11:17 PM, Elena Lepilkina via llvm-dev <llvm-dev at
lists.llvm.org<mailto:llvm-dev at lists.llvm.org>> wrote:
Of course it'll be running as now. But user will need have installed
MongoDB.

Installation on linux with support of .deb packages is quite easy.
sudo apt-key adv --keyserver
hkp://keyserver.ubuntu.com:80<http://keyserver.ubuntu.com:80> --recv
EA312927
echo "deb http://repo.mongodb.org/apt/debian wheezy/mongodb-org/3.2
main" | sudo tee /etc/apt/sources.list.d/mongodb-org-3.2.list
sudo apt-get update
sudo apt-get install -y mongodb-org
sudo service mongod start

After that mongo will be running service in localhost:27017. After that user
should create database
mongo
use <db name>

And set database name in config file. As additional fields will be host and port
for users who will do settings for their server, which will have default
values(localhost:27017).

After that old steps
~/mysandbox/bin/python ~/lnt/setup.py develop
lnt create ~/myperfdb
lnt runserver ~/myperfdb

MongoDB has detailed instructions for installing for all operating systems.

Extra 6-7 commands for install MongoDB, which takes about 2 minutes and should
be executed once shouldn't be a great problem for new users, who would like
to try LNT.

Thanks,

Elena.

-----Original Message-----
From: Kristof Beyls [mailto:Kristof.Beyls at arm.com<mailto:Kristof.Beyls at
arm.com>]
Sent: Tuesday, April 26, 2016 5:15 PM
To: James Molloy <james at jamesmolloy.co.uk<mailto:james at
jamesmolloy.co.uk>>
Cc: Elena Lepilkina <Elena.Lepilkina at
synopsys.com<mailto:Elena.Lepilkina at synopsys.com>>; Chris Matthews
<chris.matthews at apple.com<mailto:chris.matthews at apple.com>>;
llvm-dev <llvm-dev at lists.llvm.org<mailto:llvm-dev at
lists.llvm.org>>; nd <nd at arm.com<mailto:nd at arm.com>>
Subject: Re: [llvm-dev] RFC: LNT/Test-suite support for custom metrics and test
parameterization
I think it's important that it remains simple to get a simple local instance
up and running.
That will make it easier for people to give LNT a try, and also makes it easier
for LNT developers to have everything they need for developing locally.
I have no experience with NoSQL database engines. Would it be possible, assuming
you have the MongoDB/other packages installed for your system, to just run

$ ~/mysandbox/bin/python ~/lnt/setup.py develop $ lnt create ~/myperfdb $ lnt
runserver ~/myperfdb

and be up and running (which is roughly what is required at the moment)?
Of course good documentation could go a long way if a few extra steps would be
needed.

I do agree with James that if there are no major concerns for using a NoSQL
database, it would be easiest if we only supported one database engine.
For example, I had to do quite a bit of LNT regression test mangling to make
them work on both sqlite and postgres, and it remains harder than it should be
to write regression tests that test the database interface.

Thanks,

Kristof

> On 26 Apr 2016, at 11:37, James Molloy via llvm-dev <llvm-dev at
lists.llvm.org<mailto:llvm-dev at lists.llvm.org>> wrote:
>
> Hi Elena,
>
> Thanks for pushing forward with this. I like the idea of using a NoSQL
solution.
>
> My primary reservation is about adding the new NoSQL stuff as an extra
backend. I think the current database backend and its use of SQLAlchemy is
extremely complex and is in fact the most complex part of LNT. Adding something
more (as opposed to *replacing* it) would just make this worse and make it more
likely that contributors wouldn't be able to test LNT very well (having
three engines to test: SQLite, PostgreSQL and MongoDB).
>
> I think it'd be far better all around, if we decide to go with the
NoSQL solution, to just bite the bullet and force users who want to run a server
to install MongoDB.
>
> In my experience most of the teams I've seen using LNT have a single
LNT server instance and submit results to that, rather than launching small
instances to run "lnt viewcomparison".
>
> Cheers,
>
> James
>
> On Tue, 26 Apr 2016 at 09:15 Elena Lepilkina via llvm-dev <llvm-dev at
lists.llvm.org<mailto:llvm-dev at lists.llvm.org>> wrote:
> Hi everyone.
>
>
>
> Thanks to everyone who took participant in discussion of this proposal.
>
> After discussion we understood how other users use LNT and how great
datasets may be.
>
>
>
> So there is new updated proposal.
>
> (Google docs version with some images -
> https://docs.google.com/document/d/11qHNWRIQ2gc2aWH3gNBeeCUB3-JPe7AoMt
> n7n9JoCeY/edit?usp=sharing)
>
>
>
> Goal is the same.
>
> Enable LNT support of custom metrics such as: user-defined run-time and
static metrics (power, etc.) and LLVM pass statistic counters. Provide
integration with LLVM testsuite to automatically collect LLVM statistic counters
or custom metrics.
>
>
>
> Analysis of current Database
>
>
>
> Limitations
>
> 1.      This structure isn’t flexible.
>
> There is no opportunity to run another test-suite except simple one.
>
> 2.      Performance is quite bad when database has a lot of records.
>
> For example, rendering graph is too slow. On
green-dragon-07-x86_64-O3-flto:42 SingleSource/Benchmarks/Shootout/objinst  
compile_time need for rendering 191.8 seconds.
>
> 3.       It’s difficult to add new features which need queries to sample
table in database(if we use BLOB field for custom metrics).
>
> Queries will be needed for more complex analysis. For example, if we would
like to add some additional check for tests which compile time is too long, we
should get result of query where this metric is  greater than some constant.
>
> Or we would like to compare tests with different run options, so we should
get only some tests but not all.
>
> BLOB field will help to save current structure and make system a bit more
flexible. But in the nearest future it will be not enough.
>
> Getting all metrics of all tests will make work slow on great datasets. And
this way isn’t enough optimal.
>
> So we wouldn’t like to do BLOB field, which wouldn’t help to add new
features and have flexible system in future.
>
>
>
> Proposal
>
>
>
> We suggest to do third part of LNT (as Chris Matthews suggested). This part
will be used for getting custom metrics and running any test-suite.
>
> We suggest to use NoSQL database (for example, MongoDB or JSON/JSONB
extension of PostgresSQL, which let use PostgresSQL as NoSQL database) for this
part. This part will be enable if there is path to NoSQL database in config
file.
>
> It helps to have one Sample table(collection in NoSQL). If we use
schemaless feature in MongoDB, for example, then it’s possible to add new fields
when new testsuite is running.  Then there would be one table with a lot of
fields, some of which are empty. At any moment of time it will be possible to
change schema of table(document).
>
> A small prototype was made with MongoDB and ORM MongoEngine. This ORM was
choosen because MongoAlchemy doesn’t support schemaless features and last
MongoKit version has error with last pymongo release.
>
> I try it on virtual machine and get next results on 5 000 000 records.
>
> Current scheme - 13.72 seconds
>
> MongoDB – 1.35 seconds.
>
> Results of course will be better on real server machine .
>
>
>
> For use some test-suite user should describe fields in file with format
.fields such way:
>
> {
>
>  "Fields" : [{
>
>    "TestSuiteName" : "Bytecode",
>
>    "Type" : "integer",
>
>    "BiggerIsBetter" : 0,
>
>     "Show" : true
>
>  },
>
>  {
>
>    "TestSuiteName" : "GCC",
>
>    "Type" : "real",
>
>    "BiggerIsBetter" : 0,
>
>    "Name" : "GCC time"
>
>  },
>
>  {
>
>    "TestSuiteName" : "JIT",
>
>    "Type" : "real",
>
>    "BiggerIsBetter" : 0,
>
>    "Name" : "JIT Compile time",
>
>    "Show" : true
>
>  },
>
>  {
>
>    "TestSuiteName" : "GCC/LLC",
>
>    "Type" : "string",
>
>    "BiggerIsBetter" : 0
>
>  }]
>
> }
>
>
>
> There was added one field “Show” for describing if this metric should be
shown by default on web page (as James Molloy suggested). Other metrics would be
added to page if user chooses them in view options.
>
>
>
> Conclusion
>
>
>
> This change will let user to choose if he wants to use flexible powerful
system or use limited version with SQLite database.
>
> If user chooses NoSQL version his data can be copied from its old database
to new one. This will help to use new features without losing old data.
>
>
>
> The actual question is which NoSQL database will be better for LNT. We are
interested in opinions of people, who know features of LNT better.
>
>
>
> Thanks,
>
>
>
> Elena.
>
>
>
> From: llvm-dev [mailto:llvm-dev-bounces at
lists.llvm.org<mailto:llvm-dev-bounces at lists.llvm.org>] On Behalf Of
> Elena Lepilkina via llvm-dev
> Sent: Tuesday, April 26, 2016 9:07 AM
> To: chris.matthews at apple.com<mailto:chris.matthews at apple.com>
>
>
> Cc: llvm-dev <llvm-dev at lists.llvm.org<mailto:llvm-dev at
lists.llvm.org>>
> Subject: Re: [llvm-dev] RFC: LNT/Test-suite support for custom metrics
> and test parameterization
>
>
>
> Hi, Chris.
>
>
>
> Thank you for your answer about compile tests. As I understood during
looking through code of compile tests they don’t use test suite at all. Am I
right? There is lack of information and examples of running compile tests in LNT
documentation.
>
> We understood that there are two groups of users: users using servers and
collecting a lot of data and SQLite users, but these users as I think wouldn’t
have about millions of sample records.
>
> I think that it’s obvious that there is no universal solution for simple
installing process and flexible high-loaded system.
>
> I will update proposal and take into consideration your suggestion about
third part of test-suite.
>
>
>
> Thanks
>
>
>
> Elena.
>
>
>
> From: chris.matthews at apple.com<mailto:chris.matthews at apple.com>
[mailto:chris.matthews at apple.com<mailto:chris.matthews at apple.com>]
> Sent: Monday, April 25, 2016 8:06 PM
> To: Elena Lepilkina <Elena.Lepilkina at
synopsys.com<mailto:Elena.Lepilkina at synopsys.com>>
> Cc: llvm-dev <llvm-dev at lists.llvm.org<mailto:llvm-dev at
lists.llvm.org>>
> Subject: Re: [llvm-dev] RFC: LNT/Test-suite support for custom metrics
> and test parameterization
>
>
>
> I am really torn about this.
>
>
>
> When I implemented the regression tracking stuff recently, it really showed
me how badly we are scaling.  On our production server, the run ingestion can
take well over 100s.  Time is mostly spent in FieldChange generation and
regression grouping. Both have to access a lot of recent samples. This is not
the end of the world, because it runs in a background process.  Where this
really sucks is when a regression has a lot indicators. The web interface
renders these in a graph, and just trying to pull down 100 graphs worth of data
kills the server.  I ended up limiting those to a max of 10 datasets, and even
that takes 30s to load.
>
>
>
> So I do think we need some improvements to the scalability.
>
>
>
> LNT usage is spread between two groups. Users who setup big servers, with
Postgres and apache/Gunicorn. For those uses I think a NoSQL is the way to go.  
However, our second (and probably more common) user, is the people running
little instance on their own machine to do some local compiler benchmarking. 
Their setup process needs to be dead simple, and I think requiring a NoSQL
database to be setup on their machine first is a no starter.  Like we do with
sqlite, I think we need a transparent fall back for people who don’t have a
NoSQL database.
>
>
>
> Would it be helpful to anyone if I got a dump of the
llvm.org<http://llvm.org> LNT Postgres database?  It is a good dataset big
dataset to test with, and I assume everyone is okay with it being public, since
the LNT server already is.
>
>
>
>
>
> On Apr 25, 2016, at 4:33 AM, Elena Lepilkina via llvm-dev <llvm-dev at
lists.llvm.org<mailto:llvm-dev at lists.llvm.org>> wrote:
>
>
>
>
>
>
>
> From: Elena Lepilkina
> Sent: Monday, April 25, 2016 2:33 PM
> To: 'James Molloy' <james at jamesmolloy.co.uk<mailto:james
at jamesmolloy.co.uk>>; Kristof Beyls
> <Kristof.Beyls at arm.com<mailto:Kristof.Beyls at arm.com>>;
Mehdi Amini <mehdi.amini at apple.com<mailto:mehdi.amini at
apple.com>>
> Cc: nd <nd at arm.com<mailto:nd at arm.com>>; Matthias Braun
<matze at braunis.de<mailto:matze at braunis.de>>
> Subject: RE: [llvm-dev] RFC: LNT/Test-suite support for custom metrics
> and test parameterization
>
>
>
> Hi everyone,
>
>
>
> Thank you for your answer. BLOB format adds some more actions for working
with metrics. We know that ComparisonResult class makes analysis work. But it
gets all metrics by request from database, we will need additional time for work
with fields during analysis in ComparisonResult class. May be it will be better
to do one Sample table for each testsuite, as it was suggested before. It should
be more quickly, shouldn’t it? Moreover, next wished LNT changes will need
getting some metrics separately and BLOB format will add some delay in time for
queries.
>
>
>
> As we see now problem of performance is actual, because time for rendering
graph page is about 3 minutes.
>
> <image001.png>
>
> So maybe it will be better to start working with NoSql databases? I made a
small prototype with TestSuite, TestSuiteFields, Test, Run and Sample tables for
getting time metrics. It works quickly. And using NoSQL helps solve problems
with  different fields for samples metrics fields. Then it will be possible to
store different metrics for different testsuites in one table.
>
> What do you think about this proposal?
>
> I used MongoDB, but I know that there is NoSQL extension for
> PostgresSQL with JSONB fields which are more
>
> effective than JSON-encoded BLOB, because it can be included in queries
very simply and let use indexes.
>
>
>
> About proposal that not all metrics should be shown. It can be added as a
field in JSON in .fields file, which describes fields getted from test-suite. To
see other metrics user should choose them with checkboxes in view options. Will
be this solution suitable?
>
> We can make as you wrote
>
> “I'd also suggest that if we're adding many more metrics to a test,
we should create a "test sample information" page that the test link
goes to instead of just the graph. This page could contain all counter/metric
data, historic sparklines, the full graph and profiling links.
>
> ”
>
> But the render time of this page will be too great because of graph render
time. In my opinion, some users wouldn’t like to wait so long for see some
additional metrics.
>
>
>
> Thanks for your suggestions,
>
>
>
> Elena.
>
> From: llvm-dev [mailto:llvm-dev-bounces at
lists.llvm.org<mailto:llvm-dev-bounces at lists.llvm.org>] On Behalf Of
> James Molloy via llvm-dev
> Sent: Monday, April 25, 2016 12:43 PM
> To: Kristof Beyls <Kristof.Beyls at arm.com<mailto:Kristof.Beyls at
arm.com>>; Mehdi Amini
> <mehdi.amini at apple.com<mailto:mehdi.amini at apple.com>>
> Cc: llvm-dev <llvm-dev at lists.llvm.org<mailto:llvm-dev at
lists.llvm.org>>; nd <nd at arm.com<mailto:nd at arm.com>>;
Matthias
> Braun <matze at braunis.de<mailto:matze at braunis.de>>
> Subject: Re: [llvm-dev] RFC: LNT/Test-suite support for custom metrics
> and test parameterization
>
>
>
> Hi Sergey, Elena,
>
>
>
> Firstly, thanks for this RFC. It's great to see more people actively
using and modifying LNT and the test metrics support in general is rather weak
currently.
>
>
>
> Metrics
>
> -------
>
>
>
> I agree with Daniel and Kristof that your proposed schema changes have the
potential to make many queries extremely slow. Certainly for the metrics
enhancements, I don't see a reason why we need such a radical change in
schema.
>
>
>
> To add custom metrics on the fly, we need to change the schema for the
Sample table. Currently this consists of a column for each metric, but actually
we never ever query those metric values. We never query for example for
"all failing tests in a run" - when we do analyses we use the
ComparisonResult class which reads *all* samples from the database for a run and
performs the analysis entirely in Python.
>
>
>
> Therefore, having a semi-structured format where some fields are
first-class columns and the rest are in a JSON-encoded BLOB (as Daniel suggests)
seems totally acceptable. There is certainly an argument now that we're
using the wrong backend storage solution and that a key-value store might be
more suitable, but that's a very invasive change and I don't think
we've reached the point where we need to force a move from the simplicity of
SQLite.
>
>
>
> Adding an extra BLOB column would be easy - there would just need to be
logic in testsuitedb.py for reading and writing it - the Sample model class
would expose the JSON-encoded fields as normal python fields so the rest of LNT
would be isolated from this change.
>
>
>
> But I think this is a small detail compared to the bigger problem of how to
effectively *display* all this new data. Currently every new metric gets its own
separate table in the report/run views, and this does not scale well at all.
>
>
>
> I think we need some more concepts in the metric system to make it
scaleable:
>
>
>
>   * What "attribute" of the test is this metric measuring? For
example, both "exec_time" and "score" measure the same
attribute; performance of the generated code. It's superfluous to have them
displayed in separate tables. However mem_size and compile_time both measure
completely different aspects of the test.
>
>   * Is this metric useful to display at the top level? or should it only be
exposed when more data about a test result is requested?
>
>     * An example of this is the pass statistics. I don't want my daily
report view cluttered by the time spent in register allocation for every test!
OK, this is useful information when debugging a problem, but it should be
available when requested rather than by default.
>
>
>
> An example of why we need the above is your screenshots in your google doc.
I'm looking at the last screenshot, and it's incredibly difficult to
read and get useful information out of.
>
>
>
> I'd also suggest that if we're adding many more metrics to a test,
we should create a "test sample information" page that the test link
goes to instead of just the graph. This page could contain all counter/metric
data, historic sparklines, the full graph and profiling links.
>
>
>
> Cheers,
>
>
>
> James
>
>
>
> On Fri, 22 Apr 2016 at 10:17 Kristof Beyls via llvm-dev <llvm-dev at
lists.llvm.org<mailto:llvm-dev at lists.llvm.org>> wrote:
>
>
>
> On 22 Apr 2016, at 11:14, Mehdi Amini <mehdi.amini at
apple.com<mailto:mehdi.amini at apple.com>> wrote:
>
>
>
>
> On Apr 22, 2016, at 12:45 AM, Kristof Beyls via llvm-dev <llvm-dev at
lists.llvm.org<mailto:llvm-dev at lists.llvm.org>> wrote:
>
>
>
>
> On 21 Apr 2016, at 17:44, Sergey Yakoushkin <sergey.yakoushkin at
gmail.com<mailto:sergey.yakoushkin at gmail.com>> wrote:
>
>
>
> Hi Kristof,
>
>
>
>        The way we use LNT, we would run different configuration (e.g. -O3
vs -Os) as different "machines" in LNT's model.
>
>
>
> O2/O3 is indeed bad example. We're also using different machines for
Os/O3 - such parameters apply to all tests and we don't propose major
changes.
>
> Elena was only extending LNT interface a bit to ease LLVM-testsuite
execution with different compiler or HW flags.
>
>
>
> Oh I see, this boils down to extending the lnt runtest interface to be
> able to specify a set of configurations, rather than a single
> configuration and making
>
> sure configurations get submitted under different machine names? We
> kick off the different configuration runs through a script invoking
> lnt runtest multiple
>
> times. I don't see a big problem with extending the lnt runtest
> interface to do this, assuming it doesn't break the underlying
> concepts assumed throughout
>
> LNT. Maybe the only downside is that this will add even more command
> line options to lnt runtest, which already has a lot (too many?)
> command line
>
> options.
>
>
>
> Maybe some changes are required to analyze and compare metrics between
"machines": e.g. code size/performance between Os/O2/O3.
>
> Do you perform such comparisons?
>
>
>
> We typically do these kinds of comparisons when we test our patches
pre-commit, i.e. comparing for example '-O3' with '-O3 'mllvm
-enable-my-new-pass'.
>
> To stick with the LNT concepts, tests enabling new passes are stored as a
different "machine".
>
> The only way I know to be able to do a comparison between runs on 2
> different "machine"s is to manually edit the URL for run vs run
> comparison
>
> and fill in the runids of the 2 runs you want to compare.
>
> For example, the following URL is a comparison of
green-dragon-07-x86_64-O3-flto vs green-dragon-06-x86_64-O0-g on the public
llvm.org/perf<http://llvm.org/perf> server:
>
> http://llvm.org/perf/db_default/v4/nts/70644?compare_to=70634
>
> I had to manually look up and fill in the run ids 70644 and 70634.
>
> It would be great if there was a better way to be able to do these kind of
comparisons - i.e. not having to manually fill in run ids, but having a webui to
easily find and pick the runs you want to compare.
>
> (As an aside: I find it intriguing that the URL above suggests that there
are quite a few cases where "-O0 -g" produces faster code than
"-O3 -flto").
>
>
>
> Can you be more explicit which ones? I don't see any regression (other
than compared to the baseline, or on the compile time).
>
>
>
> --
>
> Mehdi
>
>
>
> D'Oh! I was misinterpreting the compile time differences as execution
time differences. Indeed, there is no unexpected result in there.
>
> Sorry for the noise!
>
>
>
> Kristof
>
>
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org>
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org>
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org>
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org>
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
_______________________________________________
LLVM Developers mailing list
llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org>
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160427/463cc5fb/attachment.html>

Hi all,

As we understood great changes will be done in LNT, so we are waiting to new LNT
version and stopped our work in LNT.

One more question about using test-suite separately with cmake. Cmake can only
build all tests and generate lit tests. After that we can run LIT and get report
which is not equal with report (simple) got with make. Cmake test-suite version
has no features to run custom metrics and generate other report type, right?

Are these features of make-version of test-suite planned to be added?

Thanks,

Elena.

From: daniel.dunbar at gmail.com [mailto:daniel.dunbar at gmail.com] On Behalf
Of Daniel Dunbar
Sent: Wednesday, April 27, 2016 10:15 AM
To: Elena Lepilkina <Elena.Lepilkina at synopsys.com>
Cc: Kristof Beyls <Kristof.Beyls at arm.com>; James Molloy <james at
jamesmolloy.co.uk>; llvm-dev <llvm-dev at lists.llvm.org>; nd <nd at
arm.com>
Subject: Re: [llvm-dev] RFC: LNT/Test-suite support for custom metrics and test
parameterization

Hi all,

First off, let me ask one question about the use case you intend to support:

Is your expectation for LLVM statistics style data that this would be present
for almost all runs in a database, or that it would only be present for a small
subset of runs?


Second, here is my general perspective on the LNT system architecture:

1. It is very important to me that LNT continue to support an
"turn-key" experience. In fact, I *wish* that experience would get
easier, not harder, for example by moving all interactions with the server to a
true admin interface. I would prefer not to introduce additional dependencies in
that case.

I will add that I am not completely opposed to moving to a "turn-key"
architecture which requires substantially more components (e.g., PostgreSQL,
memcached, etc.) as long as it was packaged in such a way that it could offer a
nice turn key experience. For example, if someone was willing to implement &
document a Vagrant or Docker based development model, that would be ok with me,
as long as there was still the option to do fully native deployments on OS X
with real system administrator support.

2. Our internal application architecture is severely lacking. I believe the
standard architecture these days would be to have (load-balancer + front-end +
[load-balancer] + back-end + database), and I think partly we are suffering from
missing most of that infrastructure. In particular, we are missing two important
components:
- There should be a separate backend, which would allow us to implement improved
caching, and a clear model for long-lived or persistent state. I filed this as:
https://llvm.org/bugs/show_bug.cgi?id=27534
- This would give us a place to manage background processing tasks, for example
jobs that reprocess the raw sample data for efficient queries.
If we had such a thing, we could consider using something like memcached for
improving caching (particularly in a larger deployment).

3. My intention was always to use JSON blobs for situations where a small % of
samples want to include arbitrary blobs of extra data. I think standardizing on
PostgreSQL/JSON is the right approach here, given the standard PaaS support for
PostgreSQL and its existing use within our deployments. SQLAlchemy has native
support for this.

4. I don't actually think that a NoSQL database buys us much if anything
(particularly when we have PostgreSQL/JSON available). We are not in a situation
where we would need something like eventual consistency around writes, which
leaves us wanting raw query performance over large sets of relatively
well-structured data. I believe that is a situation which is best served by SQL
with properly designed tables. This is also an area where having an
infrastructure that could handle background processing to load query-optimized
tables & indices would be valuable.

I think it is a mistake to believe that using NoSQL and a schema-less model
without also introducing substantial caching will somehow give better
performance than a real SQL database, and would be very interested to see
performance data showing otherwise (e.g., compare your MongoDB implementation to
one in which all data was in a full-schematized table with proper indices).

5. The database schema used by LNT (which is dynamically instantiated in
response to test suite configurations) is admittedly unorthodox. However, I
still believe this is the right answer to support a turn-key performance
tracking solution that can be customized by the user based on the fields they
wish to track (in fact, if Elena's data is present for almost all samples
then it is exactly the designed use case). I'm not yet convinced that the
right answer here isn't to improve the actual implementation of this
approach; for example, we could require the definition to be created when the
LNT instance is set up, which might solve some of the issues Chris mentioned.
Or, we could expose improved tools or a UI for interacting with the schema via
an admin interface, for example to promote individual from a JSON blob to being
first class for improved performance/reporting.

6. To echo one of James's points, I also do think that ultimately the most
complicated part of adding support for arbitrary metrics is not the database
implementation, but managing the complexity in the reporting and graphing logic
(which is already cumbersome). I am fine seeing a layered implementation
approach where we first focus on how just to get the data in, but I do think it
is worth spending some time looking at how to manage the visualizations of that
data.

 - Daniel


On Tue, Apr 26, 2016 at 11:17 PM, Elena Lepilkina via llvm-dev <llvm-dev at
lists.llvm.org<mailto:llvm-dev at lists.llvm.org>> wrote:
Of course it'll be running as now. But user will need have installed
MongoDB.

Installation on linux with support of .deb packages is quite easy.
sudo apt-key adv --keyserver
hkp://keyserver.ubuntu.com:80<http://keyserver.ubuntu.com:80> --recv
EA312927
echo "deb http://repo.mongodb.org/apt/debian wheezy/mongodb-org/3.2
main" | sudo tee /etc/apt/sources.list.d/mongodb-org-3.2.list
sudo apt-get update
sudo apt-get install -y mongodb-org
sudo service mongod start

After that mongo will be running service in localhost:27017. After that user
should create database
mongo
use <db name>

And set database name in config file. As additional fields will be host and port
for users who will do settings for their server, which will have default
values(localhost:27017).

After that old steps
~/mysandbox/bin/python ~/lnt/setup.py develop
lnt create ~/myperfdb
lnt runserver ~/myperfdb

MongoDB has detailed instructions for installing for all operating systems.

Extra 6-7 commands for install MongoDB, which takes about 2 minutes and should
be executed once shouldn't be a great problem for new users, who would like
to try LNT.

Thanks,

Elena.

-----Original Message-----
From: Kristof Beyls [mailto:Kristof.Beyls at arm.com<mailto:Kristof.Beyls at
arm.com>]
Sent: Tuesday, April 26, 2016 5:15 PM
To: James Molloy <james at jamesmolloy.co.uk<mailto:james at
jamesmolloy.co.uk>>
Cc: Elena Lepilkina <Elena.Lepilkina at
synopsys.com<mailto:Elena.Lepilkina at synopsys.com>>; Chris Matthews
<chris.matthews at apple.com<mailto:chris.matthews at apple.com>>;
llvm-dev <llvm-dev at lists.llvm.org<mailto:llvm-dev at
lists.llvm.org>>; nd <nd at arm.com<mailto:nd at arm.com>>
Subject: Re: [llvm-dev] RFC: LNT/Test-suite support for custom metrics and test
parameterization
I think it's important that it remains simple to get a simple local instance
up and running.
That will make it easier for people to give LNT a try, and also makes it easier
for LNT developers to have everything they need for developing locally.
I have no experience with NoSQL database engines. Would it be possible, assuming
you have the MongoDB/other packages installed for your system, to just run

$ ~/mysandbox/bin/python ~/lnt/setup.py develop $ lnt create ~/myperfdb $ lnt
runserver ~/myperfdb

and be up and running (which is roughly what is required at the moment)?
Of course good documentation could go a long way if a few extra steps would be
needed.

I do agree with James that if there are no major concerns for using a NoSQL
database, it would be easiest if we only supported one database engine.
For example, I had to do quite a bit of LNT regression test mangling to make
them work on both sqlite and postgres, and it remains harder than it should be
to write regression tests that test the database interface.

Thanks,

Kristof

> On 26 Apr 2016, at 11:37, James Molloy via llvm-dev <llvm-dev at
lists.llvm.org<mailto:llvm-dev at lists.llvm.org>> wrote:
>
> Hi Elena,
>
> Thanks for pushing forward with this. I like the idea of using a NoSQL
solution.
>
> My primary reservation is about adding the new NoSQL stuff as an extra
backend. I think the current database backend and its use of SQLAlchemy is
extremely complex and is in fact the most complex part of LNT. Adding something
more (as opposed to *replacing* it) would just make this worse and make it more
likely that contributors wouldn't be able to test LNT very well (having
three engines to test: SQLite, PostgreSQL and MongoDB).
>
> I think it'd be far better all around, if we decide to go with the
NoSQL solution, to just bite the bullet and force users who want to run a server
to install MongoDB.
>
> In my experience most of the teams I've seen using LNT have a single
LNT server instance and submit results to that, rather than launching small
instances to run "lnt viewcomparison".
>
> Cheers,
>
> James
>
> On Tue, 26 Apr 2016 at 09:15 Elena Lepilkina via llvm-dev <llvm-dev at
lists.llvm.org<mailto:llvm-dev at lists.llvm.org>> wrote:
> Hi everyone.
>
>
>
> Thanks to everyone who took participant in discussion of this proposal.
>
> After discussion we understood how other users use LNT and how great
datasets may be.
>
>
>
> So there is new updated proposal.
>
> (Google docs version with some images -
> https://docs.google.com/document/d/11qHNWRIQ2gc2aWH3gNBeeCUB3-JPe7AoMt
> n7n9JoCeY/edit?usp=sharing)
>
>
>
> Goal is the same.
>
> Enable LNT support of custom metrics such as: user-defined run-time and
static metrics (power, etc.) and LLVM pass statistic counters. Provide
integration with LLVM testsuite to automatically collect LLVM statistic counters
or custom metrics.
>
>
>
> Analysis of current Database
>
>
>
> Limitations
>
> 1.      This structure isn’t flexible.
>
> There is no opportunity to run another test-suite except simple one.
>
> 2.      Performance is quite bad when database has a lot of records.
>
> For example, rendering graph is too slow. On
green-dragon-07-x86_64-O3-flto:42 SingleSource/Benchmarks/Shootout/objinst  
compile_time need for rendering 191.8 seconds.
>
> 3.       It’s difficult to add new features which need queries to sample
table in database(if we use BLOB field for custom metrics).
>
> Queries will be needed for more complex analysis. For example, if we would
like to add some additional check for tests which compile time is too long, we
should get result of query where this metric is  greater than some constant.
>
> Or we would like to compare tests with different run options, so we should
get only some tests but not all.
>
> BLOB field will help to save current structure and make system a bit more
flexible. But in the nearest future it will be not enough.
>
> Getting all metrics of all tests will make work slow on great datasets. And
this way isn’t enough optimal.
>
> So we wouldn’t like to do BLOB field, which wouldn’t help to add new
features and have flexible system in future.
>
>
>
> Proposal
>
>
>
> We suggest to do third part of LNT (as Chris Matthews suggested). This part
will be used for getting custom metrics and running any test-suite.
>
> We suggest to use NoSQL database (for example, MongoDB or JSON/JSONB
extension of PostgresSQL, which let use PostgresSQL as NoSQL database) for this
part. This part will be enable if there is path to NoSQL database in config
file.
>
> It helps to have one Sample table(collection in NoSQL). If we use
schemaless feature in MongoDB, for example, then it’s possible to add new fields
when new testsuite is running.  Then there would be one table with a lot of
fields, some of which are empty. At any moment of time it will be possible to
change schema of table(document).
>
> A small prototype was made with MongoDB and ORM MongoEngine. This ORM was
choosen because MongoAlchemy doesn’t support schemaless features and last
MongoKit version has error with last pymongo release.
>
> I try it on virtual machine and get next results on 5 000 000 records.
>
> Current scheme - 13.72 seconds
>
> MongoDB – 1.35 seconds.
>
> Results of course will be better on real server machine .
>
>
>
> For use some test-suite user should describe fields in file with format
.fields such way:
>
> {
>
>  "Fields" : [{
>
>    "TestSuiteName" : "Bytecode",
>
>    "Type" : "integer",
>
>    "BiggerIsBetter" : 0,
>
>     "Show" : true
>
>  },
>
>  {
>
>    "TestSuiteName" : "GCC",
>
>    "Type" : "real",
>
>    "BiggerIsBetter" : 0,
>
>    "Name" : "GCC time"
>
>  },
>
>  {
>
>    "TestSuiteName" : "JIT",
>
>    "Type" : "real",
>
>    "BiggerIsBetter" : 0,
>
>    "Name" : "JIT Compile time",
>
>    "Show" : true
>
>  },
>
>  {
>
>    "TestSuiteName" : "GCC/LLC",
>
>    "Type" : "string",
>
>    "BiggerIsBetter" : 0
>
>  }]
>
> }
>
>
>
> There was added one field “Show” for describing if this metric should be
shown by default on web page (as James Molloy suggested). Other metrics would be
added to page if user chooses them in view options.
>
>
>
> Conclusion
>
>
>
> This change will let user to choose if he wants to use flexible powerful
system or use limited version with SQLite database.
>
> If user chooses NoSQL version his data can be copied from its old database
to new one. This will help to use new features without losing old data.
>
>
>
> The actual question is which NoSQL database will be better for LNT. We are
interested in opinions of people, who know features of LNT better.
>
>
>
> Thanks,
>
>
>
> Elena.
>
>
>
> From: llvm-dev [mailto:llvm-dev-bounces at
lists.llvm.org<mailto:llvm-dev-bounces at lists.llvm.org>] On Behalf Of
> Elena Lepilkina via llvm-dev
> Sent: Tuesday, April 26, 2016 9:07 AM
> To: chris.matthews at apple.com<mailto:chris.matthews at apple.com>
>
>
> Cc: llvm-dev <llvm-dev at lists.llvm.org<mailto:llvm-dev at
lists.llvm.org>>
> Subject: Re: [llvm-dev] RFC: LNT/Test-suite support for custom metrics
> and test parameterization
>
>
>
> Hi, Chris.
>
>
>
> Thank you for your answer about compile tests. As I understood during
looking through code of compile tests they don’t use test suite at all. Am I
right? There is lack of information and examples of running compile tests in LNT
documentation.
>
> We understood that there are two groups of users: users using servers and
collecting a lot of data and SQLite users, but these users as I think wouldn’t
have about millions of sample records.
>
> I think that it’s obvious that there is no universal solution for simple
installing process and flexible high-loaded system.
>
> I will update proposal and take into consideration your suggestion about
third part of test-suite.
>
>
>
> Thanks
>
>
>
> Elena.
>
>
>
> From: chris.matthews at apple.com<mailto:chris.matthews at apple.com>
[mailto:chris.matthews at apple.com<mailto:chris.matthews at apple.com>]
> Sent: Monday, April 25, 2016 8:06 PM
> To: Elena Lepilkina <Elena.Lepilkina at
synopsys.com<mailto:Elena.Lepilkina at synopsys.com>>
> Cc: llvm-dev <llvm-dev at lists.llvm.org<mailto:llvm-dev at
lists.llvm.org>>
> Subject: Re: [llvm-dev] RFC: LNT/Test-suite support for custom metrics
> and test parameterization
>
>
>
> I am really torn about this.
>
>
>
> When I implemented the regression tracking stuff recently, it really showed
me how badly we are scaling.  On our production server, the run ingestion can
take well over 100s.  Time is mostly spent in FieldChange generation and
regression grouping. Both have to access a lot of recent samples. This is not
the end of the world, because it runs in a background process.  Where this
really sucks is when a regression has a lot indicators. The web interface
renders these in a graph, and just trying to pull down 100 graphs worth of data
kills the server.  I ended up limiting those to a max of 10 datasets, and even
that takes 30s to load.
>
>
>
> So I do think we need some improvements to the scalability.
>
>
>
> LNT usage is spread between two groups. Users who setup big servers, with
Postgres and apache/Gunicorn. For those uses I think a NoSQL is the way to go.  
However, our second (and probably more common) user, is the people running
little instance on their own machine to do some local compiler benchmarking. 
Their setup process needs to be dead simple, and I think requiring a NoSQL
database to be setup on their machine first is a no starter.  Like we do with
sqlite, I think we need a transparent fall back for people who don’t have a
NoSQL database.
>
>
>
> Would it be helpful to anyone if I got a dump of the
llvm.org<http://llvm.org> LNT Postgres database?  It is a good dataset big
dataset to test with, and I assume everyone is okay with it being public, since
the LNT server already is.
>
>
>
>
>
> On Apr 25, 2016, at 4:33 AM, Elena Lepilkina via llvm-dev <llvm-dev at
lists.llvm.org<mailto:llvm-dev at lists.llvm.org>> wrote:
>
>
>
>
>
>
>
> From: Elena Lepilkina
> Sent: Monday, April 25, 2016 2:33 PM
> To: 'James Molloy' <james at jamesmolloy.co.uk<mailto:james
at jamesmolloy.co.uk>>; Kristof Beyls
> <Kristof.Beyls at arm.com<mailto:Kristof.Beyls at arm.com>>;
Mehdi Amini <mehdi.amini at apple.com<mailto:mehdi.amini at
apple.com>>
> Cc: nd <nd at arm.com<mailto:nd at arm.com>>; Matthias Braun
<matze at braunis.de<mailto:matze at braunis.de>>
> Subject: RE: [llvm-dev] RFC: LNT/Test-suite support for custom metrics
> and test parameterization
>
>
>
> Hi everyone,
>
>
>
> Thank you for your answer. BLOB format adds some more actions for working
with metrics. We know that ComparisonResult class makes analysis work. But it
gets all metrics by request from database, we will need additional time for work
with fields during analysis in ComparisonResult class. May be it will be better
to do one Sample table for each testsuite, as it was suggested before. It should
be more quickly, shouldn’t it? Moreover, next wished LNT changes will need
getting some metrics separately and BLOB format will add some delay in time for
queries.
>
>
>
> As we see now problem of performance is actual, because time for rendering
graph page is about 3 minutes.
>
> <image001.png>
>
> So maybe it will be better to start working with NoSql databases? I made a
small prototype with TestSuite, TestSuiteFields, Test, Run and Sample tables for
getting time metrics. It works quickly. And using NoSQL helps solve problems
with  different fields for samples metrics fields. Then it will be possible to
store different metrics for different testsuites in one table.
>
> What do you think about this proposal?
>
> I used MongoDB, but I know that there is NoSQL extension for
> PostgresSQL with JSONB fields which are more
>
> effective than JSON-encoded BLOB, because it can be included in queries
very simply and let use indexes.
>
>
>
> About proposal that not all metrics should be shown. It can be added as a
field in JSON in .fields file, which describes fields getted from test-suite. To
see other metrics user should choose them with checkboxes in view options. Will
be this solution suitable?
>
> We can make as you wrote
>
> “I'd also suggest that if we're adding many more metrics to a test,
we should create a "test sample information" page that the test link
goes to instead of just the graph. This page could contain all counter/metric
data, historic sparklines, the full graph and profiling links.
>
> ”
>
> But the render time of this page will be too great because of graph render
time. In my opinion, some users wouldn’t like to wait so long for see some
additional metrics.
>
>
>
> Thanks for your suggestions,
>
>
>
> Elena.
>
> From: llvm-dev [mailto:llvm-dev-bounces at
lists.llvm.org<mailto:llvm-dev-bounces at lists.llvm.org>] On Behalf Of
> James Molloy via llvm-dev
> Sent: Monday, April 25, 2016 12:43 PM
> To: Kristof Beyls <Kristof.Beyls at arm.com<mailto:Kristof.Beyls at
arm.com>>; Mehdi Amini
> <mehdi.amini at apple.com<mailto:mehdi.amini at apple.com>>
> Cc: llvm-dev <llvm-dev at lists.llvm.org<mailto:llvm-dev at
lists.llvm.org>>; nd <nd at arm.com<mailto:nd at arm.com>>;
Matthias
> Braun <matze at braunis.de<mailto:matze at braunis.de>>
> Subject: Re: [llvm-dev] RFC: LNT/Test-suite support for custom metrics
> and test parameterization
>
>
>
> Hi Sergey, Elena,
>
>
>
> Firstly, thanks for this RFC. It's great to see more people actively
using and modifying LNT and the test metrics support in general is rather weak
currently.
>
>
>
> Metrics
>
> -------
>
>
>
> I agree with Daniel and Kristof that your proposed schema changes have the
potential to make many queries extremely slow. Certainly for the metrics
enhancements, I don't see a reason why we need such a radical change in
schema.
>
>
>
> To add custom metrics on the fly, we need to change the schema for the
Sample table. Currently this consists of a column for each metric, but actually
we never ever query those metric values. We never query for example for
"all failing tests in a run" - when we do analyses we use the
ComparisonResult class which reads *all* samples from the database for a run and
performs the analysis entirely in Python.
>
>
>
> Therefore, having a semi-structured format where some fields are
first-class columns and the rest are in a JSON-encoded BLOB (as Daniel suggests)
seems totally acceptable. There is certainly an argument now that we're
using the wrong backend storage solution and that a key-value store might be
more suitable, but that's a very invasive change and I don't think
we've reached the point where we need to force a move from the simplicity of
SQLite.
>
>
>
> Adding an extra BLOB column would be easy - there would just need to be
logic in testsuitedb.py for reading and writing it - the Sample model class
would expose the JSON-encoded fields as normal python fields so the rest of LNT
would be isolated from this change.
>
>
>
> But I think this is a small detail compared to the bigger problem of how to
effectively *display* all this new data. Currently every new metric gets its own
separate table in the report/run views, and this does not scale well at all.
>
>
>
> I think we need some more concepts in the metric system to make it
scaleable:
>
>
>
>   * What "attribute" of the test is this metric measuring? For
example, both "exec_time" and "score" measure the same
attribute; performance of the generated code. It's superfluous to have them
displayed in separate tables. However mem_size and compile_time both measure
completely different aspects of the test.
>
>   * Is this metric useful to display at the top level? or should it only be
exposed when more data about a test result is requested?
>
>     * An example of this is the pass statistics. I don't want my daily
report view cluttered by the time spent in register allocation for every test!
OK, this is useful information when debugging a problem, but it should be
available when requested rather than by default.
>
>
>
> An example of why we need the above is your screenshots in your google doc.
I'm looking at the last screenshot, and it's incredibly difficult to
read and get useful information out of.
>
>
>
> I'd also suggest that if we're adding many more metrics to a test,
we should create a "test sample information" page that the test link
goes to instead of just the graph. This page could contain all counter/metric
data, historic sparklines, the full graph and profiling links.
>
>
>
> Cheers,
>
>
>
> James
>
>
>
> On Fri, 22 Apr 2016 at 10:17 Kristof Beyls via llvm-dev <llvm-dev at
lists.llvm.org<mailto:llvm-dev at lists.llvm.org>> wrote:
>
>
>
> On 22 Apr 2016, at 11:14, Mehdi Amini <mehdi.amini at
apple.com<mailto:mehdi.amini at apple.com>> wrote:
>
>
>
>
> On Apr 22, 2016, at 12:45 AM, Kristof Beyls via llvm-dev <llvm-dev at
lists.llvm.org<mailto:llvm-dev at lists.llvm.org>> wrote:
>
>
>
>
> On 21 Apr 2016, at 17:44, Sergey Yakoushkin <sergey.yakoushkin at
gmail.com<mailto:sergey.yakoushkin at gmail.com>> wrote:
>
>
>
> Hi Kristof,
>
>
>
>        The way we use LNT, we would run different configuration (e.g. -O3
vs -Os) as different "machines" in LNT's model.
>
>
>
> O2/O3 is indeed bad example. We're also using different machines for
Os/O3 - such parameters apply to all tests and we don't propose major
changes.
>
> Elena was only extending LNT interface a bit to ease LLVM-testsuite
execution with different compiler or HW flags.
>
>
>
> Oh I see, this boils down to extending the lnt runtest interface to be
> able to specify a set of configurations, rather than a single
> configuration and making
>
> sure configurations get submitted under different machine names? We
> kick off the different configuration runs through a script invoking
> lnt runtest multiple
>
> times. I don't see a big problem with extending the lnt runtest
> interface to do this, assuming it doesn't break the underlying
> concepts assumed throughout
>
> LNT. Maybe the only downside is that this will add even more command
> line options to lnt runtest, which already has a lot (too many?)
> command line
>
> options.
>
>
>
> Maybe some changes are required to analyze and compare metrics between
"machines": e.g. code size/performance between Os/O2/O3.
>
> Do you perform such comparisons?
>
>
>
> We typically do these kinds of comparisons when we test our patches
pre-commit, i.e. comparing for example '-O3' with '-O3 'mllvm
-enable-my-new-pass'.
>
> To stick with the LNT concepts, tests enabling new passes are stored as a
different "machine".
>
> The only way I know to be able to do a comparison between runs on 2
> different "machine"s is to manually edit the URL for run vs run
> comparison
>
> and fill in the runids of the 2 runs you want to compare.
>
> For example, the following URL is a comparison of
green-dragon-07-x86_64-O3-flto vs green-dragon-06-x86_64-O0-g on the public
llvm.org/perf<http://llvm.org/perf> server:
>
> http://llvm.org/perf/db_default/v4/nts/70644?compare_to=70634
>
> I had to manually look up and fill in the run ids 70644 and 70634.
>
> It would be great if there was a better way to be able to do these kind of
comparisons - i.e. not having to manually fill in run ids, but having a webui to
easily find and pick the runs you want to compare.
>
> (As an aside: I find it intriguing that the URL above suggests that there
are quite a few cases where "-O0 -g" produces faster code than
"-O3 -flto").
>
>
>
> Can you be more explicit which ones? I don't see any regression (other
than compared to the baseline, or on the compile time).
>
>
>
> --
>
> Mehdi
>
>
>
> D'Oh! I was misinterpreting the compile time differences as execution
time differences. Indeed, there is no unexpected result in there.
>
> Sorry for the noise!
>
>
>
> Kristof
>
>
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org>
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org>
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org>
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org>
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
_______________________________________________
LLVM Developers mailing list
llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org>
http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160513/74fc8acb/attachment-0001.html>