On 2 August 2014 00:40, Renato Golin <renato.golin at linaro.org> wrote:> If memory serves me well (it doesn't), these are the list of things we > agreed on making, and their progress: > > 1. Performance-specific test-suite: a group of specific benchmarks > that should be tracked with the LNT infrastructure. Hal proposed to > look at this, but other people helped implement it. Last I heard there > was some way of running it but I'm not sure how to do it. I'd love to > have this as a buildbot, though, so we can track its progress.We have this in LNT now which can be activated using `--benchmarking-only`. It's about 50% faster than a full run and massively reduces the number of false positives. Chris has also posted a patch to rerun tests which the server said changed. Haven't tried it yet but it looks really promising.> 2. Statistical analysis of the LNT data. A lot of work has been put > into this and I believe it's a lot better. Anton, Yi and others have > been discussing and submitting many patches to make the LNT reporting > infrastructure more stable, less prone to noise and more useful all > round. It's not perfect yet, but a lot better than last year's.There's definitely lots of room for improvement. I'm going to propose some more once we've solved the LNT stability issues.> 3. LNT website got really unstable (Internal Server Error every other > day). This is the reason I stopped submitting results to it, since it > would make my bot fail. And because I still don't have a performance > test-suite bot, I don't really care much for the results. But with the > noise reduction, it'd be really interesting to monitor the progress, > even of the full test-suite, but right now, I can't afford to have > random failures. This seriously needs looking into and would be good > to have that as a topic in this BoF.We are now testing PostgreSQL as database backend on the public perf server, replacing the SQLite db. Hopefully this can improve the stability and system performance. Also being discussing is to move the LNT server to a PaaS service, as it has higher availability and saves a lot of maintenance work. However this will need community to provide or fund the hosting service. -Yi -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20140802/e9716261/attachment.html>
On 2 August 2014 01:24, Yi Kong <kongy.dev at gmail.com> wrote:> We have this in LNT now which can be activated using `--benchmarking-only`. > It's about 50% faster than a full run and massively reduces the number of > false positives.Excellent! Does that mean I can create another LNT buildbot with an extra flag and I'll get my benchmarking bot?> We are now testing PostgreSQL as database backend on the public perf server, > replacing the SQLite db. Hopefully this can improve the stability and system > performance.I believe a real database would make things more stable but I don't think that the problem is in the DB, only that a better DB will hide the real problem even deeper. But I'm also not in the position to offer help debug the thing, so I'll go with what you guys think is the best.> Also being discussing is to move the LNT server to a PaaS service, as it has > higher availability and saves a lot of maintenance work. However this will > need community to provide or fund the hosting service.I've been thinking about this for a while and I think the LLVM Foundation could make this happen across the globe. If I got it right, part of the foundation's purpose is to make sure we have a stable, modern and effective infrastructure for building, testing and benchmarking LLVM, as well as other critical development tools like bugzilla, phabricator and svn/git repositories. My proposal is to use the foundation's charitable status to get help from around the world for redundant services. I don't know about PostgreSQL, but MySQL has a master/slave setting that works fantastically well across continents, so that there is only one master (with switch-over facility in case of failure) and multiple remote slaves. I believe that services line LNT, bugzilla, phabricator could highly benefit from this. Other services like git could also have back-up copies (maybe making an official GitHub repo as a redundant copy of out oficial repo), and buildbots would be easy to have at least one slave of the same bot config on each continent, so that if one fails, another is still up and running. The hardest one to get that distributed is SVN. If we could get universities in Europe and Asia, in addition to the ones in US that host and maintain LLVM tools, we could have a really reliable infrastructure without raising costs that much. I'd be happy to pursue this in England (maybe even Brazil) and I believe there are developers that would be glad to do this across Europe and Asia. cheers, --renato
On 2 August 2014 14:35, Renato Golin <renato.golin at linaro.org> wrote:> On 2 August 2014 01:24, Yi Kong <kongy.dev at gmail.com> wrote: >> We have this in LNT now which can be activated using `--benchmarking-only`. >> It's about 50% faster than a full run and massively reduces the number of >> false positives. > > Excellent! Does that mean I can create another LNT buildbot with an > extra flag and I'll get my benchmarking bot?Yes. But you should also let performance tracking buildbots take multiple samples. -Yi