Le 06/05/2015 21:05, Renato Golin a écrit :> On 6 May 2015 at 19:15, Sanjoy Das <sanjoy at playingwithpointers.com> wrote: >> I could not easily locate this on http://llvm.org/reports/coverage/ so >> asking here: what workload is the coverage computed over? IOW, what >> all does the bot run to get this coverage information? > Nothing is clear in that page. No revision number, no arch, no test > names, nothing. Just a date.I am running it. I don't have access to test names but I can try to add the revision number. The arch is GNU/Linux 64 bit. Joshua (as cc) worked on bringing better coverage to Thunderbird and Firefox: https://github.com/jcranmer/mozilla-coverage Maybe I could hack with his tool to get the test information to LLVM/Clang too. For now, I haven't put too much effort into it as I wasn't sure how many people are using the code coverage (btw, is it possible to have stats on our website?) FYI, It is generated by this jenkins job: http://llvm-jenkins.debian.net/job/llvm-toolchain-codecoverage-binaries/ I have plans to try https://coveralls.io/ to handle that. It should not be too hard as I already have most of the things enabled in this job.> The CMakeBuilder is used by almost all ARM/AArch64 and Windows > builders, we could potentially make it run for all archs, and adding > another step might be simple, depending on how that LCOV page works...For now, it uses autotools and it is managed by the same workflow as the packages on llvm.org/apt/ Now, moving to cmake is just a matter of time. Anyway, don't hesitate if you have other comments or suggestions. Glad to know it interests folks, despite all it limitations. Cheers, Sylvestre
On 5/6/2015 2:47 PM, Sylvestre Ledru wrote:> Le 06/05/2015 21:05, Renato Golin a écrit : >> On 6 May 2015 at 19:15, Sanjoy Das <sanjoy at playingwithpointers.com> wrote: >>> I could not easily locate this on http://llvm.org/reports/coverage/ so >>> asking here: what workload is the coverage computed over? IOW, what >>> all does the bot run to get this coverage information? >> Nothing is clear in that page. No revision number, no arch, no test >> names, nothing. Just a date. > I am running it. > I don't have access to test names but I can try to add the revision number. > The arch is GNU/Linux 64 bit. > > Joshua (as cc) worked on bringing better coverage to Thunderbird and > Firefox: > https://github.com/jcranmer/mozilla-coverage > Maybe I could hack with his tool to get the test information to > LLVM/Clang too. > For now, I haven't put too much effort into it as I wasn't sure how many > people > are using the code coverage (btw, is it possible to have stats on our > website?)An example of the tool I wrote is publicly accessible at <https://www.tjhsst.edu/~jcranmer/m-ccov/>, although it is a bit on the older side. I essentially ported lcov to Python, firstly since the lcov tool seemed to suddenly run achingly slowly, and speed is important when you're merging 100 test coverage files comprising ~1.8M counters. I also added a coverage treemap overview (<https://www.tjhsst.edu/~jcranmer/m-ccov/coverage.html?dir=js> is a link that won't kill your browser in animation), and you can view detailed code coverage of individual tests on individual files, a feature LCOV didn't have when I poked it. The only reason the tooling doesn't support clang is because clang's code coverage support was fatally broken when I started the project, necessitating the need to use gcc for tests. The automated infrastructure for building Firefox is presently using a sufficiently old version of clang on OS X that the inability to get an instrumented version of Firefox on the automation infrastructure is probably due to inherently broken code coverage, and I've lacked any way to definitively debug the issue, so it ended up being a lower priority. -- Joshua Cranmer Thunderbird module owner DXR coauthor
On 6 May 2015 at 23:50, Joshua Cranmer <pidgeot18 at gmail.com> wrote:> An example of the tool I wrote is publicly accessible at > <https://www.tjhsst.edu/~jcranmer/m-ccov/>Hi Joshua, This sure looks beautiful, but I think we need something a lot simpler than that. First, I really want to know if the tests we run regularly (check-all and test-suite) do cover most of the code. So we don't need a test-by-test report, just three states: check-all, test-suite, both. Second, the area graph can hide small files that were never tested. I'd like a report like LCOV (or your detailed report), but with the option to sort by coverage, so you can focus on the problematic cases. Third, your tool also doesn't seem to have a revision number, which makes it difficult to compare commits or even know what features are in or out. Finally, a separate report, one that only includes the changed lines for a specific commit, is the most important thing. This will tell you if the tests that you committed do touch all the logic you have added, and even if the existing tests are already exercising the patch, so a new test would be redundant. cheers, --renato