llvm dev - Oct 2016 - Getting a combined llvm-cov report for a set of tests

I am working on a large software project with many thousands of tests. A
common workflow using gcov as a coverage tool is:

1) Build code with coverage instrumentation
2) Run *all* the tests. Gcov aggregates the line coverage counters together
into its .gcda files
3) Generate a report for some set of files using those combined gcda files.
4) Analyze the report to find files/sections of code that are not covered
by any test.

Using llvm's source-based code coverage, I don't believe I can
accomplish
steps 3-4. Each binary produces a .profraw file which contains profile
counts, but the mapping of those profile counts to files is contained
inside the binary.  I can use llvm-profdata merge to merge multiple
.profraw files together, but the report can still only be generated for a
single binary, and so only the files contained in that binary's
__llvm_covmap section will be produced as part of the report.

Am I correct in this thinking? Is there a way to get a single coverage
report for a set of multiple binaries? I discovered the undocumented
'llvm-cov convert-for-testing' command which generates 'covmapping
files',
which appear to be the contents of __llvm_covmap with an additional header.
Maybe it's possible to merge multiple of these covmapping files and
generate a report from that data (plus the merged profraw files)? Or
perhaps I could use 'llvm-cov export' on each binary individually and
manually merge the results somehow (although in that case, I would not be
able to 'import' the merge to create the usual report) ? Are there other
suggestions, or is this a planned feature in 4.0? I found some discussion
on this thread, but I am not sure if this is asking the same thing:
https://groups.google.com/forum/#!topic/llvm-dev/r4qOdN-y9hc

Thanks!
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20161014/8f878a9f/attachment.html>

Hi,

There isn't yet a way to generate a single coverage report for multiple
binaries. We're planning on adding this functionality to llvm-cov soon. The
patch is up for review here: https://reviews.llvm.org/D25086.
> Maybe it's possible to merge multiple of these covmapping files and
generate a report from that data (plus the merged profraw files)?
It also isn't yet possible to merge multiple 'covmapping' files. Do
you have
other use-cases for this functionality (apart from using it to generate a
single report given multiple binaries)?
> I found some discussion on this thread, but I am not sure if this is asking
the same thing:
> https://groups.google.com/forum/#!topic/llvm-dev/r4qOdN-y9hc
Yes, that thread is about the same topic. The person who started it is the
author of the patch I linked to :).

best,
vedant

> On Oct 14, 2016, at 9:05 AM, Chris Brown via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> 
> I am working on a large software project with many thousands of tests. A
common workflow using gcov as a coverage tool is:
> 
> 1) Build code with coverage instrumentation
> 2) Run *all* the tests. Gcov aggregates the line coverage counters together
into its .gcda files
> 3) Generate a report for some set of files using those combined gcda files.
> 4) Analyze the report to find files/sections of code that are not covered
by any test.
>  
> Using llvm's source-based code coverage, I don't believe I can
accomplish steps 3-4. Each binary produces a .profraw file which contains
profile counts, but the mapping of those profile counts to files is contained
inside the binary.  I can use llvm-profdata merge to merge multiple .profraw
files together, but the report can still only be generated for a single binary,
and so only the files contained in that binary's __llvm_covmap section will
be produced as part of the report.
> 
> Am I correct in this thinking? Is there a way to get a single coverage
report for a set of multiple binaries? I discovered the undocumented
'llvm-cov convert-for-testing' command which generates 'covmapping
files', which appear to be the contents of __llvm_covmap with an additional
header. Maybe it's possible to merge multiple of these covmapping files and
generate a report from that data (plus the merged profraw files)? Or perhaps I
could use 'llvm-cov export' on each binary individually and manually
merge the results somehow (although in that case, I would not be able to
'import' the merge to create the usual report) ? Are there other
suggestions, or is this a planned feature in 4.0? I found some discussion on
this thread, but I am not sure if this is asking the same thing:
> https://groups.google.com/forum/#!topic/llvm-dev/r4qOdN-y9hc
> 
> Thanks!
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev