llvm dev - Mar 2016 - llvm-cov accepting many binary files for aggregated coverage reports

Hi All,

I want to provide a solution that presents code coverage reports that
include the aggregated code-counts  across many binaries.  Our test
engineers currently do this using gcov-mode by merged .gcda data files. We
can do a similar merge of the .profraw files, so that the many binaries are
represented in one .profdata file; However, llvm-cov will only generate
reports based on one binary at a time.

My suggested solution to this problem is for llvm-cov to accept multiple
binary files on the command line. Thereby allowing every coverage-count /
source file (that is compiled into at least one binary) to be represented
in the report from a single call to llvm-cov.  Presently the command format
is: llvm-cov show [options] <executable | object file> <Source
files>.  I
suggest adding the option "-bin=<executable | object file>" so
that more
than one binary (or object file) can be listed as: -bin=binary1.elf
-bin=binary2.elf -bin=binary3.elf.

I would like people’s opinions first so that may be acceptable when I come
to upstream this.

Thanks


-- 
Ying Yi
SN Systems Ltd - Sony Computer Entertainment Group.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160304/913c98af/attachment.html>

If many binaries share some library code, it can be confusing to show the
aggregated coverage data of the library as if they are valid for any
individual binary.

It seems to me that a per-library coverage report is more appropriate here.
llvm can take DSO, but not yet static archive file (.a) -- so  I can see it
being a useful feature if you can teach llvm-cov to handle it.  On the
other hand,  given a set of standalone object files, I don't see why a
wrapper script (that passes object file to llvm-cov one by one) won't work
well -- so I am not convinced that the new option proposed is in general
useful.

thanks,

David



On Fri, Mar 4, 2016 at 12:57 PM, Ying Yi via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> Hi All,
>
> I want to provide a solution that presents code coverage reports that
> include the aggregated code-counts  across many binaries.  Our test
> engineers currently do this using gcov-mode by merged .gcda data files. We
> can do a similar merge of the .profraw files, so that the many binaries are
> represented in one .profdata file; However, llvm-cov will only generate
> reports based on one binary at a time.
>
> My suggested solution to this problem is for llvm-cov to accept multiple
> binary files on the command line. Thereby allowing every coverage-count /
> source file (that is compiled into at least one binary) to be represented
> in the report from a single call to llvm-cov.  Presently the command format
> is: llvm-cov show [options] <executable | object file> <Source
files>.  I
> suggest adding the option "-bin=<executable | object file>"
so that more
> than one binary (or object file) can be listed as: -bin=binary1.elf
> -bin=binary2.elf -bin=binary3.elf.
>
> I would like people’s opinions first so that may be acceptable when I come
> to upstream this.
>
> Thanks
>
>
> --
> Ying Yi
> SN Systems Ltd - Sony Computer Entertainment Group.
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160304/4ac7bf41/attachment.html>

Hi David,

In my use case, our test teams want a single html index file that links to
an aggregated coverage report that covers an entire code repository.  I see
that there is a new patch of generating html reports, is there a way to
index all of the individual source files into one summary report, where
there are many binary files to consider?


*> I don't see why a wrapper script (that passes object file
to llvm-cov one by one) won't work well*

If the functionality to generate the toplevel html index page for one
binary is embedded in llvm-cov,  how will a wrapper script generate
the toplevel
html index page for a code-repository that is tested across multiple
binaries.

*> If *many binaries share some library code, it can be confusing to show
the aggregated coverage data of the library as if they are valid for any
individual binary.

The functionality to do this merge of profile data across many binaries
already exists in llvm-profdata.  Our test engineers do not find this
confusing, on the contrary they find it very helpful, where they are
looking for an aggregated coverage report.

*> It seems to me that a per-library coverage report is more appropriate
here.*

I'm not sure how this differs from generating a single report across an
entire code-repository; passing the code-repository as a single library is
just one use-case.  There are also use-cases where a single repository is
compiled into many libraries;  do you plan to support giving llvm-cov
multiple library files (or multiple object files)?

Ultimately, in my case, I would like to see a single toplevel coverage
report (preferably in html) and all of the detailed reports for each source
file that includes the coverage from many binaries.

Thanks,
Phillip
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160307/7271e954/attachment.html>