llvm dev - Mar 2016 - llvm-cov accepting many binary files for aggregated coverage reports

Hi David,

In my use case, our test teams want a single html index file that links to
an aggregated coverage report that covers an entire code repository.  I see
that there is a new patch of generating html reports, is there a way to
index all of the individual source files into one summary report, where
there are many binary files to consider?


*> I don't see why a wrapper script (that passes object file
to llvm-cov one by one) won't work well*

If the functionality to generate the toplevel html index page for one
binary is embedded in llvm-cov,  how will a wrapper script generate
the toplevel
html index page for a code-repository that is tested across multiple
binaries.

*> If *many binaries share some library code, it can be confusing to show
the aggregated coverage data of the library as if they are valid for any
individual binary.

The functionality to do this merge of profile data across many binaries
already exists in llvm-profdata.  Our test engineers do not find this
confusing, on the contrary they find it very helpful, where they are
looking for an aggregated coverage report.

*> It seems to me that a per-library coverage report is more appropriate
here.*

I'm not sure how this differs from generating a single report across an
entire code-repository; passing the code-repository as a single library is
just one use-case.  There are also use-cases where a single repository is
compiled into many libraries;  do you plan to support giving llvm-cov
multiple library files (or multiple object files)?

Ultimately, in my case, I would like to see a single toplevel coverage
report (preferably in html) and all of the detailed reports for each source
file that includes the coverage from many binaries.

Thanks,
Phillip
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160307/7271e954/attachment.html>

Ok -- this sounds like a reasonable use case so personally I don't have
strong objection to adding this feature (assuming it does not add too much
additional complexity to llvm-cov implementation). Perhaps we can discuss
it more in the actual code review?

thanks,

David

On Mon, Mar 7, 2016 at 2:46 AM, p23 power <p23power at gmail.com> wrote:
> Hi David,
>
> In my use case, our test teams want a single html index file that links to
> an aggregated coverage report that covers an entire code repository.  I see
> that there is a new patch of generating html reports, is there a way to
> index all of the individual source files into one summary report, where
> there are many binary files to consider?
>
>
> *> I don't see why a wrapper script (that passes object file
> to llvm-cov one by one) won't work well*
>
> If the functionality to generate the toplevel html index page for one
> binary is embedded in llvm-cov,  how will a wrapper script generate the
toplevel
> html index page for a code-repository that is tested across multiple
> binaries.
>
> *> If *many binaries share some library code, it can be confusing to
show
> the aggregated coverage data of the library as if they are valid for any
> individual binary.
>
> The functionality to do this merge of profile data across many binaries
> already exists in llvm-profdata.  Our test engineers do not find this
> confusing, on the contrary they find it very helpful, where they are
> looking for an aggregated coverage report.
>
> *> It seems to me that a per-library coverage report is more appropriate
> here.*
>
> I'm not sure how this differs from generating a single report across an
> entire code-repository; passing the code-repository as a single library is
> just one use-case.  There are also use-cases where a single repository is
> compiled into many libraries;  do you plan to support giving llvm-cov
> multiple library files (or multiple object files)?
>
> Ultimately, in my case, I would like to see a single toplevel coverage
> report (preferably in html) and all of the detailed reports for each source
> file that includes the coverage from many binaries.
>
> Thanks,
> Phillip
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160307/91f05693/attachment-0001.html>

Thanks David, I will look at the additional complexity for adding this
feature.


Maggie

On Mon, Mar 7, 2016 at 6:38 PM, Xinliang David Li <xinliangli at
gmail.com>
wrote:
> Ok -- this sounds like a reasonable use case so personally I don't have
> strong objection to adding this feature (assuming it does not add too much
> additional complexity to llvm-cov implementation). Perhaps we can discuss
> it more in the actual code review?
>
> thanks,
>
> David
>
> On Mon, Mar 7, 2016 at 2:46 AM, p23 power <p23power at gmail.com>
wrote:
>
>> Hi David,
>>
>> In my use case, our test teams want a single html index file that links
>> to an aggregated coverage report that covers an entire code repository.
I
>> see that there is a new patch of generating html reports, is there a
way to
>> index all of the individual source files into one summary report, where
>> there are many binary files to consider?
>>
>>
>> *> I don't see why a wrapper script (that passes object file
>> to llvm-cov one by one) won't work well*
>>
>> If the functionality to generate the toplevel html index page for one
>> binary is embedded in llvm-cov,  how will a wrapper script generate the
toplevel
>> html index page for a code-repository that is tested across multiple
>> binaries.
>>
>> *> If *many binaries share some library code, it can be confusing to
>> show the aggregated coverage data of the library as if they are valid
>> for any individual binary.
>>
>> The functionality to do this merge of profile data across many binaries
>> already exists in llvm-profdata.  Our test engineers do not find this
>> confusing, on the contrary they find it very helpful, where they are
>> looking for an aggregated coverage report.
>>
>> *> It seems to me that a per-library coverage report is more
appropriate
>> here.*
>>
>> I'm not sure how this differs from generating a single report
across an
>> entire code-repository; passing the code-repository as a single library
is
>> just one use-case.  There are also use-cases where a single repository
is
>> compiled into many libraries;  do you plan to support giving llvm-cov
>> multiple library files (or multiple object files)?
>>
>> Ultimately, in my case, I would like to see a single toplevel coverage
>> report (preferably in html) and all of the detailed reports for each
source
>> file that includes the coverage from many binaries.
>>
>> Thanks,
>> Phillip
>>
>>
>>
>

-- 
Ying Yi
SN Systems Ltd - Sony Computer Entertainment Group.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160310/aaf5c71b/attachment-0001.html>