llvm dev - May 2020 - [EXTERNAL] How to get branch coverage by using 'source-based code coverage'

Hi, Alan

Really very excited to receive your email and sorry to be slow replying, it
has been exceptionally busy over the last few days ;(

Your explanation made the problem clear to me. So gcov branch coverage
should be called condition coverage and clang region coverage
is branch coverage in fact(also known as *decision/C1*), right?

And llvm/clang will support all the following coverage criteria in future

   - Line coverage
   - Function coverage
   - Branch coverage/region coverage
   - Condition coverage
   - Modified Condition/Decision Coverage

If there is any misunderstanding here, please point it out.

Thanks,
Ted Xie



Phipps, Alan <a-phipps at ti.com> 于2020年4月28日周二 下午10:14写道：
> Hi Ted,
>
>
>
> Presently, branch coverage is not supported in clang source-based code
> coverage, but I am working on implementing branch condition coverage right
> now.  It will track True/False branches for each leaf-level condition,
> including within Boolean expressions comprised of logical operators
(“&&”,
> “||”).  Technically, this condition-based coverage is more granular than
> how “branch coverage” is often defined (tracking on that control flow
> decisions allow for all regions to be hit), and is closer to the
> granularity that GCOV provides, but it’s anchored to the source code, won’t
> count “hidden branches”, and isn’t negatively impacted by optimization
> (although the reverse isn’t true: coverage instrumentation may impact
> optimization).
>
>
>
> My work is a step closer to getting us toward Modified Condition/Decision
> Coverage (MC/DC).  I hope to upstream my work within the next few months,
> so unfortunately it won’t help you in the short-term.  Region coverage may
> be enough for what you need if you can ascertain based on the coverage that
> control-flow decisions have evaluated to both true and false, but I’ll let
> others comment here with suggestions.
>
>
>
> -Alan Phipps
>
>
>
> *From:* llvm-dev [mailto:llvm-dev-bounces at lists.llvm.org] *On Behalf Of
*LoveZhi
> Xie via llvm-dev
> *Sent:* Sunday, April 26, 2020 11:36 AM
> *To:* llvm-dev at lists.llvm.org
> *Subject:* [EXTERNAL] [llvm-dev] How to get branch coverage by using
> 'source-based code coverage'
>
>
>
> Hi, llvm/clang experts
>
>
>
> I need to get the branch coverage for some testing code. But i found gcov
> can't give a expected coverage which may
>
> count some 'hidden branch' in (See stackoverflow answer
>
<https://stackoverflow.com/questions/42003783/lcov-gcov-branch-coverage-with-c-producing-branches-all-over-the-place>).
> Instead, I turn to use clang and the 'source-based code coverage'
feature
>
> may be a right choice. But i can't find anything to describe branch
> coverage explicitly on the official site
> <https://clang.llvm.org/docs/SourceBasedCodeCoverage.html>.
>
>
>
> So how to use clang to get branch coverage by utilizing
'source-based...'
> feature?  And is *region coverage *equivalent to branch coverage?
>
> (I tested on a simple program and region coverage seems pretty close to
> branch coverage)
>
>
>
> Any help is highly appreciated
>
>
>
> Thanks,
>
> Ted Xie
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20200504/664917ab/attachment.html>

+ Vedant Kumar

Hi Ted!

I would not say that clang region coverage is branch coverage(‘aka decision
coverage’).  I would say that it may be possible to extrapolate branch decision
coverage based on region coverage.  For example, consider the following:

[region0]
if (decision) {
   [region1]
}

Region coverage will tell you that whether [region1] was executed, and this will
be enough to tell you that the branch decision evaluated to True at least once. 
However, this will not tell you how many times the decision evaluated to False,
which is also important when considering branch coverage.  In this case, you can
calculate this value by subtracting the number of times [region0] was executed
from the number of times [region1] was executed.

What I am implementing for branch condition coverage will do this for you but
will provide more granular True/False counts for individual conditions that
comprise all Boolean expression (which also generate branches), and will tally
all of these cases as branches that are either covered or not covered.

I hope that helps.

As for the future goals, I do plan on addressing what’s required for MC/DC, but
that is work that I have not yet planned.

Thanks!

-Alan Phipps

From: LoveZhi Xie [mailto:crazyforzhi1987 at gmail.com]
Sent: Sunday, May 3, 2020 11:12 AM
To: Phipps, Alan
Cc: llvm-dev at lists.llvm.org
Subject: Re: [EXTERNAL] [llvm-dev] How to get branch coverage by using
'source-based code coverage'

Hi, Alan

Really very excited to receive your email and sorry to be slow replying, it has
been exceptionally busy over the last few days ;(

Your explanation made the problem clear to me. So gcov branch coverage should be
called condition coverage and clang region coverage
is branch coverage in fact(also known as decision/C1), right?

And llvm/clang will support all the following coverage criteria in future

  *   Line coverage
  *   Function coverage
  *   Branch coverage/region coverage
  *   Condition coverage
  *   Modified Condition/Decision Coverage
If there is any misunderstanding here, please point it out.

Thanks,
Ted Xie



Phipps, Alan <a-phipps at ti.com<mailto:a-phipps at ti.com>>
于2020年4月28日周二 下午10:14写道：
Hi Ted,

Presently, branch coverage is not supported in clang source-based code coverage,
but I am working on implementing branch condition coverage right now.  It will
track True/False branches for each leaf-level condition, including within
Boolean expressions comprised of logical operators (“&&”, “||”). 
Technically, this condition-based coverage is more granular than how “branch
coverage” is often defined (tracking on that control flow decisions allow for
all regions to be hit), and is closer to the granularity that GCOV provides, but
it’s anchored to the source code, won’t count “hidden branches”, and isn’t
negatively impacted by optimization (although the reverse isn’t true: coverage
instrumentation may impact optimization).

My work is a step closer to getting us toward Modified Condition/Decision
Coverage (MC/DC).  I hope to upstream my work within the next few months, so
unfortunately it won’t help you in the short-term.  Region coverage may be
enough for what you need if you can ascertain based on the coverage that
control-flow decisions have evaluated to both true and false, but I’ll let
others comment here with suggestions.

-Alan Phipps

From: llvm-dev [mailto:llvm-dev-bounces at
lists.llvm.org<mailto:llvm-dev-bounces at lists.llvm.org>] On Behalf Of
LoveZhi Xie via llvm-dev
Sent: Sunday, April 26, 2020 11:36 AM
To: llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org>
Subject: [EXTERNAL] [llvm-dev] How to get branch coverage by using
'source-based code coverage'

Hi, llvm/clang experts

I need to get the branch coverage for some testing code. But i found gcov
can't give a expected coverage which may
count some 'hidden branch' in (See stackoverflow
answer<https://stackoverflow.com/questions/42003783/lcov-gcov-branch-coverage-with-c-producing-branches-all-over-the-place>).
Instead, I turn to use clang and the 'source-based code coverage'
feature
may be a right choice. But i can't find anything to describe branch coverage
explicitly on the official
site<https://clang.llvm.org/docs/SourceBasedCodeCoverage.html>.

So how to use clang to get branch coverage by utilizing
'source-based...' feature?  And is region coverage equivalent to branch
coverage?
(I tested on a simple program and region coverage seems pretty close to branch
coverage)

Any help is highly appreciated

Thanks,
Ted Xie


-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20200503/063e5bdc/attachment-0001.html>

Hi, Alan

Thanks for making it clear. But I was more confused now :(

I tested on a simple program and used both gcov and lcov to get branch
coverage.
The code and build commands as below:

*Example simple.cc*

#include <string>
// If not comment this line, the branch coverage won't reach to 100%
// #include <iostream>
int main(int argc, const char* argv[]) {
  std::string str = "foo";
  str = argv[1];
  if (str == "foo" || str == "bar") {
    int a = 5;
    a = 7;
  } else {
    int b = 6;
    b = 4;
  }
  return 0;
}


*Coverage build commands*

#!/bin/bash
g++ -o simple -fno-exceptions -fno-inline -fprofile-arcs -ftest-coverage
simple.cc
./simple "foo"
./simple "bar"
./simple "hello1"
./simple "hello2"
./simple "hello3"
./simple "hello4"
./simple "hello5"
./simple "hello6"
./simple "hello7"
./simple "hello8"
lcov --rc lcov_branch_coverage=1 --capture --directory . --output-file
simple.lcov.info
genhtml simple.lcov.info --function-coverage --branch-coverage
--output-directory lcov_out/
gcov -b -c simple.gcno


Both gcov and lcov will give a 100% branch coverage report:
[image: cov1.png] [image: cov2.png]


My questions:

   -  What's the "*official*" evaluation formula on branch
coverage?

            Sorry I can't find any compelling documents to describe the
accurate algorithm on this issue.
            I just thought gcov/lcov won't consider *how many times the
decision evaluated to False.*
*          P*lease see the gcov file on my example: simple.cc. There are 6
branches in total which are all hit
            So the result would be 6/6 * 100% = 100%

            [image: cov3.png]


   - Are there any existing post-processing tools based on clang/llvm
   source based coverage?

           I just saw a clion plugin
<https://github.com/zero9178/C-Cpp-Coverage-for-CLion> at github which may
generate branch coverage by using region coverage data. But this depends on
clion
           which is not easy to implement automation test.

   - How to disable STL (or other) noise in gcov branch coverage?

           Sorry, this is not clang/llvm question. But I am still looking
forward to some suggestions. For example, if I add iostream header back to
           simple.cc, then we won't get a 100% branch coverage. This may
because iostream will introduce some global destruction branch.

           [image: cov4.png]

           Or if it's impossible, can we think clang region coverage should
be a best practice *in C++ testing world*??
AFAIK, chromium is using region coverage now.
[image: cov5.png]

FYI.

Thanks,
Ted Xie

Phipps, Alan <a-phipps at ti.com> 于2020年5月4日周一 上午12:54写道：
> + Vedant Kumar
>
>
>
> Hi Ted!
>
>
>
> I would not say that clang region coverage is branch coverage(‘aka
> decision coverage’).  I would say that it may be possible to extrapolate
> branch decision coverage based on region coverage.  For example, consider
> the following:
>
>
>
> [region0]
>
> if (decision) {
>
>    [region1]
>
> }
>
>
>
> Region coverage will tell you that whether [region1] was executed, and
> this will be enough to tell you that the branch decision evaluated to True
> at least once.  However, this will not tell you how many times the decision
> evaluated to False, which is also important when considering branch
> coverage.  In this case, you can calculate this value by subtracting the
> number of times [region0] was executed from the number of times [region1]
> was executed.
>
>
>
> What I am implementing for branch condition coverage will do this for you
> but will provide more granular True/False counts for individual conditions
> that comprise all Boolean expression (which also generate branches), and
> will tally all of these cases as branches that are either covered or not
> covered.
>
>
>
> I hope that helps.
>
>
>
> As for the future goals, I do plan on addressing what’s required for
> MC/DC, but that is work that I have not yet planned.
>
>
>
> Thanks!
>
>
>
> -Alan Phipps
>
>
>
> *From:* LoveZhi Xie [mailto:crazyforzhi1987 at gmail.com]
> *Sent:* Sunday, May 3, 2020 11:12 AM
> *To:* Phipps, Alan
> *Cc:* llvm-dev at lists.llvm.org
> *Subject:* Re: [EXTERNAL] [llvm-dev] How to get branch coverage by using
> 'source-based code coverage'
>
>
>
> Hi, Alan
>
>
>
> Really very excited to receive your email and sorry to be slow replying,
> it has been exceptionally busy over the last few days ;(
>
>
>
> Your explanation made the problem clear to me. So gcov branch coverage
> should be called condition coverage and clang region coverage
>
> is branch coverage in fact(also known as *decision/C1*), right?
>
>
>
> And llvm/clang will support all the following coverage criteria in future
>
>    - Line coverage
>    - Function coverage
>    - Branch coverage/region coverage
>    - Condition coverage
>    - Modified Condition/Decision Coverage
>
> If there is any misunderstanding here, please point it out.
>
>
>
> Thanks,
>
> Ted Xie
>
>
>
>
>
>
>
> Phipps, Alan <a-phipps at ti.com> 于2020年4月28日周二 下午10:14写道：
>
> Hi Ted,
>
>
>
> Presently, branch coverage is not supported in clang source-based code
> coverage, but I am working on implementing branch condition coverage right
> now.  It will track True/False branches for each leaf-level condition,
> including within Boolean expressions comprised of logical operators
(“&&”,
> “||”).  Technically, this condition-based coverage is more granular than
> how “branch coverage” is often defined (tracking on that control flow
> decisions allow for all regions to be hit), and is closer to the
> granularity that GCOV provides, but it’s anchored to the source code, won’t
> count “hidden branches”, and isn’t negatively impacted by optimization
> (although the reverse isn’t true: coverage instrumentation may impact
> optimization).
>
>
>
> My work is a step closer to getting us toward Modified Condition/Decision
> Coverage (MC/DC).  I hope to upstream my work within the next few months,
> so unfortunately it won’t help you in the short-term.  Region coverage may
> be enough for what you need if you can ascertain based on the coverage that
> control-flow decisions have evaluated to both true and false, but I’ll let
> others comment here with suggestions.
>
>
>
> -Alan Phipps
>
>
>
> *From:* llvm-dev [mailto:llvm-dev-bounces at lists.llvm.org] *On Behalf Of
*LoveZhi
> Xie via llvm-dev
> *Sent:* Sunday, April 26, 2020 11:36 AM
> *To:* llvm-dev at lists.llvm.org
> *Subject:* [EXTERNAL] [llvm-dev] How to get branch coverage by using
> 'source-based code coverage'
>
>
>
> Hi, llvm/clang experts
>
>
>
> I need to get the branch coverage for some testing code. But i found gcov
> can't give a expected coverage which may
>
> count some 'hidden branch' in (See stackoverflow answer
>
<https://stackoverflow.com/questions/42003783/lcov-gcov-branch-coverage-with-c-producing-branches-all-over-the-place>).
> Instead, I turn to use clang and the 'source-based code coverage'
feature
>
> may be a right choice. But i can't find anything to describe branch
> coverage explicitly on the official site
> <https://clang.llvm.org/docs/SourceBasedCodeCoverage.html>.
>
>
>
> So how to use clang to get branch coverage by utilizing
'source-based...'
> feature?  And is *region coverage *equivalent to branch coverage?
>
> (I tested on a simple program and region coverage seems pretty close to
> branch coverage)
>
>
>
> Any help is highly appreciated
>
>
>
> Thanks,
>
> Ted Xie
>
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20200504/c241fdc0/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: cov1.png
Type: image/png
Size: 12113 bytes
Desc: not available
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20200504/c241fdc0/attachment-0005.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: cov2.png
Type: image/png
Size: 25337 bytes
Desc: not available
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20200504/c241fdc0/attachment-0006.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: cov3.png
Type: image/png
Size: 18297 bytes
Desc: not available
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20200504/c241fdc0/attachment-0007.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: cov4.png
Type: image/png
Size: 24148 bytes
Desc: not available
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20200504/c241fdc0/attachment-0008.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: cov5.png
Type: image/png
Size: 92921 bytes
Desc: not available
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20200504/c241fdc0/attachment-0009.png>