Lustre discuss - Jul 2012 - coverage measurement for some lustre test suites

Hi,
we did some work for collecting code coverage which generate some Lustre  
tests.

Results are available on opensfs site
http://www.opensfs.org/foswiki/bin/view/Lustre/CodeCoverage ,
please look and comment it.

Thanks,
	Roman

On 2012-07-18, at 6:49 AM, Roman Grigoryev wrote:
> we did some work for collecting code coverage which generate some Lustre  
> tests.
> 
> Results are available on opensfs site
> http://www.opensfs.org/foswiki/bin/view/Lustre/CodeCoverage ,
> please look and comment it.
This looks quite interesting.  I guess the next step is to figure out which
functions are receiving no coverage during testing, and write tests to exercise
them.

I suspect one type of function which gets very little testing is the lprocfs
functions.  Writing a simple test to read all of the functions would get
coverage, but it would be better to have actual tests that verify the content of
the files is actually correct.

Cheers, Andreas
--
Andreas Dilger                       Whamcloud, Inc.
Principal Lustre Engineer            http://www.whamcloud.com/

Andreas Dilger <adilger at whamcloud.com> wrote at  Sat, 21 Jul 2012
21:35:50
+0400:
> On 2012-07-18, at 6:49 AM, Roman Grigoryev wrote:
>> we did some work for collecting code coverage which generate some
Lustre
>> tests.
>>
>> Results are available on opensfs site
>> http://www.opensfs.org/foswiki/bin/view/Lustre/CodeCoverage ,
>> please look and comment it.
>
> This looks quite interesting.  I guess the next step is to figure out  
> which functions are receiving no coverage during testing, and write  
> tests to exercise them.
This is the most important one of possible ways for using this info.

It is possible to use this info for finding probably death code. It will  
be  good to check it with static analyzer like Coverity or Polyspace too.

Also, I think, it could be helpful to know which test cover which code  
(for example for developers, for quick testing new code after fix before  
uploading for review/wide testing and maybe put changes to these tests). I  
have coverage by test and have script for searching test by code line.
In future, we could create script for searching tests with maximum  
codecoverage(and probably we could update SLOW subset) and searching tests  
with  minimum coverage or duplicate coverage  and possible refactor them

Maybe this can be used for proving orientation of benchmark/stress tests  
when we will have coverage for them.
>
> I suspect one type of function which gets very little testing is the  
> lprocfs functions.  Writing a simple test to read all of the functions  
> would get coverage, but it would be better to have actual tests that  
> verify the content of the files is actually correct.
Absolutely agree that  just coverage is not enough for prove quality. In  
coverage theory percent of covered set of all combination of execution  
paths should be used for proving quality but this ''spherical
cow'' for the
most of application.

I think,  in real life we should use rule like this ''dummy coverage is
better than zero coverage but coverage which provide real use case is  
better dummy coverage''.

Ideally, I prefer to have real use cases in tests, dummy coverage in unit  
tests and also use advanced static code analyzers for all code.

-- 
Thanks,
	Roman
>
> Cheers, Andreas
> --
> Andreas Dilger                       Whamcloud, Inc.
> Principal Lustre Engineer            http://www.whamcloud.com/
>
>
>

Roman,
> Maybe this can be used for proving orientation of benchmark/stress tests
when we will have coverage for them.
I think the other area of interest would be failover and failback. 

hsn

________________________________________
Henry Newman | CEO/CTO
Instrumental, Inc | High Performance Innovation
1450 Energy Park Drive Suite 375
St. Paul, MN 55108-5274
Direct:????? 651-280-4801
Fax:???????? 651-280-4839
Toll-free:?? 1.800.886.6188
E-Mail:????? hsn at instrumental.com
Web:???????? http://www.instrumental.com


STRICTLY PERSONAL AND CONFIDENTIAL. This email may contain confidential and
proprietary material for the sole use of the intended recipient. Any review or
distribution by others is strictly prohibited. If you are not the intended
recipient please?contact the sender and delete all copies.

-----Original Message-----
From: discuss-bounces at lists.opensfs.org [mailto:discuss-bounces at
lists.opensfs.org] On Behalf Of Roman Grigoryev
Sent: Sunday, July 22, 2012 3:24 AM
To: Roman Grigoryev; Andreas Dilger
Cc: discuss at lists.opensfs.org; lustre-discuss at lists.lustre.org
Subject: Re: [Discuss] coverage measurement for some lustre test suites

Andreas Dilger <adilger at whamcloud.com> wrote at  Sat, 21 Jul 2012
21:35:50
+0400:
> On 2012-07-18, at 6:49 AM, Roman Grigoryev wrote:
>> we did some work for collecting code coverage which generate some 
>> Lustre tests.
>>
>> Results are available on opensfs site 
>> http://www.opensfs.org/foswiki/bin/view/Lustre/CodeCoverage , please 
>> look and comment it.
>
> This looks quite interesting.  I guess the next step is to figure out 
> which functions are receiving no coverage during testing, and write 
> tests to exercise them.
This is the most important one of possible ways for using this info.

It is possible to use this info for finding probably death code. It will be 
good to check it with static analyzer like Coverity or Polyspace too.

Also, I think, it could be helpful to know which test cover which code (for
example for developers, for quick testing new code after fix before uploading
for review/wide testing and maybe put changes to these tests). I have coverage
by test and have script for searching test by code line.
In future, we could create script for searching tests with maximum
codecoverage(and probably we could update SLOW subset) and searching tests with 
minimum coverage or duplicate coverage  and possible refactor them

Maybe this can be used for proving orientation of benchmark/stress tests when we
will have coverage for them.
>
> I suspect one type of function which gets very little testing is the  
> lprocfs functions.  Writing a simple test to read all of the functions  
> would get coverage, but it would be better to have actual tests that  
> verify the content of the files is actually correct.
Absolutely agree that  just coverage is not enough for prove quality. In  
coverage theory percent of covered set of all combination of execution  
paths should be used for proving quality but this ''spherical
cow'' for the
most of application.

I think,  in real life we should use rule like this ''dummy coverage is
better than zero coverage but coverage which provide real use case is  
better dummy coverage''.

Ideally, I prefer to have real use cases in tests, dummy coverage in unit  
tests and also use advanced static code analyzers for all code.

-- 
Thanks,
	Roman
>
> Cheers, Andreas
> --
> Andreas Dilger                       Whamcloud, Inc.
> Principal Lustre Engineer            http://www.whamcloud.com/
>
>
>
_______________________________________________
discuss mailing list
discuss at lists.opensfs.org
http://lists.opensfs.org/listinfo.cgi/discuss-opensfs.org
STRICTLY PERSONAL AND CONFIDENTIAL. This email may contain confidential and
proprietary material for the sole use of the intended recipient. Any review or
distribution by others is strictly prohibited. If you are not the intended
recipient please contact the sender and delete all copies.

On 2012-07-22, at 1:24, "Roman Grigoryev" <Roman_Grigoryev at
xyratex.com> wrote:
> Andreas Dilger <adilger at whamcloud.com> wrote at  Sat, 21 Jul 2012
21:35:50
> +0400:
> 
>> On 2012-07-18, at 6:49 AM, Roman Grigoryev wrote:
>>> we did some work for collecting code coverage which generate some
Lustre
>>> tests.
>>> 
>>> Results are available on opensfs site
>>> http://www.opensfs.org/foswiki/bin/view/Lustre/CodeCoverage ,
>>> please look and comment it.
>> 
>> This looks quite interesting.  I guess the next step is to figure out  
>> which functions are receiving no coverage during testing, and write  
>> tests to exercise them.
> 
> This is the most important one of possible ways for using this info.
> 
> It is possible to use this info for finding probably death code. It will  
> be  good to check it with static analyzer like Coverity or Polyspace too.
The problem I''ve seen with Coverity is that the output is not intended
to be exported outside their web interface. There are a couple of free code
analysis tools that could be integrated with the build system more easily, such
as "sparse" and "clang".

The problem with starting to use any static analysis tools is that someone has
to put in the time and effort to clean up the existing warnings/errors and any
false positives. At that point, they could be enabled for continuous use going
forward.
> Also, I think, it could be helpful to know which test cover which code  
> (for example for developers, for quick testing new code after fix before  
> uploading for review/wide testing and maybe put changes to these tests). I
> have coverage by test and have script for searching test by code line.
> In future, we could create script for searching tests with maximum  
> code coverage (and probably we could update SLOW subset)
Agreed - this would be useful. It would also be good if there was an automated
way to know if newly added tests are covering new code as intended.
> and searching tests with minimum coverage or duplicate coverage and
possible refactor them.
In some cases, it isn''t clear that straight code coverage is enough,
because it also doesn''t check the different states the system might be
in at the time.
> Maybe this can be used for proving orientation of benchmark/stress tests  
> when we will have coverage for them.
> 
>> 
>> I suspect one type of function which gets very little testing is the  
>> lprocfs functions.  Writing a simple test to read all of the functions
>> would get coverage, but it would be better to have actual tests that  
>> verify the content of the files is actually correct.
> 
> Absolutely agree that  just coverage is not enough for prove quality. In  
> coverage theory percent of covered set of all combination of execution  
> paths should be used for proving quality but this ''spherical
cow'' for the
> most of application.
> 
> I think,  in real life we should use rule like this ''dummy
coverage is
> better than zero coverage but coverage which provide real use case is  
> better dummy coverage''.
> 
> Ideally, I prefer to have real use cases in tests, dummy coverage in unit  
> tests and also use advanced static code analyzers for all code.
> 
> -- 
> Thanks,
>    Roman
> 
>> 
>> Cheers, Andreas
>> --
>> Andreas Dilger                       Whamcloud, Inc.
>> Principal Lustre Engineer            http://www.whamcloud.com/
>> 
>> 
>>

>> On 2012-07-18, at 6:49 AM, Roman Grigoryev wrote:
>>> we did some work for collecting code coverage which generate some
Lustre
>>> tests.
>>>
>>> Results are available on opensfs site
>>> http://www.opensfs.org/foswiki/bin/view/Lustre/CodeCoverage ,
>>> please look and comment it.
>>
>> This looks quite interesting.  I guess the next step is to figure out
>> which functions are receiving no coverage during testing, and write
>> tests to exercise them.
>
> This is the most important one of possible ways for using this info.
>
> It is possible to use this info for finding probably death code. It will
> be  good to check it with static analyzer like Coverity or Polyspace too.
>
> Also, I think, it could be helpful to know which test cover which code
> (for example for developers, for quick testing new code after fix before
> uploading for review/wide testing and maybe put changes to these tests). I
> have coverage by test and have script for searching test by code line.
> In future, we could create script for searching tests with maximum
> codecoverage(and probably we could update SLOW subset) and searching tests
> with  minimum coverage or duplicate coverage  and possible refactor them
>
> Maybe this can be used for proving orientation of benchmark/stress tests
> when we will have coverage for them.
>
>>
>> I suspect one type of function which gets very little testing is the
>> lprocfs functions.  Writing a simple test to read all of the functions
>> would get coverage, but it would be better to have actual tests that
>> verify the content of the files is actually correct.
>
> Absolutely agree that  just coverage is not enough for prove quality. In
> coverage theory percent of covered set of all combination of execution
> paths should be used for proving quality but this ''spherical
cow'' for the
> most of application.
>
> I think,  in real life we should use rule like this ''dummy
coverage is
> better than zero coverage but coverage which provide real use case is
> better dummy coverage''.
>
> Ideally, I prefer to have real use cases in tests, dummy coverage in unit
> tests and also use advanced static code analyzers for all code.
>
> --
> Thanks,
>        Roman
I think this is very valuable data and something we should update over time so 
that we can see which way it is trending. Hopefully the effort of a repeat run
is much less than the original which I sure was quite significant.

On the subject of coverage many people are sceptical of code coverage results,
and whilst it is true to that that 100% coverage doesn''t mean
everything is
tested, 0% does mean nothing is tested. I make this point because what we can
say is that overtime the % coverages should all be increasing because
it''s
difficult to image a case where less coverage is better testing.

Returning to the specifics of this set, you presumably have the
report currently on opensfs for any given test, i.e. we could have page that
listed
all ofthe lustre suites/tests that can then be clicked through to the underlying
report. Is that the case and if would you be able to publish it.

In fact as you say flipping the tree on it''s head it should be possible
to click
on any line and find out which tests exercise that line, which would be cool. Of
course we have to be careful here that data paths are well as code paths need to
be tested for any change, and data paths are real tricky to map.

Thanks

Chris
---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

Gearing, Chris <chris.gearing at intel.com> wrote at  Mon, 23 Jul 2012  
16:19:27 +0400:
>>> On 2012-07-18, at 6:49 AM, Roman Grigoryev wrote:
>>>> we did some work for collecting code coverage which generate
some
>>>> Lustre
>>>> tests.
>>>>
>>>> Results are available on opensfs site
>>>> http://www.opensfs.org/foswiki/bin/view/Lustre/CodeCoverage ,
>>>> please look and comment it.
>>>
>>> This looks quite interesting.  I guess the next step is to figure
out
>>> which functions are receiving no coverage during testing, and write
>>> tests to exercise them.
>>
>> This is the most important one of possible ways for using this info.
>>
>> It is possible to use this info for finding probably death code. It
will
>> be  good to check it with static analyzer like Coverity or Polyspace  
>> too.
>>
>> Also, I think, it could be helpful to know which test cover which code
>> (for example for developers, for quick testing new code after fix
before
>> uploading for review/wide testing and maybe put changes to these  
>> tests). I
>> have coverage by test and have script for searching test by code line.
>> In future, we could create script for searching tests with maximum
>> codecoverage(and probably we could update SLOW subset) and searching  
>> tests
>> with  minimum coverage or duplicate coverage  and possible refactor
them
>>
>> Maybe this can be used for proving orientation of benchmark/stress
tests
>> when we will have coverage for them.
>>
>>>
>>> I suspect one type of function which gets very little testing is
the
>>> lprocfs functions.  Writing a simple test to read all of the
functions
>>> would get coverage, but it would be better to have actual tests
that
>>> verify the content of the files is actually correct.
>>
>> Absolutely agree that  just coverage is not enough for prove quality.
In
>> coverage theory percent of covered set of all combination of execution
>> paths should be used for proving quality but this ''spherical
cow'' for
>> the
>> most of application.
>>
>> I think,  in real life we should use rule like this ''dummy
coverage is
>> better than zero coverage but coverage which provide real use case is
>> better dummy coverage''.
>>
>> Ideally, I prefer to have real use cases in tests, dummy coverage in  
>> unit
>> tests and also use advanced static code analyzers for all code.
>>
>> --
>> Thanks,
>>        Roman
>
> I think this is very valuable data and something we should update over  
> time so
> that we can see which way it is trending. Hopefully the effort of a  
> repeat run
> is much less than the original which I sure was quite significant.
>
> On the subject of coverage many people are sceptical of code coverage  
> results,
> and whilst it is true to that that 100% coverage doesn''t mean
everything
> is
> tested, 0% does mean nothing is tested. I make this point because what  
> we can
> say is that overtime the % coverages should all be increasing because  
> it''s
> difficult to image a case where less coverage is better testing.
I absolutely agree that code coverage is metric which need pretty accurate  
interpretation.
>
> Returning to the specifics of this set, you presumably have the
> report currently on opensfs for any given test, i.e. we could have page  
> that listed
> all ofthe lustre suites/tests that can then be clicked through to the  
> underlying
> report. Is that the case and if would you be able to publish it.
I have only summary html report and set of lcov trace files(one per test)  
and own script for searching with it. Generating lcov report for every  
test is pretty big task and produce a lot of output. Locally I can  
generate report for every test. Maybe it is good idea to just provide  
tools and lcov trace files?

>
> In fact as you say flipping the tree on it''s head it should be
possible
> to click
> on any line and find out which tests exercise that line, which would be  
> cool. Of
> course we have to be careful here that data paths are well as code paths  
> need to
> be tested for any change, and data paths are real tricky to map.
For solving this problem lcov report keeps internally sources which were  
used for collecting coverage. Also I wrote few own scripts for merging and  
transforming lcov for simplifying report generation on new place.

-- 
Thanks,
	Roman