llvm dev - Nov 2013 - [LLVMdev] Any objections to my importing GoogleMock to go with GoogleTest in LLVM?

On Thu, Nov 14, 2013 at 1:04 PM, Rafael Espíndola <
rafael.espindola at gmail.com> wrote:
> > I think the cost of carrying it around is essentially zero. I'm
happy to
> do
> > any of the maintenance. People who don't know how to use it or
want to
> learn
> > how to use it don't need to use it. If it isn't making their
job of
> writing
> > tests sufficiently easier to justify, then they don't use it. I
see this
> as
> > a good pattern.
>
> That is not the case. If the test finds any bug at all, people have to
> look at the testcase and see if it is failing.
>
> Even if not bug is found, someone doing refactoring has to change the
> test to use the new apis.

On IRC, Rafael indicated he would like to understand the specific use case
I had in mind better. I'm still working on a concrete example, but figured
I'd clarify more than my initial mail did.

I'm working on the pass manager. I need a way to test that specific passes
in the pass manager are being run and invalidated appropriately based on
the dependencies between them and the caching infrastructure. Verifying
this can be done in two ways:

1) I can add non-trivial code which essentially dumps the state at each
point, along with a command line tool and collection of fake passes, and
use FileCheck against this code to verify things. This works for asserts
builds but not for no-asserts builds, and generally feels like working
around a missing feature in our testing infrastructure. This is the same
reasons we don't use FileCheck and state serialization to test our DenseMap
implementation.

2) I can write a unittest using gtest with a completely custom collection
of N passes written for every single test, each with a different set of
integer output parameters that are incremented and decremented at the right
points, and then a verification of their final value. This will work, but
will be hard to debug (the failure is detected long after it happens) and
very verbose.

3) I can use gmock to write a specific set of expected events for a pass
and verify that they happen. It was specifically designed to make verifying
this kind of interaction explicit with little boilerplate and decent error
messages when it fails.

I'll try to come up with an example of #3 this evening or tomorrow.

-Chandler
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20131114/93db21df/attachment.html>

On Nov 14, 2013, at 2:44 PM, Chandler Carruth <chandlerc at google.com>
wrote:
> On Thu, Nov 14, 2013 at 1:04 PM, Rafael Espíndola <rafael.espindola at
gmail.com> wrote:
> > I think the cost of carrying it around is essentially zero. I'm
happy to do
> > any of the maintenance. People who don't know how to use it or
want to learn
> > how to use it don't need to use it. If it isn't making their
job of writing
> > tests sufficiently easier to justify, then they don't use it. I
see this as
> > a good pattern.
> 
> That is not the case. If the test finds any bug at all, people have to
> look at the testcase and see if it is failing.
> 
> Even if not bug is found, someone doing refactoring has to change the
> test to use the new apis.
> 
> On IRC, Rafael indicated he would like to understand the specific use case
I had in mind better. I'm still working on a concrete example, but figured
I'd clarify more than my initial mail did.
> 
> I'm working on the pass manager. I need a way to test that specific
passes in the pass manager are being run and invalidated appropriately based on
the dependencies between them and the caching infrastructure. Verifying this can
be done in two ways:
> 
> 1) I can add non-trivial code which essentially dumps the state at each
point, along with a command line tool and collection of fake passes, and use
FileCheck against this code to verify things. This works for asserts builds but
not for no-asserts builds, and generally feels like working around a missing
feature in our testing infrastructure. This is the same reasons we don't use
FileCheck and state serialization to test our DenseMap implementation.
> 
> 2) I can write a unittest using gtest with a completely custom collection
of N passes written for every single test, each with a different set of integer
output parameters that are incremented and decremented at the right points, and
then a verification of their final value. This will work, but will be hard to
debug (the failure is detected long after it happens) and very verbose.
> 
> 3) I can use gmock to write a specific set of expected events for a pass
and verify that they happen. It was specifically designed to make verifying this
kind of interaction explicit with little boilerplate and decent error messages
when it fails.
This is probably most like #1, but i would either improve (or add a verbose
option to) -debug-pass=Structure.  Then just write a test which calls opt with
some passes and uses FileCheck to verify the debug output.

This is nice as it means that if anyone ever wants to verify for performance
reasons that a given test invalidates a very specific set of passes and never
regresses from that set then they can use FileCheck for this.

And it looks like debug-pass is available on release builds.

Pete
> 
> I'll try to come up with an example of #3 this evening or tomorrow.
> 
> -Chandler
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20131114/aba9cb74/attachment.html>

On Thu, Nov 14, 2013 at 3:00 PM, Pete Cooper <peter_cooper at apple.com>
wrote:
> This is probably most like #1, but i would either improve (or add a
> verbose option to) -debug-pass=Structure.  Then just write a test which
> calls opt with some passes and uses FileCheck to verify the debug output.
>
Yes, but see the problems with it that I brought up. Note that the new pass
manager makes this significantly more complex because there isn't a linear
series of passes (regardless of nesting).

> This is nice as it means that if anyone ever wants to verify for
> performance reasons that a given test invalidates a very specific set of
> passes and never regresses from that set then they can use FileCheck for
> this.
>
I don't think these are realistic or good tests in the new model... I'm
not
really sure what we're gaining from this at all. It sounds like a much
worse case of what other people are complaining about with unittests?

>
> And it looks like debug-pass is available on release builds.
>
... not always. Certainly, with the added complexity needed to check a
system that uses active caching, I don't think we would want it in release
builds.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20131114/b3fcfdb3/attachment.html>

> This is probably most like #1, but i would either improve (or add a verbose
> option to) -debug-pass=Structure.  Then just write a test which calls opt
> with some passes and uses FileCheck to verify the debug output.
>
> This is nice as it means that if anyone ever wants to verify for
performance
> reasons that a given test invalidates a very specific set of passes and
> never regresses from that set then they can use FileCheck for this.
+1

My point on IRC was that I consider gtest a necessary evil since it
lets us test properties like "DenseMap releases memory after a series
of insertions and deletions".  To test that with FileCheck we would
need to make the test brittle (by setting a program wide memory limit
for example) or add dumping to DenseMap just to write the test.

On the example you have it looks like the extra logging would be a
nice debugging tool for finding when/why something is being recomputed
and not specific to the testcase at all.

Cheers,
Rafael