llvm dev - Nov 2013 - [LLVMdev] Any objections to my importing GoogleMock to go with GoogleTest in LLVM?

On Thu, Nov 14, 2013 at 10:58 AM, Chris Lattner <clattner at apple.com>
wrote:
> On Nov 14, 2013, at 3:16 AM, Chandler Carruth <chandlerc at
google.com>
> wrote:
>
> However, when we are adding interfaces or generic utilities to LLVM
> (admittedly, not the common case) I don't think we do ourselves any
favors
> by using only half of the available tools to write unit tests for them.
>
>
> I agree in principle, but it leads me to a different conclusion.  We have
> other great testing support, which means that the mocking *should* only be
> used sparingly.  Given that it will not be used much, the cost of carrying
> it around (and for people to learn how to use/maintain it) is high.
>
I think the cost of carrying it around is essentially zero. I'm happy to do
any of the maintenance. People who don't know how to use it or want to
learn how to use it don't need to use it. If it isn't making their job
of
writing tests sufficiently easier to justify, then they don't use it. I see
this as a good pattern.

>
> I’ve said this before, but I’m not a fan of our current use of gtest for
> unit testing.  I have never had the unit tests catch a bug, but I have had
> to update the tests countless times.  At least for my purposes, the unit
> tests cause significantly more harm than good - and it certainly isn’t
> because I write perfect code. :-)
>
I seem to recall code review spotting a bug that would have been caught by
a unittest were one written. Also, I don't see a lot of patches going in
over the last 2 years that had to re-shuffle unittests. They happen, but
they are very rare. So while I agree this can be a problem, I don't see it
being a problem in practice.

Even so, you aren't the only one we're trying to optimize for. A lot of
people have written unittests using the framework, and I think the
incremental cost of making it a slightly more powerful framework (by adding
one complementary library) is really low.

>
> There is definitely a culture/religion around testing and TDD, and I am
> well aware that many projects don’t have proper tests (which LLVM doesn’t
> suffer from).  However, there is a pragmatic balance to be struck here, and
> I personally think that adding gmock and pushing the unit tests stuff even
> further is a bad use of testing time (i.e., increases test cycles for make
> check) and maintenance time (updating tests given that we don’t have a
> stable API).
>
These two things (adding gmock and pushing unittests further) are not
necessarily related, and I don't plan to do the latter. I'm asking if
doing
the former would cause significant problems for any consumers of LLVM, and
I don't hear any statements to that effect except for David Chisnall's
which I responded to specifically.

I'm not trying to make LLVM use unittests everywhere, I'm just trying to
get a tool added to the toolbox so that a unittest I'm already writing can
be written more simply and in a more maintainable fashion.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20131114/15e6ba8a/attachment.html>

On 11/15/13 03:52 AM, Chandler Carruth wrote:
>
> I'm not trying to make LLVM use unittests everywhere, I'm just
trying
> to get a tool added to the toolbox so that a unittest I'm already 
> writing can be written more simply and in a more maintainable fashion.
You're welcome to ignore me and keep writing eloquent emails, but you 
still haven't shown an exact use case - why not write a unit test which 
demonstrates the benefit and post a patch for review? for those who are 
not familiar with gtest/gmock it makes it very clear.. Then the 
discussion moves from opinions and "feelings" to tangibles

> I think the cost of carrying it around is essentially zero. I'm happy
to do
> any of the maintenance. People who don't know how to use it or want to
learn
> how to use it don't need to use it. If it isn't making their job of
writing
> tests sufficiently easier to justify, then they don't use it. I see
this as
> a good pattern.
That is not the case. If the test finds any bug at all, people have to
look at the testcase and see if it is failing.

Even if not bug is found, someone doing refactoring has to change the
test to use the new apis.

Cheers,
Rafael

On Thu, Nov 14, 2013 at 1:04 PM, Rafael Espíndola <
rafael.espindola at gmail.com> wrote:
> > I think the cost of carrying it around is essentially zero. I'm
happy to
> do
> > any of the maintenance. People who don't know how to use it or
want to
> learn
> > how to use it don't need to use it. If it isn't making their
job of
> writing
> > tests sufficiently easier to justify, then they don't use it. I
see this
> as
> > a good pattern.
>
> That is not the case. If the test finds any bug at all, people have to
> look at the testcase and see if it is failing.
>
> Even if not bug is found, someone doing refactoring has to change the
> test to use the new apis.

Yes, but this is relatively rare in both cases. I looked at the maintenance
burden of unittest/... and it doesn't look like this is a common occurrence
in LLVM. I also expect the results of using these tools to be easier for
maintainers rather than harder in most cases. (See my reply to Sean and C.
Bergstrom...)
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20131114/141b027f/attachment.html>

On Thu, Nov 14, 2013 at 1:04 PM, "C. Bergström" <cbergstrom at
pathscale.com>wrote:
> On 11/15/13 03:52 AM, Chandler Carruth wrote:
>
>>
>> I'm not trying to make LLVM use unittests everywhere, I'm just
trying to
>> get a tool added to the toolbox so that a unittest I'm already
writing can
>> be written more simply and in a more maintainable fashion.
>>
> You're welcome to ignore me and keep writing eloquent emails, but you
> still haven't shown an exact use case - why not write a unit test which
> demonstrates the benefit and post a patch for review? for those who are not
> familiar with gtest/gmock it makes it very clear.. Then the discussion
> moves from opinions and "feelings" to tangibles
>
I'm not ignoring you, i'm working on exactly that. But it takes a bit
more
time, and so I was trying to respond promptly to the emails which weren't
asking for a specific example concurrently with working on a demo of what
I'd like to do.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20131114/2fd62389/attachment.html>

On Thu, Nov 14, 2013 at 1:04 PM, Rafael Espíndola <
rafael.espindola at gmail.com> wrote:
> > I think the cost of carrying it around is essentially zero. I'm
happy to
> do
> > any of the maintenance. People who don't know how to use it or
want to
> learn
> > how to use it don't need to use it. If it isn't making their
job of
> writing
> > tests sufficiently easier to justify, then they don't use it. I
see this
> as
> > a good pattern.
>
> That is not the case. If the test finds any bug at all, people have to
> look at the testcase and see if it is failing.
>
> Even if not bug is found, someone doing refactoring has to change the
> test to use the new apis.

On IRC, Rafael indicated he would like to understand the specific use case
I had in mind better. I'm still working on a concrete example, but figured
I'd clarify more than my initial mail did.

I'm working on the pass manager. I need a way to test that specific passes
in the pass manager are being run and invalidated appropriately based on
the dependencies between them and the caching infrastructure. Verifying
this can be done in two ways:

1) I can add non-trivial code which essentially dumps the state at each
point, along with a command line tool and collection of fake passes, and
use FileCheck against this code to verify things. This works for asserts
builds but not for no-asserts builds, and generally feels like working
around a missing feature in our testing infrastructure. This is the same
reasons we don't use FileCheck and state serialization to test our DenseMap
implementation.

2) I can write a unittest using gtest with a completely custom collection
of N passes written for every single test, each with a different set of
integer output parameters that are incremented and decremented at the right
points, and then a verification of their final value. This will work, but
will be hard to debug (the failure is detected long after it happens) and
very verbose.

3) I can use gmock to write a specific set of expected events for a pass
and verify that they happen. It was specifically designed to make verifying
this kind of interaction explicit with little boilerplate and decent error
messages when it fails.

I'll try to come up with an example of #3 this evening or tomorrow.

-Chandler
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20131114/93db21df/attachment.html>

On Nov 14, 2013, at 12:52 PM, Chandler Carruth <chandlerc at google.com>
wrote:
> On Thu, Nov 14, 2013 at 10:58 AM, Chris Lattner <clattner at
apple.com> wrote:
> On Nov 14, 2013, at 3:16 AM, Chandler Carruth <chandlerc at
google.com> wrote:
>> However, when we are adding interfaces or generic utilities to LLVM
(admittedly, not the common case) I don't think we do ourselves any favors
by using only half of the available tools to write unit tests for them.
> 
> I agree in principle, but it leads me to a different conclusion.  We have
other great testing support, which means that the mocking *should* only be used
sparingly.  Given that it will not be used much, the cost of carrying it around
(and for people to learn how to use/maintain it) is high.
> 
> I think the cost of carrying it around is essentially zero. I'm happy
to do any of the maintenance.
I'm not talking about the cost of maintaining gmock itself.  I'm talking
about the cost of maintaining tests that are ultimately written around it.  Alp
eloquently made the point up-thread.
> People who don't know how to use it or want to learn how to use it
don't need to use it. If it isn't making their job of writing tests
sufficiently easier to justify, then they don't use it. I see this as a good
pattern.
This is factually not the case.  I've had to learn a lot about gtest just to
figure out its wonky macro system, figure out how to rerun the failing tests,
etc.

It is also an empirical fact that linking all the unit tests slows down my build
in a very noticeable way... every time I build, even though I derive near zero
benefit from it.
> 
> I’ve said this before, but I’m not a fan of our current use of gtest for
unit testing.  I have never had the unit tests catch a bug, but I have had to
update the tests countless times.  At least for my purposes, the unit tests
cause significantly more harm than good - and it certainly isn’t because I write
perfect code. :-)
> 
> I seem to recall code review spotting a bug that would have been caught by
a unittest were one written.
Right, validating my assertion that while TDD and unit testing are good in
general, they may not be right for LLVM.  In LLVM, we have mature tests of other
sorts, as well as a strong process of review.

Further, even if you guys didn't notice the bug immediately, I would have
noticed it in my (planned) next commit.  That commit was the code that used it. 
You don't need a unit test to catch obvious bozo bugs like the one you are
referring to.
> Even so, you aren't the only one we're trying to optimize for. A
lot of people have written unittests using the framework, and I think the
incremental cost of making it a slightly more powerful framework (by adding one
complementary library) is really low.
I am trying to optimize for tests that add value: this means that they find real
bugs that would otherwise be expensive to find, and do so at low cost.  The goal
is not to have tests for their own sake.

I fully understand that tests and mocking and other tooling can be useful to
bring up a major new subsystem like the passmanager.  I just don't think it
makes sense to carry around a bringup tool forever in the source tree.
>  
> There is definitely a culture/religion around testing and TDD, and I am
well aware that many projects don’t have proper tests (which LLVM doesn’t suffer
from).  However, there is a pragmatic balance to be struck here, and I
personally think that adding gmock and pushing the unit tests stuff even further
is a bad use of testing time (i.e., increases test cycles for make check) and
maintenance time (updating tests given that we don’t have a stable API).
> 
> These two things (adding gmock and pushing unittests further) are not
necessarily related, and I don't plan to do the latter. I'm asking if
doing the former would cause significant problems for any consumers of LLVM, and
I don't hear any statements to that effect except for David Chisnall's
which I responded to specifically.
> 
> I'm not trying to make LLVM use unittests everywhere, I'm just
trying to get a tool added to the toolbox so that a unittest I'm already
writing can be written more simply and in a more maintainable fashion.
I see gmock tests as something that will make API evolution much more difficult,
and believe that it will be much more of a problem in practice than unit tests,
because it necessarily requires more internal implementation details to be
exposed to the testing framework.

-Chris
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20131114/36e30ed5/attachment.html>

On 15 November 2013 04:38, Chris Lattner <clattner at apple.com> wrote:
> Right, validating my assertion that while TDD and unit testing are good in
> general, they may not be right for LLVM.  In LLVM, we have mature tests of
> other sorts, as well as a strong process of review.
>
While I understand the value of TDD, I have to agree with Chris, here.

I have been bitten by having to write silly boundary checks tests that the
code would never allow, and getting 100% test coverage (of lines AND
branches), only to realize that a test engineer could break my new feature
in many different ways by misusing it on a command-line level.

I personally think we'd have a much better use of time by making the MI
layer dumpable and re-readable, so we could create lots of very specific
low-level tests, than having yet-another unit-test infrastructure.

The only place I think that unit-tests are worthy is on base libraries
(APFLoat, APInt, containers, basic algorithms), and for that, you need
nothing special.


> Further, even if you guys didn't notice the bug immediately, I would
have
> noticed it in my (planned) next commit.  That commit was the code that used
> it.  You don't need a unit test to catch obvious bozo bugs like the one
you
> are referring to.
>
That is the point. Most of us don't commit before a check-all anyway, so
looking back on the commits list won't give you a clear picture.

cheers,
--renato
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20131115/3c623c4b/attachment.html>