llvm dev - Jan 2009 - [LLVMdev] Private headers and testing

2009/1/2 Chris Lattner <clattner at apple.com>
> On Jan 2, 2009, at 12:21 PM, Misha Brukman wrote:
> Do you have a specific example of a unit test that would need these?  I
> really think these should stay private.
>
Let's take lib/Transforms/Utils/CodeExtractor.cpp .  The public interface
for it is in include/llvm/Transform/Utils/FunctionUtils.h, with only the
high-level API.

If I want to test some corner-case (consider the number of code paths in
that code), I have to strain to come up with some combination of a Function
and input BasicBlocks that will fit all the conditions from the outset.  Or,
I can just feed each individual method in the class exactly the corner cases
I will want it to handle, and verify its output (or side effects) exactly,
to make sure I pin-pointed the issue.

Or, let's say I want to write some tests for the LLParser -- the only public
API is "ParseAssemblyString", I can't unittest each individual
method.  The
unittest for this using the public API will be very brittle as it would have
to check the contents of the returned error message, instead of calling each
Parse*() function directly and analyzing its output.

Of course, we should still have a test for the overall public API, but
that's in addition to the small unit tests to make sure each one works as it
should.

Note that this is a parallel to having an on-disk representation of IR
(which LLVM has) to a lack of such (which some other compilers had and/or
have now).  In the former case, to test an optimization, one just writes
LLVM IR and runs the pass directly on the case at hand.  In the other case,
one has to strain to come up with a front-end input source that will trigger
something in the 'blackbox' middle or back-end code, knowing that it
will
get pre-digested before it gets there, and hence it's hard to know what
you're really testing on.

Misha
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20090102/d46222fa/attachment.html>

On 2 Jan 2009, at 22:48, Misha Brukman wrote:
> Let's take lib/Transforms/Utils/CodeExtractor.cpp .  The public  
> interface for it is in include/llvm/Transform/Utils/FunctionUtils.h,  
> with only the high-level API.
I believe this should be possible to handle within GNU Make. Assuming  
that the source file in question is ‘TheTest.cpp’, something like  
this should do the trick:
$(ObjDir)/TheTest.o $(ObjDir)/TheTest.lo: \
	CompileCommonOpts += -I$(PROJ_SRC_ROOT)/lib/Transforms/Utils
The specific entry in the GNU Make manual is ‘6.10 Target-specific  
Variable Values’. [1] (I used this when trying to decrease the amount  
of libraries in LLVM; an example can be seen at [2].)

I hope this helps :)

[1] <http://www.gnu.org/software/make/manual/make.html#Target_002dspecific 
 >
[2]
<http://www.bitbucket.org/danchr/llvm/src/ab1e5398289e/lib/Target/Makefile#cl-59
 >

--

Dan Villiom Podlaski Christiansen, stud. scient.,
danchr at cs.au.dk, danchr at gmail.com

On Jan 2, 2009, at 1:48 PM, Misha Brukman wrote:
> 2009/1/2 Chris Lattner <clattner at apple.com>
> On Jan 2, 2009, at 12:21 PM, Misha Brukman wrote:
> Do you have a specific example of a unit test that would need  
> these?  I really think these should stay private.
>
> Let's take lib/Transforms/Utils/CodeExtractor.cpp .  The public  
> interface for it is in include/llvm/Transform/Utils/FunctionUtils.h,  
> with only the high-level API.
>
> If I want to test some corner-case (consider the number of code  
> paths in that code), I have to strain to come up with some  
> combination of a Function and input BasicBlocks that will fit all  
> the conditions from the outset.  Or, I can just feed each individual  
> method in the class exactly the corner cases I will want it to  
> handle, and verify its output (or side effects) exactly, to make  
> sure I pin-pointed the issue.
Sure, ok.  But that is a public API, I'm specifically interested in  
cases you need private API.
> Or, let's say I want to write some tests for the LLParser -- the  
> only public API is "ParseAssemblyString", I can't unittest
each
> individual method.  The unittest for this using the public API will  
> be very brittle as it would have to check the contents of the  
> returned error message, instead of calling each Parse*() function  
> directly and analyzing its output.
The code is structured so that it is trivially easy to construct  
testcases for anything you're concerned with.  If you give me an  
example, I'll give you a testcase (in .ll form).  I don't see the need  
for unittests for LLParser.

-Chris
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20090103/34ab5a6f/attachment.html>

2009/1/3 Chris Lattner <clattner at apple.com>
> On Jan 2, 2009, at 1:48 PM, Misha Brukman wrote:
>
> 2009/1/2 Chris Lattner <clattner at apple.com>
>
>> On Jan 2, 2009, at 12:21 PM, Misha Brukman wrote:
>> Do you have a specific example of a unit test that would need these?  I
>> really think these should stay private.
>>
>
> Let's take lib/Transforms/Utils/CodeExtractor.cpp .  The public
interface
> for it is in include/llvm/Transform/Utils/FunctionUtils.h, with only the
> high-level API.
>
> If I want to test some corner-case (consider the number of code paths in
> that code), I have to strain to come up with some combination of a Function
> and input BasicBlocks that will fit all the conditions from the outset. 
Or,
> I can just feed each individual method in the class exactly the corner
cases
> I will want it to handle, and verify its output (or side effects) exactly,
> to make sure I pin-pointed the issue.
>
> Sure, ok.  But that is a public API, I'm specifically interested in
cases
> you need private API.
>
I'm not sure I understand what you mean.  What I am saying is that I want to
expose the innards of CodeExtractor.cpp in a header file, #include that in
the unittest, and move the implementation out of an anonymous namespace into
llvm::transforms::utils, or llvm::internal::transforms::utils, or similar,
so it can be unit-tested.  This is not about exposing the private API to
end-users of LLVM or even other LLVM libraries, just unittests.

The public API, while useful to the end-user, is limiting when it comes to
unit-testing.  "Unit" in this case is a single method of the actual
implementation (i.e., a method of the CodeExtractor class), not a single
function call of the high-level exported public API, which will exercise the
entire class and its many methods.

Misha
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20090107/f76ef1b4/attachment.html>