2008/12/27 Mark Kromis <greybird at mac.com>> Just a curiosity question, why push for gtest vs Boost Test or > a different test suite? > I normally use Boost, and their test suite, so I'm more familiar with that. > So I was wondering is one better then the other, or is it just that someone > makes a patch for it? >I looked more into Boost.Test to see what's in it. Boost.Test doesn't seem to be stand-alone -- I don't see a way to use Boost.Test without importing some other chunks of Boost that the testing library depends on. While Boost is a fine set of libraries, I don't think we want to increase the LLVM distribution by sizeof(Boost) just to enable unittesting, nor do we want to spend the time on maintaining a subset of Boost that's "just enough" to build and use the unittest library, along a modified configure/build process that Boost wants to use (Boost.Build? Boost.Jam?). Boost also seems to want to use exceptions, and LLVM does not want to. I'm not sure if there would be some difficulties in running a build where some libraries are compiled with no exceptions, some with, and the results are linked together. At the best case, it would complicate our build system to be able to support different set of flags for building LLVM libraries vs. Boost.Test (and the rest of Boost that we import). Sample usage of Boost.Test: http://svn.boost.org/svn/boost/trunk/libs/test/example/unit_test_example_12.cpp Note the code at the end setting up the test suite -- this is boilerplate code that I think shouldn't be necessary to setup and run tests. Google Test, on the other hand, has no external dependencies, and is distributed as a dozen of .h/.cc files; supports Makefile, SCons, and Xcode; and doesn't use exceptions or RTTI. Sample usage of GTest: http://code.google.com/p/googletest/source/browse/trunk/samples/sample5_unittest.cc GTest-specific LOC besides the #include statement: 0. Note that I'm not counting main() for either Boost or GTest, because both provide a standard main() for use with almost all test files. Misha -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20081227/eeff2436/attachment.html>
On 2008-12-27, at 17:41, Misha Brukman wrote:> 2008/12/27 Mark Kromis <greybird at mac.com> > Just a curiosity question, why push for gtest vs Boost Test or a > different test suite? > I normally use Boost, and their test suite, so I'm more familiar > with that. So I was wondering is one better then the other, or is it > just that someone makes a patch for it? > > I looked more into Boost.Test to see what's in it. Boost.Test > doesn't seem to be stand-alone -- I don't see a way to use > Boost.Test without importing some other chunks of Boost that the > testing library depends on. While Boost is a fine set of libraries, > I don't think we want to increase the LLVM distribution by > sizeof(Boost) just to enable unittesting, nor do we want to spend > the time on maintaining a subset of Boost that's "just enough" to > build and use the unittest library, along a modified configure/build > process that Boost wants to use (Boost.Build? Boost.Jam?).Indeed, Boost.Test requires approximately 500 header files, minimally. — Gordon -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20081227/95300bc7/attachment.html>
On Dec 27, 2008, at 7:41 PM, Misha Brukman wrote:> 2008/12/27 Mark Kromis <greybird at mac.com> > Just a curiosity question, why push for gtest vs Boost Test or a > different test suite? > I normally use Boost, and their test suite, so I'm more familiar > with that. So I was wondering is one better then the other, or is it > just that someone makes a patch for it? > > I looked more into Boost.Test to see what's in it. Boost.Test > doesn't seem to be stand-alone -- I don't see a way to use > Boost.Test without importing some other chunks of Boost that the > testing library depends on. While Boost is a fine set of libraries, > I don't think we want to increase the LLVM distribution by > sizeof(Boost) just to enable unittesting, nor do we want to spend > the time on maintaining a subset of Boost that's "just enough" to > build and use the unittest library, along a modified configure/build > process that Boost wants to use (Boost.Build? Boost.Jam?).So are you planning on maintaining whatever test system, or just have them as a pre-requisite. For example are you going to have the gtest incorporated, or have them install it separately first? I was under the impression that the user would have to install gtest first.> > Boost also seems to want to use exceptions, and LLVM does not want > to. I'm not sure if there would be some difficulties in running a > build where some libraries are compiled with no exceptions, some > with, and the results are linked together. At the best case, it > would complicate our build system to be able to support different > set of flags for building LLVM libraries vs. Boost.Test (and the > rest of Boost that we import).http://www.boost.org/doc/libs/1_37_0/libs/utility/throw_exception.html #define BOOST_NO_EXCEPTIONS> > Sample usage of Boost.Test: http://svn.boost.org/svn/boost/trunk/libs/test/example/unit_test_example_12.cpp > Note the code at the end setting up the test suite -- this is > boilerplate code that I think shouldn't be necessary to setup and > run tests. >http://svn.boost.org/svn/boost/trunk/libs/test/example/unit_test_example_01.cpp My test cases are not that in-depth, I'm much closer to sample 1. I haven't found a reason to go that crazy yet.> Google Test, on the other hand, has no external dependencies, and is > distributed as a dozen of .h/.cc files; supports Makefile, SCons, > and Xcode; and doesn't use exceptions or RTTI. >Gtest is much more lightweight, no comparison there. I know that llvm is not very good with exceptions, but should a test case system support that?> Sample usage of GTest: http://code.google.com/p/googletest/source/browse/trunk/samples/sample5_unittest.cc > GTest-specific LOC besides the #include statement: 0.I think it links to a library as well.> Note that I'm not counting main() for either Boost or GTest, because > both provide a standard main() for use with almost all test files. > > Misha > _______________________________________________ > LLVM Developers mailing list > LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdevAlso for a note of reference, your links to the examples are the most advanced samples. So boost can do more, thus has more weight/bloat behind it. Were the other test kits looked at? Is gtest the best solution for the project. Is this something your planning as putting in the tree, thus require pulling in changes from google (license allowing), or does user need to have the libraries/headers pre-installed? My question was not to cause a battle, but I wanted to be sure we were using the right test kit, and not just picking one just because. For example gtest is very light weight test kit, that can do the job, but will the tests outgrow what the test kit can do, and cause a conversion to a more advanced one later? Regards, Mark Kromis -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20081227/398a6713/attachment.html> -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 832 bytes Desc: not available URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20081227/398a6713/attachment.sig>
On Sat, Dec 27, 2008 at 6:56 PM, Mark Kromis <greybird at mac.com> wrote:> > On Dec 27, 2008, at 7:41 PM, Misha Brukman wrote: > > 2008/12/27 Mark Kromis <greybird at mac.com> > >> Just a curiosity question, why push for gtest vs Boost Test or >> a different test suite? >> I normally use Boost, and their test suite, so I'm more familiar with >> that. So I was wondering is one better then the other, or is it just that >> someone makes a patch for it? >> > > I looked more into Boost.Test to see what's in it. Boost.Test doesn't seem > to be stand-alone -- I don't see a way to use Boost.Test without importing > some other chunks of Boost that the testing library depends on. While Boost > is a fine set of libraries, I don't think we want to increase the LLVM > distribution by sizeof(Boost) just to enable unittesting, nor do we want to > spend the time on maintaining a subset of Boost that's "just enough" to > build and use the unittest library, along a modified configure/build process > that Boost wants to use (Boost.Build? Boost.Jam?). > > > So are you planning on maintaining whatever test system, or just have them > as a pre-requisite. For example are you going to have the gtest > incorporated, or have them install it separately first? I was under the > impression that the user would have to install gtest first. > > > Boost also seems to want to use exceptions, and LLVM does not want to. I'm > not sure if there would be some difficulties in running a build where some > libraries are compiled with no exceptions, some with, and the results are > linked together. At the best case, it would complicate our build system to > be able to support different set of flags for building LLVM libraries vs. > Boost.Test (and the rest of Boost that we import). > > > http://www.boost.org/doc/libs/1_37_0/libs/utility/throw_exception.html > #define BOOST_NO_EXCEPTIONS > > > Sample usage of Boost.Test: > http://svn.boost.org/svn/boost/trunk/libs/test/example/unit_test_example_12.cpp > Note the code at the end setting up the test suite -- this is boilerplate > code that I think shouldn't be necessary to setup and run tests. > > > > > http://svn.boost.org/svn/boost/trunk/libs/test/example/unit_test_example_01.cpp > My test cases are not that in-depth, I'm much closer to sample 1. I haven't > found a reason to go that crazy yet. > > > Google Test, on the other hand, has no external dependencies, and is > distributed as a dozen of .h/.cc files; supports Makefile, SCons, and Xcode; > and doesn't use exceptions or RTTI. > > > Gtest is much more lightweight, no comparison there. I know that llvm is > not very good with exceptions, but should a test case system support that? > > > Sample usage of GTest: > http://code.google.com/p/googletest/source/browse/trunk/samples/sample5_unittest.cc > GTest-specific LOC besides the #include statement: 0. > > > I think it links to a library as well. > > Note that I'm not counting main() for either Boost or GTest, because both > provide a standard main() for use with almost all test files. > > Misha > _______________________________________________ > LLVM Developers mailing list > LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev > > > Also for a note of reference, your links to the examples are the > most advanced samples. So boost can do more, thus has more weight/bloat > behind it. > > Were the other test kits looked at? Is gtest the best solution for the > project. > > Is this something your planning as putting in the tree, > thus require pulling in changes from google (license allowing), or does user > need to have the libraries/headers pre-installed? >Including it in the tree is the most reasonable thing to do. No point in inconveniencing the user over tiny libraries with liberal licenses.> > My question was not to cause a battle, but I wanted to be sure we were > using the right test kit, and not just picking one just because. For > example gtest is very light weight test kit, that can do the job, but will > the tests outgrow what the test kit can do, and cause a conversion to a > more advanced one later? >I am not sure what advanced features you are thinking of that gtest doesn't already offer; it's pretty sophisticated despite it's small size. Can you give an example of something gtest doesn't already support? Newer releases added support for data parameterized tests, which is really useful. Type parameterized tests are being discussed on the mailing list. All while maintaining a small codebase! One other nice feature of using gtest is that it integrates with gmock, one of the only really good C++ mocking libraries available. It was just recently released. http://code.google.com/p/googlemock Keir> Regards, > Mark Kromis > > _______________________________________________ > LLVM Developers mailing list > LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20081227/d01b5a95/attachment.html>
2008/12/27 Mark Kromis <greybird at mac.com>> So are you planning on maintaining whatever test system, or just have them > as a pre-requisite. For example are you going to have the gtest > incorporated, or have them install it separately first? I was under the > impression that the user would have to install gtest first. >The current plan is to check in the unittest library into LLVM and build it as part of the test process, so the user doesn't have to do anything separately.> > Boost also seems to want to use exceptions, and LLVM does not want to. I'm > not sure if there would be some difficulties in running a build where some > libraries are compiled with no exceptions, some with, and the results are > linked together. At the best case, it would complicate our build system to > be able to support different set of flags for building LLVM libraries vs. > Boost.Test (and the rest of Boost that we import). > > http://www.boost.org/doc/libs/1_37_0/libs/utility/throw_exception.html > #define BOOST_NO_EXCEPTIONS >Thanks for the pointer.> Sample usage of Boost.Test: > http://svn.boost.org/svn/boost/trunk/libs/test/example/unit_test_example_12.cpp > Note the code at the end setting up the test suite -- this is boilerplate > code that I think shouldn't be necessary to setup and run tests. > > > http://svn.boost.org/svn/boost/trunk/libs/test/example/unit_test_example_01.cpp > My test cases are not that in-depth, I'm much closer to sample 1. I haven't > found a reason to go that crazy yet. >You would need to use fixtures if you are testing classes and their interactions, otherwise each of your test functions will re-do everything that's in the fixture's SetUp() method -- why would you want to repeat the same setup code for every test case?> Google Test, on the other hand, has no external dependencies, and is > distributed as a dozen of .h/.cc files; supports Makefile, SCons, and Xcode; > and doesn't use exceptions or RTTI. > > Gtest is much more lightweight, no comparison there. I know that llvm is > not very good with exceptions, but should a test case system support that? >GTest allows testing for exceptions, it just doesn't require them to work properly: http://code.google.com/p/googletest/wiki/GoogleTestAdvancedGuide#Exception_Assertions> Sample usage of GTest: > http://code.google.com/p/googletest/source/browse/trunk/samples/sample5_unittest.cc > GTest-specific LOC besides the #include statement: 0. > > I think it links to a library as well. >Yes, that's true -- just about any unittesting library will have that requirement, that wasn't part of the comparison. I was pointing out how much C++ a user has to write in their test files for a reasonable test with fixtures.> Also for a note of reference, your links to the examples are the > most advanced samples. So boost can do more, thus has more weight/bloat > behind it. >Gordon Henriksen pointed out earlier in the thread that Boost.Test will require pulling in 500 headers from Boost just to work, so that's pretty heavy weight. Also, I was comparing apples-to-apples, not taking the simplest example of GTest and the most complex of Boost -- the two examples I chose were doing the same thing, i.e., tests with fixtures. What does Boost provide that GTest does not, that you think LLVM needs, such that it's worth importing the entire Boost distribution into the LLVM tree? Were the other test kits looked at? Is gtest the best solution for the> project. >There are dozens of C++ unittesting libraries out there, I admit I haven't looked at all of them. I don't know which ones are still under active development or use, but I (and a few other posters) are familiar with and have used GTest enough to say that it's very well suited for LLVM. Is this something your planning as putting in the tree, thus require pulling> in changes from google (license allowing), or does user need to have the > libraries/headers pre-installed? >I already answered this question above (first option: in the tree). GTest is under the BSD license, which is compatible with LLVM (though I am not a copyright lawyer).> For example gtest is very light weight test kit, that can do the job, but > will the tests outgrow what the test kit can do, and cause a conversion to a > more advanced one later? >As soon as LLVM outgrows what GTest can do, we can easily write wrapper macros to forward from GTest-style macros to the other, more suitable framework; however, I think you're conflating the two concepts of "light-weight" and "trivial" -- GTest is light-weight, but not trivial. Misha -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20081227/33bab223/attachment.html>
Mark Kromis wrote:> On Dec 27, 2008, at 7:41 PM, Misha Brukman wrote: >> 2008/12/27 Mark Kromis <greybird at mac.com> >> Just a curiosity question, why push for gtest vs Boost Test or a >> different test suite? >> I normally use Boost, and their test suite, so I'm more familiar with >> that. So I was wondering is one better then the other, or is it just >> that someone makes a patch for it?I did a very minimal amount of research into unit testing frameworks before I started using gtest for my own work. Since then I've become quite familiar with gtest's features, which made it a natural choice for me. I presume that other testing frameworks will have their own defenders on this list, and I am certainly willing to listen to what they have to say. The unit test comparison article mentioned earlier (http://gamesfromwithin.com/?p=29) is a good starting point for evaluating different unit test frameworks. Although gtest is too new to have been mentioned in the article directly, if you actually compare gtest's features against the various criteria used by the author of the article, it comes off quite well. Note that the author of the article later went and created his own unit test framework (UnitTest++), based on his experience with all of the other frameworks he tested. This was the only framework that I seriously looked at other than gtest, and I found it to be a bit too minimal for what I needed.>> I looked more into Boost.Test to see what's in it. Boost.Test >> doesn't seem to be stand-alone -- I don't see a way to use Boost.Test >> without importing some other chunks of Boost that the testing library >> depends on. While Boost is a fine set of libraries, I don't think we >> want to increase the LLVM distribution by sizeof(Boost) just to >> enable unittesting, nor do we want to spend the time on maintaining a >> subset of Boost that's "just enough" to build and use the unittest >> library, along a modified configure/build process that Boost wants to >> use (Boost.Build? Boost.Jam?).Although I haven't actually tried Boost.Test, I kind of figured that this would be the case - that you pretty much have to drink the "Boost Kool-Aid" in order to use it.> So are you planning on maintaining whatever test system, or just have > them as a pre-requisite. For example are you going to have the gtest > incorporated, or have them install it separately first? I was under > the impression that the user would have to install gtest first.So the plan is to take a snapshot of gtest and check it in to the LLVM tree, rather than have it installed separately. I was able to integrate gtest into LLVM's build system fairly easily, as gtest is designed to be integrated into a foreign build system - basically I just ignored the makefile that comes with gtest, and wrote an LLVM-style makefile rule for it. There's a special source file in gtest which includes all other sources that is intended for just such a purpose. I did not need to modify the gtest sources in any way. This means that keeping the gtest snapshot up to date will be trivial, since it will only require copying in the latest gtest snapshot and checking it in to LLVM - presuming that gtest remains backwards compatible, which I assume it will. Licensing-wise, both LLVM and gtest are distributed under a fairly permissive BSD-style license. I don't know who would make the judgement call as to whether or not the licenses can co-exist. However, since neither license is "viral" in the sense of wanting to apply any sort of restriction on derived works or the "work as a whole", I see no barrier to shipping a combined product with different portions falling under different licenses. Thus, the unit tests themselves would still fall under the LLVM license, and linking the unit tests with gtest would not violate either license. Of course, IANAL. From a maintenance standpoint, we have already heard from several enthusiastic volunteers who are involved in both the development of LLVM and gtest. So I doubt there will be much problems on that score. My personal goal is that one should be able to check out the head of LLVM on a generic Linux/OS-X system, with only the standard development environment (i.e. make/gcc/etc) and type "make unittest" and have the tests run. In the longer term, I'd like to see LLVM have an automated build that runs the unit tests as part of the build. I noticed that gtest has an option to output test results in XML form, although I have not played with this personally, it might be useful in this regard.>> >> Boost also seems to want to use exceptions, and LLVM does not want >> to. I'm not sure if there would be some difficulties in running a >> build where some libraries are compiled with no exceptions, some >> with, and the results are linked together. At the best case, it >> would complicate our build system to be able to support different set >> of flags for building LLVM libraries vs. Boost.Test (and the rest of >> Boost that we import). > > http://www.boost.org/doc/libs/1_37_0/libs/utility/throw_exception.html > #define BOOST_NO_EXCEPTIONS > >> >> Sample usage of Boost.Test: >> http://svn.boost.org/svn/boost/trunk/libs/test/example/unit_test_example_12.cpp >> >> Note the code at the end setting up the test suite -- this is >> boilerplate code that I think shouldn't be necessary to setup and run >> tests. >> > > > http://svn.boost.org/svn/boost/trunk/libs/test/example/unit_test_example_01.cpp > > My test cases are not that in-depth, I'm much closer to sample 1. I > haven't found a reason to go that crazy yet. > > >> Google Test, on the other hand, has no external dependencies, and is >> distributed as a dozen of .h/.cc files; supports Makefile, SCons, and >> Xcode; and doesn't use exceptions or RTTI. >> > > Gtest is much more lightweight, no comparison there. I know that llvm > is not very good with exceptions, but should a test case system > support that? > > >> Sample usage of GTest: >> http://code.google.com/p/googletest/source/browse/trunk/samples/sample5_unittest.cc >> >> GTest-specific LOC besides the #include statement: 0. > > I think it links to a library as well. > >> Note that I'm not counting main() for either Boost or GTest, because >> both provide a standard main() for use with almost all test files. >> >> Misha >> _______________________________________________ >> LLVM Developers mailing list >> LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu >> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev > > Also for a note of reference, your links to the examples are the most > advanced samples. So boost can do more, thus has more weight/bloat > behind it. > > Were the other test kits looked at? Is gtest the best solution for the > project. > > Is this something your planning as putting in the tree, thus require > pulling in changes from google (license allowing), or does user need > to have the libraries/headers pre-installed? > > My question was not to cause a battle, but I wanted to be sure we were > using the right test kit, and not just picking one just because. For > example gtest is very light weight test kit, that can do the job, but > will the tests outgrow what the test kit can do, and cause a > conversion to a more advanced one later? > > Regards, > Mark Kromis > ------------------------------------------------------------------------ > > _______________________________________________ > LLVM Developers mailing list > LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev >