> On Jul 18, 2015, at 11:27 AM, Hal Finkel <hfinkel at anl.gov> wrote: > > 2. We don't have a good set of tests for it, nor do we have a good set of tutorials/documentation for it. Our tutorials, specifically, are in C++, not in C. We could break the C API and we'd likely remain unaware for quite awhile.I think this is the most important point, that we lack testing for it. IMO, the language doesn’t matter too much. I’m happy with C or C++, but whichever (or both) or those are exposed in a stable way, we need the *users* of those APIs to help test it. How about we add a StableAPI directory in unittests? Then have a test written in C/C++ for each of the users of it. So a WebKit.c test, Go.c, SomeProject.cpp, etc. Then adding anything to the stable API must have a corresponding test, and changing the API shows us exactly which test broke, who cares about that test, and who to talk to about updating that API if we need to. If the only test which breaks is WebKit then talk to WebKit, if its Go too then add them in, and so on. Sure the tests will get large, but thats the point. It would show us exactly what API users care about. And the tests don’t need to actually run anything, just ensure that methods signatures are compatible with what they are using. Cheers, Pete -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20150720/4dd40480/attachment.html>
On Mon, Jul 20, 2015 at 9:40 AM, Pete Cooper <peter_cooper at apple.com> wrote:> > On Jul 18, 2015, at 11:27 AM, Hal Finkel <hfinkel at anl.gov> wrote: > > 2. We don't have a good set of tests for it, nor do we have a good set of > tutorials/documentation for it. Our tutorials, specifically, are in C++, > not in C. We could break the C API and we'd likely remain unaware for quite > awhile. > > I think this is the most important point, that we lack testing for it. > > IMO, the language doesn’t matter too much. I’m happy with C or C++, but > whichever (or both) or those are exposed in a stable way, we need the > *users* of those APIs to help test it. > > How about we add a StableAPI directory in unittests? Then have a test > written in C/C++ for each of the users of it. So a WebKit.c test, Go.c, > SomeProject.cpp, etc. > > Then adding anything to the stable API must have a corresponding test, and > changing the API shows us exactly which test broke, who cares about that > test, and who to talk to about updating that API if we need to. If the > only test which breaks is WebKit then talk to WebKit, if its Go too then > add them in, and so on. > > Sure the tests will get large, but thats the point. It would show us > exactly what API users care about. And the tests don’t need to actually > run anything, just ensure that methods signatures are compatible with what > they are using. > >Part (most?) of the point of having a stable API is as a way of decoupling the development processes of two separate projects (modulo well-documented release-to-release updating). Requiring our users to add tests in our tree doesn't really achieve much decoupling. As a hyperbolic analogy: imagine if libjpeg required every user to add tests into its tree for their usage of the API. -- Sean Silva> Cheers, > Pete > > > _______________________________________________ > LLVM Developers mailing list > LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20150720/b761a710/attachment.html>
> On Jul 20, 2015, at 12:36 PM, Sean Silva <chisophugis at gmail.com> wrote: > > > > On Mon, Jul 20, 2015 at 9:40 AM, Pete Cooper <peter_cooper at apple.com <mailto:peter_cooper at apple.com>> wrote: > >> On Jul 18, 2015, at 11:27 AM, Hal Finkel <hfinkel at anl.gov <mailto:hfinkel at anl.gov>> wrote: >> >> 2. We don't have a good set of tests for it, nor do we have a good set of tutorials/documentation for it. Our tutorials, specifically, are in C++, not in C. We could break the C API and we'd likely remain unaware for quite awhile. > I think this is the most important point, that we lack testing for it. > > IMO, the language doesn’t matter too much. I’m happy with C or C++, but whichever (or both) or those are exposed in a stable way, we need the *users* of those APIs to help test it. > > How about we add a StableAPI directory in unittests? Then have a test written in C/C++ for each of the users of it. So a WebKit.c test, Go.c, SomeProject.cpp, etc. > > Then adding anything to the stable API must have a corresponding test, and changing the API shows us exactly which test broke, who cares about that test, and who to talk to about updating that API if we need to. If the only test which breaks is WebKit then talk to WebKit, if its Go too then add them in, and so on. > > Sure the tests will get large, but thats the point. It would show us exactly what API users care about. And the tests don’t need to actually run anything, just ensure that methods signatures are compatible with what they are using. > > > Part (most?) of the point of having a stable API is as a way of decoupling the development processes of two separate projects (modulo well-documented release-to-release updating). Requiring our users to add tests in our tree doesn't really achieve much decoupling.I’m not sure there is much “coupling” here. The point is that we expose a C API that is supposed to be stable but is not well tested. And some part of the C API is just a wrapper around the C++ and hasn’t really been designed to be “stable” in time. It seems also that we don’t really know what part of the C API really needs to be stable and is important for the users, so I read Pete’s proposal as “let’s collect the current use-cases and make them tests in LLVM, so that we define what is part of the stable C API and so that we won’t (inadvertently) break valid use cases". Now if you see “coupling" because of the naming of the test, they can be made in a more abstract way by naming the test “compile_multiple_module_with_mcjit.test” instead of “webkit.test”.> As a hyperbolic analogy: imagine if libjpeg required every user to add tests into its tree for their usage of the API.I believe this is already what any reasonable API would do: they would define use cases and write test. The expectation being that the client would mimic the use-case present in the various tests. As I understand it, in our case we start in a situation where the coverage of the use-cases in tests is low and we would gather our clients use-cases to narrow down what needs to be supported. As of having this in-tree or out-of-tree, I’m not sure about that and there is a trade-off. — Mehdi -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20150720/ec362bc4/attachment.html>
> On Jul 20, 2015, at 12:36 PM, Sean Silva <chisophugis at gmail.com> wrote: > > > > On Mon, Jul 20, 2015 at 9:40 AM, Pete Cooper <peter_cooper at apple.com <mailto:peter_cooper at apple.com>> wrote: > >> On Jul 18, 2015, at 11:27 AM, Hal Finkel <hfinkel at anl.gov <mailto:hfinkel at anl.gov>> wrote: >> >> 2. We don't have a good set of tests for it, nor do we have a good set of tutorials/documentation for it. Our tutorials, specifically, are in C++, not in C. We could break the C API and we'd likely remain unaware for quite awhile. > I think this is the most important point, that we lack testing for it. > > IMO, the language doesn’t matter too much. I’m happy with C or C++, but whichever (or both) or those are exposed in a stable way, we need the *users* of those APIs to help test it. > > How about we add a StableAPI directory in unittests? Then have a test written in C/C++ for each of the users of it. So a WebKit.c test, Go.c, SomeProject.cpp, etc. > > Then adding anything to the stable API must have a corresponding test, and changing the API shows us exactly which test broke, who cares about that test, and who to talk to about updating that API if we need to. If the only test which breaks is WebKit then talk to WebKit, if its Go too then add them in, and so on. > > Sure the tests will get large, but thats the point. It would show us exactly what API users care about. And the tests don’t need to actually run anything, just ensure that methods signatures are compatible with what they are using. > > > Part (most?) of the point of having a stable API is as a way of decoupling the development processes of two separate projects (modulo well-documented release-to-release updating). Requiring our users to add tests in our tree doesn't really achieve much decoupling. As a hyperbolic analogy: imagine if libjpeg required every user to add tests into its tree for their usage of the API.That is a convincing analogy. I think the same applies to a separate project. We shouldn’t expect users of the stable API to maintain it in another project. Many people don’t even build clang, even fewer will build the API project, and if C++ API becomes common, then just about any commit to LLVM could break the API project. Seems like there are a few problems here. The first is that there’s disagreement over whether the C API is currently meant to be stable. It used to be, but now maybe it is, maybe it isn’t. Then, if all of the C API is stable, we need to just be much more careful about ever changing it, and having tests of everything in it. If only a subset of the C API is stable, then we really need to better define what is stable or not. Currently I think users think its all stable. Having stable and unstable in the same headers, which is what we have now, isn’t a good idea. Finally there’s whether to have a stable C++ API. This is much tricker. If we go this route, I would personally like us to define new stable C++ API and not just say that ConstantInt for example is now stable. I think stable C++ API should be in its own namespace at the very least. Cheers Pete> > -- Sean Silva > > Cheers, > Pete > > > _______________________________________________ > LLVM Developers mailing list > LLVMdev at cs.uiuc.edu <mailto:LLVMdev at cs.uiuc.edu> http://llvm.cs.uiuc.edu <http://llvm.cs.uiuc.edu/> > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev <http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev> > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20150720/f6c276d9/attachment.html>