> On Jul 20, 2015, at 12:36 PM, Sean Silva <chisophugis at gmail.com> wrote: > > > > On Mon, Jul 20, 2015 at 9:40 AM, Pete Cooper <peter_cooper at apple.com <mailto:peter_cooper at apple.com>> wrote: > >> On Jul 18, 2015, at 11:27 AM, Hal Finkel <hfinkel at anl.gov <mailto:hfinkel at anl.gov>> wrote: >> >> 2. We don't have a good set of tests for it, nor do we have a good set of tutorials/documentation for it. Our tutorials, specifically, are in C++, not in C. We could break the C API and we'd likely remain unaware for quite awhile. > I think this is the most important point, that we lack testing for it. > > IMO, the language doesn’t matter too much. I’m happy with C or C++, but whichever (or both) or those are exposed in a stable way, we need the *users* of those APIs to help test it. > > How about we add a StableAPI directory in unittests? Then have a test written in C/C++ for each of the users of it. So a WebKit.c test, Go.c, SomeProject.cpp, etc. > > Then adding anything to the stable API must have a corresponding test, and changing the API shows us exactly which test broke, who cares about that test, and who to talk to about updating that API if we need to. If the only test which breaks is WebKit then talk to WebKit, if its Go too then add them in, and so on. > > Sure the tests will get large, but thats the point. It would show us exactly what API users care about. And the tests don’t need to actually run anything, just ensure that methods signatures are compatible with what they are using. > > > Part (most?) of the point of having a stable API is as a way of decoupling the development processes of two separate projects (modulo well-documented release-to-release updating). Requiring our users to add tests in our tree doesn't really achieve much decoupling.I’m not sure there is much “coupling” here. The point is that we expose a C API that is supposed to be stable but is not well tested. And some part of the C API is just a wrapper around the C++ and hasn’t really been designed to be “stable” in time. It seems also that we don’t really know what part of the C API really needs to be stable and is important for the users, so I read Pete’s proposal as “let’s collect the current use-cases and make them tests in LLVM, so that we define what is part of the stable C API and so that we won’t (inadvertently) break valid use cases". Now if you see “coupling" because of the naming of the test, they can be made in a more abstract way by naming the test “compile_multiple_module_with_mcjit.test” instead of “webkit.test”.> As a hyperbolic analogy: imagine if libjpeg required every user to add tests into its tree for their usage of the API.I believe this is already what any reasonable API would do: they would define use cases and write test. The expectation being that the client would mimic the use-case present in the various tests. As I understand it, in our case we start in a situation where the coverage of the use-cases in tests is low and we would gather our clients use-cases to narrow down what needs to be supported. As of having this in-tree or out-of-tree, I’m not sure about that and there is a trade-off. — Mehdi -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20150720/ec362bc4/attachment.html>
> > > Part (most?) of the point of having a stable API is as a way of decoupling > the development processes of two separate projects (modulo well-documented > release-to-release updating). Requiring our users to add tests in our tree > doesn't really achieve much decoupling. > > > I’m not sure there is much “coupling” here. The point is that we expose a > C API that is supposed to be stable but is not well tested. And some part > of the C API is just a wrapper around the C++ and hasn’t really been > designed to be “stable” in time. > It seems also that we don’t really know what part of the C API really > needs to be stable and is important for the users, so I read Pete’s > proposal as “let’s collect the current use-cases and make them tests in > LLVM, so that we define what is part of the stable C API and so that we > won’t (inadvertently) break valid use cases". > >Yeah, this is just terrible though for all of the reasons I raised in my email and as you even say here "And some part of the C API is just a wrapper around the C++ and hasn’t really been designed to be “stable” in time."> > As of having this in-tree or out-of-tree, I’m not sure about that and > there is a trade-off. > >I'm regretting ever saying "out of tree" here as I don't think it's the main issue, rather the splitting of the "bindings" style of api that we see a lot of in the C API directory and the more solid ones that we see from libclang and liblto. -eric -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20150720/06dc0fce/attachment.html>
> On Jul 20, 2015, at 1:03 PM, Mehdi Amini <mehdi.amini at apple.com> wrote: > > >> On Jul 20, 2015, at 12:36 PM, Sean Silva <chisophugis at gmail.com <mailto:chisophugis at gmail.com>> wrote: >> >> >> >> On Mon, Jul 20, 2015 at 9:40 AM, Pete Cooper <peter_cooper at apple.com <mailto:peter_cooper at apple.com>> wrote: >> >>> On Jul 18, 2015, at 11:27 AM, Hal Finkel <hfinkel at anl.gov <mailto:hfinkel at anl.gov>> wrote: >>> >>> 2. We don't have a good set of tests for it, nor do we have a good set of tutorials/documentation for it. Our tutorials, specifically, are in C++, not in C. We could break the C API and we'd likely remain unaware for quite awhile. >> I think this is the most important point, that we lack testing for it. >> >> IMO, the language doesn’t matter too much. I’m happy with C or C++, but whichever (or both) or those are exposed in a stable way, we need the *users* of those APIs to help test it. >> >> How about we add a StableAPI directory in unittests? Then have a test written in C/C++ for each of the users of it. So a WebKit.c test, Go.c, SomeProject.cpp, etc. >> >> Then adding anything to the stable API must have a corresponding test, and changing the API shows us exactly which test broke, who cares about that test, and who to talk to about updating that API if we need to. If the only test which breaks is WebKit then talk to WebKit, if its Go too then add them in, and so on. >> >> Sure the tests will get large, but thats the point. It would show us exactly what API users care about. And the tests don’t need to actually run anything, just ensure that methods signatures are compatible with what they are using. >> >> >> Part (most?) of the point of having a stable API is as a way of decoupling the development processes of two separate projects (modulo well-documented release-to-release updating). Requiring our users to add tests in our tree doesn't really achieve much decoupling. > > I’m not sure there is much “coupling” here. The point is that we expose a C API that is supposed to be stable but is not well tested. And some part of the C API is just a wrapper around the C++ and hasn’t really been designed to be “stable” in time. > It seems also that we don’t really know what part of the C API really needs to be stable and is important for the users, so I read Pete’s proposal as “let’s collect the current use-cases and make them tests in LLVM, so that we define what is part of the stable C API and so that we won’t (inadvertently) break valid use cases”.Yes, exactly. We (I don’t mind who TBH) should make tests which reflect what people are doing with the API.> > Now if you see “coupling" because of the naming of the test, they can be made in a more abstract way by naming the test “compile_multiple_module_with_mcjit.test” instead of “webkit.test”.My initial thinking for actually naming the tests is that it helps to know who the main users of an API are when we do want to change it. That way they can be involved in the conversation. If you name it webkit.c, go.c, whatever and the test breaks because of something you are trying to commit, then you can see who maintains that test and ask them if they are ok with the change. Its like choosing who to ask for commit review on a pass for example. You look at the blame list for the pass and see who is actively developing it and ask for a review. I see the API users as being like the maintainers of the API they use, in that they should be on the emails asking for review when something changes. Cheers, Pete> >> As a hyperbolic analogy: imagine if libjpeg required every user to add tests into its tree for their usage of the API. > > I believe this is already what any reasonable API would do: they would define use cases and write test. The expectation being that the client would mimic the use-case present in the various tests. As I understand it, in our case we start in a situation where the coverage of the use-cases in tests is low and we would gather our clients use-cases to narrow down what needs to be supported. > > As of having this in-tree or out-of-tree, I’m not sure about that and there is a trade-off. > > — > Mehdi >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20150720/ae6a8c56/attachment.html>
Wow, this went of topic very quickly ;-) As you know I am very interested in an stable API (C and/or C++) for LLVM, but maybe we should discuss this in a separate thread. Designing a good stable API from scratch will take some time and until that point I want to document our current “tribal knowledge”. I will post a patch as you suggested. —Juergen> On Jul 20, 2015, at 1:08 PM, Eric Christopher <echristo at gmail.com> wrote: > >> >> Part (most?) of the point of having a stable API is as a way of decoupling the development processes of two separate projects (modulo well-documented release-to-release updating). Requiring our users to add tests in our tree doesn't really achieve much decoupling. > > I’m not sure there is much “coupling” here. The point is that we expose a C API that is supposed to be stable but is not well tested. And some part of the C API is just a wrapper around the C++ and hasn’t really been designed to be “stable” in time. > It seems also that we don’t really know what part of the C API really needs to be stable and is important for the users, so I read Pete’s proposal as “let’s collect the current use-cases and make them tests in LLVM, so that we define what is part of the stable C API and so that we won’t (inadvertently) break valid use cases". > > > Yeah, this is just terrible though for all of the reasons I raised in my email and as you even say here "And some part of the C API is just a wrapper around the C++ and hasn’t really been designed to be “stable” in time." > > > As of having this in-tree or out-of-tree, I’m not sure about that and there is a trade-off. > > > I'm regretting ever saying "out of tree" here as I don't think it's the main issue, rather the splitting of the "bindings" style of api that we see a lot of in the C API directory and the more solid ones that we see from libclang and liblto. > > -eric > _______________________________________________ > LLVM Developers mailing list > LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20150720/1324fcd0/attachment.html>
Forking thread> On Jul 20, 2015, at 1:08 PM, Eric Christopher <echristo at gmail.com> wrote: > >> >> Part (most?) of the point of having a stable API is as a way of decoupling the development processes of two separate projects (modulo well-documented release-to-release updating). Requiring our users to add tests in our tree doesn't really achieve much decoupling. > > I’m not sure there is much “coupling” here. The point is that we expose a C API that is supposed to be stable but is not well tested. And some part of the C API is just a wrapper around the C++ and hasn’t really been designed to be “stable” in time. > It seems also that we don’t really know what part of the C API really needs to be stable and is important for the users, so I read Pete’s proposal as “let’s collect the current use-cases and make them tests in LLVM, so that we define what is part of the stable C API and so that we won’t (inadvertently) break valid use cases". > > > Yeah, this is just terrible though for all of the reasons I raised in my email and as you even say here "And some part of the C API is just a wrapper around the C++ and hasn’t really been designed to be “stable” in time." > > > As of having this in-tree or out-of-tree, I’m not sure about that and there is a trade-off. > > > I'm regretting ever saying "out of tree" here as I don't think it's the main issue, rather the splitting of the "bindings" style of api that we see a lot of in the C API directory and the more solid ones that we see from libclang and liblto. > > -eric > _______________________________________________ > LLVM Developers mailing list > LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20150720/70e20189/attachment.html>
> On Jul 20, 2015, at 1:08 PM, Eric Christopher <echristo at gmail.com> wrote: > >> >> Part (most?) of the point of having a stable API is as a way of decoupling the development processes of two separate projects (modulo well-documented release-to-release updating). Requiring our users to add tests in our tree doesn't really achieve much decoupling. > > I’m not sure there is much “coupling” here. The point is that we expose a C API that is supposed to be stable but is not well tested. And some part of the C API is just a wrapper around the C++ and hasn’t really been designed to be “stable” in time. > It seems also that we don’t really know what part of the C API really needs to be stable and is important for the users, so I read Pete’s proposal as “let’s collect the current use-cases and make them tests in LLVM, so that we define what is part of the stable C API and so that we won’t (inadvertently) break valid use cases". > > > Yeah, this is just terrible though for all of the reasons I raised in my email and as you even say here "And some part of the C API is just a wrapper around the C++ and hasn’t really been designed to be “stable” in time." > > > As of having this in-tree or out-of-tree, I’m not sure about that and there is a trade-off. > > > I'm regretting ever saying "out of tree" here as I don't think it's the main issue, rather the splitting of the "bindings" style of api that we see a lot of in the C API directory and the more solid ones that we see from libclang and liblto.I agree with this. There’s a lot of muddy waters surrounding the C API right now. “Stable API” vs. “bindings” is a great clarifying question. Out-of-tree or not doesn’t matter too much to me, personally. It was a useful turn-of-phrase to jump-start this aspect of the conversation, but pretty much all of the deeper questions apply equally regardless of which implementation is chosen. I think you, and others who’ve observed the same basic thing, have hit it on the head that our current C API has morphed in definition over time from a very small, restricted API for very specific use-cases to something broader and more general purpose. It’s done so in a gradual and mostly organic way along the lines of “Hey, I need this widget and can’t get to it from C, so I’ll add an API to get to it.” As a consequence, it’s also morphed from a very (intentionally) stable API to something a bit more fluid. Exactly how fluid is unclear, thus the various breakages, discussions about how to deal with them, etc.. We started with something that was very much a stable API, but now we have some sort of weird mix of API and bindings, with no clear delineation of which is which or even which is intended to be which. I see value in having C APIs for both classes of use cases, but they should be separated out from one another. I strongly agree that especially for the stable API, we need very robust test cases which test not only that the API is maintained from release to release, but that we have a well defined, well documented and well tested set of semantics for that API. Basically, we need to stop doing this by half measures and actually have a real stable API. To get there, we’ll need to have two sets of headers, one for bindings and one for APIs. I suggest we start by cloning what exists today for both, with the bindings API moving to a new location. Then the bindings side can start changing (or not) as much or as little as those consumers want without muddying the stable API waters. On the stable API side, we can work on a strategy for deprecating those APIs we currently have that aren’t a great fit for a stable API and replacing them with new ones (if appropriate). -Jim -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20150720/24565095/attachment.html>