David Blaikie via llvm-dev
2016-Feb-18 18:15 UTC
[llvm-dev] RFC: Add bitcode tests to test-suite
On Thu, Feb 18, 2016 at 10:03 AM, Mehdi Amini via llvm-dev < llvm-dev at lists.llvm.org> wrote:> > > On Feb 18, 2016, at 8:42 AM, Philip Reames via llvm-dev < > llvm-dev at lists.llvm.org> wrote: > > > > On 02/18/2016 06:54 AM, Hal Finkel via llvm-dev wrote: > >> Hi Chandler, et al., > >> > >> While this proposal to put IR into the test suite technically > non-problematic, I've convinced myself that this is a suboptimal direction > for the LLVM project. Here's what I think would be better: > >> > >> - We create a test-suite/Frontends directory, and open this directory > to actively-maintained external frontends, subject to the following > restrictions: > >> > >> - The frontend must be actively maintained, and the project must > agree to actively maintain the test-suite version > >> - The frontend must use the LLVM API (either C or C++) - no printing > textual IR > >> - The frontend must have no significant (non-optional) dependencies > outside of LLVM itself, or things on which LLVM itself depends > >> - The frontend must have regression tests and benchmarks/correctness > tests providing significant coverage of the frontend and its associated > code generation > >> > >> Here's the quid pro quo: > >> > >> - The LLVM community gains additional testing coverage (which we > definitely need) > >> - The LLVM community gains extra insight into how its APIs are being > used (hopefully allowing us to make more-informed decisions about how to > update them) > >> > >> - The frontend gains free API updates > >> - The frontend's use of LLVM will be more stable > >> > >> This involves extra work for everybody, but will help us all deliver > higher-quality products. Plus, given the constant discussions about the > difficulty for external projects to follow API updates, etc., this is a > good way to help address those difficulties. > >> > >> The fact that Halide will provide extra coverage of our vector code > generation (aside from whatever we happen to produce from our > autovectorizers), and our JIT infrastructure, makes it a good candidate for > this. Intel's ispc, POCL, (maybe whatever bit of Mesa uses LLVM), etc. > would also be natural candidates should the projects be interested. > > I think this is a really bad tradeoff and am strongly opposed to this > proposal. > > > > If we want to focus on improving test coverage, that's reasonable, but > doing so at the cost of requiring LLVM contributors to maintain everyone's > frontend is not a reasonable approach. > > > > A couple of alternate approaches which might be worth considering: > > 1) The IR corpus approach mentioned previously. So long as external > teams are willing to update the corpus regularly (weekly), this gives most > of the backend coverage with none of the maintenance burden. > > 2) Use coverage information to determine which code paths Halide covers > which are not covered by existing unit tests. Work to improve those unit > tests. Using something along the lines with a mutation testing (i.e. > change the source code and see what breaks), combined with test reduction > (bugpoint), could greatly improve our test coverage in tree fairly > quickly. This would require a lot of work from a single contributor, but > that's much better than requiring a lot of work from all contributors. > > I support the view of Philip on this topic. > I think that having the bitcode without the burden of the API and linking > is the best trade-off for LLVM. > > About the bitcode, I'm not convince that it needs to be updated that > often: we're interested in coverage for LLVM, not trying to validate that > we support correctly frontend X or Y in a particular version. I'd even > argue that if Halide version 13 generates a very different IR than Halide > 12, then we should keep the Halide 12 generated bitcode in a separate > directory because it is likely to stress LLVM differently. >Potentially, though I hesitate to make test-suite too much of a dumping ground (it is already, to an extent). Quantity isn't quality and all that - making sure bots cycle quickly (in a relative sense - of course running the test-suite is going to be a more involved process than our high priority regression 'unit'-y tests) is important for them to be meaningful/actionable by developers & not just noise.> > I have more questions for Alina. What kind of tests do you have: > > - "the compiler takes the bitcode and generates code without crashing" > - "the compiled test runs without crashing" > - "the compiled test will produce an output that be checked against a > reference" > - "the compiled test is meaningful as a benchmarks" > > All these different aspects of testing can be interesting, but knowing > what we're talking may influence the way forward. > > As mentioned before, the test-suite has a mechanism of "external" suites, > it may be limited right now but could probably be expanded. Ideally there > could be multiple repositories that would just be checked out independently > (think about the way we build LLVM alone or > LLVM+clang+libcxx+compiler-rt+...). > > -- > Mehdi > > > >> > >> Thanks again, > >> Hal > >> > >> ----- Original Message ----- > >>> From: "Chandler Carruth" <chandlerc at google.com> > >>> To: "Hal Finkel" <hfinkel at anl.gov>, "Alina Sbirlea" < > alina.sbirlea at gmail.com> > >>> Cc: "llvm-dev" <llvm-dev at lists.llvm.org> > >>> Sent: Wednesday, February 17, 2016 9:34:24 PM > >>> Subject: Re: [llvm-dev] RFC: Add bitcode tests to test-suite > >>> > >>> > >>> Some perhaps relevant aspects that make testing users of LLVM like > >>> Halide challenging: > >>> > >>> > >>> Halide uses the LLVM C++ APIs, but there isn't a good way to > >>> lock-step update it. So if we were to directly test Halide, it > >>> wouldn't link against the new LLVM. > >>> > >>> > >>> Practically speaking though, the LLVM IR generated by Halide should > >>> continue to work with newer LLVM optimizations and code generation. > >>> So the idea would be to snapshot the IR in bitcode (which is at > >>> least reasonably stable) so that we could replay the tests as LLVM > >>> changes. We can freshen the bitcode by re-generating it periodically > >>> so it doesn't drift too far from what Halide actually uses. > >>> > >>> > >>> The interesting questions IMO are: > >>> > >>> > >>> 1) Are folks happy using bitcode as the format here? I agree with Hal > >>> that it should be easy since Clang will actually Do The Right Thing > >>> if given a bitcode input. > >>> > >>> > >>> 2) Are folks happy with non-execution tests in some cases? I think > >>> Alina is looking at whether we can get a runtime library that will > >>> allow some of these to actually execute, but at least some of the > >>> tests are just snap-shots of a JIT, and would need the full Halide > >>> libraries (and introspection) to execute usefully. > >>> > >>> > >>> -Chandler > >>> > >>> > >>> On Wed, Feb 17, 2016 at 7:25 PM Hal Finkel via llvm-dev < > >>> llvm-dev at lists.llvm.org > wrote: > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> From: "Alina Sbirlea via llvm-dev" < llvm-dev at lists.llvm.org > > >>> To: "llvm-dev" < llvm-dev at lists.llvm.org > > >>> Sent: Wednesday, February 17, 2016 7:25:17 PM > >>> Subject: [llvm-dev] RFC: Add bitcode tests to test-suite > >>> > >>> > >>> > >>> > >>> > >>> > >>> Hi all, > >>> > >>> > >>> TL;DR: Add *.bc to test-suite; llc *.bc; run some. > >>> > >>> > >>> > >>> We would like to propose adding bitcode tests to the llvm test-suite. > >>> > >>> > >>> Recent LLVM bugs [2-4] prompted us to look into upstreaming a subset > >>> of the tests the Halide library [1] is running and we'd like the > >>> community's feedback on moving forward with this. > >>> > >>> > >>> > >>> Halide uses LLVM and can generate bitcode, but we cannot add C++ > >>> tests to test-suite without including the library itself. > >>> This proposal is also potentially useful for other cases where there > >>> is no C++ front-end. > >>> > >>> > >>> As a first step we are interested in adding a set of correctness > >>> tests, for testing the IR without running the tests. Since these > >>> tests are generated, they are not instrumented like the .ll files in > >>> trunk, however we believe checking that llc runs without errors is > >>> still useful. > >>> The bitcode files for Halide may also be large, so including them as > >>> regression tests is not an option. If the smaller tests are found to > >>> be valuable or covering cases no other tests cover, we can > >>> instrument them and move them into the llvm trunk further along, but > >>> that is not the goal of this proposal. > >>> In addition, we're not sure whether the format for the tests should > >>> be .ll or .bc, we're open to either. > >>> > >>> > >>> After this first step, we're interested in upstreaming bitcode tests > >>> and also running them. > >>> We are very interested in tests for multiple architectures, aarch64 > >>> in particular, since this is where we have seen things break. This > >>> may motivate adding .ll files rather than .bc in order to include > >>> the "RUN:" target. > >>> Where would these tests reside and with what directory structure? > >>> (similar to test/CodeGen?) > >>> > >>> > >>> Suggestion on what's the best approach for extending the test-suite > >>> framework for this proposal are more than welcome. > >>> > >>> > >>> > >>> We already have architecture-specific tests in the test suite (e.g. > >>> SingleSource/UnitTests/Vector/{SSE,Altivec,etc.}, and Clang can deal > >>> with IR inputs. I suppose you need to compile some corresponding > >>> runtime library, but this does not seem like a big deal either. > >>> Mechanically, I don't see this as particularly complicated. I think > >>> the real question is: Is this the best way to have a kind of 'halide > >>> buildbot' that can inform the LLVM developer community? > >>> > >>> -Hal > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> This is just the high-level overview to start off the discussion, I'm > >>> sure there are many more aspects to touch on. Looking forward to > >>> your feedback! > >>> > >>> > >>> > >>> Thanks, > >>> Alina > >>> > >>> > >>> [1] http:// halide -lang.org/ > >>> [2] Broken: r259800 => Fixed: r260131 > >>> [3] Broken: r260569 => Fixed: r260701 > >>> > >>> [4] https://llvm.org/bugs/show_bug.cgi?id=26642 > >>> > >>> > >>> > >>> > >>> _______________________________________________ > >>> LLVM Developers mailing list > >>> llvm-dev at lists.llvm.org > >>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev > >>> > >>> > >>> > >>> > >>> > >>> -- > >>> > >>> Hal Finkel > >>> Assistant Computational Scientist > >>> Leadership Computing Facility > >>> Argonne National Laboratory > >>> _______________________________________________ > >>> LLVM Developers mailing list > >>> llvm-dev at lists.llvm.org > >>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev > >>> > > > > _______________________________________________ > > LLVM Developers mailing list > > llvm-dev at lists.llvm.org > > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev > > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160218/8fe9204d/attachment.html>
Mehdi Amini via llvm-dev
2016-Feb-18 18:22 UTC
[llvm-dev] RFC: Add bitcode tests to test-suite
> On Feb 18, 2016, at 10:15 AM, David Blaikie <dblaikie at gmail.com> wrote: > > > > On Thu, Feb 18, 2016 at 10:03 AM, Mehdi Amini via llvm-dev <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote: > > > On Feb 18, 2016, at 8:42 AM, Philip Reames via llvm-dev <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote: > > > > On 02/18/2016 06:54 AM, Hal Finkel via llvm-dev wrote: > >> Hi Chandler, et al., > >> > >> While this proposal to put IR into the test suite technically non-problematic, I've convinced myself that this is a suboptimal direction for the LLVM project. Here's what I think would be better: > >> > >> - We create a test-suite/Frontends directory, and open this directory to actively-maintained external frontends, subject to the following restrictions: > >> > >> - The frontend must be actively maintained, and the project must agree to actively maintain the test-suite version > >> - The frontend must use the LLVM API (either C or C++) - no printing textual IR > >> - The frontend must have no significant (non-optional) dependencies outside of LLVM itself, or things on which LLVM itself depends > >> - The frontend must have regression tests and benchmarks/correctness tests providing significant coverage of the frontend and its associated code generation > >> > >> Here's the quid pro quo: > >> > >> - The LLVM community gains additional testing coverage (which we definitely need) > >> - The LLVM community gains extra insight into how its APIs are being used (hopefully allowing us to make more-informed decisions about how to update them) > >> > >> - The frontend gains free API updates > >> - The frontend's use of LLVM will be more stable > >> > >> This involves extra work for everybody, but will help us all deliver higher-quality products. Plus, given the constant discussions about the difficulty for external projects to follow API updates, etc., this is a good way to help address those difficulties. > >> > >> The fact that Halide will provide extra coverage of our vector code generation (aside from whatever we happen to produce from our autovectorizers), and our JIT infrastructure, makes it a good candidate for this. Intel's ispc, POCL, (maybe whatever bit of Mesa uses LLVM), etc. would also be natural candidates should the projects be interested. > > I think this is a really bad tradeoff and am strongly opposed to this proposal. > > > > If we want to focus on improving test coverage, that's reasonable, but doing so at the cost of requiring LLVM contributors to maintain everyone's frontend is not a reasonable approach. > > > > A couple of alternate approaches which might be worth considering: > > 1) The IR corpus approach mentioned previously. So long as external teams are willing to update the corpus regularly (weekly), this gives most of the backend coverage with none of the maintenance burden. > > 2) Use coverage information to determine which code paths Halide covers which are not covered by existing unit tests. Work to improve those unit tests. Using something along the lines with a mutation testing (i.e. change the source code and see what breaks), combined with test reduction (bugpoint), could greatly improve our test coverage in tree fairly quickly. This would require a lot of work from a single contributor, but that's much better than requiring a lot of work from all contributors. > > I support the view of Philip on this topic. > I think that having the bitcode without the burden of the API and linking is the best trade-off for LLVM. > > About the bitcode, I'm not convince that it needs to be updated that often: we're interested in coverage for LLVM, not trying to validate that we support correctly frontend X or Y in a particular version. I'd even argue that if Halide version 13 generates a very different IR than Halide 12, then we should keep the Halide 12 generated bitcode in a separate directory because it is likely to stress LLVM differently. > > Potentially, though I hesitate to make test-suite too much of a dumping ground (it is already, to an extent). Quantity isn't quality and all that - making sure bots cycle quickly (in a relative sense - of course running the test-suite is going to be a more involved process than our high priority regression 'unit'-y tests) is important for them to be meaningful/actionable by developers & not just noise.Yeah this is why I stressed the "modular" aspect of the test-suite: I'd like this to be a *separate* repository that people (and bot owners) who want it can get it very easily through a unified test-suite infrastructure. Anyone could put any "external" test-suite on github and try to get it used. (Like if someone was to publish an "external" test-suite with a lot of cryptographic algorithm that could be benchmarked as part of the llvm test-suite infrastructure). -- Mehdi> > > I have more questions for Alina. What kind of tests do you have: > > - "the compiler takes the bitcode and generates code without crashing" > - "the compiled test runs without crashing" > - "the compiled test will produce an output that be checked against a reference" > - "the compiled test is meaningful as a benchmarks" > > All these different aspects of testing can be interesting, but knowing what we're talking may influence the way forward. > > As mentioned before, the test-suite has a mechanism of "external" suites, it may be limited right now but could probably be expanded. Ideally there could be multiple repositories that would just be checked out independently (think about the way we build LLVM alone or LLVM+clang+libcxx+compiler-rt+...). > > -- > Mehdi > > > >> > >> Thanks again, > >> Hal > >> > >> ----- Original Message ----- > >>> From: "Chandler Carruth" <chandlerc at google.com <mailto:chandlerc at google.com>> > >>> To: "Hal Finkel" <hfinkel at anl.gov <mailto:hfinkel at anl.gov>>, "Alina Sbirlea" <alina.sbirlea at gmail.com <mailto:alina.sbirlea at gmail.com>> > >>> Cc: "llvm-dev" <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> > >>> Sent: Wednesday, February 17, 2016 9:34:24 PM > >>> Subject: Re: [llvm-dev] RFC: Add bitcode tests to test-suite > >>> > >>> > >>> Some perhaps relevant aspects that make testing users of LLVM like > >>> Halide challenging: > >>> > >>> > >>> Halide uses the LLVM C++ APIs, but there isn't a good way to > >>> lock-step update it. So if we were to directly test Halide, it > >>> wouldn't link against the new LLVM. > >>> > >>> > >>> Practically speaking though, the LLVM IR generated by Halide should > >>> continue to work with newer LLVM optimizations and code generation. > >>> So the idea would be to snapshot the IR in bitcode (which is at > >>> least reasonably stable) so that we could replay the tests as LLVM > >>> changes. We can freshen the bitcode by re-generating it periodically > >>> so it doesn't drift too far from what Halide actually uses. > >>> > >>> > >>> The interesting questions IMO are: > >>> > >>> > >>> 1) Are folks happy using bitcode as the format here? I agree with Hal > >>> that it should be easy since Clang will actually Do The Right Thing > >>> if given a bitcode input. > >>> > >>> > >>> 2) Are folks happy with non-execution tests in some cases? I think > >>> Alina is looking at whether we can get a runtime library that will > >>> allow some of these to actually execute, but at least some of the > >>> tests are just snap-shots of a JIT, and would need the full Halide > >>> libraries (and introspection) to execute usefully. > >>> > >>> > >>> -Chandler > >>> > >>> > >>> On Wed, Feb 17, 2016 at 7:25 PM Hal Finkel via llvm-dev < > >>> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org> > wrote: > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> From: "Alina Sbirlea via llvm-dev" < llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org> > > >>> To: "llvm-dev" < llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org> > > >>> Sent: Wednesday, February 17, 2016 7:25:17 PM > >>> Subject: [llvm-dev] RFC: Add bitcode tests to test-suite > >>> > >>> > >>> > >>> > >>> > >>> > >>> Hi all, > >>> > >>> > >>> TL;DR: Add *.bc to test-suite; llc *.bc; run some. > >>> > >>> > >>> > >>> We would like to propose adding bitcode tests to the llvm test-suite. > >>> > >>> > >>> Recent LLVM bugs [2-4] prompted us to look into upstreaming a subset > >>> of the tests the Halide library [1] is running and we'd like the > >>> community's feedback on moving forward with this. > >>> > >>> > >>> > >>> Halide uses LLVM and can generate bitcode, but we cannot add C++ > >>> tests to test-suite without including the library itself. > >>> This proposal is also potentially useful for other cases where there > >>> is no C++ front-end. > >>> > >>> > >>> As a first step we are interested in adding a set of correctness > >>> tests, for testing the IR without running the tests. Since these > >>> tests are generated, they are not instrumented like the .ll files in > >>> trunk, however we believe checking that llc runs without errors is > >>> still useful. > >>> The bitcode files for Halide may also be large, so including them as > >>> regression tests is not an option. If the smaller tests are found to > >>> be valuable or covering cases no other tests cover, we can > >>> instrument them and move them into the llvm trunk further along, but > >>> that is not the goal of this proposal. > >>> In addition, we're not sure whether the format for the tests should > >>> be .ll or .bc, we're open to either. > >>> > >>> > >>> After this first step, we're interested in upstreaming bitcode tests > >>> and also running them. > >>> We are very interested in tests for multiple architectures, aarch64 > >>> in particular, since this is where we have seen things break. This > >>> may motivate adding .ll files rather than .bc in order to include > >>> the "RUN:" target. > >>> Where would these tests reside and with what directory structure? > >>> (similar to test/CodeGen?) > >>> > >>> > >>> Suggestion on what's the best approach for extending the test-suite > >>> framework for this proposal are more than welcome. > >>> > >>> > >>> > >>> We already have architecture-specific tests in the test suite (e.g. > >>> SingleSource/UnitTests/Vector/{SSE,Altivec,etc.}, and Clang can deal > >>> with IR inputs. I suppose you need to compile some corresponding > >>> runtime library, but this does not seem like a big deal either. > >>> Mechanically, I don't see this as particularly complicated. I think > >>> the real question is: Is this the best way to have a kind of 'halide > >>> buildbot' that can inform the LLVM developer community? > >>> > >>> -Hal > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> This is just the high-level overview to start off the discussion, I'm > >>> sure there are many more aspects to touch on. Looking forward to > >>> your feedback! > >>> > >>> > >>> > >>> Thanks, > >>> Alina > >>> > >>> > >>> [1] http:// halide -lang.org/ <http://lang.org/> > >>> [2] Broken: r259800 => Fixed: r260131 > >>> [3] Broken: r260569 => Fixed: r260701 > >>> > >>> [4] https://llvm.org/bugs/show_bug.cgi?id=26642 <https://llvm.org/bugs/show_bug.cgi?id=26642> > >>> > >>> > >>> > >>> > >>> _______________________________________________ > >>> LLVM Developers mailing list > >>> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org> > >>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev <http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev> > >>> > >>> > >>> > >>> > >>> > >>> -- > >>> > >>> Hal Finkel > >>> Assistant Computational Scientist > >>> Leadership Computing Facility > >>> Argonne National Laboratory > >>> _______________________________________________ > >>> LLVM Developers mailing list > >>> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org> > >>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev <http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev> > >>> > > > > _______________________________________________ > > LLVM Developers mailing list > > llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org> > > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev <http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev> > > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org> > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev <http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160218/0dabb513/attachment.html>
Hal Finkel via llvm-dev
2016-Feb-18 18:30 UTC
[llvm-dev] RFC: Add bitcode tests to test-suite
----- Original Message -----> From: "Mehdi Amini via llvm-dev" <llvm-dev at lists.llvm.org> > To: "David Blaikie" <dblaikie at gmail.com> > Cc: "llvm-dev" <llvm-dev at lists.llvm.org> > Sent: Thursday, February 18, 2016 12:22:59 PM > Subject: Re: [llvm-dev] RFC: Add bitcode tests to test-suite> > On Feb 18, 2016, at 10:15 AM, David Blaikie < dblaikie at gmail.com > > > wrote: >> > On Thu, Feb 18, 2016 at 10:03 AM, Mehdi Amini via llvm-dev < > > llvm-dev at lists.llvm.org > wrote: >> > > > On Feb 18, 2016, at 8:42 AM, Philip Reames via llvm-dev < > > > > llvm-dev at lists.llvm.org > wrote: > > > > > > > > > > > > > > On 02/18/2016 06:54 AM, Hal Finkel via llvm-dev wrote: > > > > > > >> Hi Chandler, et al., > > > > > > >> > > > > > > >> While this proposal to put IR into the test suite technically > > > >> non-problematic, I've convinced myself that this is a > > > >> suboptimal > > > >> direction for the LLVM project. Here's what I think would be > > > >> better: > > > > > > >> > > > > > > >> - We create a test-suite/Frontends directory, and open this > > > >> directory to actively-maintained external frontends, subject > > > >> to > > > >> the following restrictions: > > > > > > >> > > > > > > >> - The frontend must be actively maintained, and the project > > > >> must > > > >> agree to actively maintain the test-suite version > > > > > > >> - The frontend must use the LLVM API (either C or C++) - no > > > >> printing textual IR > > > > > > >> - The frontend must have no significant (non-optional) > > > >> dependencies outside of LLVM itself, or things on which LLVM > > > >> itself depends > > > > > > >> - The frontend must have regression tests and > > > >> benchmarks/correctness tests providing significant coverage of > > > >> the frontend and its associated code generation > > > > > > >> > > > > > > >> Here's the quid pro quo: > > > > > > >> > > > > > > >> - The LLVM community gains additional testing coverage (which > > > >> we > > > >> definitely need) > > > > > > >> - The LLVM community gains extra insight into how its APIs are > > > >> being used (hopefully allowing us to make more-informed > > > >> decisions > > > >> about how to update them) > > > > > > >> > > > > > > >> - The frontend gains free API updates > > > > > > >> - The frontend's use of LLVM will be more stable > > > > > > >> > > > > > > >> This involves extra work for everybody, but will help us all > > > >> deliver higher-quality products. Plus, given the constant > > > >> discussions about the difficulty for external projects to > > > >> follow > > > >> API updates, etc., this is a good way to help address those > > > >> difficulties. > > > > > > >> > > > > > > >> The fact that Halide will provide extra coverage of our vector > > > >> code generation (aside from whatever we happen to produce from > > > >> our autovectorizers), and our JIT infrastructure, makes it a > > > >> good > > > >> candidate for this. Intel's ispc, POCL, (maybe whatever bit of > > > >> Mesa uses LLVM), etc. would also be natural candidates should > > > >> the > > > >> projects be interested. > > > > > > > I think this is a really bad tradeoff and am strongly opposed > > > > to > > > > this proposal. > > > > > > > > > > > > > > If we want to focus on improving test coverage, that's > > > > reasonable, > > > > but doing so at the cost of requiring LLVM contributors to > > > > maintain everyone's frontend is not a reasonable approach. > > > > > > > > > > > > > > A couple of alternate approaches which might be worth > > > > considering: > > > > > > > 1) The IR corpus approach mentioned previously. So long as > > > > external > > > > teams are willing to update the corpus regularly (weekly), this > > > > gives most of the backend coverage with none of the maintenance > > > > burden. > > > > > > > 2) Use coverage information to determine which code paths > > > > Halide > > > > covers which are not covered by existing unit tests. Work to > > > > improve those unit tests. Using something along the lines with > > > > a > > > > mutation testing (i.e. change the source code and see what > > > > breaks), combined with test reduction (bugpoint), could greatly > > > > improve our test coverage in tree fairly quickly. This would > > > > require a lot of work from a single contributor, but that's > > > > much > > > > better than requiring a lot of work from all contributors. > > >> > > I support the view of Philip on this topic. > > > > > > I think that having the bitcode without the burden of the API and > > > linking is the best trade-off for LLVM. > > >> > > About the bitcode, I'm not convince that it needs to be updated > > > that > > > often: we're interested in coverage for LLVM, not trying to > > > validate > > > that we support correctly frontend X or Y in a particular > > > version. > > > I'd even argue that if Halide version 13 generates a very > > > different > > > IR than Halide 12, then we should keep the Halide 12 generated > > > bitcode in a separate directory because it is likely to stress > > > LLVM > > > differently. > > >> > Potentially, though I hesitate to make test-suite too much of a > > dumping ground (it is already, to an extent). Quantity isn't > > quality > > and all that - making sure bots cycle quickly (in a relative sense > > - > > of course running the test-suite is going to be a more involved > > process than our high priority regression 'unit'-y tests) is > > important for them to be meaningful/actionable by developers & not > > just noise. >> Yeah this is why I stressed the "modular" aspect of the test-suite: > I'd like this to be a *separate* repository that people (and bot > owners) who want it can get it very easily through a unified > test-suite infrastructure. Anyone could put any "external" > test-suite on github and try to get it used. (Like if someone was to > publish an "external" test-suite with a lot of cryptographic > algorithm that could be benchmarked as part of the llvm test-suite > infrastructure).Regardless of anything else, I support this idea. The reason why I think it is better for these things to be in our own repositories is that most (although not all) API updates are trivial, and it is better for us to make them ourselves as opposed to having to disable some external test-suite "module" while we await external developer action. -Hal> -- > Mehdi> > > I have more questions for Alina. What kind of tests do you have: > > >> > > - "the compiler takes the bitcode and generates code without > > > crashing" > > > > > > - "the compiled test runs without crashing" > > > > > > - "the compiled test will produce an output that be checked > > > against > > > a > > > reference" > > > > > > - "the compiled test is meaningful as a benchmarks" > > >> > > All these different aspects of testing can be interesting, but > > > knowing what we're talking may influence the way forward. > > >> > > As mentioned before, the test-suite has a mechanism of "external" > > > suites, it may be limited right now but could probably be > > > expanded. > > > Ideally there could be multiple repositories that would just be > > > checked out independently (think about the way we build LLVM > > > alone > > > or LLVM+clang+libcxx+compiler-rt+...). > > >> > > -- > > > > > > Mehdi > > >> > > >> > > > > > > >> Thanks again, > > > > > > >> Hal > > > > > > >> > > > > > > >> ----- Original Message ----- > > >> > > >>> From: "Chandler Carruth" < chandlerc at google.com > > > > > > > >>> To: "Hal Finkel" < hfinkel at anl.gov >, "Alina Sbirlea" < > > > >>> alina.sbirlea at gmail.com > > > > > > > >>> Cc: "llvm-dev" < llvm-dev at lists.llvm.org > > > > > > > >>> Sent: Wednesday, February 17, 2016 9:34:24 PM > > > > > > >>> Subject: Re: [llvm-dev] RFC: Add bitcode tests to test-suite > > > > > > >>> > > > > > > >>> > > > > > > >>> Some perhaps relevant aspects that make testing users of LLVM > > > >>> like > > > > > > >>> Halide challenging: > > > > > > >>> > > > > > > >>> > > > > > > >>> Halide uses the LLVM C++ APIs, but there isn't a good way to > > > > > > >>> lock-step update it. So if we were to directly test Halide, > > > >>> it > > > > > > >>> wouldn't link against the new LLVM. > > > > > > >>> > > > > > > >>> > > > > > > >>> Practically speaking though, the LLVM IR generated by Halide > > > >>> should > > > > > > >>> continue to work with newer LLVM optimizations and code > > > >>> generation. > > > > > > >>> So the idea would be to snapshot the IR in bitcode (which is > > > >>> at > > > > > > >>> least reasonably stable) so that we could replay the tests as > > > >>> LLVM > > > > > > >>> changes. We can freshen the bitcode by re-generating it > > > >>> periodically > > > > > > >>> so it doesn't drift too far from what Halide actually uses. > > > > > > >>> > > > > > > >>> > > > > > > >>> The interesting questions IMO are: > > > > > > >>> > > > > > > >>> > > > > > > >>> 1) Are folks happy using bitcode as the format here? I agree > > > >>> with > > > >>> Hal > > > > > > >>> that it should be easy since Clang will actually Do The Right > > > >>> Thing > > > > > > >>> if given a bitcode input. > > > > > > >>> > > > > > > >>> > > > > > > >>> 2) Are folks happy with non-execution tests in some cases? I > > > >>> think > > > > > > >>> Alina is looking at whether we can get a runtime library that > > > >>> will > > > > > > >>> allow some of these to actually execute, but at least some of > > > >>> the > > > > > > >>> tests are just snap-shots of a JIT, and would need the full > > > >>> Halide > > > > > > >>> libraries (and introspection) to execute usefully. > > > > > > >>> > > > > > > >>> > > > > > > >>> -Chandler > > > > > > >>> > > > > > > >>> > > > > > > >>> On Wed, Feb 17, 2016 at 7:25 PM Hal Finkel via llvm-dev < > > > > > > >>> llvm-dev at lists.llvm.org > wrote: > > > > > > >>> > > > > > > >>> > > > > > > >>> > > > > > > >>> > > > > > > >>> > > > > > > >>> > > > > > > >>> > > > > > > >>> > > > > > > >>> From: "Alina Sbirlea via llvm-dev" < llvm-dev at lists.llvm.org > > > >>> > > > > > > > >>> To: "llvm-dev" < llvm-dev at lists.llvm.org > > > > > > > >>> Sent: Wednesday, February 17, 2016 7:25:17 PM > > > > > > >>> Subject: [llvm-dev] RFC: Add bitcode tests to test-suite > > > > > > >>> > > > > > > >>> > > > > > > >>> > > > > > > >>> > > > > > > >>> > > > > > > >>> > > > > > > >>> Hi all, > > > > > > >>> > > > > > > >>> > > > > > > >>> TL;DR: Add *.bc to test-suite; llc *.bc; run some. > > > > > > >>> > > > > > > >>> > > > > > > >>> > > > > > > >>> We would like to propose adding bitcode tests to the llvm > > > >>> test-suite. > > > > > > >>> > > > > > > >>> > > > > > > >>> Recent LLVM bugs [2-4] prompted us to look into upstreaming a > > > >>> subset > > > > > > >>> of the tests the Halide library [1] is running and we'd like > > > >>> the > > > > > > >>> community's feedback on moving forward with this. > > > > > > >>> > > > > > > >>> > > > > > > >>> > > > > > > >>> Halide uses LLVM and can generate bitcode, but we cannot add > > > >>> C++ > > > > > > >>> tests to test-suite without including the library itself. > > > > > > >>> This proposal is also potentially useful for other cases > > > >>> where > > > >>> there > > > > > > >>> is no C++ front-end. > > > > > > >>> > > > > > > >>> > > > > > > >>> As a first step we are interested in adding a set of > > > >>> correctness > > > > > > >>> tests, for testing the IR without running the tests. Since > > > >>> these > > > > > > >>> tests are generated, they are not instrumented like the .ll > > > >>> files > > > >>> in > > > > > > >>> trunk, however we believe checking that llc runs without > > > >>> errors > > > >>> is > > > > > > >>> still useful. > > > > > > >>> The bitcode files for Halide may also be large, so including > > > >>> them > > > >>> as > > > > > > >>> regression tests is not an option. If the smaller tests are > > > >>> found > > > >>> to > > > > > > >>> be valuable or covering cases no other tests cover, we can > > > > > > >>> instrument them and move them into the llvm trunk further > > > >>> along, > > > >>> but > > > > > > >>> that is not the goal of this proposal. > > > > > > >>> In addition, we're not sure whether the format for the tests > > > >>> should > > > > > > >>> be .ll or .bc, we're open to either. > > > > > > >>> > > > > > > >>> > > > > > > >>> After this first step, we're interested in upstreaming > > > >>> bitcode > > > >>> tests > > > > > > >>> and also running them. > > > > > > >>> We are very interested in tests for multiple architectures, > > > >>> aarch64 > > > > > > >>> in particular, since this is where we have seen things break. > > > >>> This > > > > > > >>> may motivate adding .ll files rather than .bc in order to > > > >>> include > > > > > > >>> the "RUN:" target. > > > > > > >>> Where would these tests reside and with what directory > > > >>> structure? > > > > > > >>> (similar to test/CodeGen?) > > > > > > >>> > > > > > > >>> > > > > > > >>> Suggestion on what's the best approach for extending the > > > >>> test-suite > > > > > > >>> framework for this proposal are more than welcome. > > > > > > >>> > > > > > > >>> > > > > > > >>> > > > > > > >>> We already have architecture-specific tests in the test suite > > > >>> (e.g. > > > > > > >>> SingleSource/UnitTests/Vector/{SSE,Altivec,etc.}, and Clang > > > >>> can > > > >>> deal > > > > > > >>> with IR inputs. I suppose you need to compile some > > > >>> corresponding > > > > > > >>> runtime library, but this does not seem like a big deal > > > >>> either. > > > > > > >>> Mechanically, I don't see this as particularly complicated. I > > > >>> think > > > > > > >>> the real question is: Is this the best way to have a kind of > > > >>> 'halide > > > > > > >>> buildbot' that can inform the LLVM developer community? > > > > > > >>> > > > > > > >>> -Hal > > > > > > >>> > > > > > > >>> > > > > > > >>> > > > > > > >>> > > > > > > >>> > > > > > > >>> > > > > > > >>> > > > > > > >>> > > > > > > >>> > > > > > > >>> > > > > > > >>> This is just the high-level overview to start off the > > > >>> discussion, > > > >>> I'm > > > > > > >>> sure there are many more aspects to touch on. Looking forward > > > >>> to > > > > > > >>> your feedback! > > > > > > >>> > > > > > > >>> > > > > > > >>> > > > > > > >>> Thanks, > > > > > > >>> Alina > > > > > > >>> > > > > > > >>> > > > > > > >>> [1] http:// halide - lang.org/ > > > > > > >>> [2] Broken: r259800 => Fixed: r260131 > > > > > > >>> [3] Broken: r260569 => Fixed: r260701 > > > > > > >>> > > > > > > >>> [4] https://llvm.org/bugs/show_bug.cgi?id=26642 > > > > > > >>> > > > > > > >>> > > > > > > >>> > > > > > > >>> > > > > > > >>> _______________________________________________ > > > > > > >>> LLVM Developers mailing list > > > > > > >>> llvm-dev at lists.llvm.org > > > > > > >>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev > > > > > > >>> > > > > > > >>> > > > > > > >>> > > > > > > >>> > > > > > > >>> > > > > > > >>> -- > > > > > > >>> > > > > > > >>> Hal Finkel > > > > > > >>> Assistant Computational Scientist > > > > > > >>> Leadership Computing Facility > > > > > > >>> Argonne National Laboratory > > > > > > >>> _______________________________________________ > > > > > > >>> LLVM Developers mailing list > > > > > > >>> llvm-dev at lists.llvm.org > > > > > > >>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev > > > > > > >>> > > > > > > > > > > > > > > _______________________________________________ > > > > > > > LLVM Developers mailing list > > > > > > > llvm-dev at lists.llvm.org > > > > > > > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev > > >> > > _______________________________________________ > > > > > > LLVM Developers mailing list > > > > > > llvm-dev at lists.llvm.org > > > > > > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev > > > > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev-- Hal Finkel Assistant Computational Scientist Leadership Computing Facility Argonne National Laboratory -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160218/45d1dccc/attachment-0001.html>
Hal Finkel via llvm-dev
2016-Feb-18 18:35 UTC
[llvm-dev] RFC: Add bitcode tests to test-suite
----- Original Message -----> From: "David Blaikie via llvm-dev" <llvm-dev at lists.llvm.org> > To: "Mehdi Amini" <mehdi.amini at apple.com> > Cc: "llvm-dev" <llvm-dev at lists.llvm.org> > Sent: Thursday, February 18, 2016 12:15:52 PM > Subject: Re: [llvm-dev] RFC: Add bitcode tests to test-suite> On Thu, Feb 18, 2016 at 10:03 AM, Mehdi Amini via llvm-dev < > llvm-dev at lists.llvm.org > wrote:> > > On Feb 18, 2016, at 8:42 AM, Philip Reames via llvm-dev < > > > llvm-dev at lists.llvm.org > wrote: > > > > > > > > On 02/18/2016 06:54 AM, Hal Finkel via llvm-dev wrote: > > > >> Hi Chandler, et al., > > > >> > > > >> While this proposal to put IR into the test suite technically > > >> non-problematic, I've convinced myself that this is a suboptimal > > >> direction for the LLVM project. Here's what I think would be > > >> better: > > > >> > > > >> - We create a test-suite/Frontends directory, and open this > > >> directory to actively-maintained external frontends, subject to > > >> the following restrictions: > > > >> > > > >> - The frontend must be actively maintained, and the project must > > >> agree to actively maintain the test-suite version > > > >> - The frontend must use the LLVM API (either C or C++) - no > > >> printing textual IR > > > >> - The frontend must have no significant (non-optional) > > >> dependencies outside of LLVM itself, or things on which LLVM > > >> itself depends > > > >> - The frontend must have regression tests and > > >> benchmarks/correctness tests providing significant coverage of > > >> the frontend and its associated code generation > > > >> > > > >> Here's the quid pro quo: > > > >> > > > >> - The LLVM community gains additional testing coverage (which we > > >> definitely need) > > > >> - The LLVM community gains extra insight into how its APIs are > > >> being used (hopefully allowing us to make more-informed > > >> decisions > > >> about how to update them) > > > >> > > > >> - The frontend gains free API updates > > > >> - The frontend's use of LLVM will be more stable > > > >> > > > >> This involves extra work for everybody, but will help us all > > >> deliver higher-quality products. Plus, given the constant > > >> discussions about the difficulty for external projects to follow > > >> API updates, etc., this is a good way to help address those > > >> difficulties. > > > >> > > > >> The fact that Halide will provide extra coverage of our vector > > >> code generation (aside from whatever we happen to produce from > > >> our autovectorizers), and our JIT infrastructure, makes it a > > >> good > > >> candidate for this. Intel's ispc, POCL, (maybe whatever bit of > > >> Mesa uses LLVM), etc. would also be natural candidates should > > >> the > > >> projects be interested. > > > > I think this is a really bad tradeoff and am strongly opposed to > > > this proposal. > > > > > > > > If we want to focus on improving test coverage, that's > > > reasonable, > > > but doing so at the cost of requiring LLVM contributors to > > > maintain everyone's frontend is not a reasonable approach. > > > > > > > > A couple of alternate approaches which might be worth > > > considering: > > > > 1) The IR corpus approach mentioned previously. So long as > > > external > > > teams are willing to update the corpus regularly (weekly), this > > > gives most of the backend coverage with none of the maintenance > > > burden. > > > > 2) Use coverage information to determine which code paths Halide > > > covers which are not covered by existing unit tests. Work to > > > improve those unit tests. Using something along the lines with a > > > mutation testing (i.e. change the source code and see what > > > breaks), combined with test reduction (bugpoint), could greatly > > > improve our test coverage in tree fairly quickly. This would > > > require a lot of work from a single contributor, but that's much > > > better than requiring a lot of work from all contributors. >> > I support the view of Philip on this topic. > > > I think that having the bitcode without the burden of the API and > > linking is the best trade-off for LLVM. >> > About the bitcode, I'm not convince that it needs to be updated > > that > > often: we're interested in coverage for LLVM, not trying to > > validate > > that we support correctly frontend X or Y in a particular version. > > I'd even argue that if Halide version 13 generates a very different > > IR than Halide 12, then we should keep the Halide 12 generated > > bitcode in a separate directory because it is likely to stress LLVM > > differently. >> Potentially, though I hesitate to make test-suite too much of a > dumping ground (it is already, to an extent). Quantity isn't quality > and all thatI agree with this.> - making sure bots cycle quickly (in a relative sense - of course > running the test-suite is going to be a more involved process than > our high priority regression 'unit'-y tests) is important for them > to be meaningful/actionable by developers & not just noise.However, our test suite is already much too small. It is still quite common that correctness bugs show up in self hosting and not in the test suite, and that's not good. Fast turn around is important, but that can be done by having more extensive 'nightly' test sets and smaller 'hourly' ones, etc. -Hal> > I have more questions for Alina. What kind of tests do you have: >> > - "the compiler takes the bitcode and generates code without > > crashing" > > > - "the compiled test runs without crashing" > > > - "the compiled test will produce an output that be checked against > > a > > reference" > > > - "the compiled test is meaningful as a benchmarks" >> > All these different aspects of testing can be interesting, but > > knowing what we're talking may influence the way forward. >> > As mentioned before, the test-suite has a mechanism of "external" > > suites, it may be limited right now but could probably be expanded. > > Ideally there could be multiple repositories that would just be > > checked out independently (think about the way we build LLVM alone > > or LLVM+clang+libcxx+compiler-rt+...). >> > -- > > > Mehdi >> > >> > > > >> Thanks again, > > > >> Hal > > > >> > > > >> ----- Original Message ----- >> > >>> From: "Chandler Carruth" < chandlerc at google.com > > > > >>> To: "Hal Finkel" < hfinkel at anl.gov >, "Alina Sbirlea" < > > >>> alina.sbirlea at gmail.com > > > > >>> Cc: "llvm-dev" < llvm-dev at lists.llvm.org > > > > >>> Sent: Wednesday, February 17, 2016 9:34:24 PM > > > >>> Subject: Re: [llvm-dev] RFC: Add bitcode tests to test-suite > > > >>> > > > >>> > > > >>> Some perhaps relevant aspects that make testing users of LLVM > > >>> like > > > >>> Halide challenging: > > > >>> > > > >>> > > > >>> Halide uses the LLVM C++ APIs, but there isn't a good way to > > > >>> lock-step update it. So if we were to directly test Halide, it > > > >>> wouldn't link against the new LLVM. > > > >>> > > > >>> > > > >>> Practically speaking though, the LLVM IR generated by Halide > > >>> should > > > >>> continue to work with newer LLVM optimizations and code > > >>> generation. > > > >>> So the idea would be to snapshot the IR in bitcode (which is at > > > >>> least reasonably stable) so that we could replay the tests as > > >>> LLVM > > > >>> changes. We can freshen the bitcode by re-generating it > > >>> periodically > > > >>> so it doesn't drift too far from what Halide actually uses. > > > >>> > > > >>> > > > >>> The interesting questions IMO are: > > > >>> > > > >>> > > > >>> 1) Are folks happy using bitcode as the format here? I agree > > >>> with > > >>> Hal > > > >>> that it should be easy since Clang will actually Do The Right > > >>> Thing > > > >>> if given a bitcode input. > > > >>> > > > >>> > > > >>> 2) Are folks happy with non-execution tests in some cases? I > > >>> think > > > >>> Alina is looking at whether we can get a runtime library that > > >>> will > > > >>> allow some of these to actually execute, but at least some of > > >>> the > > > >>> tests are just snap-shots of a JIT, and would need the full > > >>> Halide > > > >>> libraries (and introspection) to execute usefully. > > > >>> > > > >>> > > > >>> -Chandler > > > >>> > > > >>> > > > >>> On Wed, Feb 17, 2016 at 7:25 PM Hal Finkel via llvm-dev < > > > >>> llvm-dev at lists.llvm.org > wrote: > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> From: "Alina Sbirlea via llvm-dev" < llvm-dev at lists.llvm.org > > > > >>> To: "llvm-dev" < llvm-dev at lists.llvm.org > > > > >>> Sent: Wednesday, February 17, 2016 7:25:17 PM > > > >>> Subject: [llvm-dev] RFC: Add bitcode tests to test-suite > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> Hi all, > > > >>> > > > >>> > > > >>> TL;DR: Add *.bc to test-suite; llc *.bc; run some. > > > >>> > > > >>> > > > >>> > > > >>> We would like to propose adding bitcode tests to the llvm > > >>> test-suite. > > > >>> > > > >>> > > > >>> Recent LLVM bugs [2-4] prompted us to look into upstreaming a > > >>> subset > > > >>> of the tests the Halide library [1] is running and we'd like > > >>> the > > > >>> community's feedback on moving forward with this. > > > >>> > > > >>> > > > >>> > > > >>> Halide uses LLVM and can generate bitcode, but we cannot add > > >>> C++ > > > >>> tests to test-suite without including the library itself. > > > >>> This proposal is also potentially useful for other cases where > > >>> there > > > >>> is no C++ front-end. > > > >>> > > > >>> > > > >>> As a first step we are interested in adding a set of > > >>> correctness > > > >>> tests, for testing the IR without running the tests. Since > > >>> these > > > >>> tests are generated, they are not instrumented like the .ll > > >>> files > > >>> in > > > >>> trunk, however we believe checking that llc runs without errors > > >>> is > > > >>> still useful. > > > >>> The bitcode files for Halide may also be large, so including > > >>> them > > >>> as > > > >>> regression tests is not an option. If the smaller tests are > > >>> found > > >>> to > > > >>> be valuable or covering cases no other tests cover, we can > > > >>> instrument them and move them into the llvm trunk further > > >>> along, > > >>> but > > > >>> that is not the goal of this proposal. > > > >>> In addition, we're not sure whether the format for the tests > > >>> should > > > >>> be .ll or .bc, we're open to either. > > > >>> > > > >>> > > > >>> After this first step, we're interested in upstreaming bitcode > > >>> tests > > > >>> and also running them. > > > >>> We are very interested in tests for multiple architectures, > > >>> aarch64 > > > >>> in particular, since this is where we have seen things break. > > >>> This > > > >>> may motivate adding .ll files rather than .bc in order to > > >>> include > > > >>> the "RUN:" target. > > > >>> Where would these tests reside and with what directory > > >>> structure? > > > >>> (similar to test/CodeGen?) > > > >>> > > > >>> > > > >>> Suggestion on what's the best approach for extending the > > >>> test-suite > > > >>> framework for this proposal are more than welcome. > > > >>> > > > >>> > > > >>> > > > >>> We already have architecture-specific tests in the test suite > > >>> (e.g. > > > >>> SingleSource/UnitTests/Vector/{SSE,Altivec,etc.}, and Clang can > > >>> deal > > > >>> with IR inputs. I suppose you need to compile some > > >>> corresponding > > > >>> runtime library, but this does not seem like a big deal either. > > > >>> Mechanically, I don't see this as particularly complicated. I > > >>> think > > > >>> the real question is: Is this the best way to have a kind of > > >>> 'halide > > > >>> buildbot' that can inform the LLVM developer community? > > > >>> > > > >>> -Hal > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> This is just the high-level overview to start off the > > >>> discussion, > > >>> I'm > > > >>> sure there are many more aspects to touch on. Looking forward > > >>> to > > > >>> your feedback! > > > >>> > > > >>> > > > >>> > > > >>> Thanks, > > > >>> Alina > > > >>> > > > >>> > > > >>> [1] http:// halide - lang.org/ > > > >>> [2] Broken: r259800 => Fixed: r260131 > > > >>> [3] Broken: r260569 => Fixed: r260701 > > > >>> > > > >>> [4] https://llvm.org/bugs/show_bug.cgi?id=26642 > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> _______________________________________________ > > > >>> LLVM Developers mailing list > > > >>> llvm-dev at lists.llvm.org > > > >>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> -- > > > >>> > > > >>> Hal Finkel > > > >>> Assistant Computational Scientist > > > >>> Leadership Computing Facility > > > >>> Argonne National Laboratory > > > >>> _______________________________________________ > > > >>> LLVM Developers mailing list > > > >>> llvm-dev at lists.llvm.org > > > >>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev > > > >>> > > > > > > > > _______________________________________________ > > > > LLVM Developers mailing list > > > > llvm-dev at lists.llvm.org > > > > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >> > _______________________________________________ > > > LLVM Developers mailing list > > > llvm-dev at lists.llvm.org > > > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >> _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev-- Hal Finkel Assistant Computational Scientist Leadership Computing Facility Argonne National Laboratory -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160218/f10ad2b6/attachment.html>
Hal Finkel via llvm-dev
2016-Feb-18 18:52 UTC
[llvm-dev] RFC: Add bitcode tests to test-suite
----- Original Message -----> From: "Hal Finkel via llvm-dev" <llvm-dev at lists.llvm.org> > To: "David Blaikie" <dblaikie at gmail.com> > Cc: "llvm-dev" <llvm-dev at lists.llvm.org> > Sent: Thursday, February 18, 2016 12:35:39 PM > Subject: Re: [llvm-dev] RFC: Add bitcode tests to test-suite> ----- Original Message -----> > From: "David Blaikie via llvm-dev" <llvm-dev at lists.llvm.org> > > > To: "Mehdi Amini" <mehdi.amini at apple.com> > > > Cc: "llvm-dev" <llvm-dev at lists.llvm.org> > > > Sent: Thursday, February 18, 2016 12:15:52 PM > > > Subject: Re: [llvm-dev] RFC: Add bitcode tests to test-suite >> > On Thu, Feb 18, 2016 at 10:03 AM, Mehdi Amini via llvm-dev < > > llvm-dev at lists.llvm.org > wrote: >> > > > On Feb 18, 2016, at 8:42 AM, Philip Reames via llvm-dev < > > > > llvm-dev at lists.llvm.org > wrote: > > > > > > > > > > > > > > On 02/18/2016 06:54 AM, Hal Finkel via llvm-dev wrote: > > > > > > >> Hi Chandler, et al., > > > > > > >> > > > > > > >> While this proposal to put IR into the test suite technically > > > >> non-problematic, I've convinced myself that this is a > > > >> suboptimal > > > >> direction for the LLVM project. Here's what I think would be > > > >> better: > > > > > > >> > > > > > > >> - We create a test-suite/Frontends directory, and open this > > > >> directory to actively-maintained external frontends, subject > > > >> to > > > >> the following restrictions: > > > > > > >> > > > > > > >> - The frontend must be actively maintained, and the project > > > >> must > > > >> agree to actively maintain the test-suite version > > > > > > >> - The frontend must use the LLVM API (either C or C++) - no > > > >> printing textual IR > > > > > > >> - The frontend must have no significant (non-optional) > > > >> dependencies outside of LLVM itself, or things on which LLVM > > > >> itself depends > > > > > > >> - The frontend must have regression tests and > > > >> benchmarks/correctness tests providing significant coverage of > > > >> the frontend and its associated code generation > > > > > > >> > > > > > > >> Here's the quid pro quo: > > > > > > >> > > > > > > >> - The LLVM community gains additional testing coverage (which > > > >> we > > > >> definitely need) > > > > > > >> - The LLVM community gains extra insight into how its APIs are > > > >> being used (hopefully allowing us to make more-informed > > > >> decisions > > > >> about how to update them) > > > > > > >> > > > > > > >> - The frontend gains free API updates > > > > > > >> - The frontend's use of LLVM will be more stable > > > > > > >> > > > > > > >> This involves extra work for everybody, but will help us all > > > >> deliver higher-quality products. Plus, given the constant > > > >> discussions about the difficulty for external projects to > > > >> follow > > > >> API updates, etc., this is a good way to help address those > > > >> difficulties. > > > > > > >> > > > > > > >> The fact that Halide will provide extra coverage of our vector > > > >> code generation (aside from whatever we happen to produce from > > > >> our autovectorizers), and our JIT infrastructure, makes it a > > > >> good > > > >> candidate for this. Intel's ispc, POCL, (maybe whatever bit of > > > >> Mesa uses LLVM), etc. would also be natural candidates should > > > >> the > > > >> projects be interested. > > > > > > > I think this is a really bad tradeoff and am strongly opposed > > > > to > > > > this proposal. > > > > > > > > > > > > > > If we want to focus on improving test coverage, that's > > > > reasonable, > > > > but doing so at the cost of requiring LLVM contributors to > > > > maintain everyone's frontend is not a reasonable approach. > > > > > > > > > > > > > > A couple of alternate approaches which might be worth > > > > considering: > > > > > > > 1) The IR corpus approach mentioned previously. So long as > > > > external > > > > teams are willing to update the corpus regularly (weekly), this > > > > gives most of the backend coverage with none of the maintenance > > > > burden. > > > > > > > 2) Use coverage information to determine which code paths > > > > Halide > > > > covers which are not covered by existing unit tests. Work to > > > > improve those unit tests. Using something along the lines with > > > > a > > > > mutation testing (i.e. change the source code and see what > > > > breaks), combined with test reduction (bugpoint), could greatly > > > > improve our test coverage in tree fairly quickly. This would > > > > require a lot of work from a single contributor, but that's > > > > much > > > > better than requiring a lot of work from all contributors. > > >> > > I support the view of Philip on this topic. > > > > > > I think that having the bitcode without the burden of the API and > > > linking is the best trade-off for LLVM. > > >> > > About the bitcode, I'm not convince that it needs to be updated > > > that > > > often: we're interested in coverage for LLVM, not trying to > > > validate > > > that we support correctly frontend X or Y in a particular > > > version. > > > I'd even argue that if Halide version 13 generates a very > > > different > > > IR than Halide 12, then we should keep the Halide 12 generated > > > bitcode in a separate directory because it is likely to stress > > > LLVM > > > differently. > > >> > Potentially, though I hesitate to make test-suite too much of a > > dumping ground (it is already, to an extent). Quantity isn't > > quality > > and all that > > I agree with this.Also, the relative value of compile-only tests seems unclear to me. Not that we don't need better testing, because we do, but this kind of testing can also be done with random input generators (llvm-stress, csmith, etc.), and it seems like investing in configurations/improvements for that kind of technology would give us a much better coverage than a few pre-defined bitcode files. Plus, crashes are much easier to debug without a full setup capable of running the code. A lot of the value in the test suite is that the details of how to compile and run the various tests are all uniformly specified, so that, when a miscompile happens, we don't need to worry about how to download/configure/build/run the code to reproduce the problem. -Hal> > - making sure bots cycle quickly (in a relative sense - of course > > running the test-suite is going to be a more involved process than > > our high priority regression 'unit'-y tests) is important for them > > to be meaningful/actionable by developers & not just noise. >> However, our test suite is already much too small. It is still quite > common that correctness bugs show up in self hosting and not in the > test suite, and that's not good. Fast turn around is important, but > that can be done by having more extensive 'nightly' test sets and > smaller 'hourly' ones, etc.> -Hal> > > I have more questions for Alina. What kind of tests do you have: > > >> > > - "the compiler takes the bitcode and generates code without > > > crashing" > > > > > > - "the compiled test runs without crashing" > > > > > > - "the compiled test will produce an output that be checked > > > against > > > a > > > reference" > > > > > > - "the compiled test is meaningful as a benchmarks" > > >> > > All these different aspects of testing can be interesting, but > > > knowing what we're talking may influence the way forward. > > >> > > As mentioned before, the test-suite has a mechanism of "external" > > > suites, it may be limited right now but could probably be > > > expanded. > > > Ideally there could be multiple repositories that would just be > > > checked out independently (think about the way we build LLVM > > > alone > > > or LLVM+clang+libcxx+compiler-rt+...). > > >> > > -- > > > > > > Mehdi > > >> > > >> > > > > > > >> Thanks again, > > > > > > >> Hal > > > > > > >> > > > > > > >> ----- Original Message ----- > > >> > > >>> From: "Chandler Carruth" < chandlerc at google.com > > > > > > > >>> To: "Hal Finkel" < hfinkel at anl.gov >, "Alina Sbirlea" < > > > >>> alina.sbirlea at gmail.com > > > > > > > >>> Cc: "llvm-dev" < llvm-dev at lists.llvm.org > > > > > > > >>> Sent: Wednesday, February 17, 2016 9:34:24 PM > > > > > > >>> Subject: Re: [llvm-dev] RFC: Add bitcode tests to test-suite > > > > > > >>> > > > > > > >>> > > > > > > >>> Some perhaps relevant aspects that make testing users of LLVM > > > >>> like > > > > > > >>> Halide challenging: > > > > > > >>> > > > > > > >>> > > > > > > >>> Halide uses the LLVM C++ APIs, but there isn't a good way to > > > > > > >>> lock-step update it. So if we were to directly test Halide, > > > >>> it > > > > > > >>> wouldn't link against the new LLVM. > > > > > > >>> > > > > > > >>> > > > > > > >>> Practically speaking though, the LLVM IR generated by Halide > > > >>> should > > > > > > >>> continue to work with newer LLVM optimizations and code > > > >>> generation. > > > > > > >>> So the idea would be to snapshot the IR in bitcode (which is > > > >>> at > > > > > > >>> least reasonably stable) so that we could replay the tests as > > > >>> LLVM > > > > > > >>> changes. We can freshen the bitcode by re-generating it > > > >>> periodically > > > > > > >>> so it doesn't drift too far from what Halide actually uses. > > > > > > >>> > > > > > > >>> > > > > > > >>> The interesting questions IMO are: > > > > > > >>> > > > > > > >>> > > > > > > >>> 1) Are folks happy using bitcode as the format here? I agree > > > >>> with > > > >>> Hal > > > > > > >>> that it should be easy since Clang will actually Do The Right > > > >>> Thing > > > > > > >>> if given a bitcode input. > > > > > > >>> > > > > > > >>> > > > > > > >>> 2) Are folks happy with non-execution tests in some cases? I > > > >>> think > > > > > > >>> Alina is looking at whether we can get a runtime library that > > > >>> will > > > > > > >>> allow some of these to actually execute, but at least some of > > > >>> the > > > > > > >>> tests are just snap-shots of a JIT, and would need the full > > > >>> Halide > > > > > > >>> libraries (and introspection) to execute usefully. > > > > > > >>> > > > > > > >>> > > > > > > >>> -Chandler > > > > > > >>> > > > > > > >>> > > > > > > >>> On Wed, Feb 17, 2016 at 7:25 PM Hal Finkel via llvm-dev < > > > > > > >>> llvm-dev at lists.llvm.org > wrote: > > > > > > >>> > > > > > > >>> > > > > > > >>> > > > > > > >>> > > > > > > >>> > > > > > > >>> > > > > > > >>> > > > > > > >>> > > > > > > >>> From: "Alina Sbirlea via llvm-dev" < llvm-dev at lists.llvm.org > > > >>> > > > > > > > >>> To: "llvm-dev" < llvm-dev at lists.llvm.org > > > > > > > >>> Sent: Wednesday, February 17, 2016 7:25:17 PM > > > > > > >>> Subject: [llvm-dev] RFC: Add bitcode tests to test-suite > > > > > > >>> > > > > > > >>> > > > > > > >>> > > > > > > >>> > > > > > > >>> > > > > > > >>> > > > > > > >>> Hi all, > > > > > > >>> > > > > > > >>> > > > > > > >>> TL;DR: Add *.bc to test-suite; llc *.bc; run some. > > > > > > >>> > > > > > > >>> > > > > > > >>> > > > > > > >>> We would like to propose adding bitcode tests to the llvm > > > >>> test-suite. > > > > > > >>> > > > > > > >>> > > > > > > >>> Recent LLVM bugs [2-4] prompted us to look into upstreaming a > > > >>> subset > > > > > > >>> of the tests the Halide library [1] is running and we'd like > > > >>> the > > > > > > >>> community's feedback on moving forward with this. > > > > > > >>> > > > > > > >>> > > > > > > >>> > > > > > > >>> Halide uses LLVM and can generate bitcode, but we cannot add > > > >>> C++ > > > > > > >>> tests to test-suite without including the library itself. > > > > > > >>> This proposal is also potentially useful for other cases > > > >>> where > > > >>> there > > > > > > >>> is no C++ front-end. > > > > > > >>> > > > > > > >>> > > > > > > >>> As a first step we are interested in adding a set of > > > >>> correctness > > > > > > >>> tests, for testing the IR without running the tests. Since > > > >>> these > > > > > > >>> tests are generated, they are not instrumented like the .ll > > > >>> files > > > >>> in > > > > > > >>> trunk, however we believe checking that llc runs without > > > >>> errors > > > >>> is > > > > > > >>> still useful. > > > > > > >>> The bitcode files for Halide may also be large, so including > > > >>> them > > > >>> as > > > > > > >>> regression tests is not an option. If the smaller tests are > > > >>> found > > > >>> to > > > > > > >>> be valuable or covering cases no other tests cover, we can > > > > > > >>> instrument them and move them into the llvm trunk further > > > >>> along, > > > >>> but > > > > > > >>> that is not the goal of this proposal. > > > > > > >>> In addition, we're not sure whether the format for the tests > > > >>> should > > > > > > >>> be .ll or .bc, we're open to either. > > > > > > >>> > > > > > > >>> > > > > > > >>> After this first step, we're interested in upstreaming > > > >>> bitcode > > > >>> tests > > > > > > >>> and also running them. > > > > > > >>> We are very interested in tests for multiple architectures, > > > >>> aarch64 > > > > > > >>> in particular, since this is where we have seen things break. > > > >>> This > > > > > > >>> may motivate adding .ll files rather than .bc in order to > > > >>> include > > > > > > >>> the "RUN:" target. > > > > > > >>> Where would these tests reside and with what directory > > > >>> structure? > > > > > > >>> (similar to test/CodeGen?) > > > > > > >>> > > > > > > >>> > > > > > > >>> Suggestion on what's the best approach for extending the > > > >>> test-suite > > > > > > >>> framework for this proposal are more than welcome. > > > > > > >>> > > > > > > >>> > > > > > > >>> > > > > > > >>> We already have architecture-specific tests in the test suite > > > >>> (e.g. > > > > > > >>> SingleSource/UnitTests/Vector/{SSE,Altivec,etc.}, and Clang > > > >>> can > > > >>> deal > > > > > > >>> with IR inputs. I suppose you need to compile some > > > >>> corresponding > > > > > > >>> runtime library, but this does not seem like a big deal > > > >>> either. > > > > > > >>> Mechanically, I don't see this as particularly complicated. I > > > >>> think > > > > > > >>> the real question is: Is this the best way to have a kind of > > > >>> 'halide > > > > > > >>> buildbot' that can inform the LLVM developer community? > > > > > > >>> > > > > > > >>> -Hal > > > > > > >>> > > > > > > >>> > > > > > > >>> > > > > > > >>> > > > > > > >>> > > > > > > >>> > > > > > > >>> > > > > > > >>> > > > > > > >>> > > > > > > >>> > > > > > > >>> This is just the high-level overview to start off the > > > >>> discussion, > > > >>> I'm > > > > > > >>> sure there are many more aspects to touch on. Looking forward > > > >>> to > > > > > > >>> your feedback! > > > > > > >>> > > > > > > >>> > > > > > > >>> > > > > > > >>> Thanks, > > > > > > >>> Alina > > > > > > >>> > > > > > > >>> > > > > > > >>> [1] http:// halide - lang.org/ > > > > > > >>> [2] Broken: r259800 => Fixed: r260131 > > > > > > >>> [3] Broken: r260569 => Fixed: r260701 > > > > > > >>> > > > > > > >>> [4] https://llvm.org/bugs/show_bug.cgi?id=26642 > > > > > > >>> > > > > > > >>> > > > > > > >>> > > > > > > >>> > > > > > > >>> _______________________________________________ > > > > > > >>> LLVM Developers mailing list > > > > > > >>> llvm-dev at lists.llvm.org > > > > > > >>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev > > > > > > >>> > > > > > > >>> > > > > > > >>> > > > > > > >>> > > > > > > >>> > > > > > > >>> -- > > > > > > >>> > > > > > > >>> Hal Finkel > > > > > > >>> Assistant Computational Scientist > > > > > > >>> Leadership Computing Facility > > > > > > >>> Argonne National Laboratory > > > > > > >>> _______________________________________________ > > > > > > >>> LLVM Developers mailing list > > > > > > >>> llvm-dev at lists.llvm.org > > > > > > >>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev > > > > > > >>> > > > > > > > > > > > > > > _______________________________________________ > > > > > > > LLVM Developers mailing list > > > > > > > llvm-dev at lists.llvm.org > > > > > > > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev > > >> > > _______________________________________________ > > > > > > LLVM Developers mailing list > > > > > > llvm-dev at lists.llvm.org > > > > > > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev > > >> > _______________________________________________ > > > LLVM Developers mailing list > > > llvm-dev at lists.llvm.org > > > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >> --> Hal Finkel > Assistant Computational Scientist > Leadership Computing Facility > Argonne National Laboratory> _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev-- Hal Finkel Assistant Computational Scientist Leadership Computing Facility Argonne National Laboratory -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160218/625d6432/attachment.html>