Alina Sbirlea via llvm-dev
2016-Feb-18 01:25 UTC
[llvm-dev] RFC: Add bitcode tests to test-suite
Hi all, TL;DR: Add *.bc to test-suite; llc *.bc; run some. We would like to propose adding bitcode tests to the llvm test-suite. Recent LLVM bugs [2-4] prompted us to look into upstreaming a subset of the tests the Halide library [1] is running and we'd like the community's feedback on moving forward with this. Halide uses LLVM and can generate bitcode, but we cannot add C++ tests to test-suite without including the library itself. This proposal is also potentially useful for other cases where there is no C++ front-end. As a first step we are interested in adding a set of correctness tests, for testing the IR without running the tests. Since these tests are generated, they are not instrumented like the .ll files in trunk, however we believe checking that llc runs without errors is still useful. The bitcode files for Halide may also be large, so including them as regression tests is not an option. If the smaller tests are found to be valuable or covering cases no other tests cover, we can instrument them and move them into the llvm trunk further along, but that is not the goal of this proposal. In addition, we're not sure whether the format for the tests should be .ll or .bc, we're open to either. After this first step, we're interested in upstreaming bitcode tests and also running them. We are very interested in tests for multiple architectures, aarch64 in particular, since this is where we have seen things break. This may motivate adding .ll files rather than .bc in order to include the "RUN:" target. Where would these tests reside and with what directory structure? (similar to test/CodeGen?) Suggestion on what's the best approach for extending the test-suite framework for this proposal are more than welcome. This is just the high-level overview to start off the discussion, I'm sure there are many more aspects to touch on. Looking forward to your feedback! Thanks, Alina [1] http://halide-lang.org/ [2] Broken: r259800 => Fixed: r260131 [3] Broken: r260569 => Fixed: r260701 [4] https://llvm.org/bugs/show_bug.cgi?id=26642 -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160217/d8e48b0e/attachment.html>
Hal Finkel via llvm-dev
2016-Feb-18 03:25 UTC
[llvm-dev] RFC: Add bitcode tests to test-suite
----- Original Message -----> From: "Alina Sbirlea via llvm-dev" <llvm-dev at lists.llvm.org> > To: "llvm-dev" <llvm-dev at lists.llvm.org> > Sent: Wednesday, February 17, 2016 7:25:17 PM > Subject: [llvm-dev] RFC: Add bitcode tests to test-suite> Hi all,> TL;DR: Add *.bc to test-suite; llc *.bc; run some.> We would like to propose adding bitcode tests to the llvm test-suite.> Recent LLVM bugs [2-4] prompted us to look into upstreaming a subset > of the tests the Halide library [1] is running and we'd like the > community's feedback on moving forward with this.> Halide uses LLVM and can generate bitcode, but we cannot add C++ > tests to test-suite without including the library itself. > This proposal is also potentially useful for other cases where there > is no C++ front-end.> As a first step we are interested in adding a set of correctness > tests, for testing the IR without running the tests. Since these > tests are generated, they are not instrumented like the .ll files in > trunk, however we believe checking that llc runs without errors is > still useful. > The bitcode files for Halide may also be large, so including them as > regression tests is not an option. If the smaller tests are found to > be valuable or covering cases no other tests cover, we can > instrument them and move them into the llvm trunk further along, but > that is not the goal of this proposal. > In addition, we're not sure whether the format for the tests should > be .ll or .bc, we're open to either.> After this first step, we're interested in upstreaming bitcode tests > and also running them. > We are very interested in tests for multiple architectures, aarch64 > in particular, since this is where we have seen things break. This > may motivate adding .ll files rather than .bc in order to include > the "RUN:" target. > Where would these tests reside and with what directory structure? > (similar to test/CodeGen?)> Suggestion on what's the best approach for extending the test-suite > framework for this proposal are more than welcome.We already have architecture-specific tests in the test suite (e.g. SingleSource/UnitTests/Vector/{SSE,Altivec,etc.}, and Clang can deal with IR inputs. I suppose you need to compile some corresponding runtime library, but this does not seem like a big deal either. Mechanically, I don't see this as particularly complicated. I think the real question is: Is this the best way to have a kind of 'halide buildbot' that can inform the LLVM developer community? -Hal> This is just the high-level overview to start off the discussion, I'm > sure there are many more aspects to touch on. Looking forward to > your feedback!> Thanks, > Alina> [1] http:// halide -lang.org/ > [2] Broken: r259800 => Fixed: r260131 > [3] Broken: r260569 => Fixed: r260701> [4] https://llvm.org/bugs/show_bug.cgi?id=26642> _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev-- Hal Finkel Assistant Computational Scientist Leadership Computing Facility Argonne National Laboratory -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160217/2296bd36/attachment.html>
Chandler Carruth via llvm-dev
2016-Feb-18 03:34 UTC
[llvm-dev] RFC: Add bitcode tests to test-suite
Some perhaps relevant aspects that make testing users of LLVM like Halide challenging: Halide uses the LLVM C++ APIs, but there isn't a good way to lock-step update it. So if we were to directly test Halide, it wouldn't link against the new LLVM. Practically speaking though, the LLVM IR generated by Halide should continue to work with newer LLVM optimizations and code generation. So the idea would be to snapshot the IR in bitcode (which is at least reasonably stable) so that we could replay the tests as LLVM changes. We can freshen the bitcode by re-generating it periodically so it doesn't drift too far from what Halide actually uses. The interesting questions IMO are: 1) Are folks happy using bitcode as the format here? I agree with Hal that it should be easy since Clang will actually Do The Right Thing if given a bitcode input. 2) Are folks happy with non-execution tests in some cases? I think Alina is looking at whether we can get a runtime library that will allow some of these to actually execute, but at least some of the tests are just snap-shots of a JIT, and would need the full Halide libraries (and introspection) to execute usefully. -Chandler On Wed, Feb 17, 2016 at 7:25 PM Hal Finkel via llvm-dev < llvm-dev at lists.llvm.org> wrote:> > ------------------------------ > > *From: *"Alina Sbirlea via llvm-dev" <llvm-dev at lists.llvm.org> > *To: *"llvm-dev" <llvm-dev at lists.llvm.org> > *Sent: *Wednesday, February 17, 2016 7:25:17 PM > *Subject: *[llvm-dev] RFC: Add bitcode tests to test-suite > > > > Hi all, > > TL;DR: Add *.bc to test-suite; llc *.bc; run some. > > We would like to propose adding bitcode tests to the llvm test-suite. > > Recent LLVM bugs [2-4] prompted us to look into upstreaming a subset of > the tests the Halide library [1] is running and we'd like the community's > feedback on moving forward with this. > > Halide uses LLVM and can generate bitcode, but we cannot add C++ tests to > test-suite without including the library itself. > This proposal is also potentially useful for other cases where there is no > C++ front-end. > > As a first step we are interested in adding a set of correctness tests, > for testing the IR without running the tests. Since these tests are > generated, they are not instrumented like the .ll files in trunk, however > we believe checking that llc runs without errors is still useful. > The bitcode files for Halide may also be large, so including them as > regression tests is not an option. If the smaller tests are found to be > valuable or covering cases no other tests cover, we can instrument them and > move them into the llvm trunk further along, but that is not the goal of > this proposal. > In addition, we're not sure whether the format for the tests should be .ll > or .bc, we're open to either. > > After this first step, we're interested in upstreaming bitcode tests and > also running them. > We are very interested in tests for multiple architectures, aarch64 in > particular, since this is where we have seen things break. This may > motivate adding .ll files rather than .bc in order to include the "RUN:" > target. > Where would these tests reside and with what directory structure? (similar > to test/CodeGen?) > > Suggestion on what's the best approach for extending the test-suite > framework for this proposal are more than welcome. > > We already have architecture-specific tests in the test suite (e.g. > SingleSource/UnitTests/Vector/{SSE,Altivec,etc.}, and Clang can deal with > IR inputs. I suppose you need to compile some corresponding runtime > library, but this does not seem like a big deal either. Mechanically, I > don't see this as particularly complicated. I think the real question is: > Is this the best way to have a kind of 'halide buildbot' that can inform > the LLVM developer community? > > -Hal > > > > This is just the high-level overview to start off the discussion, I'm sure > there are many more aspects to touch on. Looking forward to your feedback! > > Thanks, > Alina > > [1] http://halide-lang.org/ > [2] Broken: r259800 => Fixed: r260131 > [3] Broken: r260569 => Fixed: r260701 > [4] https://llvm.org/bugs/show_bug.cgi?id=26642 > > > > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev > > > > > -- > Hal Finkel > Assistant Computational Scientist > Leadership Computing Facility > Argonne National Laboratory > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160218/7c554c6a/attachment.html>
Philip Reames via llvm-dev
2016-Feb-18 04:34 UTC
[llvm-dev] RFC: Add bitcode tests to test-suite
I'm a bit confused as to what's being proposed for immediate action. Is the proposal essentially to add a set of binary bitcode files and ensure that running each of them through LLC does not trigger any assertions? If so, arranging that as an external build bot would seem entirely reasonable. If on the other hand, we were testing for equivalence of the output assembly, that would probably NOT be okay, just because of false positive rate. (Running with the assumption I rephrased the proposal correctly... if not, this will be totally off topic.) Beyond Halide, this sounds like a potentially useful general mechanism for integration testing (not unit testing). We (upstream llvm) have something sorta similar today in the "self host clang, see what breaks" workflow. We (my downstream team) have a similar mechanism where we've collected a corpus of IR files from key benchmarks (regenerated weekly), that we use to stress test tricky commits before submission. Standardizing such a mechanism and policy around it's usage seems useful. My major concerns are a) keeping the corpus small enough to be useful, b) not having them intermixed with unit tests and c) making sure that the *frontend* authors pay the primary cost to maintain them. Philip On 02/17/2016 05:25 PM, Alina Sbirlea via llvm-dev wrote:> Hi all, > > TL;DR: Add *.bc to test-suite; llc *.bc; run some. > > We would like to propose adding bitcode tests to the llvm test-suite. > > Recent LLVM bugs [2-4] prompted us to look into upstreaming a subset > of the tests the Halide library [1] is running and we'd like the > community's feedback on moving forward with this. > > Halide uses LLVM and can generate bitcode, but we cannot add C++ tests > to test-suite without including the library itself. > This proposal is also potentially useful for other cases where there > is no C++ front-end. > > As a first step we are interested in adding a set of correctness > tests, for testing the IR without running the tests. Since these tests > are generated, they are not instrumented like the .ll files in trunk, > however we believe checking that llc runs without errors is still useful. > The bitcode files for Halide may also be large, so including them as > regression tests is not an option. If the smaller tests are found to > be valuable or covering cases no other tests cover, we can instrument > them and move them into the llvm trunk further along, but that is not > the goal of this proposal. > In addition, we're not sure whether the format for the tests should be > .ll or .bc, we're open to either. > > After this first step, we're interested in upstreaming bitcode tests > and also running them. > We are very interested in tests for multiple architectures, aarch64 in > particular, since this is where we have seen things break. This may > motivate adding .ll files rather than .bc in order to include the > "RUN:" target. > Where would these tests reside and with what directory structure? > (similar to test/CodeGen?) > > Suggestion on what's the best approach for extending the test-suite > framework for this proposal are more than welcome. > > This is just the high-level overview to start off the discussion, I'm > sure there are many more aspects to touch on. Looking forward to your > feedback! > > Thanks, > Alina > > [1] http://halide-lang.org/ <http://halide-lang.org/> > [2] Broken: r259800 => Fixed: r260131 > [3] Broken: r260569 => Fixed: r260701 > [4] https://llvm.org/bugs/show_bug.cgi?id=26642 > > > > > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160217/f08c378e/attachment.html>
Alina Sbirlea via llvm-dev
2016-Feb-18 05:59 UTC
[llvm-dev] RFC: Add bitcode tests to test-suite
On Wed, Feb 17, 2016 at 8:34 PM, Philip Reames <listmail at philipreames.com> wrote:> I'm a bit confused as to what's being proposed for immediate action. Is > the proposal essentially to add a set of binary bitcode files and ensure > that running each of them through LLC does not trigger any assertions? If > so, arranging that as an external build bot would seem entirely > reasonable. If on the other hand, we were testing for equivalence of the > output assembly, that would probably NOT be okay, just because of false > positive rate. >No, we are not planning for equivalence of output assembly.> > (Running with the assumption I rephrased the proposal correctly... if not, > this will be totally off topic.) > > Beyond Halide, this sounds like a potentially useful general mechanism for > integration testing (not unit testing). We (upstream llvm) have something > sorta similar today in the "self host clang, see what breaks" workflow. We > (my downstream team) have a similar mechanism where we've collected a > corpus of IR files from key benchmarks (regenerated weekly), that we use to > stress test tricky commits before submission. > > Standardizing such a mechanism and policy around it's usage seems useful. > My major concerns are a) keeping the corpus small enough to be useful, b) > not having them intermixed with unit tests and c) making sure that the > *frontend* authors pay the primary cost to maintain them. >>From my side I would like to have this standardized. As I said, I thinkthis proposal's benefits are not just for Halide. a) I believe the frontend authors need to ensure this. For Halide we would like to include a larger set of unit tests and a small set of large applications. The unit tests are meant to make sure no assertions are triggered. The applications are meant to run and compare with reference results. Do you consider such a corpus of IR files reasonable? b) I agree. How/where should these be integrated into test-suite? c) Primarily the tests would be maintained by the frontend authors, but may be updated by llvm devs. For example, in the event of a change in llvm IR the easiest would be to have the tests as bitcode files.> > Philip > > > On 02/17/2016 05:25 PM, Alina Sbirlea via llvm-dev wrote: > > Hi all, > > TL;DR: Add *.bc to test-suite; llc *.bc; run some. > > We would like to propose adding bitcode tests to the llvm test-suite. > > Recent LLVM bugs [2-4] prompted us to look into upstreaming a subset of > the tests the Halide library [1] is running and we'd like the community's > feedback on moving forward with this. > > Halide uses LLVM and can generate bitcode, but we cannot add C++ tests to > test-suite without including the library itself. > This proposal is also potentially useful for other cases where there is no > C++ front-end. > > As a first step we are interested in adding a set of correctness tests, > for testing the IR without running the tests. Since these tests are > generated, they are not instrumented like the .ll files in trunk, however > we believe checking that llc runs without errors is still useful. > The bitcode files for Halide may also be large, so including them as > regression tests is not an option. If the smaller tests are found to be > valuable or covering cases no other tests cover, we can instrument them and > move them into the llvm trunk further along, but that is not the goal of > this proposal. > In addition, we're not sure whether the format for the tests should be .ll > or .bc, we're open to either. > > After this first step, we're interested in upstreaming bitcode tests and > also running them. > We are very interested in tests for multiple architectures, aarch64 in > particular, since this is where we have seen things break. This may > motivate adding .ll files rather than .bc in order to include the "RUN:" > target. > Where would these tests reside and with what directory structure? (similar > to test/CodeGen?) > > Suggestion on what's the best approach for extending the test-suite > framework for this proposal are more than welcome. > > This is just the high-level overview to start off the discussion, I'm sure > there are many more aspects to touch on. Looking forward to your feedback! > > Thanks, > Alina > > [1] <http://halide-lang.org/>http://halide-lang.org/ > [2] Broken: r259800 => Fixed: r260131 > [3] Broken: r260569 => Fixed: r260701 > [4] https://llvm.org/bugs/show_bug.cgi?id=26642 > > > > > _______________________________________________ > LLVM Developers mailing listllvm-dev at lists.llvm.orghttp://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev > > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160217/123a791b/attachment.html>