Hi all, As you may or may not have noticed, I have gradually been replacing the testing infrastructure in LLVM. Most of this work is moving LLVM and Clang to share a single testing tool, 'lit'. See: http://llvm.org/cmds/lit.html for documentation on the tool itself. If you haven't already tried it, please consider switch to 'make check-lit' as an alternative to 'make check'. If it doesn't work for you, or you find it doesn't do something DejaGNU did and you like, please let me know. My eventual plan is to move to lit entirely and drop DejaGNU support, so consider yourself warned. A couple key points about 'make check-lit': 1. It runs the unittests as well, there is no separate 'make unittests' step. If you like, there is also a 'make check-all' which will also run the Clang tests at the same time if it happens to be checked out in the standard location (tools/clang). 2. The public buildbots have switched to it. It shouldn't happen, but if you do find a discrepancy between DejaGNU and lit, let me know. If you aren't already familiar with it, here are some of the advantages of lit: 1. It is portable. It already is being used on the Windows buildbot for Clang, and almost all of our tests "just work". 2. It uses multiple threads to test. While it doesn't yet scale quite as well as I would like, switching the Linux x86_64 buildbot to it dropped its cycle time by over 30% (just because running the tests got about 3x faster). 3. I've tried hard to make it easy to use, and will keep working in this direction. Eventually I want to replace the TestRunner scripts as well so that all that is needed to run a single test is 'lit path/to/the/test'. I also would like to improve the output for failures to be more informative (particularly when run via buildbot). See http://llvm.org/PR5217 for my TODO list before dropping DejaGNU, feel free to add comments, requests, or blocking bugs to that one. - Daniel
Daniel Dunbar <daniel at zuster.org> writes: [snip]> 1. It is portable. It already is being used on the Windows buildbot > for Clang, and almost all of our tests "just work".Nice. Thanks Daniel! -- Óscar
On Oct 18, 2009, at 7:49 PM, Daniel Dunbar wrote:> Hi all, > > As you may or may not have noticed, I have gradually been replacing > the testing infrastructure in LLVM. Most of this work is moving LLVM > and Clang to share a single testing tool, 'lit'. See: > http://llvm.org/cmds/lit.html > for documentation on the tool itself. > > If you haven't already tried it, please consider switch to 'make > check-lit' as an alternative to 'make check'. If it doesn't work for > you, or you find it doesn't do something DejaGNU did and you like, > please let me know. My eventual plan is to move to lit entirely and > drop DejaGNU support, so consider yourself warned.This is great work Daniel, One question: Why not switch 'make check' to default to lit and offer 'make check-dejagnu' as a workaround. That forces adoption :) -Chris> > A couple key points about 'make check-lit': > 1. It runs the unittests as well, there is no separate 'make > unittests' step. If you like, there is also a 'make check-all' which > will also run the Clang tests at the same time if it happens to be > checked out in the standard location (tools/clang). > 2. The public buildbots have switched to it. It shouldn't happen, but > if you do find a discrepancy between DejaGNU and lit, let me know. > > If you aren't already familiar with it, here are some of the > advantages of lit: > 1. It is portable. It already is being used on the Windows buildbot > for Clang, and almost all of our tests "just work". > 2. It uses multiple threads to test. While it doesn't yet scale quite > as well as I would like, switching the Linux x86_64 buildbot to it > dropped its cycle time by over 30% (just because running the tests got > about 3x faster). > 3. I've tried hard to make it easy to use, and will keep working in > this direction. Eventually I want to replace the TestRunner scripts as > well so that all that is needed to run a single test is 'lit > path/to/the/test'. I also would like to improve the output for > failures to be more informative (particularly when run via buildbot). > > See http://llvm.org/PR5217 for my TODO list before dropping DejaGNU, > feel free to add comments, requests, or blocking bugs to that one. > > - Daniel > _______________________________________________ > LLVM Developers mailing list > LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
On Sun, Oct 18, 2009 at 9:00 PM, Chris Lattner <clattner at apple.com> wrote:> > On Oct 18, 2009, at 7:49 PM, Daniel Dunbar wrote: > >> Hi all, >> >> As you may or may not have noticed, I have gradually been replacing >> the testing infrastructure in LLVM. Most of this work is moving LLVM >> and Clang to share a single testing tool, 'lit'. See: >> http://llvm.org/cmds/lit.html >> for documentation on the tool itself. >> >> If you haven't already tried it, please consider switch to 'make >> check-lit' as an alternative to 'make check'. If it doesn't work for >> you, or you find it doesn't do something DejaGNU did and you like, >> please let me know. My eventual plan is to move to lit entirely and >> drop DejaGNU support, so consider yourself warned. > > This is great work Daniel, > > One question: Why not switch 'make check' to default to lit and offer 'make > check-dejagnu' as a workaround. That forces adoption :)I intend to as soon as it won't break anything. The sole blocker for doing that is making sure it doesn't break the nightly testing infrastructure. - Daniel> -Chris > >> >> A couple key points about 'make check-lit': >> 1. It runs the unittests as well, there is no separate 'make >> unittests' step. If you like, there is also a 'make check-all' which >> will also run the Clang tests at the same time if it happens to be >> checked out in the standard location (tools/clang). >> 2. The public buildbots have switched to it. It shouldn't happen, but >> if you do find a discrepancy between DejaGNU and lit, let me know. >> >> If you aren't already familiar with it, here are some of the advantages of >> lit: >> 1. It is portable. It already is being used on the Windows buildbot >> for Clang, and almost all of our tests "just work". >> 2. It uses multiple threads to test. While it doesn't yet scale quite >> as well as I would like, switching the Linux x86_64 buildbot to it >> dropped its cycle time by over 30% (just because running the tests got >> about 3x faster). >> 3. I've tried hard to make it easy to use, and will keep working in >> this direction. Eventually I want to replace the TestRunner scripts as >> well so that all that is needed to run a single test is 'lit >> path/to/the/test'. I also would like to improve the output for >> failures to be more informative (particularly when run via buildbot). >> >> See http://llvm.org/PR5217 for my TODO list before dropping DejaGNU, >> feel free to add comments, requests, or blocking bugs to that one. >> >> - Daniel >> _______________________________________________ >> LLVM Developers mailing list >> LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu >> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev > >
On Oct 18, 2009, at 7:49 PM, Daniel Dunbar wrote:> If you haven't already tried it, please consider switch to 'make > check-lit' as an alternative to 'make check'. If it doesn't work for > you, or you find it doesn't do something DejaGNU did and you like, > please let me know. My eventual plan is to move to lit entirely and > drop DejaGNU support, so consider yourself warned.I've swapped all of my testing to lit since all I do is native and it's been great. I'm definitely in favor of Chris's suggestion that we move to that as the default. One quick feature question: one of the nifty things about dejagnu (really the only nifty thing) is that it supported remote execution of binaries - I don't use that right now, but other people may. Any idea if you plan on having lit support that? Honestly I doubt it would be hard in python. Probably a "if someone has a need" feature? :) -eric
On Mon, Oct 19, 2009 at 2:03 PM, Eric Christopher <echristo at apple.com> wrote:> On Oct 18, 2009, at 7:49 PM, Daniel Dunbar wrote: >> If you haven't already tried it, please consider switch to 'make >> check-lit' as an alternative to 'make check'. If it doesn't work for >> you, or you find it doesn't do something DejaGNU did and you like, >> please let me know. My eventual plan is to move to lit entirely and >> drop DejaGNU support, so consider yourself warned. > > I've swapped all of my testing to lit since all I do is native and it's been > great. I'm definitely in favor of Chris's suggestion that we move to that > as the default.Yay!> One quick feature question: one of the nifty things about dejagnu (really > the only nifty thing) is that it supported remote execution of binaries - I > don't use that right now, but other people may. Any idea if you plan on > having lit support that? Honestly I doubt it would be hard in python. > Probably a "if someone has a need" feature? :)Should be possible, although I don't know anything about the dejagnu feature. However, 'lit' has a reasonably complete shell and Tcl parser so it knows what commands are being called, so at least in theory it should be able to rewrite them. - Daniel
On Sun, Oct 18, 2009 at 7:49 PM, Daniel Dunbar <daniel at zuster.org> wrote:> Hi all, > > As you may or may not have noticed, I have gradually been replacing > the testing infrastructure in LLVM. Most of this work is moving LLVM > and Clang to share a single testing tool, 'lit'. See: > http://llvm.org/cmds/lit.html > for documentation on the tool itself. > > If you haven't already tried it, please consider switch to 'make > check-lit' as an alternative to 'make check'. If it doesn't work for > you, or you find it doesn't do something DejaGNU did and you like, > please let me know. My eventual plan is to move to lit entirely and > drop DejaGNU support, so consider yourself warned. > > A couple key points about 'make check-lit': > 1. It runs the unittests as well, there is no separate 'make > unittests' step. If you like, there is also a 'make check-all' which > will also run the Clang tests at the same time if it happens to be > checked out in the standard location (tools/clang).Daniel, This is really nice :) - Devang
On Sun, Oct 18, 2009 at 7:49 PM, Daniel Dunbar <daniel at zuster.org> wrote:> Hi all, > > As you may or may not have noticed, I have gradually been replacing > the testing infrastructure in LLVM. Most of this work is moving LLVM > and Clang to share a single testing tool, 'lit'. See: > http://llvm.org/cmds/lit.html > for documentation on the tool itself. > > If you haven't already tried it, please consider switch to 'make > check-lit' as an alternative to 'make check'. If it doesn't work for > you, or you find it doesn't do something DejaGNU did and you like, > please let me know. My eventual plan is to move to lit entirely and > drop DejaGNU support, so consider yourself warned. > > A couple key points about 'make check-lit': > 1. It runs the unittests as well, there is no separate 'make > unittests' step. If you like, there is also a 'make check-all' which > will also run the Clang tests at the same time if it happens to be > checked out in the standard location (tools/clang).Is there a way to run only clang test ? -- Jean-Daniel