On 10/18/2012 10:36 AM, David Blaikie wrote:> On Thu, Oct 18, 2012 at 12:48 AM, Renato Golin<rengolin at
systemcall.org> wrote:
>> On 18 October 2012 05:11, Rick Foos<rfoos at codeaurora.org>
wrote:
>>> I don't think GDB testsuite should block a commit, it can vary
by a few
>>> tests, they rarely if ever all pass 100%. Tracking the results over
time can
>>> catch big regressions, as well as the ones that slowly increase the
failed
>>> tests.
>> Agreed. It should at least serve as comparison between two branches,
>> but hopefully, being actively monitored.
>>
>> Maybe would be good to add a directory (if there isn't one yet) to
the
>> testsuite repository, or at least the code necessary to make it run on
>> LLVM.
>
> The clang-tests repository (
> http://llvm.org/viewvc/llvm-project/clang-tests/ ) includes an Apple
> GCC 4.2 compatible version of the GCC and GDB test suites that Apple
> run internally. I'm working on bringing up an equivalent public
> buildbot at least for the GDB suite here (
> http://lab.llvm.org:8011/builders/clang-x86_64-darwin10-gdb-gcc ) -
> just a few timing out tests I need to look at to get that green.
> Apparently it's fairly stable.
>
> Beyond that I'll be trying to bring up one with the latest suite (7.4
> is what I've been playing with) on Linux as well.
>
Since you're going to bring a bot up in zorg, I'll stop working on
bringing mine testsuite runner forward. A couple thoughts:
1) I've been running on the latest test suite, polling once a day. I
think Eric and anyone working dwarf 4/5 should be running against the
upstream testsuite. (I have no problems with running 7.4 too)
It's been stable to run at the tip of GDB this way, the test results
aren't varying much.
2) A surprise benefit of running this way is that hundreds of obsolete
tests, or broken tests are getting removed. This hasn't resulted in any
broken backwards compatibility here at least. Saves tons of time
debugging tests that don't work, and developing around compatible things
that reasonable people have decided no longer matter.
3) Testsuite runs against two compilers at a time makes it easier to see
regressions. By comparing against a known stable compiler, or GCC,
regressions are visible by summary numbers.
4) I have plots of the summary numbers online with a window of a month
or two. The trend allows you to see regressions occurring, and remaining
as regressions. Sometimes GDB Testsuite or a compiler has a bad day. The
trend let's you see a stable regression, and when you get a round toit,
tells you when the regression started.
<soapbox>
I've been doing this with Jenkins. It's fairly easy to set up, and does
the plotting. Developers can grab a copy of the script to duplicate a
run on their broken compiler. Running the testsuite under JNLP increased
the number of executed tests - don't know why just did.
</soapbox>
> - David
>
>> --
>> cheers,
>> --renato
>>
>> http://systemcall.org/
>> _______________________________________________
>> LLVM Developers mailing list
>> LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu
>> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
--
Rick Foos
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The
Linux Foundation