llvm dev - Nov 2015 - How LLVM guarantee the qualify of the product from the limited test suit?

Hi, All,
After searching the whole project, I only find about ~10000 cases from
"llvm/test" for each commit, and a separate testsuit wrote with high
level language(i.e. C/C++) to verify the quality and performance. As a general
Backend, you know, it must be strong enough to cope with all the IR generated by
Frontend. I cannot believe what I see. Did I miss something ? As far as I know,
many commerce compilers usually contains hundreds of thousands test cases.
Further, I notice that, under llvm project repo, there is also a clang-tests
that using gcc test
suits(http://llvm.org/svn/llvm-project/clang-tests/trunk/gcc-4_2-testsuite/). Is
that test suit used by llvm to guarantee the quality before the release ?
Any input is appreciated. Thank you.
 
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20151109/c9337353/attachment.html>

On 9 November 2015 at 09:04, via llvm-dev <llvm-dev at lists.llvm.org>
wrote:
> After searching the whole project, I only find about ~10000 cases from
> "llvm/test" for each commit, and a separate testsuit wrote with
high level
> language(i.e. C/C++) to verify the quality and performance. As a general
> Backend, you know, it must be strong enough to cope with all the IR
> generated by Frontend. I cannot believe what I see. Did I miss something ?
Hi,

You're missing all the Clang, compiler-rt, libc++ tests on their own
repositories. That'll add a few more tens of thousands of tests into
the bundle. But that's just the regression / base validation, the
release has a second stage which is less visible than, I believe, for
GCC. We also run the test-suite and benchmarks on trunk validation,
which is something I believe GCC doesn't do, so releases have a lot
less bugs than GCC to begin with. GCC focus on fixing all bugs at
release time, while LLVM focus on fixing them as they happen, which is
much easier and more stable for all developers.

> Further, I notice that, under llvm project repo, there is also a
clang-tests
> that using gcc test
>
suits(http://llvm.org/svn/llvm-project/clang-tests/trunk/gcc-4_2-testsuite/).
> Is that test suit used by llvm to guarantee the quality before the release
?
The GCC test-suite, AFAIK, has very poor quality on what's considered
a pass or a failure, and it's common to release GCC with thousands of
failures on those tests. Some people may run it, but I honestly don't
trust it myself, nor have the time to sift through every single test
to make sure my errors are compiler errors or test errors. You can't
assume that just because GCC runs *more* tests, that what they're
testing is more *thorough*. There are also lots of tests that have
erratic behaviour, which only adds noise to the process.

The release process also involves passing standard compiler benchmarks
from the part of the base testers, and higher level applications (like
Chromium) from the community. Different targets may get different
community interest, but most targets have an additional phase inside
companies like, ARM, Mips, Intel, Apple, Google, Sony, Qualcomm, etc.
They all have internal work-loads that represent a larger piece of
real world code that the test-suite can offer. Whenever those work
loads fail, we get bug reports. It's also good practice to add a
snippet to the test-suite or the regression tests in these cases.

As a separate quality control, there are a few efforts tracking
trunk/releases to build the Linux kernel, Debian, FreeBSD, Mandriva,
OpenEmbedded and other large scale projects. Whenever something breaks
on those projects, bugs are reported and fixed on the next stable
release possible.

I think it's a pretty solid validation process for both trunk and releases.

cheers,
--renato

On 9 November 2015 at 09:44, Renato Golin via llvm-dev
<llvm-dev at lists.llvm.org> wrote:
> The GCC test-suite, AFAIK, has very poor quality on what's considered
> a pass or a failure, and it's common to release GCC with thousands of
> failures on those tests. Some people may run it, but I honestly don't
> trust it myself, nor have the time to sift through every single test
> to make sure my errors are compiler errors or test errors. You can't
> assume that just because GCC runs *more* tests, that what they're
> testing is more *thorough*. There are also lots of tests that have
> erratic behaviour, which only adds noise to the process.
I've found the gcc test suite pretty useful for my out-of-tree
research backend, though it does require some initial work in
disabling tests that exercise GCC features unsupported by Clang. I
haven't yet switched to using it, but Ed Jones at Embecosm did some
work on making the gcc test suite easier to use with Clang
http://www.embecosm.com/2015/04/21/flexible-runtime-testing-of-llvm-on-embedded-systems/

Alex

> -----Original Message-----
> From: llvm-dev [mailto:llvm-dev-bounces at lists.llvm.org] On Behalf Of
> Renato Golin via llvm-dev
> Sent: Monday, November 09, 2015 1:45 AM
> To: bluedream_zqs at sina.com
> Cc: llvm-dev
> Subject: Re: [llvm-dev] How LLVM guarantee the qualify of the product from
> the limited test suit?
> 
> On 9 November 2015 at 09:04, via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> > After searching the whole project, I only find about ~10000 cases from
> > "llvm/test" for each commit, and a separate testsuit wrote
with high
> level
> > language(i.e. C/C++) to verify the quality and performance. As a
general
> > Backend, you know, it must be strong enough to cope with all the IR
> > generated by Frontend. I cannot believe what I see. Did I miss
something
> ?
> 
> Hi,
> 
> You're missing all the Clang, compiler-rt, libc++ tests on their own
> repositories. That'll add a few more tens of thousands of tests into
> the bundle. But that's just the regression / base validation, the
> release has a second stage which is less visible than, I believe, for
> GCC. We also run the test-suite and benchmarks on trunk validation,
> which is something I believe GCC doesn't do, so releases have a lot
> less bugs than GCC to begin with. GCC focus on fixing all bugs at
> release time, while LLVM focus on fixing them as they happen, which is
> much easier and more stable for all developers.
> 
> 
> > Further, I notice that, under llvm project repo, there is also a
clang-
> tests
> > that using gcc test
> > suits(http://llvm.org/svn/llvm-project/clang-tests/trunk/gcc-4_2-
> testsuite/).
> > Is that test suit used by llvm to guarantee the quality before the
> release ?
> 
> The GCC test-suite, AFAIK, has very poor quality on what's considered
> a pass or a failure, and it's common to release GCC with thousands of
> failures on those tests. Some people may run it, but I honestly don't
> trust it myself, nor have the time to sift through every single test
> to make sure my errors are compiler errors or test errors. You can't
> assume that just because GCC runs *more* tests, that what they're
> testing is more *thorough*. There are also lots of tests that have
> erratic behaviour, which only adds noise to the process.
> 
> The release process also involves passing standard compiler benchmarks
> from the part of the base testers, and higher level applications (like
> Chromium) from the community. Different targets may get different
> community interest, but most targets have an additional phase inside
> companies like, ARM, Mips, Intel, Apple, Google, Sony, Qualcomm, etc.
> They all have internal work-loads that represent a larger piece of
> real world code that the test-suite can offer. Whenever those work
> loads fail, we get bug reports. It's also good practice to add a
> snippet to the test-suite or the regression tests in these cases.
Hear, hear.  The llvm/test/... and clang/test/... suites are no more
than "smoke tests" in my opinion.  We do a LOT more than that
internally.
--paulr
> 
> As a separate quality control, there are a few efforts tracking
> trunk/releases to build the Linux kernel, Debian, FreeBSD, Mandriva,
> OpenEmbedded and other large scale projects. Whenever something breaks
> on those projects, bugs are reported and fixed on the next stable
> release possible.
> 
> I think it's a pretty solid validation process for both trunk and
> releases.
> 
> cheers,
> --renato
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

>
>
> The GCC test-suite, AFAIK, has very poor quality on what's considered
> a pass or a failure,


?????
What makes you say this


> and it's common to release GCC with thousands of
> failures on those tests.


Also not correct.

https://gcc.gnu.org/gcc-4.4/criteria.html

It is a zero regression policy for primary platforms.

Look, I love LLVM as much as the next guy, but in the 15 years i worked on
GCC, through lots of major and minor releases, i can't remember a single
release with "thousands" of failures.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20151109/e8bfe0ad/attachment.html>