llvm dev - Nov 2015 - 回复：Re: How LLVM guarantee the qualify of the product from the limited test suit?

> On 9 November 2015 at 09:04, via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> > After searching the whole project, I only find about ~10000 cases from
> > "llvm/test" for each commit, and a separate testsuit wrote
with high level
> > language(i.e. C/C++) to verify the quality and performance. As a
general
> > Backend, you know, it must be strong enough to cope with all the IR
> > generated by Frontend. I cannot believe what I see. Did I miss
something ?
> Hi,
> You're missing all the Clang, compiler-rt, libc++ tests on their own
> repositories. That'll add a few more tens of thousands of tests into
> the bundle. But that's just the regression / base validation, the
> release has a second stage which is less visible than, I believe, for
> GCC. We also run the test-suite and benchmarks on trunk validation,
> which is something I believe GCC doesn't do, so releases have a lot
> less bugs than GCC to begin with. GCC focus on fixing all bugs at
> release time, while LLVM focus on fixing them as they happen, which is
> much easier and more stable for all developers.
Yes, there are many test cases that we could leverage as LLVM is widely used.
However, I am not sure if these cases are part of the release process,as they
are not part of this project. Is it that, when LLVM is going to release, we must
pass all the regression tests and test-suit to make sure that, the basic
functionality and performance is good. Then, release it. Clang/rt and other
project would use the new LLVM and report bugs to us if have issues?
The regression test and test-suit is our release criteria, and people would do
more tests for us from other projects. Is it right?
> > Further, I notice that, under llvm project repo, there is also a
clang-tests
> > that using gcc test
> >
suits(http://llvm.org/svn/llvm-project/clang-tests/trunk/gcc-4_2-testsuite/).
> > Is that test suit used by llvm to guarantee the quality before the
release ?
> The GCC test-suite, AFAIK, has very poor quality on what's considered
> a pass or a failure, and it's common to release GCC with thousands of
> failures on those tests. Some people may run it, but I honestly don't
> trust it myself, nor have the time to sift through every single test
> to make sure my errors are compiler errors or test errors. You can't
> assume that just because GCC runs *more* tests, that what they're
> testing is more *thorough*. There are also lots of tests that have
> erratic behaviour, which only adds noise to the process.
> The release process also involves passing standard compiler benchmarks
> from the part of the base testers, and higher level applications (like
> Chromium) from the community. Different targets may get different
> community interest, but most targets have an additional phase inside
> companies like, ARM, Mips, Intel, Apple, Google, Sony, Qualcomm, etc.
> They all have internal work-loads that represent a larger piece of
> real world code that the test-suite can offer. Whenever those work
> loads fail, we get bug reports. It's also good practice to add a
> snippet to the test-suite or the regression tests in these cases.
> As a separate quality control, there are a few efforts tracking
> trunk/releases to build the Linux kernel, Debian, FreeBSD, Mandriva,
> OpenEmbedded and other large scale projects. Whenever something breaks
> on those projects, bugs are reported and fixed on the next stable
> release possible.
> I think it's a pretty solid validation process for both trunk and
releases.
> cheers,
> --renatoAgree that, LLVM tests have much better quality than GCC's. So
all the tests are most provided by developers and bugzilla. Do we have the test
team to do that ?
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20151110/a8fa58ae/attachment.html>