llvm dev - Nov 2008 - [LLVMdev] Validation Criteria

As we begin designing the proposed validation process, we should think about 
what the criteria for a successful validation should be.

In the initial proposal I suggested the "make check" on llvm should
pass as
should the tests in llvm-test.

According to Tanya, not all of llvm-test passes.  Do we have a sense of how 
far away from a full passing llvm-test we are?

I also know that a lot of work has gone into making the llvm-gcc DejaGNU tests 
pass.  Where are we in that process?

Initially, I'll throw out the idea that if llvm passes its internal tests 
("make check") and successfully bootstraps llvm-gcc-4.2 we can
consider the
validation successful.  Does that sound like a reasonable starting point?  The 
assumption here is that as regressions in other tests (llvm-test, SPEC, etc.) 
are found, testcases will be submitted to the llvm testsuite.

Once llvm-test passes, we can add it to the list of required tests to pass.  
Ditto llvm-gcc-4.2's tests and anything else we might want to add.

Note that as tests get added to existing testsuites, we'll require the new 
tests to pass as well.  So once llvm-test is added to the validation 
criteria, anything added to llvm-test after that will also have to pass in 
order for a validation to succeed.  I don't want to get into the business of
tracking individual test regressions, XFAIL'ing things that run outside 
DejaGNU, etc.  It's a lot of extra work for not much gain, IMHO.

Would anyone want to add other criteria?

                                            -Dave

Hi David,
> According to Tanya, not all of llvm-test passes.  Do we have a sense of how
> far away from a full passing llvm-test we are?
Since some time, we've been calling these tests the "test-suite"
(as opposed
to the DejaGNU test), the name llvm-test has been deprecated.

Gr.

Matthijs
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20081121/31bb1e18/attachment.sig>

On Nov 20, 2008, at 3:14 PM, David Greene wrote:
> As we begin designing the proposed validation process, we should  
> think about
> what the criteria for a successful validation should be.
>
> In the initial proposal I suggested the "make check" on llvm
should
> pass as
> should the tests in llvm-test.
>
> According to Tanya, not all of llvm-test passes.  Do we have a sense  
> of how
> far away from a full passing llvm-test we are?
>
> I also know that a lot of work has gone into making the llvm-gcc  
> DejaGNU tests
> pass.  Where are we in that process?
>
> Initially, I'll throw out the idea that if llvm passes its internal  
> tests
> ("make check") and successfully bootstraps llvm-gcc-4.2 we can  
> consider the
> validation successful.  Does that sound like a reasonable starting  
> point?  The
> assumption here is that as regressions in other tests (llvm-test,  
> SPEC, etc.)
> are found, testcases will be submitted to the llvm testsuite.
>
> Once llvm-test passes, we can add it to the list of required tests  
> to pass.
> Ditto llvm-gcc-4.2's tests and anything else we might want to add.
>
> Note that as tests get added to existing testsuites, we'll require  
> the new
> tests to pass as well.  So once llvm-test is added to the validation
> criteria, anything added to llvm-test after that will also have to  
> pass in
> order for a validation to succeed.  I don't want to get into the  
> business of
> tracking individual test regressions, XFAIL'ing things that run  
> outside
> DejaGNU, etc.  It's a lot of extra work for not much gain, IMHO.
>
> Would anyone want to add other criteria?
>
Hi Dave,

(Sorry that this is late in coming.)

This is what I'd like to see for a valid LLVM:

1. A full bootstrap of LLVM-GCC in Release mode compiles successfully.
2. "make check" on LLVM with (1) passes without regressions.
3. A self-hosted full bootstrap of LLVM-GCC in Release mode compiles  
successfully.
4. "make check" on LLVM with (3) passes without regressions.
5. A "make check" of LLVM-GCC with (1) passes without any *major*  
regressions.
6. No major regressions in the nightly tests from the last verification.

I think that this would be a good start to making sure that a verified  
LLVM is of good quality. I leave the term "major" up for grabs.

-bw