llvm dev - Apr 2008 - [LLVMdev] Proposal for improving the llvm nightly tester

hi all,
After having some discussions in the IRC, I am trying here to come up with a
proposal for GSoC 2008 for improving the llvm nightly tester[1].Following
are the ideas and suggestions that came up in the discussion, if you have
any comment  or any other suggestion please add them to the list. I have
some doubts in some places.

1.  Improvements to the perl script which manage actual testing
(./util/NewNightlyTest.pl)
     (a).  compiling llvm-gcc
     (b).  allowing someone to use an existing tree versus checking out a
new one

2.  The improvements to the email that get sent to the llvm-testresults
      (a). finding a point in time where some test failed
      (b). comparing two different sets of test results that aren't just 2
days apart
  here if you have any suggestions which help someone analysis test results
would be grateful.

3. Running llvm testsuite for all targets, target variants and operating
systems(using an simulator for hardware which is not available) ex:gcc
project's compile farm [2]

4. Running Fortran,  Ada, C/C++/ObjC tests as well with test results emailed
to the llvm-testresults.

Most of the server side and the database design has already done by
"tonic"(IRC nickname at #llvm), so suggestions regarding others would
be
important, so that we have a fully featured nightly tester .And finally all
the php scripts need to be updated according to the required new features.
Thanks in advance. Please add your comments.!
Regards,
-Rajika

[1] - http://llvm.org/nightlytest/
[2] - http://gcc.gnu.org/wiki/CompileFarm

-- 
gautamabuddha - http://www.gautamabuddha.org/
comp.lang.c - http://groups.google.com/group/comp.lang.c/topics
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20080405/53a41e79/attachment.html>

My main complaints about the nightly tester are:
Loading the web pages is waaay tooo sllooooow
The information provided is not terribly useful for tracking down  
bugs.  What bug fixers need is environment+command line options  
adequate to reproduce the problem.  Access to the run logs would be a  
good start.

On Apr 4, 2008, at 9:22 PM, Rajika Kumarasiri wrote:
> hi all,
> After having some discussions in the IRC, I am trying here to come  
> up with a proposal for GSoC 2008 for improving the llvm nightly  
> tester[1].Following are the ideas and suggestions that came up in  
> the discussion, if you have  any comment  or any other suggestion  
> please add them to the list. I have some doubts in some places.
>
> 1.  Improvements to the perl script which manage actual testing (./ 
> util/NewNightlyTest.pl)
>      (a).  compiling llvm-gcc
>      (b).  allowing someone to use an existing tree versus checking  
> out a new one
>
> 2.  The improvements to the email that get sent to the llvm- 
> testresults
>       (a). finding a point in time where some test failed
>       (b). comparing two different sets of test results that aren't  
> just 2 days apart
>   here if you have any suggestions which help someone analysis test  
> results would be grateful.
>
> 3. Running llvm testsuite for all targets, target variants and  
> operating systems(using an simulator for hardware which is not  
> available) ex:gcc project's compile farm [2]
>
> 4. Running Fortran,  Ada, C/C++/ObjC tests as well with test results  
> emailed to the llvm-testresults.
>
> Most of the server side and the database design has already done by  
> "tonic"(IRC nickname at #llvm), so suggestions regarding others  
> would be important, so that we have a fully featured nightly  
> tester .And finally all the php scripts need to be updated according  
> to the required new features.
> Thanks in advance. Please add your comments.!
> Regards,
> -Rajika
>
> [1] - http://llvm.org/nightlytest/
> [2] - http://gcc.gnu.org/wiki/CompileFarm
>
> -- 
> gautamabuddha - http://www.gautamabuddha.org/
> comp.lang.c - http://groups.google.com/group/comp.lang.c/topics
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20080404/4cd5ed19/attachment.html>

Hi Dale,

On Sat, Apr 5, 2008 at 10:27 AM, Dale Johannesen <dalej at apple.com>
wrote:
> My main complaints about the nightly tester are:
> Loading the web pages is waaay tooo sllooooow
>
One of  the complaints there was that the nightly tester is too slow. I am
looking forward to hear any idea from some one to improve that.

> The information provided is not terribly useful for tracking down bugs.
>  What bug fixers need is environment+command line options adequate to
> reproduce the
>
problem.  Access to the run logs would be a good start.
>
This information can be added to the email that get delivered to the
llvm-testresults. Thanks for the inputs!

-Rajika
>
> On Apr 4, 2008, at 9:22 PM, Rajika Kumarasiri wrote:
>
> hi all,
> After having some discussions in the IRC, I am trying here to come up with
> a proposal for GSoC 2008 for improving the llvm nightly tester[1].Following
> are the ideas and suggestions that came up in the discussion, if you have
> any comment  or any other suggestion please add them to the list. I have
> some doubts in some places.
>
> 1.  Improvements to the perl script which manage actual testing
> (./util/NewNightlyTest.pl)
>      (a).  compiling llvm-gcc
>      (b).  allowing someone to use an existing tree versus checking out a
> new one
>
> 2.  The improvements to the email that get sent to the llvm-testresults
>       (a). finding a point in time where some test failed
>       (b). comparing two different sets of test results that aren't
just 2
> days apart
>   here if you have any suggestions which help someone analysis test
> results would be grateful.
>
> 3. Running llvm testsuite for all targets, target variants and operating
> systems(using an simulator for hardware which is not available) ex:gcc
> project's compile farm [2]
>
> 4. Running Fortran,  Ada, C/C++/ObjC tests as well with test results
> emailed to the llvm-testresults.
>
> Most of the server side and the database design has already done by
> "tonic"(IRC nickname at #llvm), so suggestions regarding others
would be
> important, so that we have a fully featured nightly tester .And finally all
> the php scripts need to be updated according to the required new features.
> Thanks in advance. Please add your comments.!
> Regards,
> -Rajika
>
> [1] - http://llvm.org/nightlytest/
> [2] - http://gcc.gnu.org/wiki/CompileFarm
>
> --
> gautamabuddha - http://www.gautamabuddha.org/
> comp.lang.c - http://groups.google.com/group/comp.lang.c/topics
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>
>
>
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>
>

-- 
gautamabuddha - http://www.gautamabuddha.org/
comp.lang.c - http://groups.google.com/group/comp.lang.c/topics
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20080405/994ff4d2/attachment.html>

> 1.  Improvements to the perl script which manage actual testing
> (./util/NewNightlyTest.pl)
>     (a).  compiling llvm-gcc
>     (b).  allowing someone to use an existing tree versus checking out a
> new one
> 2.  The improvements to the email that get sent to the llvm-testresults
>      (a). finding a point in time where some test failed
>      (b). comparing two different sets of test results that aren't just
2
> days apart
>  here if you have any suggestions which help someone analysis test results
> would be grateful.
I think you need to expand on this. Basically, these would be improvements 
to how the data is displayed and analyzed. The second part is making the 
emails that get sent out better. They are separate things.

You should come up with ideas on your own about how someone might want to 
interpret the data and allow that flexibility via the web pages.
> 3. Running llvm testsuite for all targets, target variants and operating
> systems(using an simulator for hardware which is not available) ex:gcc
> project's compile farm [2]
How will you test this?
> 4. Running Fortran,  Ada, C/C++/ObjC tests as well with test results
emailed
> to the llvm-testresults.
Are these different or new tests? Why wouldn't they just be a part of the 
llvm-test module? Then you don't have to do any modifications to the 
scripts.

-Tanya

> Most of the server side and the database design has already done by
> "tonic"(IRC nickname at #llvm), so suggestions regarding others
would be
> important, so that we have a fully featured nightly tester .And finally all
> the php scripts need to be updated according to the required new features.
> Thanks in advance. Please add your comments.!
> Regards,
> -Rajika
>
> [1] - http://llvm.org/nightlytest/
> [2] - http://gcc.gnu.org/wiki/CompileFarm
>
>

> > 4. Running Fortran,  Ada, C/C++/ObjC tests as well with test results
emailed
> > to the llvm-testresults.
> 
> Are these different or new tests? Why wouldn't they just be a part of
the
> llvm-test module? Then you don't have to do any modifications to the 
> scripts.
Probably means the gcc testsuite.

Ciao,

Duncan.