llvm dev - Jul 2010 - [LLVMdev] I would like to merge PARSEC into test-suite

On Tue, Jul 20, 2010 at 1:30 AM, Owen Anderson <resistor at mac.com>
wrote:
> Patrick,
>
> On Jul 20, 2010, at 1:18 AM, Patrick Simmons wrote:
>> It is open-source and redistributable, and I have added LICENSE.TXT
>> files to all the pieces I want to merge.  These are blackscholes,
>> canneal, dedup, fluidanimate, freqmine, streamcluster, and swaptions.
 I
>> will disable the tests by default on the initial merge and test
>> thoroughly on Linux and MacOS before enabling them.  May I please
commit
>> my changes directly to test-suite?
Awesome! Can we see a patch?

The PARSEC webpage mentions that the benchmark suite is quite large.
Does your patch actually commit a specific version to the test-suite,
or is it setup like the externals tests where the user is expected to
download PARSEC on the side if they wish to test it?
> I'm not convinced that the PARSEC tests are appropriate for addition
the LLVM testsuite, because they are multithreaded tests.  The testsuite needs
to be low-volatility in terms of performance variation, as well as easy to
verify correctness, and easy for the compiler developers to debug
miscompilations.   I don't think PARSEC is a good match on these fronts.
I don't agree, more tests are always better. We can always invent
infrastructure to ignore the performance of certain tests if that
becomes an issue.

 - Daniel
> --Owen
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>

Daniel Dunbar wrote:
> On Tue, Jul 20, 2010 at 1:30 AM, Owen Anderson <resistor at mac.com>
wrote:
>   
>> Patrick,
>>
>> On Jul 20, 2010, at 1:18 AM, Patrick Simmons wrote:
>>     
>>> It is open-source and redistributable, and I have added LICENSE.TXT
>>> files to all the pieces I want to merge.  These are blackscholes,
>>> canneal, dedup, fluidanimate, freqmine, streamcluster, and
swaptions.  I
>>> will disable the tests by default on the initial merge and test
>>> thoroughly on Linux and MacOS before enabling them.  May I please
commit
>>> my changes directly to test-suite?
>>>       
>
> Awesome! Can we see a patch?
>   
Hi!  I'm the person that asked Patrick to email llvmdev about 
integrating PARSEC.  We needed to make PARSEC work with test-suite 
because we use test-suite as the foundation of our testing 
infrastructure, and since we've gone through the trouble to make it 
work, we might as well let others benefit from our effort.

I advised Patrick to *not* send a patch because it would be very large 
and mostly contain PARSEC source code.  However, I've since realized 
that we can place it on our web site and send out a URL for it.  
Patrick, I'll let you know how to do that.
> The PARSEC webpage mentions that the benchmark suite is quite large.
> Does your patch actually commit a specific version to the test-suite,
> or is it setup like the externals tests where the user is expected to
> download PARSEC on the side if they wish to test it?
>   
We can integrate PARSEC either as a regular internal test with its 
source code inside the test suite or as an external test like SPEC, 
depending on what people think is best.  We wanted to know if there were 
objections to either approach.
>   
>> I'm not convinced that the PARSEC tests are appropriate for
addition the LLVM testsuite, because they are multithreaded tests.  The
testsuite needs to be low-volatility in terms of performance variation, as well
as easy to verify correctness, and easy for the compiler developers to debug
miscompilations.   I don't think PARSEC is a good match on these fronts.
>>     
>
> I don't agree, more tests are always better. We can always invent
> infrastructure to ignore the performance of certain tests if that
> becomes an issue.
>   
As an aside, I'll offer up my opinion on the purpose of test-suite:

IMHO, the test suite is for the LLVM community.  Benchmarking LLVM is 
one of its uses.  However, I think the test suite is also useful for 
stress testing LLVM's correctness and for research (e.g., we use the 
SPEC and MultiSource/Benchmark tests in our papers).  As long as it does 
not grow too large, I think test-suite should continue to fill these 
needs.  If it does grow too large (or some tests cause problems), we can 
split it into several smaller test suites tailored to different 
subgroups of the community.

So, with that said, are there still objections to integrating it as an 
internal test?  If not, I'll ask Patrick to create a patch, and we'll 
send a URL for it.  If there are still objections, would integrating it 
as an external test (like SPEC) be okay?

-- John T.
>  - Daniel
>
>   
>> --Owen
>> _______________________________________________
>> LLVM Developers mailing list
>> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
>> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>>
>>     
>
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>

On 07/20/2010 09:18 AM, Daniel Dunbar wrote:
> On Tue, Jul 20, 2010 at 1:30 AM, Owen Anderson<resistor at mac.com> 
wrote:
>
>> Patrick,
>>
>> On Jul 20, 2010, at 1:18 AM, Patrick Simmons wrote:
>>
>>> It is open-source and redistributable, and I have added LICENSE.TXT
>>> files to all the pieces I want to merge.  These are blackscholes,
>>> canneal, dedup, fluidanimate, freqmine, streamcluster, and
swaptions.  I
>>> will disable the tests by default on the initial merge and test
>>> thoroughly on Linux and MacOS before enabling them.  May I please
commit
>>> my changes directly to test-suite?
>>>
> Awesome! Can we see a patch?
>
> The PARSEC webpage mentions that the benchmark suite is quite large.
> Does your patch actually commit a specific version to the test-suite,
> or is it setup like the externals tests where the user is expected to
> download PARSEC on the side if they wish to test it?
>
My plan was to commit a specific version to test-suite.  The exact 
programs I want to commit are blackscholes, canneal, dedup, 
fluidanimate, freqmine, streamcluster, and swaptions.

John has mentioned the possibility of my putting a patch on a website 
somewhere, so I will do that.  The patch is really too large to post here.

--Patrick

On Wed, Jul 21, 2010 at 11:57 AM, John Criswell <criswell at uiuc.edu>
wrote:
> Daniel Dunbar wrote:
>>
>> On Tue, Jul 20, 2010 at 1:30 AM, Owen Anderson <resistor at
mac.com> wrote:
>>
>>>
>>> Patrick,
>>>
>>> On Jul 20, 2010, at 1:18 AM, Patrick Simmons wrote:
>>>
>>>>
>>>> It is open-source and redistributable, and I have added
LICENSE.TXT
>>>> files to all the pieces I want to merge.  These are
blackscholes,
>>>> canneal, dedup, fluidanimate, freqmine, streamcluster, and
swaptions.  I
>>>> will disable the tests by default on the initial merge and test
>>>> thoroughly on Linux and MacOS before enabling them.  May I
please commit
>>>> my changes directly to test-suite?
>>>>
>>
>> Awesome! Can we see a patch?
>>
>
> Hi!  I'm the person that asked Patrick to email llvmdev about
integrating
> PARSEC.  We needed to make PARSEC work with test-suite because we use
> test-suite as the foundation of our testing infrastructure, and since
we've
> gone through the trouble to make it work, we might as well let others
> benefit from our effort.
>
> I advised Patrick to *not* send a patch because it would be very large and
> mostly contain PARSEC source code.  However, I've since realized that
we can
> place it on our web site and send out a URL for it.  Patrick, I'll let
you
> know how to do that.
>
>> The PARSEC webpage mentions that the benchmark suite is quite large.
>> Does your patch actually commit a specific version to the test-suite,
>> or is it setup like the externals tests where the user is expected to
>> download PARSEC on the side if they wish to test it?
>>
>
> We can integrate PARSEC either as a regular internal test with its source
> code inside the test suite or as an external test like SPEC, depending on
> what people think is best.  We wanted to know if there were objections to
> either approach.
I would probably object to integrating it directly, depending on the
size of code. I don't like the idea of *having* to check out a huge
test suite, just to run the SingleSource tests, for example.

OTOH, I strongly encourage integrating it as an external supplement like SPEC.
>
>>
>>>
>>> I'm not convinced that the PARSEC tests are appropriate for
addition the
>>> LLVM testsuite, because they are multithreaded tests.  The
testsuite needs
>>> to be low-volatility in terms of performance variation, as well as
easy to
>>> verify correctness, and easy for the compiler developers to debug
>>> miscompilations.   I don't think PARSEC is a good match on
these fronts.
>>>
>>
>> I don't agree, more tests are always better. We can always invent
>> infrastructure to ignore the performance of certain tests if that
>> becomes an issue.
>>
>
> As an aside, I'll offer up my opinion on the purpose of test-suite:
>
> IMHO, the test suite is for the LLVM community.  Benchmarking LLVM is one
of
> its uses.  However, I think the test suite is also useful for stress
testing
> LLVM's correctness and for research (e.g., we use the SPEC and
> MultiSource/Benchmark tests in our papers).  As long as it does not grow
too
> large, I think test-suite should continue to fill these needs.  If it does
> grow too large (or some tests cause problems), we can split it into several
> smaller test suites tailored to different subgroups of the community.
Yes, I generally agree with this philosophy, with the caveat as above
that I think keeping large bodies of code in external supplements is
easier to manage.
> So, with that said, are there still objections to integrating it as an
> internal test?  If not, I'll ask Patrick to create a patch, and
we'll send a
> URL for it.  If there are still objections, would integrating it as an
> external test (like SPEC) be okay?
Can we start by integrating it externally? I think it would be fine to
check the external bits into another part of the LLVM repo, I just
don't want to be required to grab it whenever I (or my buildbot
minions) grab llvm-test-suite.

 - Daniel
> -- John T.
>
>>  - Daniel
>>
>>
>>>
>>> --Owen
>>> _______________________________________________
>>> LLVM Developers mailing list
>>> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
>>> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>>>
>>>
>>
>> _______________________________________________
>> LLVM Developers mailing list
>> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
>> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>>
>
>