llvm dev - Jul 2010 - [LLVMdev] I would like to merge PARSEC into test-suite

On Wed, Jul 21, 2010 at 11:57 AM, John Criswell <criswell at uiuc.edu>
wrote:
> Daniel Dunbar wrote:
>>
>> On Tue, Jul 20, 2010 at 1:30 AM, Owen Anderson <resistor at
mac.com> wrote:
>>
>>>
>>> Patrick,
>>>
>>> On Jul 20, 2010, at 1:18 AM, Patrick Simmons wrote:
>>>
>>>>
>>>> It is open-source and redistributable, and I have added
LICENSE.TXT
>>>> files to all the pieces I want to merge.  These are
blackscholes,
>>>> canneal, dedup, fluidanimate, freqmine, streamcluster, and
swaptions.  I
>>>> will disable the tests by default on the initial merge and test
>>>> thoroughly on Linux and MacOS before enabling them.  May I
please commit
>>>> my changes directly to test-suite?
>>>>
>>
>> Awesome! Can we see a patch?
>>
>
> Hi!  I'm the person that asked Patrick to email llvmdev about
integrating
> PARSEC.  We needed to make PARSEC work with test-suite because we use
> test-suite as the foundation of our testing infrastructure, and since
we've
> gone through the trouble to make it work, we might as well let others
> benefit from our effort.
>
> I advised Patrick to *not* send a patch because it would be very large and
> mostly contain PARSEC source code.  However, I've since realized that
we can
> place it on our web site and send out a URL for it.  Patrick, I'll let
you
> know how to do that.
>
>> The PARSEC webpage mentions that the benchmark suite is quite large.
>> Does your patch actually commit a specific version to the test-suite,
>> or is it setup like the externals tests where the user is expected to
>> download PARSEC on the side if they wish to test it?
>>
>
> We can integrate PARSEC either as a regular internal test with its source
> code inside the test suite or as an external test like SPEC, depending on
> what people think is best.  We wanted to know if there were objections to
> either approach.
I would probably object to integrating it directly, depending on the
size of code. I don't like the idea of *having* to check out a huge
test suite, just to run the SingleSource tests, for example.

OTOH, I strongly encourage integrating it as an external supplement like SPEC.
>
>>
>>>
>>> I'm not convinced that the PARSEC tests are appropriate for
addition the
>>> LLVM testsuite, because they are multithreaded tests.  The
testsuite needs
>>> to be low-volatility in terms of performance variation, as well as
easy to
>>> verify correctness, and easy for the compiler developers to debug
>>> miscompilations.   I don't think PARSEC is a good match on
these fronts.
>>>
>>
>> I don't agree, more tests are always better. We can always invent
>> infrastructure to ignore the performance of certain tests if that
>> becomes an issue.
>>
>
> As an aside, I'll offer up my opinion on the purpose of test-suite:
>
> IMHO, the test suite is for the LLVM community.  Benchmarking LLVM is one
of
> its uses.  However, I think the test suite is also useful for stress
testing
> LLVM's correctness and for research (e.g., we use the SPEC and
> MultiSource/Benchmark tests in our papers).  As long as it does not grow
too
> large, I think test-suite should continue to fill these needs.  If it does
> grow too large (or some tests cause problems), we can split it into several
> smaller test suites tailored to different subgroups of the community.
Yes, I generally agree with this philosophy, with the caveat as above
that I think keeping large bodies of code in external supplements is
easier to manage.
> So, with that said, are there still objections to integrating it as an
> internal test?  If not, I'll ask Patrick to create a patch, and
we'll send a
> URL for it.  If there are still objections, would integrating it as an
> external test (like SPEC) be okay?
Can we start by integrating it externally? I think it would be fine to
check the external bits into another part of the LLVM repo, I just
don't want to be required to grab it whenever I (or my buildbot
minions) grab llvm-test-suite.

 - Daniel
> -- John T.
>
>>  - Daniel
>>
>>
>>>
>>> --Owen
>>> _______________________________________________
>>> LLVM Developers mailing list
>>> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
>>> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>>>
>>>
>>
>> _______________________________________________
>> LLVM Developers mailing list
>> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
>> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>>
>
>

It's not *that* big.  The patch is 634K uncompressed.  I'd say it would 
be annoying to have that attached to an email message, but it's no 
bigger than existing MultiSource tests.

--Patrick

On 07/21/2010 08:13 PM, Daniel Dunbar wrote:
> On Wed, Jul 21, 2010 at 11:57 AM, John Criswell<criswell at uiuc.edu>
wrote:
>    
>> Daniel Dunbar wrote:
>>      
>>> On Tue, Jul 20, 2010 at 1:30 AM, Owen Anderson<resistor at
mac.com>  wrote:
>>>
>>>        
>>>> Patrick,
>>>>
>>>> On Jul 20, 2010, at 1:18 AM, Patrick Simmons wrote:
>>>>
>>>>          
>>>>> It is open-source and redistributable, and I have added
LICENSE.TXT
>>>>> files to all the pieces I want to merge.  These are
blackscholes,
>>>>> canneal, dedup, fluidanimate, freqmine, streamcluster, and
swaptions.  I
>>>>> will disable the tests by default on the initial merge and
test
>>>>> thoroughly on Linux and MacOS before enabling them.  May I
please commit
>>>>> my changes directly to test-suite?
>>>>>
>>>>>            
>>> Awesome! Can we see a patch?
>>>
>>>        
>> Hi!  I'm the person that asked Patrick to email llvmdev about
integrating
>> PARSEC.  We needed to make PARSEC work with test-suite because we use
>> test-suite as the foundation of our testing infrastructure, and since
we've
>> gone through the trouble to make it work, we might as well let others
>> benefit from our effort.
>>
>> I advised Patrick to *not* send a patch because it would be very large
and
>> mostly contain PARSEC source code.  However, I've since realized
that we can
>> place it on our web site and send out a URL for it.  Patrick, I'll
let you
>> know how to do that.
>>
>>      
>>> The PARSEC webpage mentions that the benchmark suite is quite
large.
>>> Does your patch actually commit a specific version to the
test-suite,
>>> or is it setup like the externals tests where the user is expected
to
>>> download PARSEC on the side if they wish to test it?
>>>
>>>        
>> We can integrate PARSEC either as a regular internal test with its
source
>> code inside the test suite or as an external test like SPEC, depending
on
>> what people think is best.  We wanted to know if there were objections
to
>> either approach.
>>      
> I would probably object to integrating it directly, depending on the
> size of code. I don't like the idea of *having* to check out a huge
> test suite, just to run the SingleSource tests, for example.
>
> OTOH, I strongly encourage integrating it as an external supplement like
SPEC.
>
>    
>>      
>>>        
>>>> I'm not convinced that the PARSEC tests are appropriate for
addition the
>>>> LLVM testsuite, because they are multithreaded tests.  The
testsuite needs
>>>> to be low-volatility in terms of performance variation, as well
as easy to
>>>> verify correctness, and easy for the compiler developers to
debug
>>>> miscompilations.   I don't think PARSEC is a good match on
these fronts.
>>>>
>>>>          
>>> I don't agree, more tests are always better. We can always
invent
>>> infrastructure to ignore the performance of certain tests if that
>>> becomes an issue.
>>>
>>>        
>> As an aside, I'll offer up my opinion on the purpose of test-suite:
>>
>> IMHO, the test suite is for the LLVM community.  Benchmarking LLVM is
one of
>> its uses.  However, I think the test suite is also useful for stress
testing
>> LLVM's correctness and for research (e.g., we use the SPEC and
>> MultiSource/Benchmark tests in our papers).  As long as it does not
grow too
>> large, I think test-suite should continue to fill these needs.  If it
does
>> grow too large (or some tests cause problems), we can split it into
several
>> smaller test suites tailored to different subgroups of the community.
>>      
> Yes, I generally agree with this philosophy, with the caveat as above
> that I think keeping large bodies of code in external supplements is
> easier to manage.
>
>    
>> So, with that said, are there still objections to integrating it as an
>> internal test?  If not, I'll ask Patrick to create a patch, and
we'll send a
>> URL for it.  If there are still objections, would integrating it as an
>> external test (like SPEC) be okay?
>>      
> Can we start by integrating it externally? I think it would be fine to
> check the external bits into another part of the LLVM repo, I just
> don't want to be required to grab it whenever I (or my buildbot
> minions) grab llvm-test-suite.
>
>   - Daniel
>
>    
>> -- John T.
>>
>>      
>>>   - Daniel
>>>
>>>
>>>        
>>>> --Owen
>>>> _______________________________________________
>>>> LLVM Developers mailing list
>>>> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
>>>> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>>>>
>>>>
>>>>          
>>> _______________________________________________
>>> LLVM Developers mailing list
>>> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
>>> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>>>
>>>        
>>
>>      
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>

On Thu, Jul 22, 2010 at 10:45 AM, Patrick Alexander Simmons
<simmon12 at cs.uiuc.edu> wrote:
> It's not *that* big.  The patch is 634K uncompressed.  I'd say it
would
> be annoying to have that attached to an email message, but it's no
> bigger than existing MultiSource tests.
Oh, sure, thats totally fine. I read the "3GB" on the website and got
scared. :)

 - Daniel
> --Patrick
>
> On 07/21/2010 08:13 PM, Daniel Dunbar wrote:
>> On Wed, Jul 21, 2010 at 11:57 AM, John Criswell<criswell at
uiuc.edu>  wrote:
>>
>>> Daniel Dunbar wrote:
>>>
>>>> On Tue, Jul 20, 2010 at 1:30 AM, Owen Anderson<resistor at
mac.com>  wrote:
>>>>
>>>>
>>>>> Patrick,
>>>>>
>>>>> On Jul 20, 2010, at 1:18 AM, Patrick Simmons wrote:
>>>>>
>>>>>
>>>>>> It is open-source and redistributable, and I have added
LICENSE.TXT
>>>>>> files to all the pieces I want to merge.  These are
blackscholes,
>>>>>> canneal, dedup, fluidanimate, freqmine, streamcluster,
and swaptions.  I
>>>>>> will disable the tests by default on the initial merge
and test
>>>>>> thoroughly on Linux and MacOS before enabling them.
 May I please commit
>>>>>> my changes directly to test-suite?
>>>>>>
>>>>>>
>>>> Awesome! Can we see a patch?
>>>>
>>>>
>>> Hi!  I'm the person that asked Patrick to email llvmdev about
integrating
>>> PARSEC.  We needed to make PARSEC work with test-suite because we
use
>>> test-suite as the foundation of our testing infrastructure, and
since we've
>>> gone through the trouble to make it work, we might as well let
others
>>> benefit from our effort.
>>>
>>> I advised Patrick to *not* send a patch because it would be very
large and
>>> mostly contain PARSEC source code.  However, I've since
realized that we can
>>> place it on our web site and send out a URL for it.  Patrick,
I'll let you
>>> know how to do that.
>>>
>>>
>>>> The PARSEC webpage mentions that the benchmark suite is quite
large.
>>>> Does your patch actually commit a specific version to the
test-suite,
>>>> or is it setup like the externals tests where the user is
expected to
>>>> download PARSEC on the side if they wish to test it?
>>>>
>>>>
>>> We can integrate PARSEC either as a regular internal test with its
source
>>> code inside the test suite or as an external test like SPEC,
depending on
>>> what people think is best.  We wanted to know if there were
objections to
>>> either approach.
>>>
>> I would probably object to integrating it directly, depending on the
>> size of code. I don't like the idea of *having* to check out a huge
>> test suite, just to run the SingleSource tests, for example.
>>
>> OTOH, I strongly encourage integrating it as an external supplement
like SPEC.
>>
>>
>>>
>>>>
>>>>> I'm not convinced that the PARSEC tests are appropriate
for addition the
>>>>> LLVM testsuite, because they are multithreaded tests.  The
testsuite needs
>>>>> to be low-volatility in terms of performance variation, as
well as easy to
>>>>> verify correctness, and easy for the compiler developers to
debug
>>>>> miscompilations.   I don't think PARSEC is a good match
on these fronts.
>>>>>
>>>>>
>>>> I don't agree, more tests are always better. We can always
invent
>>>> infrastructure to ignore the performance of certain tests if
that
>>>> becomes an issue.
>>>>
>>>>
>>> As an aside, I'll offer up my opinion on the purpose of
test-suite:
>>>
>>> IMHO, the test suite is for the LLVM community.  Benchmarking LLVM
is one of
>>> its uses.  However, I think the test suite is also useful for
stress testing
>>> LLVM's correctness and for research (e.g., we use the SPEC and
>>> MultiSource/Benchmark tests in our papers).  As long as it does not
grow too
>>> large, I think test-suite should continue to fill these needs.  If
it does
>>> grow too large (or some tests cause problems), we can split it into
several
>>> smaller test suites tailored to different subgroups of the
community.
>>>
>> Yes, I generally agree with this philosophy, with the caveat as above
>> that I think keeping large bodies of code in external supplements is
>> easier to manage.
>>
>>
>>> So, with that said, are there still objections to integrating it as
an
>>> internal test?  If not, I'll ask Patrick to create a patch, and
we'll send a
>>> URL for it.  If there are still objections, would integrating it as
an
>>> external test (like SPEC) be okay?
>>>
>> Can we start by integrating it externally? I think it would be fine to
>> check the external bits into another part of the LLVM repo, I just
>> don't want to be required to grab it whenever I (or my buildbot
>> minions) grab llvm-test-suite.
>>
>>   - Daniel
>>
>>
>>> -- John T.
>>>
>>>
>>>>   - Daniel
>>>>
>>>>
>>>>
>>>>> --Owen
>>>>> _______________________________________________
>>>>> LLVM Developers mailing list
>>>>> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
>>>>> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>>>>>
>>>>>
>>>>>
>>>> _______________________________________________
>>>> LLVM Developers mailing list
>>>> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
>>>> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>>>>
>>>>
>>>
>>>
>> _______________________________________________
>> LLVM Developers mailing list
>> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
>> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>>
>
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>