llvm dev - Jul 2018 - Separate test harness for LLVM test-suite

If this is mainly about size then I would rather propose to split the
"ABI-Testsuite" folder into a separate repository. With 1.5G it takes
up more space than the rest of the repository combined yet it is coneptually
independent of the rest of the testsuite and most people are not running it.

- Matthias
> On Jul 9, 2018, at 9:11 AM, Michael Kruse via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> 
> If I understand correctly, you wish to put the the test-suite
> framework (cmake/, litsupport/, litsupport-tests/, tools/, utils/)
> into a different repository than the test sources (MultiSource/,
> SingleSource, etc)?
> 
> Conceptually, the test framework (llvm-lit) is already separated,
> living in the main LLVM repository. The build instructions (cmake
> files) are hard to separate from the source files to compile.
> 
> I don't find that the External/ part works well. It was created for
> licensing reasons, for proprietary source that cannot be included in
> the repository. I have yet to successfully make any non-SPEC program
> work. For instance, I cannot find the source of Povray that works with
> test-suite's makefile/CMakeLists.txt
> 
> Michael
> 
> 
> 
> 
> 2018-07-09 2:26 GMT-05:00 Elena Lepilkina via llvm-dev
> <llvm-dev at lists.llvm.org>:
>> Dear community,
>> 
>> 
>> 
>> LLVM test-suite includes both default applications (benchmarks, tests)
and
>> test harness (CMake modules, lit-support, some scripts, etc.). Test
suite
>> provides extension interface to add “external” tests suite and folders.
This
>> worked well, we enhanced basic harness and integrated large set of
internal
>> benchmarks and applications.
>> 
>> 
>> 
>> However, we have users interested in LLVM test harness to build,
profile,
>> test applications and experiment with tools/compilers without using
>> applications from full LLVM test-suite (~2,5 Gb).
>> 
>> 
>> 
>> Are there reasons why separation of test-suite into 1) basic
test-harness
>> and 2) applications is undesirable or impossible?
>> 
>> 
>> 
>> We will be glad to hear opinions of community.
>> 
>> 
>> 
>> Best regards,
>> 
>> Elena Lepilkina,
>> 
>> Synopsys Corp, ARC processors
>> 
>> 
>> 
>> 
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>> 
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

On 07/09/2018 12:32 PM, Matthias Braun via llvm-dev
wrote:
> If this is mainly about size then I would rather propose to split the
"ABI-Testsuite" folder into a separate repository. With 1.5G it takes
up more space than the rest of the repository combined yet it is coneptually
independent of the rest of the testsuite and most people are not running it.
I think is a good idea regardless.

Actually, do we have any buildbots running the ABI testsuite?

 -Hal
>
> - Matthias
>
>> On Jul 9, 2018, at 9:11 AM, Michael Kruse via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
>>
>> If I understand correctly, you wish to put the the test-suite
>> framework (cmake/, litsupport/, litsupport-tests/, tools/, utils/)
>> into a different repository than the test sources (MultiSource/,
>> SingleSource, etc)?
>>
>> Conceptually, the test framework (llvm-lit) is already separated,
>> living in the main LLVM repository. The build instructions (cmake
>> files) are hard to separate from the source files to compile.
>>
>> I don't find that the External/ part works well. It was created for
>> licensing reasons, for proprietary source that cannot be included in
>> the repository. I have yet to successfully make any non-SPEC program
>> work. For instance, I cannot find the source of Povray that works with
>> test-suite's makefile/CMakeLists.txt
>>
>> Michael
>>
>>
>>
>>
>> 2018-07-09 2:26 GMT-05:00 Elena Lepilkina via llvm-dev
>> <llvm-dev at lists.llvm.org>:
>>> Dear community,
>>>
>>>
>>>
>>> LLVM test-suite includes both default applications (benchmarks,
tests) and
>>> test harness (CMake modules, lit-support, some scripts, etc.). Test
suite
>>> provides extension interface to add “external” tests suite and
folders. This
>>> worked well, we enhanced basic harness and integrated large set of
internal
>>> benchmarks and applications.
>>>
>>>
>>>
>>> However, we have users interested in LLVM test harness to build,
profile,
>>> test applications and experiment with tools/compilers without using
>>> applications from full LLVM test-suite (~2,5 Gb).
>>>
>>>
>>>
>>> Are there reasons why separation of test-suite into 1) basic
test-harness
>>> and 2) applications is undesirable or impossible?
>>>
>>>
>>>
>>> We will be glad to hear opinions of community.
>>>
>>>
>>>
>>> Best regards,
>>>
>>> Elena Lepilkina,
>>>
>>> Synopsys Corp, ARC processors
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> LLVM Developers mailing list
>>> llvm-dev at lists.llvm.org
>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
-- 
Hal Finkel
Lead, Compiler Technology and Programming Languages
Leadership Computing Facility
Argonne National Laboratory

I personally was hoping that maybe when we move to github and have to reorganize
the repositories anyway, we can split out "ABI-Testsuite" and
"ClangAnalyzer" from the llvm-testsuite repository, given that they
pretty orthogonal to the rest.


Judging from builder names we one ABI-Testuite builder:
http://lab.llvm.org:8011/builders/clang-x86_64-linux-abi-test
<http://lab.llvm.org:8011/builders/clang-x86_64-linux-abi-test>
(and BTW: I am very happy that we have it!)

- Matthias
> On Jul 9, 2018, at 10:48 AM, Hal Finkel <hfinkel at anl.gov> wrote:
> 
> 
> On 07/09/2018 12:32 PM, Matthias Braun via llvm-dev wrote:
>> If this is mainly about size then I would rather propose to split the
"ABI-Testsuite" folder into a separate repository. With 1.5G it takes
up more space than the rest of the repository combined yet it is coneptually
independent of the rest of the testsuite and most people are not running it.
> 
> I think is a good idea regardless.
> 
> Actually, do we have any buildbots running the ABI testsuite?
> 
>  -Hal
> 
>> 
>> - Matthias
>> 
>>> On Jul 9, 2018, at 9:11 AM, Michael Kruse via llvm-dev <llvm-dev
at lists.llvm.org> wrote:
>>> 
>>> If I understand correctly, you wish to put the the test-suite
>>> framework (cmake/, litsupport/, litsupport-tests/, tools/, utils/)
>>> into a different repository than the test sources (MultiSource/,
>>> SingleSource, etc)?
>>> 
>>> Conceptually, the test framework (llvm-lit) is already separated,
>>> living in the main LLVM repository. The build instructions (cmake
>>> files) are hard to separate from the source files to compile.
>>> 
>>> I don't find that the External/ part works well. It was created
for
>>> licensing reasons, for proprietary source that cannot be included
in
>>> the repository. I have yet to successfully make any non-SPEC
program
>>> work. For instance, I cannot find the source of Povray that works
with
>>> test-suite's makefile/CMakeLists.txt
>>> 
>>> Michael
>>> 
>>> 
>>> 
>>> 
>>> 2018-07-09 2:26 GMT-05:00 Elena Lepilkina via llvm-dev
>>> <llvm-dev at lists.llvm.org>:
>>>> Dear community,
>>>> 
>>>> 
>>>> 
>>>> LLVM test-suite includes both default applications (benchmarks,
tests) and
>>>> test harness (CMake modules, lit-support, some scripts, etc.).
Test suite
>>>> provides extension interface to add “external” tests suite and
folders. This
>>>> worked well, we enhanced basic harness and integrated large set
of internal
>>>> benchmarks and applications.
>>>> 
>>>> 
>>>> 
>>>> However, we have users interested in LLVM test harness to
build, profile,
>>>> test applications and experiment with tools/compilers without
using
>>>> applications from full LLVM test-suite (~2,5 Gb).
>>>> 
>>>> 
>>>> 
>>>> Are there reasons why separation of test-suite into 1) basic
test-harness
>>>> and 2) applications is undesirable or impossible?
>>>> 
>>>> 
>>>> 
>>>> We will be glad to hear opinions of community.
>>>> 
>>>> 
>>>> 
>>>> Best regards,
>>>> 
>>>> Elena Lepilkina,
>>>> 
>>>> Synopsys Corp, ARC processors
>>>> 
>>>> 
>>>> 
>>>> 
>>>> _______________________________________________
>>>> LLVM Developers mailing list
>>>> llvm-dev at lists.llvm.org
>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>> 
>>> _______________________________________________
>>> LLVM Developers mailing list
>>> llvm-dev at lists.llvm.org
>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> 
> -- 
> Hal Finkel
> Lead, Compiler Technology and Programming Languages
> Leadership Computing Facility
> Argonne National Laboratory
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20180709/f9985d9c/attachment-0001.html>