llvm dev - Apr 2020 - Adding a new External Suite to test-suite

Greetings LLVMers,

I would like to add the licensed part of AnghaBench
(http://cuda.dcc.ufmg.br/angha/) as an External Test in the LLVM
test-suite. We have 128,411 files with their original licenses.
(http://www.dcc.ufmg.br/~fernando/coisas/c_files_with_licenses.tar.gz)
Each file is a single function, that compiles as is---no dependencies are
needed. For instance, the commands below work for each individual
file:

$ clang $compile_flags -Xclang -disable-O0-optnone -S -c -emit-llvm \
    "$file" -o "$individual_bc"
$ opt -mem2reg -O0 ${user_passes[@]} -instcount -stats -S \
    "$individual_bc" -disable-output 2>&1"

So, these are my questions:

Q1: What do I have to do to start integrating my collection of .c files as
an
External Suite? They are all single source, single function,
compilable headerless files extracted from open source repositories.
They are grouped in folders, in a similar way the original project
was.

Q2: I read that using glob might not be a good idea, even though there is a
partial workaround. However, when trying to understand what the other
External
suites were doing, I noticed several of them were using it.
May I ignore the note at
https://cmake.org/cmake/help/latest/command/file.html#glob ?

Q3: The C files were not extracted/reconstructed with execution in mind.
For now, we're mainly interested in collecting stats and testing the
compiler. Since we don't generate executables, should I call cmake's
add_library() for each .c file?

Kind regards,
Bruno Kind
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20200406/0c5478bb/attachment.html>

I suggest to have a look into the CMakeLists.txt of the test-suite.
These are mostly self-explanatory.

Am Mo., 6. Apr. 2020 um 04:25 Uhr schrieb Bruno Conde Kind via
llvm-dev <llvm-dev at lists.llvm.org>:
> Q1: What do I have to do to start integrating my collection of .c files as
an
> External Suite? They are all single source, single function,
> compilable headerless files extracted from open source repositories.
> They are grouped in folders, in a similar way the original project
> was.
External tests are in
https://github.com/llvm/llvm-test-suite/tree/master/External.
Tests where each source file is one executable are in
https://github.com/llvm/llvm-test-suite/tree/master/SingleSource
Tests where each source file in a directory compiles to object files
that is linked to an executable are in
https://github.com/llvm/llvm-test-suite/tree/master/MultiSource

> Q2: I read that using glob might not be a good idea, even though there is a
> partial workaround. However, when trying to understand what the other
External
> suites were doing, I noticed several of them were using it.
> May I ignore the note at
> https://cmake.org/cmake/help/latest/command/file.html#glob ?
The typical use case of the test-suite is to configure and build it
from scratch every time with a different compiler executable. There is
no typical edit-compile-debug cycle.
> Q3: The C files were not extracted/reconstructed with execution in mind.
> For now, we're mainly interested in collecting stats and testing the
> compiler. Since we don't generate executables, should I call
cmake's
> add_library() for each .c file?
See compile time benchmarks in
https://github.com/llvm/llvm-test-suite/tree/master/CTMark

Are the C files intended to compile-time benchmarking as well? If not,
what are they useful for?

Michael

On 4/6/20 8:46 AM, Michael Kruse via llvm-dev wrote:
> I suggest to have a look into the CMakeLists.txt of the test-suite.
> These are mostly self-explanatory.
>
> Am Mo., 6. Apr. 2020 um 04:25 Uhr schrieb Bruno Conde Kind via
> llvm-dev <llvm-dev at lists.llvm.org>:
>> Q1: What do I have to do to start integrating my collection of .c files
as an
>> External Suite? They are all single source, single function,
>> compilable headerless files extracted from open source repositories.
>> They are grouped in folders, in a similar way the original project
>> was.
> External tests are in
> https://github.com/llvm/llvm-test-suite/tree/master/External.
> Tests where each source file is one executable are in
> https://github.com/llvm/llvm-test-suite/tree/master/SingleSource
> Tests where each source file in a directory compiles to object files
> that is linked to an executable are in
> https://github.com/llvm/llvm-test-suite/tree/master/MultiSource
>
>
>> Q2: I read that using glob might not be a good idea, even though there
is a
>> partial workaround. However, when trying to understand what the other
External
>> suites were doing, I noticed several of them were using it.
>> May I ignore the note at
>> https://cmake.org/cmake/help/latest/command/file.html#glob ?
> The typical use case of the test-suite is to configure and build it
> from scratch every time with a different compiler executable. There is
> no typical edit-compile-debug cycle.
>
>> Q3: The C files were not extracted/reconstructed with execution in
mind.
>> For now, we're mainly interested in collecting stats and testing
the
>> compiler. Since we don't generate executables, should I call
cmake's
>> add_library() for each .c file?
> See compile time benchmarks in
> https://github.com/llvm/llvm-test-suite/tree/master/CTMark
>
> Are the C files intended to compile-time benchmarking as well? If not,
> what are they useful for?
I'd also like to know what the intention here is. What is tested and how?

Looking at a few of these it seems there is not much you can do as it is

little code with a lot of unknown function calls and global symbols.


Cheers,

   Johannes


> Michael
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev