llvm dev - Jan 2018 - Testing End-To-End Functionality of Specific Optimization

Ah, thank you, that makes sense.

Is there any chance you can point me toward some documentation for how 
to add tests to the test-suite? I've found a fair bit on how to run the 
tests, but nothing on how to add new ones. Is it just as simple as 
adding testcase.c and testscase.reference_output in the appropriate 
directory?

Thanks!


On 01/02/2018 02:50 PM, Nemanja Ivanovic wrote:
> Typically the "end-to-end" test cases reside in
`projects/test-suite`.
> All the test cases there build executable binaries from source, run 
> those and compare the output (as well as the return code) to what's 
> expected. You should probably put your test cases in there. Please 
> note also that there are subcategories there - there are single source 
> file and multiple source file programs. There are those that are meant 
> as benchmarks, unit tests, full applications, etc. I'm sure you can 
> find an appropriate place to add your test cases to.
>
> On Tue, Jan 2, 2018 at 8:16 PM, Eli Davis via llvm-dev 
> <llvm-dev at lists.llvm.org <mailto:llvm-dev at
lists.llvm.org>> wrote:
>
>     Hello!
>
>     I am writing an instrumentation pass using LLVM. I have unit tests
>     using llvm-lit in the style of the tests in llvm/test to make sure
>     the IR changes show up in the correct place. I would also like to
>     have some end-to-end tests that start from a .c file, compile the
>     program, run it, and check the output.
>
>     I imagine I could do this as a test in the same style, but I'm
>     wondering if that's the llvm-correct way of doing it. Can anyone
>     advise?
>
>     Thank you!
>
>     Eli
>
>     _______________________________________________
>     LLVM Developers mailing list
>     llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
>     http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>     <http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20180102/67ea7b1c/attachment-0001.html>

I am far from an expert on this since I've only ever added one or two of
these IIRC. But as far as I remember, for most purposes, that's all you
need. Add a source file, add an expected file and add the source file to
the CMakeLists.txt (along with any additional CFLAGS/CXXFLAGS/LDFLAGS)...
Of course, if you need anything custom, there are likely examples in the
test suite that you can use for guidance.
Also, this may be of some help:
https://llvm.org/docs/TestSuiteMakefileGuide.html

On Tue, Jan 2, 2018 at 9:14 PM, Eli Davis <eli at aarno-labs.com> wrote:
> Ah, thank you, that makes sense.
>
> Is there any chance you can point me toward some documentation for how to
> add tests to the test-suite? I've found a fair bit on how to run the
tests,
> but nothing on how to add new ones. Is it just as simple as adding
> testcase.c and testscase.reference_output in the appropriate directory?
>
> Thanks!
>
> On 01/02/2018 02:50 PM, Nemanja Ivanovic wrote:
>
> Typically the "end-to-end" test cases reside in
`projects/test-suite`. All
> the test cases there build executable binaries from source, run those and
> compare the output (as well as the return code) to what's expected. You
> should probably put your test cases in there. Please note also that there
> are subcategories there - there are single source file and multiple source
> file programs. There are those that are meant as benchmarks, unit tests,
> full applications, etc. I'm sure you can find an appropriate place to
add
> your test cases to.
>
> On Tue, Jan 2, 2018 at 8:16 PM, Eli Davis via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>> Hello!
>>
>> I am writing an instrumentation pass using LLVM. I have unit tests
using
>> llvm-lit in the style of the tests in llvm/test to make sure the IR
changes
>> show up in the correct place. I would also like to have some end-to-end
>> tests that start from a .c file, compile the program, run it, and check
the
>> output.
>>
>> I imagine I could do this as a test in the same style, but I'm
wondering
>> if that's the llvm-correct way of doing it. Can anyone advise?
>>
>> Thank you!
>>
>> Eli
>>
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20180102/14d7baf1/attachment.html>

Thank you very much. I'll give it a shot!


On 01/02/2018 03:25 PM, Nemanja Ivanovic wrote:
> I am far from an expert on this since I've only ever added one or two 
> of these IIRC. But as far as I remember, for most purposes, that's all 
> you need. Add a source file, add an expected file and add the source 
> file to the CMakeLists.txt (along with any additional 
> CFLAGS/CXXFLAGS/LDFLAGS)...
> Of course, if you need anything custom, there are likely examples in 
> the test suite that you can use for guidance.
> Also, this may be of some help:
> https://llvm.org/docs/TestSuiteMakefileGuide.html
>
> On Tue, Jan 2, 2018 at 9:14 PM, Eli Davis <eli at aarno-labs.com 
> <mailto:eli at aarno-labs.com>> wrote:
>
>     Ah, thank you, that makes sense.
>
>     Is there any chance you can point me toward some documentation for
>     how to add tests to the test-suite? I've found a fair bit on how
>     to run the tests, but nothing on how to add new ones. Is it just
>     as simple as adding testcase.c and testscase.reference_output in
>     the appropriate directory?
>
>     Thanks!
>
>
>     On 01/02/2018 02:50 PM, Nemanja Ivanovic wrote:
>>     Typically the "end-to-end" test cases reside in
>>     `projects/test-suite`. All the test cases there build executable
>>     binaries from source, run those and compare the output (as well
>>     as the return code) to what's expected. You should probably put
>>     your test cases in there. Please note also that there are
>>     subcategories there - there are single source file and multiple
>>     source file programs. There are those that are meant as
>>     benchmarks, unit tests, full applications, etc. I'm sure you
can
>>     find an appropriate place to add your test cases to.
>>
>>     On Tue, Jan 2, 2018 at 8:16 PM, Eli Davis via llvm-dev
>>     <llvm-dev at lists.llvm.org <mailto:llvm-dev at
lists.llvm.org>> wrote:
>>
>>         Hello!
>>
>>         I am writing an instrumentation pass using LLVM. I have unit
>>         tests using llvm-lit in the style of the tests in llvm/test
>>         to make sure the IR changes show up in the correct place. I
>>         would also like to have some end-to-end tests that start from
>>         a .c file, compile the program, run it, and check the output.
>>
>>         I imagine I could do this as a test in the same style, but
>>         I'm wondering if that's the llvm-correct way of doing
it. Can
>>         anyone advise?
>>
>>         Thank you!
>>
>>         Eli
>>
>>         _______________________________________________
>>         LLVM Developers mailing list
>>         llvm-dev at lists.llvm.org <mailto:llvm-dev at
lists.llvm.org>
>>         http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>         <http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20180102/04c984b5/attachment-0001.html>