llvm dev - Jan 2018 - Testing End-To-End Functionality of Specific Optimization

Hello!

I am writing an instrumentation pass using LLVM. I have unit tests using 
llvm-lit in the style of the tests in llvm/test to make sure the IR 
changes show up in the correct place. I would also like to have some 
end-to-end tests that start from a .c file, compile the program, run it, 
and check the output.

I imagine I could do this as a test in the same style, but I'm wondering 
if that's the llvm-correct way of doing it. Can anyone advise?

Thank you!

Eli

Typically the "end-to-end" test cases reside in `projects/test-suite`.
All
the test cases there build executable binaries from source, run those and
compare the output (as well as the return code) to what's expected. You
should probably put your test cases in there. Please note also that there
are subcategories there - there are single source file and multiple source
file programs. There are those that are meant as benchmarks, unit tests,
full applications, etc. I'm sure you can find an appropriate place to add
your test cases to.

On Tue, Jan 2, 2018 at 8:16 PM, Eli Davis via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> Hello!
>
> I am writing an instrumentation pass using LLVM. I have unit tests using
> llvm-lit in the style of the tests in llvm/test to make sure the IR changes
> show up in the correct place. I would also like to have some end-to-end
> tests that start from a .c file, compile the program, run it, and check the
> output.
>
> I imagine I could do this as a test in the same style, but I'm
wondering
> if that's the llvm-correct way of doing it. Can anyone advise?
>
> Thank you!
>
> Eli
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20180102/218fe619/attachment.html>

Ah, thank you, that makes sense.

Is there any chance you can point me toward some documentation for how 
to add tests to the test-suite? I've found a fair bit on how to run the 
tests, but nothing on how to add new ones. Is it just as simple as 
adding testcase.c and testscase.reference_output in the appropriate 
directory?

Thanks!


On 01/02/2018 02:50 PM, Nemanja Ivanovic wrote:
> Typically the "end-to-end" test cases reside in
`projects/test-suite`.
> All the test cases there build executable binaries from source, run 
> those and compare the output (as well as the return code) to what's 
> expected. You should probably put your test cases in there. Please 
> note also that there are subcategories there - there are single source 
> file and multiple source file programs. There are those that are meant 
> as benchmarks, unit tests, full applications, etc. I'm sure you can 
> find an appropriate place to add your test cases to.
>
> On Tue, Jan 2, 2018 at 8:16 PM, Eli Davis via llvm-dev 
> <llvm-dev at lists.llvm.org <mailto:llvm-dev at
lists.llvm.org>> wrote:
>
>     Hello!
>
>     I am writing an instrumentation pass using LLVM. I have unit tests
>     using llvm-lit in the style of the tests in llvm/test to make sure
>     the IR changes show up in the correct place. I would also like to
>     have some end-to-end tests that start from a .c file, compile the
>     program, run it, and check the output.
>
>     I imagine I could do this as a test in the same style, but I'm
>     wondering if that's the llvm-correct way of doing it. Can anyone
>     advise?
>
>     Thank you!
>
>     Eli
>
>     _______________________________________________
>     LLVM Developers mailing list
>     llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
>     http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>     <http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20180102/67ea7b1c/attachment-0001.html>

Typically this would be tested alongside the runtime component in
compiler-rt.
For example, PGO instrumentation has its "end-to-end" tests in
https://github.com/llvm-mirror/compiler-rt/tree/master/test/profile

-- Sean Silva

On Tue, Jan 2, 2018 at 11:16 AM, Eli Davis via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> Hello!
>
> I am writing an instrumentation pass using LLVM. I have unit tests using
> llvm-lit in the style of the tests in llvm/test to make sure the IR changes
> show up in the correct place. I would also like to have some end-to-end
> tests that start from a .c file, compile the program, run it, and check the
> output.
>
> I imagine I could do this as a test in the same style, but I'm
wondering
> if that's the llvm-correct way of doing it. Can anyone advise?
>
> Thank you!
>
> Eli
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20180103/7a689e75/attachment.html>

Ah, that makes a lot of sense. Thank you! Do you have any tips (or 
points to documentation) for how to run compiler-rt tests? Just invoking 
llvm-lit doesn't seem to work


On 01/03/2018 11:17 PM, Sean Silva wrote:
> Typically this would be tested alongside the runtime component in 
> compiler-rt.
> For example, PGO instrumentation has its "end-to-end" tests in 
> https://github.com/llvm-mirror/compiler-rt/tree/master/test/profile
>
> -- Sean Silva
>
> On Tue, Jan 2, 2018 at 11:16 AM, Eli Davis via llvm-dev 
> <llvm-dev at lists.llvm.org <mailto:llvm-dev at
lists.llvm.org>> wrote:
>
>     Hello!
>
>     I am writing an instrumentation pass using LLVM. I have unit tests
>     using llvm-lit in the style of the tests in llvm/test to make sure
>     the IR changes show up in the correct place. I would also like to
>     have some end-to-end tests that start from a .c file, compile the
>     program, run it, and check the output.
>
>     I imagine I could do this as a test in the same style, but I'm
>     wondering if that's the llvm-correct way of doing it. Can anyone
>     advise?
>
>     Thank you!
>
>     Eli
>
>     _______________________________________________
>     LLVM Developers mailing list
>     llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
>     http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>     <http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20180104/d36b4194/attachment-0001.html>