llvm dev - Dec 2014 - [LLVMdev] Newbee question: LLVM backend regression tests for thumb1 targets on simulator possible?

Hello,

as a newbee, I'd appreciate some support on regression test setup.

Specifically, I am interrested in the feature of tail call optimizations for
the ARM v6m targets. This feature currently seems to be completely
deactivated at the moment (v6m being based on thumb1 ?!). According to my
code-reading, this feature will involve some modifications in epilogue
generation.

My work on a gcc backend did show me that for a beginner like me, it is rather
likely that the first attempts will break something.

Thus, being completely new to the llvm project I think that it's
essential to first establish a suitable regression test setup. My plan
is to run the tests for the v4t platform in thumb-only mode where free
simulators are available and which behaves (at least with respect to the
epilogue) very much like v6m.

 From former work on gcc I'm used to the possibilty to run regression
tests in a simulator for the target platform (there with the expect
script mechanism). In the documentation on the web, I did, however, not
yet find information on a comparable feature for llvm.

Therefore the specific questions:

- Is there a mechanism to run backend testsuite runs in a simulation
framework and if yes, could you give me a hint on how to find information
- If not, how is regression testing for the thumb1 targets currently
implemented?

Yours,

Björn Haase

P.S.: Of course, any hint with respect to sibling call optimization
would also be appreciated in a second step. I actually already found some
helpful
information in the sources.

On 12/15/14 4:29 AM, Bjoern Haase wrote:
>
> Hello,
>
> as a newbee, I'd appreciate some support on regression test setup.
>
> Specifically, I am interrested in the feature of tail call optimizations
for
> the ARM v6m targets. This feature currently seems to be completely
> deactivated at the moment (v6m being based on thumb1 ?!). According to my
> code-reading, this feature will involve some modifications in epilogue
> generation.
>
> My work on a gcc backend did show me that for a beginner like me, it is
rather
> likely that the first attempts will break something.
:)
>
> Thus, being completely new to the llvm project I think that it's
> essential to first establish a suitable regression test setup. My plan
> is to run the tests for the v4t platform in thumb-only mode where free
> simulators are available and which behaves (at least with respect to the
> epilogue) very much like v6m.
Why not test your v6m changes on a Cortex-m0 QEMU? Semihosted applications are 
pretty easy to get going, something like:

$ qemu-system-arm -semihosting -M integratorcp -cpu cortex-m0 -kernel a.out

Thumv4t support is a bit spotty in llvm as it's not very well tested. The 
problem I run into most (and fixed several times), is that thumbv4t doesn't
have
a lo->lo mov instruction that doesn't clobber cpsr. That being said, it
does
work for the most part... just something to watch out
for.
>
>  From former work on gcc I'm used to the possibilty to run regression
> tests in a simulator for the target platform (there with the expect
> script mechanism). In the documentation on the web, I did, however, not
> yet find information on a comparable feature for llvm.
>
> Therefore the specific questions:
>
> - Is there a mechanism to run backend testsuite runs in a simulation
> framework and if yes, could you give me a hint on how to find information
Not that I know of, no.

I routinely run the libc++ & libc++abi test suites in QEMU, but that's
more of a
whole-toolchain test, rather that just a backend test. You can get surprisingly 
far without simulator testing, so for most things, LLVM doesn't use it for 
testing. This makes it easier for any random developer to check out the llvm 
sources, build them, and run the testsuite, and not have to have simulators 
installed for all the different backends.
> - If not, how is regression testing for the thumb1 targets currently
> implemented?
We use the LIT framework http://llvm.org/docs/CommandGuide/lit.html to test the 
compiler. Mostly this means feeding in llvm IR, and using FileCheck 
http://llvm.org/docs/CommandGuide/FileCheck.html (which is basically a glorified
grep) to verify that the assembly produced is the same as what the test expects 
to be generated.

To run these tests, use the 'make check-all' target from the build
directory.


Cheers,

Jon
>
> Yours,
>
> Björn Haase
>
> P.S.: Of course, any hint with respect to sibling call optimization
> would also be appreciated in a second step. I actually already found some
helpful
> information in the sources.
>
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
-- 
Jon Roelofs
jonathan at codesourcery.com
CodeSourcery / Mentor Embedded

Am 15.12.2014 19:31, schrieb Jonathan Roelofs:
> Why not test your v6m changes on a Cortex-m0 QEMU? Semihosted 
> applications are pretty easy to get going, something like:
>
> $ qemu-system-arm -semihosting -M integratorcp -cpu cortex-m0 -kernel 
> a.out
>
Thank you, Jonathan, for your reply. I will be checking this out. To my 
knowledge there was no free simulator for cortex M0! Great to see that 
there *is* an option.
> Thumv4t support is a bit spotty in llvm as it's not very well tested. 
> The problem I run into most (and fixed several times), is that 
> thumbv4t doesn't have a lo->lo mov instruction that doesn't
clobber
> cpsr. That being said, it does work for the most part... just 
> something to watch out for.
>>
OK, good to know this. I considered using v4t only because I assumed 
that licensing prohibited free simulators for v6m.
>
> I routinely run the libc++ & libc++abi test suites in QEMU, but
that's
> more of a whole-toolchain test, rather that just a backend test. You 
> can get surprisingly far without simulator testing, so for most 
> things, LLVM doesn't use it for testing. This makes it easier for any 
> random developer to check out the llvm sources, build them, and run 
> the testsuite, and not have to have simulators installed for all the 
> different backends.
>> - If not, how is regression testing for the thumb1 targets currently
>> implemented?
> We use the LIT framework http://llvm.org/docs/CommandGuide/lit.html to 
> test the compiler. Mostly this means feeding in llvm IR, and using 
> FileCheck http://llvm.org/docs/CommandGuide/FileCheck.html (which is 
> basically a glorified grep) to verify that the assembly produced is 
> the same as what the test expects to be generated.
>
OK, I will look after this, It might take me some time to understand the 
mechanisms.

Thank's again for your support.

Björn.

I've been wondering too about how to get better ARM v6m compile-and-execute
testing going.

As you say Jon, the non-execution-based regression tests are surprisingly
good at catching issues; but they're no full substitute for executing the
code produced by the backend for a reasonably-sized test suite.

If somehow it would be possible to compile and run the LLVM test-suite
for v6m, I think that would be a good step forward. It would also allow
to get a buildbot going without too much effort continuously checking
basic correctness of v6m code generation.

My guess is that the biggest hurdle would be to get linux or a similar
operating system going on a v6m/thumb1 simulator. Does anyone have an idea
if this is feasible or completely impossible?

Thanks,

Kristof
> -----Original Message-----
> From: llvmdev-bounces at cs.uiuc.edu [mailto:llvmdev-bounces at
cs.uiuc.edu]
> On Behalf Of Jonathan Roelofs
> Sent: 15 December 2014 18:31
> To: Bjoern Haase; llvmdev at cs.uiuc.edu
> Subject: Re: [LLVMdev] Newbee question: LLVM backend regression tests
> for thumb1 targets on simulator possible?
> 
> 
> 
> On 12/15/14 4:29 AM, Bjoern Haase wrote:
> >
> > Hello,
> >
> > as a newbee, I'd appreciate some support on regression test setup.
> >
> > Specifically, I am interrested in the feature of tail call
> > optimizations for the ARM v6m targets. This feature currently seems to
> > be completely deactivated at the moment (v6m being based on thumb1
> > ?!). According to my code-reading, this feature will involve some
> > modifications in epilogue generation.
> >
> > My work on a gcc backend did show me that for a beginner like me, it
> > is rather likely that the first attempts will break something.
> :)
> >
> > Thus, being completely new to the llvm project I think that it's
> > essential to first establish a suitable regression test setup. My plan
> > is to run the tests for the v4t platform in thumb-only mode where free
> > simulators are available and which behaves (at least with respect to
> > the
> > epilogue) very much like v6m.
> Why not test your v6m changes on a Cortex-m0 QEMU? Semihosted
> applications are pretty easy to get going, something like:
> 
> $ qemu-system-arm -semihosting -M integratorcp -cpu cortex-m0 -kernel
> a.out
> 
> Thumv4t support is a bit spotty in llvm as it's not very well tested.
> The problem I run into most (and fixed several times), is that thumbv4t
> doesn't have a lo->lo mov instruction that doesn't clobber cpsr.
That
> being said, it does work for the most part... just something to watch
> out for.
> >
> >  From former work on gcc I'm used to the possibilty to run
regression
> > tests in a simulator for the target platform (there with the expect
> > script mechanism). In the documentation on the web, I did, however,
> > not yet find information on a comparable feature for llvm.
> >
> > Therefore the specific questions:
> >
> > - Is there a mechanism to run backend testsuite runs in a simulation
> > framework and if yes, could you give me a hint on how to find
> > information
> Not that I know of, no.
> 
> I routinely run the libc++ & libc++abi test suites in QEMU, but
that's
> more of a whole-toolchain test, rather that just a backend test. You can
> get surprisingly far without simulator testing, so for most things, LLVM
> doesn't use it for testing. This makes it easier for any random
> developer to check out the llvm sources, build them, and run the
> testsuite, and not have to have simulators installed for all the
> different backends.
> > - If not, how is regression testing for the thumb1 targets currently
> > implemented?
> We use the LIT framework http://llvm.org/docs/CommandGuide/lit.html to
> test the compiler. Mostly this means feeding in llvm IR, and using
> FileCheck http://llvm.org/docs/CommandGuide/FileCheck.html (which is
> basically a glorified
> grep) to verify that the assembly produced is the same as what the test
> expects to be generated.
> 
> To run these tests, use the 'make check-all' target from the build
> directory.
> 
> 
> Cheers,
> 
> Jon
> >
> > Yours,
> >
> > Björn Haase
> >
> > P.S.: Of course, any hint with respect to sibling call optimization
> > would also be appreciated in a second step. I actually already found
> > some helpful information in the sources.
> >
> > _______________________________________________
> > LLVM Developers mailing list
> > LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
> 
> --
> Jon Roelofs
> jonathan at codesourcery.com
> CodeSourcery / Mentor Embedded
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev