llvm dev - Dec 2014 - [LLVMdev] Newbee question: LLVM backend regression tests for thumb1 targets on simulator possible?

I've been wondering too about how to get better ARM v6m compile-and-execute
testing going.

As you say Jon, the non-execution-based regression tests are surprisingly
good at catching issues; but they're no full substitute for executing the
code produced by the backend for a reasonably-sized test suite.

If somehow it would be possible to compile and run the LLVM test-suite
for v6m, I think that would be a good step forward. It would also allow
to get a buildbot going without too much effort continuously checking
basic correctness of v6m code generation.

My guess is that the biggest hurdle would be to get linux or a similar
operating system going on a v6m/thumb1 simulator. Does anyone have an idea
if this is feasible or completely impossible?

Thanks,

Kristof
> -----Original Message-----
> From: llvmdev-bounces at cs.uiuc.edu [mailto:llvmdev-bounces at
cs.uiuc.edu]
> On Behalf Of Jonathan Roelofs
> Sent: 15 December 2014 18:31
> To: Bjoern Haase; llvmdev at cs.uiuc.edu
> Subject: Re: [LLVMdev] Newbee question: LLVM backend regression tests
> for thumb1 targets on simulator possible?
> 
> 
> 
> On 12/15/14 4:29 AM, Bjoern Haase wrote:
> >
> > Hello,
> >
> > as a newbee, I'd appreciate some support on regression test setup.
> >
> > Specifically, I am interrested in the feature of tail call
> > optimizations for the ARM v6m targets. This feature currently seems to
> > be completely deactivated at the moment (v6m being based on thumb1
> > ?!). According to my code-reading, this feature will involve some
> > modifications in epilogue generation.
> >
> > My work on a gcc backend did show me that for a beginner like me, it
> > is rather likely that the first attempts will break something.
> :)
> >
> > Thus, being completely new to the llvm project I think that it's
> > essential to first establish a suitable regression test setup. My plan
> > is to run the tests for the v4t platform in thumb-only mode where free
> > simulators are available and which behaves (at least with respect to
> > the
> > epilogue) very much like v6m.
> Why not test your v6m changes on a Cortex-m0 QEMU? Semihosted
> applications are pretty easy to get going, something like:
> 
> $ qemu-system-arm -semihosting -M integratorcp -cpu cortex-m0 -kernel
> a.out
> 
> Thumv4t support is a bit spotty in llvm as it's not very well tested.
> The problem I run into most (and fixed several times), is that thumbv4t
> doesn't have a lo->lo mov instruction that doesn't clobber cpsr.
That
> being said, it does work for the most part... just something to watch
> out for.
> >
> >  From former work on gcc I'm used to the possibilty to run
regression
> > tests in a simulator for the target platform (there with the expect
> > script mechanism). In the documentation on the web, I did, however,
> > not yet find information on a comparable feature for llvm.
> >
> > Therefore the specific questions:
> >
> > - Is there a mechanism to run backend testsuite runs in a simulation
> > framework and if yes, could you give me a hint on how to find
> > information
> Not that I know of, no.
> 
> I routinely run the libc++ & libc++abi test suites in QEMU, but
that's
> more of a whole-toolchain test, rather that just a backend test. You can
> get surprisingly far without simulator testing, so for most things, LLVM
> doesn't use it for testing. This makes it easier for any random
> developer to check out the llvm sources, build them, and run the
> testsuite, and not have to have simulators installed for all the
> different backends.
> > - If not, how is regression testing for the thumb1 targets currently
> > implemented?
> We use the LIT framework http://llvm.org/docs/CommandGuide/lit.html to
> test the compiler. Mostly this means feeding in llvm IR, and using
> FileCheck http://llvm.org/docs/CommandGuide/FileCheck.html (which is
> basically a glorified
> grep) to verify that the assembly produced is the same as what the test
> expects to be generated.
> 
> To run these tests, use the 'make check-all' target from the build
> directory.
> 
> 
> Cheers,
> 
> Jon
> >
> > Yours,
> >
> > Björn Haase
> >
> > P.S.: Of course, any hint with respect to sibling call optimization
> > would also be appreciated in a second step. I actually already found
> > some helpful information in the sources.
> >
> > _______________________________________________
> > LLVM Developers mailing list
> > LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
> 
> --
> Jon Roelofs
> jonathan at codesourcery.com
> CodeSourcery / Mentor Embedded
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev

Am 16.12.2014 11:53, schrieb Kristof Beyls:
> If somehow it would be possible to compile and run the LLVM test-suite
> for v6m, I think that would be a good step forward. It would also allow
> to get a buildbot going without too much effort continuously checking
> basic correctness of v6m code generation.
IIRC, any linux will be requiring some MMU unit? If I understand 
correctly, the v6m instruction set is just a binary compatible subset of 
v7. So what about generating code with restrictions for v6m and running 
tests on some linux on, say, A15 machine?

Björn.

On 12/16/14 3:53 AM, Kristof Beyls wrote:
> I've been wondering too about how to get better ARM v6m
compile-and-execute
> testing going.
>
> As you say Jon, the non-execution-based regression tests are surprisingly
> good at catching issues; but they're no full substitute for executing
the
> code produced by the backend for a reasonably-sized test suite.
>
> If somehow it would be possible to compile and run the LLVM test-suite
> for v6m, I think that would be a good step forward. It would also allow
> to get a buildbot going without too much effort continuously checking
> basic correctness of v6m code generation.
>
> My guess is that the biggest hurdle would be to get linux or a similar
> operating system going on a v6m/thumb1 simulator. Does anyone have an idea
> if this is feasible or completely impossible?
I'm working on a baremetal cross toolchain, and I would like to set up such
a
buildbot. There are several pieces needed before that can happen though, and the
important one is remote testing support in LIT (which I'm working on)... in
due
time :)

Jon

-- 
Jon Roelofs
jonathan at codesourcery.com
CodeSourcery / Mentor Embedded

Am 16.12.2014 15:27, schrieb Jonathan Roelofs:
> I'm working on a baremetal cross toolchain, and I would like to set up 
> such a buildbot. There are several pieces needed before that can 
> happen though, and the important one is remote testing support in LIT 
> (which I'm working on)... in due time :)
>
> Jon
This would be perfect.

Actually, you suggested to use

 > $qemu-system-arm -semihosting -M integratorcp -cpu cortex-m0 -kernel 
a.out

You have some special version of qemu? My default distribution's qemu 
package did not have a cortex-M0 nor has qemu head freshly taken from git:

$ qemu-arm -cpu ?
Available CPUs:
   arm926  arm946 arm1026  arm1136  arm1136-r2 arm1176  arm11mpcore 
cortex-m3 cortex-a8
   cortex-a8-r2 cortex-a9 cortex-a15 ti925t pxa250 sa1100 sa1110 pxa255 
pxa260
   pxa261 pxa262 pxa270 pxa270-a0 pxa270-a1 pxa270-b0 pxa270-b1 xa270-c0
   pxa270-c5 any

Still your procedure will work, just by using the m3 instead :-).

Yours,

Björn.

On Wed Dec 17 2014 at 10:51:51 AM Kristof Beyls <kristof.beyls at arm.com>
wrote:
> I've been wondering too about how to get better ARM v6m
compile-and-execute
> testing going.
>
> As you say Jon, the non-execution-based regression tests are surprisingly
> good at catching issues; but they're no full substitute for executing
the
> code produced by the backend for a reasonably-sized test suite.
>
> If somehow it would be possible to compile and run the LLVM test-suite
> for v6m, I think that would be a good step forward. It would also allow
> to get a buildbot going without too much effort continuously checking
> basic correctness of v6m code generation.
>
> My guess is that the biggest hurdle would be to get linux or a similar
> operating system going on a v6m/thumb1 simulator. Does anyone have an idea
> if this is feasible or completely impossible?
>
>
Not sure feasibility of the particular implementation, but qemu + linux
works pretty well for testing other targets.

-eric

> Thanks,
>
> Kristof
>
> > -----Original Message-----
> > From: llvmdev-bounces at cs.uiuc.edu [mailto:llvmdev-bounces at
cs.uiuc.edu]
> > On Behalf Of Jonathan Roelofs
> > Sent: 15 December 2014 18:31
> > To: Bjoern Haase; llvmdev at cs.uiuc.edu
> > Subject: Re: [LLVMdev] Newbee question: LLVM backend regression tests
> > for thumb1 targets on simulator possible?
> >
> >
> >
> > On 12/15/14 4:29 AM, Bjoern Haase wrote:
> > >
> > > Hello,
> > >
> > > as a newbee, I'd appreciate some support on regression test
setup.
> > >
> > > Specifically, I am interrested in the feature of tail call
> > > optimizations for the ARM v6m targets. This feature currently
seems to
> > > be completely deactivated at the moment (v6m being based on
thumb1
> > > ?!). According to my code-reading, this feature will involve some
> > > modifications in epilogue generation.
> > >
> > > My work on a gcc backend did show me that for a beginner like me,
it
> > > is rather likely that the first attempts will break something.
> > :)
> > >
> > > Thus, being completely new to the llvm project I think that
it's
> > > essential to first establish a suitable regression test setup. My
plan
> > > is to run the tests for the v4t platform in thumb-only mode where
free
> > > simulators are available and which behaves (at least with respect
to
> > > the
> > > epilogue) very much like v6m.
> > Why not test your v6m changes on a Cortex-m0 QEMU? Semihosted
> > applications are pretty easy to get going, something like:
> >
> > $ qemu-system-arm -semihosting -M integratorcp -cpu cortex-m0 -kernel
> > a.out
> >
> > Thumv4t support is a bit spotty in llvm as it's not very well
tested.
> > The problem I run into most (and fixed several times), is that
thumbv4t
> > doesn't have a lo->lo mov instruction that doesn't clobber
cpsr. That
> > being said, it does work for the most part... just something to watch
> > out for.
> > >
> > >  From former work on gcc I'm used to the possibilty to run
regression
> > > tests in a simulator for the target platform (there with the
expect
> > > script mechanism). In the documentation on the web, I did,
however,
> > > not yet find information on a comparable feature for llvm.
> > >
> > > Therefore the specific questions:
> > >
> > > - Is there a mechanism to run backend testsuite runs in a
simulation
> > > framework and if yes, could you give me a hint on how to find
> > > information
> > Not that I know of, no.
> >
> > I routinely run the libc++ & libc++abi test suites in QEMU, but
that's
> > more of a whole-toolchain test, rather that just a backend test. You
can
> > get surprisingly far without simulator testing, so for most things,
LLVM
> > doesn't use it for testing. This makes it easier for any random
> > developer to check out the llvm sources, build them, and run the
> > testsuite, and not have to have simulators installed for all the
> > different backends.
> > > - If not, how is regression testing for the thumb1 targets
currently
> > > implemented?
> > We use the LIT framework http://llvm.org/docs/CommandGuide/lit.html to
> > test the compiler. Mostly this means feeding in llvm IR, and using
> > FileCheck http://llvm.org/docs/CommandGuide/FileCheck.html (which is
> > basically a glorified
> > grep) to verify that the assembly produced is the same as what the
test
> > expects to be generated.
> >
> > To run these tests, use the 'make check-all' target from the
build
> > directory.
> >
> >
> > Cheers,
> >
> > Jon
> > >
> > > Yours,
> > >
> > > Björn Haase
> > >
> > > P.S.: Of course, any hint with respect to sibling call
optimization
> > > would also be appreciated in a second step. I actually already
found
> > > some helpful information in the sources.
> > >
> > > _______________________________________________
> > > LLVM Developers mailing list
> > > LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> > > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
> >
> > --
> > Jon Roelofs
> > jonathan at codesourcery.com
> > CodeSourcery / Mentor Embedded
> > _______________________________________________
> > LLVM Developers mailing list
> > LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>
>
>
>
>
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20141217/8a5f547e/attachment.html>