llvm dev - Sep 2010 - [LLVMdev] Need advise on adding tests - Was: Re: ARM MC .s status?

Hi everyone,

I am trying to get up to speed on the MC object file emission for ARM,
the first cut being for ELF, and the testing required for that.

Obviously, we want the tests for the .o emission to ultimately test
the entire .ll -> many llvm passes -> .bc -> .o, but as a first cut,
my instinct tells me that a simple .cpp unit tests that directly
invokes the MC code to generate (and self-check) the ARM ELF might be
the way to go, but I am wondering where the best place for something
like this is? I.e. From what I can tell, in the test-suite, even
single file .c files are (according to the readme at least) are run as
integration tests that compare llvm .bc / lli output to th eoutput of
a native compile ...

The testing document in llvm.org only talk about pretty heavy weight
integration tests (Tcl + FileCheck, .cpp -> (many compilers  ...)
which is fine, but, but as a sanity check, I can't seem to find an
example of a test.cpp file that invokes a certain subset of LLVM API,
and self-checks the result - Or is the heavyweight "compile a .cpp
file and cross check LLVM-clang's output to gcc" the only way
something like this supported?

Thanks,

-Jason.
-jason

Is the rationale that llvm has matured/advanced to the point where
integration tests are the only ones to make sense from a test density
perspective?

On Thu, Sep 16, 2010 at 5:05 PM, Jason Kim <jasonwkim at google.com>
wrote:
> Hi everyone,
>
> I am trying to get up to speed on the MC object file emission for ARM,
> the first cut being for ELF, and the testing required for that.
>
> Obviously, we want the tests for the .o emission to ultimately test
> the entire .ll -> many llvm passes -> .bc -> .o, but as a first
cut,
> my instinct tells me that a simple .cpp unit tests that directly
> invokes the MC code to generate (and self-check) the ARM ELF might be
> the way to go, but I am wondering where the best place for something
> like this is? I.e. From what I can tell, in the test-suite, even
> single file .c files are (according to the readme at least) are run as
> integration tests that compare llvm .bc / lli output to th eoutput of
> a native compile ...
>
> The testing document in llvm.org only talk about pretty heavy weight
> integration tests (Tcl + FileCheck, .cpp -> (many compilers  ...)
> which is fine, but, but as a sanity check, I can't seem to find an
> example of a test.cpp file that invokes a certain subset of LLVM API,
> and self-checks the result - Or is the heavyweight "compile a .cpp
> file and cross check LLVM-clang's output to gcc" the only way
> something like this supported?
>
> Thanks,
>
> -Jason.
> -jason
>

On Sep 16, 2010, at 5:30 PM, Jason Kim wrote:
> Is the rationale that llvm has matured/advanced to the point where
> integration tests are the only ones to make sense from a test density
> perspective?
We prefer unit tests based on decomposed command line tools in the llvm/test
directory.  These are simple to write, maintain, and understand/debug when
something breaks.

This approach works great for regression and simple feature tests.  More
elaborate things get put into other test suites.  Is this what you mean?

-Chris
> 
> On Thu, Sep 16, 2010 at 5:05 PM, Jason Kim <jasonwkim at google.com>
wrote:
>> Hi everyone,
>> 
>> I am trying to get up to speed on the MC object file emission for ARM,
>> the first cut being for ELF, and the testing required for that.
>> 
>> Obviously, we want the tests for the .o emission to ultimately test
>> the entire .ll -> many llvm passes -> .bc -> .o, but as a
first cut,
>> my instinct tells me that a simple .cpp unit tests that directly
>> invokes the MC code to generate (and self-check) the ARM ELF might be
>> the way to go, but I am wondering where the best place for something
>> like this is? I.e. From what I can tell, in the test-suite, even
>> single file .c files are (according to the readme at least) are run as
>> integration tests that compare llvm .bc / lli output to th eoutput of
>> a native compile ...
>> 
>> The testing document in llvm.org only talk about pretty heavy weight
>> integration tests (Tcl + FileCheck, .cpp -> (many compilers  ...)
>> which is fine, but, but as a sanity check, I can't seem to find an
>> example of a test.cpp file that invokes a certain subset of LLVM API,
>> and self-checks the result - Or is the heavyweight "compile a .cpp
>> file and cross check LLVM-clang's output to gcc" the only way
>> something like this supported?
>> 
>> Thanks,
>> 
>> -Jason.
>> -jason
>> 
> 
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev

We do tend to focus on the integration test style, but using our own
tools which allow decomposing commands. Take a look at the tests in
the tests/MC directory for examples of MC tests. Most tests are pretty
lightweight, definitely much much lighter than .cpp -> .o.

As far as ARM / MC goes, we are still just in the preliminary stages
of getting things working. We have some instruction parsing support,
still need substantial work on instruction matching, and still need
substantial work in the encoder and backend. We also still need
substantial work on the codegen side to MC'ize things.

 - Daniel

On Thu, Sep 16, 2010 at 5:05 PM, Jason Kim <jasonwkim at google.com>
wrote:
> Hi everyone,
>
> I am trying to get up to speed on the MC object file emission for ARM,
> the first cut being for ELF, and the testing required for that.
>
> Obviously, we want the tests for the .o emission to ultimately test
> the entire .ll -> many llvm passes -> .bc -> .o, but as a first
cut,
> my instinct tells me that a simple .cpp unit tests that directly
> invokes the MC code to generate (and self-check) the ARM ELF might be
> the way to go, but I am wondering where the best place for something
> like this is? I.e. From what I can tell, in the test-suite, even
> single file .c files are (according to the readme at least) are run as
> integration tests that compare llvm .bc / lli output to th eoutput of
> a native compile ...
>
> The testing document in llvm.org only talk about pretty heavy weight
> integration tests (Tcl + FileCheck, .cpp -> (many compilers  ...)
> which is fine, but, but as a sanity check, I can't seem to find an
> example of a test.cpp file that invokes a certain subset of LLVM API,
> and self-checks the result - Or is the heavyweight "compile a .cpp
> file and cross check LLVM-clang's output to gcc" the only way
> something like this supported?
>
> Thanks,
>
> -Jason.
> -jason
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>

On 16 September 2010 20:30, Jason Kim <jasonwkim at google.com>
wrote:
> Is the rationale that llvm has matured/advanced to the point where
> integration tests are the only ones to make sense from a test density
> perspective?
Since you are working on producing .o files, I would suggest first
getting llvm-mc to the point where it can produce some very basic .o
files from a .s (maybe an empty .s?). With that you can start adding
tests to test/MC/ELF and checking the output with elf-dump.

Cheers,
-- 
Rafael Ávila de Espíndola