llvm dev - Feb 2011 - [LLVMdev] Running the programs in the LLVM test suite with a pass and without it.

Hi guys,

I have a pass called "MyPass". And I can run this pass in the LLVM
test
suite programs and produce a report with the statistics of the execution of
this pass. Now, I would like to run the programs in LLVM test suite and
generate a report comparing the results obtained without using the pass with
the results obtained using the pass. Is there any way to do this?

Thank you,

Douglas
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20110207/d07c4bfe/attachment.html>

Hi Douglas,

Yes, you can use the "llc-beta" test suite option to do what
you're looking for. That runs the tests with and without a set of command
line options and provides a comparison.

1. Make your pass disabled by default and enabled via a command line option
(e.g., --enable-mypass),
2. Tell the test-suite Makefiles to run your pass as the beta option*,. Look in
<llvm/projects/test-suite/Makefile.programs*,
3. Run the testsuite in night test mode (the "simple" mode does not
support beta options at this time) via something like "make TEST=nightly
report.html",
4. Open the report to see your results.


* Look in <llvm/projects/test-suite/Makefile.programs. The options are
enabled by target, search for "BETAOPTION" to find them. You'll
see examples for setting the LLC beta option via LLCBETAOPTION. If your pass is
a machine function pass, that's where you'll want to add it. If it's
run by 'opt' instead, you'll want to use OPTBETAOPTION.

Regards,
-Jim


On Feb 7, 2011, at 4:20 AM, Douglas do Couto Teixeira wrote:
> Hi guys,
> 
> I have a pass called "MyPass". And I can run this pass in the
LLVM test suite programs and produce a report with the statistics of the
execution of this pass. Now, I would like to run the programs in LLVM test suite
and generate a report comparing the results obtained without using the pass with
the results obtained using the pass. Is there any way to do this?
> 
> Thank you,
> 
> Douglas
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev

On Feb 7, 2011, at 4:20 AM, Douglas do Couto Teixeira wrote:
> Hi guys,
> 
> I have a pass called "MyPass". And I can run this pass in the
LLVM test suite programs and produce a report with the statistics of the
execution of this pass. Now, I would like to run the programs in LLVM test suite
and generate a report comparing the results obtained without using the pass with
the results obtained using the pass. Is there any way to do this?
If by "stats" you mean execution time, then you may use the Makefile
support for "llc beta" as Jim suggested. I use the following without
editing the Makefiles:

TEST=nightly ENABLE_LLCBETA=1 LLCBETAOPTION="-my-pass" report.csv

To compare across separate runs, I just load the report.csv into a spreadsheet
and define some formulas.

If you actually want to compare output from the -stats option, then you can
compare the *.llc.s.info files (one for each benchmark) from two separate runs.
I attached the script that I use to do this.

-Andy

-------------- next part --------------
A non-text attachment was scrubbed...
Name: comparestats.py
Type: text/x-python-script
Size: 2416 bytes
Desc: not available
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20110209/0b90ce5b/attachment.bin>