llvm dev - May 2008 - [LLVMdev] Using the test suite to benchmark patches

On Wed, 21 May 2008, Mike Stump wrote:
> On May 21, 2008, at 8:09 AM, Matthijs Kooijman wrote:
>> Any thoughts or suggestions on how to do this testing in a
>> structured manner?
>
> I think that if what you're doing is sound, and you get the results
> you want, say, on compiling something like gcc with it and others
> review the basic idea (hi evan or chris) and like it, just checking it
> in and watching the performance numbers for the next day seems
> reasonable to me.  You can always revert the patch if there are
> unexpected downsides that make it not worth while.
It depends on the scope of the change.  If it is a relatively minor 
change, getting the code approved, testing it for correctness, and adding 
a regression test is sufficient.  If it is major (adding a new pass, 
significantly changing pass ordering etc) then the bar is much higher.

We don't have a great way of diffing performance runs, other than the 
nightly tester.  Devang has an experimental "opt-beta" mode that can
be
used for experimenting with optimization passes, and we have
"llc-beta"
which is great for measuring the impact of codegen changes.

The usual approach is to decide that the patch is good, check it in, then 
watch for unexpected fallout on the nightly testers.

-Chris

-- 
http://nondot.org/sabre/
http://llvm.org/

>> On May 21, 2008, at 8:09 AM, Matthijs Kooijman wrote:
>>> Any thoughts or suggestions on how to do this testing in a
>>> structured manner?
>>
>> I think that if what you're doing is sound, and you get the results
>> you want, say, on compiling something like gcc with it and others
>> review the basic idea (hi evan or chris) and like it, just checking it
>> in and watching the performance numbers for the next day seems
>> reasonable to me.  You can always revert the patch if there are
>> unexpected downsides that make it not worth while.
>
> It depends on the scope of the change.  If it is a relatively minor
> change, getting the code approved, testing it for correctness, and adding
> a regression test is sufficient.  If it is major (adding a new pass,
> significantly changing pass ordering etc) then the bar is much higher.
>
> We don't have a great way of diffing performance runs, other than the
> nightly tester.  Devang has an experimental "opt-beta" mode that
can be
> used for experimenting with optimization passes, and we have
"llc-beta"
> which is great for measuring the impact of codegen changes.
>
> The usual approach is to decide that the patch is good, check it in, then
> watch for unexpected fallout on the nightly testers.
Btw, this is all spelled out in the developers policy, which I like to 
mention in case people are not aware of it:
http://llvm.org/docs/DeveloperPolicy.html#quality

-Tanya

Matthijs,

On May 21, 2008, at 1:43 PM, Chris Lattner wrote:
> We don't have a great way of diffing performance runs, other than the
> nightly tester.  Devang has an experimental "opt-beta" mode that
can
> be
> used for experimenting with optimization passes, and we have "llc- 
> beta"
> which is great for measuring the impact of codegen changes.

opt-beta allows you to compare "opt -std-compile-opts" vs. "opt
<your
sequence of optimization passes>"
For your use, you can have local command line option to trigger your  
simplifycfg patch and then you can try
	ENABLE_OPTBETA=1 OPTBETAOPTIONS="-your-command-line-flag -std-compile- 
opts"

Let me know, if you try this and run into issues.
-
Devang

Hi,

I've polished my changes to HTMLColDiff.pl a bit and it should now be a
fairly
useful tool for finding performance changes. It's not the most robust tool,
but it should be able to compare different test runs and output something
useful.
> opt-beta allows you to compare "opt -std-compile-opts" vs.
"opt <your
> sequence of optimization passes>"
> For your use, you can have local command line option to trigger your  
> simplifycfg patch and then you can try
> 	ENABLE_OPTBETA=1 OPTBETAOPTIONS="-your-command-line-flag
-std-compile-
> opts"
This also sounds like an (even more useful) option, in particular to quickly
compare a single program or change. I will be trying this next.

Thanks for all the pointers!

Gr.

Matthijs
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20080522/f0a08a69/attachment.sig>

Hi Devang,

I've tried the OPTBETA approach, and it now runs with and without my patch
succesfully. I've found two problems, however:
 1. The output of the nightly report does not include the figures from
    opt-beta by default. I've modified the TEST.nightly.report script to add
		two columns (OPT-BETA and LLC/OPT-BETA), but committing this change would
		mean that most users will be having a lot of useless columns (though they
		already have that for the LLC-BETA stuff).

		Is there a way to access the ENABLE_* variables from the makefile in the
		report script? If so, you could only show the columns that will contain
		actual output, which will make the report a lot easier to read as well.
 2. The opt-beta output is always processed by llc, so you can only compare
    llc results with and without a change. It seems that for my change, the
		largest degradations are in the cbe test, not llc. I don't think there is
		an easy way to solve this, though.

Gr.

Matthijs
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20080522/2a47ece7/attachment.sig>