llvm dev - Apr 2010 - [LLVMdev] FYI: Changing RunSafely.sh to only track user time

I am going to change the LLVM test-suite RunSafely.sh to only track
'user' time, instead of reporting 'user' + 'sys' time as
it currently
does. This will probably cause a spike in nightly test numbers,
although hopefully it will be limited to the smaller tests.

The eventual goal is to report all numbers (independently), so that we
can control for noise better. However, until that happens it is better
to track the most stable & interesting number.

 - Daniel

Daniel Dunbar wrote:
> I am going to change the LLVM test-suite RunSafely.sh to only track
> 'user' time, instead of reporting 'user' + 'sys'
time as it currently
> does. This will probably cause a spike in nightly test numbers,
> although hopefully it will be limited to the smaller tests.
>
> The eventual goal is to report all numbers (independently), so that we
> can control for noise better. However, until that happens it is better
> to track the most stable & interesting number.
>   
First, you should be aware that the test-suite infrastructure is used by 
the Automatic Pool Allocation and SAFECode projects for their testing 
infrastructure.  It is also used by our internal research projects and 
may be used by other research projects at other universities.  We use 
this infrastructure for our research, so changes you make can affect us.

Second, why are you only interested in user time?  The reason why we had 
RunSafely.sh measure user + system time is that it gives a more accurate 
depiction of how well an optimization works.  If a program spends most 
of its time in the OS, increasing speed in user-space doesn't gain us 
much.  If a transform decreases user time but increases system time, 
then measuring only user time may show a speedup when measuring 
user+system will show a loss.

If noise is your concern, I think it would make sense to run the tests 
several times and report averages and standard deviations.  System time 
should only be reporting OS CPU time, so delays due to interrupts, I/O, 
etc. should not affect the results.  User+System reported for several 
runs should provide the most accurate evaluation of how well an 
optimization is improving performance.

Third, IIRC, I think you can change the nightly testers to report just 
user time by changing how they grab the time measurements in 
TEST.nightly.Makefile.  I think the .time files created by the test 
suite record user, system, and user+system, and the testing Makefiles 
just grab the number that they want.

Fourth, if you are intent on changing RunSafely.sh to measure just user 
time, why not add a feature that toggles whether to measure user+system 
or user time and *then* change the default behavior to measure user 
time?  That way, I can easily toggle it back.  What you've suggested 
above (more or less) is to remove a feature, add a feature, and then add 
back the feature you removed.  It makes more sense to me to add the one 
feature you want and then to change the default behavior to use the new 
feature instead of the old one.

-- John T.
>  - Daniel
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>

On Apr 18, 2010, at 11:08 PM, John Criswell wrote:
> Second, why are you only interested in user time?  The reason why we had 
> RunSafely.sh measure user + system time is that it gives a more accurate 
> depiction of how well an optimization works.  If a program spends most 
> of its time in the OS, increasing speed in user-space doesn't gain us 
> much.  If a transform decreases user time but increases system time, 
> then measuring only user time may show a speedup when measuring 
> user+system will show a loss.
What kind of optimization might change the system time?

The problem with measuring system time is that it can depend on many variable
factors that have nothing to do with the process being tested. For instance, the
number of files on disk, the amount of free space on disk, the total number of
processes on the system, the amount of free memory pages on the system, and the
size of the buffer cache can all affect how much work a system call has to do.

/jakob

On Apr 18, 2010, at 11:08 PM, John Criswell wrote:
> Daniel Dunbar wrote:
>> I am going to change the LLVM test-suite RunSafely.sh to only track
>> 'user' time, instead of reporting 'user' +
'sys' time as it currently
>> does. This will probably cause a spike in nightly test numbers,
>> although hopefully it will be limited to the smaller tests.
>> 
>> The eventual goal is to report all numbers (independently), so that we
>> can control for noise better. However, until that happens it is better
>> to track the most stable & interesting number.
>> 
> 
> First, you should be aware that the test-suite infrastructure is used by 
> the Automatic Pool Allocation and SAFECode projects for their testing 
> infrastructure.  It is also used by our internal research projects and 
> may be used by other research projects at other universities.  We use 
> this infrastructure for our research, so changes you make can affect us.
You or other people are welcome to genericize this by adding a knob to control
it.  However, mainline llvm should be most concerned with itself, not with
projects with out of tree work.  This is very similar to an API change.

I don't really expect it, but worse case, if the harness becomes completely
unsuitable for you guys, you could just clone all the makefiles in your tree as
a 'report'.  This would be insulate you from changes.

-Chris