llvm dev - May 2014 - [LLVMdev] Use perf tool for more accurate time measuring on Linux

On 16 May 2014 18:08, Hal Finkel <hfinkel at anl.gov>
wrote:
> ----- Original Message -----
>> From: "Yi Kong" <kongy.dev at gmail.com>
>> To: "Hal Finkel" <hfinkel at anl.gov>, "Renato
Golin" <renato.golin at linaro.org>, "Tobias Grosser"
<tobias at grosser.es>
>> Cc: "LLVM Dev" <llvmdev at cs.uiuc.edu>
>> Sent: Friday, May 16, 2014 11:37:28 AM
>> Subject: Use perf tool for more accurate time measuring on Linux
>>
>> Hi all,
>>
>> The LLVM benchmarking system produces very noisy results even on
>> quiet
>> machines. One of the sources of inaccuracy is the timing tool we are
>> using. Because it is a user-space tool, the OS can context switch it
>> and we will get an outlier result. Perf stat uses SW_TASK_CLOCK
>> counter in kernel to measure time, therefore more accurate. It also
>> does not get context switched.
>>
>> I've implemented a wrapper script over perf stat which mimics the
>> behaviour of timeit tool in test suite, so that nothing else needs to
>> be modified. The script is not yet feature complete as timeit, but
>> enough to run nightly tests.
>>
>> I carried out experiments on several machines and saw different level
>> of improvements. I am no longer seeing outlier results, and MAD is
>> considerably lower. The run-by-run changes results over the same
>> revision shrank from around 10 to only 2-3. The MAD reduced from
>> around 0.01 to 0.003 on a quiet machine.
>
> That sounds great, thanks for working on this!
>
> First, we'd definitely need more documentation on what perf is and how
to get it. The testing guide (and lnt dependencies), at least, need to be
updated. FWIW, I don't have any machines on which this is already installed
(and so it certainly is not installed by default). On Ubuntu, it is claimed that
both linux-base and linux-tools-common provide a 'pref' utility, and on
rpm systems it looks like there is a perf package.
>
> +# FIXME: How to measure sys time?
> +echo sys 0.0000 >> $REPORT
>
> Is this just the difference between the real time and the task time (or
would that be a reasonable approximation)?
Not on some occasions. But for most programs, that should a reasonable
approximation. Since test suite does not care about the sys time, we
can leave it for now.
$ time -p sleep 5
real 5.00
user 0.00
sys 0.00
>
> Thanks again,
> Hal
>
>>
>> I've attached the patch and please experiment with it.
>>
>> Cheers,
>> Yi Kong
>>
>
> --
> Hal Finkel
> Assistant Computational Scientist
> Leadership Computing Facility
> Argonne National Laboratory

Why not use the cycle count which perf exposes from hardware? That would
seem even better to me, but data would be better. =]


On Fri, May 16, 2014 at 11:17 AM, Yi Kong <kongy.dev at gmail.com> wrote:
> On 16 May 2014 18:08, Hal Finkel <hfinkel at anl.gov> wrote:
> > ----- Original Message -----
> >> From: "Yi Kong" <kongy.dev at gmail.com>
> >> To: "Hal Finkel" <hfinkel at anl.gov>,
"Renato Golin" <
> renato.golin at linaro.org>, "Tobias Grosser" <tobias at
grosser.es>
> >> Cc: "LLVM Dev" <llvmdev at cs.uiuc.edu>
> >> Sent: Friday, May 16, 2014 11:37:28 AM
> >> Subject: Use perf tool for more accurate time measuring on Linux
> >>
> >> Hi all,
> >>
> >> The LLVM benchmarking system produces very noisy results even on
> >> quiet
> >> machines. One of the sources of inaccuracy is the timing tool we
are
> >> using. Because it is a user-space tool, the OS can context switch
it
> >> and we will get an outlier result. Perf stat uses SW_TASK_CLOCK
> >> counter in kernel to measure time, therefore more accurate. It
also
> >> does not get context switched.
> >>
> >> I've implemented a wrapper script over perf stat which mimics
the
> >> behaviour of timeit tool in test suite, so that nothing else needs
to
> >> be modified. The script is not yet feature complete as timeit, but
> >> enough to run nightly tests.
> >>
> >> I carried out experiments on several machines and saw different
level
> >> of improvements. I am no longer seeing outlier results, and MAD is
> >> considerably lower. The run-by-run changes results over the same
> >> revision shrank from around 10 to only 2-3. The MAD reduced from
> >> around 0.01 to 0.003 on a quiet machine.
> >
> > That sounds great, thanks for working on this!
> >
> > First, we'd definitely need more documentation on what perf is and
how
> to get it. The testing guide (and lnt dependencies), at least, need to be
> updated. FWIW, I don't have any machines on which this is already
installed
> (and so it certainly is not installed by default). On Ubuntu, it is claimed
> that both linux-base and linux-tools-common provide a 'pref'
utility, and
> on rpm systems it looks like there is a perf package.
> >
> > +# FIXME: How to measure sys time?
> > +echo sys 0.0000 >> $REPORT
> >
> > Is this just the difference between the real time and the task time
(or
> would that be a reasonable approximation)?
>
> Not on some occasions. But for most programs, that should a reasonable
> approximation. Since test suite does not care about the sys time, we
> can leave it for now.
> $ time -p sleep 5
> real 5.00
> user 0.00
> sys 0.00
>
> >
> > Thanks again,
> > Hal
> >
> >>
> >> I've attached the patch and please experiment with it.
> >>
> >> Cheers,
> >> Yi Kong
> >>
> >
> > --
> > Hal Finkel
> > Assistant Computational Scientist
> > Leadership Computing Facility
> > Argonne National Laboratory
>
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20140516/7484aed8/attachment.html>

On 16 May 2014 18:40, "Chandler Carruth" <chandlerc at
google.com> wrote:
>
> Why not use the cycle count which perf exposes from hardware? That would
seem even better to me, but data would be better. =]

That's an interesting idea. However I'm concerned if that will miss some
aspects of compiler optimization. For example frequent cache misses would
have much smaller impact on the result if the processor goes to lower
frequency during the stall period. Nonetheless it's definitely worth to try
out.
>
>
> On Fri, May 16, 2014 at 11:17 AM, Yi Kong <kongy.dev at gmail.com>
wrote:
>>
>> On 16 May 2014 18:08, Hal Finkel <hfinkel at anl.gov> wrote:
>> > ----- Original Message -----
>> >> From: "Yi Kong" <kongy.dev at gmail.com>
>> >> To: "Hal Finkel" <hfinkel at anl.gov>,
"Renato Golin" <
renato.golin at linaro.org>, "Tobias Grosser" <tobias at
grosser.es>
>> >> Cc: "LLVM Dev" <llvmdev at cs.uiuc.edu>
>> >> Sent: Friday, May 16, 2014 11:37:28 AM
>> >> Subject: Use perf tool for more accurate time measuring on
Linux
>> >>
>> >> Hi all,
>> >>
>> >> The LLVM benchmarking system produces very noisy results even
on
>> >> quiet
>> >> machines. One of the sources of inaccuracy is the timing tool
we are
>> >> using. Because it is a user-space tool, the OS can context
switch it
>> >> and we will get an outlier result. Perf stat uses
SW_TASK_CLOCK
>> >> counter in kernel to measure time, therefore more accurate. It
also
>> >> does not get context switched.
>> >>
>> >> I've implemented a wrapper script over perf stat which
mimics the
>> >> behaviour of timeit tool in test suite, so that nothing else
needs to
>> >> be modified. The script is not yet feature complete as timeit,
but
>> >> enough to run nightly tests.
>> >>
>> >> I carried out experiments on several machines and saw
different level
>> >> of improvements. I am no longer seeing outlier results, and
MAD is
>> >> considerably lower. The run-by-run changes results over the
same
>> >> revision shrank from around 10 to only 2-3. The MAD reduced
from
>> >> around 0.01 to 0.003 on a quiet machine.
>> >
>> > That sounds great, thanks for working on this!
>> >
>> > First, we'd definitely need more documentation on what perf is
and how
to get it. The testing guide (and lnt dependencies), at least, need to be
updated. FWIW, I don't have any machines on which this is already installed
(and so it certainly is not installed by default). On Ubuntu, it is claimed
that both linux-base and linux-tools-common provide a 'pref' utility,
and
on rpm systems it looks like there is a perf package.
>> >
>> > +# FIXME: How to measure sys time?
>> > +echo sys 0.0000 >> $REPORT
>> >
>> > Is this just the difference between the real time and the task
time
(or would that be a reasonable approximation)?
>>
>> Not on some occasions. But for most programs, that should a reasonable
>> approximation. Since test suite does not care about the sys time, we
>> can leave it for now.
>> $ time -p sleep 5
>> real 5.00
>> user 0.00
>> sys 0.00
>>
>> >
>> > Thanks again,
>> > Hal
>> >
>> >>
>> >> I've attached the patch and please experiment with it.
>> >>
>> >> Cheers,
>> >> Yi Kong
>> >>
>> >
>> > --
>> > Hal Finkel
>> > Assistant Computational Scientist
>> > Leadership Computing Facility
>> > Argonne National Laboratory
>>
>> _______________________________________________
>> LLVM Developers mailing list
>> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
>> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20140516/957f6bf0/attachment.html>