[llvm-dev] CTMark - regular LLVM and CLANG compile-time tracking

Matthias Braun via llvm-dev
2016-11-17T23:15:05Z

> On Nov 17, 2016, at 3:10 PM, Mehdi Amini <mehdi.amini at apple.com>
wrote:
>
>>
>> On Nov 17, 2016, at 3:00 PM, Matthias Braun <mbraun at apple.com
<mailto:mbraun at apple.com>> wrote:
>>
>>
>>> On Nov 17, 2016, at 2:55 PM, Mehdi Amini <mehdi.amini at
apple.com <mailto:mehdi.amini at apple.com>> wrote:
>>>
>>> Hi Gerolf,
>>>
>>> This is really cool!
>>> I’m very excited about this initiative and I hope we’ll be able to
get to a stage where compile time regression are handled like other regression:
if they are not expected / justified by the commit author promptly, the commit
should be reverted in the meantime!
>>>
>>> I’d like to suggest adding to CTMark the “empty” compile test (and
maybe “empty + one empty function”), unless it is too noisy to measure.
>>> It is an interesting test to complete the existing ones because it
measures the general overhead of setting up all the “infrastructure” (static
initializers, creating a pass pipeline, etc.)
>> That would indeed be a very interesting test, however this will be way
too short to measure predictably on its own.
>
> Are you afraid of the measurement noise for this?
>
>> I could see it working if we had a flag that artifically runs the
compilation pipeline hundreds of times or alternatively puts the whole compiler
into something like googlebenchmark.
>
> Since I’m interested in the startup time in general, it’d have to be a loop
in a shell script that invokes clang ~1000 times (or better: with a statistical
measurement of “confidence” that stops the loop when it reaches a threshold or
don’t make progress anymore), and returns something like the geometric mean.
Can someone report actual experiences with this? I would expect extra noise if
we get the operating system involved as well creating thousands of processes.

>
> I think I remember Michael G. doing something like that for Swift
performance testing?
Well to come back to the end-to-end vs. unittest comparison. If startup time
gets out of hand it will show up in CTMark as well. If it does not get out of
hand and requires special measurement techniques because it is a lot less than
0.5s then probably end user won't care as well.

- Matthias
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20161117/d0ac46df/attachment-0001.html>