llvm dev - Jun 2016 - RFC: Comprehensive Static Instrumentation

> On Jun 17, 2016, at 11:27 AM, TB Schardl via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> 
> Hey Ben,
> 
> Thank you for your comments.  I've put my response inline.
> 
> Cheers,
> TB
> 
> On Fri, Jun 17, 2016 at 6:29 AM, Craig, Ben via llvm-dev <llvm-dev at
lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote:
> On 6/16/2016 2:48 PM, Mehdi Amini via llvm-dev wrote:
>> 
>>> On Jun 16, 2016, at 9:01 AM, TB Schardl via llvm-dev <llvm-dev
at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote:
>>> 
>>> The CSI framework inserts instrumentation hooks at salient
locations throughout the compiled code of a program-under-test, such as function
entry and exit points, basic-block entry and exit points, before and after each
memory operation, etc.  Tool writers can instrument a program-under-test simply
by first writing a library that defines the semantics of relevant hooks
>>> and then statically linking their compiled library with the
program-under-test.
>>> 
>>> At first glance, this brute-force method of inserting hooks at
every salient location in the program-under-test seems to be replete with
overheads.  CSI overcomes these overheads through the use of
link-time-optimization (LTO), which is now readily available in most major
compilers, including GCC and LLVM.  Using LTO, instrumentation hooks that are
not used by a particular tool can be elided, allowing the overheads of these
hooks to be avoided when the
>> 
>> I don't understand this flow: the front-end emits all the possible
instrumentation but the useless calls to the runtime will be removed during the
link?
>> It means that the final binary is specialized for a given tool right?
What is the advantage of generating this useless instrumentation in the first
place then? I'm missing a piece here...
>> 
> Suppose I want to build a production build, and one build for each of ASAN,
MSAN, UBSAN, and TSAN.
> 
> With the current approach, I need to compile my source five different
times, and link five different times.
> 
> With the CSI approach (assuming it was the backing technology behind the
sanitizers), I need to compile twice (once for production, once for
instrumentation), then LTO-link five times.  I can reuse my .o files across the
sanitizer types.
> 
> This reduction in the number of compile operations needed, and in the
number intermediate object/bitcode files produced, is indeed an advantage of the
CSI approach.
It is a very artificial advantage, what are you saving? Temporary Disk Space?

> As an aside, we've been experimenting with linking CSI-instrumented
bitcodes against the "null tool," which implements every
instrumentation hook as a nop, and comparing the performance of those binaries
against production binaries.  Our preliminary tests have shown some promising
results.  For generating main executables, using LTO to link CSI-instrumented
bitcodes with the null tool produces executables that are as fast as the
production executables.  For generating dynamic libraries, however, using LTO to
link the CSI-instrumented bitcode of a dynamic library with the null tool seems
to produce a binary that is slower than production.  (The Apache HTTP server
benchmark we've tried runs roughly 30% slower when using such
null-tool-instrumented dynamic libraries.)  These results suggest that using LTO
to link CSI-instrumented bitcodes with the null tool is almost, but not quite,
able to produce binaries with production performance
These results suggests that “adding instrumentation has a cost” nothing more,
and is unrelated to LTO at all.
You would provide the runtime to the compiler directly during the compile phase
and you would get the same results.

> , which would allow tool users to only compile their sources once.
LTO means basically “compiles during the link”. You won’t save much.

I haven’t seen a single compelling argument to *tie* CSI to LTO in this thread
until now.


— 
Mehdi




>  
> 
> It's possible that the math doesn't really work out in practice if
the cost of the LTO-link dwarfs the compile times.
> -- 
> Employee of Qualcomm Innovation Center, Inc.
> Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, a Linux
Foundation Collaborative Project
> 
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
<http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
> 
> 
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160617/c34d1aec/attachment-0001.html>

Mehdi, I think TB is saying that LTO is actually *more* incremental than
standard compilation for people writing instrumentation tools.

It's far simpler to rebuild your CSI instrumentation hooks and re-run LTO
on Apache than it is to re-run someone else's complicated build system. In
fact, you save all the time that clang would spend parsing C and C++ source
code, since you're starting with several blobs of semi-optimized LLVM IR.

It's worth exploring what you mentioned up-thread, where you inline the
instrumentation into the code during normal compliation instead of waiting
until link time, but I think it makes more sense to make it easy to build
tools before we try to make those new tools compile faster. It seems like a
nice-to-have feature for instrumentation tools that manage to graduate from
research idea to production tool.

On Fri, Jun 17, 2016 at 11:42 AM, Mehdi Amini via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> This reduction in the number of compile operations needed, and in the
> number intermediate object/bitcode files produced, is indeed an advantage
> of the CSI approach.
>
>
> It is a very artificial advantage, what are you saving? Temporary Disk
> Space?
>
>
> As an aside, we've been experimenting with linking CSI-instrumented
> bitcodes against the "null tool," which implements every
instrumentation
> hook as a nop, and comparing the performance of those binaries against
> production binaries.  Our preliminary tests have shown some promising
> results.  For generating main executables, using LTO to link
> CSI-instrumented bitcodes with the null tool produces executables that are
> as fast as the production executables.  For generating dynamic libraries,
> however, using LTO to link the CSI-instrumented bitcode of a dynamic
> library with the null tool seems to produce a binary that is slower than
> production.  (The Apache HTTP server benchmark we've tried runs roughly
30%
> slower when using such null-tool-instrumented dynamic libraries.)  These
> results suggest that using LTO to link CSI-instrumented bitcodes with the
> null tool is almost, but not quite, able to produce binaries with
> production performance
>
>
> These results suggests that “adding instrumentation has a cost” nothing
> more, and is unrelated to LTO at all.
> You would provide the runtime to the compiler directly during the compile
> phase and you would get the same results.
>
>
> , which would allow tool users to only compile their sources once.
>
>
> LTO means basically “compiles during the link”. You won’t save much.
>
> I haven’t seen a single compelling argument to *tie* CSI to LTO in this
> thread until now.
>
>
> —
> Mehdi
>
>
>
>
>
>
>
>>
>> It's possible that the math doesn't really work out in practice
if the
>> cost of the LTO-link dwarfs the compile times.
>>
>> --
>> Employee of Qualcomm Innovation Center, Inc.
>> Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, a
Linux Foundation Collaborative Project
>>
>>
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
>>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160617/92d784f6/attachment.html>

> On Jun 17, 2016, at 11:56 AM, Reid Kleckner <rnk at google.com>
wrote:
> 
> Mehdi, I think TB is saying that LTO is actually *more* incremental than
standard compilation for people writing instrumentation tools.
> 
> It's far simpler to rebuild your CSI instrumentation hooks and re-run
LTO on Apache than it is to re-run someone else's complicated build system.
I *highly* doubt about that. I’d even say that I’m heavily convinced of the
opposite: any build system has a facility to add a compiler flag. What could be
easier than adding `-csi=path_to_my_tool` to the CFLAGS?

On the other hand, LTO is a whole other beast. First you also need to modify the
CFLAGS to enable LTO, so you don’t avoid the “complicated build system”
interaction.
But then you’ll need to make sure the rest of the toolchain is compatible (Never
had an issue with static archives? I did a lot of time!). Also what if the
projects involves subprojects with dylib? You need to make sure every link is
correctly performed with the instrumented runtime.

Nothing “impossible” here, but I don’t buy at all the “it is simpler” argument.
> In fact, you save all the time that clang would spend parsing C and C++
source code, since you're starting with several blobs of semi-optimized LLVM
IR.
LTO alone on the clang binary is longer than a full clean build (and if you
build all of the llvm binaries, we one or two order of magnitudes away), so it
won't buy you anything. And even if it would (apparently on Apache httpd it
would), I doubt it would be significant enough to make it an argument to *tie*
the design to LTO.
> It's worth exploring what you mentioned up-thread, where you inline the
instrumentation into the code during normal compliation instead of waiting until
link time, but I think it makes more sense to make it easy to build tools before
we try to make those new tools compile faster.
So a few line earlier your argument for relying on LTO was to save “parsing C
and C++” (so basically compile time), and now you’re saying that “making those
new tools compile faster” should not drive the design?

I agree with “making it easy to build tools” part, I just don’t see any reason
why forcing to use LTO supports this goal.
> It seems like a nice-to-have feature for instrumentation tools that manage
to graduate from research idea to production tool.
I believe this last sentence isn’t related to the LTO aspect, CSI looks like a
great idea, let’s just do it in a way that it does not require LTO.

— 
Mehdi

> 
> On Fri, Jun 17, 2016 at 11:42 AM, Mehdi Amini via llvm-dev <llvm-dev at
lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote:
>> This reduction in the number of compile operations needed, and in the
number intermediate object/bitcode files produced, is indeed an advantage of the
CSI approach.
> 
> It is a very artificial advantage, what are you saving? Temporary Disk
Space?
> 
> 
>> As an aside, we've been experimenting with linking CSI-instrumented
bitcodes against the "null tool," which implements every
instrumentation hook as a nop, and comparing the performance of those binaries
against production binaries.  Our preliminary tests have shown some promising
results.  For generating main executables, using LTO to link CSI-instrumented
bitcodes with the null tool produces executables that are as fast as the
production executables.  For generating dynamic libraries, however, using LTO to
link the CSI-instrumented bitcode of a dynamic library with the null tool seems
to produce a binary that is slower than production.  (The Apache HTTP server
benchmark we've tried runs roughly 30% slower when using such
null-tool-instrumented dynamic libraries.)  These results suggest that using LTO
to link CSI-instrumented bitcodes with the null tool is almost, but not quite,
able to produce binaries with production performance
> 
> These results suggests that “adding instrumentation has a cost” nothing
more, and is unrelated to LTO at all.
> You would provide the runtime to the compiler directly during the compile
phase and you would get the same results.
> 
> 
>> , which would allow tool users to only compile their sources once.
> 
> LTO means basically “compiles during the link”. You won’t save much.
> 
> I haven’t seen a single compelling argument to *tie* CSI to LTO in this
thread until now.
> 
> 
> — 
> Mehdi
> 
> 
> 
> 
> 
>>  
>> 
>> It's possible that the math doesn't really work out in practice
if the cost of the LTO-link dwarfs the compile times.
>> -- 
>> Employee of Qualcomm Innovation Center, Inc.
>> Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, a
Linux Foundation Collaborative Project
>> 
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
<http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
>> 
>> 
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
<http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
> 
> 
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
<http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160617/4c07dbbd/attachment.html>

There seems to be two main concerns with the use of LTO.  The first one is
the hit on build time, and the second one is the argument of simplicity ---
that is, it may not actually be simpler to use LTO.

Regarding the performance hit on build time, we would like to emphasize
that, CSI is not tied to the use of LTO --- it does not rely on LTO to work
correctly; LTO is simply a mean to optimize the final executable.  Having
that being said, as Kostya pointed it out, and based what our understanding
of thinLTO, we believe that thinLTO can do most of the optimizations that
CSI will benefit from.  CSI uses LTO only for optimization, but it also
seems that later down the road, such an optimization will cost less.

Regarding simplicity, Mehdi, you raise a good point, but I think Reid's
point that, linking is more incremental still stands.  In particular, we
can separate out the application developer and the tool user --- the tool
user does not have to be the one that build the application to generate the
CSI-enabled bitcode files.  These days, there are versions of libc that you
can get w/ all the debug symbols compiled in versus the version that
doesn't.  Imagine if by default, the applications / libraries also come
with just a collection of bitcode files that are CSI enabled, then the tool
user doesn't even need to have the source directly nor mess with the
application's build system --- he / she can simply link the CSI-compatible
bitcodes with any tool that he / she intends to use.

Cheers,
Angelina

----
I-Ting Angelina Lee
Assistant Professor, Washington University in St. Louis
Department of Computer Science and Engineering
Homepage: *http://www.cse.wustl.edu/~angelee/
<http://www.cse.wustl.edu/~angelee/>*

On Fri, Jun 17, 2016 at 3:22 PM, Mehdi Amini <mehdi.amini at apple.com>
wrote:
>
> On Jun 17, 2016, at 11:56 AM, Reid Kleckner <rnk at google.com>
wrote:
>
> Mehdi, I think TB is saying that LTO is actually *more* incremental than
> standard compilation for people writing instrumentation tools.
>
>
> It's far simpler to rebuild your CSI instrumentation hooks and re-run
LTO
> on Apache than it is to re-run someone else's complicated build system.
>
>
> I *highly* doubt about that. I’d even say that I’m heavily convinced of
> the opposite: any build system has a facility to add a compiler flag. What
> could be easier than adding `-csi=path_to_my_tool` to the CFLAGS?
>
> On the other hand, LTO is a whole other beast. First you also need to
> modify the CFLAGS to enable LTO, so you don’t avoid the “complicated build
> system” interaction.
> But then you’ll need to make sure the rest of the toolchain is compatible
> (Never had an issue with static archives? I did a lot of time!). Also what
> if the projects involves subprojects with dylib? You need to make sure
> every link is correctly performed with the instrumented runtime.
>
> Nothing “impossible” here, but I don’t buy at all the “it is simpler”
> argument.
>
> In fact, you save all the time that clang would spend parsing C and C++
> source code, since you're starting with several blobs of semi-optimized
> LLVM IR.
>
>
> LTO alone on the clang binary is longer than a full clean build (and if
> you build all of the llvm binaries, we one or two order of magnitudes
> away), so it won't buy you anything. And even if it would (apparently
on
> Apache httpd it would), I doubt it would be significant enough to make it
> an argument to *tie* the design to LTO.
>
> It's worth exploring what you mentioned up-thread, where you inline the
> instrumentation into the code during normal compliation instead of waiting
> until link time, but I think it makes more sense to make it easy to build
> tools before we try to make those new tools compile faster.
>
>
> So a few line earlier your argument for relying on LTO was to save
> “parsing C and C++” (so basically compile time), and now you’re saying that
> “making those new tools compile faster” should not drive the design?
>
> I agree with “making it easy to build tools” part, I just don’t see any
> reason why forcing to use LTO supports this goal.
>
> It seems like a nice-to-have feature for instrumentation tools that manage
> to graduate from research idea to production tool.
>
>
> I believe this last sentence isn’t related to the LTO aspect, CSI looks
> like a great idea, let’s just do it in a way that it does not require LTO.
>
> —
> Mehdi
>
>
>
> On Fri, Jun 17, 2016 at 11:42 AM, Mehdi Amini via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>> This reduction in the number of compile operations needed, and in the
>> number intermediate object/bitcode files produced, is indeed an
advantage
>> of the CSI approach.
>>
>>
>> It is a very artificial advantage, what are you saving? Temporary Disk
>> Space?
>>
>>
>> As an aside, we've been experimenting with linking CSI-instrumented
>> bitcodes against the "null tool," which implements every
instrumentation
>> hook as a nop, and comparing the performance of those binaries against
>> production binaries.  Our preliminary tests have shown some promising
>> results.  For generating main executables, using LTO to link
>> CSI-instrumented bitcodes with the null tool produces executables that
are
>> as fast as the production executables.  For generating dynamic
libraries,
>> however, using LTO to link the CSI-instrumented bitcode of a dynamic
>> library with the null tool seems to produce a binary that is slower
than
>> production.  (The Apache HTTP server benchmark we've tried runs
roughly 30%
>> slower when using such null-tool-instrumented dynamic libraries.) 
These
>> results suggest that using LTO to link CSI-instrumented bitcodes with
the
>> null tool is almost, but not quite, able to produce binaries with
>> production performance
>>
>>
>> These results suggests that “adding instrumentation has a cost” nothing
>> more, and is unrelated to LTO at all.
>> You would provide the runtime to the compiler directly during the
compile
>> phase and you would get the same results.
>>
>>
>> , which would allow tool users to only compile their sources once.
>>
>>
>> LTO means basically “compiles during the link”. You won’t save much.
>>
>> I haven’t seen a single compelling argument to *tie* CSI to LTO in this
>> thread until now.
>>
>>
>> —
>> Mehdi
>>
>>
>>
>>
>>
>>
>>
>>>
>>> It's possible that the math doesn't really work out in
practice if the
>>> cost of the LTO-link dwarfs the compile times.
>>>
>>> --
>>> Employee of Qualcomm Innovation Center, Inc.
>>> Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
a Linux Foundation Collaborative Project
>>>
>>>
>>> _______________________________________________
>>> LLVM Developers mailing list
>>> llvm-dev at lists.llvm.org
>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>
>>>
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
>>
>>
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160619/63d502a1/attachment-0001.html>

Hi Angelina,
> On Jun 19, 2016, at 7:51 AM, Angelina Lee <angelee at seas.wustl.edu>
wrote:
> 
> There seems to be two main concerns with the use of LTO. 
To be clear: for the purpose of CSI, the concern is not about making is friendly
with LTO, it is about *requiring* LTO to make it efficient. This is design flaw
that has no justification.
> The first one is the hit on build time, and the second one is the argument
of simplicity --- that is, it may not actually be simpler to use LTO.
> 
> Regarding the performance hit on build time, we would like to emphasize
that, CSI is not tied to the use of LTO --- it does not rely on LTO to work
correctly; LTO is simply a mean to optimize the final executable.  Having that
being said, as Kostya pointed it out, and based what our understanding of
thinLTO, we believe that thinLTO can do most of the optimizations that CSI will
benefit from.  CSI uses LTO only for optimization, but it also seems that later
down the road, such an optimization will cost less.
I understand this, and I still don’t see any reason to force the use of LTO to
optimize the CSI use-case. We should offer a mechanism that makes is no
different for the tool writer, but does not require LTO. I also believe such
mechanism can be implemented in the compiler easily.
And I haven’t seen any reasons raised by anyone not to do it in this thread.

> Regarding simplicity, Mehdi, you raise a good point, but I think Reid's
point that, linking is more incremental still stands. In particular, we can
separate out the application developer and the tool user --- the tool user does
not have to be the one that build the application to generate the CSI-enabled
bitcode files.  These days, there are versions of libc that you can get w/ all
the debug symbols compiled in versus the version that doesn't.  Imagine if
by default, the applications / libraries also come with just a collection of
bitcode files that are CSI enabled, then the tool user doesn't even need to
have the source directly nor mess with the application's build system --- he
/ she can simply link the CSI-compatible
> bitcodes with any tool that he / she intends to use.
All of what you describe is interesting, and a good justification for making CSI
*friendly* with LTO (which I never contested by the way). However it does not
seem to invalidate my point that *requiring* LTO is fundamentally unnecessary
here to support the “mainstream” flow that already exists with ASAN and other
sanitizer for instance.

— 
Mehdi


> 
> Cheers,
> Angelina
> 
> ----
> I-Ting Angelina Lee
> Assistant Professor, Washington University in St. Louis
> Department of Computer Science and Engineering 
> Homepage: http://www.cse.wustl.edu/~angelee/
<http://www.cse.wustl.edu/~angelee/>
> On Fri, Jun 17, 2016 at 3:22 PM, Mehdi Amini <mehdi.amini at apple.com
<mailto:mehdi.amini at apple.com>> wrote:
> 
>> On Jun 17, 2016, at 11:56 AM, Reid Kleckner <rnk at google.com
<mailto:rnk at google.com>> wrote:
>> 
>> Mehdi, I think TB is saying that LTO is actually *more* incremental
than standard compilation for people writing instrumentation tools.
>> 
>> It's far simpler to rebuild your CSI instrumentation hooks and
re-run LTO on Apache than it is to re-run someone else's complicated build
system.
> 
> I *highly* doubt about that. I’d even say that I’m heavily convinced of the
opposite: any build system has a facility to add a compiler flag. What could be
easier than adding `-csi=path_to_my_tool` to the CFLAGS?
> 
> On the other hand, LTO is a whole other beast. First you also need to
modify the CFLAGS to enable LTO, so you don’t avoid the “complicated build
system” interaction.
> But then you’ll need to make sure the rest of the toolchain is compatible
(Never had an issue with static archives? I did a lot of time!). Also what if
the projects involves subprojects with dylib? You need to make sure every link
is correctly performed with the instrumented runtime.
> 
> Nothing “impossible” here, but I don’t buy at all the “it is simpler”
argument.
> 
>> In fact, you save all the time that clang would spend parsing C and C++
source code, since you're starting with several blobs of semi-optimized LLVM
IR.
> 
> LTO alone on the clang binary is longer than a full clean build (and if you
build all of the llvm binaries, we one or two order of magnitudes away), so it
won't buy you anything. And even if it would (apparently on Apache httpd it
would), I doubt it would be significant enough to make it an argument to *tie*
the design to LTO.
> 
>> It's worth exploring what you mentioned up-thread, where you inline
the instrumentation into the code during normal compliation instead of waiting
until link time, but I think it makes more sense to make it easy to build tools
before we try to make those new tools compile faster.
> 
> So a few line earlier your argument for relying on LTO was to save “parsing
C and C++” (so basically compile time), and now you’re saying that “making those
new tools compile faster” should not drive the design?
> 
> I agree with “making it easy to build tools” part, I just don’t see any
reason why forcing to use LTO supports this goal.
> 
>> It seems like a nice-to-have feature for instrumentation tools that
manage to graduate from research idea to production tool.
> 
> I believe this last sentence isn’t related to the LTO aspect, CSI looks
like a great idea, let’s just do it in a way that it does not require LTO.
> 
> — 
> Mehdi
> 
> 
>> 
>> On Fri, Jun 17, 2016 at 11:42 AM, Mehdi Amini via llvm-dev <llvm-dev
at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote:
>>> This reduction in the number of compile operations needed, and in
the number intermediate object/bitcode files produced, is indeed an advantage of
the CSI approach.
>> 
>> It is a very artificial advantage, what are you saving? Temporary Disk
Space?
>> 
>> 
>>> As an aside, we've been experimenting with linking
CSI-instrumented bitcodes against the "null tool," which implements
every instrumentation hook as a nop, and comparing the performance of those
binaries against production binaries.  Our preliminary tests have shown some
promising results.  For generating main executables, using LTO to link
CSI-instrumented bitcodes with the null tool produces executables that are as
fast as the production executables.  For generating dynamic libraries, however,
using LTO to link the CSI-instrumented bitcode of a dynamic library with the
null tool seems to produce a binary that is slower than production.  (The Apache
HTTP server benchmark we've tried runs roughly 30% slower when using such
null-tool-instrumented dynamic libraries.)  These results suggest that using LTO
to link CSI-instrumented bitcodes with the null tool is almost, but not quite,
able to produce binaries with production performance
>> 
>> These results suggests that “adding instrumentation has a cost” nothing
more, and is unrelated to LTO at all.
>> You would provide the runtime to the compiler directly during the
compile phase and you would get the same results.
>> 
>> 
>>> , which would allow tool users to only compile their sources once.
>> 
>> LTO means basically “compiles during the link”. You won’t save much.
>> 
>> I haven’t seen a single compelling argument to *tie* CSI to LTO in this
thread until now.
>> 
>> 
>> — 
>> Mehdi
>> 
>> 
>> 
>> 
>> 
>>>  
>>> 
>>> It's possible that the math doesn't really work out in
practice if the cost of the LTO-link dwarfs the compile times.
>>> -- 
>>> Employee of Qualcomm Innovation Center, Inc.
>>> Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
a Linux Foundation Collaborative Project
>>> 
>>> _______________________________________________
>>> LLVM Developers mailing list
>>> llvm-dev at lists.llvm.org <mailto:llvm-dev at
lists.llvm.org>
>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
<http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
>>> 
>>> 
>>> _______________________________________________
>>> LLVM Developers mailing list
>>> llvm-dev at lists.llvm.org <mailto:llvm-dev at
lists.llvm.org>
>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
<http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
>> 
>> 
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
<http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160619/4dde6732/attachment.html>