llvm dev - Nov 2014 - [LLVMdev] [RFC] New ToolsSupport library for stuff that only tools need

Here is my limited perspective on this topic:

1. I would like to see some of the code in LLVM's ADT and Support libraries
available for reuse in other projects. This also has some potential for
being useful to Clang -- one could imagine a world where Clang directly
linked the ADT & Support libraries, but used the compiler itself via a
clear dynamic-library vended API (which would be unrealistic to expect the
ADT & Support libraries would ever want to provide).

2. I agree with Chandler in that I don't see a good need to try hard to
factor out code from the Support library that can just be conditional
disabled or would be unused by normal .a link semantics. For example,
whether or not the regex or YAML code belongs in Support doesn't seem worth
worrying too much about, because they are very isolated, don't introduce
extra dependencies, and won't be linked by projects that don't use them.

3. I also see a good argument for having a ToolsSupport library, as a place
for all of the very-LLVM(the compiler) specific functionality that would
have no place in a shared Support library (for any number of reasons: lack
of generality, invasiveness, or because they add substantial other
dependencies).

 - Daniel


On Thu, Nov 6, 2014 at 6:53 PM, Pete Cooper <peter_cooper at apple.com>
wrote:
>
> On Nov 6, 2014, at 4:52 PM, Chris Bieneman <beanz at apple.com>
wrote:
>
> I think for the main goal of cleaning up the Mac-specific hack, a
> CrashRecovery library would work equally well. Juergen is more familiar
> with the WebKit side of things, so he may be aware of something I’m not
> thinking of.
>
> Chandler, does splitting out a CrashRecovery library instead seem sane?
>
> FWIW, I prefer a ToolsSupport library.  It gives us the opportunity to put
> other tools specific things in there if we find them.  I can’t, for
> example, think of the dylib needing YAML right now, although I could be
> wrong.
>
> At least if we made it ToolsSupport then we wouldn’t have come back in a
> few months/year and ask to rename it from CrashRecovery.
>
> Pete
>
>
> Other than code organization and naming, the general idea of splitting out
> a CrashRecovery library would be the same as the other patches I sent out.
> I was thinking of taking the approach of moving one symbol, fixing
> everything, then repeat.
>
> Does that seem like the right approach?
>
> -Chris
>
> On Nov 6, 2014, at 3:43 PM, Reid Kleckner <rnk at google.com> wrote:
>
> On Thu, Nov 6, 2014 at 3:22 PM, Chris Bieneman <beanz at apple.com>
wrote:
>>
>> The other thought I had which motivated this solution was that if we
>> could strip all the functionality that is only really used by tools out
>> into a separate library it would offer cleaner organization of code.
>> Support seems to often get used as a dumping ground for stuff that just
>> doesn’t fit anywhere else.
>>
>
>
>> Based on your feedback and Chandler’s maybe this just isn’t the right
>> separation. I can look into a solution to address our hackiness without
>> creating a separate library.
>>
>
> What other stuff do you think belongs in ToolsSupport that doesn't
belong
> in Support? Looking back at the initial email, you have command line
> parsing and ToolOutputFile.
>
> We could split out command line parsing, but it doesn't seem worth it,
> given that we're still carrying regex support, Unicode conversion,
dynamic
> library support, and other things that probably aren't absolutely
necessary.
>
> What about splitting out a CrashRecovery library instead? That seems a lot
> more targeted and meaningful. We'd probably put ToolOutputFile.cpp in
there.
>
>
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>
>
>
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20141107/9f8dab7b/attachment.html>

On Fri, Nov 7, 2014 at 10:57 AM, Daniel Dunbar <daniel at zuster.org>
wrote:
> 1. I would like to see some of the code in LLVM's ADT and Support
> libraries available for reuse in other projects. This also has some
> potential for being useful to Clang -- one could imagine a world where
> Clang directly linked the ADT & Support libraries, but used the
compiler
> itself via a clear dynamic-library vended API (which would be unrealistic
> to expect the ADT & Support libraries would ever want to provide).

I don't think this is relevant or possible. The IR interfaces used by Clang
are extensive and not likely to be reasonable dynamic library vended APIs.
And in general, I don't think we should design based on this very
speculative future.

3. I also see a good argument for having a ToolsSupport library, as a
place
> for all of the very-LLVM(the compiler) specific functionality that would
> have no place in a shared Support library (for any number of reasons: lack
> of generality, invasiveness, or because they add substantial other
> dependencies).

I agree in principle, but I don't think the signal handling or crash
recovery are actually relevant for this -- they're very general, just only
useful for executable applications as opposed to useful for things like the
LLVM libraries in WebKit.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20141107/e11aa4ca/attachment.html>

> On Nov 7, 2014, at 8:57 AM, Daniel Dunbar <daniel at zuster.org>
wrote:
> 
> 2. I agree with Chandler in that I don't see a good need to try hard to
factor out code from the Support library that can just be conditional disabled
or would be unused by normal .a link semantics. For example, whether or not the
regex or YAML code belongs in Support doesn't seem worth worrying too much
about, because they are very isolated, don't introduce extra dependencies,
and won't be linked by projects that don't use them.
This assumes that the client is statically linking against Support.  Chris has
been pretty explicit about his goals of building and using LLVM as a monolithic
dynamic library, in which case things like regex or YAML support will *not* be
automatically removed.

—Owen
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20141107/2671e5c9/attachment.html>

On Fri, Nov 7, 2014 at 12:20 PM, Owen Anderson <resistor at mac.com>
wrote:
> On Nov 7, 2014, at 8:57 AM, Daniel Dunbar <daniel at zuster.org>
wrote:
>
> 2. I agree with Chandler in that I don't see a good need to try hard to
> factor out code from the Support library that can just be conditional
> disabled or would be unused by normal .a link semantics. For example,
> whether or not the regex or YAML code belongs in Support doesn't seem
worth
> worrying too much about, because they are very isolated, don't
introduce
> extra dependencies, and won't be linked by projects that don't use
them.
>
>
> This assumes that the client is statically linking against Support.  Chris
> has been pretty explicit about his goals of building and using LLVM as a
> monolithic dynamic library, in which case things like regex or YAML support
> will *not* be automatically removed.
>
I see...

Maybe what we really need is (lacking a complete export list for the
monolithic dynamic library) to use visibility so that these kinds of things
can be dropped from the final dynamic library if they aren't used within
it? I suspect would could mark essentially all of the support library, as
unless it is used in the interface of some other library, we probably don't
want to export it... (Having just started thinking about this, I've no idea
whether folks using the LLVM dynamic library are independently using bits
of the ADT for example....)
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20141107/e5b9a620/attachment.html>

To further this point, it was recently brought to my attention that the CMake
libLLVM.so on Linux is empty because the GNU linker on linux doesn’t treat
symbols mentioned in an export list as being used, so the linker dead strips it
all.

-Chris
> On Nov 7, 2014, at 12:28 PM, Jim Grosbach <grosbach at apple.com>
wrote:
> 
>> 
>> On Nov 7, 2014, at 11:05 AM, Chandler Carruth <chandlerc at
google.com <mailto:chandlerc at google.com>> wrote:
>> 
>> 
>> On Fri, Nov 7, 2014 at 12:20 PM, Owen Anderson <resistor at mac.com
<mailto:resistor at mac.com>> wrote:
>>> On Nov 7, 2014, at 8:57 AM, Daniel Dunbar <daniel at zuster.org
<mailto:daniel at zuster.org>> wrote:
>>> 
>>> 2. I agree with Chandler in that I don't see a good need to try
hard to factor out code from the Support library that can just be conditional
disabled or would be unused by normal .a link semantics. For example, whether or
not the regex or YAML code belongs in Support doesn't seem worth worrying
too much about, because they are very isolated, don't introduce extra
dependencies, and won't be linked by projects that don't use them.
>> 
>> This assumes that the client is statically linking against Support. 
Chris has been pretty explicit about his goals of building and using LLVM as a
monolithic dynamic library, in which case things like regex or YAML support will
*not* be automatically removed.
>> 
>> I see...
>> 
>> Maybe what we really need is (lacking a complete export list for the
monolithic dynamic library) to use visibility so that these kinds of things can
be dropped from the final dynamic library if they aren't used within it? I
suspect would could mark essentially all of the support library, as unless it is
used in the interface of some other library, we probably don't want to
export it... (Having just started thinking about this, I've no idea whether
folks using the LLVM dynamic library are independently using bits of the ADT for
example….)
> 
> To the best of my understanding, many linkers don’t have sufficiently
granular dead code stripping to rely on it for this.
> 
>> _______________________________________________
>> LLVM Developers mailing list
>> LLVMdev at cs.uiuc.edu <mailto:LLVMdev at cs.uiuc.edu>        
http://llvm.cs.uiuc.edu <http://llvm.cs.uiuc.edu/>
>> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
<http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev>
> 
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu <mailto:LLVMdev at cs.uiuc.edu>        
http://llvm.cs.uiuc.edu <http://llvm.cs.uiuc.edu/>
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
<http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20141107/02178f4b/attachment.html>

I’ve been playing around with trying to accomplish my goal without a new library
and I think that while it isn’t impossible, it is nasty and not ideal.

To recap, the goal I’m trying to accomplish is being able to build (from a
single configure) LLVM libraries that exclude crash handler support
(specifically the Darwin-specific abort and raise overrides), and LLVM tools
which include that functionality.

The simple naive solution to this would be to make those hooks implemented in
the headers, and conditionally compile it based on whether you’re building for a
tool or a library. Unfortunately this doesn’t really work in this case. Header
implemented functions need to be weak so that the linker can properly resolve
the duplicates. These symbols need to be strong so that they override the
versions present in the C library. That means that the only way to do this in a
header gets unwieldy because it would need to be a header that is included once
and only once in a tool, otherwise you’ll fail linking.

This just comes back to the ideal solution for this (to me) being separating out
this functionality into a new library.

Thoughts anyone?
-Chris
> On Nov 7, 2014, at 1:22 PM, Chris Bieneman <beanz at apple.com>
wrote:
> 
> To further this point, it was recently brought to my attention that the
CMake libLLVM.so on Linux is empty because the GNU linker on linux doesn’t treat
symbols mentioned in an export list as being used, so the linker dead strips it
all.
> 
> -Chris
> 
>> On Nov 7, 2014, at 12:28 PM, Jim Grosbach <grosbach at apple.com
<mailto:grosbach at apple.com>> wrote:
>> 
>>> 
>>> On Nov 7, 2014, at 11:05 AM, Chandler Carruth <chandlerc at
google.com <mailto:chandlerc at google.com>> wrote:
>>> 
>>> 
>>> On Fri, Nov 7, 2014 at 12:20 PM, Owen Anderson <resistor at
mac.com <mailto:resistor at mac.com>> wrote:
>>>> On Nov 7, 2014, at 8:57 AM, Daniel Dunbar <daniel at
zuster.org <mailto:daniel at zuster.org>> wrote:
>>>> 
>>>> 2. I agree with Chandler in that I don't see a good need to
try hard to factor out code from the Support library that can just be
conditional disabled or would be unused by normal .a link semantics. For
example, whether or not the regex or YAML code belongs in Support doesn't
seem worth worrying too much about, because they are very isolated, don't
introduce extra dependencies, and won't be linked by projects that don't
use them.
>>> 
>>> This assumes that the client is statically linking against Support.
Chris has been pretty explicit about his goals of building and using LLVM as a
monolithic dynamic library, in which case things like regex or YAML support will
*not* be automatically removed.
>>> 
>>> I see...
>>> 
>>> Maybe what we really need is (lacking a complete export list for
the monolithic dynamic library) to use visibility so that these kinds of things
can be dropped from the final dynamic library if they aren't used within it?
I suspect would could mark essentially all of the support library, as unless it
is used in the interface of some other library, we probably don't want to
export it... (Having just started thinking about this, I've no idea whether
folks using the LLVM dynamic library are independently using bits of the ADT for
example….)
>> 
>> To the best of my understanding, many linkers don’t have sufficiently
granular dead code stripping to rely on it for this.
>> 
>>> _______________________________________________
>>> LLVM Developers mailing list
>>> LLVMdev at cs.uiuc.edu <mailto:LLVMdev at cs.uiuc.edu>       
http://llvm.cs.uiuc.edu <http://llvm.cs.uiuc.edu/>
>>> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
<http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev>
>> 
>> _______________________________________________
>> LLVM Developers mailing list
>> LLVMdev at cs.uiuc.edu <mailto:LLVMdev at cs.uiuc.edu>        
http://llvm.cs.uiuc.edu <http://llvm.cs.uiuc.edu/>
>> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
<http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev>
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20141110/70825b5f/attachment.html>

> On Nov 10, 2014, at 1:01 PM, Chris Bieneman <beanz at apple.com>
wrote:
> 
> I’ve been playing around with trying to accomplish my goal without a new
library and I think that while it isn’t impossible, it is nasty and not ideal.
> 
> To recap, the goal I’m trying to accomplish is being able to build (from a
single configure) LLVM libraries that exclude crash handler support
(specifically the Darwin-specific abort and raise overrides), and LLVM tools
which include that functionality.
> 
> The simple naive solution to this would be to make those hooks implemented
in the headers, and conditionally compile it based on whether you’re building
for a tool or a library. Unfortunately this doesn’t really work in this case.
Header implemented functions need to be weak so that the linker can properly
resolve the duplicates. These symbols need to be strong so that they override
the versions present in the C library. That means that the only way to do this
in a header gets unwieldy because it would need to be a header that is included
once and only once in a tool, otherwise you’ll fail linking.
> 
> This just comes back to the ideal solution for this (to me) being
separating out this functionality into a new library.
> 
> Thoughts anyone?
SGTM.  I’d prefer to avoid building bits I don’t need than relying on linker
stripping anyway, so a separate library is the only way to achieve that too.

Pete
> -Chris
> 
>> On Nov 7, 2014, at 1:22 PM, Chris Bieneman <beanz at apple.com
<mailto:beanz at apple.com>> wrote:
>> 
>> To further this point, it was recently brought to my attention that the
CMake libLLVM.so on Linux is empty because the GNU linker on linux doesn’t treat
symbols mentioned in an export list as being used, so the linker dead strips it
all.
>> 
>> -Chris
>> 
>>> On Nov 7, 2014, at 12:28 PM, Jim Grosbach <grosbach at apple.com
<mailto:grosbach at apple.com>> wrote:
>>> 
>>>> 
>>>> On Nov 7, 2014, at 11:05 AM, Chandler Carruth <chandlerc at
google.com <mailto:chandlerc at google.com>> wrote:
>>>> 
>>>> 
>>>> On Fri, Nov 7, 2014 at 12:20 PM, Owen Anderson <resistor at
mac.com <mailto:resistor at mac.com>> wrote:
>>>>> On Nov 7, 2014, at 8:57 AM, Daniel Dunbar <daniel at
zuster.org <mailto:daniel at zuster.org>> wrote:
>>>>> 
>>>>> 2. I agree with Chandler in that I don't see a good
need to try hard to factor out code from the Support library that can just be
conditional disabled or would be unused by normal .a link semantics. For
example, whether or not the regex or YAML code belongs in Support doesn't
seem worth worrying too much about, because they are very isolated, don't
introduce extra dependencies, and won't be linked by projects that don't
use them.
>>>> 
>>>> This assumes that the client is statically linking against
Support.  Chris has been pretty explicit about his goals of building and using
LLVM as a monolithic dynamic library, in which case things like regex or YAML
support will *not* be automatically removed.
>>>> 
>>>> I see...
>>>> 
>>>> Maybe what we really need is (lacking a complete export list
for the monolithic dynamic library) to use visibility so that these kinds of
things can be dropped from the final dynamic library if they aren't used
within it? I suspect would could mark essentially all of the support library, as
unless it is used in the interface of some other library, we probably don't
want to export it... (Having just started thinking about this, I've no idea
whether folks using the LLVM dynamic library are independently using bits of the
ADT for example….)
>>> 
>>> To the best of my understanding, many linkers don’t have
sufficiently granular dead code stripping to rely on it for this.
>>> 
>>>> _______________________________________________
>>>> LLVM Developers mailing list
>>>> LLVMdev at cs.uiuc.edu <mailto:LLVMdev at cs.uiuc.edu>   
http://llvm.cs.uiuc.edu <http://llvm.cs.uiuc.edu/>
>>>> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
<http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev>
>>> 
>>> _______________________________________________
>>> LLVM Developers mailing list
>>> LLVMdev at cs.uiuc.edu <mailto:LLVMdev at cs.uiuc.edu>       
http://llvm.cs.uiuc.edu <http://llvm.cs.uiuc.edu/>
>>> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
<http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev>
>> _______________________________________________
>> LLVM Developers mailing list
>> LLVMdev at cs.uiuc.edu <mailto:LLVMdev at cs.uiuc.edu>        
http://llvm.cs.uiuc.edu <http://llvm.cs.uiuc.edu/>
>> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
<http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev>
> 
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20141110/e27f234a/attachment.html>

On 10 November 2014 16:30, Pete Cooper <peter_cooper at apple.com>
wrote:
>
> On Nov 10, 2014, at 1:01 PM, Chris Bieneman <beanz at apple.com>
wrote:
>
> I’ve been playing around with trying to accomplish my goal without a new
> library and I think that while it isn’t impossible, it is nasty and not
> ideal.
>
> To recap, the goal I’m trying to accomplish is being able to build (from a
> single configure) LLVM libraries that exclude crash handler support
> (specifically the Darwin-specific abort and raise overrides), and LLVM
tools
> which include that functionality.
>
> The simple naive solution to this would be to make those hooks implemented
> in the headers, and conditionally compile it based on whether you’re
> building for a tool or a library. Unfortunately this doesn’t really work in
> this case. Header implemented functions need to be weak so that the linker
> can properly resolve the duplicates. These symbols need to be strong so
that
> they override the versions present in the C library. That means that the
> only way to do this in a header gets unwieldy because it would need to be a
> header that is included once and only once in a tool, otherwise you’ll fail
> linking.
>
> This just comes back to the ideal solution for this (to me) being
separating
> out this functionality into a new library.
>
> Thoughts anyone?
>
> SGTM.  I’d prefer to avoid building bits I don’t need than relying on
linker
> stripping anyway, so a separate library is the only way to achieve that
too.
+1, lib/Support is in the critical path of a lot of builds.

Cheers,
Rafael

Here are updated patches for adding libToolsSupport.

I still haven’t done the Windows-side of the code, but I did modify the patches
based on some earlier feedback.

Does this seem like a good approach?

-Chris
> On Nov 10, 2014, at 2:34 PM, Rafael Espíndola <rafael.espindola at
gmail.com> wrote:
> 
> On 10 November 2014 16:30, Pete Cooper <peter_cooper at apple.com
<mailto:peter_cooper at apple.com>> wrote:
>> 
>> On Nov 10, 2014, at 1:01 PM, Chris Bieneman <beanz at apple.com>
wrote:
>> 
>> I’ve been playing around with trying to accomplish my goal without a
new
>> library and I think that while it isn’t impossible, it is nasty and not
>> ideal.
>> 
>> To recap, the goal I’m trying to accomplish is being able to build
(from a
>> single configure) LLVM libraries that exclude crash handler support
>> (specifically the Darwin-specific abort and raise overrides), and LLVM
tools
>> which include that functionality.
>> 
>> The simple naive solution to this would be to make those hooks
implemented
>> in the headers, and conditionally compile it based on whether you’re
>> building for a tool or a library. Unfortunately this doesn’t really
work in
>> this case. Header implemented functions need to be weak so that the
linker
>> can properly resolve the duplicates. These symbols need to be strong so
that
>> they override the versions present in the C library. That means that
the
>> only way to do this in a header gets unwieldy because it would need to
be a
>> header that is included once and only once in a tool, otherwise you’ll
fail
>> linking.
>> 
>> This just comes back to the ideal solution for this (to me) being
separating
>> out this functionality into a new library.
>> 
>> Thoughts anyone?
>> 
>> SGTM.  I’d prefer to avoid building bits I don’t need than relying on
linker
>> stripping anyway, so a separate library is the only way to achieve that
too.
> 
> +1, lib/Support is in the critical path of a lot of builds.
> 
> Cheers,
> Rafael
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20141111/731dfef4/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: libLLVMToolsSupport.diff
Type: application/octet-stream
Size: 41606 bytes
Desc: not available
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20141111/731dfef4/attachment.obj>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20141111/731dfef4/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: libLLVMToolsSupport-clang.diff
Type: application/octet-stream
Size: 5160 bytes
Desc: not available
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20141111/731dfef4/attachment-0001.obj>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20141111/731dfef4/attachment-0002.html>