llvm dev - Apr 2018 - [cfe-dev] [RFC] Open sourcing and contributing TAPI back to the LLVM community

On Mon, Apr 9, 2018 at 10:11 PM, John Ericson via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> > Regardless of any of that, given that TBD files _are_ an integral part
> of the apple platform, supporting them is certainly a necessity in order to
> have a working apple linker. So, if making LLD work for Apple/MachO is the
> justification for adding TBD support to LLVM, that seems self-evidently a
> reasonable thing to do. On the other hand, it looks like the LLD mach-o
> code is unmaintained and nobody seems to be much interested in it. And
> having code for reading TBD files in LLVM seems not terribly interesting,
> unless it is as part of a project to make the LLD MachO linker actually
> functional and supported.
>
> Yes. I hope this can be reason enough. Hobbyists could push for LLD
> support for Mach-O besides Apple, and if LLD is to displace other linkers
> this is a necessary component as you say. Better to upstream now before the
> code diverges than more work later? Conversely if nothing happens, I doubt
> libtapi would be a greater drag on the codebase than the MachO LLD code, so
> whatever cost/benefit analysis exists for keeping that around could also
> apply to this.


Speaking for the Zig project here, our goal is to support cross-compilation
for any target, on any target, without requiring installation of any
target-specific SDK. So, for example, these use cases:
 * on linux, compile & link a binary targeting macos
 * on windows, compile & link a binary targeting macos

This works today, although it depends on a patch to LLD to fix the MACH-O
linker that is not high enough quality to upstream.

So we have a vested interest in improving the MACH-O linker, and in fact a
Zig community member has fixed at least one bug in MACH-O LLD:
reviews.llvm.org/D35387

I don't fully understand how TBD or TAPI works, but I hope that it results
in improvements to the MACH-O linker.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20180410/8d503873/attachment.html>

Benifits of TBD:
1) It's human readable and diffs on TBDs correspond to changes in the ABI.
Diffs can be automatically added to review processes to ensure that changes
to the ABI are reviewed. The TBDs also document your precise ABI.
2) The size is smaller which means they can be shipped in an SDK instead of
binaries to reduce the size of an SDK
3) Stubs are producible from TBDs (or should be) which means stubs for
linking can be produced even if we don't directly support them in LLD. This
lets you ship the smaller TBD files in place of larger binaries and still
link things without direct linker support (assuming you already ship a
toolchain with your SDK or expect your users to have this tool)

Since stubs are producible from TBDs I don't really see a downside. I think
we need both, I was going to propose a yaml based representation for ELF
for the above reasons anyhow.

On Tue, Apr 10, 2018 at 1:14 PM Andrew Kelley via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> On Mon, Apr 9, 2018 at 10:11 PM, John Ericson via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>> > Regardless of any of that, given that TBD files _are_ an integral
part
>> of the apple platform, supporting them is certainly a necessity in
order to
>> have a working apple linker. So, if making LLD work for Apple/MachO is
the
>> justification for adding TBD support to LLVM, that seems self-evidently
a
>> reasonable thing to do. On the other hand, it looks like the LLD mach-o
>> code is unmaintained and nobody seems to be much interested in it. And
>> having code for reading TBD files in LLVM seems not terribly
interesting,
>> unless it is as part of a project to make the LLD MachO linker actually
>> functional and supported.
>>
>> Yes. I hope this can be reason enough. Hobbyists could push for LLD
>> support for Mach-O besides Apple, and if LLD is to displace other
linkers
>> this is a necessary component as you say. Better to upstream now before
the
>> code diverges than more work later? Conversely if nothing happens, I
doubt
>> libtapi would be a greater drag on the codebase than the MachO LLD
code, so
>> whatever cost/benefit analysis exists for keeping that around could
also
>> apply to this.
>
>
>
> Speaking for the Zig project here, our goal is to support
> cross-compilation for any target, on any target, without requiring
> installation of any target-specific SDK. So, for example, these use cases:
>  * on linux, compile & link a binary targeting macos
>  * on windows, compile & link a binary targeting macos
>
> This works today, although it depends on a patch to LLD to fix the MACH-O
> linker that is not high enough quality to upstream.
>
> So we have a vested interest in improving the MACH-O linker, and in fact a
> Zig community member has fixed at least one bug in MACH-O LLD:
> reviews.llvm.org/D35387
>
> I don't fully understand how TBD or TAPI works, but I hope that it
results
> in improvements to the MACH-O linker.
>
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20180410/6d4b0b14/attachment-0001.html>

Seems like there are a few of us interested in this then. I new around 
here and don't really know how decisions are made, so what's next? Just 
open a diff with the entire library??

John

On 04/10/2018 05:33 PM, Jake Ehrlich wrote:
> Benifits of TBD:
> 1) It's human readable and diffs on TBDs correspond to changes in the 
> ABI. Diffs can be automatically added to review processes to ensure 
> that changes to the ABI are reviewed. The TBDs also document your 
> precise ABI.
> 2) The size is smaller which means they can be shipped in an SDK 
> instead of binaries to reduce the size of an SDK
> 3) Stubs are producible from TBDs (or should be) which means stubs for 
> linking can be produced even if we don't directly support them in LLD. 
> This lets you ship the smaller TBD files in place of larger binaries 
> and still link things without direct linker support (assuming you 
> already ship a toolchain with your SDK or expect your users to have 
> this tool)
>
> Since stubs are producible from TBDs I don't really see a downside. I 
> think we need both, I was going to propose a yaml based representation 
> for ELF for the above reasons anyhow.
>
> On Tue, Apr 10, 2018 at 1:14 PM Andrew Kelley via llvm-dev 
> <llvm-dev at lists.llvm.org <mailto:llvm-dev at
lists.llvm.org>> wrote:
>
>     On Mon, Apr 9, 2018 at 10:11 PM, John Ericson via llvm-dev
>     <llvm-dev at lists.llvm.org <mailto:llvm-dev at
lists.llvm.org>> wrote:
>
>         > Regardless of any of that, given that TBD files _are_ an
integral part
>         of the apple platform, supporting them is certainly a
>         necessity in order to have a working apple linker. So, if
>         making LLD work for Apple/MachO is the justification for
>         adding TBD support to LLVM, that seems self-evidently a
>         reasonable thing to do. On the other hand, it looks like the
>         LLD mach-o code is unmaintained and nobody seems to be much
>         interested in it. And having code for reading TBD files in
>         LLVM seems not terribly interesting, unless it is as part of a
>         project to make the LLD MachO linker actually functional and
>         supported.
>
>         Yes. I hope this can be reason enough. Hobbyists could push
>         for LLD support for Mach-O besides Apple, and if LLD is to
>         displace other linkers this is a necessary component as you
>         say. Better to upstream now before the code diverges than more
>         work later? Conversely if nothing happens, I doubt libtapi
>         would be a greater drag on the codebase than the MachO LLD
>         code, so whatever cost/benefit analysis exists for keeping
>         that around could also apply to this.
>
>
>
>     Speaking for the Zig project here, our goal is to support
>     cross-compilation for any target, on any target, without requiring
>     installation of any target-specific SDK. So, for example, these
>     use cases:
>      * on linux, compile & link a binary targeting macos
>      * on windows, compile & link a binary targeting macos
>
>     This works today, although it depends on a patch to LLD to fix the
>     MACH-O linker that is not high enough quality to upstream.
>
>     So we have a vested interest in improving the MACH-O linker, and
>     in fact a Zig community member has fixed at least one bug in
>     MACH-O LLD: reviews.llvm.org/D35387
<http://reviews.llvm.org/D35387>
>
>     I don't fully understand how TBD or TAPI works, but I hope that it
>     results in improvements to the MACH-O linker.
>
>     _______________________________________________
>     LLVM Developers mailing list
>     llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
>     http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20180410/ed551f3f/attachment.html>

On Tue, Apr 10, 2018 at 5:33 PM Jake Ehrlich via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> Benifits of TBD:
> 1) It's human readable and diffs on TBDs correspond to changes in the
ABI.
> Diffs can be automatically added to review processes to ensure that changes
> to the ABI are reviewed. The TBDs also document your precise ABI.
> 2) The size is smaller which means they can be shipped in an SDK instead
> of binaries to reduce the size of an SDK
>
I'm still skeptical that this is significant.

3) Stubs are producible from TBDs (or should be) which means stubs
for
> linking can be produced even if we don't directly support them in LLD.
This
> lets you ship the smaller TBD files in place of larger binaries and still
> link things without direct linker support (assuming you already ship a
> toolchain with your SDK or expect your users to have this tool)
>
> Since stubs are producible from TBDs I don't really see a downside. I
> think we need both, I was going to propose a yaml based representation for
> ELF for the above reasons anyhow.
>
Yea, a tool which can produce a .so from a textual description is certainly
much less concerning than adding linker support for a new textual
description format. If it's an official linker-supported format, it'd be
yet another format that potentially needs to be standardized across
multiple linkers, and kept compatible for"ever", etc. I just don't
think
that seems worthwhile for ELF.

OTOH, a standalone tool which can convert from a "full" shared-object
to an
interface shared-object would be _great_ to have. If that tool also has
some auxiliary textual I/O format it supports, I guess that's fine, too.
(We do have some existing yaml <-> ELF support, via the
"obj2yaml" and
"yaml2obj" tools.)

I'd note that reproducing all the things that are required/used from an ELF
shared object during linking -- symbol type, binding-type, visibility,
version, alignment (!), .gnu.warning messages, various important
"SHT_NOTE"
sections, and whatever other things I've forgotten about, will need to be a
_significantly_ different format than what Apple has as their "TBD"
format.
Apple's format also has a bunch of special cases in it to make it easier to
use for their platform, but a rather less generic tool. E.g., symbols
starting with "_OBJC_CLASS_$" are recorded in the
"objc-classes" field with
the prefix removed, instead of just recording it as-is.

So, I'd also caution that while the project of "import apple's
libtapi into
LLVM for LLD/MachO" and "Make a scheme to do interface shared-libs for
ELF"
might seem superficially related, I'd be very surprised if that actually
ended up being the case. I would really not expect it to share just about
anything at all other than the concept of being a textual description for a
library.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20180411/b9064b3c/attachment.html>