llvm dev - Sep 2018 - [cfe-dev] [RFC] Open sourcing and contributing TAPI back to the LLVM community

That sounds great to me, thanks Jake. I'm not Jurgen either, of course, 
but I'm happy to assist you if he is unavailable. I'm not also not 
qualified to audit the license, but do note Apple formally also released 
some code at https://opensource.apple.com/tarballs/tapi/. If there's 
anything else I can do to help, let me know.

Cheers,

John


On 04/10/2018 06:13 PM, Jake Ehrlich wrote:
> Ideally Jurgen would cut up the code on github, put up an initial diff 
> for a minimal viable tool, and then we would review it and then 
> continue to copy code from the github repo into llvm and review it. 
> I'm also willing to do that if Jurgen doesn't want to at this point
> though. I'd like the OK from Jurgen on that and I'd also like the
OK
> from someone that the license stuff is all good to go (I'm not sure 
> who should check licence stuff).
>
> Best,
> Jake
>
> On Tue, Apr 10, 2018 at 2:39 PM John Ericson 
> <john.ericson at obsidian.systems> wrote:
>
>     Seems like there are a few of us interested in this then. I new
>     around here and don't really know how decisions are made, so
>     what's next? Just open a diff with the entire library??
>
>     John
>
>     On 04/10/2018 05:33 PM, Jake Ehrlich wrote:
>>     Benifits of TBD:
>>     1) It's human readable and diffs on TBDs correspond to changes
in
>>     the ABI. Diffs can be automatically added to review processes to
>>     ensure that changes to the ABI are reviewed. The TBDs also
>>     document your precise ABI.
>>     2) The size is smaller which means they can be shipped in an SDK
>>     instead of binaries to reduce the size of an SDK
>>     3) Stubs are producible from TBDs (or should be) which means
>>     stubs for linking can be produced even if we don't directly
>>     support them in LLD. This lets you ship the smaller TBD files in
>>     place of larger binaries and still link things without direct
>>     linker support (assuming you already ship a toolchain with your
>>     SDK or expect your users to have this tool)
>>
>>     Since stubs are producible from TBDs I don't really see a
>>     downside. I think we need both, I was going to propose a yaml
>>     based representation for ELF for the above reasons anyhow.
>>
>>     On Tue, Apr 10, 2018 at 1:14 PM Andrew Kelley via llvm-dev
>>     <llvm-dev at lists.llvm.org <mailto:llvm-dev at
lists.llvm.org>> wrote:
>>
>>         On Mon, Apr 9, 2018 at 10:11 PM, John Ericson via llvm-dev
>>         <llvm-dev at lists.llvm.org <mailto:llvm-dev at
lists.llvm.org>> wrote:
>>
>>             > Regardless of any of that, given that TBD files _are_
an
>>             integral part of the apple platform, supporting them is
>>             certainly a necessity in order to have a working apple
>>             linker. So, if making LLD work for Apple/MachO is the
>>             justification for adding TBD support to LLVM, that seems
>>             self-evidently a reasonable thing to do. On the other
>>             hand, it looks like the LLD mach-o code is unmaintained
>>             and nobody seems to be much interested in it. And having
>>             code for reading TBD files in LLVM seems not terribly
>>             interesting, unless it is as part of a project to make
>>             the LLD MachO linker actually functional and supported.
>>
>>             Yes. I hope this can be reason enough. Hobbyists could
>>             push for LLD support for Mach-O besides Apple, and if LLD
>>             is to displace other linkers this is a necessary
>>             component as you say. Better to upstream now before the
>>             code diverges than more work later? Conversely if nothing
>>             happens, I doubt libtapi would be a greater drag on the
>>             codebase than the MachO LLD code, so whatever
>>             cost/benefit analysis exists for keeping that around
>>             could also apply to this.
>>
>>
>>
>>         Speaking for the Zig project here, our goal is to support
>>         cross-compilation for any target, on any target, without
>>         requiring installation of any target-specific SDK. So, for
>>         example, these use cases:
>>          * on linux, compile & link a binary targeting macos
>>          * on windows, compile & link a binary targeting macos
>>
>>         This works today, although it depends on a patch to LLD to
>>         fix the MACH-O linker that is not high enough quality to
>>         upstream.
>>
>>         So we have a vested interest in improving the MACH-O linker,
>>         and in fact a Zig community member has fixed at least one bug
>>         in MACH-O LLD: reviews.llvm.org/D35387
>>         <http://reviews.llvm.org/D35387>
>>
>>         I don't fully understand how TBD or TAPI works, but I hope
>>         that it results in improvements to the MACH-O linker.
>>
>>         _______________________________________________
>>         LLVM Developers mailing list
>>         llvm-dev at lists.llvm.org <mailto:llvm-dev at
lists.llvm.org>
>>         http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20180410/b5e844b3/attachment.html>

Also I mainly care about getting the ELF part of this working so it would
be nice to have an informal owner of the MachO part.

On Tue, Apr 10, 2018 at 4:15 PM John Ericson <john.ericson at
obsidian.systems>
wrote:
> That sounds great to me, thanks Jake. I'm not Jurgen either, of course,
> but I'm happy to assist you if he is unavailable. I'm not also not
> qualified to audit the license, but do note Apple formally also released
> some code at https://opensource.apple.com/tarballs/tapi/. If there's
> anything else I can do to help, let me know.
>
> Cheers,
>
> John
>
> On 04/10/2018 06:13 PM, Jake Ehrlich wrote:
>
> Ideally Jurgen would cut up the code on github, put up an initial diff for
> a minimal viable tool, and then we would review it and then continue to
> copy code from the github repo into llvm and review it. I'm also
willing to
> do that if Jurgen doesn't want to at this point though. I'd like
the OK
> from Jurgen on that and I'd also like the OK from someone that the
license
> stuff is all good to go (I'm not sure who should check licence stuff).
>
> Best,
> Jake
>
> On Tue, Apr 10, 2018 at 2:39 PM John Ericson
> <john.ericson at obsidian.systems> <john.ericson at
obsidian.systems> wrote:
>
>> Seems like there are a few of us interested in this then. I new around
>> here and don't really know how decisions are made, so what's
next? Just
>> open a diff with the entire library??
>>
>> John
>> On 04/10/2018 05:33 PM, Jake Ehrlich wrote:
>>
>> Benifits of TBD:
>> 1) It's human readable and diffs on TBDs correspond to changes in
the
>> ABI. Diffs can be automatically added to review processes to ensure
that
>> changes to the ABI are reviewed. The TBDs also document your precise
ABI.
>> 2) The size is smaller which means they can be shipped in an SDK
instead
>> of binaries to reduce the size of an SDK
>> 3) Stubs are producible from TBDs (or should be) which means stubs for
>> linking can be produced even if we don't directly support them in
LLD. This
>> lets you ship the smaller TBD files in place of larger binaries and
still
>> link things without direct linker support (assuming you already ship a
>> toolchain with your SDK or expect your users to have this tool)
>>
>> Since stubs are producible from TBDs I don't really see a downside.
I
>> think we need both, I was going to propose a yaml based representation
for
>> ELF for the above reasons anyhow.
>>
>> On Tue, Apr 10, 2018 at 1:14 PM Andrew Kelley via llvm-dev <
>> llvm-dev at lists.llvm.org> wrote:
>>
>>> On Mon, Apr 9, 2018 at 10:11 PM, John Ericson via llvm-dev <
>>> llvm-dev at lists.llvm.org> wrote:
>>>
>>>> > Regardless of any of that, given that TBD files _are_ an
integral
>>>> part of the apple platform, supporting them is certainly a
necessity in
>>>> order to have a working apple linker. So, if making LLD work
for
>>>> Apple/MachO is the justification for adding TBD support to
LLVM, that seems
>>>> self-evidently a reasonable thing to do. On the other hand, it
looks like
>>>> the LLD mach-o code is unmaintained and nobody seems to be much
interested
>>>> in it. And having code for reading TBD files in LLVM seems not
terribly
>>>> interesting, unless it is as part of a project to make the LLD
MachO linker
>>>> actually functional and supported.
>>>>
>>>> Yes. I hope this can be reason enough. Hobbyists could push for
LLD
>>>> support for Mach-O besides Apple, and if LLD is to displace
other linkers
>>>> this is a necessary component as you say. Better to upstream
now before the
>>>> code diverges than more work later? Conversely if nothing
happens, I doubt
>>>> libtapi would be a greater drag on the codebase than the MachO
LLD code, so
>>>> whatever cost/benefit analysis exists for keeping that around
could also
>>>> apply to this.
>>>
>>>
>>>
>>> Speaking for the Zig project here, our goal is to support
>>> cross-compilation for any target, on any target, without requiring
>>> installation of any target-specific SDK. So, for example, these use
cases:
>>>  * on linux, compile & link a binary targeting macos
>>>  * on windows, compile & link a binary targeting macos
>>>
>>> This works today, although it depends on a patch to LLD to fix the
>>> MACH-O linker that is not high enough quality to upstream.
>>>
>>> So we have a vested interest in improving the MACH-O linker, and in
fact
>>> a Zig community member has fixed at least one bug in MACH-O LLD:
>>> reviews.llvm.org/D35387
>>>
>>> I don't fully understand how TBD or TAPI works, but I hope that
it
>>> results in improvements to the MACH-O linker.
>>>
>>>
>>> _______________________________________________
>>> LLVM Developers mailing list
>>> llvm-dev at lists.llvm.org
>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20180410/675887d8/attachment-0001.html>

The code on opensource.apple.com is the minimal code needed for libtapi to
read TBD files with ld64. The code I pushed to GitHub on the other side
includes the full TAPI tool source code. If you check the code you will see
that I already use the LLVM license for everything, because the goal has
always been to contribute this back to the community.

I attached an initial patch for libtapi and nm support to my original
email. I also have some updated patches for that too, but somehow got
derailed into "lets rewrite libobject" discussions.

On Tue, Apr 10, 2018 at 4:15 PM, John Ericson via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> That sounds great to me, thanks Jake. I'm not Jurgen either, of course,
> but I'm happy to assist you if he is unavailable. I'm not also not
> qualified to audit the license, but do note Apple formally also released
> some code at https://opensource.apple.com/tarballs/tapi/. If there's
> anything else I can do to help, let me know.
>
> Cheers,
>
> John
>
> On 04/10/2018 06:13 PM, Jake Ehrlich wrote:
>
> Ideally Jurgen would cut up the code on github, put up an initial diff for
> a minimal viable tool, and then we would review it and then continue to
> copy code from the github repo into llvm and review it. I'm also
willing to
> do that if Jurgen doesn't want to at this point though. I'd like
the OK
> from Jurgen on that and I'd also like the OK from someone that the
license
> stuff is all good to go (I'm not sure who should check licence stuff).
>
> Best,
> Jake
>
> On Tue, Apr 10, 2018 at 2:39 PM John Ericson <john.ericson at obsidian.
> systems> <john.ericson at obsidian.systems> wrote:
>
>> Seems like there are a few of us interested in this then. I new around
>> here and don't really know how decisions are made, so what's
next? Just
>> open a diff with the entire library??
>>
>> John
>> On 04/10/2018 05:33 PM, Jake Ehrlich wrote:
>>
>> Benifits of TBD:
>> 1) It's human readable and diffs on TBDs correspond to changes in
the
>> ABI. Diffs can be automatically added to review processes to ensure
that
>> changes to the ABI are reviewed. The TBDs also document your precise
ABI.
>> 2) The size is smaller which means they can be shipped in an SDK
instead
>> of binaries to reduce the size of an SDK
>> 3) Stubs are producible from TBDs (or should be) which means stubs for
>> linking can be produced even if we don't directly support them in
LLD. This
>> lets you ship the smaller TBD files in place of larger binaries and
still
>> link things without direct linker support (assuming you already ship a
>> toolchain with your SDK or expect your users to have this tool)
>>
>> Since stubs are producible from TBDs I don't really see a downside.
I
>> think we need both, I was going to propose a yaml based representation
for
>> ELF for the above reasons anyhow.
>>
>> On Tue, Apr 10, 2018 at 1:14 PM Andrew Kelley via llvm-dev <
>> llvm-dev at lists.llvm.org> wrote:
>>
>>> On Mon, Apr 9, 2018 at 10:11 PM, John Ericson via llvm-dev <
>>> llvm-dev at lists.llvm.org> wrote:
>>>
>>>> > Regardless of any of that, given that TBD files _are_ an
integral
>>>> part of the apple platform, supporting them is certainly a
necessity in
>>>> order to have a working apple linker. So, if making LLD work
for
>>>> Apple/MachO is the justification for adding TBD support to
LLVM, that seems
>>>> self-evidently a reasonable thing to do. On the other hand, it
looks like
>>>> the LLD mach-o code is unmaintained and nobody seems to be much
interested
>>>> in it. And having code for reading TBD files in LLVM seems not
terribly
>>>> interesting, unless it is as part of a project to make the LLD
MachO linker
>>>> actually functional and supported.
>>>>
>>>> Yes. I hope this can be reason enough. Hobbyists could push for
LLD
>>>> support for Mach-O besides Apple, and if LLD is to displace
other linkers
>>>> this is a necessary component as you say. Better to upstream
now before the
>>>> code diverges than more work later? Conversely if nothing
happens, I doubt
>>>> libtapi would be a greater drag on the codebase than the MachO
LLD code, so
>>>> whatever cost/benefit analysis exists for keeping that around
could also
>>>> apply to this.
>>>
>>>
>>>
>>> Speaking for the Zig project here, our goal is to support
>>> cross-compilation for any target, on any target, without requiring
>>> installation of any target-specific SDK. So, for example, these use
cases:
>>>  * on linux, compile & link a binary targeting macos
>>>  * on windows, compile & link a binary targeting macos
>>>
>>> This works today, although it depends on a patch to LLD to fix the
>>> MACH-O linker that is not high enough quality to upstream.
>>>
>>> So we have a vested interest in improving the MACH-O linker, and in
fact
>>> a Zig community member has fixed at least one bug in MACH-O LLD:
>>> reviews.llvm.org/D35387
>>>
>>> I don't fully understand how TBD or TAPI works, but I hope that
it
>>> results in improvements to the MACH-O linker.
>>>
>>>
>>> _______________________________________________
>>> LLVM Developers mailing list
>>> llvm-dev at lists.llvm.org
>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>
>>
>>
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20180425/9e0d95fa/attachment-0001.html>

Was there any progress in the upstreaming effort? I'd be interested in
having lld be able to link against tbd files, and I think it'd be cool if
libtool -static could write tbd files (similar to thin archives on linux)
since that should make archiving much faster.

Juergen, maybe uploading your initial patch to phabricator instead of
attaching might get more traction?

On Wed, Apr 25, 2018 at 8:53 AM Juergen Ributzka via cfe-dev <
cfe-dev at lists.llvm.org> wrote:
> The code on opensource.apple.com is the minimal code needed for libtapi
> to read TBD files with ld64. The code I pushed to GitHub on the other side
> includes the full TAPI tool source code. If you check the code you will see
> that I already use the LLVM license for everything, because the goal has
> always been to contribute this back to the community.
>
> I attached an initial patch for libtapi and nm support to my original
> email. I also have some updated patches for that too, but somehow got
> derailed into "lets rewrite libobject" discussions.
>
> On Tue, Apr 10, 2018 at 4:15 PM, John Ericson via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>> That sounds great to me, thanks Jake. I'm not Jurgen either, of
course,
>> but I'm happy to assist you if he is unavailable. I'm not also
not
>> qualified to audit the license, but do note Apple formally also
released
>> some code at https://opensource.apple.com/tarballs/tapi/. If
there's
>> anything else I can do to help, let me know.
>>
>> Cheers,
>>
>> John
>>
>> On 04/10/2018 06:13 PM, Jake Ehrlich wrote:
>>
>> Ideally Jurgen would cut up the code on github, put up an initial diff
>> for a minimal viable tool, and then we would review it and then
continue to
>> copy code from the github repo into llvm and review it. I'm also
willing to
>> do that if Jurgen doesn't want to at this point though. I'd
like the OK
>> from Jurgen on that and I'd also like the OK from someone that the
license
>> stuff is all good to go (I'm not sure who should check licence
stuff).
>>
>> Best,
>> Jake
>>
>> On Tue, Apr 10, 2018 at 2:39 PM John Ericson
>> <john.ericson at obsidian.systems> <john.ericson at
obsidian.systems> wrote:
>>
>>> Seems like there are a few of us interested in this then. I new
around
>>> here and don't really know how decisions are made, so
what's next? Just
>>> open a diff with the entire library??
>>>
>>> John
>>> On 04/10/2018 05:33 PM, Jake Ehrlich wrote:
>>>
>>> Benifits of TBD:
>>> 1) It's human readable and diffs on TBDs correspond to changes
in the
>>> ABI. Diffs can be automatically added to review processes to ensure
that
>>> changes to the ABI are reviewed. The TBDs also document your
precise ABI.
>>> 2) The size is smaller which means they can be shipped in an SDK
instead
>>> of binaries to reduce the size of an SDK
>>> 3) Stubs are producible from TBDs (or should be) which means stubs
for
>>> linking can be produced even if we don't directly support them
in LLD. This
>>> lets you ship the smaller TBD files in place of larger binaries and
still
>>> link things without direct linker support (assuming you already
ship a
>>> toolchain with your SDK or expect your users to have this tool)
>>>
>>> Since stubs are producible from TBDs I don't really see a
downside. I
>>> think we need both, I was going to propose a yaml based
representation for
>>> ELF for the above reasons anyhow.
>>>
>>> On Tue, Apr 10, 2018 at 1:14 PM Andrew Kelley via llvm-dev <
>>> llvm-dev at lists.llvm.org> wrote:
>>>
>>>> On Mon, Apr 9, 2018 at 10:11 PM, John Ericson via llvm-dev <
>>>> llvm-dev at lists.llvm.org> wrote:
>>>>
>>>>> > Regardless of any of that, given that TBD files _are_
an integral
>>>>> part of the apple platform, supporting them is certainly a
necessity in
>>>>> order to have a working apple linker. So, if making LLD
work for
>>>>> Apple/MachO is the justification for adding TBD support to
LLVM, that seems
>>>>> self-evidently a reasonable thing to do. On the other hand,
it looks like
>>>>> the LLD mach-o code is unmaintained and nobody seems to be
much interested
>>>>> in it. And having code for reading TBD files in LLVM seems
not terribly
>>>>> interesting, unless it is as part of a project to make the
LLD MachO linker
>>>>> actually functional and supported.
>>>>>
>>>>> Yes. I hope this can be reason enough. Hobbyists could push
for LLD
>>>>> support for Mach-O besides Apple, and if LLD is to displace
other linkers
>>>>> this is a necessary component as you say. Better to
upstream now before the
>>>>> code diverges than more work later? Conversely if nothing
happens, I doubt
>>>>> libtapi would be a greater drag on the codebase than the
MachO LLD code, so
>>>>> whatever cost/benefit analysis exists for keeping that
around could also
>>>>> apply to this.
>>>>
>>>>
>>>>
>>>> Speaking for the Zig project here, our goal is to support
>>>> cross-compilation for any target, on any target, without
requiring
>>>> installation of any target-specific SDK. So, for example, these
use cases:
>>>>  * on linux, compile & link a binary targeting macos
>>>>  * on windows, compile & link a binary targeting macos
>>>>
>>>> This works today, although it depends on a patch to LLD to fix
the
>>>> MACH-O linker that is not high enough quality to upstream.
>>>>
>>>> So we have a vested interest in improving the MACH-O linker,
and in
>>>> fact a Zig community member has fixed at least one bug in
MACH-O LLD:
>>>> reviews.llvm.org/D35387
>>>>
>>>> I don't fully understand how TBD or TAPI works, but I hope
that it
>>>> results in improvements to the MACH-O linker.
>>>>
>>>>
>>>> _______________________________________________
>>>> LLVM Developers mailing list
>>>> llvm-dev at lists.llvm.org
>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>
>>>
>>>
>>
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
>>
> _______________________________________________
> cfe-dev mailing list
> cfe-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20180920/75fa648c/attachment-0001.html>