llvm dev - Sep 2018 - [cfe-dev] [RFC] Open sourcing and contributing TAPI back to the LLVM community

Was there any progress in the upstreaming effort? I'd be interested in
having lld be able to link against tbd files, and I think it'd be cool if
libtool -static could write tbd files (similar to thin archives on linux)
since that should make archiving much faster.

Juergen, maybe uploading your initial patch to phabricator instead of
attaching might get more traction?

On Wed, Apr 25, 2018 at 8:53 AM Juergen Ributzka via cfe-dev <
cfe-dev at lists.llvm.org> wrote:
> The code on opensource.apple.com is the minimal code needed for libtapi
> to read TBD files with ld64. The code I pushed to GitHub on the other side
> includes the full TAPI tool source code. If you check the code you will see
> that I already use the LLVM license for everything, because the goal has
> always been to contribute this back to the community.
>
> I attached an initial patch for libtapi and nm support to my original
> email. I also have some updated patches for that too, but somehow got
> derailed into "lets rewrite libobject" discussions.
>
> On Tue, Apr 10, 2018 at 4:15 PM, John Ericson via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>> That sounds great to me, thanks Jake. I'm not Jurgen either, of
course,
>> but I'm happy to assist you if he is unavailable. I'm not also
not
>> qualified to audit the license, but do note Apple formally also
released
>> some code at https://opensource.apple.com/tarballs/tapi/. If
there's
>> anything else I can do to help, let me know.
>>
>> Cheers,
>>
>> John
>>
>> On 04/10/2018 06:13 PM, Jake Ehrlich wrote:
>>
>> Ideally Jurgen would cut up the code on github, put up an initial diff
>> for a minimal viable tool, and then we would review it and then
continue to
>> copy code from the github repo into llvm and review it. I'm also
willing to
>> do that if Jurgen doesn't want to at this point though. I'd
like the OK
>> from Jurgen on that and I'd also like the OK from someone that the
license
>> stuff is all good to go (I'm not sure who should check licence
stuff).
>>
>> Best,
>> Jake
>>
>> On Tue, Apr 10, 2018 at 2:39 PM John Ericson
>> <john.ericson at obsidian.systems> <john.ericson at
obsidian.systems> wrote:
>>
>>> Seems like there are a few of us interested in this then. I new
around
>>> here and don't really know how decisions are made, so
what's next? Just
>>> open a diff with the entire library??
>>>
>>> John
>>> On 04/10/2018 05:33 PM, Jake Ehrlich wrote:
>>>
>>> Benifits of TBD:
>>> 1) It's human readable and diffs on TBDs correspond to changes
in the
>>> ABI. Diffs can be automatically added to review processes to ensure
that
>>> changes to the ABI are reviewed. The TBDs also document your
precise ABI.
>>> 2) The size is smaller which means they can be shipped in an SDK
instead
>>> of binaries to reduce the size of an SDK
>>> 3) Stubs are producible from TBDs (or should be) which means stubs
for
>>> linking can be produced even if we don't directly support them
in LLD. This
>>> lets you ship the smaller TBD files in place of larger binaries and
still
>>> link things without direct linker support (assuming you already
ship a
>>> toolchain with your SDK or expect your users to have this tool)
>>>
>>> Since stubs are producible from TBDs I don't really see a
downside. I
>>> think we need both, I was going to propose a yaml based
representation for
>>> ELF for the above reasons anyhow.
>>>
>>> On Tue, Apr 10, 2018 at 1:14 PM Andrew Kelley via llvm-dev <
>>> llvm-dev at lists.llvm.org> wrote:
>>>
>>>> On Mon, Apr 9, 2018 at 10:11 PM, John Ericson via llvm-dev <
>>>> llvm-dev at lists.llvm.org> wrote:
>>>>
>>>>> > Regardless of any of that, given that TBD files _are_
an integral
>>>>> part of the apple platform, supporting them is certainly a
necessity in
>>>>> order to have a working apple linker. So, if making LLD
work for
>>>>> Apple/MachO is the justification for adding TBD support to
LLVM, that seems
>>>>> self-evidently a reasonable thing to do. On the other hand,
it looks like
>>>>> the LLD mach-o code is unmaintained and nobody seems to be
much interested
>>>>> in it. And having code for reading TBD files in LLVM seems
not terribly
>>>>> interesting, unless it is as part of a project to make the
LLD MachO linker
>>>>> actually functional and supported.
>>>>>
>>>>> Yes. I hope this can be reason enough. Hobbyists could push
for LLD
>>>>> support for Mach-O besides Apple, and if LLD is to displace
other linkers
>>>>> this is a necessary component as you say. Better to
upstream now before the
>>>>> code diverges than more work later? Conversely if nothing
happens, I doubt
>>>>> libtapi would be a greater drag on the codebase than the
MachO LLD code, so
>>>>> whatever cost/benefit analysis exists for keeping that
around could also
>>>>> apply to this.
>>>>
>>>>
>>>>
>>>> Speaking for the Zig project here, our goal is to support
>>>> cross-compilation for any target, on any target, without
requiring
>>>> installation of any target-specific SDK. So, for example, these
use cases:
>>>>  * on linux, compile & link a binary targeting macos
>>>>  * on windows, compile & link a binary targeting macos
>>>>
>>>> This works today, although it depends on a patch to LLD to fix
the
>>>> MACH-O linker that is not high enough quality to upstream.
>>>>
>>>> So we have a vested interest in improving the MACH-O linker,
and in
>>>> fact a Zig community member has fixed at least one bug in
MACH-O LLD:
>>>> reviews.llvm.org/D35387
>>>>
>>>> I don't fully understand how TBD or TAPI works, but I hope
that it
>>>> results in improvements to the MACH-O linker.
>>>>
>>>>
>>>> _______________________________________________
>>>> LLVM Developers mailing list
>>>> llvm-dev at lists.llvm.org
>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>
>>>
>>>
>>
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
>>
> _______________________________________________
> cfe-dev mailing list
> cfe-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20180920/75fa648c/attachment-0001.html>

A member of my team +Armando Montanez <amontanez at google.com> is going
to
drop a proposal for the ELF part of this soon (like sometime next week) and
will be working on the implementation. I'll be one of the reviewers for
anything that comes out of that so we can be sure it will get reviewed as
well. The goal is to have a single tool/library that would include
Juergen's original code and the new ELF code.

On Thu, Sep 20, 2018 at 10:45 AM Nico Weber <thakis at chromium.org>
wrote:
> Was there any progress in the upstreaming effort? I'd be interested in
> having lld be able to link against tbd files, and I think it'd be cool
if
> libtool -static could write tbd files (similar to thin archives on linux)
> since that should make archiving much faster.
>
> Juergen, maybe uploading your initial patch to phabricator instead of
> attaching might get more traction?
>
> On Wed, Apr 25, 2018 at 8:53 AM Juergen Ributzka via cfe-dev <
> cfe-dev at lists.llvm.org> wrote:
>
>> The code on opensource.apple.com is the minimal code needed for libtapi
>> to read TBD files with ld64. The code I pushed to GitHub on the other
side
>> includes the full TAPI tool source code. If you check the code you will
see
>> that I already use the LLVM license for everything, because the goal
has
>> always been to contribute this back to the community.
>>
>> I attached an initial patch for libtapi and nm support to my original
>> email. I also have some updated patches for that too, but somehow got
>> derailed into "lets rewrite libobject" discussions.
>>
>> On Tue, Apr 10, 2018 at 4:15 PM, John Ericson via llvm-dev <
>> llvm-dev at lists.llvm.org> wrote:
>>
>>> That sounds great to me, thanks Jake. I'm not Jurgen either, of
course,
>>> but I'm happy to assist you if he is unavailable. I'm not
also not
>>> qualified to audit the license, but do note Apple formally also
released
>>> some code at https://opensource.apple.com/tarballs/tapi/. If
there's
>>> anything else I can do to help, let me know.
>>>
>>> Cheers,
>>>
>>> John
>>>
>>> On 04/10/2018 06:13 PM, Jake Ehrlich wrote:
>>>
>>> Ideally Jurgen would cut up the code on github, put up an initial
diff
>>> for a minimal viable tool, and then we would review it and then
continue to
>>> copy code from the github repo into llvm and review it. I'm
also willing to
>>> do that if Jurgen doesn't want to at this point though. I'd
like the OK
>>> from Jurgen on that and I'd also like the OK from someone that
the license
>>> stuff is all good to go (I'm not sure who should check licence
stuff).
>>>
>>> Best,
>>> Jake
>>>
>>> On Tue, Apr 10, 2018 at 2:39 PM John Ericson
>>> <john.ericson at obsidian.systems> <john.ericson at
obsidian.systems> wrote:
>>>
>>>> Seems like there are a few of us interested in this then. I new
around
>>>> here and don't really know how decisions are made, so
what's next? Just
>>>> open a diff with the entire library??
>>>>
>>>> John
>>>> On 04/10/2018 05:33 PM, Jake Ehrlich wrote:
>>>>
>>>> Benifits of TBD:
>>>> 1) It's human readable and diffs on TBDs correspond to
changes in the
>>>> ABI. Diffs can be automatically added to review processes to
ensure that
>>>> changes to the ABI are reviewed. The TBDs also document your
precise ABI.
>>>> 2) The size is smaller which means they can be shipped in an
SDK
>>>> instead of binaries to reduce the size of an SDK
>>>> 3) Stubs are producible from TBDs (or should be) which means
stubs for
>>>> linking can be produced even if we don't directly support
them in LLD. This
>>>> lets you ship the smaller TBD files in place of larger binaries
and still
>>>> link things without direct linker support (assuming you already
ship a
>>>> toolchain with your SDK or expect your users to have this tool)
>>>>
>>>> Since stubs are producible from TBDs I don't really see a
downside. I
>>>> think we need both, I was going to propose a yaml based
representation for
>>>> ELF for the above reasons anyhow.
>>>>
>>>> On Tue, Apr 10, 2018 at 1:14 PM Andrew Kelley via llvm-dev <
>>>> llvm-dev at lists.llvm.org> wrote:
>>>>
>>>>> On Mon, Apr 9, 2018 at 10:11 PM, John Ericson via llvm-dev
<
>>>>> llvm-dev at lists.llvm.org> wrote:
>>>>>
>>>>>> > Regardless of any of that, given that TBD files
_are_ an integral
>>>>>> part of the apple platform, supporting them is
certainly a necessity in
>>>>>> order to have a working apple linker. So, if making LLD
work for
>>>>>> Apple/MachO is the justification for adding TBD support
to LLVM, that seems
>>>>>> self-evidently a reasonable thing to do. On the other
hand, it looks like
>>>>>> the LLD mach-o code is unmaintained and nobody seems to
be much interested
>>>>>> in it. And having code for reading TBD files in LLVM
seems not terribly
>>>>>> interesting, unless it is as part of a project to make
the LLD MachO linker
>>>>>> actually functional and supported.
>>>>>>
>>>>>> Yes. I hope this can be reason enough. Hobbyists could
push for LLD
>>>>>> support for Mach-O besides Apple, and if LLD is to
displace other linkers
>>>>>> this is a necessary component as you say. Better to
upstream now before the
>>>>>> code diverges than more work later? Conversely if
nothing happens, I doubt
>>>>>> libtapi would be a greater drag on the codebase than
the MachO LLD code, so
>>>>>> whatever cost/benefit analysis exists for keeping that
around could also
>>>>>> apply to this.
>>>>>
>>>>>
>>>>>
>>>>> Speaking for the Zig project here, our goal is to support
>>>>> cross-compilation for any target, on any target, without
requiring
>>>>> installation of any target-specific SDK. So, for example,
these use cases:
>>>>>  * on linux, compile & link a binary targeting macos
>>>>>  * on windows, compile & link a binary targeting macos
>>>>>
>>>>> This works today, although it depends on a patch to LLD to
fix the
>>>>> MACH-O linker that is not high enough quality to upstream.
>>>>>
>>>>> So we have a vested interest in improving the MACH-O
linker, and in
>>>>> fact a Zig community member has fixed at least one bug in
MACH-O LLD:
>>>>> reviews.llvm.org/D35387
>>>>>
>>>>> I don't fully understand how TBD or TAPI works, but I
hope that it
>>>>> results in improvements to the MACH-O linker.
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> LLVM Developers mailing list
>>>>> llvm-dev at lists.llvm.org
>>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>>
>>>>
>>>>
>>>
>>> _______________________________________________
>>> LLVM Developers mailing list
>>> llvm-dev at lists.llvm.org
>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>
>>>
>> _______________________________________________
>> cfe-dev mailing list
>> cfe-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20180920/1df470d7/attachment.html>

Great to hear, thanks!

On Thu, Sep 20, 2018 at 1:56 PM Jake Ehrlich <jakehehrlich at google.com>
wrote:
> A member of my team +Armando Montanez <amontanez at google.com> is
going to
> drop a proposal for the ELF part of this soon (like sometime next week) and
> will be working on the implementation. I'll be one of the reviewers for
> anything that comes out of that so we can be sure it will get reviewed as
> well. The goal is to have a single tool/library that would include
> Juergen's original code and the new ELF code.
>
> On Thu, Sep 20, 2018 at 10:45 AM Nico Weber <thakis at chromium.org>
wrote:
>
>> Was there any progress in the upstreaming effort? I'd be interested
in
>> having lld be able to link against tbd files, and I think it'd be
cool if
>> libtool -static could write tbd files (similar to thin archives on
linux)
>> since that should make archiving much faster.
>>
>> Juergen, maybe uploading your initial patch to phabricator instead of
>> attaching might get more traction?
>>
>> On Wed, Apr 25, 2018 at 8:53 AM Juergen Ributzka via cfe-dev <
>> cfe-dev at lists.llvm.org> wrote:
>>
>>> The code on opensource.apple.com is the minimal code needed for
libtapi
>>> to read TBD files with ld64. The code I pushed to GitHub on the
other side
>>> includes the full TAPI tool source code. If you check the code you
will see
>>> that I already use the LLVM license for everything, because the
goal has
>>> always been to contribute this back to the community.
>>>
>>> I attached an initial patch for libtapi and nm support to my
original
>>> email. I also have some updated patches for that too, but somehow
got
>>> derailed into "lets rewrite libobject" discussions.
>>>
>>> On Tue, Apr 10, 2018 at 4:15 PM, John Ericson via llvm-dev <
>>> llvm-dev at lists.llvm.org> wrote:
>>>
>>>> That sounds great to me, thanks Jake. I'm not Jurgen
either, of course,
>>>> but I'm happy to assist you if he is unavailable. I'm
not also not
>>>> qualified to audit the license, but do note Apple formally also
released
>>>> some code at https://opensource.apple.com/tarballs/tapi/. If
there's
>>>> anything else I can do to help, let me know.
>>>>
>>>> Cheers,
>>>>
>>>> John
>>>>
>>>> On 04/10/2018 06:13 PM, Jake Ehrlich wrote:
>>>>
>>>> Ideally Jurgen would cut up the code on github, put up an
initial diff
>>>> for a minimal viable tool, and then we would review it and then
continue to
>>>> copy code from the github repo into llvm and review it. I'm
also willing to
>>>> do that if Jurgen doesn't want to at this point though.
I'd like the OK
>>>> from Jurgen on that and I'd also like the OK from someone
that the license
>>>> stuff is all good to go (I'm not sure who should check
licence stuff).
>>>>
>>>> Best,
>>>> Jake
>>>>
>>>> On Tue, Apr 10, 2018 at 2:39 PM John Ericson
>>>> <john.ericson at obsidian.systems> <john.ericson at
obsidian.systems> wrote:
>>>>
>>>>> Seems like there are a few of us interested in this then. I
new around
>>>>> here and don't really know how decisions are made, so
what's next? Just
>>>>> open a diff with the entire library??
>>>>>
>>>>> John
>>>>> On 04/10/2018 05:33 PM, Jake Ehrlich wrote:
>>>>>
>>>>> Benifits of TBD:
>>>>> 1) It's human readable and diffs on TBDs correspond to
changes in the
>>>>> ABI. Diffs can be automatically added to review processes
to ensure that
>>>>> changes to the ABI are reviewed. The TBDs also document
your precise ABI.
>>>>> 2) The size is smaller which means they can be shipped in
an SDK
>>>>> instead of binaries to reduce the size of an SDK
>>>>> 3) Stubs are producible from TBDs (or should be) which
means stubs for
>>>>> linking can be produced even if we don't directly
support them in LLD. This
>>>>> lets you ship the smaller TBD files in place of larger
binaries and still
>>>>> link things without direct linker support (assuming you
already ship a
>>>>> toolchain with your SDK or expect your users to have this
tool)
>>>>>
>>>>> Since stubs are producible from TBDs I don't really see
a downside. I
>>>>> think we need both, I was going to propose a yaml based
representation for
>>>>> ELF for the above reasons anyhow.
>>>>>
>>>>> On Tue, Apr 10, 2018 at 1:14 PM Andrew Kelley via llvm-dev
<
>>>>> llvm-dev at lists.llvm.org> wrote:
>>>>>
>>>>>> On Mon, Apr 9, 2018 at 10:11 PM, John Ericson via
llvm-dev <
>>>>>> llvm-dev at lists.llvm.org> wrote:
>>>>>>
>>>>>>> > Regardless of any of that, given that TBD
files _are_ an integral
>>>>>>> part of the apple platform, supporting them is
certainly a necessity in
>>>>>>> order to have a working apple linker. So, if making
LLD work for
>>>>>>> Apple/MachO is the justification for adding TBD
support to LLVM, that seems
>>>>>>> self-evidently a reasonable thing to do. On the
other hand, it looks like
>>>>>>> the LLD mach-o code is unmaintained and nobody
seems to be much interested
>>>>>>> in it. And having code for reading TBD files in
LLVM seems not terribly
>>>>>>> interesting, unless it is as part of a project to
make the LLD MachO linker
>>>>>>> actually functional and supported.
>>>>>>>
>>>>>>> Yes. I hope this can be reason enough. Hobbyists
could push for LLD
>>>>>>> support for Mach-O besides Apple, and if LLD is to
displace other linkers
>>>>>>> this is a necessary component as you say. Better to
upstream now before the
>>>>>>> code diverges than more work later? Conversely if
nothing happens, I doubt
>>>>>>> libtapi would be a greater drag on the codebase
than the MachO LLD code, so
>>>>>>> whatever cost/benefit analysis exists for keeping
that around could also
>>>>>>> apply to this.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Speaking for the Zig project here, our goal is to
support
>>>>>> cross-compilation for any target, on any target,
without requiring
>>>>>> installation of any target-specific SDK. So, for
example, these use cases:
>>>>>>  * on linux, compile & link a binary targeting
macos
>>>>>>  * on windows, compile & link a binary targeting
macos
>>>>>>
>>>>>> This works today, although it depends on a patch to LLD
to fix the
>>>>>> MACH-O linker that is not high enough quality to
upstream.
>>>>>>
>>>>>> So we have a vested interest in improving the MACH-O
linker, and in
>>>>>> fact a Zig community member has fixed at least one bug
in MACH-O LLD:
>>>>>> reviews.llvm.org/D35387
>>>>>>
>>>>>> I don't fully understand how TBD or TAPI works, but
I hope that it
>>>>>> results in improvements to the MACH-O linker.
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> LLVM Developers mailing list
>>>>>> llvm-dev at lists.llvm.org
>>>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>>>
>>>>>
>>>>>
>>>>
>>>> _______________________________________________
>>>> LLVM Developers mailing list
>>>> llvm-dev at lists.llvm.org
>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>
>>>>
>>> _______________________________________________
>>> cfe-dev mailing list
>>> cfe-dev at lists.llvm.org
>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20180920/03f02f9d/attachment.html>