Hi all, We’re planning to contribute a new implementation of LLD for Mach-O, using the same design as the COFF and ELF ports. This design has proven to work very well for those ports, and we’re keen to explore it for Mach-O as well. Our work is based on an initial prototype created by Peter Collingbourne and Rui Ueyama. Our initial commit is up for review at https://reviews.llvm.org/D75382. We’ve intentionally stripped down this initial commit as much as possible to ease reviewing; we’ve kept it to the absolute minimum needed to produce and test a working macOS x86-64 executable for that prints “Hello World” via a syscall. We have several short-term follow-ups planned to add important functionality, such as linking against archives, universal binaries, dylibs, and tbd files, performing subsection splitting (atomization), and producing dylibs. The follow-ups should give a good sense of the overall design while still keeping each piece easily reviewable and testable individually. Our end goal is to create a full-featured Mach-O linker, and we’ll be working toward that goal over the next several months (and years, in all likelihood). We’d appreciate feedback and reviews. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20200228/67914114/attachment.html>
Nice! Your plan sounds great, and it'll be awesome to finally have a good MachO LLD available. On Fri, Feb 28, 2020 at 4:32 PM Shoaib Meenai via llvm-dev < llvm-dev at lists.llvm.org> wrote:> Hi all, > > We’re planning to contribute a new implementation of LLD for Mach-O, using > the same design as the COFF and ELF ports. This design has proven to work > very well for those ports, and we’re keen to explore it for Mach-O as well. > Our work is based on an initial prototype created by Peter Collingbourne > and Rui Ueyama. > > Our initial commit is up for review at https://reviews.llvm.org/D75382. > We’ve intentionally stripped down this initial commit as much as possible > to ease reviewing; we’ve kept it to the absolute minimum needed to produce > and test a working macOS x86-64 executable for that prints “Hello World” > via a syscall. We have several short-term follow-ups planned to add > important functionality, such as linking against archives, universal > binaries, dylibs, and tbd files, performing subsection splitting > (atomization), and producing dylibs. The follow-ups should give a good > sense of the overall design while still keeping each piece easily > reviewable and testable individually. Our end goal is to create a > full-featured Mach-O linker, and we’ll be working toward that goal over the > next several months (and years, in all likelihood). We’d appreciate > feedback and reviews. > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20200228/13d764ac/attachment.html>
On 2020-02-28, James Y Knight via llvm-dev wrote:>Nice! > >Your plan sounds great, and it'll be awesome to finally have a good MachO >LLD available. > >On Fri, Feb 28, 2020 at 4:32 PM Shoaib Meenai via llvm-dev < >llvm-dev at lists.llvm.org> wrote: > >> Hi all, >> >> We’re planning to contribute a new implementation of LLD for Mach-O, using >> the same design as the COFF and ELF ports. This design has proven to work >> very well for those ports, and we’re keen to explore it for Mach-O as well. >> Our work is based on an initial prototype created by Peter Collingbourne >> and Rui Ueyama. >> >> Our initial commit is up for review at https://reviews.llvm.org/D75382. >> We’ve intentionally stripped down this initial commit as much as possible >> to ease reviewing; we’ve kept it to the absolute minimum needed to produce >> and test a working macOS x86-64 executable for that prints “Hello World” >> via a syscall. We have several short-term follow-ups planned to add >> important functionality, such as linking against archives, universal >> binaries, dylibs, and tbd files, performing subsection splitting >> (atomization), and producing dylibs. The follow-ups should give a good >> sense of the overall design while still keeping each piece easily >> reviewable and testable individually. Our end goal is to create a >> full-featured Mach-O linker, and we’ll be working toward that goal over the >> next several months (and years, in all likelihood). We’d appreciate >> feedback and reviews.The existing Mach-O port https://reviews.llvm.org/D38290#882910 had been unmaintained when the ld64.lld alias was added. If Jez and the team are committed to maintain the new Mach-O port and we think the existing port is a dead end, we may assign the flavor `darwin` to it (`lld -flavor darwin`) and rename the existing flavor to `darwin-old` or `darwin-legacy`.
Awesome :) out of curiosity, are there any particular features/characteristics/benefits you're hoping for compared to the MacOS system linker? On Fri, Feb 28, 2020 at 1:32 PM Shoaib Meenai via llvm-dev < llvm-dev at lists.llvm.org> wrote:> Hi all, > > We’re planning to contribute a new implementation of LLD for Mach-O, using > the same design as the COFF and ELF ports. This design has proven to work > very well for those ports, and we’re keen to explore it for Mach-O as well. > Our work is based on an initial prototype created by Peter Collingbourne > and Rui Ueyama. > > Our initial commit is up for review at https://reviews.llvm.org/D75382. > We’ve intentionally stripped down this initial commit as much as possible > to ease reviewing; we’ve kept it to the absolute minimum needed to produce > and test a working macOS x86-64 executable for that prints “Hello World” > via a syscall. We have several short-term follow-ups planned to add > important functionality, such as linking against archives, universal > binaries, dylibs, and tbd files, performing subsection splitting > (atomization), and producing dylibs. The follow-ups should give a good > sense of the overall design while still keeping each piece easily > reviewable and testable individually. Our end goal is to create a > full-featured Mach-O linker, and we’ll be working toward that goal over the > next several months (and years, in all likelihood). We’d appreciate > feedback and reviews. > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20200229/f9009f40/attachment.html>
We’re hoping to get some linking speedups, similar to how LLD for ELF and COFF ended up being faster than the existing system linkers, though of course it’s hard to say how ld64’s design and efficiency compares to those other system linkers. In particular, we’re excited about the potential of speedups from parallelization and using the LLVM data structures instead of the STL ones. From: David Blaikie <dblaikie at gmail.com> Date: Saturday, February 29, 2020 at 2:15 PM To: Shoaib Meenai <smeenai at fb.com> Cc: "llvm-dev at lists.llvm.org" <llvm-dev at lists.llvm.org>, Jez Ng <jezng at fb.com> Subject: Re: [llvm-dev] Contributing LLD for Mach-O Awesome :) out of curiosity, are there any particular features/characteristics/benefits you're hoping for compared to the MacOS system linker? On Fri, Feb 28, 2020 at 1:32 PM Shoaib Meenai via llvm-dev <llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org>> wrote: Hi all, We’re planning to contribute a new implementation of LLD for Mach-O, using the same design as the COFF and ELF ports. This design has proven to work very well for those ports, and we’re keen to explore it for Mach-O as well. Our work is based on an initial prototype created by Peter Collingbourne and Rui Ueyama. Our initial commit is up for review at https://reviews.llvm.org/D75382<https://urldefense.proofpoint.com/v2/url?u=https-3A__reviews.llvm.org_D75382&d=DwMFaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=o3kDXzdBUE3ljQXKeTWOMw&m=FsUJ3zm22ZnHwLpD3M0eika9YJ5VYGAKgWXVbeDI5gs&s=upNikuhsap52-7XAVMILW83qrO15V8-QgRnwZvBc5Tg&e=>. We’ve intentionally stripped down this initial commit as much as possible to ease reviewing; we’ve kept it to the absolute minimum needed to produce and test a working macOS x86-64 executable for that prints “Hello World” via a syscall. We have several short-term follow-ups planned to add important functionality, such as linking against archives, universal binaries, dylibs, and tbd files, performing subsection splitting (atomization), and producing dylibs. The follow-ups should give a good sense of the overall design while still keeping each piece easily reviewable and testable individually. Our end goal is to create a full-featured Mach-O linker, and we’ll be working toward that goal over the next several months (and years, in all likelihood). We’d appreciate feedback and reviews. _______________________________________________ LLVM Developers mailing list llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev<https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.llvm.org_cgi-2Dbin_mailman_listinfo_llvm-2Ddev&d=DwMFaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=o3kDXzdBUE3ljQXKeTWOMw&m=FsUJ3zm22ZnHwLpD3M0eika9YJ5VYGAKgWXVbeDI5gs&s=DwpI7nEy97J4KI4Me0IdFo60JS3T68M63dMfRUnmF9s&e=> -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20200302/4a6ce0a0/attachment.html>
Hi Shoaib, Has there been any recent discussion with anyone on Apple's side? Is there a way forward that results in a single unified open source linker? If not at the start, how would it work if they later want to take maintainership? Have you thought about a compatibility test suite? If so, I'm curious what the approach will be. Also for what it's worth, there's a recent fork of ld64's recent source drop, which is optimized for incremental compilation. It also uses non-STL data structures, parallelism, and a disk cache. See https://github.com/michaeleisel/zld On Fri, Feb 28, 2020 at 1:32 PM Shoaib Meenai via llvm-dev < llvm-dev at lists.llvm.org> wrote:> Hi all, > > We’re planning to contribute a new implementation of LLD for Mach-O, using > the same design as the COFF and ELF ports. This design has proven to work > very well for those ports, and we’re keen to explore it for Mach-O as well. > Our work is based on an initial prototype created by Peter Collingbourne > and Rui Ueyama. > > Our initial commit is up for review at https://reviews.llvm.org/D75382. > We’ve intentionally stripped down this initial commit as much as possible > to ease reviewing; we’ve kept it to the absolute minimum needed to produce > and test a working macOS x86-64 executable for that prints “Hello World” > via a syscall. We have several short-term follow-ups planned to add > important functionality, such as linking against archives, universal > binaries, dylibs, and tbd files, performing subsection splitting > (atomization), and producing dylibs. The follow-ups should give a good > sense of the overall design while still keeping each piece easily > reviewable and testable individually. Our end goal is to create a > full-featured Mach-O linker, and we’ll be working toward that goal over the > next several months (and years, in all likelihood). We’d appreciate > feedback and reviews. > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20200303/73f1a02a/attachment.html>
On 2020-03-03, Dave Lee via llvm-dev wrote:>Hi Shoaib, > >Has there been any recent discussion with anyone on Apple's side? Is there >a way forward that results in a single unified open source linker? If not >at the start, how would it work if they later want to take maintainership? > >Have you thought about a compatibility test suite? If so, I'm curious what >the approach will be. > >Also for what it's worth, there's a recent fork of ld64's recent source >drop, which is optimized for incremental compilation. It also uses non-STL >data structures, parallelism, and a disk cache. See >https://github.com/michaeleisel/zld% cat llvm-project/lld/CODE_OWNERS.TXT ... N: Lang Hames, Nick Kledzik E: lhames at gmail.com, kledzik at apple.com D: Mach-O backend ... Lang resigned (https://reviews.llvm.org/D75382#1906108) and agreed to delete the existing Mach-O port, but I think we should also get Nick's confirmation.> >On Fri, Feb 28, 2020 at 1:32 PM Shoaib Meenai via llvm-dev < >llvm-dev at lists.llvm.org> wrote: > >> Hi all, >> >> We’re planning to contribute a new implementation of LLD for Mach-O, using >> the same design as the COFF and ELF ports. This design has proven to work >> very well for those ports, and we’re keen to explore it for Mach-O as well. >> Our work is based on an initial prototype created by Peter Collingbourne >> and Rui Ueyama. >> >> Our initial commit is up for review at https://reviews.llvm.org/D75382. >> We’ve intentionally stripped down this initial commit as much as possible >> to ease reviewing; we’ve kept it to the absolute minimum needed to produce >> and test a working macOS x86-64 executable for that prints “Hello World” >> via a syscall. We have several short-term follow-ups planned to add >> important functionality, such as linking against archives, universal >> binaries, dylibs, and tbd files, performing subsection splitting >> (atomization), and producing dylibs. The follow-ups should give a good >> sense of the overall design while still keeping each piece easily >> reviewable and testable individually. Our end goal is to create a >> full-featured Mach-O linker, and we’ll be working toward that goal over the >> next several months (and years, in all likelihood). We’d appreciate >> feedback and reviews. >> _______________________________________________ >> LLVM Developers mailing list >> llvm-dev at lists.llvm.org >> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >>>_______________________________________________ >LLVM Developers mailing list >llvm-dev at lists.llvm.org >https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
We’ve discussed this with Jim Grosbach at Apple (CC’d). My understanding is that at this time, the officially supported linker for Apple’s platforms is ld64. We’d love to collaborate with Apple (and any other interested parties, for that matter) on LLD for Mach-O, and we’d be delighted if it were to become officially supported at some point, but there’s a lot of work to be done first on reaching feature parity with ld64 before that could even be considered :) Once we reach feature parity, I can envision several good reasons both for sticking with ld64 and for switching to LLD. On our end, we aim to create a feature-complete Mach-O linker. We also aim for the end product to be compelling enough that a switch could be considered, and we’d be happy to work with Apple on that front if it turns out to be. What sort of compatibility test suite did you have in mind? We’re adding lit-style unit tests as we add features (as is standard for LLVM), but we didn’t have anything in mind beyond that right now. We saw zld, and the speedups achieved by it make us hopeful of being able to achieve similar results with LLD (since we’ll also have better parallelization and better data structures in the form of the LLVM ones). (I’d also be curious if zld’s improvements could be contributed back to ld64 so that everyone can take advantage of them, but that’s completely tangential, of course.) From: Dave Lee <davelee.com at gmail.com> Date: Tuesday, March 3, 2020 at 7:37 AM To: Shoaib Meenai <smeenai at fb.com> Cc: "llvm-dev at lists.llvm.org" <llvm-dev at lists.llvm.org>, Jez Ng <jezng at fb.com> Subject: Re: [llvm-dev] Contributing LLD for Mach-O Hi Shoaib, Has there been any recent discussion with anyone on Apple's side? Is there a way forward that results in a single unified open source linker? If not at the start, how would it work if they later want to take maintainership? Have you thought about a compatibility test suite? If so, I'm curious what the approach will be. Also for what it's worth, there's a recent fork of ld64's recent source drop, which is optimized for incremental compilation. It also uses non-STL data structures, parallelism, and a disk cache. See https://github.com/michaeleisel/zld On Fri, Feb 28, 2020 at 1:32 PM Shoaib Meenai via llvm-dev <llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org>> wrote: Hi all, We’re planning to contribute a new implementation of LLD for Mach-O, using the same design as the COFF and ELF ports. This design has proven to work very well for those ports, and we’re keen to explore it for Mach-O as well. Our work is based on an initial prototype created by Peter Collingbourne and Rui Ueyama. Our initial commit is up for review at https://reviews.llvm.org/D75382<https://urldefense.proofpoint.com/v2/url?u=https-3A__reviews.llvm.org_D75382&d=DwMFaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=o3kDXzdBUE3ljQXKeTWOMw&m=Du_g8hPBNR5iofm4w6TR-gFx_aFK-4RNxpEmlx-FMNc&s=z1H0UW2F3uSaikSViTE-eRlQmyNhhrkHNq0aj2VxbfE&e=>. We’ve intentionally stripped down this initial commit as much as possible to ease reviewing; we’ve kept it to the absolute minimum needed to produce and test a working macOS x86-64 executable for that prints “Hello World” via a syscall. We have several short-term follow-ups planned to add important functionality, such as linking against archives, universal binaries, dylibs, and tbd files, performing subsection splitting (atomization), and producing dylibs. The follow-ups should give a good sense of the overall design while still keeping each piece easily reviewable and testable individually. Our end goal is to create a full-featured Mach-O linker, and we’ll be working toward that goal over the next several months (and years, in all likelihood). We’d appreciate feedback and reviews. _______________________________________________ LLVM Developers mailing list llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev<https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.llvm.org_cgi-2Dbin_mailman_listinfo_llvm-2Ddev&d=DwMFaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=o3kDXzdBUE3ljQXKeTWOMw&m=Du_g8hPBNR5iofm4w6TR-gFx_aFK-4RNxpEmlx-FMNc&s=WEM8UzKRSd0-QQZOXRcgL_Rkt5cgC1soudjdbkMnbgA&e=> -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20200305/954ec44e/attachment.html>