llvm dev - Jun 2021 - RFC: Revisiting LLD-as-a-library design

On Fri, Jun 11, 2021 at 3:56 AM Peter Smith via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> One of the earliest discussions about the LLD as a library design, at
> least after it had matured enough to be practical was this rather long
> thread https://lists.llvm.org/pipermail/llvm-dev/2016-December/107981.html
>
>
>
> I don’t have any objections about making LLD more useable as a library.
>
>
>
> What I would say is that we should come up with a good idea of what the
> functionality needed from the library is. For example I can see one use
> case with a relatively coarse interface that is similar to running LLD from
> the command line, with object files passed in memory buffers. I can see
> that working as an extension of the existing design. A more ambitious use
> case would permit more fine grain control of the pipeline, or construct LLD
> data structures directly rather than using object files could require quite
> a bit more work. I think people are thinking along the lines of the former,
> but it is worth making sure.
>
Not sure it's super important to decide much of that up-front. Once the
general philosophy is agreed upon and the most fundamental issues are
addressed for even the narrowest/simplest library use - other things can be
done as enhancements beyond that.

A stretch/eventual goal could be to share some code between lld and LLVM's
ORC JITLink - so that JIT and AOT linking share more code/diverge less.

>
>
> I think one of the reasons the library use case faltered was that there
> was no-one with a use case that was able to spend enough time to make it
> happen. The existing maintainers had enough work to do to catch up with
> Gold and BFD. Do we have someone willing to do the work?
>
>
>
> Peter
>
>
>
> *From:* llvm-dev <llvm-dev-bounces at lists.llvm.org> *On Behalf Of
*James
> Henderson via llvm-dev
> *Sent:* 11 June 2021 08:42
> *To:* Reid Kleckner <rnk at google.com>
> *Cc:* llvm-dev <llvm-dev at lists.llvm.org>; jezng at fb.com
> *Subject:* Re: [llvm-dev] RFC: Revisiting LLD-as-a-library design
>
>
>
> No objections here (although I don't have a specific use-case
currently).
>
>
>
> Regarding the error handling, I support some sort of callback approach to
> report the errors (https://www.youtube.com/watch?v=YSEY4pg1YB0). This
> doesn't solve the problem of what to do after a fatal error has been
> reported. In the debug line parsing code which inspired that talk, we had a
> concept of unrecoverable and recoverable errors, whereby the parser would
> either stop parsing if it found something it couldn't recover from, by
> bailing out of the function, or it would set some assumed values and
> continue parsing. This may work for some cases in LLD, but the fatal cases
> need to stop the linking completely, so we'll need some way to bail out
of
> the LLD call stack in those cases still somehow - personally, I think we
> should use llvm::Error for that up to the point of public interface with
> the library, to avoid the failure being unchecked. The error callbacks
> could then return Error to allow a client to force LLD to stop, even if the
> error would otherwise be non-fatal.
>
>
>
> James
>
>
>
> On Thu, 10 Jun 2021 at 19:15, Reid Kleckner via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
> Hey all,
>
>
>
> Long ago, the LLD project contributors decided that they weren't going
to
> design LLD as a library, which stands in opposition to the way that the
> rest of LLVM strives to be a reusable library. Part of the reasoning was
> that, at the time, LLD wasn't done yet, and the top priority was to
finish
> making LLD a fast, useful, usable product. If sacrificing reusability
> helped LLD achieve its project goals, the contributors at the time felt
> that was the right tradeoff, and that carried the day.
>
>
>
> However, it is now ${YEAR} 2021, and I think we ought to reconsider this
> design decision. LLD was a great success: it works, it is fast, it is
> simple, many users have adopted it, it has many ports
> (COFF/ELF/mingw/wasm/new MachO). Today, we have actual users who want to
> run the linker as a library, and they aren't satisfied with the option
of
> launching a child process. Some users are interested in process reuse as a
> performance optimization, some are including the linker in the frontend.
> Who knows. I try not to pre-judge any of these efforts, I think we should
> do what we can to enable experimentation.
>
>
>
> So, concretely, what could change? The main points of reusability are:
>
> - Fatal errors and warnings exit the process without returning control to
> the caller
>
> - Conflicts over global variables between threads
>
>
>
> Error recovery is the big imposition here. To avoid a giant rewrite of all
> error handling code in LLD, I think we should *avoid* returning failure via
> the llvm::Error class or std::error_code. We should instead use an approach
> more like clang, where diagnostics are delivered to a diagnostic consumer
> on the side. The success of the link is determined by whether any errors
> were reported. Functions may return a simple success boolean in cases where
> higher level functions need to exit early. This has worked reasonably well
> for clang. The main failure mode here is that we miss an error check, and
> crash or report useless follow-on errors after an error that would normally
> have been fatal.
>
>
>
> Another motivation for all of this is increasing the use of parallelism in
> LLD. Emitting errors in parallel from threads and then exiting the process
> is risky business. A new diagnostic context or consumer could make this
> more reliable. MLIR has this issue as well, and I believe they use this
> pattern. They use some kind of thread shard index to order the diagnostics,
> LLD could do the same.
>
>
>
> Finally, we'd work to eliminate globals. I think this is mainly a small
> matter of programming (SMOP) and doesn't need much discussion, although
the
> `make` template presents interesting challenges.
>
>
>
> Thoughts? Tomatoes? Flowers? I apologize for the lack of context links to
> the original discussions. It takes more time than I have to dig those up.
>
> Reid
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20210611/c76cf38f/attachment.html>

As one of the people working on the new Mach-O backend, my main concerns are:

  1.  The Mach-O backend is still very much in flux, and will likely remain so
until the end of this year. Whoever undertakes this library-fication should sync
up with us to avoid e.g. awkward merge conflicts.
  2.  Performance is very important to us, and this library-fication should not
regress it. Right now, we don't have a good benchmarking service set up (the
LNT server only benchmarks LLD-ELF); we've been mostly just profiling things
locally. I can send these benchmarking instructions to whomever takes on this
effort.

Jez
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20210611/788f0053/attachment.html>