llvm dev - Jun 2021 - RFC: Revisiting LLD-as-a-library design

A big +1 from our side since we have a potential use case for
LLD-as-a-library (I was going to write a similar RFC but you beat me to it).

On Thu, Jun 10, 2021 at 11:15 AM Reid Kleckner via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> Hey all,
>
> Long ago, the LLD project contributors decided that they weren't going
to
> design LLD as a library, which stands in opposition to the way that the
> rest of LLVM strives to be a reusable library. Part of the reasoning was
> that, at the time, LLD wasn't done yet, and the top priority was to
finish
> making LLD a fast, useful, usable product. If sacrificing reusability
> helped LLD achieve its project goals, the contributors at the time felt
> that was the right tradeoff, and that carried the day.
>
> However, it is now ${YEAR} 2021, and I think we ought to reconsider this
> design decision. LLD was a great success: it works, it is fast, it is
> simple, many users have adopted it, it has many ports
> (COFF/ELF/mingw/wasm/new MachO). Today, we have actual users who want to
> run the linker as a library, and they aren't satisfied with the option
of
> launching a child process. Some users are interested in process reuse as a
> performance optimization, some are including the linker in the frontend.
> Who knows. I try not to pre-judge any of these efforts, I think we should
> do what we can to enable experimentation.
>
> So, concretely, what could change? The main points of reusability are:
> - Fatal errors and warnings exit the process without returning control to
> the caller
> - Conflicts over global variables between threads
>
> Error recovery is the big imposition here. To avoid a giant rewrite of all
> error handling code in LLD, I think we should *avoid* returning failure via
> the llvm::Error class or std::error_code. We should instead use an approach
> more like clang, where diagnostics are delivered to a diagnostic consumer
> on the side. The success of the link is determined by whether any errors
> were reported. Functions may return a simple success boolean in cases where
> higher level functions need to exit early. This has worked reasonably well
> for clang. The main failure mode here is that we miss an error check, and
> crash or report useless follow-on errors after an error that would normally
> have been fatal.
>
> Another motivation for all of this is increasing the use of parallelism in
> LLD. Emitting errors in parallel from threads and then exiting the process
> is risky business. A new diagnostic context or consumer could make this
> more reliable. MLIR has this issue as well, and I believe they use this
> pattern. They use some kind of thread shard index to order the diagnostics,
> LLD could do the same.
>
> Finally, we'd work to eliminate globals. I think this is mainly a small
> matter of programming (SMOP) and doesn't need much discussion, although
the
> `make` template presents interesting challenges.
>
> Thoughts? Tomatoes? Flowers? I apologize for the lack of context links to
> the original discussions. It takes more time than I have to dig those up.
>
> Reid
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20210610/07c7191b/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3996 bytes
Desc: S/MIME Cryptographic Signature
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20210610/07c7191b/attachment-0001.bin>

+1 generally.

Petr - if you have particular context for your use case, might be handy to
have as reference/motivation.
Reid - if you have any particular use case of your own in mind, or links to
other discussions/users who are having friction with the current state of
affairs, would be hand to have.

(though I don't generally want the thread to become about picking apart
those use cases - library-based design and flexibility is a fairly core
tenant of the LLVM project overall regardless of the validity of specific
use cases)

On Thu, Jun 10, 2021 at 11:27 AM Petr Hosek via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> A big +1 from our side since we have a potential use case for
> LLD-as-a-library (I was going to write a similar RFC but you beat me to
it).
>
> On Thu, Jun 10, 2021 at 11:15 AM Reid Kleckner via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>> Hey all,
>>
>> Long ago, the LLD project contributors decided that they weren't
going to
>> design LLD as a library, which stands in opposition to the way that the
>> rest of LLVM strives to be a reusable library. Part of the reasoning
was
>> that, at the time, LLD wasn't done yet, and the top priority was to
finish
>> making LLD a fast, useful, usable product. If sacrificing reusability
>> helped LLD achieve its project goals, the contributors at the time felt
>> that was the right tradeoff, and that carried the day.
>>
>> However, it is now ${YEAR} 2021, and I think we ought to reconsider
this
>> design decision. LLD was a great success: it works, it is fast, it is
>> simple, many users have adopted it, it has many ports
>> (COFF/ELF/mingw/wasm/new MachO). Today, we have actual users who want
to
>> run the linker as a library, and they aren't satisfied with the
option of
>> launching a child process. Some users are interested in process reuse
as a
>> performance optimization, some are including the linker in the
frontend.
>> Who knows. I try not to pre-judge any of these efforts, I think we
should
>> do what we can to enable experimentation.
>>
>> So, concretely, what could change? The main points of reusability are:
>> - Fatal errors and warnings exit the process without returning control
to
>> the caller
>> - Conflicts over global variables between threads
>>
>> Error recovery is the big imposition here. To avoid a giant rewrite of
>> all error handling code in LLD, I think we should *avoid* returning
failure
>> via the llvm::Error class or std::error_code. We should instead use an
>> approach more like clang, where diagnostics are delivered to a
diagnostic
>> consumer on the side. The success of the link is determined by whether
any
>> errors were reported. Functions may return a simple success boolean in
>> cases where higher level functions need to exit early. This has worked
>> reasonably well for clang. The main failure mode here is that we miss
an
>> error check, and crash or report useless follow-on errors after an
error
>> that would normally have been fatal.
>>
>> Another motivation for all of this is increasing the use of parallelism
>> in LLD. Emitting errors in parallel from threads and then exiting the
>> process is risky business. A new diagnostic context or consumer could
make
>> this more reliable. MLIR has this issue as well, and I believe they use
>> this pattern. They use some kind of thread shard index to order the
>> diagnostics, LLD could do the same.
>>
>> Finally, we'd work to eliminate globals. I think this is mainly a
small
>> matter of programming (SMOP) and doesn't need much discussion,
although the
>> `make` template presents interesting challenges.
>>
>> Thoughts? Tomatoes? Flowers? I apologize for the lack of context links
to
>> the original discussions. It takes more time than I have to dig those
up.
>>
>> Reid
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20210610/cdbb5404/attachment.html>