llvm dev - Jun 2021 - RFC: Revisiting LLD-as-a-library design

+1 on this proposal from Reid (thanks for bringing this to the list!)

I'll drop a brain dump of why we (Unity) would like this proposal: we're
running into the same problems that Andrew noted for Zig above. At present
we are using LLD to do an on-disk JIT effectively:

   - this gets us debugging support for 'free' (write a PDB to disk next
to
   the DLL, et voila you can debug)
   - it lets us cache binaries that don't change across source file changes
   to reduce overall build times

We've seen cases where we call LLD as-a-subprocess 2000 times with this
approach. While I noted on the Windows/COFF call that we are taking steps
to try and mitigate the number of calls to LLD, we'll still likely be in
the 100~ ish DLLs built and thus 100~ ish calls to LLD. The cost of
spawning all these LLD subprocesses is a good 15% of our build pipeline. As
an experiment I tried running LLD-as-a-library and serialized the accesses
to the linker, and it was only a little bit slower than having N threads
launching N instances of LLD. That gives me good hope that having an actual
thread-safe way to run LLD will substantially reduce our build times,
meaning happy users.

-Neil.

On Sat, Jun 12, 2021 at 7:41 PM David Blaikie via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> On Sat, Jun 12, 2021 at 10:54 AM Erik McClure <erik2003 at gmail.com>
wrote:
>
>> The point of using LLVM for compiling WASM is to take advantage of
>> ahead-of-time optimizations that could cause hitches in a JIT.
>>
>
> Curious what sort of hitches you're referring to - but probably far
enough
> off-topic from this thread. (certainly the goal of the ORC JIT is to be as
> robust as AOT compilation, providing the same semantics, etc)
>
>
>> For example, it integrates polly to try to recover vectorization
>> optimizations. The resulting DLL can then be cached and loaded
instantly on
>> every subsequent playthrough,
>>
>
> Fair enough - that's what I was curious about, or whether there were
some
> other circumstances/motivations for using LLD as a library (eg: perhaps
> some bugs/missing features of ORC that could be addressed, or lack of
> documentation/visibility/etc).
>
>
>> without any possibility of hitching. Microsoft Flight Simulator 2020
also
>> ships pre-compiled plugin DLLs on Xbox, which does not allow JITing
code,
>> but because these are compiled on developer machines the linker problem
>> doesn't really apply in that situation. If they wanted to JIT
webassembly,
>> there are plenty of JIT runtimes to do that.
>>
>> Regardless, I think it's kind of silly to say that instead of using
a
>> perfectly functional linker that LLVM has, someone should JIT the code.
>>
>
> I didn't mean to suggest that - but that it sounded pretty close to a
> JIT-like use case & was curious if there was some non-fundamental
blocker
> that lead to the use of LLD rather than ORC for what appeared like it might
> be a JIT-like use case.
>
>
>> LLVM is a compiler backend - it should support using its own linker the
>> same way people use LLVM, and if LLVM can be used as a library, then
LLD
>> should be usable as a library. Furthermore, there is no technical
reason
>> for LLD to not be a library. It's already almost all the way there,
the
>> maintainers simply don't bother testing to see when they forget to
clean up
>> one of the global caches.
>>
>
> I don't disagree that LLD, like the rest of LLVM, would benefit from
> having a library-centric design.
>
> - Dave
>
>
>> --
>> Sincerely, Erik McClure
>>
>>
>> On Sat, Jun 12, 2021 at 10:24 AM David Blaikie <dblaikie at
gmail.com>
>> wrote:
>>
>>> Is this a JIT use case? Perhaps ORC would be applicable there.
>>>
>>> Or is the intent to make on-disk linked shared libraries so they
can be
>>> cached over multiple executions/etc, perhaps?
>>>
>>> On Sat, Jun 12, 2021 at 10:09 AM Erik McClure via llvm-dev <
>>> llvm-dev at lists.llvm.org> wrote:
>>>
>>>> I use LLVM to compile WebAssembly to native code. The primary
use-case
>>>> for this is compiling WASM plugins for games - this is what
Microsoft
>>>> Flight Simulator 2020 uses it for. Using the system linker is
not an option
>>>> on Windows, which does not ship link.exe by default, making LLD
a mandatory
>>>> requirement if you are using LLVM in any kind of end-user
plugin scenario,
>>>> as the average user has not installed Visual Studio.
>>>>
>>>> This puts users of LLVM's library capabilities on windows
in an awkward
>>>> position, because in order to use LLVM as a library when
compiling a
>>>> plugin, one must use LLD, which cannot be used as a library. My
current
>>>> solution is to use LLD as a library anyway and maintain a fork
of LLVM with
>>>> the various global cleanup bugs patched (most of which have now
made it
>>>> into stable), along with a helper function that allows me to
use LLD to
>>>> read out the symbols of a given shared library (which is used
to perform
>>>> link-time validation of webassembly modules, because LLD makes
it difficult
>>>> to access any errors that happen).
>>>>
>>>> If LLD wanted to become an actual library, I think it would
need a
>>>> better method of reporting errors than simply an stdout and
stderr stream,
>>>> although I don't know what this would look like. It would
also be nice for
>>>> it to expose the different link stages like LLVM does so that
the
>>>> application has a bit more control over what's going on.
However, I don't
>>>> really have any concrete ideas about what LLD should look like
as a
>>>> library, only that I would like it to stop crashing when I
attempt to use
>>>> it as one.
>>>>
>>>> --
>>>> Sincerely, Erik McClure
>>>>
>>>>
>>>> On Fri, Jun 11, 2021 at 8:20 PM Michael Spencer <bigcheesegs
at gmail.com>
>>>> wrote:
>>>>
>>>>> Adding Erik (not subscribed) who has previously had issues
with LLD
>>>>> not being a library to provide some additional use cases.
>>>>>
>>>>> - Michael Spencer
>>>>>
>>>>>
>>>>> On Thu, Jun 10, 2021 at 12:15 PM Reid Kleckner via llvm-dev
<
>>>>> llvm-dev at lists.llvm.org> wrote:
>>>>>
>>>>>> Hey all,
>>>>>>
>>>>>> Long ago, the LLD project contributors decided that
they weren't
>>>>>> going to design LLD as a library, which stands in
opposition to the way
>>>>>> that the rest of LLVM strives to be a reusable library.
Part of the
>>>>>> reasoning was that, at the time, LLD wasn't done
yet, and the top priority
>>>>>> was to finish making LLD a fast, useful, usable
product. If sacrificing
>>>>>> reusability helped LLD achieve its project goals, the
contributors at the
>>>>>> time felt that was the right tradeoff, and that carried
the day.
>>>>>>
>>>>>> However, it is now ${YEAR} 2021, and I think we ought
to reconsider
>>>>>> this design decision. LLD was a great success: it
works, it is fast, it is
>>>>>> simple, many users have adopted it, it has many ports
>>>>>> (COFF/ELF/mingw/wasm/new MachO). Today, we have actual
users who want to
>>>>>> run the linker as a library, and they aren't
satisfied with the option of
>>>>>> launching a child process. Some users are interested in
process reuse as a
>>>>>> performance optimization, some are including the linker
in the frontend.
>>>>>> Who knows. I try not to pre-judge any of these efforts,
I think we should
>>>>>> do what we can to enable experimentation.
>>>>>>
>>>>>> So, concretely, what could change? The main points of
reusability are:
>>>>>> - Fatal errors and warnings exit the process without
returning
>>>>>> control to the caller
>>>>>> - Conflicts over global variables between threads
>>>>>>
>>>>>> Error recovery is the big imposition here. To avoid a
giant rewrite
>>>>>> of all error handling code in LLD, I think we should
*avoid* returning
>>>>>> failure via the llvm::Error class or std::error_code.
We should instead use
>>>>>> an approach more like clang, where diagnostics are
delivered to a
>>>>>> diagnostic consumer on the side. The success of the
link is determined by
>>>>>> whether any errors were reported. Functions may return
a simple success
>>>>>> boolean in cases where higher level functions need to
exit early. This has
>>>>>> worked reasonably well for clang. The main failure mode
here is that we
>>>>>> miss an error check, and crash or report useless
follow-on errors after an
>>>>>> error that would normally have been fatal.
>>>>>>
>>>>>> Another motivation for all of this is increasing the
use of
>>>>>> parallelism in LLD. Emitting errors in parallel from
threads and then
>>>>>> exiting the process is risky business. A new diagnostic
context or consumer
>>>>>> could make this more reliable. MLIR has this issue as
well, and I believe
>>>>>> they use this pattern. They use some kind of thread
shard index to order
>>>>>> the diagnostics, LLD could do the same.
>>>>>>
>>>>>> Finally, we'd work to eliminate globals. I think
this is mainly a
>>>>>> small matter of programming (SMOP) and doesn't need
much discussion,
>>>>>> although the `make` template presents interesting
challenges.
>>>>>>
>>>>>> Thoughts? Tomatoes? Flowers? I apologize for the lack
of context
>>>>>> links to the original discussions. It takes more time
than I have to dig
>>>>>> those up.
>>>>>>
>>>>>> Reid
>>>>>> _______________________________________________
>>>>>> LLVM Developers mailing list
>>>>>> llvm-dev at lists.llvm.org
>>>>>>
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>>>
>>>>> _______________________________________________
>>>> LLVM Developers mailing list
>>>> llvm-dev at lists.llvm.org
>>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>
>>> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>

-- 
Neil Henning
Senior Software Engineer Compiler
unity.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20210614/37a34b12/attachment.html>

Hello,
Im trying to gather complete data about each use case of this new libified
lld.

Can I ask more details on difference between workflow/scenario with 2000
spawns of lld process and 100ish spawns/runs in threads for linking 100ish
DLLs?
Is same number of binaries built in each scenario?

Where do 2000 spawns of lld come from? Are there 2000 binaries built or do
You run in modifycode-recompile-relink same binary in succession workflow?

Please bear with me. Im still learning. I would like to understand all
details of requirements for all subtasks of this task to try to fix all
issues mentioned.

I got one more concern with "running lld in threads" which is
requiring
thread safety between runs. Is some data shared between those threads or
could we in worst scenario duplicate it in each threads context?

Best regards,
Pawel Kunio

pon., 14.06.2021, 10:45 użytkownik Neil Henning via llvm-dev <
llvm-dev at lists.llvm.org> napisał:
> +1 on this proposal from Reid (thanks for bringing this to the list!)
>
> I'll drop a brain dump of why we (Unity) would like this proposal:
we're
> running into the same problems that Andrew noted for Zig above. At present
> we are using LLD to do an on-disk JIT effectively:
>
>    - this gets us debugging support for 'free' (write a PDB to disk
next
>    to the DLL, et voila you can debug)
>    - it lets us cache binaries that don't change across source file
>    changes to reduce overall build times
>
> We've seen cases where we call LLD as-a-subprocess 2000 times with this
> approach. While I noted on the Windows/COFF call that we are taking steps
> to try and mitigate the number of calls to LLD, we'll still likely be
in
> the 100~ ish DLLs built and thus 100~ ish calls to LLD. The cost of
> spawning all these LLD subprocesses is a good 15% of our build pipeline. As
> an experiment I tried running LLD-as-a-library and serialized the accesses
> to the linker, and it was only a little bit slower than having N threads
> launching N instances of LLD. That gives me good hope that having an actual
> thread-safe way to run LLD will substantially reduce our build times,
> meaning happy users.
>
> -Neil.
>
> On Sat, Jun 12, 2021 at 7:41 PM David Blaikie via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>> On Sat, Jun 12, 2021 at 10:54 AM Erik McClure <erik2003 at
gmail.com> wrote:
>>
>>> The point of using LLVM for compiling WASM is to take advantage of
>>> ahead-of-time optimizations that could cause hitches in a JIT.
>>>
>>
>> Curious what sort of hitches you're referring to - but probably far
>> enough off-topic from this thread. (certainly the goal of the ORC JIT
is to
>> be as robust as AOT compilation, providing the same semantics, etc)
>>
>>
>>> For example, it integrates polly to try to recover vectorization
>>> optimizations. The resulting DLL can then be cached and loaded
instantly on
>>> every subsequent playthrough,
>>>
>>
>> Fair enough - that's what I was curious about, or whether there
were some
>> other circumstances/motivations for using LLD as a library (eg: perhaps
>> some bugs/missing features of ORC that could be addressed, or lack of
>> documentation/visibility/etc).
>>
>>
>>> without any possibility of hitching. Microsoft Flight Simulator
2020
>>> also ships pre-compiled plugin DLLs on Xbox, which does not allow
JITing
>>> code, but because these are compiled on developer machines the
linker
>>> problem doesn't really apply in that situation. If they wanted
to JIT
>>> webassembly, there are plenty of JIT runtimes to do that.
>>>
>>> Regardless, I think it's kind of silly to say that instead of
using a
>>> perfectly functional linker that LLVM has, someone should JIT the
code.
>>>
>>
>> I didn't mean to suggest that - but that it sounded pretty close to
a
>> JIT-like use case & was curious if there was some non-fundamental
blocker
>> that lead to the use of LLD rather than ORC for what appeared like it
might
>> be a JIT-like use case.
>>
>>
>>> LLVM is a compiler backend - it should support using its own linker
the
>>> same way people use LLVM, and if LLVM can be used as a library,
then LLD
>>> should be usable as a library. Furthermore, there is no technical
reason
>>> for LLD to not be a library. It's already almost all the way
there, the
>>> maintainers simply don't bother testing to see when they forget
to clean up
>>> one of the global caches.
>>>
>>
>> I don't disagree that LLD, like the rest of LLVM, would benefit
from
>> having a library-centric design.
>>
>> - Dave
>>
>>
>>> --
>>> Sincerely, Erik McClure
>>>
>>>
>>> On Sat, Jun 12, 2021 at 10:24 AM David Blaikie <dblaikie at
gmail.com>
>>> wrote:
>>>
>>>> Is this a JIT use case? Perhaps ORC would be applicable there.
>>>>
>>>> Or is the intent to make on-disk linked shared libraries so
they can be
>>>> cached over multiple executions/etc, perhaps?
>>>>
>>>> On Sat, Jun 12, 2021 at 10:09 AM Erik McClure via llvm-dev <
>>>> llvm-dev at lists.llvm.org> wrote:
>>>>
>>>>> I use LLVM to compile WebAssembly to native code. The
primary use-case
>>>>> for this is compiling WASM plugins for games - this is what
Microsoft
>>>>> Flight Simulator 2020 uses it for. Using the system linker
is not an option
>>>>> on Windows, which does not ship link.exe by default, making
LLD a mandatory
>>>>> requirement if you are using LLVM in any kind of end-user
plugin scenario,
>>>>> as the average user has not installed Visual Studio.
>>>>>
>>>>> This puts users of LLVM's library capabilities on
windows in an
>>>>> awkward position, because in order to use LLVM as a library
when compiling
>>>>> a plugin, one must use LLD, which cannot be used as a
library. My current
>>>>> solution is to use LLD as a library anyway and maintain a
fork of LLVM with
>>>>> the various global cleanup bugs patched (most of which have
now made it
>>>>> into stable), along with a helper function that allows me
to use LLD to
>>>>> read out the symbols of a given shared library (which is
used to perform
>>>>> link-time validation of webassembly modules, because LLD
makes it difficult
>>>>> to access any errors that happen).
>>>>>
>>>>> If LLD wanted to become an actual library, I think it would
need a
>>>>> better method of reporting errors than simply an stdout and
stderr stream,
>>>>> although I don't know what this would look like. It
would also be nice for
>>>>> it to expose the different link stages like LLVM does so
that the
>>>>> application has a bit more control over what's going
on. However, I don't
>>>>> really have any concrete ideas about what LLD should look
like as a
>>>>> library, only that I would like it to stop crashing when I
attempt to use
>>>>> it as one.
>>>>>
>>>>> --
>>>>> Sincerely, Erik McClure
>>>>>
>>>>>
>>>>> On Fri, Jun 11, 2021 at 8:20 PM Michael Spencer
<bigcheesegs at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Adding Erik (not subscribed) who has previously had
issues with LLD
>>>>>> not being a library to provide some additional use
cases.
>>>>>>
>>>>>> - Michael Spencer
>>>>>>
>>>>>>
>>>>>> On Thu, Jun 10, 2021 at 12:15 PM Reid Kleckner via
llvm-dev <
>>>>>> llvm-dev at lists.llvm.org> wrote:
>>>>>>
>>>>>>> Hey all,
>>>>>>>
>>>>>>> Long ago, the LLD project contributors decided that
they weren't
>>>>>>> going to design LLD as a library, which stands in
opposition to the way
>>>>>>> that the rest of LLVM strives to be a reusable
library. Part of the
>>>>>>> reasoning was that, at the time, LLD wasn't
done yet, and the top priority
>>>>>>> was to finish making LLD a fast, useful, usable
product. If sacrificing
>>>>>>> reusability helped LLD achieve its project goals,
the contributors at the
>>>>>>> time felt that was the right tradeoff, and that
carried the day.
>>>>>>>
>>>>>>> However, it is now ${YEAR} 2021, and I think we
ought to reconsider
>>>>>>> this design decision. LLD was a great success: it
works, it is fast, it is
>>>>>>> simple, many users have adopted it, it has many
ports
>>>>>>> (COFF/ELF/mingw/wasm/new MachO). Today, we have
actual users who want to
>>>>>>> run the linker as a library, and they aren't
satisfied with the option of
>>>>>>> launching a child process. Some users are
interested in process reuse as a
>>>>>>> performance optimization, some are including the
linker in the frontend.
>>>>>>> Who knows. I try not to pre-judge any of these
efforts, I think we should
>>>>>>> do what we can to enable experimentation.
>>>>>>>
>>>>>>> So, concretely, what could change? The main points
of reusability
>>>>>>> are:
>>>>>>> - Fatal errors and warnings exit the process
without returning
>>>>>>> control to the caller
>>>>>>> - Conflicts over global variables between threads
>>>>>>>
>>>>>>> Error recovery is the big imposition here. To avoid
a giant rewrite
>>>>>>> of all error handling code in LLD, I think we
should *avoid* returning
>>>>>>> failure via the llvm::Error class or
std::error_code. We should instead use
>>>>>>> an approach more like clang, where diagnostics are
delivered to a
>>>>>>> diagnostic consumer on the side. The success of the
link is determined by
>>>>>>> whether any errors were reported. Functions may
return a simple success
>>>>>>> boolean in cases where higher level functions need
to exit early. This has
>>>>>>> worked reasonably well for clang. The main failure
mode here is that we
>>>>>>> miss an error check, and crash or report useless
follow-on errors after an
>>>>>>> error that would normally have been fatal.
>>>>>>>
>>>>>>> Another motivation for all of this is increasing
the use of
>>>>>>> parallelism in LLD. Emitting errors in parallel
from threads and then
>>>>>>> exiting the process is risky business. A new
diagnostic context or consumer
>>>>>>> could make this more reliable. MLIR has this issue
as well, and I believe
>>>>>>> they use this pattern. They use some kind of thread
shard index to order
>>>>>>> the diagnostics, LLD could do the same.
>>>>>>>
>>>>>>> Finally, we'd work to eliminate globals. I
think this is mainly a
>>>>>>> small matter of programming (SMOP) and doesn't
need much discussion,
>>>>>>> although the `make` template presents interesting
challenges.
>>>>>>>
>>>>>>> Thoughts? Tomatoes? Flowers? I apologize for the
lack of context
>>>>>>> links to the original discussions. It takes more
time than I have to dig
>>>>>>> those up.
>>>>>>>
>>>>>>> Reid
>>>>>>> _______________________________________________
>>>>>>> LLVM Developers mailing list
>>>>>>> llvm-dev at lists.llvm.org
>>>>>>>
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>>>>
>>>>>> _______________________________________________
>>>>> LLVM Developers mailing list
>>>>> llvm-dev at lists.llvm.org
>>>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>>
>>>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
>
>
> --
> Neil Henning
> Senior Software Engineer Compiler
> unity.com
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20210614/4b5d5ee4/attachment.html>