llvm dev - Sep 2021 - Supporting LLVM_BUILD_LLVM_DYLIB on Windows

Hello llvm-dev,

One of the current limitations on LLVM on Windows is that you cannot use
LLVM_BUILD_LLVM_DYLIB:
https://github.com/llvm/llvm-project/blob/main/llvm/tools/llvm-shlib/CMakeLists.txt#L14-L16
 I am interested in trying to see if we can lift this limitation.  There
are others in the community that also seem to be interested in seeing LLVM
being possible to use as a DLL on Windows and the topic does come up on the
mailing lists every so often.

When you build a distribution of a LLVM based toolchain currently, the
result on Windows is ~2GiB for a trimmed down toolset.  This is largely due
to the static linking used for all the tools.  I would like to be able to
use the shared LLVM build for building a toolset on Windows.

Unlike Unix platforms, the default on Windows is that all symbols are
treated as `dso_local` (that is `-fvisibility-default=hidden`).  Symbols
which are meant to participate in dynamic linking are to be attributed as
`__declspec(dllexport)` in the module and `__declspec(dllimport)` external
to the module.  This is similar to Unix platforms where
`__attribute__((__visibility__(...)))` controls the same type of behaviour
with `-fvisibility-default=hidden`.

For the case of distributions, it would remain valuable to minimize the
number of shared objects to reduce the files that require to be shipped but
also to minimize the number of cross-module calls which are not entirely
free (i.e. PLT+GOT or IAT costs).  At the same time, the number of possible
labels which can be exposed from a single module on Windows is limited to
64K.  Experience from MSys2 indicates that LLVM with all the backends is
likely to exceed this count (with a subset of targets, the number already
is close to 60K).  This means that it may be that we would need two
libraries on Windows.

With the LLVM community being diverse, people often build on different
platforms with different configurations, and I am concerned that adding
more differences in how we build libraries complicates how maintainable
LLVM is.  I would suggest that we actually change the behavior of the Unix
builds to match that of Windows by building with
`-fvisibility-default=hidden`.  Although this is a change, it is not
without value.  By explicitly marking the interfaces which are vended by a
library and making everything else internal, it does enable some potential
optimization options for the compiler and linker (to be clear, I am not
suggesting that this will have a guaranteed benefit, just that it can
potentially enable additional opportunities for optimizations and size
reductions).  This should incidentally help static linking.

In order to achieve this, we would need to have a module specific
annotation to indicate what symbols are meant to be used outside of the
module when built in a shared configuration.  The same annotation would
apply to all targets and is expected to be applied uniformly.  This of
course has a cost associated with it: the public interfaces would need to
be decorated appropriately.  However, by having the same behaviour on all
the platforms, developers would not be impacted by the platform differences
in their day-to-day development.  The only time that developers would need
to be aware of this is when they are working on the module boundary, that
is, changes which do not change the API surface of LLVM would not need to
consider the annotations.

Concretely, what I believe is required to enable building with
LLVM_BUILD_LLVM_DYLIB on Windows is:
- introduce module specific decoration (e.g. LLVM_SUPPORT_ABI, ...) to mark
public interfaces of shared library modules
- decorate all the public interfaces of the shared library modules with the
new decoration
- switching the builds to use `-fvisibility-default=hidden` by default

I believe that these can be done mostly independently and staged in the
order specified.  Until the last phase, it would have no actual impact on
the builds.  However, by staging it, we could allow others to experiment
with the option while it is under development, and allows for an easier
path for switching the builds over.

Although this would enable LLVM_BUILD_LLVM_DYLIB on Windows, give us better
uniformity between Windows and non-Windows platforms, potentially enable
additional optimization benefits, improve binary sizes for a distribution
of the toolchain (though less on Linux where distributors are already using
the build configuration ignoring the official suggestions in the LLVM
guides), and help with runtime costs of the toolchain (by making the core
of the tools a shared library, the backing pages can now be shared across
multiple instances), it is not entirely without downsides.  The primary
downsides that I see are:
- it becomes less enticing to support both LLVM_BUILD_LLVM_DYLIB and
BUILD_SHARED_LIBS: while technically possible, interfaces will need to be
decorated for both forms of the build
- LLVM_DYLIB_COMPONENTS becomes less tractable: in theory it is possible to
apply enough CPP magic to determine where a symbol is homed, but allowing a
symbol to be homed in a shared or static library is significantly more
complex
- BUILD_SHARED_LIBS becomes more expensive to maintain: the decoration is
per-module, which requires that we would need to decorate the symbols of
each module with module specific annotations as well

One argument that people make for BUILD_SHARED_LIBS is that it reduces the
overall time build-test cycle.  With the combination of lld, DWARF Fission,
and LLVM_BUILD_LLVM_DYLIB, I believe that most of the benefits still can be
had.  The cost of linking all the tools is amortized across the link of a
single library, which while not as small as the a singular library, is
offset by the following:
- The LLVM_BUILD_LLVM_DYLIB would not require the re-linking of all the
libraries for each tool.
- DWARF Fission would avoid the need to relink all of the DWARF information.
- lld is faster than the gold and bfd linkers

Header changes would still ripple through the system as before, requiring
rebuilding the transitive closure.  Source file changes do not have the
same impact of course.

For those would like a more concrete example of what a change like this may
shape up into: https://reviews.llvm.org/D109192 contains
`LLVMSupportExports.h` which has the expected structure for declaring the
decoration macros with the rest of the change primarily being focused on
applying the decoration.  Please ignore the CMake changes as they are there
to ensure that the CI validates this without changing the configuration and
not intended to be part of the final version of the change.

-- 
Saleem Abdulrasool
compnerd (at) compnerd (dot) org
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20210908/e7690ec7/attachment.html>

On 9/8/21 3:52 PM, Saleem Abdulrasool via llvm-dev
wrote:
> Hello llvm-dev,
> 
In general, I am in favor of this change and think it would be a good
improvement.  A few comments below:
> One of the current limitations on LLVM on Windows is that you cannot use
LLVM_BUILD_LLVM_DYLIB:
https://github.com/llvm/llvm-project/blob/main/llvm/tools/llvm-shlib/CMakeLists.txt#L14-L16
<https://github.com/llvm/llvm-project/blob/main/llvm/tools/llvm-shlib/CMakeLists.txt#L14-L16>
 I am interested in trying to see if we can lift this limitation.  There are
others in the community that also seem to be interested in seeing LLVM being
possible to use as a DLL on Windows and the topic does come up on the mailing
lists every so often.
> 
> When you build a distribution of a LLVM based toolchain currently, the
result on Windows is ~2GiB for a trimmed down toolset.  This is largely due to
the static linking used for all the tools.  I would like to be able to use the
shared LLVM build for building a toolset on Windows.
> 
> Unlike Unix platforms, the default on Windows is that all symbols are
treated as `dso_local` (that is `-fvisibility-default=hidden`).  Symbols which
are meant to participate in dynamic linking are to be attributed as
`__declspec(dllexport)` in the module and `__declspec(dllimport)` external to
the module.  This is similar to Unix platforms where
`__attribute__((__visibility__(...)))` controls the same type of behaviour with
`-fvisibility-default=hidden`.
> 
> For the case of distributions, it would remain valuable to minimize the
number of shared objects to reduce the files that require to be shipped but also
to minimize the number of cross-module calls which are not entirely free (i.e.
PLT+GOT or IAT costs).  At the same time, the number of possible labels which
can be exposed from a single module on Windows is limited to 64K.  Experience
from MSys2 indicates that LLVM with all the backends is likely to exceed this
count (with a subset of targets, the number already is close to 60K).  This
means that it may be that we would need two libraries on Windows.
> 
The backends only need to export a handful of symbols, so I don't think
enabling more targets will have that big of impact on the number
of exported symbols.
> With the LLVM community being diverse, people often build on different
platforms with different configurations, and I am concerned that adding more
differences in how we build libraries complicates how maintainable LLVM is.  I
would suggest that we actually change the behavior of the Unix builds to match
that of Windows by building with `-fvisibility-default=hidden`.  Although this
is a change, it is not without value.  By explicitly marking the interfaces
which are vended by a library and making everything else internal, it does
enable some potential optimization options for the compiler and linker (to be
clear, I am not suggesting that this will have a guaranteed benefit, just that
it can potentially enable additional opportunities for optimizations and size
reductions).  This should incidentally help static linking.
> 
> In order to achieve this, we would need to have a module specific
annotation to indicate what symbols are meant to be used outside of the module
when built in a shared configuration.  The same annotation would apply to all
targets and is expected to be applied uniformly.  This of course has a cost
associated with it: the public interfaces would need to be decorated
appropriately.  However, by having the same behaviour on all the platforms,
developers would not be impacted by the platform differences in their day-to-day
development.  The only time that developers would need to be aware of this is
when they are working on the module boundary, that is, changes which do not
change the API surface of LLVM would not need to consider the annotations.
> 
> Concretely, what I believe is required to enable building with
LLVM_BUILD_LLVM_DYLIB on Windows is:
> - introduce module specific decoration (e.g. LLVM_SUPPORT_ABI, ...) to mark
public interfaces of shared library modules
> - decorate all the public interfaces of the shared library modules with the
new decoration
> - switching the builds to use `-fvisibility-default=hidden` by default
> 
My recommendation would be to start with a generic decoration (e.g.
LLVM_EXPORTED_API)
and decorate all the currently exported symbols (which currently is all of them
except
for the lib/Target symbols on Linux).  This way you could switch to
-fvisibility-default=hidden
and it would have no impact on library users.  I know it's tedious work, but
you can split it up
and recruit others (like me) to help make the changes.

Once you have that working we can start trimming down the ABI and determine
whether or
not we actually need to have two libraries or not.  If the full build with a few
targets
enabled is already under the limit, I think there is a good chance we can get
a single library to work.

The advantage of this approach is that you can make a lot of progress in a very
non-invasive way.
> I believe that these can be done mostly independently and staged in the
order specified.  Until the last phase, it would have no actual impact on the
builds.  However, by staging it, we could allow others to experiment with the
option while it is under development, and allows for an easier path for
switching the builds over.
> 
> Although this would enable LLVM_BUILD_LLVM_DYLIB on Windows, give us better
uniformity between Windows and non-Windows platforms, potentially enable
additional optimization benefits, improve binary sizes for a distribution of the
toolchain (though less on Linux where distributors are already using the build
configuration ignoring the official suggestions in the LLVM guides), and help
with runtime costs of the toolchain (by making the core of the tools a shared
library, the backing pages can now be shared across multiple instances), it is
not entirely without downsides.  The primary downsides that I see are:
> - it becomes less enticing to support both LLVM_BUILD_LLVM_DYLIB and
BUILD_SHARED_LIBS: while technically possible, interfaces will need to be
decorated for both forms of the build
I don't think we should make any changes to the BUILD_SHARED_LIBS build.
This is intended for developer convenience only and not recommend for
end users or distros[1].

Supporting BUILD_SHARED_LIBS will require annotating more of the API
and create more work both initially and also in ongoing maintenance,
and I don't think it is worth it.

-Tom
> - LLVM_DYLIB_COMPONENTS becomes less tractable: in theory it is possible to
apply enough CPP magic to determine where a symbol is homed, but allowing a
symbol to be homed in a shared or static library is significantly more complex
> - BUILD_SHARED_LIBS becomes more expensive to maintain: the decoration is
per-module, which requires that we would need to decorate the symbols of each
module with module specific annotations as well
> 
> One argument that people make for BUILD_SHARED_LIBS is that it reduces the
overall time build-test cycle.  With the combination of lld, DWARF Fission, and
LLVM_BUILD_LLVM_DYLIB, I believe that most of the benefits still can be had. 
The cost of linking all the tools is amortized across the link of a single
library, which while not as small as the a singular library, is offset by the
following:
> - The LLVM_BUILD_LLVM_DYLIB would not require the re-linking of all the
libraries for each tool.
> - DWARF Fission would avoid the need to relink all of the DWARF
information.
> - lld is faster than the gold and bfd linkers
> 
> Header changes would still ripple through the system as before, requiring
rebuilding the transitive closure.  Source file changes do not have the same
impact of course.
> 
> For those would like a more concrete example of what a change like this may
shape up into: https://reviews.llvm.org/D109192
<https://reviews.llvm.org/D109192> contains `LLVMSupportExports.h` which
has the expected structure for declaring the decoration macros with the rest of
the change primarily being focused on applying the decoration.  Please ignore
the CMake changes as they are there to ensure that the CI validates this without
changing the configuration and not intended to be part of the final version of
the change.
> 
[1] https://llvm.org/docs/CMake.html#llvm-related-variables

> -- 
> Saleem Abdulrasool
> compnerd (at) compnerd (dot) org
> 
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>

Hi Saleem,

I am concerned that your change will increase the maintenance burden for
those of us who would prefer to develop without shared libraries. Since it
is unclear a priori where the macros will be required, developers will need
to build both with and without shared libraries in order to verify that
they aren't breaking the build for shared library users -- in effect
slowing down the development for folks who prefer to develop without shared
libraries.

I think your goal should be achievable without littering the code with
macros. Perhaps on Windows you can achieve your goal with a variant of
Leonard Chan's "busybox" proposal [1] with some adjustments to
account for
a lack of symlink support on Windows. Perhaps something like:

- Create a <tool name>_main() entry point for each tool that does not use
llvm::cl to parse options.
- Create a llvm.dll in the bin directory that links together all the <tool
name>_main() entry points.
- Each tool <tool name>.exe consists of:
int main() {
  <tool name>_main();
}
- Tools that use llvm::cl will need to be linked with all of their code in
the .exe for now. However, they can be incrementally switched away from
llvm::cl and moved into llvm.dll.

Peter

[1] https://lists.llvm.org/pipermail/llvm-dev/2021-June/151321.html

On Wed, Sep 8, 2021 at 3:52 PM Saleem Abdulrasool via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> Hello llvm-dev,
>
> One of the current limitations on LLVM on Windows is that you cannot use
> LLVM_BUILD_LLVM_DYLIB:
>
https://github.com/llvm/llvm-project/blob/main/llvm/tools/llvm-shlib/CMakeLists.txt#L14-L16
>  I am interested in trying to see if we can lift this limitation.  There
> are others in the community that also seem to be interested in seeing LLVM
> being possible to use as a DLL on Windows and the topic does come up on the
> mailing lists every so often.
>
> When you build a distribution of a LLVM based toolchain currently, the
> result on Windows is ~2GiB for a trimmed down toolset.  This is largely due
> to the static linking used for all the tools.  I would like to be able to
> use the shared LLVM build for building a toolset on Windows.
>
> Unlike Unix platforms, the default on Windows is that all symbols are
> treated as `dso_local` (that is `-fvisibility-default=hidden`).  Symbols
> which are meant to participate in dynamic linking are to be attributed as
> `__declspec(dllexport)` in the module and `__declspec(dllimport)` external
> to the module.  This is similar to Unix platforms where
> `__attribute__((__visibility__(...)))` controls the same type of behaviour
> with `-fvisibility-default=hidden`.
>
> For the case of distributions, it would remain valuable to minimize the
> number of shared objects to reduce the files that require to be shipped but
> also to minimize the number of cross-module calls which are not entirely
> free (i.e. PLT+GOT or IAT costs).  At the same time, the number of possible
> labels which can be exposed from a single module on Windows is limited to
> 64K.  Experience from MSys2 indicates that LLVM with all the backends is
> likely to exceed this count (with a subset of targets, the number already
> is close to 60K).  This means that it may be that we would need two
> libraries on Windows.
>
> With the LLVM community being diverse, people often build on different
> platforms with different configurations, and I am concerned that adding
> more differences in how we build libraries complicates how maintainable
> LLVM is.  I would suggest that we actually change the behavior of the Unix
> builds to match that of Windows by building with
> `-fvisibility-default=hidden`.  Although this is a change, it is not
> without value.  By explicitly marking the interfaces which are vended by a
> library and making everything else internal, it does enable some potential
> optimization options for the compiler and linker (to be clear, I am not
> suggesting that this will have a guaranteed benefit, just that it can
> potentially enable additional opportunities for optimizations and size
> reductions).  This should incidentally help static linking.
>
> In order to achieve this, we would need to have a module specific
> annotation to indicate what symbols are meant to be used outside of the
> module when built in a shared configuration.  The same annotation would
> apply to all targets and is expected to be applied uniformly.  This of
> course has a cost associated with it: the public interfaces would need to
> be decorated appropriately.  However, by having the same behaviour on all
> the platforms, developers would not be impacted by the platform differences
> in their day-to-day development.  The only time that developers would need
> to be aware of this is when they are working on the module boundary, that
> is, changes which do not change the API surface of LLVM would not need to
> consider the annotations.
>
> Concretely, what I believe is required to enable building with
> LLVM_BUILD_LLVM_DYLIB on Windows is:
> - introduce module specific decoration (e.g. LLVM_SUPPORT_ABI, ...) to
> mark public interfaces of shared library modules
> - decorate all the public interfaces of the shared library modules with
> the new decoration
> - switching the builds to use `-fvisibility-default=hidden` by default
>
> I believe that these can be done mostly independently and staged in the
> order specified.  Until the last phase, it would have no actual impact on
> the builds.  However, by staging it, we could allow others to experiment
> with the option while it is under development, and allows for an easier
> path for switching the builds over.
>
> Although this would enable LLVM_BUILD_LLVM_DYLIB on Windows, give us
> better uniformity between Windows and non-Windows platforms, potentially
> enable additional optimization benefits, improve binary sizes for a
> distribution of the toolchain (though less on Linux where distributors are
> already using the build configuration ignoring the official suggestions in
> the LLVM guides), and help with runtime costs of the toolchain (by making
> the core of the tools a shared library, the backing pages can now be shared
> across multiple instances), it is not entirely without downsides.  The
> primary downsides that I see are:
> - it becomes less enticing to support both LLVM_BUILD_LLVM_DYLIB and
> BUILD_SHARED_LIBS: while technically possible, interfaces will need to be
> decorated for both forms of the build
> - LLVM_DYLIB_COMPONENTS becomes less tractable: in theory it is possible
> to apply enough CPP magic to determine where a symbol is homed, but
> allowing a symbol to be homed in a shared or static library is
> significantly more complex
> - BUILD_SHARED_LIBS becomes more expensive to maintain: the decoration is
> per-module, which requires that we would need to decorate the symbols of
> each module with module specific annotations as well
>
> One argument that people make for BUILD_SHARED_LIBS is that it reduces the
> overall time build-test cycle.  With the combination of lld, DWARF Fission,
> and LLVM_BUILD_LLVM_DYLIB, I believe that most of the benefits still can be
> had.  The cost of linking all the tools is amortized across the link of a
> single library, which while not as small as the a singular library, is
> offset by the following:
> - The LLVM_BUILD_LLVM_DYLIB would not require the re-linking of all the
> libraries for each tool.
> - DWARF Fission would avoid the need to relink all of the DWARF
> information.
> - lld is faster than the gold and bfd linkers
>
> Header changes would still ripple through the system as before, requiring
> rebuilding the transitive closure.  Source file changes do not have the
> same impact of course.
>
> For those would like a more concrete example of what a change like this
> may shape up into: https://reviews.llvm.org/D109192 contains
> `LLVMSupportExports.h` which has the expected structure for declaring the
> decoration macros with the rest of the change primarily being focused on
> applying the decoration.  Please ignore the CMake changes as they are there
> to ensure that the CI validates this without changing the configuration and
> not intended to be part of the final version of the change.
>
> --
> Saleem Abdulrasool
> compnerd (at) compnerd (dot) org
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>

-- 
-- 
Peter
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20210908/2bf5dfe8/attachment.html>

+1

I would even go as far to make libLLVM.dll (and libClang.dll?) the
only (supported) configuration on Windows. The reason is that we
currently cannot properly support plugins on Windows because these
plugins need to know from which file to import symbols from.
llvm_add_library has a PLUGIN_TOOL parameter to set this, but
consequently the plugin can only be used with that tool (typically
opt.exe), but to have a plugin for another executable like clang.exe,
one would need another plugin binary and yet another one different one
for clang-cl.exe etc. Having a canonical libLLVM.so for all
tools+clang would finally allow us to support plugins one Windows(*).

Also note previous discussion on this, e.g. [1]. [2] already suggests
to introduce component-specific headers, from where
dllimport/dllexport could be controlled.

Michael

(*) IMHO, the plugin system is currently broken under Linux as well.
The plugin may need to link against symbols which simply do not exist
in the target executable because the tool itself does not need them
and have not been added by the linker.
[1] https://lists.llvm.org/pipermail/llvm-dev/2021-June/150860.html
[2] https://lists.llvm.org/pipermail/llvm-dev/2021-June/151187.html

Let me start by saying that I think this is important work that you are
undertaking. This is a huge task, and it you pull it off you will be a hero. Do
you have any notion of how big the distribution would be on Windows with the dll
build? 2 GB is ridiculous!

At a previous workplace, we had a similar problem, and we solved it in much the
way that you propose: by having a magic export macro that we decorated all
external symbols with. The main problem was the human factor: there are those
that don’t develop on Windows (we didn’t disable exporting everything by default
on Linux and Mac), and it was a constant battle to get these developers to use
the export macro. If we disable exporting everything on not-Windows, that will
at least reduce the amount of people that fail to use the macro. (as they will
get linker failures and presumably the build bots will complain) However, there
will always be a contingent of people saying things like “This problem that you
are solving doesn’t affect me, and now I a am being asked to do extra work! I
don’t like this!” My personal opinion is that:

```
class LLVM_SUPPORT_ABI Foo { …
```

… is not materially harder to write than:

```
class Foo { …
```

… and this sort of stuff is just a fact of life in writing C++ code that is
portable between Windows and Linux.

However, if you don’t dynamically link in your day-to-day work, then it’s easy
to mess this up. Is it `void LLVM_SUPPORT_ABI bar()` or `LLVM_SUPPORT_ABI void
bar()`? Do I have to decorate class members, or just the class? When can I omit
it? Do I have to decorate templated code? If you’re statically linking
`LLVM_SUPPORT_ABI` preprocesses to nothing, so you can put it literally anywhere
and your code will compile. I can’t think of how to do it offhand, and maybe
it’s not possible, but if the system could somehow be rigged to fail to compile
if you do it wrong, even for statically linked builds that would be nice. If
this could be done, it would also help with the transition. At some point you’re
going to flip the switch on this and everybody is going to get random breakages
for a few days while they hunt down the stragglers.

All I can say is: “good luck, I’ll be rooting for you!”

Thanks,
   Chris Tetreault

From: llvm-dev <llvm-dev-bounces at lists.llvm.org> On Behalf Of Saleem
Abdulrasool via llvm-dev
Sent: Wednesday, September 8, 2021 3:52 PM
To: llvm-dev <llvm-dev at lists.llvm.org>
Cc: Saleem Abdulrasool <abdulras at google.com>; Hans Wennborg
<hwennborg at google.com>
Subject: [llvm-dev] Supporting LLVM_BUILD_LLVM_DYLIB on Windows


WARNING: This email originated from outside of Qualcomm. Please be wary of any
links or attachments, and do not enable macros.
Hello llvm-dev,

One of the current limitations on LLVM on Windows is that you cannot use
LLVM_BUILD_LLVM_DYLIB:
https://github.com/llvm/llvm-project/blob/main/llvm/tools/llvm-shlib/CMakeLists.txt#L14-L16
I am interested in trying to see if we can lift this limitation.  There are
others in the community that also seem to be interested in seeing LLVM being
possible to use as a DLL on Windows and the topic does come up on the mailing
lists every so often.

When you build a distribution of a LLVM based toolchain currently, the result on
Windows is ~2GiB for a trimmed down toolset.  This is largely due to the static
linking used for all the tools.  I would like to be able to use the shared LLVM
build for building a toolset on Windows.

Unlike Unix platforms, the default on Windows is that all symbols are treated as
`dso_local` (that is `-fvisibility-default=hidden`).  Symbols which are meant to
participate in dynamic linking are to be attributed as `__declspec(dllexport)`
in the module and `__declspec(dllimport)` external to the module.  This is
similar to Unix platforms where `__attribute__((__visibility__(...)))` controls
the same type of behaviour with `-fvisibility-default=hidden`.

For the case of distributions, it would remain valuable to minimize the number
of shared objects to reduce the files that require to be shipped but also to
minimize the number of cross-module calls which are not entirely free (i.e.
PLT+GOT or IAT costs).  At the same time, the number of possible labels which
can be exposed from a single module on Windows is limited to 64K.  Experience
from MSys2 indicates that LLVM with all the backends is likely to exceed this
count (with a subset of targets, the number already is close to 60K).  This
means that it may be that we would need two libraries on Windows.

With the LLVM community being diverse, people often build on different platforms
with different configurations, and I am concerned that adding more differences
in how we build libraries complicates how maintainable LLVM is.  I would suggest
that we actually change the behavior of the Unix builds to match that of Windows
by building with `-fvisibility-default=hidden`.  Although this is a change, it
is not without value.  By explicitly marking the interfaces which are vended by
a library and making everything else internal, it does enable some potential
optimization options for the compiler and linker (to be clear, I am not
suggesting that this will have a guaranteed benefit, just that it can
potentially enable additional opportunities for optimizations and size
reductions).  This should incidentally help static linking.

In order to achieve this, we would need to have a module specific annotation to
indicate what symbols are meant to be used outside of the module when built in a
shared configuration.  The same annotation would apply to all targets and is
expected to be applied uniformly.  This of course has a cost associated with it:
the public interfaces would need to be decorated appropriately.  However, by
having the same behaviour on all the platforms, developers would not be impacted
by the platform differences in their day-to-day development.  The only time that
developers would need to be aware of this is when they are working on the module
boundary, that is, changes which do not change the API surface of LLVM would not
need to consider the annotations.

Concretely, what I believe is required to enable building with
LLVM_BUILD_LLVM_DYLIB on Windows is:
- introduce module specific decoration (e.g. LLVM_SUPPORT_ABI, ...) to mark
public interfaces of shared library modules
- decorate all the public interfaces of the shared library modules with the new
decoration
- switching the builds to use `-fvisibility-default=hidden` by default

I believe that these can be done mostly independently and staged in the order
specified.  Until the last phase, it would have no actual impact on the builds. 
However, by staging it, we could allow others to experiment with the option
while it is under development, and allows for an easier path for switching the
builds over.

Although this would enable LLVM_BUILD_LLVM_DYLIB on Windows, give us better
uniformity between Windows and non-Windows platforms, potentially enable
additional optimization benefits, improve binary sizes for a distribution of the
toolchain (though less on Linux where distributors are already using the build
configuration ignoring the official suggestions in the LLVM guides), and help
with runtime costs of the toolchain (by making the core of the tools a shared
library, the backing pages can now be shared across multiple instances), it is
not entirely without downsides.  The primary downsides that I see are:
- it becomes less enticing to support both LLVM_BUILD_LLVM_DYLIB and
BUILD_SHARED_LIBS: while technically possible, interfaces will need to be
decorated for both forms of the build
- LLVM_DYLIB_COMPONENTS becomes less tractable: in theory it is possible to
apply enough CPP magic to determine where a symbol is homed, but allowing a
symbol to be homed in a shared or static library is significantly more complex
- BUILD_SHARED_LIBS becomes more expensive to maintain: the decoration is
per-module, which requires that we would need to decorate the symbols of each
module with module specific annotations as well

One argument that people make for BUILD_SHARED_LIBS is that it reduces the
overall time build-test cycle.  With the combination of lld, DWARF Fission, and
LLVM_BUILD_LLVM_DYLIB, I believe that most of the benefits still can be had. 
The cost of linking all the tools is amortized across the link of a single
library, which while not as small as the a singular library, is offset by the
following:
- The LLVM_BUILD_LLVM_DYLIB would not require the re-linking of all the
libraries for each tool.
- DWARF Fission would avoid the need to relink all of the DWARF information.
- lld is faster than the gold and bfd linkers

Header changes would still ripple through the system as before, requiring
rebuilding the transitive closure.  Source file changes do not have the same
impact of course.

For those would like a more concrete example of what a change like this may
shape up into: https://reviews.llvm.org/D109192 contains `LLVMSupportExports.h`
which has the expected structure for declaring the decoration macros with the
rest of the change primarily being focused on applying the decoration.  Please
ignore the CMake changes as they are there to ensure that the CI validates this
without changing the configuration and not intended to be part of the final
version of the change.

--
Saleem Abdulrasool
compnerd (at) compnerd (dot) org
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20210909/b3c34219/attachment.html>