llvm dev - May 2021 - LLVM_DYLIB and CLANG_DYLIB with MSVC

Hi,

Clang 12 can be configured on Windows with MinGW (either GNU or LLVM) 
with the following CMake parameters:

  * LLVM_BUILD_LLVM_DYLIB=ON
  * LLVM_LINK_LLVM_DYLIB=ON
  * CLANG_LINK_CLANG_DYLIB=ON

which has some effect on the binary size of the build.

I configured the llvm-project with the following parameters:

  * CMAKE_BUILD_TYPE=Release
  * LLVM_TARGETS_TO_BUILD=X86
  * LLVM_ENABLE_PROJECTS=clang;clang-tools-extra

The installed (stripped) build of Clang 12 with llvm-mingw release 
12.0.0 <https://github.com/mstorsjo/llvm-mingw/releases/tag/20210423> 
resulted in:

  * Normal build: 1,76 GB
  * shlib build: 481 MB

Due to the nature of MSVC regarding default visibility of symbols 
(hidden by default, whereas MinGW has visible by default), one needs to 
generate a .def file with the symbols needed to be exported.

This is done already in two cases for LLVM_BUILD_LLVM_C_DYLIB 
(llvm/tools/llvm-shlib/gen-msvc-exports.py) and for 
LLVM_EXPORT_SYMBOLS_FOR_PLUGINS (llvm/utils/extract_symbols.py).

I've put together a patch 
<https://github.com/cristianadam/llvm-project/commit/3a3b8a7df17a49ba7c0153b0c9a7ee25705ede46>
that enables LLVM_DYLIB and CLANG_DYLIB for MSVC.

I tested with clang-cl from the official Clang 12 x64 Windows binary 
release:

  * Normal build: 1,42 GB
  * shlib build: 536 MB

The shlib release build compiled and linked fine with LLVM.dll and 
clang-cpp.dll, unfortunately it crashes at runtime. For example llvm-nm:

$ llvm-nm
PLEASE submit a bug report to https://bugs.llvm.org/ and include the 
crash backtrace.
Stack dump:
0.      Program arguments: llvm-nm
#0 0x00007ffd32807d43 llvm::StringMap<llvm::cl::Option 
*,llvm::MallocAllocator>::begin 
C:\Projects\llvm-project\repo\llvm\include\llvm\ADT\StringMap.h:204:0
#1 0x00007ffd32807d43 llvm::cl::HideUnrelatedOptions(class 
llvm::cl::OptionCategory &, class llvm::cl::SubCommand &) 
C:\Projects\llvm-project\repo\llvm\lib\Support\CommandLine.cpp:2589:0
#2 0x00007ff689df2b13 llvm::StringRef::StringRef 
C:\Projects\llvm-project\repo\llvm\include\llvm\ADT\StringRef.h:107:0
#3 0x00007ff689df2b13 main 
C:\Projects\llvm-project\repo\llvm\tools\llvm-nm\llvm-nm.cpp:2232:0
#4 0x00007ff689e26d04 invoke_main 
D:\agent\_work\10\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:78:0
#5 0x00007ff689e26d04 __scrt_common_main_seh 
D:\agent\_work\10\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288:0
#6 0x00007ffd9a7f7034 (C:\Windows\System32\KERNEL32.DLL+0x17034)
#7 0x00007ffd9b742651 (C:\Windows\SYSTEM32\ntdll.dll+0x52651)

This crash is due to llvm::cl::HideUnrelatedOptions which 
accessesTopLevelSubCommand, which is defined as:
extern ManagedStatic<SubCommand> TopLevelSubCommand;

The MSVC 2019 build behaves in the same way as the clang-cl build.

I have tried building without LLVM_ENABLE_THREADS, or by linking to the 
CRT statically LLVM_USE_CRT_RELEASE=MT, didn't help.

The MSVC 2019 build sizes were:

  * Normal build: 1,74 GB
  * shlib build: 949 MB

I would appreciate any help in getting the shlib build running. It works 
fine with llvm-mingw, I think it should also work with clang-cl / cl.

Cheers,
Cristian.

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20210530/682a36d8/attachment.html>

I agree, for the reasons you outline, LLVM needs to do more to support
producing DLLs on Windows. However, I don't really have time to dig into
your proposal and the issues you are running into.

Personally, I think LLVM should move away from that .def file generation
python script, and towards source-level annotations. It would allow us to
use fvisibility=hidden in the shared library build on other platforms. That
would greatly reduce the number of dynamic symbols in LLVM, which I
understand is desirable. Today we used Bsymbolic-functions, so we may have
already captured most of the benefits of fvisibility=hidden, but hidden
visibility is always better for performance and binary size.

I put out this alternative proposal mainly to see if there is any
enthusiasm for it. If not, you are welcome to continue forward with our
existing solutions. But if there is broad interest in adding API
annotations to LLVM, I think that would be a better way to go long term.

On Sun, May 30, 2021 at 7:17 AM Cristian Adam via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> Hi,
>
> Clang 12 can be configured on Windows with MinGW (either GNU or LLVM) with
> the following CMake parameters:
>
>    - LLVM_BUILD_LLVM_DYLIB=ON
>    - LLVM_LINK_LLVM_DYLIB=ON
>    - CLANG_LINK_CLANG_DYLIB=ON
>
> which has some effect on the binary size of the build.
>
> I configured the llvm-project with the following parameters:
>
>    - CMAKE_BUILD_TYPE=Release
>    - LLVM_TARGETS_TO_BUILD=X86
>    - LLVM_ENABLE_PROJECTS=clang;clang-tools-extra
>
> The installed (stripped) build of Clang 12 with llvm-mingw release 12.0.0
> <https://github.com/mstorsjo/llvm-mingw/releases/tag/20210423>
resulted
> in:
>
>    - Normal build: 1,76 GB
>    - shlib build: 481 MB
>
> Due to the nature of MSVC regarding default visibility of symbols (hidden
> by default, whereas MinGW has visible by default), one needs to generate a
> .def file with the symbols needed to be exported.
>
> This is done already in two cases for LLVM_BUILD_LLVM_C_DYLIB
(llvm/tools/llvm-shlib/gen-msvc-exports.py)
> and for LLVM_EXPORT_SYMBOLS_FOR_PLUGINS (llvm/utils/extract_symbols.py).
>
> I've put together a patch
>
<https://github.com/cristianadam/llvm-project/commit/3a3b8a7df17a49ba7c0153b0c9a7ee25705ede46>
> that enables LLVM_DYLIB and CLANG_DYLIB for MSVC.
>
> I tested with clang-cl from the official Clang 12 x64 Windows binary
> release:
>
>    - Normal build: 1,42 GB
>    - shlib build: 536 MB
>
> The shlib release build compiled and linked fine with LLVM.dll and
> clang-cpp.dll, unfortunately it crashes at runtime. For example llvm-nm:
>
> $ llvm-nm
> PLEASE submit a bug report to https://bugs.llvm.org/ and include the
> crash backtrace.
> Stack dump:
> 0.      Program arguments: llvm-nm
> #0 0x00007ffd32807d43 llvm::StringMap<llvm::cl::Option
> *,llvm::MallocAllocator>::begin
> C:\Projects\llvm-project\repo\llvm\include\llvm\ADT\StringMap.h:204:0
> #1 0x00007ffd32807d43 llvm::cl::HideUnrelatedOptions(class
> llvm::cl::OptionCategory &, class llvm::cl::SubCommand &)
> C:\Projects\llvm-project\repo\llvm\lib\Support\CommandLine.cpp:2589:0
> #2 0x00007ff689df2b13 llvm::StringRef::StringRef
> C:\Projects\llvm-project\repo\llvm\include\llvm\ADT\StringRef.h:107:0
> #3 0x00007ff689df2b13 main
> C:\Projects\llvm-project\repo\llvm\tools\llvm-nm\llvm-nm.cpp:2232:0
> #4 0x00007ff689e26d04 invoke_main
>
D:\agent\_work\10\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:78:0
> #5 0x00007ff689e26d04 __scrt_common_main_seh
>
D:\agent\_work\10\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288:0
> #6 0x00007ffd9a7f7034 (C:\Windows\System32\KERNEL32.DLL+0x17034)
> #7 0x00007ffd9b742651 (C:\Windows\SYSTEM32\ntdll.dll+0x52651)
>
> This crash is due to llvm::cl::HideUnrelatedOptions which accesses
> TopLevelSubCommand, which is defined as:
> extern ManagedStatic<SubCommand> TopLevelSubCommand;
>
> The MSVC 2019 build behaves in the same way as the clang-cl build.
>
> I have tried building without LLVM_ENABLE_THREADS, or by linking to the
> CRT statically LLVM_USE_CRT_RELEASE=MT, didn't help.
>
> The MSVC 2019 build sizes were:
>
>    - Normal build: 1,74 GB
>    - shlib build: 949 MB
>
> I would appreciate any help in getting the shlib build running. It works
> fine with llvm-mingw, I think it should also work with clang-cl / cl.
>
> Cheers,
> Cristian.
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20210603/2154c017/attachment.html>

On Sun, 30 May 2021, Cristian Adam via llvm-dev wrote:
> Due to the nature of MSVC regarding default visibility of symbols (hidden
by
> default, whereas MinGW has visible by default), one needs to generate a
.def
> file with the symbols needed to be exported.
> 
> This is done already in two cases for LLVM_BUILD_LLVM_C_DYLIB
> (llvm/tools/llvm-shlib/gen-msvc-exports.py) and for
> LLVM_EXPORT_SYMBOLS_FOR_PLUGINS (llvm/utils/extract_symbols.py).
> 
> I've put together a patch that enables LLVM_DYLIB and CLANG_DYLIB for
MSVC.
> 
> I tested with clang-cl from the official Clang 12 x64 Windows binary
> release:
>
>  *  Normal build: 1,42 GB
>  *  shlib build: 536 MB
> 
> The shlib release build compiled and linked fine with LLVM.dll and
> clang-cpp.dll, unfortunately it crashes at runtime.
Without digging into the scripts, I have one hunch:

Does the def generator script differentiate between code and data symbols?

For the cases where accessing a data symbol from another DLL, the caller 
would have to have seen a declaration with the dllimport attribute. For 
functions, it doesn't matter (it just does an extra hop via the import 
thunk), but for data variables it matters. If the def file would have 
proper DATA annotations for such symbols, you would end up with linker 
errors (where you'd have an undefined reference to dataSymbol, where the 
import library only provides __imp_dataSymbol).

This is fixed up by the autoimport feature when linking in mingw mode 
(which, in general, requires you to link against the mingw runtime too); 
for cases where the caller references dataSymbol but you only have 
__imp_dataSymbol available, the linker adds an entry to a list of pseudo 
relocations, which the mingw runtime handles when loaded, which then maps 
sections as writable and patches up the addresses to where they are 
located in another DLL.

So to avoid this, we would either need to actually provide proper 
dllimport declarations at least for all data symbols, or avoid cross-DLL 
data accesses (by using e.g. accessor functions instead).


// Martin