llvm dev - Nov 2018 - Generating exported wasm functions

Using LLVM v5, my Cone compiler (https://github.com/jondgoodwin/cone)
automatically directly generated all functions as named and exported for
both wasm text and binary files (not using clang/lld etc, but using the
API).

Upgrading to LLVM v7, generated wasm object files now fail, exhibiting
markedly different behavior, behavior that varies between the text and
binary files. For example, the binary wasm file (only) now automatically
adds two import directives that were not there before. More importantly,
function names, visible in the wat file, are stripped from the binary wasm
file and are not exported. It seems as if LLVM webassembly codegen has
acted as if the linker were at play and had defaulted to options like:
-fvisibility=hidden, -W1, --import-memory etc.

Is there a way to programmatically (via the LLVM APIs) alter this default
behavior, so that I can make public functions once again visible and
exported?  Also, is there documentation somewhere that describes these and
other breaking changes in the way that LLVM v7  now generates webassembly
object files?
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20181117/ab0f10e5/attachment.html>

+Sam Clegg <sbc at google.com>, +Wouter van Oortmerssen <wvo at
google.com>
Hi Jonathan,
A lot has changed with the wasm backend since LLVM v5! One of the reasons
we remained an experimental target was that the interfaces (e.g. the text
and binary formats, and the LLVM APIs) were not stable or well-documented.
We have been investing a lot of work in these formats since then; for
example LLVM's wasm object file format and lld support for static linking
are now feature-complete and mostly documented (
https://github.com/WebAssembly/tool-conventions/blob/master/Linking.md).
Our plan is to additionally document the C ABI and declare it (and the
object file format) as "version 1" by the LLVM 8.0 release in hopes of
keeping that version stable. LLVM's wasm text assembly format is currently
under active development and I wouldn't consider the text format in either
LLVM v7 or current trunk to be stable. We also hope to have some stability
for it by LLVM 8.0.

With respect to the details of symbol visibility and exports, there are
some details about that that we are currently hashing out, as we are
currently designing and implementing dynamic linking to go with our static
linking support. I think there wasm will always have some differences
compared to the model used by e.g. ELF/COFF/MachO. One reason is that wasm
includes the notion of having (a specified set of) symbols exported from a
finally-linked binary; in wasm this is used to export a set of functions to
the external environment (e.g. JavaScript). ELF has --export-dynamic but I
think that's probably not enough control for the web use case.

We definitely do want to enable control over the visibility and export of
public functions (at the source, IR, and linker level). For the IR I'd
expect this would be reflected as attributes on functions or GlobalValues,
which should be controllable via the LLVM APIs.

In any case we'd be interested in hearing about your feedback or use cases;
most of the discussion about the conventions used by LLVM and related tools
are on the GitHub repo we use for our docs:
https://github.com/WebAssembly/tool-conventions/. Feel free to file an
issue or chime in on an existing one.

Best,
-Derek

On Sat, Nov 17, 2018 at 2:04 AM Jonathan Goodwin via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> Using LLVM v5, my Cone compiler (https://github.com/jondgoodwin/cone)
> automatically directly generated all functions as named and exported for
> both wasm text and binary files (not using clang/lld etc, but using the
> API).
>
> Upgrading to LLVM v7, generated wasm object files now fail, exhibiting
> markedly different behavior, behavior that varies between the text and
> binary files. For example, the binary wasm file (only) now automatically
> adds two import directives that were not there before. More importantly,
> function names, visible in the wat file, are stripped from the binary wasm
> file and are not exported. It seems as if LLVM webassembly codegen has
> acted as if the linker were at play and had defaulted to options like:
> -fvisibility=hidden, -W1, --import-memory etc.
>
> Is there a way to programmatically (via the LLVM APIs) alter this default
> behavior, so that I can make public functions once again visible and
> exported?  Also, is there documentation somewhere that describes these and
> other breaking changes in the way that LLVM v7  now generates webassembly
> object files?
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20181128/b1f24c38/attachment.html>

Derek:

Thank you for your detailed and helpful response. Since writing that
e-mail, I have done more research about the work being done. I am very
pleased by the excellent progress on the linker, the new wasm file
conventions, and the emergent work on dynamic linking. I think these are
all very positive developments.

A couple of days ago, I did open an issue on the
https://github.com/WebAssembly/tool-conventions/ repository to explore
these issues in more depth:
https://github.com/WebAssembly/tool-conventions/issues/81. Sam responded
and I have responded back to him. Although I am pleased with what's been
accomplished, there are a few areas where I hope we can do better, both in
terms of the architecture's features and the way we communicate them.

These are important issues, and I am willing to take the time to exchange
perspectives and feedback in detail. I am also willing to write helpful
documentation and/or contribute to LLVM code changes if that too would be
helpful.

Cheers,
Jonathan

On Thu, Nov 29, 2018 at 5:43 AM Derek Schuff <dschuff at google.com>
wrote:
> +Sam Clegg <sbc at google.com>, +Wouter van Oortmerssen <wvo at
google.com>
> Hi Jonathan,
> A lot has changed with the wasm backend since LLVM v5! One of the reasons
> we remained an experimental target was that the interfaces (e.g. the text
> and binary formats, and the LLVM APIs) were not stable or well-documented.
> We have been investing a lot of work in these formats since then; for
> example LLVM's wasm object file format and lld support for static
linking
> are now feature-complete and mostly documented (
> https://github.com/WebAssembly/tool-conventions/blob/master/Linking.md).
> Our plan is to additionally document the C ABI and declare it (and the
> object file format) as "version 1" by the LLVM 8.0 release in
hopes of
> keeping that version stable. LLVM's wasm text assembly format is
currently
> under active development and I wouldn't consider the text format in
either
> LLVM v7 or current trunk to be stable. We also hope to have some stability
> for it by LLVM 8.0.
>
> With respect to the details of symbol visibility and exports, there are
> some details about that that we are currently hashing out, as we are
> currently designing and implementing dynamic linking to go with our static
> linking support. I think there wasm will always have some differences
> compared to the model used by e.g. ELF/COFF/MachO. One reason is that wasm
> includes the notion of having (a specified set of) symbols exported from a
> finally-linked binary; in wasm this is used to export a set of functions to
> the external environment (e.g. JavaScript). ELF has --export-dynamic but I
> think that's probably not enough control for the web use case.
>
> We definitely do want to enable control over the visibility and export of
> public functions (at the source, IR, and linker level). For the IR I'd
> expect this would be reflected as attributes on functions or GlobalValues,
> which should be controllable via the LLVM APIs.
>
> In any case we'd be interested in hearing about your feedback or use
> cases; most of the discussion about the conventions used by LLVM and
> related tools are on the GitHub repo we use for our docs:
> https://github.com/WebAssembly/tool-conventions/. Feel free to file an
> issue or chime in on an existing one.
>
> Best,
> -Derek
>
> On Sat, Nov 17, 2018 at 2:04 AM Jonathan Goodwin via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>> Using LLVM v5, my Cone compiler (https://github.com/jondgoodwin/cone)
>> automatically directly generated all functions as named and exported
for
>> both wasm text and binary files (not using clang/lld etc, but using the
>> API).
>>
>> Upgrading to LLVM v7, generated wasm object files now fail, exhibiting
>> markedly different behavior, behavior that varies between the text and
>> binary files. For example, the binary wasm file (only) now
automatically
>> adds two import directives that were not there before. More
importantly,
>> function names, visible in the wat file, are stripped from the binary
wasm
>> file and are not exported. It seems as if LLVM webassembly codegen has
>> acted as if the linker were at play and had defaulted to options like:
>> -fvisibility=hidden, -W1, --import-memory etc.
>>
>> Is there a way to programmatically (via the LLVM APIs) alter this
default
>> behavior, so that I can make public functions once again visible and
>> exported?  Also, is there documentation somewhere that describes these
and
>> other breaking changes in the way that LLVM v7  now generates
webassembly
>> object files?
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20181129/ee8e9dc9/attachment.html>