llvm dev - Dec 2011 - [LLVMdev] [RFC]Extending lib/Linker to support bitcode "shared objects"

On Sun, Dec 11, 2011 at 2:19 PM, Rafael Ávila de Espíndola <
rafael.espindola at gmail.com> wrote:
> On 08/12/11 03:56 PM, Ivan Krasin wrote:
> > Hi llvm team!
> >
> > I'm currently working on the extended version of llvm-ld, which
has an
> > ability to check if all the symbols present (and fail if some symbols
> > are not resolved), treat archives in the right way (link all the
object
> > files in the archive if it's specified as the regular input, not
as -l)
>
> Is that the "right way"? I think all native linker require a
> --whole-archive or similar to do that, no?
>
My mistake. Please, read the phrase as "support -whole-archive".

>
> > and the most important to my project feature: to link against bitcode
> > "shared objects". The semantics is pretty simple. The module
treated as
> > shared object is only used as the source of symbol names which be
> > excluded from the list unresolved symbols.
>
> In summary, linking foo.bc and bar.bcso (do you use a different
> extension to differentiate from a "static" bc?) would be
equivalent (but
> faster, easier) to
>
> $ llc bar.bc -filetype=obj -o bar.o
> $ clang -shared -o bar.so bar.o
> $ clang -use-gold-plugin foo.o bar.so -o t
>
> Is that correct? In particular, "lld t" should show a dependency
on bar.
> Any particular reason for not adding this to the plugin api?
>
The result is a native .so here. My goal is to have a bitcode result.

gold plugin with mods supports that as well, but it would be nice to avoid
dependency on gold and gold plugin to simplify things.

>
> > I feel like the similar functionality might be useful for other
projects
> > as well. For example, Emscripten (as of 3 months ago, when I have
> > checked it) does not check for undefined symbols at link time, because
> > it does not have a way to link against glibc (the symbols from which
> > would be added at runtime as usual javascript functions). This adds
> > additional overhead to the developer who ports the software to
> > javascript using Emscripten.
>
> Is that the same case? It seems that it would be easier to just include
> a stub ELF libc.so with the symbols from libc that Emscripten supports.
>
I believe that Emscripten does not use gold and gold plugin at the moment.
They use llvm-link:
https://github.com/kripken/emscripten/wiki/Building-Projects

I fully agree that all the features I want from the bitcode linker may be
achieved with gold + gold plugin (with mods to both parts), but I would
like to stay away from this dependency. Partly, because maintaining
modified gold and gold plugin is no fun, partly because gold + gold plugin
is hardly broken under cygwin. See
http://code.google.com/p/nativeclient/issues/detail?id=2286 for more
details. In short, it's because cygwin does not fully support dlopen.
https://www.google.com/search?sourceid=chrome&ie=UTF-8&q=cygwin+dlopen

It may be fixed by linking gold plugin to gold statically, like here:
http://codereview.chromium.org/8713008/ but it only adds the complexity.
>
> > Any objections? Comments?
> >
> > Ivan Krasin
>
> Cheers,
> Rafael
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20111212/86621978/attachment.html>

>     $ llc bar.bc -filetype=obj -o bar.o
>     $ clang -shared -o bar.so bar.o
>     $ clang -use-gold-plugin foo.o bar.so -o t
> 
>     Is that correct? In particular, "lld t" should show a
dependency on bar.
>     Any particular reason for not adding this to the plugin api?
> 
> The result is a native .so here. My goal is to have a bitcode result.
> 
> gold plugin with mods supports that as well, but it would be nice to
> avoid dependency on gold and gold plugin to simplify things.
Sorry, I meant equivalent only in the produced executable (t). In the
above example the bar.bc is your "shared IL library" which I assume
already exists. You want to produce a executable (t) which has a
dependency on a shared library with the symbols defined in foo.bc.

Is that the use case?
> I believe that Emscripten does not use gold and gold plugin at the
> moment. They use
> llvm-link: https://github.com/kripken/emscripten/wiki/Building-Projects
I think that is the current implementation, yes.
> I fully agree that all the features I want from the bitcode linker may
> be achieved with gold + gold plugin (with mods to both parts), but I
> would like to stay away from this dependency. Partly, because
> maintaining modified gold and gold plugin is no fun, partly because gold
> + gold plugin is hardly broken under cygwin.
I understand. I was just pointing that emacscriten case looks different
from what you have. I don't see where an LLVM IL library (libc in your
example) would fit. All that is needed is a stub with the functions that
will be defined in JS in the end.

Cheers,
Rafael

On Tue, Dec 13, 2011 at 9:27 PM, Rafael Ávila de Espíndola <
rafael.espindola at gmail.com> wrote:
>
> >     $ llc bar.bc -filetype=obj -o bar.o
> >     $ clang -shared -o bar.so bar.o
> >     $ clang -use-gold-plugin foo.o bar.so -o t
> >
> >     Is that correct? In particular, "lld t" should show a
dependency on
> bar.
> >     Any particular reason for not adding this to the plugin api?
> >
> > The result is a native .so here. My goal is to have a bitcode result.
> >
> > gold plugin with mods supports that as well, but it would be nice to
> > avoid dependency on gold and gold plugin to simplify things.
>
> Sorry, I meant equivalent only in the produced executable (t). In the
> above example the bar.bc is your "shared IL library" which I
assume
> already exists. You want to produce a executable (t) which has a
> dependency on a shared library with the symbols defined in foo.bc.
>
> Is that the use case?
>
Almost. The difference is that I want to have a bitcode module as the
result of linking, not executable file.
>
> > I believe that Emscripten does not use gold and gold plugin at the
> > moment. They use
> > llvm-link:
https://github.com/kripken/emscripten/wiki/Building-Projects
>
> I think that is the current implementation, yes.
>
> > I fully agree that all the features I want from the bitcode linker may
> > be achieved with gold + gold plugin (with mods to both parts), but I
> > would like to stay away from this dependency. Partly, because
> > maintaining modified gold and gold plugin is no fun, partly because
gold
> > + gold plugin is hardly broken under cygwin.
>
> I understand. I was just pointing that emacscriten case looks different
> from what you have. I don't see where an LLVM IL library (libc in your
> example) would fit. All that is needed is a stub with the functions that
> will be defined in JS in the end.
>
bitcode "so" is supposed to have the form of:

define void @printf() {
  ret void
}
...

In this case, Empscripten might want to generate such stubs based on the
list of javascript functions they provide and in my case, the generation of
this LL stub is a simple script based on nm, sed and llvm-as that takes
native .so as the input and produces a bitcode file with stub definitions
of the public symbols.

This is about the same use case as Emscripten with the only difference in
the source of the defined symbols (native .so vs js file)

Is it more clear now? If not, I would like to give it another try and write
much more details and examples.

krasin
>
> Cheers,
> Rafael
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20111213/3a3af8c9/attachment.html>