llvm dev - Dec 2011 - [LLVMdev] [RFC]Extending lib/Linker to support bitcode "shared objects"

Hi llvm team!

I'm currently working on the extended version of llvm-ld, which has an
ability to check if all the symbols present (and fail if some symbols are
not resolved), treat archives in the right way (link all the object files
in the archive if it's specified as the regular input, not as -l) and the
most important to my project feature: to link against bitcode "shared
objects". The semantics is pretty simple. The module treated as shared
object is only used as the source of symbol names which be excluded from
the list unresolved symbols. For example:

foo.c:

int main(void) {
  return bar()
}

bar.c:

int bar() {
  return 123;
}

In the native case, you have three options to link these files.

1. Classic static linking:

$ clang foo.c bar.c


2. Static linking with the archive.

$ clang -c bar.c
$ ar q libbar.a bar.o
$ clang foo.c -lbar -L.


3. Dynamic linking:

$ clang -c foo.c
$ clang -shared bar.c -o libbar.so
$ clang foo.c -lbar -L.
$ nm a.out
...
U bar
...
=========================================

Currently, lib/Linker supports first two cases, but not the third. I would
like to be able add this functionality and it even exists as the prototype:
https://github.com/krasin/bitlink
To build, just place it under llvm/tools and register in
tools/CMakefiles.txt

The utility itself is probably not interesting, because it's just a
modified llvm-ld. At the same time, it uses lib/Linker with the added
capability of dynamic linking with bitcode stubs:

$ clang -emit-llvm -c foo.c
clang -emit-llvm -c bar.c -o libbar.o

# Fail if there're undefined symbols:
$ bin/llvm-bitlink foo.o
llvm-bitlink: undefined reference to bar
llvm-bitlink: Linker aborted due to undefined symbols

# Add bitcode "shared object" and link against it:
$ bin/llvm-bitlink foo.o -shared-library bar.o

Check a.out.ll:
$ llvm-dis a.out
$ cat a.out.ll
...
define i32 @main(i32 %argc, i8** nocapture %argv) nounwind uwtable {
entry:
  %call = call i32 (...)* @bar() nounwind
  ret i32 %call
}

declare i32 @bar(...)
=========================================

I know, it's a basic example and it's not anyhow impressive, but
hopefully
it provides the clue what I'm talking about. Our plans include generating
bitcode link-time stubs for glibc and doing the pure bitcode linking for
PNaCl.
I feel like the similar functionality might be useful for other projects as
well. For example, Emscripten (as of 3 months ago, when I have checked it)
does not check for undefined symbols at link time, because it does not have
a way to link against glibc (the symbols from which would be added at
runtime as usual javascript functions). This adds additional overhead to
the developer who ports the software to javascript using Emscripten.

Any objections? Comments?

Ivan Krasin
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20111208/d4e7f914/attachment.html>

On 08/12/11 03:56 PM, Ivan Krasin wrote:
> Hi llvm team!
> 
> I'm currently working on the extended version of llvm-ld, which has an
> ability to check if all the symbols present (and fail if some symbols
> are not resolved), treat archives in the right way (link all the object
> files in the archive if it's specified as the regular input, not as -l)
Is that the "right way"? I think all native linker require a
--whole-archive or similar to do that, no?
> and the most important to my project feature: to link against bitcode
> "shared objects". The semantics is pretty simple. The module
treated as
> shared object is only used as the source of symbol names which be
> excluded from the list unresolved symbols.
In summary, linking foo.bc and bar.bcso (do you use a different
extension to differentiate from a "static" bc?) would be equivalent
(but
faster, easier) to

$ llc bar.bc -filetype=obj -o bar.o
$ clang -shared -o bar.so bar.o
$ clang -use-gold-plugin foo.o bar.so -o t

Is that correct? In particular, "lld t" should show a dependency on
bar.
Any particular reason for not adding this to the plugin api?
> I feel like the similar functionality might be useful for other projects
> as well. For example, Emscripten (as of 3 months ago, when I have
> checked it) does not check for undefined symbols at link time, because
> it does not have a way to link against glibc (the symbols from which
> would be added at runtime as usual javascript functions). This adds
> additional overhead to the developer who ports the software to
> javascript using Emscripten.
Is that the same case? It seems that it would be easier to just include
a stub ELF libc.so with the symbols from libc that Emscripten supports.
> Any objections? Comments?
> 
> Ivan Krasin
Cheers,
Rafael

On Sun, Dec 11, 2011 at 2:19 PM, Rafael Ávila de Espíndola <
rafael.espindola at gmail.com> wrote:
> On 08/12/11 03:56 PM, Ivan Krasin wrote:
> > Hi llvm team!
> >
> > I'm currently working on the extended version of llvm-ld, which
has an
> > ability to check if all the symbols present (and fail if some symbols
> > are not resolved), treat archives in the right way (link all the
object
> > files in the archive if it's specified as the regular input, not
as -l)
>
> Is that the "right way"? I think all native linker require a
> --whole-archive or similar to do that, no?
>
My mistake. Please, read the phrase as "support -whole-archive".

>
> > and the most important to my project feature: to link against bitcode
> > "shared objects". The semantics is pretty simple. The module
treated as
> > shared object is only used as the source of symbol names which be
> > excluded from the list unresolved symbols.
>
> In summary, linking foo.bc and bar.bcso (do you use a different
> extension to differentiate from a "static" bc?) would be
equivalent (but
> faster, easier) to
>
> $ llc bar.bc -filetype=obj -o bar.o
> $ clang -shared -o bar.so bar.o
> $ clang -use-gold-plugin foo.o bar.so -o t
>
> Is that correct? In particular, "lld t" should show a dependency
on bar.
> Any particular reason for not adding this to the plugin api?
>
The result is a native .so here. My goal is to have a bitcode result.

gold plugin with mods supports that as well, but it would be nice to avoid
dependency on gold and gold plugin to simplify things.

>
> > I feel like the similar functionality might be useful for other
projects
> > as well. For example, Emscripten (as of 3 months ago, when I have
> > checked it) does not check for undefined symbols at link time, because
> > it does not have a way to link against glibc (the symbols from which
> > would be added at runtime as usual javascript functions). This adds
> > additional overhead to the developer who ports the software to
> > javascript using Emscripten.
>
> Is that the same case? It seems that it would be easier to just include
> a stub ELF libc.so with the symbols from libc that Emscripten supports.
>
I believe that Emscripten does not use gold and gold plugin at the moment.
They use llvm-link:
https://github.com/kripken/emscripten/wiki/Building-Projects

I fully agree that all the features I want from the bitcode linker may be
achieved with gold + gold plugin (with mods to both parts), but I would
like to stay away from this dependency. Partly, because maintaining
modified gold and gold plugin is no fun, partly because gold + gold plugin
is hardly broken under cygwin. See
http://code.google.com/p/nativeclient/issues/detail?id=2286 for more
details. In short, it's because cygwin does not fully support dlopen.
https://www.google.com/search?sourceid=chrome&ie=UTF-8&q=cygwin+dlopen

It may be fixed by linking gold plugin to gold statically, like here:
http://codereview.chromium.org/8713008/ but it only adds the complexity.
>
> > Any objections? Comments?
> >
> > Ivan Krasin
>
> Cheers,
> Rafael
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20111212/86621978/attachment.html>