llvm dev - Dec 2013 - [LLVMdev] [LLVM] What has happened to LLVM bitcode archive support?

Hi Rafael and other LLVM devs,

I'm currently upgrading a project that uses LLVM that links a bitcode
archive (a C library) with a module. Originally we used
Linker::LinkInFile() but that was removed by r172749. So I started
looking for an alternative and I found
Archive::findModulesDefiningSymbols() which looked very promising as
it would allow me to very easily implement linking a bitcode archive
with a module efficiently (i.e. only call Linker::LinkModules() on the
relevant bitcode modules inside the bitcode archive).

Before I started I thought I'd better check that this API was still
going to be available in the upcoming 3.4 release and to my dismay
``llvm/Bitcode/Archive.h`` was moved by you in r184083 and finally
removed by you in r186197.

So...

* Is LLVM completely moving away from the idea have bitcode archive
runtime libraries? The LLVM Make build system seems to support
building these but LLVM doesn't seem to have an API for linking
runtime bitcode archives into a bitcode module any more (or have I
missed something?). I have seen suggestions of using the LLVM gold
plug-in but that's not an API I can use, is it?

* I'm guessing I'm supposed to use the Object/Archive.h interface now?
This doesn't seem provide a nice interface for getting the set of
Modules I want to link with, so what am I supposed to do instead?

Any help would be appreciated,

Thanks,
Dan Liew.

On 20 December 2013 05:39, Daniel Liew <daniel.liew at imperial.ac.uk>
wrote:
> Hi Rafael and other LLVM devs,
>
> I'm currently upgrading a project that uses LLVM that links a bitcode
> archive (a C library) with a module. Originally we used
> Linker::LinkInFile() but that was removed by r172749. So I started
> looking for an alternative and I found
> Archive::findModulesDefiningSymbols() which looked very promising as
> it would allow me to very easily implement linking a bitcode archive
> with a module efficiently (i.e. only call Linker::LinkModules() on the
> relevant bitcode modules inside the bitcode archive).
>
> Before I started I thought I'd better check that this API was still
> going to be available in the upcoming 3.4 release and to my dismay
> ``llvm/Bitcode/Archive.h`` was moved by you in r184083 and finally
> removed by you in r186197.
>
> So...
>
> * Is LLVM completely moving away from the idea have bitcode archive
> runtime libraries? The LLVM Make build system seems to support
> building these but LLVM doesn't seem to have an API for linking
> runtime bitcode archives into a bitcode module any more (or have I
> missed something?). I have seen suggestions of using the LLVM gold
> plug-in but that's not an API I can use, is it?
>
> * I'm guessing I'm supposed to use the Object/Archive.h interface
now?
> This doesn't seem provide a nice interface for getting the set of
> Modules I want to link with, so what am I supposed to do instead?
We are going this way, yes. The overview is

* In the old days llvm had its own archive index format for IL files
and special code to use it.
* llvm-ld was very incomplete, and not designed to grow into real
linker, so an alternative for LTO was needed.
* apple implemented libLTO and changed their linker to use it. A
wrapper (LLVMgold.so) was created for gold and bfd.
* Using this the gnu nm, ar and ld (and gold) were able to do LTO. In
particular, a IL files can be put in archives and show up in the
regular symbol table.
* llvm-ld was deprecated and then removed.
* with nothing able to read the old LLVM only symbol table, that was
also removed along with other archive related now dead code.

The more recent story is that llvm has grow support for object files,
so that llvm-ar is now a "real" (if immature) ar and lld is growing to
be a real linker. Currently llvm-ar is missing support for putting IL
files in the symbol table. The main reason is the desire to now get
this right this time, which required quiet a bit of refactoring. I
posted what I think is the last big patch for review (making the
Mangler not depend on target). With that in things should move fairly
quickly to have llvm-ar just work with IL files and with that
lib/Object should have the features you need.
> Any help would be appreciated,
>
> Thanks,
> Dan Liew.
Cheers,
Rafael

Hi Rafael,

On 20 Dec 2013, at 14:30, Rafael Espíndola wrote:
> On 20 December 2013 05:39, Daniel Liew <daniel.liew at
imperial.ac.uk> wrote:
>> Hi Rafael and other LLVM devs,
>> 
>> I'm currently upgrading a project that uses LLVM that links a
bitcode
>> archive (a C library) with a module. Originally we used
>> Linker::LinkInFile() but that was removed by r172749. So I started
>> looking for an alternative and I found
>> Archive::findModulesDefiningSymbols() which looked very promising as
>> it would allow me to very easily implement linking a bitcode archive
>> with a module efficiently (i.e. only call Linker::LinkModules() on the
>> relevant bitcode modules inside the bitcode archive).
>> 
>> Before I started I thought I'd better check that this API was still
>> going to be available in the upcoming 3.4 release and to my dismay
>> ``llvm/Bitcode/Archive.h`` was moved by you in r184083 and finally
>> removed by you in r186197.
>> 
>> So...
>> 
>> * Is LLVM completely moving away from the idea have bitcode archive
>> runtime libraries? The LLVM Make build system seems to support
>> building these but LLVM doesn't seem to have an API for linking
>> runtime bitcode archives into a bitcode module any more (or have I
>> missed something?). I have seen suggestions of using the LLVM gold
>> plug-in but that's not an API I can use, is it?
>> 
>> * I'm guessing I'm supposed to use the Object/Archive.h
interface now?
>> This doesn't seem provide a nice interface for getting the set of
>> Modules I want to link with, so what am I supposed to do instead?
> 
> We are going this way, yes. The overview is
> 
> * In the old days llvm had its own archive index format for IL files
> and special code to use it.
> * llvm-ld was very incomplete, and not designed to grow into real
> linker, so an alternative for LTO was needed.
> * apple implemented libLTO and changed their linker to use it. A
> wrapper (LLVMgold.so) was created for gold and bfd.
> * Using this the gnu nm, ar and ld (and gold) were able to do LTO. In
> particular, a IL files can be put in archives and show up in the
> regular symbol table.
> * llvm-ld was deprecated and then removed.
> * with nothing able to read the old LLVM only symbol table, that was
> also removed along with other archive related now dead code.
> 
> The more recent story is that llvm has grow support for object files,
> so that llvm-ar is now a "real" (if immature) ar and lld is
growing to
> be a real linker. Currently llvm-ar is missing support for putting IL
> files in the symbol table. The main reason is the desire to now get
> this right this time, which required quiet a bit of refactoring. I
> posted what I think is the last big patch for review (making the
> Mangler not depend on target). With that in things should move fairly
> quickly to have llvm-ar just work with IL files and with that
> lib/Object should have the features you need.
Could I just note (for the record) there is/are [at least one ;)] bare-metal
targets that have neither static nor dynamic linkers.

We have one such (out-of-tree) target and make use of llvm-link to link bitcode
- the linked bit code is translated directly into the executable.  We would,
however, welcome a solution that allowed the 'link' to be extended to
include bitcode archives.

However, it's probably a non-starter (project constraints) to implement a
static linker solely for this purpose.

cheers
Iain