llvm dev - Jun 2020 - RFC: Contributing llvm-libtool (LLVM version of Apple's libtool)

We're planning to contribute an LLVM version of Apple's libtool utility.
To quote its man page, "Libtool with -static is intended to replace ar(5)
and ranlib". In other words, libtool is the preferred tool to create
archives for Apple platforms, and it also has good defaults for that platform
(e.g. automatically creating a table of contents with the right format). It
provides a convenient interface for developers, and we have a lot of usage of it
(instead of ar) for that reason in various builds. In particular, it handles
universal and archive input files well; for universal input files, you can
either specify a single architecture or have libtool create a universal output
file, and for archive input files, the individual objects in the input archive
are added to the output archive. LLVM's build system uses libtool on Apple
platforms to handle universal binaries correctly, for example.
 
To be clear, we're *not* planning to contribute an equivalent to GNU
libtool, which has the same name but (as far as I can tell) a very different
scope. Any naming ideas to avoid confusion on this front would be appreciated.
 
One design point we'd appreciate input on is how to structure this new tool.
Apple's libtool supports creating either static or dynamic libraries. It
produces static libraries directly, but for dynamic libraries, it calls the
linker. Our initial focus and contribution will be the functionality for static
libraries, but we want our design to take the possibility of the dynamic library
generation being added later into account. For this reason, we're planning
to create llvm-libtool as a standalone tool instead of as a frontend for
llvm-ar. (llvm-ranlib is an example of a frontend for llvm-ar, where it's
the same underlying binary which just does different option parsing depending on
how it's invoked.) Much of the archive writing logic that llvm-ar uses is
already factored out to libObject (the writeArchive function), and we can factor
more of it out as needed, so we shouldn't end up with too much duplicated
code between llvm-ar and llvm-libtool. At the same time, making llvm-libtool its
own tool gives us the freedom to e.g. factor out parts of llvm-lipo into a
library to handle universal inputs and outputs directly in llvm-libtool rather
than having to invoke lipo separately, handle dynamic library generation in a
natural way (whereas it would be odd for llvm-ar to contain code for generating
dynamic libraries if we were to make llvm-libtool a frontend for llvm-ar, since
llvm-ar is a tool specifically for creating archives), and potentially integrate
with LLD for Mach-O as a library in the future instead of spawning the linker
separately. Are there any concerns with this plan?

On Jun 9, 2020, at 4:05 PM, Shoaib Meenai via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> We're planning to contribute an LLVM version of Apple's libtool
utility. To quote its man page, "Libtool with -static is intended to
replace ar(5) and ranlib". In other words, libtool is the preferred tool to
create archives for Apple platforms, and it also has good defaults for that
platform (e.g. automatically creating a table of contents with the right
format). It provides a convenient interface for developers, and we have a lot of
usage of it (instead of ar) for that reason in various builds. In particular, it
handles universal and archive input files well; for universal input files, you
can either specify a single architecture or have libtool create a universal
output file, and for archive input files, the individual objects in the input
archive are added to the output archive. LLVM's build system uses libtool on
Apple platforms to handle universal binaries correctly, for example.
Nice!
> To be clear, we're *not* planning to contribute an equivalent to GNU
libtool, which has the same name but (as far as I can tell) a very different
scope. Any naming ideas to avoid confusion on this front would be appreciated.
I think that this will be a common point of confusion.  How is this handled in
the lld space?

If you need a specific name, I’d recommend either naming this something like
llvm-libtool-macho or llvm-libtool-darwin.

An alternative is to plan for the tool to eventually grow to support the
features of GNU libtool, allowing a contributor who may become interested in
that to work on it within the same code base.
> One design point we'd appreciate input on is how to structure this new
tool. Apple's libtool supports creating either static or dynamic libraries.
It produces static libraries directly, but for dynamic libraries, it calls the
linker. Our initial focus and contribution will be the functionality for static
libraries, but we want our design to take the possibility of the dynamic library
generation being added later into account. For this reason, we're planning
to create llvm-libtool as a standalone tool instead of as a frontend for
llvm-ar. (llvm-ranlib is an example of a frontend for llvm-ar, where it's
the same underlying binary which just does different option parsing depending on
how it's invoked.)
Yes, I agree.  LLVM’s library based design makes this pretty natural.
> Much of the archive writing logic that llvm-ar uses is already factored out
to libObject (the writeArchive function), and we can factor more of it out as
needed, so we shouldn't end up with too much duplicated code between llvm-ar
and llvm-libtool. At the same time, making llvm-libtool its own tool gives us
the freedom to e.g. factor out parts of llvm-lipo into a library to handle
universal inputs and outputs directly in llvm-libtool rather than having to
invoke lipo separately, handle dynamic library generation in a natural way
(whereas it would be odd for llvm-ar to contain code for generating dynamic
libraries if we were to make llvm-libtool a frontend for llvm-ar, since llvm-ar
is a tool specifically for creating archives), and potentially integrate with
LLD for Mach-O as a library in the future instead of spawning the linker
separately. Are there any concerns with this plan?
+1 from me.

-Chris

(replies inline, as best as Outlook can manage ... the parts I'm replying to
should be indented)

On 6/10/20, 11:13 AM, "Chris Lattner" <clattner at nondot.org>
wrote:

    On Jun 9, 2020, at 4:05 PM, Shoaib Meenai via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
    > We're planning to contribute an LLVM version of Apple's libtool
utility. To quote its man page, "Libtool with -static is intended to
replace ar(5) and ranlib". In other words, libtool is the preferred tool to
create archives for Apple platforms, and it also has good defaults for that
platform (e.g. automatically creating a table of contents with the right
format). It provides a convenient interface for developers, and we have a lot of
usage of it (instead of ar) for that reason in various builds. In particular, it
handles universal and archive input files well; for universal input files, you
can either specify a single architecture or have libtool create a universal
output file, and for archive input files, the individual objects in the input
archive are added to the output archive. LLVM's build system uses libtool on
Apple platforms to handle universal binaries correctly, for example.

    Nice!

Thank you!

    > To be clear, we're *not* planning to contribute an equivalent to
GNU libtool, which has the same name but (as far as I can tell) a very different
scope. Any naming ideas to avoid confusion on this front would be appreciated.

    I think that this will be a common point of confusion.  How is this handled
in the lld space?

For the linker executables, the MSVC linker is called link.exe, so the Windows
port of LLD was just named lld-link. The ELF and Darwin system linkers are both
called "ld", but Apple's linker is named ld64, so the ELF port
became ld.lld and the Mach-O port became ld64.lld.

For the source code layout, the lld directory has subdirectories for the
different ports (COFF, ELF, Mach-O, WASM).

    If you need a specific name, I’d recommend either naming this something like
llvm-libtool-macho or llvm-libtool-darwin.

Yeah, those make sense. We already have an option in the build system for
optionally creating symlinks for tool names (e.g. if you want an objdump symlink
that points to llvm-objdump), so we can just add the option to create a
"libtool" symlink for people who want a drop-in replacement. I like
the llvm-libtool-darwin name better, since libObject already refers to the
specific archive format as "Darwin", but that's not a strong
preference.

    An alternative is to plan for the tool to eventually grow to support the
features of GNU libtool, allowing a contributor who may become interested in
that to work on it within the same code base.

Unfortunately, GNU libtool appears to have a significantly different scope and
interface, to the point where I'm not sure how much code you'd actually
be able to share in the libtool code base itself (versus just making use of the
same libObject primitives). GNU libtool supports compilation, installation and
uninstallation, and program execution with library paths set (e.g. for running
gdb), in addition to just creating static and dynamic libraries. On the other
hand, universal libraries are a Darwin-specific concept which GNU libtool has no
notion of.

We could take the approach used by LLD and llvm-objcopy, where we create a
directory for the tool (llvm/tools/llvm-libtool in this case), and have
subdirectories for the ports of the tool (Darwin in our case, and someone
interested could create a GNU subdirectory later). That way there's a
natural place to put any code that's common between the ports.

    > One design point we'd appreciate input on is how to structure this
new tool. Apple's libtool supports creating either static or dynamic
libraries. It produces static libraries directly, but for dynamic libraries, it
calls the linker. Our initial focus and contribution will be the functionality
for static libraries, but we want our design to take the possibility of the
dynamic library generation being added later into account. For this reason,
we're planning to create llvm-libtool as a standalone tool instead of as a
frontend for llvm-ar. (llvm-ranlib is an example of a frontend for llvm-ar,
where it's the same underlying binary which just does different option
parsing depending on how it's invoked.)

    Yes, I agree.  LLVM’s library based design makes this pretty natural.

    > Much of the archive writing logic that llvm-ar uses is already factored
out to libObject (the writeArchive function), and we can factor more of it out
as needed, so we shouldn't end up with too much duplicated code between
llvm-ar and llvm-libtool. At the same time, making llvm-libtool its own tool
gives us the freedom to e.g. factor out parts of llvm-lipo into a library to
handle universal inputs and outputs directly in llvm-libtool rather than having
to invoke lipo separately, handle dynamic library generation in a natural way
(whereas it would be odd for llvm-ar to contain code for generating dynamic
libraries if we were to make llvm-libtool a frontend for llvm-ar, since llvm-ar
is a tool specifically for creating archives), and potentially integrate with
LLD for Mach-O as a library in the future instead of spawning the linker
separately. Are there any concerns with this plan?

    +1 from me.

Great, thanks!

    -Chris

On Tue, Jun 9, 2020 at 5:06 PM Shoaib Meenai via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> and potentially integrate with LLD for Mach-O as a library in the future
> instead of spawning the linker separately. Are there any concerns with this
> plan?
>
Just note that in general LLD is quite hostile to being used as a library,
although this specific use case (only invoking it once for Mach-O) may not
run into any issues.

  - Michael Spencer
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20200615/8fb45b2e/attachment.html>