llvm dev - Apr 2016 - lld: --start-lib and --end-lib

Rafael,

You seems to be the person who added --start-lib and --end-lib to gold
(*1), so I think you are the best person to ask about it. :)

What is the intended use case of the options? Do you think we should
support that in LLD?

If we want to support that, then I think we can think of two implementation
strategies:

 - Link object files between --start-lib and --end-lib normally and garbage
collect unused sections, or
 - create lazy symbols for object files between --start-lib and --end-lib
and handle just like archive files.

What do you think?

*1 https://sourceware.org/ml/binutils/2010-03/msg00233.html
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160405/b9707463/attachment.html>

I hacked up a patch to support --start-lib and --end-lib in the second way.

http://reviews.llvm.org/D18814

In this patch, I introduced a new notion, LazyObject. That is analogous to
Archive file type, but that works for a single object file instead of an
archive file. This patch is +160 and -26 lines, so I can say that it is
pretty simple.

This is not directly related to --start-lib and --end-lib, but LazyObject
may be useful for --gc-sections as well. This is the suggestion I got in
the EuroLLVM 2016 meeting (I'm sorry but I don't remember who his name),
but he suggested handling all object files as lazy object files for
--gc-sections. It means by default no object files are linked. The entry
point symbol would link one object file, and that triggers all used object
files to be instantiated. I think this is very interesting idea because we
don't need to do mark-sweep garbage collection to do --gc-sections;
instead, everything will be linked only when they are needed.

On Tue, Apr 5, 2016 at 7:51 PM, Rui Ueyama <ruiu at google.com> wrote:
> Rafael,
>
> You seems to be the person who added --start-lib and --end-lib to gold
> (*1), so I think you are the best person to ask about it. :)
>
> What is the intended use case of the options? Do you think we should
> support that in LLD?
>
> If we want to support that, then I think we can think of two
> implementation strategies:
>
>  - Link object files between --start-lib and --end-lib normally and
> garbage collect unused sections, or
>  - create lazy symbols for object files between --start-lib and --end-lib
> and handle just like archive files.
>
> What do you think?
>
> *1 https://sourceware.org/ml/binutils/2010-03/msg00233.html
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160405/6ebc026b/attachment.html>

I believe the intent is that "--start-lib foo1.o foo2.o --end-lib"
should
act exactly the same as if you had given it an archive "tmp.a" which
contained "foo1.o" and "foo2.o" in it.

That doesn't really seem related to gc-sections: either way, you still need
to have the normal logic to find the correct set of object files to
include, at least in order to get the correct set of global constructors
and such.

On Wed, Apr 6, 2016 at 12:07 AM, Rui Ueyama via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> I hacked up a patch to support --start-lib and --end-lib in the second way.
>
> http://reviews.llvm.org/D18814
>
> In this patch, I introduced a new notion, LazyObject. That is analogous to
> Archive file type, but that works for a single object file instead of an
> archive file. This patch is +160 and -26 lines, so I can say that it is
> pretty simple.
>
> This is not directly related to --start-lib and --end-lib, but LazyObject
> may be useful for --gc-sections as well. This is the suggestion I got in
> the EuroLLVM 2016 meeting (I'm sorry but I don't remember who his
name),
> but he suggested handling all object files as lazy object files for
> --gc-sections. It means by default no object files are linked. The entry
> point symbol would link one object file, and that triggers all used object
> files to be instantiated. I think this is very interesting idea because we
> don't need to do mark-sweep garbage collection to do --gc-sections;
> instead, everything will be linked only when they are needed.
>
> On Tue, Apr 5, 2016 at 7:51 PM, Rui Ueyama <ruiu at google.com>
wrote:
>
>> Rafael,
>>
>> You seems to be the person who added --start-lib and --end-lib to gold
>> (*1), so I think you are the best person to ask about it. :)
>>
>> What is the intended use case of the options? Do you think we should
>> support that in LLD?
>>
>> If we want to support that, then I think we can think of two
>> implementation strategies:
>>
>>  - Link object files between --start-lib and --end-lib normally and
>> garbage collect unused sections, or
>>  - create lazy symbols for object files between --start-lib and
--end-lib
>> and handle just like archive files.
>>
>> What do you think?
>>
>> *1 https://sourceware.org/ml/binutils/2010-03/msg00233.html
>>
>
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160406/257c52e8/attachment.html>

On 5 April 2016 at 22:51, Rui Ueyama <ruiu at google.com>
wrote:
> Rafael,
>
> You seems to be the person who added --start-lib and --end-lib to gold
(*1),
> so I think you are the best person to ask about it. :)
>
> What is the intended use case of the options? Do you think we should
support
> that in LLD?
It was initially created for cluster builds. Running ar is really fast
if using thin archives, but one still has to copy the files to one
machine to create the archive.

Using --start-lib/--end-lib solves that problem. I find it a neat
feature, but to the best of my knowledge to this day it is only used
within google.

> If we want to support that, then I think we can think of two implementation
> strategies:
>
>  - Link object files between --start-lib and --end-lib normally and garbage
> collect unused sections, or
>  - create lazy symbols for object files between --start-lib and --end-lib
> and handle just like archive files.
>
> What do you think?
You have to create lazy symbols. The semantics are not the same given
that some things (static constructors) are not gced.

Cheers,
Rafael

On 6 April 2016 at 00:07, Rui Ueyama <ruiu at google.com>
wrote:
> I hacked up a patch to support --start-lib and --end-lib in the second way.
>
> http://reviews.llvm.org/D18814
>
> In this patch, I introduced a new notion, LazyObject. That is analogous to
> Archive file type, but that works for a single object file instead of an
> archive file. This patch is +160 and -26 lines, so I can say that it is
> pretty simple.
The description sounds reasonable. Taking a look.

> This is not directly related to --start-lib and --end-lib, but LazyObject
> may be useful for --gc-sections as well. This is the suggestion I got in
the
> EuroLLVM 2016 meeting (I'm sorry but I don't remember who his
name), but he
> suggested handling all object files as lazy object files for --gc-sections.
> It means by default no object files are linked. The entry point symbol
would
> link one object file, and that triggers all used object files to be
> instantiated. I think this is very interesting idea because we don't
need to
> do mark-sweep garbage collection to do --gc-sections; instead, everything
> will be linked only when they are needed.
That is a very big change in semantics given static constructors. Even
llvm itself uses static constructors to register passes, so including
a file and then gcing is not the same an not gcing.

Cheers,
Rafael

On Tue, Apr 5, 2016 at 9:07 PM, Rui Ueyama via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> I hacked up a patch to support --start-lib and --end-lib in the second way.
>
> http://reviews.llvm.org/D18814
>
> In this patch, I introduced a new notion, LazyObject. That is analogous to
> Archive file type, but that works for a single object file instead of an
> archive file. This patch is +160 and -26 lines, so I can say that it is
> pretty simple.
>
Have you tried creating a thin archive in memory and loading it in the
normal way? That seems potentially simpler.

-- Sean Silva

>
> This is not directly related to --start-lib and --end-lib, but LazyObject
> may be useful for --gc-sections as well. This is the suggestion I got in
> the EuroLLVM 2016 meeting (I'm sorry but I don't remember who his
name),
> but he suggested handling all object files as lazy object files for
> --gc-sections. It means by default no object files are linked. The entry
> point symbol would link one object file, and that triggers all used object
> files to be instantiated. I think this is very interesting idea because we
> don't need to do mark-sweep garbage collection to do --gc-sections;
> instead, everything will be linked only when they are needed.
>
> On Tue, Apr 5, 2016 at 7:51 PM, Rui Ueyama <ruiu at google.com>
wrote:
>
>> Rafael,
>>
>> You seems to be the person who added --start-lib and --end-lib to gold
>> (*1), so I think you are the best person to ask about it. :)
>>
>> What is the intended use case of the options? Do you think we should
>> support that in LLD?
>>
>> If we want to support that, then I think we can think of two
>> implementation strategies:
>>
>>  - Link object files between --start-lib and --end-lib normally and
>> garbage collect unused sections, or
>>  - create lazy symbols for object files between --start-lib and
--end-lib
>> and handle just like archive files.
>>
>> What do you think?
>>
>> *1 https://sourceware.org/ml/binutils/2010-03/msg00233.html
>>
>
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160406/8588f669/attachment.html>