llvm dev - Sep 2013 - [LLVMdev] [lld] Modeling ELF FileNodes/ControlNodes (Group's) in lld

Hi Nick,

These are the below modifications needed in lld to start processing 
groups :-

1) LinkerInput would be moved to FileNode that contains the following 
functions
     -  getBuffer
      - takeBuffer
      - getPath

2) The driver will process the vector of InputElements and call 
/*process */on each of them.
      process() would create a lld::File object within the InputElement 
if its a FileNode
      process() would create a vector of lld::File objects within the 
InputElement if its a ControlNode

3) The resolver will not process each file but it would process 
InputElements and would do the following :-
     3.1 ) if the InputElement corresponds to a fileNode, the resolver 
will call in
             inputElement->processAtoms(*this) and marks the 
InputElement as already processed.
*This would essentially call resolver.processFile(_currentFile)*

      3.2) If the inputElement corresponds to a GroupNode, the resolver 
will call inputElement->processAtom(*this) on
*This* *would essentiall call  
resolver.processGroup(std::vector<Files>), the vector of files is what 
makes it in the current group*

4) InputFiles would be removed
5) LinkerInput would be removed
6) Functions that are unused forEachInitialAtom etc

_*Issues*_*

*This will essentially break the Darwin model thats in lld, as the 
current functionalities processes all the object files and only then 
processes
archive files.

Can we control by using a resolution policy ? which is a set of boolean 
flags to control the resolution behavior ? Do you have any other approach ?

_*Questions*_

1) There might be other types of control nodes, what would we need to 
do, can the resolver be controlled by each flavor ?

Thanks

Shankar Easwaran


On 9/4/2013 5:50 PM, Nick Kledzik wrote:
> On Sep 4, 2013, at 2:32 PM, Shankar Easwaran <shankare at
codeaurora.org> wrote:
>> On 9/4/2013 4:04 PM, Nick Kledzik wrote:
>>> I do think we have too many classes.
>> Agree.
>>>   I thought InputGraph was going to replace InputFiles.
>> Interesting idea.
>>>   It seems link LinkerInput could be merged into FileNode.
>> Agree.
>>> Originally InputFiles was the abstract interface that he Resolver
used to see all the inputs.  If InputGraph supported the methods 
forEachInitalAtom() and searchLibraries() then we could get rid of InputFiles
and have the Resolver uses InputGraph directly.
>> Yes, this would be nice. I will try to write a proposal in the next few
days.
>>> What should the interface be that the Resolver uses for handling
groups?
>> bool resolveUndefines(std::vector<File> &files) ?
>>
>> This has to iterate over the files until it reaches a stable point
(that no more resolution is possible).
> It is a little more complicated.  After initial .o files are processed,
there are a bunch of undefines left.  The resolver needs to start loading
archive members that fulfill those undefines, but loading a member can introduce
even more undefines, so it has to iterate.
>
> I think the existing searchLibraries(StringRef name, bool options..)
interface almost works for InputGraph.  To support groups we just need to keep
track of where in the InputGraph we currently are, so that searchLibraries() can
spin in a group until no more undefs are fulfilled, then move on in the graph.
>
> -Nick
>
>
>>> On Sep 4, 2013, at 1:42 PM, Shankar Easwaran <shankare at
codeaurora.org> wrote:
>>>
>>>> Hi,
>>>>
>>>> With the inputGraph now, lld models command line options, input
files as nodes in the InputGraph called InputElements.
>>>>
>>>> In the current approach, each InputElement is converted to a
LinkerInput, which works if all lld deals with individual files.
>>>>
>>>> Dealing with ControlNodes (Groups), have a problem with it, on
how to model that as a LinkerInput.
>>>>
>>>> Joerg/Me were chatting on the IRC about this and we came up
with the following approach.
>>>>
>>>> - LinkerInput will contain a single file(lld::File), if the
node that its pointing to appears to be a FileNode
>>>> - LinkerInput will contain a vector(lld::Group) of
files(lld::Files) , if the node that its pointing appears to be a Group
>>>>
>>>> The resolver would need to be modified to consider lld::Groups
in addition to lld::File.
>>>>
>>>> Does this sound like the approach we want to take ?
>>>>
>>>> Thanks
>>>>
>>>> Shankar Easwaran
>>>>
>>>> -- 
>>>> Qualcomm Innovation Center, Inc. is a member of Code Aurora
Forum, hosted by the Linux Foundation
>>>
>>
>> -- 
>> Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
hosted by the Linux Foundation
>
>

-- 
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by the
Linux Foundation

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20130904/6469ffc9/attachment.html>

On Sep 4, 2013, at 9:28 PM, Shankar Easwaran <shankare at codeaurora.org>
wrote:
> Hi Nick,
> 
> These are the below modifications needed in lld to start processing groups
:-
> 
> 1) LinkerInput would be moved to FileNode that contains the following
functions
>     -  getBuffer
>      - takeBuffer
>      - getPath
> 
> 2) The driver will process the vector of InputElements and call process on
each of them.
>      process() would create a lld::File object within the InputElement if
its a FileNode
>      process() would create a vector of lld::File objects within the
InputElement if its a ControlNode
> 
> 3) The resolver will not process each file but it would process
InputElements and would do the following :-
>     3.1 ) if the InputElement corresponds to a fileNode, the resolver will
call in
>             inputElement->processAtoms(*this) and marks the InputElement
as already processed.
>             This would essentially call resolver.processFile(_currentFile)
> 
>      3.2) If the inputElement corresponds to a GroupNode, the resolver will
call inputElement->processAtom(*this) on
>             This would essentiall call 
resolver.processGroup(std::vector<Files>), the vector of files is what
makes it in the current group
> 
> 4) InputFiles would be removed
> 5) LinkerInput would be removed
> 6) Functions that are unused forEachInitialAtom etc
> 
> Issues
> 
> This will essentially break the Darwin model thats in lld, as the current
functionalities processes all the object files and only then processes
> archive files. 
> 
> Can we control by using a resolution policy ? which is a set of boolean
flags to control the resolution behavior ? Do you have any other approach ?
The way darwin works with the current scheme is that the files are added to
InputFiles in command line order, then forEachInitialAtom() walks the whole list
but only operates on the non-library (i.e. object files) and searchLibraries()
only operates on library files.

If we have the Resolver walk the graph, then either:
1) We need some option for darwin and gnu to work differently, or 
2) Have the darwin driver construct the graph with all libraries in one group at
the end.  As long as that is straight forward to do, it seems simpler and also
captures the darwin difference that all libraries are always repeatedly
searched.

-Nick

> 
> Questions
> 
> 1) There might be other types of control nodes, what would we need to do,
can the resolver be controlled by each flavor ?
> 
> Thanks
> 
> Shankar Easwaran
> 
> 
> On 9/4/2013 5:50 PM, Nick Kledzik wrote:
>> On Sep 4, 2013, at 2:32 PM, Shankar Easwaran <shankare at
codeaurora.org> wrote:
>>> On 9/4/2013 4:04 PM, Nick Kledzik wrote:
>>>> I do think we have too many classes.
>>> Agree.
>>>>  I thought InputGraph was going to replace InputFiles.
>>> Interesting idea.
>>>>  It seems link LinkerInput could be merged into FileNode.
>>> Agree.
>>>> Originally InputFiles was the abstract interface that he
Resolver used to see all the inputs.  If InputGraph supported the methods 
forEachInitalAtom() and searchLibraries() then we could get rid of InputFiles
and have the Resolver uses InputGraph directly.
>>> Yes, this would be nice. I will try to write a proposal in the next
few days.
>>>> What should the interface be that the Resolver uses for
handling groups?
>>> bool resolveUndefines(std::vector<File> &files) ?
>>> 
>>> This has to iterate over the files until it reaches a stable point
(that no more resolution is possible).
>> It is a little more complicated.  After initial .o files are processed,
there are a bunch of undefines left.  The resolver needs to start loading
archive members that fulfill those undefines, but loading a member can introduce
even more undefines, so it has to iterate.
>> 
>> I think the existing searchLibraries(StringRef name, bool options..)
interface almost works for InputGraph.  To support groups we just need to keep
track of where in the InputGraph we currently are, so that searchLibraries() can
spin in a group until no more undefs are fulfilled, then move on in the graph.
>> 
>> -Nick
>> 
>> 
>>>> On Sep 4, 2013, at 1:42 PM, Shankar Easwaran <shankare at
codeaurora.org> wrote:
>>>> 
>>>>> Hi,
>>>>> 
>>>>> With the inputGraph now, lld models command line options,
input files as nodes in the InputGraph called InputElements.
>>>>> 
>>>>> In the current approach, each InputElement is converted to
a LinkerInput, which works if all lld deals with individual files.
>>>>> 
>>>>> Dealing with ControlNodes (Groups), have a problem with it,
on how to model that as a LinkerInput.
>>>>> 
>>>>> Joerg/Me were chatting on the IRC about this and we came up
with the following approach.
>>>>> 
>>>>> - LinkerInput will contain a single file(lld::File), if the
node that its pointing to appears to be a FileNode
>>>>> - LinkerInput will contain a vector(lld::Group) of
files(lld::Files) , if the node that its pointing appears to be a Group
>>>>> 
>>>>> The resolver would need to be modified to consider
lld::Groups in addition to lld::File.
>>>>> 
>>>>> Does this sound like the approach we want to take ?
>>>>> 
>>>>> Thanks
>>>>> 
>>>>> Shankar Easwaran
>>>>> 
>>>>> -- 
>>>>> Qualcomm Innovation Center, Inc. is a member of Code Aurora
Forum, hosted by the Linux Foundation
>>> -- 
>>> Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
hosted by the Linux Foundation
>> 
> 
> 
> -- 
> Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted
by the Linux Foundation
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20130905/53e4e87c/attachment.html>

On 9/5/2013 1:40 PM, Nick Kledzik wrote:
> The way darwin works with the current scheme is that the files are added to
InputFiles in command line order, then forEachInitialAtom() walks the whole list
but only operates on the non-library (i.e. object files) and searchLibraries()
only operates on library files.
>
> If we have the Resolver walk the graph, then either:
> 1) We need some option for darwin and gnu to work differently, or
> 2) Have the darwin driver construct the graph with all libraries in one
group at the end.  As long as that is straight forward to do, it seems simpler
and also captures the darwin difference that all libraries are always repeatedly
searched.
>
> -Nick
One way, I can think of this being done is to get all the libraries that 
are specified in the command line and create a group node, and add it 
after the command line is processed to the inputGraph.

PS : Does Darwin have a command line option to handle files in the order 
specified in the command line for libraries ? If so, then the above 
would break

I am planning to support other operations too like

inputElement->setPosition(InputGraph::Top)
inputElement->setPosition(InputGraph::Last)
inputElement->setPosition(InputGraph::Position, <value>)

The user has to explicitly call a seperate api so that the elements are 
ordered according to whatever the user wants.

inputGraph->insertInputElementAt(InputGraph::Top, 
std::vector<std::unique_ptr<InputElement>>&)
inputGraph->insertInputElementAt(InputGraph::Last, 
std::vector<std::unique_ptr<InputElement>>&)
inputGraph->insertInputElementAt(InputGraph::Position, element, 
std::vector<std::unique_ptr<InputElement>>&)

The user has to explicitly call a seperate api so that the elements for 
ordinals to be set appropriately.

Thanks

Shankar Easwaran

-- 
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by the
Linux Foundation