llvm dev - Apr 2008 - [LLVMdev] Representing the dependencies of a bitcode module

I've been using SCons as the build tool for my frontend application, and 
I'm getting to the point where it would be useful to create a custom 
scanner for my generated bitcode files so that SCons can do proper 
dependency analysis. At the moment, SCons has no way to know which 
source files a particular bitcode file depends on, so the only way to do 
a "correct" build is to rebuild everything. Mostly I've been
getting by
with doing "incorrect" builds, meaning that I only build the files
that
actually changed. However, this has tripped me up one or two times and 
I'd like to solve it.

SCons has the ability to write custom scanners in Python, but I'm 
thinking that it will be easier in the long run to do this work in C++. 
So the idea would be to write a command-line tool that would spit out 
the list of dependencies, and then write a Python wrapper to call it 
from SCons.

My code generator keeps a list of what source files were imported during 
compilation, and at the moment what it does is it creates an 
internal-linkage array of strings, one string per import. The 
command-line tool can then load the bitcode file and read the string 
array. Since the strings are internally linked, and since nothing else 
in the bitcode file refers to them, they ought to be dropped during 
optimization (I hope.)

However, I notice that getting access to the strings from within the 
command-line tool is a little complicated, since I have to decompose the 
various constant getElementPtr expressions in order to get at the actual 
string. I'm wondering if there's a better way to represent this 
information. Maybe using the debug API, or perhaps annotation intrinsics?

-- Talin

On Apr 28, 2008, at 8:21 PM, Talin wrote:
> However, I notice that getting access to the strings from within the
> command-line tool is a little complicated, since I have to decompose  
> the
> various constant getElementPtr expressions in order to get at the  
> actual
> string. I'm wondering if there's a better way to represent this
> information. Maybe using the debug API, or perhaps annotation  
> intrinsics?
There isn't a good interface to the annotation intrinsics yet, it  
would be nice to have them, then this could be encoded as annotations.

-Chris

On 2008-04-28, at 23:21, Talin wrote:
> I've been using SCons as the build tool for my frontend application,  
> and
> I'm getting to the point where it would be useful to create a custom
> scanner for my generated bitcode files so that SCons can do proper
> dependency analysis. At the moment, SCons has no way to know which
> source files a particular bitcode file depends on, so the only way  
> to do
> a "correct" build is to rebuild everything. Mostly I've been
getting
> by
> with doing "incorrect" builds, meaning that I only build the
files
> that
> actually changed. However, this has tripped me up one or two times and
> I'd like to solve it.
>
> SCons has the ability to write custom scanners in Python, but I'm
> thinking that it will be easier in the long run to do this work in C+ 
> +.
> So the idea would be to write a command-line tool that would spit out
> the list of dependencies, and then write a Python wrapper to call it
> from SCons.
>
> My code generator keeps a list of what source files were imported  
> during
> compilation, and at the moment what it does is it creates an
> internal-linkage array of strings, one string per import. The
> command-line tool can then load the bitcode file and read the string
> array. Since the strings are internally linked, and since nothing else
> in the bitcode file refers to them, they ought to be dropped during
> optimization (I hope.)
>
> However, I notice that getting access to the strings from within the
> command-line tool is a little complicated, since I have to decompose  
> the
> various constant getElementPtr expressions in order to get at the  
> actual
> string. I'm wondering if there's a better way to represent this
> information. Maybe using the debug API, or perhaps annotation  
> intrinsics?

For something equivalent to #include (where the dependencies are not  
fully specified on the command line), have you considered a solution  
like 'gcc -c -MF foo.d foo.c'? This spits out the dependencies into a  
file (foo.d) as a side-effect of compilation. The dependencies can be  
directly included into 'make' on subsequent runs. If the output is not  
present, it will necessarily be rebuilt. If the output is present,  
then sufficient dependencies will be listed in the file. (They may not  
be "up to date", but it doesn't matter if you think it through.)  
Granted that you're not using make, but the principal is perfectly  
sound.

For linkage command lines, lazily updating 'response files' (sorry,  
Windows terminology...) can provide a complete solution. Something like:

LINK_CMD := $(LINKER) $(LINK_INPUTS)

all: cmds

link_cmd:
	if [ "`cat $(INT_DIR)/link_cmd`" != "$(LINK_CMD)" ]; then
echo "$
(LINK_CMD)" > link_cmd; fi

output.so: link_cmd $(LINK_INPUTS)
	$(LINK_CMD)

Of course, these are complimentary depending on your needs. Using a  
response file for compilation could protect against changes to the  
include path which are not captured by the direct file dependencies.  
Using a depends file for a linker component could protect against  
files being added to the linker search path.

— Gordon

On 2008-04-29, at 00:11, Gordon Henriksen wrote:
> For linkage command lines, lazily updating 'response files' (sorry,
> Windows terminology...) can provide a complete solution. Something  
> like:
>
> LINK_CMD := $(LINKER) $(LINK_INPUTS)
>
> all: cmds
>
> link_cmd:
> 	if [ "`cat $(INT_DIR)/link_cmd`" != "$(LINK_CMD)" ];
then echo "$
> (LINK_CMD)" > link_cmd; fi
>
> output.so: link_cmd $(LINK_INPUTS)
> 	$(LINK_CMD)
Whoops, I didn't finish this example. Should be:

LINK_CMD := $(LINKER) $(LINK_INPUTS)

all: cmds output.so

cmds:
	[ "`cat $(INT_DIR)/link_cmd`" = "$(LINK_CMD)" ] || echo
"$(LINK_CMD)"
 > link_cmd

output.so: link_cmd $(LINK_INPUTS)
	$(LINK_CMD)

— Gordon

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20080429/4787d33e/attachment.html>