llvm dev - Dec 2008 - [LLVMdev] Multiple directories in a single library

Hi all,

I've previously posted this patch on llvm-commits, but due to a lack of
replies, and the fact that this patch is more something to discuss than
something to apply, I'm posting this again here.

While working on my own backend, I found that things got really messy real
quickly, partly caused by the fact that all .cpp files must be in the same
directory (lib/Target/TargetName). Simply putting code in different
directories and using DIRS in the Makefile doesn't cut it, since that
produces
different libraries for each directory, which don't get linked in by the
programs that need them.

There is some code in llvm-config which fixes the library problem for the
TargetNameAsmPrinter library, but that isn't really the clean way IMHO. Is
this also meant for cleanup, or is there another reason why the AsmPrinter
should be in a seperate library?

Anyway, I thought it would be good if one could simply compile .cpp files in
subdirectories as if they are in the current directory, without a makefile in
the subdirectory. These cpp files would end up in the library as defined by
the parent directory's Makefile.

Is there any interest in this, or am I the only one who would like to have
something like this?

I've attached a small patch which facilitates this. It allows you to define
the EXTRA_SOURCES variable to explicitely add extra source files (one could
also explicitly define SOURCES, but this way you don't loose the default
all-sources-in-the-current-dir feature). For example, I would add:
	EXTRA_SOURCES = $(wildcard $(PROJ_SRC_DIR)/SubDir/*.cpp)

to get all the cpp files in SubDir compiled as well.

It would be more elegant to specify EXTRA_DIRS instead (or perhaps a better
name) as a list of directories, and let Makefile.rules handle the wildcarding
here. However, I've not found a way to write
.for DIR in $(EXTRA_DIRS)
	SOURCES += $(patsubst $(PROJ_SRC_DIR)/%, %, wildcard
$(PROJ_SRC_DIR)/$DIR/*.cpp))
.endfor

in the Makefile (the above appears to be valid in NetBSD make, any suggestions
on how to do this in GNU make?) Doing this in shell seems tricky, because then
the wildcarding must be done in shell as well AFAICS.

The second change in the patch makes sure that the proper directory is created
below $(ObjDir). My first attempt was to make the build rules depend on
$(ObjDir)/SubDir/.dir but this didn't quite work out (I didn't manage to
get
just "SubDir" from the filename in a clean way, and make threw away
the .dir
files afterwards, since they were intermediate targets).

The alternative in the patch is to just directly mkdir in the build target,
which works.

Gr.

Matthijs
-------------- next part --------------
Index: Makefile.rules
==================================================================---
Makefile.rules	(revision 59957)
+++ Makefile.rules	(working copy)
@@ -541,6 +541,10 @@
   Sources := $(SOURCES)
 endif 
 
+ifdef EXTRA_SOURCES
+  Sources += $(patsubst $(PROJ_SRC_DIR)/%, %, $(EXTRA_SOURCES))
+endif
+
 ifdef BUILT_SOURCES
 Sources += $(filter %.cpp %.c %.cc %.y %.l,$(BUILT_SOURCES))
 endif
@@ -1123,16 +1127,19 @@
                   else $(RM) "$(ObjDir)/$*.d.tmp"; exit 1; fi
 
 $(ObjDir)/%.lo $(ObjDir)/%.o: %.cpp $(ObjDir)/.dir $(BUILT_SOURCES)
+	$(Verb) $(MKDIR) $(dir $@) > /dev/null
 	$(Echo) "Compiling $*.cpp for $(BuildMode) build " $(PIC_FLAG)
 	$(Verb) if $(MAYBE_PIC_Compile.CXX) $(DEPEND_OPTIONS) $< -o $(ObjDir)/$*.o
; \
 	        $(DEPEND_MOVEFILE)
 
 $(ObjDir)/%.lo $(ObjDir)/%.o: %.cc $(ObjDir)/.dir $(BUILT_SOURCES)
+	$(Verb) $(MKDIR) $(dir $@) > /dev/null
 	$(Echo) "Compiling $*.cc for $(BuildMode) build" $(PIC_FLAG)
 	$(Verb) if $(MAYBE_PIC_Compile.CXX) $(DEPEND_OPTIONS) $< -o $(ObjDir)/$*.o
; \
 	        $(DEPEND_MOVEFILE)
 
 $(ObjDir)/%.lo $(ObjDir)/%.o: %.c $(ObjDir)/.dir $(BUILT_SOURCES)
+	$(Verb) $(MKDIR) $(dir $@) > /dev/null
 	$(Echo) "Compiling $*.c for $(BuildMode) build" $(PIC_FLAG)
 	$(Verb) if $(MAYBE_PIC_Compile.C) $(DEPEND_OPTIONS) $< -o $(ObjDir)/$*.o ;
\
 	        $(DEPEND_MOVEFILE)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20081201/fc6c511f/attachment.sig>

On 1 Dec 2008, at 11:42, Matthijs Kooijman wrote:
> While working on my own backend, I found that things got really  
> messy real
> quickly, partly caused by the fact that all .cpp files must be in  
> the same
> directory (lib/Target/TargetName). Simply putting code in different
> directories and using DIRS in the Makefile doesn't cut it, since  
> that produces
> different libraries for each directory, which don't get linked in by  
> the
> programs that need them.
For what it's worth, I have lib/Target compiling into one library. It  
works for building dynamic libraries, but I'm having some trouble with  
getting static archive builds to load targets. All subdirectories are  
included, identical base names should be handled gracefully, and  
compilation in a subdirectory will have all parent (sub)directories in  
their header search path.

A few examples of how I did it:

<http://www.bitbucket.org/danchr/llvm/src/253dfa2dabed/lib/Target/Makefile#cl-10
 >
<http://www.bitbucket.org/danchr/llvm/src/253dfa2dabed/Makefile.config.in#cl-82
 >
<http://www.bitbucket.org/danchr/llvm/src/253dfa2dabed/Makefile.rules#cl-532 
 >
<http://www.bitbucket.org/danchr/llvm/src/253dfa2dabed/Makefile.rules#cl-589 
 >
<http://www.bitbucket.org/danchr/llvm/src/253dfa2dabed/Makefile.rules#cl-607 
 >

(The server seems rather slow when querying individual files; sorry…)

--

Dan Villiom Podlaski Christiansen, stud. scient.
danchr at gmail.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20081201/b695b99a/attachment.html>

Hello, Matthijs
> There is some code in llvm-config which fixes the library problem for the
> TargetNameAsmPrinter library, but that isn't really the clean way IMHO.
Is
> this also meant for cleanup, or is there another reason why the AsmPrinter
> should be in a seperate library?
The main reason of such split was codesize concerns for JIT users:
they don't need asmprinting at all. The ugly include paths hacks were
due to current lack of possibility to "export private headers".
> Anyway, I thought it would be good if one could simply compile .cpp files
in
> subdirectories as if they are in the current directory, without a makefile
in
> the subdirectory. These cpp files would end up in the library as defined by
> the parent directory's Makefile.
Then, this again "merges" asmprinters into whole target library,
right?

-- 
With best regards, Anton Korobeynikov
Faculty of Mathematics and Mechanics, Saint Petersburg State University

Hi Anton,
> The main reason of such split was codesize concerns for JIT users:
> they don't need asmprinting at all. The ugly include paths hacks were
> due to current lack of possibility to "export private headers".
Ah, I suspected as much.
> > Anyway, I thought it would be good if one could simply compile .cpp
files in
> > subdirectories as if they are in the current directory, without a
makefile in
> > the subdirectory. These cpp files would end up in the library as
defined by
> > the parent directory's Makefile.
> Then, this again "merges" asmprinters into whole target library,
right?
If you would use the EXTRA_SOURCES for the AsmPrinter dir, then yes. But there
is nothing to stop you from keeping AsmPrinter in DIRS with its own makefile,
and putting other dirs in EXTRA_SOURCES.

Gr.

Matthijs
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20081201/a66af5e4/attachment.sig>