llvm dev - Oct 2011 - [LLVMdev] [cfe-dev] RFC: Upcoming Build System Changes

Hi all,

As you might have inferred, I'm in the process of working on some changes to
the
way LLVM builds. I have outlined a rough proposal below, unless there are any
major objections I will probably start committing stuff next week.

This message may be verbose, if you want the executive summary, skip
to 'What This
Means For Jane "LLVM Developer" Doe' at the bottom.

Motivation
----------

We currently maintain two build systems (make, CMake). This has a couple of
problems:

 * Duplication: the two build systems duplicate a significant amount of implicit
   logic (higher level dependencies), some config files, and other miscellaneous
   features.

 * Maintenance: Some parts of the build systems requires regular maintenance
   (the CMake file lists, CMake dependency lists, module structure). Having one
   more thing to maintain is annoying.

 * Inconsistency: The two build systems behave inconsistently in some ways. If
   we want both to officially supported systems, it would be nice for them to
   behave as identically as possible.

   For example, CMake now uses explicit dependencies which are hard coded into
   the CMake files, but the Makefiles work completely differently.

There are also other general issues with the way LLVM is built now:

 * LLVM has a higher level structure for its organization, but this is not
   explicit. LLVM is roughly organized into components (libraries, tools, etc.)
   by directory. It would be nice to have this be more explicit.

 * Much of the project build structure is implicit in the Makefiles or
   CMakeFiles. It is not particularly explicit anywhere that, say, parts of
   VMCore depend on building the Intrinsics, which depend on building tblgen.

 * The Make system is not very efficient. We use recursive Makefiles which make
   the current system (relatively) simple in some ways, but mean the Make build
   is not nearly as scalable as it could be. In particular, the current
   organization means the built is often serialized on something that is not a
   strict dependency. It also makes it much more likely to do things like a
   stampeding link of all the tools, even though many tools could have been
   built earlier.


Specific Goals
--------------

 * Move both build systems to use explicit library dependencies, in a clean
   fashion. The CMake files do this now, but I don't think it has been made
   clear to developers when they are supposed to edit these files, or how (other
   than when something breaks, I guess).

 * Explicitly describe as much of the project structure as necessary to support
   builds. This means explicitly specifying how the project is organized, and
   the dependencies among the components required to build (e.g., Intrinsics
   before VMCore).

   I believe a number of other projects/users (FreeBSD, fbuild) have
built there own
   build system for LLVM. Encoding the project structure clearly should make it
   easier for such projects, or for other future users that want to do the same.

   This should make it easier to experiment with the build system, for example
   we could just directly generate good Makefiles for our project, or could
   experiment with systems like Ninja which expect to be targetted by a
   generator of some kind.


Proposal
--------

My initial proposal is focused at moving us to use explicit library
dependencies, but paves the way for centralizing more "build systemy"
stuff in
the future.

 * Every LLVM "component" (roughly corresponds to each place we have a
Makefile
   or CMakeLists.txt currently) will get a 'LLVMBuild.txt' file.

   This file will be an LLVM specific description of that component. Initially,
   it will look something like this::

     [component]
     # The kind of component this is (currently library, tool, build tool). More
     # types will be defined over time.
     type = Library

     # The name of the component.
     name = VMCore

     # The name of the component to logically group this in. This is just for
     # organization purposes.
     parent = Libraries

     # The names of the library components that are also required when linking
     # with this library. More on this later.
     required_libraries = Support

   The exact structure of the format is TBD (and easy to change), currently the
   format is INI style but I may decide to change to JSON once all the pieces
   are in place.

   The LLVM web pages will have clear documentation on what these files should
   look like, what is required, what is supported, and so on.


 * I will add a new tool in utils/llvm-build which is designed to load and work
   with these files. This tool will be written in Python, and the expectation is
   that it can be run at configure time.

   TO BE CLEAR: I intend to introduce a hard dependency on requiring Python in
   order to build LLVM.

   For the Makefiles, this is no worse than requiring Perl, so I don't think
   there is much to argue with.

   For CMake, this is a new dependency. However, I feel this is unavoidable:

     * I see no way to support multiple build systems including CMake without
       either introducing a dependency on some extra tool (which can be shared),
       or duplicating a significant amount of logic in CMake.

       I believe that duplicating logic is worse than adding the Python
       dependency, and I think we already have more CMake code (a relatively
       horrible language) than can be expected for most LLVM developers to deal
       with.

   Additionally, we already require Python for running our tests, so anyone
   doing serious LLVM development should have it.

   The one use case I believe this particularly hurts is users who just want to
   download and play with LLVM, but I believe the Right Answer (tm) for that
   case would be for us to provide nice installer packages anyway.


 * utils/llvm-build will be run during configure time (both for Make and CMake),
   and will:

   * Generate the library dependency information required to link tools, in
     whatever format makes the most system for the build system in use.

   * Generate a C++ .inc file containing the dependency table for use by
     llvm-config (which I am going to rewrite in C++).

   * Add dependencies on the LLVMBuild.txt files to the build system, so that
     the build will reconfigure appropriately when the library
requirements change.


 * Remove GenLibDeps.pl, find-cycles.pl, etc.

   We will no longer be using these after llvm-config has moved over.


 * Add explicit source file lists to the LLVMBuild.txt files. Unfortunately,
   this is inevitable if we want to support CMake users projects automatically
   reconfiguring themselves when new files get added. I can make it easier to
   manage (for example, provide build targets that will automatically add any
   new files).


 * Move both Make and CMake over to using the explicit file lists in the
   LLVMBuild files. This ensures that both systems get the same behavior
   (instead of Make users being able to add files and forget to update
   CMakeLists.txt).


 * Add new 'lit' tests to check that the library dependency
information is correct.

   This seems a nicer place to do the checking which is currently partially
   handled by find-cycles, and we should also be able to do a better job of the
   checking (for example, verifying that the dependency list is
"exact" -- only
   specifies the minimum set of libraries required, and isn't allowed to
specify
   libraries which are only indirectly required).

   It would be particularly cool if we could just write these tests using our
   Object libraries.

   This is one piece I haven't prototyped yet. I can obviously do something
as
   good as the current find-cycles.pl, but I hope to do better (eventually).


 * These are just the first steps, after this I will continue to incrementally
   try and move as much common information out of the Make and CMake systems so
   there is "one source of truth" with regard to the project
definition.


 * I assume it is obvious, but when I say "LLVM" here I am referring
to both
   LLVM and Clang. I have not looked at lldb yet, but if it uses the LLVM build
   system (and llvm-config) functionality I will try to make sure it
continues to work.


What This Means For Jane "LLVM Developer" Doe
---------------------------------------------

In practice, this means:

 * LLVM requires Python to build.

 * When you add a file to LLVM, you will need to edit LLVMBuild.txt instead of
   CMakeLists.txt, which will be in a slightly different, but otherwise totally
   obvious format.

   If you forget to do this, your file will not be built (which will most likely
   cause a link error eventually). This is better than it being built by Make,
   but causing CMake build failures when you check in.

 * When you add a new library requirement to an existing component, you will be
   required to edit LLVMBuild.txt instead of CMakeLists.txt, which will be in a
   slightly different, but otherwise totally obvious (hopefully) format.

   If you forget to do this, you will either get a link error or a test
   failure. This is better than library you need transparently getting linked in
   (with make) because it forces you to think about whether you actually should
   be adding that dependency.

   The goal is that this also ensures that if LLVM links and passes tests on
   your system, then it should for everyone else as well.

 * Developers not actively touching the build system should never need to touch
   a Makefile or a CMake file.

Overall, I believe this should be a quality of life improvement for the
developer community. The only downside is having to deal with a new non-standard
LLVM specific format, but I plan to solve this through documentation.

Comments?

 - Daniel

I have a very high level comment, and you may be able to directly shed light
on it before I dig into a lot more detail.

Why not simply standardize on CMake? It's not my favorite tool, but it seems
to work well, we have established usage of it, and several people involved
in the project who understand how it works. It doesn't seem like a
significantly more burdensome dependency than Python when developing, and it
remains possible to build installable packages for the casual hacker.

I can see some objections to CMake, but it's not clear to me that they
should carry the day. I'm also probably missing some.

The one I see most clearly is that the CMake build, as it stands, involves
Too Much Magic. I don't at all disagree. That said, I strongly believe this
could be completely addressed.

- If we moved to CMake as the standard build system, numerous kludgy aspects
of the current build would go away. They are often in existence purely to
support interoperation with the old system.

- It would be very straight forward to centralize all of the library
dependencies and descriptions in the single top-level CMakeLists.txt file,
making it easily consumable by your average developer. It would have a
format no harder to edit or understand than the one you propose, and they
would both (at worst) be unfamiliar to existing developers.

- It would likely improve the quality of our CMake builds by ensuring it was
well tested and always in a consistent state.

- It already has a relatively optimized makefile-generation system, so we
wouldn't need to re-invent this wheel again.


The biggest downside to making CMake the standard build system is the
dependence on CMake to my eyes. However, among the cross-platform users of
LLVM, I think CMake is often the preferred build system. I know of folks
using it under xcode, visual studio, mingw, cygwin, and all flavors of
Linux.


Anyways, I'm sure there are more considerations than just these, I just
think it would be beneficial to seriously consider using an existing
meta-build system rather than rolling our own.
-Chandler

On Thu, Oct 27, 2011 at 6:11 PM, Daniel Dunbar <daniel at zuster.org>
wrote:
> Hi all,
>
> As you might have inferred, I'm in the process of working on some
changes
> to the
> way LLVM builds. I have outlined a rough proposal below, unless there are
> any
> major objections I will probably start committing stuff next week.
>
> This message may be verbose, if you want the executive summary, skip
> to 'What This
> Means For Jane "LLVM Developer" Doe' at the bottom.
>
> Motivation
> ----------
>
> We currently maintain two build systems (make, CMake). This has a couple of
> problems:
>
>  * Duplication: the two build systems duplicate a significant amount of
> implicit
>   logic (higher level dependencies), some config files, and other
> miscellaneous
>   features.
>
>  * Maintenance: Some parts of the build systems requires regular
> maintenance
>   (the CMake file lists, CMake dependency lists, module structure). Having
> one
>   more thing to maintain is annoying.
>
>  * Inconsistency: The two build systems behave inconsistently in some ways.
> If
>   we want both to officially supported systems, it would be nice for them
> to
>   behave as identically as possible.
>
>   For example, CMake now uses explicit dependencies which are hard coded
> into
>   the CMake files, but the Makefiles work completely differently.
>
> There are also other general issues with the way LLVM is built now:
>
>  * LLVM has a higher level structure for its organization, but this is not
>   explicit. LLVM is roughly organized into components (libraries, tools,
> etc.)
>   by directory. It would be nice to have this be more explicit.
>
>  * Much of the project build structure is implicit in the Makefiles or
>   CMakeFiles. It is not particularly explicit anywhere that, say, parts of
>   VMCore depend on building the Intrinsics, which depend on building
> tblgen.
>
>  * The Make system is not very efficient. We use recursive Makefiles which
> make
>   the current system (relatively) simple in some ways, but mean the Make
> build
>   is not nearly as scalable as it could be. In particular, the current
>   organization means the built is often serialized on something that is not
> a
>   strict dependency. It also makes it much more likely to do things like a
>   stampeding link of all the tools, even though many tools could have been
>   built earlier.
>
>
> Specific Goals
> --------------
>
>  * Move both build systems to use explicit library dependencies, in a clean
>   fashion. The CMake files do this now, but I don't think it has been
made
>   clear to developers when they are supposed to edit these files, or how
> (other
>   than when something breaks, I guess).
>
>  * Explicitly describe as much of the project structure as necessary to
> support
>   builds. This means explicitly specifying how the project is organized,
> and
>   the dependencies among the components required to build (e.g., Intrinsics
>   before VMCore).
>
>   I believe a number of other projects/users (FreeBSD, fbuild) have
> built there own
>   build system for LLVM. Encoding the project structure clearly should make
> it
>   easier for such projects, or for other future users that want to do the
> same.
>
>   This should make it easier to experiment with the build system, for
> example
>   we could just directly generate good Makefiles for our project, or could
>   experiment with systems like Ninja which expect to be targetted by a
>   generator of some kind.
>
>
> Proposal
> --------
>
> My initial proposal is focused at moving us to use explicit library
> dependencies, but paves the way for centralizing more "build
systemy" stuff
> in
> the future.
>
>  * Every LLVM "component" (roughly corresponds to each place we
have a
> Makefile
>   or CMakeLists.txt currently) will get a 'LLVMBuild.txt' file.
>
>   This file will be an LLVM specific description of that component.
> Initially,
>   it will look something like this::
>
>     [component]
>     # The kind of component this is (currently library, tool, build tool).
> More
>     # types will be defined over time.
>     type = Library
>
>     # The name of the component.
>     name = VMCore
>
>     # The name of the component to logically group this in. This is just
> for
>     # organization purposes.
>     parent = Libraries
>
>     # The names of the library components that are also required when
> linking
>     # with this library. More on this later.
>     required_libraries = Support
>
>   The exact structure of the format is TBD (and easy to change), currently
> the
>   format is INI style but I may decide to change to JSON once all the
> pieces
>   are in place.
>
>   The LLVM web pages will have clear documentation on what these files
> should
>   look like, what is required, what is supported, and so on.
>
>
>  * I will add a new tool in utils/llvm-build which is designed to load and
> work
>   with these files. This tool will be written in Python, and the
> expectation is
>   that it can be run at configure time.
>
>   TO BE CLEAR: I intend to introduce a hard dependency on requiring Python
> in
>   order to build LLVM.
>
>   For the Makefiles, this is no worse than requiring Perl, so I don't
think
>   there is much to argue with.
>
>   For CMake, this is a new dependency. However, I feel this is unavoidable:
>
>     * I see no way to support multiple build systems including CMake
> without
>       either introducing a dependency on some extra tool (which can be
> shared),
>       or duplicating a significant amount of logic in CMake.
>
>       I believe that duplicating logic is worse than adding the Python
>       dependency, and I think we already have more CMake code (a relatively
>       horrible language) than can be expected for most LLVM developers to
> deal
>       with.
>
>   Additionally, we already require Python for running our tests, so anyone
>   doing serious LLVM development should have it.
>
>   The one use case I believe this particularly hurts is users who just want
> to
>   download and play with LLVM, but I believe the Right Answer (tm) for that
>   case would be for us to provide nice installer packages anyway.
>
>
>  * utils/llvm-build will be run during configure time (both for Make and
> CMake),
>   and will:
>
>   * Generate the library dependency information required to link tools, in
>     whatever format makes the most system for the build system in use.
>
>   * Generate a C++ .inc file containing the dependency table for use by
>     llvm-config (which I am going to rewrite in C++).
>
>   * Add dependencies on the LLVMBuild.txt files to the build system, so
> that
>     the build will reconfigure appropriately when the library
> requirements change.
>
>
>  * Remove GenLibDeps.pl, find-cycles.pl, etc.
>
>   We will no longer be using these after llvm-config has moved over.
>
>
>  * Add explicit source file lists to the LLVMBuild.txt files.
> Unfortunately,
>   this is inevitable if we want to support CMake users projects
> automatically
>   reconfiguring themselves when new files get added. I can make it easier
> to
>   manage (for example, provide build targets that will automatically add
> any
>   new files).
>
>
>  * Move both Make and CMake over to using the explicit file lists in the
>   LLVMBuild files. This ensures that both systems get the same behavior
>   (instead of Make users being able to add files and forget to update
>   CMakeLists.txt).
>
>
>  * Add new 'lit' tests to check that the library dependency
> information is correct.
>
>   This seems a nicer place to do the checking which is currently partially
>   handled by find-cycles, and we should also be able to do a better job of
> the
>   checking (for example, verifying that the dependency list is
"exact" --
> only
>   specifies the minimum set of libraries required, and isn't allowed to
> specify
>   libraries which are only indirectly required).
>
>   It would be particularly cool if we could just write these tests using
> our
>   Object libraries.
>
>   This is one piece I haven't prototyped yet. I can obviously do
something
> as
>   good as the current find-cycles.pl, but I hope to do better
> (eventually).
>
>
>  * These are just the first steps, after this I will continue to
> incrementally
>   try and move as much common information out of the Make and CMake systems
> so
>   there is "one source of truth" with regard to the project
definition.
>
>
>  * I assume it is obvious, but when I say "LLVM" here I am
referring to
> both
>   LLVM and Clang. I have not looked at lldb yet, but if it uses the LLVM
> build
>   system (and llvm-config) functionality I will try to make sure it
> continues to work.
>
>
> What This Means For Jane "LLVM Developer" Doe
> ---------------------------------------------
>
> In practice, this means:
>
>  * LLVM requires Python to build.
>
>  * When you add a file to LLVM, you will need to edit LLVMBuild.txt instead
> of
>   CMakeLists.txt, which will be in a slightly different, but otherwise
> totally
>   obvious format.
>
>   If you forget to do this, your file will not be built (which will most
> likely
>   cause a link error eventually). This is better than it being built by
> Make,
>   but causing CMake build failures when you check in.
>
>  * When you add a new library requirement to an existing component, you
> will be
>   required to edit LLVMBuild.txt instead of CMakeLists.txt, which will be
> in a
>   slightly different, but otherwise totally obvious (hopefully) format.
>
>   If you forget to do this, you will either get a link error or a test
>   failure. This is better than library you need transparently getting
> linked in
>   (with make) because it forces you to think about whether you actually
> should
>   be adding that dependency.
>
>   The goal is that this also ensures that if LLVM links and passes tests on
>   your system, then it should for everyone else as well.
>
>  * Developers not actively touching the build system should never need to
> touch
>   a Makefile or a CMake file.
>
> Overall, I believe this should be a quality of life improvement for the
> developer community. The only downside is having to deal with a new
> non-standard
> LLVM specific format, but I plan to solve this through documentation.
>
> Comments?
>
>  - Daniel
> _______________________________________________
> cfe-dev mailing list
> cfe-dev at cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20111027/ac5ff575/attachment.html>

On Thu, Oct 27, 2011 at 6:34 PM, Chandler Carruth <chandlerc at
google.com>wrote:
> I have a very high level comment, and you may be able to directly shed
> light on it before I dig into a lot more detail.
>
> Why not simply standardize on CMake? It's not my favorite tool, but it
> seems to work well, we have established usage of it, and several people
> involved in the project who understand how it works. It doesn't seem
like a
> significantly more burdensome dependency than Python when developing, and
it
> remains possible to build installable packages for the casual hacker.
>
> I can see some objections to CMake, but it's not clear to me that they
> should carry the day. I'm also probably missing some.
>
> The one I see most clearly is that the CMake build, as it stands, involves
> Too Much Magic. I don't at all disagree. That said, I strongly believe
this
> could be completely addressed.
>
> - If we moved to CMake as the standard build system, numerous kludgy
> aspects of the current build would go away. They are often in existence
> purely to support interoperation with the old system.
>
> - It would be very straight forward to centralize all of the library
> dependencies and descriptions in the single top-level CMakeLists.txt file,
> making it easily consumable by your average developer. It would have a
> format no harder to edit or understand than the one you propose, and they
> would both (at worst) be unfamiliar to existing developers.
>
> - It would likely improve the quality of our CMake builds by ensuring it
> was well tested and always in a consistent state.
>
> - It already has a relatively optimized makefile-generation system, so we
> wouldn't need to re-invent this wheel again.
>
Something else I wanted to mention, although I don't know how relevant it
really is to most LLVM and/or Clang developers is that we have several Clang
developers who are actually contributing to CMake specifically around
integration with Clang and related tools. I expect these to increase over
the next year...
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20111027/f10edb55/attachment.html>

On Oct 27, 2011, at 6:34 PM, Chandler Carruth wrote:
> I have a very high level comment, and you may be able to directly shed
light on it before I dig into a lot more detail.
> 
> Why not simply standardize on CMake? It's not my favorite tool, but it
seems to work well, we have established usage of it, and several people involved
in the project who understand how it works. It doesn't seem like a
significantly more burdensome dependency than Python when developing, and it
remains possible to build installable packages for the casual hacker.
> 
> I can see some objections to CMake, but it's not clear to me that they
should carry the day. I'm also probably missing some.
There are several major problems with CMake IMO:

1. It generates really slow build systems.

2. The build system generated by cmake is absolute garbage, at least for Xcode. 
The build times of it are really bad, and having to work with it in the IDE is
even more terrible.

3. I'd really like us to get to explicit and principled library
dependencies, where the build horks if you accidentally add a library
dependency.  Without this, I don't see how LLVM will ever scale up.

4. I'd really like us to move in a direction where LLVM is less monolithic. 
Right now "llvm" contains all the MC stuff, all the targets, all the
mid-level optimizer, etc.  Adding a new LLVM MC-based linker will cause it to
naturally get dropped into the llvm project, which is great but not helping the
monolithicness. :)

To me at least, I see this as a first step to getting LLVM to be a more scalable
project.  Things like llvm-config are essential, but not extensible to
subprojects like Clang, etc.
> Anyways, I'm sure there are more considerations than just these, I just
think it would be beneficial to seriously consider using an existing meta-build
system rather than rolling our own.
It seems that all sufficiently large open source projects evolve their own meta
build systems!

-Chris
> -Chandler
> 
> On Thu, Oct 27, 2011 at 6:11 PM, Daniel Dunbar <daniel at zuster.org>
wrote:
> Hi all,
> 
> As you might have inferred, I'm in the process of working on some
changes to the
> way LLVM builds. I have outlined a rough proposal below, unless there are
any
> major objections I will probably start committing stuff next week.
> 
> This message may be verbose, if you want the executive summary, skip
> to 'What This
> Means For Jane "LLVM Developer" Doe' at the bottom.
> 
> Motivation
> ----------
> 
> We currently maintain two build systems (make, CMake). This has a couple of
> problems:
> 
>  * Duplication: the two build systems duplicate a significant amount of
implicit
>   logic (higher level dependencies), some config files, and other
miscellaneous
>   features.
> 
>  * Maintenance: Some parts of the build systems requires regular
maintenance
>   (the CMake file lists, CMake dependency lists, module structure). Having
one
>   more thing to maintain is annoying.
> 
>  * Inconsistency: The two build systems behave inconsistently in some ways.
If
>   we want both to officially supported systems, it would be nice for them
to
>   behave as identically as possible.
> 
>   For example, CMake now uses explicit dependencies which are hard coded
into
>   the CMake files, but the Makefiles work completely differently.
> 
> There are also other general issues with the way LLVM is built now:
> 
>  * LLVM has a higher level structure for its organization, but this is not
>   explicit. LLVM is roughly organized into components (libraries, tools,
etc.)
>   by directory. It would be nice to have this be more explicit.
> 
>  * Much of the project build structure is implicit in the Makefiles or
>   CMakeFiles. It is not particularly explicit anywhere that, say, parts of
>   VMCore depend on building the Intrinsics, which depend on building
tblgen.
> 
>  * The Make system is not very efficient. We use recursive Makefiles which
make
>   the current system (relatively) simple in some ways, but mean the Make
build
>   is not nearly as scalable as it could be. In particular, the current
>   organization means the built is often serialized on something that is not
a
>   strict dependency. It also makes it much more likely to do things like a
>   stampeding link of all the tools, even though many tools could have been
>   built earlier.
> 
> 
> Specific Goals
> --------------
> 
>  * Move both build systems to use explicit library dependencies, in a clean
>   fashion. The CMake files do this now, but I don't think it has been
made
>   clear to developers when they are supposed to edit these files, or how
(other
>   than when something breaks, I guess).
> 
>  * Explicitly describe as much of the project structure as necessary to
support
>   builds. This means explicitly specifying how the project is organized,
and
>   the dependencies among the components required to build (e.g., Intrinsics
>   before VMCore).
> 
>   I believe a number of other projects/users (FreeBSD, fbuild) have
> built there own
>   build system for LLVM. Encoding the project structure clearly should make
it
>   easier for such projects, or for other future users that want to do the
same.
> 
>   This should make it easier to experiment with the build system, for
example
>   we could just directly generate good Makefiles for our project, or could
>   experiment with systems like Ninja which expect to be targetted by a
>   generator of some kind.
> 
> 
> Proposal
> --------
> 
> My initial proposal is focused at moving us to use explicit library
> dependencies, but paves the way for centralizing more "build
systemy" stuff in
> the future.
> 
>  * Every LLVM "component" (roughly corresponds to each place we
have a Makefile
>   or CMakeLists.txt currently) will get a 'LLVMBuild.txt' file.
> 
>   This file will be an LLVM specific description of that component.
Initially,
>   it will look something like this::
> 
>     [component]
>     # The kind of component this is (currently library, tool, build tool).
More
>     # types will be defined over time.
>     type = Library
> 
>     # The name of the component.
>     name = VMCore
> 
>     # The name of the component to logically group this in. This is just
for
>     # organization purposes.
>     parent = Libraries
> 
>     # The names of the library components that are also required when
linking
>     # with this library. More on this later.
>     required_libraries = Support
> 
>   The exact structure of the format is TBD (and easy to change), currently
the
>   format is INI style but I may decide to change to JSON once all the
pieces
>   are in place.
> 
>   The LLVM web pages will have clear documentation on what these files
should
>   look like, what is required, what is supported, and so on.
> 
> 
>  * I will add a new tool in utils/llvm-build which is designed to load and
work
>   with these files. This tool will be written in Python, and the
expectation is
>   that it can be run at configure time.
> 
>   TO BE CLEAR: I intend to introduce a hard dependency on requiring Python
in
>   order to build LLVM.
> 
>   For the Makefiles, this is no worse than requiring Perl, so I don't
think
>   there is much to argue with.
> 
>   For CMake, this is a new dependency. However, I feel this is unavoidable:
> 
>     * I see no way to support multiple build systems including CMake
without
>       either introducing a dependency on some extra tool (which can be
shared),
>       or duplicating a significant amount of logic in CMake.
> 
>       I believe that duplicating logic is worse than adding the Python
>       dependency, and I think we already have more CMake code (a relatively
>       horrible language) than can be expected for most LLVM developers to
deal
>       with.
> 
>   Additionally, we already require Python for running our tests, so anyone
>   doing serious LLVM development should have it.
> 
>   The one use case I believe this particularly hurts is users who just want
to
>   download and play with LLVM, but I believe the Right Answer (tm) for that
>   case would be for us to provide nice installer packages anyway.
> 
> 
>  * utils/llvm-build will be run during configure time (both for Make and
CMake),
>   and will:
> 
>   * Generate the library dependency information required to link tools, in
>     whatever format makes the most system for the build system in use.
> 
>   * Generate a C++ .inc file containing the dependency table for use by
>     llvm-config (which I am going to rewrite in C++).
> 
>   * Add dependencies on the LLVMBuild.txt files to the build system, so
that
>     the build will reconfigure appropriately when the library
> requirements change.
> 
> 
>  * Remove GenLibDeps.pl, find-cycles.pl, etc.
> 
>   We will no longer be using these after llvm-config has moved over.
> 
> 
>  * Add explicit source file lists to the LLVMBuild.txt files.
Unfortunately,
>   this is inevitable if we want to support CMake users projects
automatically
>   reconfiguring themselves when new files get added. I can make it easier
to
>   manage (for example, provide build targets that will automatically add
any
>   new files).
> 
> 
>  * Move both Make and CMake over to using the explicit file lists in the
>   LLVMBuild files. This ensures that both systems get the same behavior
>   (instead of Make users being able to add files and forget to update
>   CMakeLists.txt).
> 
> 
>  * Add new 'lit' tests to check that the library dependency
> information is correct.
> 
>   This seems a nicer place to do the checking which is currently partially
>   handled by find-cycles, and we should also be able to do a better job of
the
>   checking (for example, verifying that the dependency list is
"exact" -- only
>   specifies the minimum set of libraries required, and isn't allowed to
specify
>   libraries which are only indirectly required).
> 
>   It would be particularly cool if we could just write these tests using
our
>   Object libraries.
> 
>   This is one piece I haven't prototyped yet. I can obviously do
something as
>   good as the current find-cycles.pl, but I hope to do better (eventually).
> 
> 
>  * These are just the first steps, after this I will continue to
incrementally
>   try and move as much common information out of the Make and CMake systems
so
>   there is "one source of truth" with regard to the project
definition.
> 
> 
>  * I assume it is obvious, but when I say "LLVM" here I am
referring to both
>   LLVM and Clang. I have not looked at lldb yet, but if it uses the LLVM
build
>   system (and llvm-config) functionality I will try to make sure it
> continues to work.
> 
> 
> What This Means For Jane "LLVM Developer" Doe
> ---------------------------------------------
> 
> In practice, this means:
> 
>  * LLVM requires Python to build.
> 
>  * When you add a file to LLVM, you will need to edit LLVMBuild.txt instead
of
>   CMakeLists.txt, which will be in a slightly different, but otherwise
totally
>   obvious format.
> 
>   If you forget to do this, your file will not be built (which will most
likely
>   cause a link error eventually). This is better than it being built by
Make,
>   but causing CMake build failures when you check in.
> 
>  * When you add a new library requirement to an existing component, you
will be
>   required to edit LLVMBuild.txt instead of CMakeLists.txt, which will be
in a
>   slightly different, but otherwise totally obvious (hopefully) format.
> 
>   If you forget to do this, you will either get a link error or a test
>   failure. This is better than library you need transparently getting
linked in
>   (with make) because it forces you to think about whether you actually
should
>   be adding that dependency.
> 
>   The goal is that this also ensures that if LLVM links and passes tests on
>   your system, then it should for everyone else as well.
> 
>  * Developers not actively touching the build system should never need to
touch
>   a Makefile or a CMake file.
> 
> Overall, I believe this should be a quality of life improvement for the
> developer community. The only downside is having to deal with a new
non-standard
> LLVM specific format, but I plan to solve this through documentation.
> 
> Comments?
> 
>  - Daniel
> _______________________________________________
> cfe-dev mailing list
> cfe-dev at cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
> 
> _______________________________________________
> cfe-dev mailing list
> cfe-dev at cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20111027/bd13f2bd/attachment.html>

Hi Daniel,

On 10/27/2011 09:11 PM, Daniel Dunbar wrote:
> Hi all,
>
> As you might have inferred, I'm in the process of working on some
changes to the
> way LLVM builds. I have outlined a rough proposal below, unless there are
any
> major objections I will probably start committing stuff next week.
>
>
I'm not an LLVM dev, but I am the maintainer for all of the LLVM-related 
packages for my distro. I have a few thoughts on changes to the build 
system that could make my life easier, but they're largely orthogonal to 
the work you described. Are you interested in hearing about them now, or 
would you rather just focus on the (rather large) task you've already 
set out? I've been planning on doing a writeup to send to the list 
anyway, but if there's a chance you'll integrate the changes in work 
starting next week I'll be sure to collect all my thoughts and send them 
sooner.

Regards,
Shea Levy

On Oct 27, 2011, at 11:28 PM, Óscar Fuentes wrote:
> Chris Lattner <clattner at apple.com> writes:
> 
>> There are several major problems with CMake IMO:
>> 
>> 1. It generates really slow build systems.
> 
> In my Linux box, last time I checked (long time ago) the cmake build was
> a bit faster than the Makefiles. But this is a tricky terrain, because
> they are not identical, not on the features supported (and some have an
> impact on build-time) nor even on the options passed to the compiler.
The makefiles are known to be really slow, among other problems being based on
recursive make.  One goal of this is to get a non-recursive makefile generated. 
We've prototyped this in the past and found it to be substantially faster
than the recursive makefile.
>> 2. The build system generated by cmake is absolute garbage, at least
>> for Xcode.  The build times of it are really bad, and having to work
>> with it in the IDE is even more terrible.
> 
> AFAIK there is a Xcode project file on the LLVM source tree.
Nope, there was once but it was removed a long time ago though.
> Are the
> LLVM makefiles used by the Xcode project?
No, it is generated by Cmake.
> If the Xcode project files
> generated by cmake is not satisfactory, can't they use the Makefiles
> generated by cmake instead?
Xcode can drive a makefile, but it doesn't provide any of the IDE
integration features, e.g. clang code completion.

-Chris

Chris Lattner <clattner at apple.com> writes:
>>> 1. It generates really slow build systems.
>> 
>> In my Linux box, last time I checked (long time ago) the cmake build
was
>> a bit faster than the Makefiles. But this is a tricky terrain, because
>> they are not identical, not on the features supported (and some have an
>> impact on build-time) nor even on the options passed to the compiler.
>
> The makefiles are known to be really slow, among other problems being
> based on recursive make.  One goal of this is to get a non-recursive
> makefile generated.  We've prototyped this in the past and found it to
> be substantially faster than the recursive makefile.
A good measure of how fast a set of Makefile are is to run the build
with all targets up-to-date. Both builds takes a few seconds (3 or so)
on my Linux quad core box. Whatever improvement can be achieved on this
seems pretty insignifant.

Furthermore, recursive make is necessary for automatic generation of
header dependencies, among other things. The makefiles generated by
cmake are "partially" recursive for that reason:

http://www.cmake.org/Wiki/CMake_FAQ#Why_does_CMake_generate_recursive_Makefiles.3F
>>> 2. The build system generated by cmake is absolute garbage, at
least
>>> for Xcode.  The build times of it are really bad, and having to
work
>>> with it in the IDE is even more terrible.
>> 
>> AFAIK there is a Xcode project file on the LLVM source tree.
>
> Nope, there was once but it was removed a long time ago though.
>
>> Are the
>> LLVM makefiles used by the Xcode project?
>
> No, it is generated by Cmake.
So, the cmake-generated Xcode file was considered good enough or... ?

[snip]

Anyways, if you wish to avoid duplicating info on both Makefile and
CMakeLists.txt there is a simple solution: read and parse the Makefile
from the corresponding CMakeLists.txt. For instance, if you put the
library dependencies on the Makefile like this:

LLVMLIBDEPS := foo zoo bar

obtaining that info from the CMakeLists.txt and generating the cmake
library dependencies is very simple, nor even you have to put anything
new on all those CMakeLists.txt, just modify one of the macros that are
(indirectly) called from each CMakeLists.txt.

+1: Extract full, exact (meta) data in pure form in easy to handle 
format and generate bits for concrete build system on demand - simply great!

As a side effect, probably LLVMBuild.txt data would help the 
distribution maintainers in the support of the package specifications too.

On 28.10.2011 04:11, Daniel Dunbar wrote:
>   * Every LLVM "component" (roughly corresponds to each place we
have a Makefile
>     or CMakeLists.txt currently) will get a 'LLVMBuild.txt' file.
>
>     This file will be an LLVM specific description of that component.
Initially,
>     it will look something like this::
>
>       [component]
>       # The kind of component this is (currently library, tool, build
tool). More
>       # types will be defined over time.
>       type = Library
>
>       # The name of the component.
>       name = VMCore
>
>       # The name of the component to logically group this in. This is just
for
>       # organization purposes.
>       parent = Libraries
>
>       # The names of the library components that are also required when
linking
>       # with this library. More on this later.
>       required_libraries = Support
>

On 28 Oct 2011, at 02:11, Daniel Dunbar wrote:
>   TO BE CLEAR: I intend to introduce a hard dependency on requiring Python
in
>   order to build LLVM.
> 
>   For the Makefiles, this is no worse than requiring Perl, so I don't
think
>   there is much to argue with.
I disagree there.  Perl is pretty much guaranteed to be installed on any UNIXish
system.  Even FreeBSD, which has removed it from the base system, tends to
install the Perl package by default.  In contrast, a lot of the machines I use
don't have Python installed.  I need to install it if I'm doing LLVM
development because it's needed for the tests, but needing it just to build
seems like massive overkill.

That said, if the information required for the build is going to be made
explicit, maybe this isn't such a problem, as other tools can be written to
parse it and run the build.

David

On Fri, Oct 28, 2011 at 3:34 AM, Chandler Carruth 
wrote:
> I have a very high level comment, and you may be able to directly shed
light
> on it before I dig into a lot more detail.
> Why not simply standardize on CMake?
That would establish a hard dependency on CMake. Not every system has
CMake whereas most systems do have Python by default (on the machines
I use daily, Python has a 5-1 lead).
See also David Chisnall's mail about Perl > Python.

Csaba
-- 
GCS a+ e++ d- C++ ULS$ L+$ !E- W++ P+++$ w++$ tv+ b++ DI D++ 5++
The Tao of math: The numbers you can count are not the real numbers.
Life is complex, with real and imaginary parts.
"Ok, it boots. Which means it must be bug-free and perfect. " -- Linus
Torvalds
"People disagree with me. I just ignore them." -- Linus Torvalds

I wouldn't say that. I know quite a few systems here around that even try to
avoid python where possible. but cmake however, as a build system, is
welcomed by all of us (working as a sysop in a unix environment).

I'd also (as a non-llvm-dev but llvm-userdev) vote for NOT reinventing the
wheel but to use the tool the fits you the best, personally that's even
cmake, too. it has a well list of great backing companies / projects and is
still improving well, e.g. Qt planned (I do not know how up-to-date this
info is) improve it in a way to make it more suitable for IDEs, however,
from the sysop point of view, it's much more a pleasure to work with cmake
than with autotools, and when you introduce (yet) another new build system,
it would be just a headache :)

Best regards,
Christian Parpart.


On Fri, Oct 28, 2011 at 12:55 PM, Csaba Raduly <rcsaba at gmail.com>
wrote:
> On Fri, Oct 28, 2011 at 3:34 AM, Chandler Carruth  wrote:
> > I have a very high level comment, and you may be able to directly shed
> light
> > on it before I dig into a lot more detail.
> > Why not simply standardize on CMake?
>
> That would establish a hard dependency on CMake. Not every system has
> CMake whereas most systems do have Python by default (on the machines
> I use daily, Python has a 5-1 lead).
> See also David Chisnall's mail about Perl > Python.
>
> Csaba
> --
> GCS a+ e++ d- C++ ULS$ L+$ !E- W++ P+++$ w++$ tv+ b++ DI D++ 5++
> The Tao of math: The numbers you can count are not the real numbers.
> Life is complex, with real and imaginary parts.
> "Ok, it boots. Which means it must be bug-free and perfect. " --
Linus
> Torvalds
> "People disagree with me. I just ignore them." -- Linus Torvalds
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20111028/704d1b42/attachment.html>

Le 28 oct. 2011 à 15:40, Christian Parpart a écrit :
> I wouldn't say that. I know quite a few systems here around that even
try to avoid python where possible. but cmake however, as a build system, is
welcomed by all of us (working as a sysop in a unix environment).
> 
> I'd also (as a non-llvm-dev but llvm-userdev) vote for NOT reinventing
the wheel but to use the tool the fits you the best, personally that's even
cmake, too. it has a well list of great backing companies / projects and is
still improving well, e.g. Qt planned (I do not know how up-to-date this info
is) improve it in a way to make it more suitable for IDEs, however, from the
sysop point of view, it's much more a pleasure to work with cmake than with
autotools, and when you introduce (yet) another new build system, it would be
just a headache :)
If I understand the proposal correctly, from a Jan Doe (llvm-user) point of
view, you will just continue to use cmake. The only difference will be that
cmake will call a python script to generate a bunch of files used by cmake  or
make.

That said, wouldn't it be possible to not require this script to be run for
the users, but simply add resulting files in the repository (just like many
project do not require you run autoconf, and just distribute the generated
configure script).

Like that, the python dependency will be required only if you change the module
description files, and not for casual developers and users who just plan to
recompile llvm.
> Best regards,
> Christian Parpart.
> 
> 
> On Fri, Oct 28, 2011 at 12:55 PM, Csaba Raduly <rcsaba at gmail.com>
wrote:
> On Fri, Oct 28, 2011 at 3:34 AM, Chandler Carruth  wrote:
> > I have a very high level comment, and you may be able to directly shed
light
> > on it before I dig into a lot more detail.
> > Why not simply standardize on CMake?
> 
> That would establish a hard dependency on CMake. Not every system has
> CMake whereas most systems do have Python by default (on the machines
> I use daily, Python has a 5-1 lead).
> See also David Chisnall's mail about Perl > Python.
> 
> Csaba
> --
> GCS a+ e++ d- C++ ULS$ L+$ !E- W++ P+++$ w++$ tv+ b++ DI D++ 5++
> The Tao of math: The numbers you can count are not the real numbers.
> Life is complex, with real and imaginary parts.
> "Ok, it boots. Which means it must be bug-free and perfect. " --
Linus Torvalds
> "People disagree with me. I just ignore them." -- Linus Torvalds
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
> 
> 
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
-- Jean-Daniel




-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20111028/d47158a3/attachment.html>

+1

We build our OpenCL SDK (for windows and Linux) using CMake.  We’ve  integrated
LLVM’s Cmake hierarchy into our own (customizing LLVM external parameters like
build and install directories, added passes, etc)

Migrating LLVM’s build system from CMake to something else would require us to
change the way we currently do things.



From: llvmdev-bounces at cs.uiuc.edu [mailto:llvmdev-bounces at cs.uiuc.edu] On
Behalf Of Chandler Carruth
Sent: Friday, October 28, 2011 03:35
To: Daniel Dunbar
Cc: cfe-dev; LLVM Developers Mailing List
Subject: Re: [LLVMdev] [cfe-dev] RFC: Upcoming Build System Changes

I have a very high level comment, and you may be able to directly shed light on
it before I dig into a lot more detail.

Why not simply standardize on CMake? It's not my favorite tool, but it seems
to work well, we have established usage of it, and several people involved in
the project who understand how it works. It doesn't seem like a
significantly more burdensome dependency than Python when developing, and it
remains possible to build installable packages for the casual hacker.

I can see some objections to CMake, but it's not clear to me that they
should carry the day. I'm also probably missing some.

The one I see most clearly is that the CMake build, as it stands, involves Too
Much Magic. I don't at all disagree. That said, I strongly believe this
could be completely addressed.

- If we moved to CMake as the standard build system, numerous kludgy aspects of
the current build would go away. They are often in existence purely to support
interoperation with the old system.

- It would be very straight forward to centralize all of the library
dependencies and descriptions in the single top-level CMakeLists.txt file,
making it easily consumable by your average developer. It would have a format no
harder to edit or understand than the one you propose, and they would both (at
worst) be unfamiliar to existing developers.

- It would likely improve the quality of our CMake builds by ensuring it was
well tested and always in a consistent state.

- It already has a relatively optimized makefile-generation system, so we
wouldn't need to re-invent this wheel again.


The biggest downside to making CMake the standard build system is the dependence
on CMake to my eyes. However, among the cross-platform users of LLVM, I think
CMake is often the preferred build system. I know of folks using it under xcode,
visual studio, mingw, cygwin, and all flavors of Linux.


Anyways, I'm sure there are more considerations than just these, I just
think it would be beneficial to seriously consider using an existing meta-build
system rather than rolling our own.
-Chandler

On Thu, Oct 27, 2011 at 6:11 PM, Daniel Dunbar <daniel at
zuster.org<mailto:daniel at zuster.org>> wrote:
Hi all,

As you might have inferred, I'm in the process of working on some changes to
the
way LLVM builds. I have outlined a rough proposal below, unless there are any
major objections I will probably start committing stuff next week.

This message may be verbose, if you want the executive summary, skip
to 'What This
Means For Jane "LLVM Developer" Doe' at the bottom.

Motivation
----------

We currently maintain two build systems (make, CMake). This has a couple of
problems:

 * Duplication: the two build systems duplicate a significant amount of implicit
  logic (higher level dependencies), some config files, and other miscellaneous
  features.

 * Maintenance: Some parts of the build systems requires regular maintenance
  (the CMake file lists, CMake dependency lists, module structure). Having one
  more thing to maintain is annoying.

 * Inconsistency: The two build systems behave inconsistently in some ways. If
  we want both to officially supported systems, it would be nice for them to
  behave as identically as possible.

  For example, CMake now uses explicit dependencies which are hard coded into
  the CMake files, but the Makefiles work completely differently.

There are also other general issues with the way LLVM is built now:

 * LLVM has a higher level structure for its organization, but this is not
  explicit. LLVM is roughly organized into components (libraries, tools, etc.)
  by directory. It would be nice to have this be more explicit.

 * Much of the project build structure is implicit in the Makefiles or
  CMakeFiles. It is not particularly explicit anywhere that, say, parts of
  VMCore depend on building the Intrinsics, which depend on building tblgen.

 * The Make system is not very efficient. We use recursive Makefiles which make
  the current system (relatively) simple in some ways, but mean the Make build
  is not nearly as scalable as it could be. In particular, the current
  organization means the built is often serialized on something that is not a
  strict dependency. It also makes it much more likely to do things like a
  stampeding link of all the tools, even though many tools could have been
  built earlier.


Specific Goals
--------------

 * Move both build systems to use explicit library dependencies, in a clean
  fashion. The CMake files do this now, but I don't think it has been made
  clear to developers when they are supposed to edit these files, or how (other
  than when something breaks, I guess).

 * Explicitly describe as much of the project structure as necessary to support
  builds. This means explicitly specifying how the project is organized, and
  the dependencies among the components required to build (e.g., Intrinsics
  before VMCore).

  I believe a number of other projects/users (FreeBSD, fbuild) have
built there own
  build system for LLVM. Encoding the project structure clearly should make it
  easier for such projects, or for other future users that want to do the same.

  This should make it easier to experiment with the build system, for example
  we could just directly generate good Makefiles for our project, or could
  experiment with systems like Ninja which expect to be targetted by a
  generator of some kind.


Proposal
--------

My initial proposal is focused at moving us to use explicit library
dependencies, but paves the way for centralizing more "build systemy"
stuff in
the future.

 * Every LLVM "component" (roughly corresponds to each place we have a
Makefile
  or CMakeLists.txt currently) will get a 'LLVMBuild.txt' file.

  This file will be an LLVM specific description of that component. Initially,
  it will look something like this::

    [component]
    # The kind of component this is (currently library, tool, build tool). More
    # types will be defined over time.
    type = Library

    # The name of the component.
    name = VMCore

    # The name of the component to logically group this in. This is just for
    # organization purposes.
    parent = Libraries

    # The names of the library components that are also required when linking
    # with this library. More on this later.
    required_libraries = Support

  The exact structure of the format is TBD (and easy to change), currently the
  format is INI style but I may decide to change to JSON once all the pieces
  are in place.

  The LLVM web pages will have clear documentation on what these files should
  look like, what is required, what is supported, and so on.


 * I will add a new tool in utils/llvm-build which is designed to load and work
  with these files. This tool will be written in Python, and the expectation is
  that it can be run at configure time.

  TO BE CLEAR: I intend to introduce a hard dependency on requiring Python in
  order to build LLVM.

  For the Makefiles, this is no worse than requiring Perl, so I don't think
  there is much to argue with.

  For CMake, this is a new dependency. However, I feel this is unavoidable:

    * I see no way to support multiple build systems including CMake without
      either introducing a dependency on some extra tool (which can be shared),
      or duplicating a significant amount of logic in CMake.

      I believe that duplicating logic is worse than adding the Python
      dependency, and I think we already have more CMake code (a relatively
      horrible language) than can be expected for most LLVM developers to deal
      with.

  Additionally, we already require Python for running our tests, so anyone
  doing serious LLVM development should have it.

  The one use case I believe this particularly hurts is users who just want to
  download and play with LLVM, but I believe the Right Answer (tm) for that
  case would be for us to provide nice installer packages anyway.


 * utils/llvm-build will be run during configure time (both for Make and CMake),
  and will:

  * Generate the library dependency information required to link tools, in
    whatever format makes the most system for the build system in use.

  * Generate a C++ .inc file containing the dependency table for use by
    llvm-config (which I am going to rewrite in C++).

  * Add dependencies on the LLVMBuild.txt files to the build system, so that
    the build will reconfigure appropriately when the library
requirements change.


 * Remove GenLibDeps.pl, find-cycles.pl<http://find-cycles.pl>, etc.

  We will no longer be using these after llvm-config has moved over.


 * Add explicit source file lists to the LLVMBuild.txt files. Unfortunately,
  this is inevitable if we want to support CMake users projects automatically
  reconfiguring themselves when new files get added. I can make it easier to
  manage (for example, provide build targets that will automatically add any
  new files).


 * Move both Make and CMake over to using the explicit file lists in the
  LLVMBuild files. This ensures that both systems get the same behavior
  (instead of Make users being able to add files and forget to update
  CMakeLists.txt).


 * Add new 'lit' tests to check that the library dependency
information is correct.

  This seems a nicer place to do the checking which is currently partially
  handled by find-cycles, and we should also be able to do a better job of the
  checking (for example, verifying that the dependency list is "exact"
-- only
  specifies the minimum set of libraries required, and isn't allowed to
specify
  libraries which are only indirectly required).

  It would be particularly cool if we could just write these tests using our
  Object libraries.

  This is one piece I haven't prototyped yet. I can obviously do something
as
  good as the current find-cycles.pl<http://find-cycles.pl>, but I hope to
do better (eventually).


 * These are just the first steps, after this I will continue to incrementally
  try and move as much common information out of the Make and CMake systems so
  there is "one source of truth" with regard to the project
definition.


 * I assume it is obvious, but when I say "LLVM" here I am referring
to both
  LLVM and Clang. I have not looked at lldb yet, but if it uses the LLVM build
  system (and llvm-config) functionality I will try to make sure it
continues to work.


What This Means For Jane "LLVM Developer" Doe
---------------------------------------------

In practice, this means:

 * LLVM requires Python to build.

 * When you add a file to LLVM, you will need to edit LLVMBuild.txt instead of
  CMakeLists.txt, which will be in a slightly different, but otherwise totally
  obvious format.

  If you forget to do this, your file will not be built (which will most likely
  cause a link error eventually). This is better than it being built by Make,
  but causing CMake build failures when you check in.

 * When you add a new library requirement to an existing component, you will be
  required to edit LLVMBuild.txt instead of CMakeLists.txt, which will be in a
  slightly different, but otherwise totally obvious (hopefully) format.

  If you forget to do this, you will either get a link error or a test
  failure. This is better than library you need transparently getting linked in
  (with make) because it forces you to think about whether you actually should
  be adding that dependency.

  The goal is that this also ensures that if LLVM links and passes tests on
  your system, then it should for everyone else as well.

 * Developers not actively touching the build system should never need to touch
  a Makefile or a CMake file.

Overall, I believe this should be a quality of life improvement for the
developer community. The only downside is having to deal with a new non-standard
LLVM specific format, but I plan to solve this through documentation.

Comments?

 - Daniel
_______________________________________________
cfe-dev mailing list
cfe-dev at cs.uiuc.edu<mailto:cfe-dev at cs.uiuc.edu>
http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev

---------------------------------------------------------------------
Intel Israel (74) Limited

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20111028/7e8ae754/attachment.html>

Hi David,

On 28.10.2011 13:05, David Chisnall wrote:
> I disagree there.  Perl is pretty much guaranteed to be installed on any
UNIXish system.  Even FreeBSD, which has removed it from the base system, tends
to install the Perl package by default.  In contrast, a lot of the machines I
use don't have Python installed.  I need to install it if I'm doing LLVM
development because it's needed for the tests, but needing it just to build
seems like massive overkill.
It is possible that you are right about the overall index of
"presence".
Just to mention that in Fedora derivatives (RedHat, CentOS, maybe 35-50% 
of Linux dev stations - I don't know) Python is guaranteed because of 
yum (the package manager).
>
> That said, if the information required for the build is going to be made
explicit, maybe this isn't such a problem, as other tools can be written to
parse it and run the build.
Absolutely - once the generators are prototyped and tested in Python, if 
current Perl (presence) > Python's, they can be easily ported to Perl.

Kind Regards,
Alek

Hi,

my 2 cents:

1) If you're targeting python 2.6 or less, I prefer a python
dependency over cmake 'cause it's already installed on all machines I
build clang on. If you're targeting 2.7+, I don't care either way.

2) On the chromium project, we're using a custom meta build system
written in python, and it's been a lot more useful than we initially
expected. Based on this experience, your proposal sounds good to me.

Nico

On Thu, Oct 27, 2011 at 6:11 PM, Daniel Dunbar <daniel at zuster.org>
wrote:
> Hi all,
>
> As you might have inferred, I'm in the process of working on some
changes to the
> way LLVM builds. I have outlined a rough proposal below, unless there are
any
> major objections I will probably start committing stuff next week.
>
> This message may be verbose, if you want the executive summary, skip
> to 'What This
> Means For Jane "LLVM Developer" Doe' at the bottom.
>
> Motivation
> ----------
>
> We currently maintain two build systems (make, CMake). This has a couple of
> problems:
>
>  * Duplication: the two build systems duplicate a significant amount of
implicit
>   logic (higher level dependencies), some config files, and other
miscellaneous
>   features.
>
>  * Maintenance: Some parts of the build systems requires regular
maintenance
>   (the CMake file lists, CMake dependency lists, module structure). Having
one
>   more thing to maintain is annoying.
>
>  * Inconsistency: The two build systems behave inconsistently in some ways.
If
>   we want both to officially supported systems, it would be nice for them
to
>   behave as identically as possible.
>
>   For example, CMake now uses explicit dependencies which are hard coded
into
>   the CMake files, but the Makefiles work completely differently.
>
> There are also other general issues with the way LLVM is built now:
>
>  * LLVM has a higher level structure for its organization, but this is not
>   explicit. LLVM is roughly organized into components (libraries, tools,
etc.)
>   by directory. It would be nice to have this be more explicit.
>
>  * Much of the project build structure is implicit in the Makefiles or
>   CMakeFiles. It is not particularly explicit anywhere that, say, parts of
>   VMCore depend on building the Intrinsics, which depend on building
tblgen.
>
>  * The Make system is not very efficient. We use recursive Makefiles which
make
>   the current system (relatively) simple in some ways, but mean the Make
build
>   is not nearly as scalable as it could be. In particular, the current
>   organization means the built is often serialized on something that is not
a
>   strict dependency. It also makes it much more likely to do things like a
>   stampeding link of all the tools, even though many tools could have been
>   built earlier.
>
>
> Specific Goals
> --------------
>
>  * Move both build systems to use explicit library dependencies, in a clean
>   fashion. The CMake files do this now, but I don't think it has been
made
>   clear to developers when they are supposed to edit these files, or how
(other
>   than when something breaks, I guess).
>
>  * Explicitly describe as much of the project structure as necessary to
support
>   builds. This means explicitly specifying how the project is organized,
and
>   the dependencies among the components required to build (e.g., Intrinsics
>   before VMCore).
>
>   I believe a number of other projects/users (FreeBSD, fbuild) have
> built there own
>   build system for LLVM. Encoding the project structure clearly should make
it
>   easier for such projects, or for other future users that want to do the
same.
>
>   This should make it easier to experiment with the build system, for
example
>   we could just directly generate good Makefiles for our project, or could
>   experiment with systems like Ninja which expect to be targetted by a
>   generator of some kind.
>
>
> Proposal
> --------
>
> My initial proposal is focused at moving us to use explicit library
> dependencies, but paves the way for centralizing more "build
systemy" stuff in
> the future.
>
>  * Every LLVM "component" (roughly corresponds to each place we
have a Makefile
>   or CMakeLists.txt currently) will get a 'LLVMBuild.txt' file.
>
>   This file will be an LLVM specific description of that component.
Initially,
>   it will look something like this::
>
>     [component]
>     # The kind of component this is (currently library, tool, build tool).
More
>     # types will be defined over time.
>     type = Library
>
>     # The name of the component.
>     name = VMCore
>
>     # The name of the component to logically group this in. This is just
for
>     # organization purposes.
>     parent = Libraries
>
>     # The names of the library components that are also required when
linking
>     # with this library. More on this later.
>     required_libraries = Support
>
>   The exact structure of the format is TBD (and easy to change), currently
the
>   format is INI style but I may decide to change to JSON once all the
pieces
>   are in place.
>
>   The LLVM web pages will have clear documentation on what these files
should
>   look like, what is required, what is supported, and so on.
>
>
>  * I will add a new tool in utils/llvm-build which is designed to load and
work
>   with these files. This tool will be written in Python, and the
expectation is
>   that it can be run at configure time.
>
>   TO BE CLEAR: I intend to introduce a hard dependency on requiring Python
in
>   order to build LLVM.
>
>   For the Makefiles, this is no worse than requiring Perl, so I don't
think
>   there is much to argue with.
>
>   For CMake, this is a new dependency. However, I feel this is unavoidable:
>
>     * I see no way to support multiple build systems including CMake
without
>       either introducing a dependency on some extra tool (which can be
shared),
>       or duplicating a significant amount of logic in CMake.
>
>       I believe that duplicating logic is worse than adding the Python
>       dependency, and I think we already have more CMake code (a relatively
>       horrible language) than can be expected for most LLVM developers to
deal
>       with.
>
>   Additionally, we already require Python for running our tests, so anyone
>   doing serious LLVM development should have it.
>
>   The one use case I believe this particularly hurts is users who just want
to
>   download and play with LLVM, but I believe the Right Answer (tm) for that
>   case would be for us to provide nice installer packages anyway.
>
>
>  * utils/llvm-build will be run during configure time (both for Make and
CMake),
>   and will:
>
>   * Generate the library dependency information required to link tools, in
>     whatever format makes the most system for the build system in use.
>
>   * Generate a C++ .inc file containing the dependency table for use by
>     llvm-config (which I am going to rewrite in C++).
>
>   * Add dependencies on the LLVMBuild.txt files to the build system, so
that
>     the build will reconfigure appropriately when the library
> requirements change.
>
>
>  * Remove GenLibDeps.pl, find-cycles.pl, etc.
>
>   We will no longer be using these after llvm-config has moved over.
>
>
>  * Add explicit source file lists to the LLVMBuild.txt files.
Unfortunately,
>   this is inevitable if we want to support CMake users projects
automatically
>   reconfiguring themselves when new files get added. I can make it easier
to
>   manage (for example, provide build targets that will automatically add
any
>   new files).
>
>
>  * Move both Make and CMake over to using the explicit file lists in the
>   LLVMBuild files. This ensures that both systems get the same behavior
>   (instead of Make users being able to add files and forget to update
>   CMakeLists.txt).
>
>
>  * Add new 'lit' tests to check that the library dependency
> information is correct.
>
>   This seems a nicer place to do the checking which is currently partially
>   handled by find-cycles, and we should also be able to do a better job of
the
>   checking (for example, verifying that the dependency list is
"exact" -- only
>   specifies the minimum set of libraries required, and isn't allowed to
specify
>   libraries which are only indirectly required).
>
>   It would be particularly cool if we could just write these tests using
our
>   Object libraries.
>
>   This is one piece I haven't prototyped yet. I can obviously do
something as
>   good as the current find-cycles.pl, but I hope to do better (eventually).
>
>
>  * These are just the first steps, after this I will continue to
incrementally
>   try and move as much common information out of the Make and CMake systems
so
>   there is "one source of truth" with regard to the project
definition.
>
>
>  * I assume it is obvious, but when I say "LLVM" here I am
referring to both
>   LLVM and Clang. I have not looked at lldb yet, but if it uses the LLVM
build
>   system (and llvm-config) functionality I will try to make sure it
> continues to work.
>
>
> What This Means For Jane "LLVM Developer" Doe
> ---------------------------------------------
>
> In practice, this means:
>
>  * LLVM requires Python to build.
>
>  * When you add a file to LLVM, you will need to edit LLVMBuild.txt instead
of
>   CMakeLists.txt, which will be in a slightly different, but otherwise
totally
>   obvious format.
>
>   If you forget to do this, your file will not be built (which will most
likely
>   cause a link error eventually). This is better than it being built by
Make,
>   but causing CMake build failures when you check in.
>
>  * When you add a new library requirement to an existing component, you
will be
>   required to edit LLVMBuild.txt instead of CMakeLists.txt, which will be
in a
>   slightly different, but otherwise totally obvious (hopefully) format.
>
>   If you forget to do this, you will either get a link error or a test
>   failure. This is better than library you need transparently getting
linked in
>   (with make) because it forces you to think about whether you actually
should
>   be adding that dependency.
>
>   The goal is that this also ensures that if LLVM links and passes tests on
>   your system, then it should for everyone else as well.
>
>  * Developers not actively touching the build system should never need to
touch
>   a Makefile or a CMake file.
>
> Overall, I believe this should be a quality of life improvement for the
> developer community. The only downside is having to deal with a new
non-standard
> LLVM specific format, but I plan to solve this through documentation.
>
> Comments?
>
>  - Daniel
> _______________________________________________
> cfe-dev mailing list
> cfe-dev at cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
>

On Oct 27, 2011, at 6:11 PM, Daniel Dunbar wrote:
> 
> 
> In practice, this means:
> 
> * LLVM requires Python to build.
> 
> * When you add a file to LLVM, you will need to edit LLVMBuild.txt instead
of
>   CMakeLists.txt, which will be in a slightly different, but otherwise
totally
>   obvious format.
> 
>   If you forget to do this, your file will not be built (which will most
likely
>   cause a link error eventually). This is better than it being built by
Make,
>   but causing CMake build failures when you check in.
Yay!
> 
> * Developers not actively touching the build system should never need to
touch
>   a Makefile or a CMake file.
That'd be great. This is the biggest benefit of your proposal IMO.
-
Devang

On Oct 27, 2011, at 6:11 PM, Daniel Dunbar wrote:
> Overall, I believe this should be a quality of life improvement for the
> developer community. The only downside is having to deal with a new
non-standard
> LLVM specific format, but I plan to solve this through documentation.
> 
> Comments?
This all sounds great. And having seen the quality of your documentation
I have no apprehension whatsoever.

Thanks for all of this!

-eric
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20111028/18f1ff8a/attachment.html>

Daniel,

I'm just a 42 year old former NeXT and Apple Dev who back in Mechanical 
Engineering is excited about OpenCL/OpenGL and all that is with 
LLVM/Clang, but I've got to say that even though my plans include 
learning Python for various areas of development the last I want to deal 
with is a Build system needing it to compile a Compiler Suite.

Autotools is a bag of hurt, always has been.

Cmake improves considerably on it with it's CmakeCache.txt giving a 
cleanly prepared configuration one expects to see their project follow 
and doesn't drown one in a list of ten thousand lines of crap just to 
build something as common as Gimp, Inkscape, GTK+, GNUstep, Poppler, 
Freetype, POVRay, Bullet Library, Blender and any other project that 
also has their own internal build stages whether it's SCons or Python.

Why in the hell should one need to become well-versed in another 
language like Python just to customize how Cmake can build a project, 
when fixing the documentation and explanation of Cmake seems a more 
astute solution?

I've read that Cmake is a dog on Xcode. Fix support for it in Xcode.

Autotools and make for Project Builder/IB/EOF/WOF was garbage and had 
tons of custom hacks at NeXT and Apple. They didn't abandon it. They 
improved upon it to suite the needs of our products.

Then this complaint about build times and extra CPU cycles when you're 
living in a world of systems soon to average 16GB of RAM, 4-12 cores and 
GPUs that would make any old Animator dream back in the '90s really 
makes me laugh.

Reinventing the wheel 50 ways instead of improving the wheel until it's 
consistent, clean and easily extendable by third parties seems to be a 
worthy endeavor.

I love my Apple stock and all the hard work folks have put in to making 
great projects under Steve's vision, but to read about how Cmake is in 
bad shape with Xcode is something I cannot imagine Tevanian, Enderby, 
Ozer or other Engineers would have ever thought were such a daunting 
task that it cannot be resolved.

With all the brilliant minds on this list, you'd think they'd be able to
coordinate a Proposal to improve Cmake upstream and provide a solution 
all parties could love.

You all get paid exceedingly well in an industry we all work in and to 
read about having to maintain a system that isn't perfect won't garner 
any compassion from me, or any reasonable person who knows 90% of life 
is a grind.

I would think these minds could convince the Cmake devs to improve their 
system and make it scalable as Lattner envisions without having to add 
Python or any other third party language into the mix and ultimately 
pair back down to Cmake.

Fix the engine. Worry about the suspension, handling, lights, sound 
system and the rest of the fluff later.

I read a lot of conjecture, hypotheticals and opinion.

Oscar seems to recognize the Keep It Simple Stupid mantra that any first 
year Mechanical Engineering student learns studying Machine Design so I 
would implore you, Oscar, to lead a Documentatio Project on Cmake with 
LLVM/Clang so us ``Jane `LLVM Developer' Doe'' types cannot only
digest
an abstract but also drill down into the manual with clear and concise 
examples--something I've always found lacking in the majority of the Man 
System legacy of UNIX.

I used to write NeXTAnswer solutions and support Enterprise Clients who 
could give a rat's behind about learning a bazillion little wrinkles on 
getting a solution to their trouble ticket(s).

Working in SQA at NeXT taught me to wear many hats. Relevant 
documentation cross-reference by revisions of a project where solutions 
worked is also a must. [Works under llvm-2.8, llvm-2.9, clang-2.7, 
clang-2.8, etc]

Like I said, my voice doesn't mean squat, but I'd suggest before trying 
to put lipstick on a pig, you have the solution completed, tested and 
fully documented. It goes much farther in convincing people to learn it.

By the way, the default Python version on Debian [and it's many spawned 
Distros] is 2.7. On OS X I've got 2.7-3.2. I've got the same on Linux.

On both I have only the current Cmake.

Reading through the thread tells me there is a lot of brainstorming to 
do and a lot of issues that are triggered, depending on how the 
implementation gets decided.

However it falls out, I'd hope a fully documented, living system of 
documentation is produced so no one has to waste the time of most senior 
developers on this list on basic ways of building, testing and deploying 
projects build against LLVM/Clang, whether it's on OS X, Linux, FreeBSD, 
Windows, Solaris, or whatever other platform supports LLVM/Clang.

It seems to me that part of being an architect [not my job] and 
implementing such a vast collection of projects under the LLVM/Clang 
umbrella requires those same folks to be fluent Technical Writers who 
can drill down to the average lamen.

- Marc J. Driftmeyer


On 10/27/2011 06:11 PM, Daniel Dunbar wrote:
> Hi all,
>
> As you might have inferred, I'm in the process of working on some
changes to the
> way LLVM builds. I have outlined a rough proposal below, unless there are
any
> major objections I will probably start committing stuff next week.
>
> This message may be verbose, if you want the executive summary, skip
> to 'What This
> Means For Jane "LLVM Developer" Doe' at the bottom.
>
> Motivation
> ----------
>
> We currently maintain two build systems (make, CMake). This has a couple of
> problems:
>
>   * Duplication: the two build systems duplicate a significant amount of
implicit
>     logic (higher level dependencies), some config files, and other
miscellaneous
>     features.
>
>   * Maintenance: Some parts of the build systems requires regular
maintenance
>     (the CMake file lists, CMake dependency lists, module structure).
Having one
>     more thing to maintain is annoying.
>
>   * Inconsistency: The two build systems behave inconsistently in some
ways. If
>     we want both to officially supported systems, it would be nice for them
to
>     behave as identically as possible.
>
>     For example, CMake now uses explicit dependencies which are hard coded
into
>     the CMake files, but the Makefiles work completely differently.
>
> There are also other general issues with the way LLVM is built now:
>
>   * LLVM has a higher level structure for its organization, but this is not
>     explicit. LLVM is roughly organized into components (libraries, tools,
etc.)
>     by directory. It would be nice to have this be more explicit.
>
>   * Much of the project build structure is implicit in the Makefiles or
>     CMakeFiles. It is not particularly explicit anywhere that, say, parts
of
>     VMCore depend on building the Intrinsics, which depend on building
tblgen.
>
>   * The Make system is not very efficient. We use recursive Makefiles which
make
>     the current system (relatively) simple in some ways, but mean the Make
build
>     is not nearly as scalable as it could be. In particular, the current
>     organization means the built is often serialized on something that is
not a
>     strict dependency. It also makes it much more likely to do things like
a
>     stampeding link of all the tools, even though many tools could have
been
>     built earlier.
>
>
> Specific Goals
> --------------
>
>   * Move both build systems to use explicit library dependencies, in a
clean
>     fashion. The CMake files do this now, but I don't think it has been
made
>     clear to developers when they are supposed to edit these files, or how
(other
>     than when something breaks, I guess).
>
>   * Explicitly describe as much of the project structure as necessary to
support
>     builds. This means explicitly specifying how the project is organized,
and
>     the dependencies among the components required to build (e.g.,
Intrinsics
>     before VMCore).
>
>     I believe a number of other projects/users (FreeBSD, fbuild) have
> built there own
>     build system for LLVM. Encoding the project structure clearly should
make it
>     easier for such projects, or for other future users that want to do the
same.
>
>     This should make it easier to experiment with the build system, for
example
>     we could just directly generate good Makefiles for our project, or
could
>     experiment with systems like Ninja which expect to be targetted by a
>     generator of some kind.
>
>
> Proposal
> --------
>
> My initial proposal is focused at moving us to use explicit library
> dependencies, but paves the way for centralizing more "build
systemy" stuff in
> the future.
>
>   * Every LLVM "component" (roughly corresponds to each place we
have a Makefile
>     or CMakeLists.txt currently) will get a 'LLVMBuild.txt' file.
>
>     This file will be an LLVM specific description of that component.
Initially,
>     it will look something like this::
>
>       [component]
>       # The kind of component this is (currently library, tool, build
tool). More
>       # types will be defined over time.
>       type = Library
>
>       # The name of the component.
>       name = VMCore
>
>       # The name of the component to logically group this in. This is just
for
>       # organization purposes.
>       parent = Libraries
>
>       # The names of the library components that are also required when
linking
>       # with this library. More on this later.
>       required_libraries = Support
>
>     The exact structure of the format is TBD (and easy to change),
currently the
>     format is INI style but I may decide to change to JSON once all the
pieces
>     are in place.
>
>     The LLVM web pages will have clear documentation on what these files
should
>     look like, what is required, what is supported, and so on.
>
>
>   * I will add a new tool in utils/llvm-build which is designed to load and
work
>     with these files. This tool will be written in Python, and the
expectation is
>     that it can be run at configure time.
>
>     TO BE CLEAR: I intend to introduce a hard dependency on requiring
Python in
>     order to build LLVM.
>
>     For the Makefiles, this is no worse than requiring Perl, so I don't
think
>     there is much to argue with.
>
>     For CMake, this is a new dependency. However, I feel this is
unavoidable:
>
>       * I see no way to support multiple build systems including CMake
without
>         either introducing a dependency on some extra tool (which can be
shared),
>         or duplicating a significant amount of logic in CMake.
>
>         I believe that duplicating logic is worse than adding the Python
>         dependency, and I think we already have more CMake code (a
relatively
>         horrible language) than can be expected for most LLVM developers to
deal
>         with.
>
>     Additionally, we already require Python for running our tests, so
anyone
>     doing serious LLVM development should have it.
>
>     The one use case I believe this particularly hurts is users who just
want to
>     download and play with LLVM, but I believe the Right Answer (tm) for
that
>     case would be for us to provide nice installer packages anyway.
>
>
>   * utils/llvm-build will be run during configure time (both for Make and
CMake),
>     and will:
>
>     * Generate the library dependency information required to link tools,
in
>       whatever format makes the most system for the build system in use.
>
>     * Generate a C++ .inc file containing the dependency table for use by
>       llvm-config (which I am going to rewrite in C++).
>
>     * Add dependencies on the LLVMBuild.txt files to the build system, so
that
>       the build will reconfigure appropriately when the library
> requirements change.
>
>
>   * Remove GenLibDeps.pl, find-cycles.pl, etc.
>
>     We will no longer be using these after llvm-config has moved over.
>
>
>   * Add explicit source file lists to the LLVMBuild.txt files.
Unfortunately,
>     this is inevitable if we want to support CMake users projects
automatically
>     reconfiguring themselves when new files get added. I can make it easier
to
>     manage (for example, provide build targets that will automatically add
any
>     new files).
>
>
>   * Move both Make and CMake over to using the explicit file lists in the
>     LLVMBuild files. This ensures that both systems get the same behavior
>     (instead of Make users being able to add files and forget to update
>     CMakeLists.txt).
>
>
>   * Add new 'lit' tests to check that the library dependency
> information is correct.
>
>     This seems a nicer place to do the checking which is currently
partially
>     handled by find-cycles, and we should also be able to do a better job
of the
>     checking (for example, verifying that the dependency list is
"exact" -- only
>     specifies the minimum set of libraries required, and isn't allowed
to specify
>     libraries which are only indirectly required).
>
>     It would be particularly cool if we could just write these tests using
our
>     Object libraries.
>
>     This is one piece I haven't prototyped yet. I can obviously do
something as
>     good as the current find-cycles.pl, but I hope to do better
(eventually).
>
>
>   * These are just the first steps, after this I will continue to
incrementally
>     try and move as much common information out of the Make and CMake
systems so
>     there is "one source of truth" with regard to the project
definition.
>
>
>   * I assume it is obvious, but when I say "LLVM" here I am
referring to both
>     LLVM and Clang. I have not looked at lldb yet, but if it uses the LLVM
build
>     system (and llvm-config) functionality I will try to make sure it
> continues to work.
>
>
> What This Means For Jane "LLVM Developer" Doe
> ---------------------------------------------
>
> In practice, this means:
>
>   * LLVM requires Python to build.
>
>   * When you add a file to LLVM, you will need to edit LLVMBuild.txt
instead of
>     CMakeLists.txt, which will be in a slightly different, but otherwise
totally
>     obvious format.
>
>     If you forget to do this, your file will not be built (which will most
likely
>     cause a link error eventually). This is better than it being built by
Make,
>     but causing CMake build failures when you check in.
>
>   * When you add a new library requirement to an existing component, you
will be
>     required to edit LLVMBuild.txt instead of CMakeLists.txt, which will be
in a
>     slightly different, but otherwise totally obvious (hopefully) format.
>
>     If you forget to do this, you will either get a link error or a test
>     failure. This is better than library you need transparently getting
linked in
>     (with make) because it forces you to think about whether you actually
should
>     be adding that dependency.
>
>     The goal is that this also ensures that if LLVM links and passes tests
on
>     your system, then it should for everyone else as well.
>
>   * Developers not actively touching the build system should never need to
touch
>     a Makefile or a CMake file.
>
> Overall, I believe this should be a quality of life improvement for the
> developer community. The only downside is having to deal with a new
non-standard
> LLVM specific format, but I plan to solve this through documentation.
>
> Comments?
>
>   - Daniel
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
-- 
Marc J. Driftmeyer
Email :: mjd at reanimality.com <mailto:mjd at reanimality.com>
Web :: http://www.reanimality.com
Cell :: (509) 435-5212
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20111031/77553f29/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mjd.vcf
Type: text/x-vcard
Size: 317 bytes
Desc: not available
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20111031/77553f29/attachment.vcf>

Am 01.11.2011 05:59, schrieb Marc J. Driftmeyer:
> Then this complaint about build times and extra CPU cycles when you're
> living in a world of systems soon to average 16GB of RAM, 4-12 cores and
> GPUs that would make any old Animator dream back in the '90s really
> makes me laugh.
Not disagreeing about the rest, but here I have to. In today's projects, 
full rebuilds take hours where they should take minutes, and minutes 
where they should take seconds.
Hour-long rebuilds turn releases into things that must be organized and 
coordinated; minute-long rebuilds still disrupt workflow. Time and 
effort spent on getting the build process fast are well invested.

On the reasons why make-based builds are slow, Peter Miller has some 
insight to offer:
http://miller.emu.id.au/pmiller/books/rmch/ .
I'm not sure how widely recognized that paper is. Maybe it's widely 
known and today's build times stem from other things than recursive make.

Regards,
Jo

"Marc J. Driftmeyer" <mjd at reanimality.com> writes:
> Then this complaint about build times and extra CPU cycles when you're
> living in a world of systems soon to average 16GB of RAM, 4-12 cores
> and GPUs that would make any old Animator dream back in the '90s
> really makes me laugh.
The way the LLVM Makefiles are written prevents use of those 4-12
cores.  Make literally cannot see the parallelism.

I don't know how that works in the CMake world.  I avoid it whenever
possible.
> Reinventing the wheel 50 ways instead of improving the wheel until
> it's consistent, clean and easily extendable by third parties seems to
> be a worthy endeavor.
But CMake is a broken wheel.  It always has been.  The fact that it
generates recursive Makefiles is a big clue.  I believe it is
fundamentally architected incorrectly.  The whole premise is wrong
because it's a meta-build system.  The last thing we need for a process
as simple as "build" is lots of indirection.

Now, nothing Daniel is proposing really addresses that but he doesn't
intend to.  He just wants to make builds simpler and faster.  I for one
am all for that.

But don't claim CMake is well-designed, clean and easily extensible.
It's not.
> With all the brilliant minds on this list, you'd think they'd be
able
> to coordinate a Proposal to improve Cmake upstream and provide a
> solution all parties could love.
That's imposing an upstream dependency that LLVM does not want to have.
LLVM has different goals and those are going to conflict with the goals
of CMake's upstream.  It would be a nightmare to get parches approved,
for example.

                          -Dave