llvm dev - Oct 2011 - [LLVMdev] RFC/Heads Up: Deprecating External Build Support

Hi all,

I would like to deprecate the LLVM project's "official" support of
setting up the Makefiles / autoconf configurations in such a way that
external projects are encouraged to leverage them in their own build.

I am mostly referring to the things documented in docs/Projects.html
and projects/sample.

My justification:

(1) I believe few people use these capabilities.

(1a) The primary consumer that people have used in the past was
llvm-test-suite. That was broken in its own way, and the test suite
can now be used independently of an LLVM installation. This makes it
much more useful as a production test suite for compilers (vs. a test
suite for a research project).

(2) They are fragile and really not a very good way to structure an
external project. It is much better for external projects to just
structure themselves as any other open source project would and use
tools like llvm-config to gather build information.

(3) The official support is not particularly well known, and is easy
to break. Considering Makefiles and autoconf files as part of a
project API is painful, at best, and makes refactoring of project
structure difficult.

Object now, or hold your peace.

 - Daniel

Hi Daniel,

Will Clang be changed to follow this or will it be special-cased? My own python
compiler currently uses the same setup Clang does (as you describe), and it
works rather well. I'd like to continue to do whatever Clang (or other
frontends) do...

Cheers,

James
________________________________________
From: llvmdev-bounces at cs.uiuc.edu [llvmdev-bounces at cs.uiuc.edu] On Behalf
Of Daniel Dunbar [daniel at zuster.org]
Sent: 11 October 2011 21:24
To: LLVM Developers Mailing List
Subject: [LLVMdev] RFC/Heads Up: Deprecating External Build Support

Hi all,

I would like to deprecate the LLVM project's "official" support of
setting up the Makefiles / autoconf configurations in such a way that
external projects are encouraged to leverage them in their own build.

I am mostly referring to the things documented in docs/Projects.html
and projects/sample.

My justification:

(1) I believe few people use these capabilities.

(1a) The primary consumer that people have used in the past was
llvm-test-suite. That was broken in its own way, and the test suite
can now be used independently of an LLVM installation. This makes it
much more useful as a production test suite for compilers (vs. a test
suite for a research project).

(2) They are fragile and really not a very good way to structure an
external project. It is much better for external projects to just
structure themselves as any other open source project would and use
tools like llvm-config to gather build information.

(3) The official support is not particularly well known, and is easy
to break. Considering Makefiles and autoconf files as part of a
project API is painful, at best, and makes refactoring of project
structure difficult.

Object now, or hold your peace.

 - Daniel
_______________________________________________
LLVM Developers mailing list
LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev


-- IMPORTANT NOTICE: The contents of this email and any attachments are
confidential and may also be privileged. If you are not the intended recipient,
please notify the sender immediately and do not disclose the contents to any
other person, use it for any purpose, or store or copy the information in any
medium.  Thank you.

On Oct 11, 2011, at 1:24 PM, Daniel Dunbar wrote:
> Hi all,
> 
> I would like to deprecate the LLVM project's "official"
support of
> setting up the Makefiles / autoconf configurations in such a way that
> external projects are encouraged to leverage them in their own build.
+1.  Please make sure Projects.html is updated, and please add a note to the 3.0
release notes.

-Chris
> 
> I am mostly referring to the things documented in docs/Projects.html
> and projects/sample.
> 
> My justification:
> 
> (1) I believe few people use these capabilities.
> 
> (1a) The primary consumer that people have used in the past was
> llvm-test-suite. That was broken in its own way, and the test suite
> can now be used independently of an LLVM installation. This makes it
> much more useful as a production test suite for compilers (vs. a test
> suite for a research project).
> 
> (2) They are fragile and really not a very good way to structure an
> external project. It is much better for external projects to just
> structure themselves as any other open source project would and use
> tools like llvm-config to gather build information.
> 
> (3) The official support is not particularly well known, and is easy
> to break. Considering Makefiles and autoconf files as part of a
> project API is painful, at best, and makes refactoring of project
> structure difficult.
> 
> Object now, or hold your peace.
> 
> - Daniel
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev

On 10/11/11 3:24 PM, Daniel Dunbar wrote:
> Hi all,
>
> I would like to deprecate the LLVM project's "official"
support of
> setting up the Makefiles / autoconf configurations in such a way that
> external projects are encouraged to leverage them in their own build.
>
> I am mostly referring to the things documented in docs/Projects.html
> and projects/sample.
>
> My justification:
>
> (1) I believe few people use these capabilities.
I use these capabilities extensively in numerous research and 
open-source projects.  Those projects include:

1) Automatic Pool Allocation (publicly available in the poolalloc module)
2) SAFECode (publicly available in the safecode module)
3) Giri (publicly available but currently of no real consequence)
4) A diagnosis tool developed by one of our graduate students
5) An integer overflow detector tool

The first two I care very much about; we are tracking mainline and want 
to keep them working so that we can have something for people to use 
when LLVM 3.0 hits the streets.

Additionally, SAFECode copies Clang into its own source tree (for a 
variety of reasons which I'll skip over for the purposes of this 
discussion).  The project build system actually makes this very easy; 
very few Clang Makefiles needed modification to enable Clang to compile 
within SAFECode.  If my projects have to create their own build system, 
then I'll have to duplicate LLVM's build machinery in the SAFECode 
project to build Clang.

The other projects I listed are less important; I merely list them so 
that you have an idea of how much work you'll make for me if you remove 
the Projects functionality.
:)
>
> (1a) The primary consumer that people have used in the past was
> llvm-test-suite. That was broken in its own way, and the test suite
> can now be used independently of an LLVM installation. This makes it
> much more useful as a production test suite for compilers (vs. a test
> suite for a research project).
>
> (2) They are fragile and really not a very good way to structure an
> external project. It is much better for external projects to just
> structure themselves as any other open source project would and use
> tools like llvm-config to gather build information.
IMHO, the LLVM Projects setup is a very good way for people to write 
their own LLVM passes without having to patch the LLVM source tree.  It 
alleviates the need to create a whole new build system, it allows one to 
build LLVM and one's project(s) all in a single go, and one's project 
source code is organized just as it would be within LLVM's source base 
(more or less).

I suspect that removing the Project functionality will also increase the 
demand for a switch to git as people will prefer to write their LLVM 
passes as patches to the LLVM source base instead of as a separate project.
>
> (3) The official support is not particularly well known, and is easy
> to break. Considering Makefiles and autoconf files as part of a
> project API is painful, at best, and makes refactoring of project
> structure difficult.

Building the Projects functionality was a pain (I'm the one that 
originally added the support for it, so I know).  If it's getting in the 
way of important changes to the build system, then perhaps it should be 
removed despite the fact that it'll create more work (headaches) for me.

That said, I ask for two things:

1) Please don't remove functionality like this right before an LLVM 
release.  Both SAFECode and Poolalloc use this feature, and I don't want 
to have to scramble at the last minute to make them work again with the 
upcoming LLVM 3.0.  If it has to go, please wait until development for 
LLVM 3.1 begins (waiting longer would be even better).

2) Please only remove it if it's currently getting in the way of some 
important bug fix, refactoring, or enhancement.  If you're removing it 
just because LLVM Core doesn't need it, then you're not fixing anything 
in LLVM, and you're breaking things that use LLVM.

-- John T.
>
> Object now, or hold your peace.
>
>   - Daniel
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev

Initially I don't actually plan on breaking anything, just removing
the idea that this is something we encourage people to do. So Clang
and your project would keep working.

Longer down the road, if I started changing things I would naturally
be keeping Clang up to date (I don't regard it as "External"), but
you
would end up having to make the same changes to your project.

 - Daniel

On Tue, Oct 11, 2011 at 1:56 PM, James Molloy <James.Molloy at arm.com>
wrote:
> Hi Daniel,
>
> Will Clang be changed to follow this or will it be special-cased? My own
python compiler currently uses the same setup Clang does (as you describe), and
it works rather well. I'd like to continue to do whatever Clang (or other
frontends) do...
>
> Cheers,
>
> James
> ________________________________________
> From: llvmdev-bounces at cs.uiuc.edu [llvmdev-bounces at cs.uiuc.edu] On
Behalf Of Daniel Dunbar [daniel at zuster.org]
> Sent: 11 October 2011 21:24
> To: LLVM Developers Mailing List
> Subject: [LLVMdev] RFC/Heads Up: Deprecating External Build Support
>
> Hi all,
>
> I would like to deprecate the LLVM project's "official"
support of
> setting up the Makefiles / autoconf configurations in such a way that
> external projects are encouraged to leverage them in their own build.
>
> I am mostly referring to the things documented in docs/Projects.html
> and projects/sample.
>
> My justification:
>
> (1) I believe few people use these capabilities.
>
> (1a) The primary consumer that people have used in the past was
> llvm-test-suite. That was broken in its own way, and the test suite
> can now be used independently of an LLVM installation. This makes it
> much more useful as a production test suite for compilers (vs. a test
> suite for a research project).
>
> (2) They are fragile and really not a very good way to structure an
> external project. It is much better for external projects to just
> structure themselves as any other open source project would and use
> tools like llvm-config to gather build information.
>
> (3) The official support is not particularly well known, and is easy
> to break. Considering Makefiles and autoconf files as part of a
> project API is painful, at best, and makes refactoring of project
> structure difficult.
>
> Object now, or hold your peace.
>
>  - Daniel
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>
>
> -- IMPORTANT NOTICE: The contents of this email and any attachments are
confidential and may also be privileged. If you are not the intended recipient,
please notify the sender immediately and do not disclose the contents to any
other person, use it for any purpose, or store or copy the information in any
medium.  Thank you.
>
>

On Oct 11, 2011, at 2:33 PM, Chris Lattner wrote:
> 
> On Oct 11, 2011, at 1:24 PM, Daniel Dunbar wrote:
> 
>> Hi all,
>> 
>> I would like to deprecate the LLVM project's "official"
support of
>> setting up the Makefiles / autoconf configurations in such a way that
>> external projects are encouraged to leverage them in their own build.
> 
> +1.  Please make sure Projects.html is updated, and please add a note to
the 3.0 release notes.
To be clear, I support saying that these are deprecated (and will be removed in
3.1) in the 3.0 release notes, but we shouldn't rip it out of mainline until
after the branch.

-Chris

Hi John,

I knew I would hear from you. :)

On Tue, Oct 11, 2011 at 2:38 PM, John Criswell <criswell at illinois.edu>
wrote:
> On 10/11/11 3:24 PM, Daniel Dunbar wrote:
>>
>> Hi all,
>>
>> I would like to deprecate the LLVM project's "official"
support of
>> setting up the Makefiles / autoconf configurations in such a way that
>> external projects are encouraged to leverage them in their own build.
>>
>> I am mostly referring to the things documented in docs/Projects.html
>> and projects/sample.
>>
>> My justification:
>>
>> (1) I believe few people use these capabilities.
>
> I use these capabilities extensively in numerous research and open-source
> projects.  Those projects include:
As it happens, I knew of two people who used this feature -- myself
(KLEE) and you. :)
> 1) Automatic Pool Allocation (publicly available in the poolalloc module)
> 2) SAFECode (publicly available in the safecode module)
> 3) Giri (publicly available but currently of no real consequence)
> 4) A diagnosis tool developed by one of our graduate students
> 5) An integer overflow detector tool
>
> The first two I care very much about; we are tracking mainline and want to
> keep them working so that we can have something for people to use when LLVM
> 3.0 hits the streets.
>
> Additionally, SAFECode copies Clang into its own source tree (for a variety
> of reasons which I'll skip over for the purposes of this discussion).
 The
> project build system actually makes this very easy; very few Clang
Makefiles
> needed modification to enable Clang to compile within SAFECode.  If my
> projects have to create their own build system, then I'll have to
duplicate
> LLVM's build machinery in the SAFECode project to build Clang.
>
> The other projects I listed are less important; I merely list them so that
> you have an idea of how much work you'll make for me if you remove the
> Projects functionality.
> :)
>
>>
>> (1a) The primary consumer that people have used in the past was
>> llvm-test-suite. That was broken in its own way, and the test suite
>> can now be used independently of an LLVM installation. This makes it
>> much more useful as a production test suite for compilers (vs. a test
>> suite for a research project).
>
>
>>
>> (2) They are fragile and really not a very good way to structure an
>> external project. It is much better for external projects to just
>> structure themselves as any other open source project would and use
>> tools like llvm-config to gather build information.
>
> IMHO, the LLVM Projects setup is a very good way for people to write their
> own LLVM passes without having to patch the LLVM source tree.  It
alleviates
> the need to create a whole new build system, it allows one to build LLVM
and
> one's project(s) all in a single go, and one's project source code
is
> organized just as it would be within LLVM's source base (more or less).
I wouldn't agree it is a "very good" way, but I agree it is a way.

The problem is that the "solution" we are providing for this problem
is too intimately tied to the build system of LLVM itself.

If the community thinks this is an important problem for LLVM to
address, then I think the ideal solution is that we provide a set of
autoconf/makefile/whatever foo that uses llvm-config and so on, but
doesn't try and do things like include the LLVM Makefiles or the LLVM
autoconf files. This would also provide a migration solution for
existing projects using the current functionality.
> I suspect that removing the Project functionality will also increase the
> demand for a switch to git as people will prefer to write their LLVM passes
> as patches to the LLVM source base instead of as a separate project.
>
>>
>> (3) The official support is not particularly well known, and is easy
>> to break. Considering Makefiles and autoconf files as part of a
>> project API is painful, at best, and makes refactoring of project
>> structure difficult.
>
>
> Building the Projects functionality was a pain (I'm the one that
originally
> added the support for it, so I know).  If it's getting in the way of
> important changes to the build system, then perhaps it should be removed
> despite the fact that it'll create more work (headaches) for me.
>
> That said, I ask for two things:
>
> 1) Please don't remove functionality like this right before an LLVM
release.
>  Both SAFECode and Poolalloc use this feature, and I don't want to have
to
> scramble at the last minute to make them work again with the upcoming LLVM
> 3.0.  If it has to go, please wait until development for LLVM 3.1 begins
> (waiting longer would be even better).
I have no intention of removing or breaking it now, I just want to
deprecate it (as in, declare somewhere that we don't officially
support it, and stop publicizing it).
> 2) Please only remove it if it's currently getting in the way of some
> important bug fix, refactoring, or enhancement.  If you're removing it
just
> because LLVM Core doesn't need it, then you're not fixing anything
in LLVM,
> and you're breaking things that use LLVM.
Right, I completely agree with this, and am completely familiar with
the pain of maintaining an external-to-LLVM client. I'll do my best to
maintain things when/if I ever start actually making changes.

 - Daniel
>
> -- John T.
>
>>
>> Object now, or hold your peace.
>>
>>  - Daniel
>> _______________________________________________
>> LLVM Developers mailing list
>> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
>> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>
>

On Tue, Oct 11, 2011 at 01:24:13PM -0700, Daniel Dunbar
wrote:
> Object now, or hold your peace.
We use it as well for an internal project. Not maintaining a build
system is really a nice feature that got us up to speed very quickly. We
have been using it for the past 4 years, and it's been proven reliable.
We will eventually switch to CMake, but if we can keep it around as long
as possible, it would be nice!

Thanks,
Julien
> 
>  - Daniel
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
-- 
Julien Lerouge
PGP Key Id: 0xB1964A62
PGP Fingerprint: 392D 4BAD DB8B CE7F 4E5F FA3C 62DB 4AA7 B196 4A62
PGP Public Key from: keyserver.pgp.com

Hello everybody,
> We use it as well for an internal project. Not maintaining a build
> system is really a nice feature that got us up to speed very quickly. We
> have been using it for the past 4 years, and it's been proven reliable.
> We will eventually switch to CMake, but if we can keep it around as long
> as possible, it would be nice!
the same holds for me, too. So, I would also be glad if this feature
persists, otherwise, I will presumable move my project to some
git-based solution.

Ciao, Fabian