On Wednesday 20 October 2004 12:01, Reid Spencer wrote:> I'm re-thinking my penchant for automake. automake is great for many > standard applications that just need to get portable makefiles up and > running quickly. However, it turns out that LLVM is "different enough" > from a standard application that automake might not be the best choice.I might just here to suggest that Boost.Build (http://boost.org/boost-build2) might be good.> Here's some of the problems I've run into: > > 1. There's no way to tell automake to build bytecode. Without a lot of > customization of automake itself, it just can't grok the fact that there > might be multiple ways to compile a c/c++ program producing different > results and requiring different tools.Boost.Build supports this very good. In particular, on my project on work I can run: bjam toolset=gcc bjam toolset=nm bjam toolset=nmm and the first will do a regular build, the third will compile for some embedded processor, and the third will pass the sources via some annotation tool and produce an annotated x86 binary. Besides, "toolset=gcc" can become "toolset=msvc" on windows, and it will have high chances of working.> 2. The entire llvm-test project would have to either be completely > rethought or stay the way it is, it just can't be easily automake'd > because it doesn't follow the automake pattern. > > 3. The llvm/test directory would require significant rework to get it to > automake correctly (via dejagnu).No comments on the two points above yet.> 4. There's no way to avoid listing all the sources for every library > (I've tried several alternatives with no good results).Easy, I've already have a half-working setup (only the targets I use are converted), and, for example, lib/Analysis/DataStructure/Jamfile has nothing but: llvm-lib datastructure ;> 5. There's no way to install bytecode libraries somewhere separately > from other libraries.Should be possible.> 6. Creating a distribution tarball would require additional Makefile.am > files to be inserted in all the directories under llvm/include, the > traversal of which would cause additional overhead for every non-dist > target. That's 17 gratuitous "make" processes per build. There doesn't > seem to be a way around this.Why distribution tarball has anything to do with build system?> 7. I can't get automake to stop doing a double configure. That is, when > you configure all the Makefiles are updated. You then go do a build and > it thinks it needs to reconfigure again so it does it twice. This is > just a waste of time (40 seconds on my machine).Ehm.... I'd say that configure should be orthogonal to building. No?> 8. automake's notion of buiding a library is very fixed. It basically > supports two things: libtool built shared libraries (linking) and ar > built libraries that are also run through ranlib. While its possible to > override AR and LINK, there isn't a way to override RANLIB on a > per-target basis so you can't build both a regular library (requiring > RANLIB=ranlib) and a bytecode library (requiring something like > RANLIB=true) in the same directory. We have in LLVM at least 4 ways to > build libraries: regular .a, pre-linked .o (combination of .o files), > shared-library, and bytecode-library. While I figured out a hack to do > pre-linked .o, it basically uses GNU Make, not automake to make it work > and therefore breaks automake's "make" portability.I think given Boost.Build's "properties" (like "toolset" above), this should be doable.> 1. Get dependency generation in a single pass like automake. This will > give us about a 30% speed up on builds .. and one of the main reasons > for moving to automake is taken away. Estimate: 1-2 hours.Well, Boost.Build is single pass already. Need to compare performance on LLVM, though. Basically, I can try to finish my attempt to add Boost.Build files to LLVM. But I wonder if that makes sense? Is it already decided to just improve the current system? Is anything with "Boost" in name will be rejected right away? - Volodya
On Thursday 21 October 2004 01:54, Vladimir Prus wrote:> On Wednesday 20 October 2004 12:01, Reid Spencer wrote: > > I'm re-thinking my penchant for automake. automake is great for many > > standard applications that just need to get portable makefiles up and > > running quickly. However, it turns out that LLVM is "different enough" > > from a standard application that automake might not be the best choice. > > I might just here to suggest that Boost.Build > (http://boost.org/boost-build2) might be good. > > > Here's some of the problems I've run into: > > > > 1. There's no way to tell automake to build bytecode. Without a lot of > > customization of automake itself, it just can't grok the fact that there > > might be multiple ways to compile a c/c++ program producing different > > results and requiring different tools. > > Boost.Build supports this very good. In particular, on my project on work I > can run: > > bjam toolset=gcc > bjam toolset=nm > bjam toolset=nmm > > and the first will do a regular build, the third will compile for some > embedded processor, and the third will pass the sources via some annotation > tool and produce an annotated x86 binary. > > Besides, "toolset=gcc" can become "toolset=msvc" on windows, and it will > have high chances of working.I don't think Reid was referring to the use of different compilers on (possibly) different platforms. If you look at runtime/GC/SemiSpace: those files are not compiled using the compiler used to compile the rest of the project (llc, lli and so on). They use the llvm-gcc-frontend to compile and link those libraries into llvm bytecode. The resulting files end up in runtime/GC/SemiSpace/BytecodeObj> > 2. The entire llvm-test project would have to either be completely > > rethought or stay the way it is, it just can't be easily automake'd > > because it doesn't follow the automake pattern. > > > > 3. The llvm/test directory would require significant rework to get it to > > automake correctly (via dejagnu). > > No comments on the two points above yet.Any ideas if the boost testing framework can apply to these two points?> > 4. There's no way to avoid listing all the sources for every library > > (I've tried several alternatives with no good results). > > Easy, I've already have a half-working setup (only the targets I use are > converted), and, for example, > > lib/Analysis/DataStructure/Jamfile > > has nothing but: > > llvm-lib datastructure ; > > > 5. There's no way to install bytecode libraries somewhere separately > > from other libraries. > > Should be possible. > > > 6. Creating a distribution tarball would require additional Makefile.am > > files to be inserted in all the directories under llvm/include, the > > traversal of which would cause additional overhead for every non-dist > > target. That's 17 gratuitous "make" processes per build. There doesn't > > seem to be a way around this. > > Why distribution tarball has anything to do with build system?This is done in automake. Because the distribution consists of files processed by automake, the "distcheck" target makes sure that those files are in the distribution and the distribution tarball can build properly without needing automake/autoconf.> > 7. I can't get automake to stop doing a double configure. That is, when > > you configure all the Makefiles are updated. You then go do a build and > > it thinks it needs to reconfigure again so it does it twice. This is > > just a waste of time (40 seconds on my machine). > > Ehm.... I'd say that configure should be orthogonal to building. No?I totally agree with this as well.> > 8. automake's notion of buiding a library is very fixed. It basically > > supports two things: libtool built shared libraries (linking) and ar > > built libraries that are also run through ranlib. While its possible to > > override AR and LINK, there isn't a way to override RANLIB on a > > per-target basis so you can't build both a regular library (requiring > > RANLIB=ranlib) and a bytecode library (requiring something like > > RANLIB=true) in the same directory. We have in LLVM at least 4 ways to > > build libraries: regular .a, pre-linked .o (combination of .o files), > > shared-library, and bytecode-library. While I figured out a hack to do > > pre-linked .o, it basically uses GNU Make, not automake to make it work > > and therefore breaks automake's "make" portability. > > I think given Boost.Build's "properties" (like "toolset" above), this > should be doable. > > > 1. Get dependency generation in a single pass like automake. This will > > give us about a 30% speed up on builds .. and one of the main reasons > > for moving to automake is taken away. Estimate: 1-2 hours. > > Well, Boost.Build is single pass already. Need to compare performance on > LLVM, though. > > Basically, I can try to finish my attempt to add Boost.Build files to LLVM. > But I wonder if that makes sense? Is it already decided to just improve the > current system?I don't think it is set on stone to keep the current system. We are always open to new ideas and if they improve things we will most likely adopt them. I would say you should proceed on finishing what you have and we can test it and see how it compares with the current system. Just for clarification: the only requirement for Boost.Build is bjam itself, right?> Is anything with "Boost" in name will be rejected right away?Absolutely not! We were using one of the boost libraries before and we replaced it because it was easy to do and it removed one external dependency for us, not because it had the "Boost" in name :-) -- Alkis
Alkis Evlogimenos wrote: [snip]> >>Is anything with "Boost" in name will be rejected right away? > > > Absolutely not! We were using one of the boost libraries before and we > replaced it because it was easy to do and it removed one external dependency > for us, not because it had the "Boost" in name :-) >Another consideration is that we try to limit the number of external tools needed to build LLVM. One difficulty with using a non-make build system is that users would need to download another build tool before building LLVM, and for some, that is too much work. Either that, or we have to include the tool with the LLVM distribution and build it with gmake. So whatever benefits we get from using another build system have to outweigh the inconvenience of an additional external dependency. -- John T. -- ********************************************************************* * John T. Criswell Email: criswell at uiuc.edu * * Research Programmer * * University of Illinois at Urbana-Champaign * * * * "It's today!" said Piglet. "My favorite day," said Pooh. * *********************************************************************
On Thursday 21 October 2004 21:51, Alkis Evlogimenos wrote:> > Boost.Build supports this very good. In particular, on my project on work > > I can run: > > > > bjam toolset=gcc > > bjam toolset=nm > > bjam toolset=nmm > > > > and the first will do a regular build, the third will compile for some > > embedded processor, and the third will pass the sources via some > > annotation tool and produce an annotated x86 binary. > > > > Besides, "toolset=gcc" can become "toolset=msvc" on windows, and it will > > have high chances of working. > > I don't think Reid was referring to the use of different compilers on > (possibly) different platforms. If you look at runtime/GC/SemiSpace: those > files are not compiled using the compiler used to compile the rest of the > project (llc, lli and so on). They use the llvm-gcc-frontend to compile and > link those libraries into llvm bytecode. The resulting files end up in > runtime/GC/SemiSpace/BytecodeObjYes, that should work. That's exactly different compiler: the sources must be compiled with llvm-gcc, not with regular gcc.> > > 2. The entire llvm-test project would have to either be completely > > > rethought or stay the way it is, it just can't be easily automake'd > > > because it doesn't follow the automake pattern. > > > > > > 3. The llvm/test directory would require significant rework to get it > > > to automake correctly (via dejagnu). > > > > No comments on the two points above yet. > > Any ideas if the boost testing framework can apply to these two points?Not yet. Boost frameworks allows to run executable, check if it failed or not, and gather that in a nice table. It's hard to say what hidden problems are there.> > > 6. Creating a distribution tarball would require additional Makefile.am > > > files to be inserted in all the directories under llvm/include, the > > > traversal of which would cause additional overhead for every non-dist > > > target. That's 17 gratuitous "make" processes per build. There doesn't > > > seem to be a way around this. > > > > Why distribution tarball has anything to do with build system? > > This is done in automake. Because the distribution consists of files > processed by automake, the "distcheck" target makes sure that those files > are in the distribution and the distribution tarball can build properly > without needing automake/autoconf.Ah... should not be necessary with Boost.Build.> > Basically, I can try to finish my attempt to add Boost.Build files to > > LLVM. But I wonder if that makes sense? Is it already decided to just > > improve the current system? > > I don't think it is set on stone to keep the current system. We are always > open to new ideas and if they improve things we will most likely adopt > them. I would say you should proceed on finishing what you have and we can > test it and see how it compares with the current system.Thanks. I'll proceed then.> Just for clarification: the only requirement for Boost.Build is bjam > itself, right?Right, plus the Boost.Build implementation files. But since they are in an interpreted language, they pose much less problems. They even can be added to distribution.> > Is anything with "Boost" in name will be rejected right away? > > Absolutely not! We were using one of the boost libraries before and we > replaced it because it was easy to do and it removed one external > dependency for us, not because it had the "Boost" in name :-)Ok, just was just making sure ;-) - Volodya