Chandler Carruth
2015-Jan-20 02:33 UTC
[LLVMdev] Can we establish layering for the LLD libraries? Current state is a bit of a mess...
I wanted to go through and map out the layering of LLD's libraries today and found that it's essentially impossible. I think some serious cleanup is needed here. Let's start with the purely link-level dependencies encoded in the CMake build: Curently the Core library depends on the ReaderWriter/Native library, which links against the ReaderWriter library, which links against the Core library. This clearly cannot work. The same cycle exists with Core -> YAML -> ReaderWrite -> Core. The situation seems a bit worse for includes. If you start from LinkingContext.h I think this becomes quite clear. This is ostensibly part of the Core library, but it has methods that produce types from the ReaderWriter library. Combined with the fact that ReaderWriter depends on Core (not the other way around) and ReaderWriter/ELF subclasses LinkingContext, I can't make heads or tails of what was intended here. My vague guess is that Core should actually be two libraries. One that doesn't depend on anything (other than Config) and provides the very "core" types for LLD. And another, perhaps called "Linker" which is a much higher-level library and provides useful APIs for actually doing linking, depends on ReaderWriter and provides methods that manipulate it. I could even see needing to spilt the target libraries in a similar manner. But I don't know LLD's design well, I'm just trying to stitch the build system back together in a reasonable way, so maybe I've missed things completely. So help me out. I'd like to understand a reasonable DAG in which to construct libraries for LLD. Having this should allow proper layering, layering checks, and eventually building with C++ modules. All of which seem essentially impossible today. -Chandler -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20150119/bd88d1b0/attachment.html>
Greg Fitzgerald
2015-Jan-21 00:02 UTC
[LLVMdev] Can we establish layering for the LLD libraries? Current state is a bit of a mess...
Hi Chandler,> Let's start with the purely link-level dependencies encoded in the CMake buildI've been playing around a bit with this. Looks like we can solve most of the issues by dissolving lldReaderWriter. Reader.cpp/Writer.cpp go into lldCore and the rest go into lldDriver. I don't know if that makes conceptual sense, but it's what breaks the cyclic dependencies. -Greg On Mon, Jan 19, 2015 at 6:33 PM, Chandler Carruth <chandlerc at google.com> wrote:> I wanted to go through and map out the layering of LLD's libraries today and > found that it's essentially impossible. I think some serious cleanup is > needed here. > > Let's start with the purely link-level dependencies encoded in the CMake > build: > > Curently the Core library depends on the ReaderWriter/Native library, which > links against the ReaderWriter library, which links against the Core > library. This clearly cannot work. The same cycle exists with Core -> YAML > -> ReaderWrite -> Core. > > > The situation seems a bit worse for includes. If you start from > LinkingContext.h I think this becomes quite clear. This is ostensibly part > of the Core library, but it has methods that produce types from the > ReaderWriter library. Combined with the fact that ReaderWriter depends on > Core (not the other way around) and ReaderWriter/ELF subclasses > LinkingContext, I can't make heads or tails of what was intended here. > > > My vague guess is that Core should actually be two libraries. One that > doesn't depend on anything (other than Config) and provides the very "core" > types for LLD. And another, perhaps called "Linker" which is a much > higher-level library and provides useful APIs for actually doing linking, > depends on ReaderWriter and provides methods that manipulate it. I could > even see needing to spilt the target libraries in a similar manner. > > > But I don't know LLD's design well, I'm just trying to stitch the build > system back together in a reasonable way, so maybe I've missed things > completely. So help me out. I'd like to understand a reasonable DAG in which > to construct libraries for LLD. Having this should allow proper layering, > layering checks, and eventually building with C++ modules. All of which seem > essentially impossible today. > > -Chandler > > _______________________________________________ > LLVM Developers mailing list > LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev >
Chandler Carruth
2015-Jan-21 00:19 UTC
[LLVMdev] Can we establish layering for the LLD libraries? Current state is a bit of a mess...
On Tue, Jan 20, 2015 at 4:02 PM, Greg Fitzgerald <garious at gmail.com> wrote:> Hi Chandler, > > > Let's start with the purely link-level dependencies encoded in the CMake > build > > I've been playing around a bit with this. Looks like we can solve > most of the issues by dissolving lldReaderWriter. > Reader.cpp/Writer.cpp go into lldCore and the rest go into lldDriver. > I don't know if that makes conceptual sense, but it's what breaks the > cyclic dependencies. >Does that break just the *link time* dependencies? If you form a DAG of cross-library header inclusion and a DAG of the link time dependencies, they should be compatible, and that's the layering we should use. I don't know much about LLD -- maybe it would help for you to describe the DAG you're envisioning? -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20150120/00456a8a/attachment.html>
Nick Kledzik
2015-Jan-21 01:35 UTC
[LLVMdev] Can we establish layering for the LLD libraries? Current state is a bit of a mess...
On Jan 19, 2015, at 6:33 PM, Chandler Carruth <chandlerc at google.com> wrote:> I wanted to go through and map out the layering of LLD's libraries today and found that it's essentially impossible. I think some serious cleanup is needed here. > > Let's start with the purely link-level dependencies encoded in the CMake build: > > Curently the Core library depends on the ReaderWriter/Native library, which links against the ReaderWriter library, which links against the Core library. This clearly cannot work. The same cycle exists with Core -> YAML -> ReaderWrite -> Core.How are you determining these cycles? How is Core dependent on YAML? (cause that seems wrong). I just build all of lld in an Xcode projects (which compiles each .o file and links them all together). I never see any of the layering...> The situation seems a bit worse for includes. If you start from LinkingContext.h I think this becomes quite clear. This is ostensibly part of the Core library, but it has methods that produce types from the ReaderWriter library. Combined with the fact that ReaderWriter depends on Core (not the other way around) and ReaderWriter/ELF subclasses LinkingContext, I can't make heads or tails of what was intended here. > > > My vague guess is that Core should actually be two libraries. One that doesn't depend on anything (other than Config) and provides the very "core" types for LLD. And another, perhaps called "Linker" which is a much higher-level library and provides useful APIs for actually doing linking, depends on ReaderWriter and provides methods that manipulate it. I could even see needing to spilt the target libraries in a similar manner. > > > But I don't know LLD's design well, I'm just trying to stitch the build system back together in a reasonable way, so maybe I've missed things completely. So help me out. I'd like to understand a reasonable DAG in which to construct libraries for LLD. Having this should allow proper layering, layering checks, and eventually building with C++ modules. All of which seem essentially impossible today.One question I have is what should the granularity of the libraries be? At one point I asked about building an lld for OSX that only had mach-o support and got push-back that lld should always be a universal linker. If that is the case, why have so many small libraries? Besides lld the command line tool, I can see lld libraries being used to construct a JIT linker. And in that case, I can see wanting a very pared down linker - just one file format (not yaml) and no driver.
Chandler Carruth
2015-Jan-21 01:42 UTC
[LLVMdev] Can we establish layering for the LLD libraries? Current state is a bit of a mess...
On Tue, Jan 20, 2015 at 5:35 PM, Nick Kledzik <kledzik at apple.com> wrote:> On Jan 19, 2015, at 6:33 PM, Chandler Carruth <chandlerc at google.com> > wrote: > > > I wanted to go through and map out the layering of LLD's libraries today > and found that it's essentially impossible. I think some serious cleanup is > needed here. > > > > Let's start with the purely link-level dependencies encoded in the CMake > build: > > > > Curently the Core library depends on the ReaderWriter/Native library, > which links against the ReaderWriter library, which links against the Core > library. This clearly cannot work. The same cycle exists with Core -> YAML > -> ReaderWrite -> Core. > How are you determining these cycles? How is Core dependent on YAML? > (cause that seems wrong). >This is in the CMake build system today. You can see it with CMakeLists.txt. Also you can look at Greg's patch for some of the issues here.> > I just build all of lld in an Xcode projects (which compiles each .o file > and links them all together). I never see any of the layering... >The CMake build works on Mac OS X too. =] With Ninja, it is even quite speedy (or so I understand). I'm hopeful in the not-too-distant-future we'll actually start enforcing even header file layering using Clang's Module's support (when it happens to be the host compiler). When that happens this stuff should actually be really explicit and enforced easily. Until then, I also have access to some crazy (and sadly internal) tools that check header file layering between libraries. Hopefully we just modules soon though.> > > The situation seems a bit worse for includes. If you start from > LinkingContext.h I think this becomes quite clear. This is ostensibly part > of the Core library, but it has methods that produce types from the > ReaderWriter library. Combined with the fact that ReaderWriter depends on > Core (not the other way around) and ReaderWriter/ELF subclasses > LinkingContext, I can't make heads or tails of what was intended here. > > > > > > My vague guess is that Core should actually be two libraries. One that > doesn't depend on anything (other than Config) and provides the very "core" > types for LLD. And another, perhaps called "Linker" which is a much > higher-level library and provides useful APIs for actually doing linking, > depends on ReaderWriter and provides methods that manipulate it. I could > even see needing to spilt the target libraries in a similar manner. > > > > > > But I don't know LLD's design well, I'm just trying to stitch the build > system back together in a reasonable way, so maybe I've missed things > completely. So help me out. I'd like to understand a reasonable DAG in > which to construct libraries for LLD. Having this should allow proper > layering, layering checks, and eventually building with C++ modules. All of > which seem essentially impossible today. > > One question I have is what should the granularity of the libraries be? > At one point I asked about building an lld for OSX that only had mach-o > support and got push-back that lld should always be a universal linker. If > that is the case, why have so many small libraries? > > Besides lld the command line tool, I can see lld libraries being used to > construct a JIT linker. And in that case, I can see wanting a very pared > down linker - just one file format (not yaml) and no driver.So, I think there is a good rationale for supporting both fine grained libraries and LLD always being a universal / cross linker. The libraries make it *possible* (as you indicate with the JIT stuff) to build a narrowly targeted linker that has no excess functionality. We should still choose to produce a full cross-linker binary called "lld" which uses all the libraries. Does that make sense? While I think that generally layering also helps abstractly organize the code, I don't think it is worth thinking about layering that serves no purpose. So, if it isn't *possible* to use library A without using library B as well, it doesn't make sense to separate them. I'm happy to separate them if it is possible even if the users for this separation haven't yet materialized. -Chandler -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20150120/277e243a/attachment.html>
Joerg Sonnenberger
2015-Jan-21 16:10 UTC
[LLVMdev] Can we establish layering for the LLD libraries? Current state is a bit of a mess...
On Tue, Jan 20, 2015 at 05:35:53PM -0800, Nick Kledzik wrote:> How are you determining these cycles? How is Core dependent on YAML? > (cause that seems wrong).Try building LLVM + LLD with shared libraries on Linux. That will reliably fail due to dependency cycles between the components. Joerg
Seemingly Similar Threads
- [LLVMdev] Can we establish layering for the LLD libraries? Current state is a bit of a mess...
- [LLVMdev] Can we establish layering for the LLD libraries? Current state is a bit of a mess...
- [LLVMdev] [lld] LLD's software architecture
- [LLVMdev] [lld] LLD's software architecture (update)
- [LLVMdev] [lld] LLD's software architecture (update)