Hi all, We have been thinking internally about a lightweight llvm-based ptracer. To address one question up front: the primary way in which this differs from LLDB is that it targets a more narrow use case -- there is no scripting support, no clang integration, no dynamic extensibility, no support for running jitted code in the target, and no user interface. We have several use cases internally that call for varying levels of functionality from such a utility, and being able to use as little as possible of the library as is necessary for the given task is important for the scale in which we wish to use it. We are still in early discussions and planning, but I think this would be a good addition to the LLVM upstream. Since we’re approaching this as a set of small isolated components, my thinking is to work on this completely upstream, directly under the llvm project (as opposed to making a separate subproject), but I’m open to discussion if anyone feels differently. LLDB has solved a lot of the difficult problems needed for such a tool. So in the spirit of code reuse, we think it’s worth trying componentize LLDB by sinking pieces into LLVM and rebasing LLDB as well as these smaller tools on top of these components, so that smaller tools can reduce code duplication and contribute to the overall health of the code base. At the same time we think that in doing so we can break things up into more granular pieces, ultimately exposing a larger testing surface and enabling us to create exhaustive tests, giving LLDB more fine grained testing of important subsystems. A good example of this would be LLDB’s DWARF parsing code, which is more featureful than LLVM’s but has kind of evolved in parallel. Sinking this into LLVM would be one early target of such an effort, although over time there would likely be more. Anyone have any thoughts / strong opinions on this proposal, or where the code should live? Also, does anyone have any suggestions on things they’d like to see come out of this? Whether it’s a specific new tool, new functionality to an existing tool, an architectural or design change to some existing tool or library, or something else entirely, all feedback and ideas are welcome. Thanks, Zach -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20180626/5aa395d5/attachment.html>
Just to be clear, by "no clang integration" do you mean "no expression parser" or do you mean something more radical? For instance, adding a TypeSystem and its DWARF parser for C family languages that uses a different underlying representation than Clang AST's to store the results would be a lot of work that wouldn't be terribly interesting to lldb. I don't think that's what you meant, but wanted to be sure. Jim> On Jun 26, 2018, at 11:58 AM, Zachary Turner via lldb-dev <lldb-dev at lists.llvm.org> wrote: > > Hi all, > > We have been thinking internally about a lightweight llvm-based ptracer. To address one question up front: the primary way in which this differs from LLDB is that it targets a more narrow use case -- there is no scripting support, no clang integration, no dynamic extensibility, no support for running jitted code in the target, and no user interface. We have several use cases internally that call for varying levels of functionality from such a utility, and being able to use as little as possible of the library as is necessary for the given task is important for the scale in which we wish to use it. > > We are still in early discussions and planning, but I think this would be a good addition to the LLVM upstream. Since we’re approaching this as a set of small isolated components, my thinking is to work on this completely upstream, directly under the llvm project (as opposed to making a separate subproject), but I’m open to discussion if anyone feels differently. > > LLDB has solved a lot of the difficult problems needed for such a tool. So in the spirit of code reuse, we think it’s worth trying componentize LLDB by sinking pieces into LLVM and rebasing LLDB as well as these smaller tools on top of these components, so that smaller tools can reduce code duplication and contribute to the overall health of the code base. At the same time we think that in doing so we can break things up into more granular pieces, ultimately exposing a larger testing surface and enabling us to create exhaustive tests, giving LLDB more fine grained testing of important subsystems. > > A good example of this would be LLDB’s DWARF parsing code, which is more featureful than LLVM’s but has kind of evolved in parallel. Sinking this into LLVM would be one early target of such an effort, although over time there would likely be more. > > Anyone have any thoughts / strong opinions on this proposal, or where the code should live? Also, does anyone have any suggestions on things they’d like to see come out of this? Whether it’s a specific new tool, new functionality to an existing tool, an architectural or design change to some existing tool or library, or something else entirely, all feedback and ideas are welcome. > > Thanks, > Zach > > _______________________________________________ > lldb-dev mailing list > lldb-dev at lists.llvm.org > http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
no expression parser or knowledge of any specific programming language. Basically I just mean that the parsing of the native DWARF format itself is in scope, but anything beyond that is out of scope. For symbolication we have things like llvm-symbolizer that already just work and are built on top of LLVM's dwarf parsing code. Similarly, LLDB's type system could be built on top of it as well. Given that I think everyone mostly agrees that unifying on one DWARF parser is a good idea in principle, this would mean no functional change from LLDB's point of view, it would just continue to do exactly what it does regarding parsing C++ expressions and converting these into types that clang understands. It will probably be useful someday to have an expression parser and language specific type system, but when that comes I don't think we'd want anything radically different than what LLDB already has. On Tue, Jun 26, 2018 at 12:26 PM Jim Ingham <jingham at apple.com> wrote:> Just to be clear, by "no clang integration" do you mean "no expression > parser" or do you mean something more radical? For instance, adding a > TypeSystem and its DWARF parser for C family languages that uses a > different underlying representation than Clang AST's to store the results > would be a lot of work that wouldn't be terribly interesting to lldb. I > don't think that's what you meant, but wanted to be sure. > > Jim > > > On Jun 26, 2018, at 11:58 AM, Zachary Turner via lldb-dev < > lldb-dev at lists.llvm.org> wrote: > > > > Hi all, > > > > We have been thinking internally about a lightweight llvm-based > ptracer. To address one question up front: the primary way in which this > differs from LLDB is that it targets a more narrow use case -- there is no > scripting support, no clang integration, no dynamic extensibility, no > support for running jitted code in the target, and no user interface. We > have several use cases internally that call for varying levels of > functionality from such a utility, and being able to use as little as > possible of the library as is necessary for the given task is important for > the scale in which we wish to use it. > > > > We are still in early discussions and planning, but I think this would > be a good addition to the LLVM upstream. Since we’re approaching this as a > set of small isolated components, my thinking is to work on this completely > upstream, directly under the llvm project (as opposed to making a separate > subproject), but I’m open to discussion if anyone feels differently. > > > > LLDB has solved a lot of the difficult problems needed for such a tool. > So in the spirit of code reuse, we think it’s worth trying componentize > LLDB by sinking pieces into LLVM and rebasing LLDB as well as these smaller > tools on top of these components, so that smaller tools can reduce code > duplication and contribute to the overall health of the code base. At the > same time we think that in doing so we can break things up into more > granular pieces, ultimately exposing a larger testing surface and enabling > us to create exhaustive tests, giving LLDB more fine grained testing of > important subsystems. > > > > A good example of this would be LLDB’s DWARF parsing code, which is more > featureful than LLVM’s but has kind of evolved in parallel. Sinking this > into LLVM would be one early target of such an effort, although over time > there would likely be more. > > > > Anyone have any thoughts / strong opinions on this proposal, or where > the code should live? Also, does anyone have any suggestions on things > they’d like to see come out of this? Whether it’s a specific new tool, new > functionality to an existing tool, an architectural or design change to > some existing tool or library, or something else entirely, all feedback and > ideas are welcome. > > > > Thanks, > > Zach > > > > _______________________________________________ > > lldb-dev mailing list > > lldb-dev at lists.llvm.org > > http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20180626/08b099f2/attachment.html>
> On Jun 26, 2018, at 11:58 AM, Zachary Turner via llvm-dev <llvm-dev at lists.llvm.org> wrote: > > Hi all, > > We have been thinking internally about a lightweight llvm-based ptracer. To address one question up front: the primary way in which this differs from LLDB is that it targets a more narrow use case -- there is no scripting support, no clang integration, no dynamic extensibility, no support for running jitted code in the target, and no user interface. We have several use cases internally that call for varying levels of functionality from such a utility, and being able to use as little as possible of the library as is necessary for the given task is important for the scale in which we wish to use it. > > We are still in early discussions and planning, but I think this would be a good addition to the LLVM upstream. Since we’re approaching this as a set of small isolated components, my thinking is to work on this completely upstream, directly under the llvm project (as opposed to making a separate subproject), but I’m open to discussion if anyone feels differently. > > LLDB has solved a lot of the difficult problems needed for such a tool. So in the spirit of code reuse, we think it’s worth trying componentize LLDB by sinking pieces into LLVM and rebasing LLDB as well as these smaller tools on top of these components, so that smaller tools can reduce code duplication and contribute to the overall health of the code base.Do you have a rough idea of what components specifically the new tool would need to function?> At the same time we think that in doing so we can break things up into more granular pieces, ultimately exposing a larger testing surface and enabling us to create exhaustive tests, giving LLDB more fine grained testing of important subsystems.Are you thinking of the new utility as something that would naturally live in llvm/tools or as something that would live in the LLDB repository?> > A good example of this would be LLDB’s DWARF parsing code, which is more featureful than LLVM’s but has kind of evolved in parallel. Sinking this into LLVM would be one early target of such an effort, although over time there would likely be more.As you are undoubtedly aware we've been carefully rearchitecting LLVM's DWARF parser over the last few years to eventually become featureful enough so that LLDB could use it, so any help on that front would be most welcome. As long as we are careful to not regress in performance/lazyness, features and fault-tolerance, deduplicating the implementations can only be good for LLVM and LLDB. -- adrian> > Anyone have any thoughts / strong opinions on this proposal, or where the code should live? Also, does anyone have any suggestions on things they’d like to see come out of this? Whether it’s a specific new tool, new functionality to an existing tool, an architectural or design change to some existing tool or library, or something else entirely, all feedback and ideas are welcome. > > Thanks, > Zach > > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
On Tue, Jun 26, 2018 at 1:28 PM Adrian Prantl <aprantl at apple.com> wrote:> > > > On Jun 26, 2018, at 11:58 AM, Zachary Turner via llvm-dev < > llvm-dev at lists.llvm.org> wrote: > > > > Hi all, > > > > We have been thinking internally about a lightweight llvm-based > ptracer. To address one question up front: the primary way in which this > differs from LLDB is that it targets a more narrow use case -- there is no > scripting support, no clang integration, no dynamic extensibility, no > support for running jitted code in the target, and no user interface. We > have several use cases internally that call for varying levels of > functionality from such a utility, and being able to use as little as > possible of the library as is necessary for the given task is important for > the scale in which we wish to use it. > > > > We are still in early discussions and planning, but I think this would > be a good addition to the LLVM upstream. Since we’re approaching this as a > set of small isolated components, my thinking is to work on this completely > upstream, directly under the llvm project (as opposed to making a separate > subproject), but I’m open to discussion if anyone feels differently. > > > > LLDB has solved a lot of the difficult problems needed for such a tool. > So in the spirit of code reuse, we think it’s worth trying componentize > LLDB by sinking pieces into LLVM and rebasing LLDB as well as these smaller > tools on top of these components, so that smaller tools can reduce code > duplication and contribute to the overall health of the code base. > > Do you have a rough idea of what components specifically the new tool > would need to function? >* process & thread control * platform agnostic ptrace wrapper (not all platforms even have ptrace, and those that do the usage and capabilities vary quite a bit) * install various kinds of traps * monitor cpu performance counters * symbol file parsing * symbol resolution (name <-> addr and line <-> addr) * unwinding and backtrace generation> > > At the same time we think that in doing so we can break things up into > more granular pieces, ultimately exposing a larger testing surface and > enabling us to create exhaustive tests, giving LLDB more fine grained > testing of important subsystems. > > Are you thinking of the new utility as something that would naturally live > in llvm/tools or as something that would live in the LLDB repository? >I would rather put it under LLDB and then link LLDB against certain pieces in cases where that makes sense.> > > > > A good example of this would be LLDB’s DWARF parsing code, which is more > featureful than LLVM’s but has kind of evolved in parallel. Sinking this > into LLVM would be one early target of such an effort, although over time > there would likely be more. > > As you are undoubtedly aware we've been carefully rearchitecting LLVM's > DWARF parser over the last few years to eventually become featureful enough > so that LLDB could use it, so any help on that front would be most welcome. > As long as we are careful to not regress in performance/lazyness, features > and fault-tolerance, deduplicating the implementations can only be good for > LLVM and LLDB. > > Yea, this is the general idea. Has anyone actively been working on thisspecific effort recently? To my knowledge someone started and then never finished, but the efforts also never made it upstream, so my understanding is that it's a goal, but one that nobody has made significant headway on. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20180626/20c97453/attachment.html>