On Tue, Jun 26, 2018 at 1:28 PM Adrian Prantl <aprantl at apple.com> wrote:> > > > On Jun 26, 2018, at 11:58 AM, Zachary Turner via llvm-dev < > llvm-dev at lists.llvm.org> wrote: > > > > Hi all, > > > > We have been thinking internally about a lightweight llvm-based > ptracer. To address one question up front: the primary way in which this > differs from LLDB is that it targets a more narrow use case -- there is no > scripting support, no clang integration, no dynamic extensibility, no > support for running jitted code in the target, and no user interface. We > have several use cases internally that call for varying levels of > functionality from such a utility, and being able to use as little as > possible of the library as is necessary for the given task is important for > the scale in which we wish to use it. > > > > We are still in early discussions and planning, but I think this would > be a good addition to the LLVM upstream. Since we’re approaching this as a > set of small isolated components, my thinking is to work on this completely > upstream, directly under the llvm project (as opposed to making a separate > subproject), but I’m open to discussion if anyone feels differently. > > > > LLDB has solved a lot of the difficult problems needed for such a tool. > So in the spirit of code reuse, we think it’s worth trying componentize > LLDB by sinking pieces into LLVM and rebasing LLDB as well as these smaller > tools on top of these components, so that smaller tools can reduce code > duplication and contribute to the overall health of the code base. > > Do you have a rough idea of what components specifically the new tool > would need to function? >* process & thread control * platform agnostic ptrace wrapper (not all platforms even have ptrace, and those that do the usage and capabilities vary quite a bit) * install various kinds of traps * monitor cpu performance counters * symbol file parsing * symbol resolution (name <-> addr and line <-> addr) * unwinding and backtrace generation> > > At the same time we think that in doing so we can break things up into > more granular pieces, ultimately exposing a larger testing surface and > enabling us to create exhaustive tests, giving LLDB more fine grained > testing of important subsystems. > > Are you thinking of the new utility as something that would naturally live > in llvm/tools or as something that would live in the LLDB repository? >I would rather put it under LLDB and then link LLDB against certain pieces in cases where that makes sense.> > > > > A good example of this would be LLDB’s DWARF parsing code, which is more > featureful than LLVM’s but has kind of evolved in parallel. Sinking this > into LLVM would be one early target of such an effort, although over time > there would likely be more. > > As you are undoubtedly aware we've been carefully rearchitecting LLVM's > DWARF parser over the last few years to eventually become featureful enough > so that LLDB could use it, so any help on that front would be most welcome. > As long as we are careful to not regress in performance/lazyness, features > and fault-tolerance, deduplicating the implementations can only be good for > LLVM and LLDB. > > Yea, this is the general idea. Has anyone actively been working on thisspecific effort recently? To my knowledge someone started and then never finished, but the efforts also never made it upstream, so my understanding is that it's a goal, but one that nobody has made significant headway on. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20180626/20c97453/attachment.html>
> On Jun 26, 2018, at 1:38 PM, Zachary Turner <zturner at google.com> wrote: > >> On Tue, Jun 26, 2018 at 1:28 PM Adrian Prantl <aprantl at apple.com> wrote: >> >>> > On Jun 26, 2018, at 11:58 AM, Zachary Turner via llvm-dev <llvm-dev at lists.llvm.org> wrote: >>> > A good example of this would be LLDB’s DWARF parsing code, which is more featureful than LLVM’s but has kind of evolved in parallel. Sinking this into LLVM would be one early target of such an effort, although over time there would likely be more. >>> >>> As you are undoubtedly aware we've been carefully rearchitecting LLVM's DWARF parser over the last few years to eventually become featureful enough so that LLDB could use it, so any help on that front would be most welcome. As long as we are careful to not regress in performance/lazyness, features and fault-tolerance, deduplicating the implementations can only be good for LLVM and LLDB. >>> >> Yea, this is the general idea. Has anyone actively been working on this specific effort recently? To my knowledge someone started and then never finished, but the efforts also never made it upstream, so my understanding is that it's a goal, but one that nobody has made significant headway on. >That's not true. Greg Clayton started the effort in 2016 and landed many of the ground-breaking changes. The design ideas fleshed out during that initial effort (thanks to David Blaikie who spent a lot of time reviewing the new interfaces!) such as improved error handling where then picked up the entire team of contributors who worked on DWARF 5 support in LLVM and we've continued down that path ever since. The greatly improved llvm-dwarfdump was also born out of this effort, for example. We also payed attention that every refactoring of LLDB DWARF parser code would bring it closer to the new LLVM parser interface to narrow the gaps between the implementations. -- adrian
Ahh, thanks. I thought those changes never landed, but it's good to hear that they did. On Tue, Jun 26, 2018 at 1:49 PM Adrian Prantl <aprantl at apple.com> wrote:> > > On Jun 26, 2018, at 1:38 PM, Zachary Turner <zturner at google.com> wrote: > > > >> On Tue, Jun 26, 2018 at 1:28 PM Adrian Prantl <aprantl at apple.com> > wrote: > >> > >>> > On Jun 26, 2018, at 11:58 AM, Zachary Turner via llvm-dev < > llvm-dev at lists.llvm.org> wrote: > >>> > A good example of this would be LLDB’s DWARF parsing code, which is > more featureful than LLVM’s but has kind of evolved in parallel. Sinking > this into LLVM would be one early target of such an effort, although over > time there would likely be more. > >>> > >>> As you are undoubtedly aware we've been carefully rearchitecting > LLVM's DWARF parser over the last few years to eventually become featureful > enough so that LLDB could use it, so any help on that front would be most > welcome. As long as we are careful to not regress in performance/lazyness, > features and fault-tolerance, deduplicating the implementations can only be > good for LLVM and LLDB. > >>> > >> Yea, this is the general idea. Has anyone actively been working on > this specific effort recently? To my knowledge someone started and then > never finished, but the efforts also never made it upstream, so my > understanding is that it's a goal, but one that nobody has made significant > headway on. > > > That's not true. Greg Clayton started the effort in 2016 and landed many > of the ground-breaking changes. The design ideas fleshed out during that > initial effort (thanks to David Blaikie who spent a lot of time reviewing > the new interfaces!) such as improved error handling where then picked up > the entire team of contributors who worked on DWARF 5 support in LLVM and > we've continued down that path ever since. The greatly improved > llvm-dwarfdump was also born out of this effort, for example. We also payed > attention that every refactoring of LLDB DWARF parser code would bring it > closer to the new LLVM parser interface to narrow the gaps between the > implementations. > > -- adrian > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20180626/934f3562/attachment.html>
One important question, does this tool need to work remotely? I'm guessing the answer to this is no, since if you are working remotely you won't have a performant enough solution to really be an effective tracer. And if the guts of the debugger are remote, you care a lot less about the complexity of the remote part. If you can always debug with the Host platform - in lldb terms - then it really does seem like you want to start with the NativeProcess classes. That won't get you macOS hosting, but OTOH this would be good reason to get macOS onto the NativeProcess classes/lldb-server and off of debugserver...> On Jun 26, 2018, at 1:38 PM, Zachary Turner via lldb-dev <lldb-dev at lists.llvm.org> wrote: > > > > On Tue, Jun 26, 2018 at 1:28 PM Adrian Prantl <aprantl at apple.com> wrote: > > > > On Jun 26, 2018, at 11:58 AM, Zachary Turner via llvm-dev <llvm-dev at lists.llvm.org> wrote: > > > > Hi all, > > > > We have been thinking internally about a lightweight llvm-based ptracer. To address one question up front: the primary way in which this differs from LLDB is that it targets a more narrow use case -- there is no scripting support, no clang integration, no dynamic extensibility, no support for running jitted code in the target, and no user interface. We have several use cases internally that call for varying levels of functionality from such a utility, and being able to use as little as possible of the library as is necessary for the given task is important for the scale in which we wish to use it. > > > > We are still in early discussions and planning, but I think this would be a good addition to the LLVM upstream. Since we’re approaching this as a set of small isolated components, my thinking is to work on this completely upstream, directly under the llvm project (as opposed to making a separate subproject), but I’m open to discussion if anyone feels differently. > > > > LLDB has solved a lot of the difficult problems needed for such a tool. So in the spirit of code reuse, we think it’s worth trying componentize LLDB by sinking pieces into LLVM and rebasing LLDB as well as these smaller tools on top of these components, so that smaller tools can reduce code duplication and contribute to the overall health of the code base. > > Do you have a rough idea of what components specifically the new tool would need to function? > > * process & thread control > * platform agnostic ptrace wrapper (not all platforms even have ptrace, and those that do the usage and capabilities vary quite a bit) > * install various kinds of traps > * monitor cpu performance countersThis part is all the job of the NativeProcess classes. That's not terribly surprising, since their whole reason for being was as a low-level abstraction for process control without any of the higher-level work that lldb does.> * symbol file parsing > * symbol resolution (name <-> addr and line <-> addr)This will involve getting object-file readers and a symbol file reader into your trace tool. These should be pretty easy to extract from lldb, though you probably don't need the plugin architecture.> * unwinding and backtrace generationJason says this will be somewhat tricky to pull out of lldb. OTOH much of the complexity of unwind is reconstructing all the non-volatile registers, and if you don't care about values, you don't really need that. So some kind of lightweight pc/sp only backtrace would be more appropriate, and probably faster for your needs. Jim> > > > > At the same time we think that in doing so we can break things up into more granular pieces, ultimately exposing a larger testing surface and enabling us to create exhaustive tests, giving LLDB more fine grained testing of important subsystems. > > Are you thinking of the new utility as something that would naturally live in llvm/tools or as something that would live in the LLDB repository? > I would rather put it under LLDB and then link LLDB against certain pieces in cases where that makes sense. > > > > > > A good example of this would be LLDB’s DWARF parsing code, which is more featureful than LLVM’s but has kind of evolved in parallel. Sinking this into LLVM would be one early target of such an effort, although over time there would likely be more. > > As you are undoubtedly aware we've been carefully rearchitecting LLVM's DWARF parser over the last few years to eventually become featureful enough so that LLDB could use it, so any help on that front would be most welcome. As long as we are careful to not regress in performance/lazyness, features and fault-tolerance, deduplicating the implementations can only be good for LLVM and LLDB. > > Yea, this is the general idea. Has anyone actively been working on this specific effort recently? To my knowledge someone started and then never finished, but the efforts also never made it upstream, so my understanding is that it's a goal, but one that nobody has made significant headway on. > _______________________________________________ > lldb-dev mailing list > lldb-dev at lists.llvm.org > http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
> On Jun 26, 2018, at 2:00 PM, Jim Ingham via lldb-dev <lldb-dev at lists.llvm.org> wrote: > > >> * unwinding and backtrace generation > > Jason says this will be somewhat tricky to pull out of lldb. OTOH much of the complexity of unwind is reconstructing all the non-volatile registers, and if you don't care about values, you don't really need that. So some kind of lightweight pc/sp only backtrace would be more appropriate, and probably faster for your needs.If it were me & performance were the utmost concern, and I had a restricted platform set that I needed to support where I can assume the presence of eh_frame and that it is trustworthy in prologue/epilogues, then I'd probably just write a simple Unwind/RegisterContext plugin pair that exclusively live off of that. If it's just stack walking, and we can assume no omit-frame-pointer code and we can assume the 0th function is always stopped in a non-prologue/epilogue location, then even simpler would be the old RegisterContextMacOSXFrameBackchain plugin would get you there. That's what we used before we had the modern unwind/registercontext plugin that we use today. It doesn't track spilled registers at all, it just looks for saved pc/framepointer values on the stack. A general problem with stopping the inferior process and examining things is that it is slow. Even if you use a NativeHost approach and get debugserver/lldb-server out of the equation, if you stop in a hot location it's very difficult to make this performant. We've prototyped things like this in the past and it was always far too slow. I don't know what your use case looks like, but I do worry about having one process controlling an inferior process in general for fast-turnaround data collection/experiments, it doesn't seem like the best way to go about it.> > Jim > >> >> >> >>> At the same time we think that in doing so we can break things up into more granular pieces, ultimately exposing a larger testing surface and enabling us to create exhaustive tests, giving LLDB more fine grained testing of important subsystems. >> >> Are you thinking of the new utility as something that would naturally live in llvm/tools or as something that would live in the LLDB repository? >> I would rather put it under LLDB and then link LLDB against certain pieces in cases where that makes sense. >> >> >>> >>> A good example of this would be LLDB’s DWARF parsing code, which is more featureful than LLVM’s but has kind of evolved in parallel. Sinking this into LLVM would be one early target of such an effort, although over time there would likely be more. >> >> As you are undoubtedly aware we've been carefully rearchitecting LLVM's DWARF parser over the last few years to eventually become featureful enough so that LLDB could use it, so any help on that front would be most welcome. As long as we are careful to not regress in performance/lazyness, features and fault-tolerance, deduplicating the implementations can only be good for LLVM and LLDB. >> >> Yea, this is the general idea. Has anyone actively been working on this specific effort recently? To my knowledge someone started and then never finished, but the efforts also never made it upstream, so my understanding is that it's a goal, but one that nobody has made significant headway on. >> _______________________________________________ >> lldb-dev mailing list >> lldb-dev at lists.llvm.org >> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev > > _______________________________________________ > lldb-dev mailing list > lldb-dev at lists.llvm.org > http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev