On Wed, 27 Jun 2018 at 01:14, Zachary Turner via lldb-dev <lldb-dev at lists.llvm.org> wrote:> > Yes that’s what I’ve been thinking about as well. > > One thing I’ve been giving a lot of thought to is whether to serialize the handling of trace events. I want to balance the “this is a library and you should be able to get it to work for you no matter what your use case is” aspect with the “you really just don’t want to go there, we know what’s best for you” aspect. Then there’s the fact that not all platforms behave the same, but we’d like a consistent set of expectations that makes it easy to use for everyone. > > So I’m leaning towards having the library serialize all tace events, because it’s a nice common denominator that every platform can implement. > > To be clear though, I don’t mean that if 2 processes are being traced simultaneously and A stops followed by B stopping, then the tool will necessarily block before handling B’s stop. I just mean that A and B’s stop handlers will be invoked on a single thread (not the threads which are tracing A or B). > > So A stops, posts its stop event on the blessed thread and waits. Then B stops and does the same thing. A’s handler runs, for whatever reason decides it will continue later, saves off the event somewhere, then processes B’s. Later something happens, it decides to continue A, signals A’s thread which wakes up. > > I think this kind of design eliminates a large class of race conditions without sacrificing any performance. >Does this mean that you will always have to have at least two threads (the one doing the tracing and the one where stop handlers are invoked)? Because if that's true, then I'm not sure I buy the no-performance-sacrifice part. Given that with ptrace (on linux at least, but I think that holds for some other OSs too), all debugging operations have to happen on a specific thread, if that thread is not the one where the core logic happens, you will have to do a lot of ping-pong to do all the debugging operations (read/write registers/memory, set breakpoints, etc.). Of all the use cases, the one where this matters most may be actually yours -- I'm not sure I understand it fully but if the goal is to have as little impact on the traced process, then this is going to be a problem, because every microsecond you spend context-switching between these two threads is a microsecond when the target process is not executing. In lldb-server we avoid these context switches (and race conditions!) by being single threaded. It think it would be good to keep things this way by having the new api (the lowest layers of it?) accessible in a single-threaded manner, at least on platforms where this is possible (everything except windows, I guess).
suppose process A (single threaded) is tracing process B (2 threads). If trace events happen on both threads of B, then the second thread can’t continue until both threads’ trace events have been fully handled, synchronously. If process A has a second thread though, the tracer thread can enqueue work via a lock free queue (or worst case scenario, a mutex), and continue immediately. So it seems less overhead this way. That said, there seems to be no harm in exposing the lowest levels of the API with all of their os specific quirks, and one could be built on top that standardizes the assumptions and requirements On Wed, Jun 27, 2018 at 12:56 AM Pavel Labath <labath at google.com> wrote:> On Wed, 27 Jun 2018 at 01:14, Zachary Turner via lldb-dev > <lldb-dev at lists.llvm.org> wrote: > > > > Yes that’s what I’ve been thinking about as well. > > > > One thing I’ve been giving a lot of thought to is whether to serialize > the handling of trace events. I want to balance the “this is a library and > you should be able to get it to work for you no matter what your use case > is” aspect with the “you really just don’t want to go there, we know what’s > best for you” aspect. Then there’s the fact that not all platforms behave > the same, but we’d like a consistent set of expectations that makes it easy > to use for everyone. > > > > So I’m leaning towards having the library serialize all tace events, > because it’s a nice common denominator that every platform can implement. > > > > To be clear though, I don’t mean that if 2 processes are being traced > simultaneously and A stops followed by B stopping, then the tool will > necessarily block before handling B’s stop. I just mean that A and B’s > stop handlers will be invoked on a single thread (not the threads which are > tracing A or B). > > > > So A stops, posts its stop event on the blessed thread and waits. Then > B stops and does the same thing. A’s handler runs, for whatever reason > decides it will continue later, saves off the event somewhere, then > processes B’s. Later something happens, it decides to continue A, signals > A’s thread which wakes up. > > > > I think this kind of design eliminates a large class of race conditions > without sacrificing any performance. > > > > Does this mean that you will always have to have at least two threads > (the one doing the tracing and the one where stop handlers are > invoked)? Because if that's true, then I'm not sure I buy the > no-performance-sacrifice part. Given that with ptrace (on linux at > least, but I think that holds for some other OSs too), all debugging > operations have to happen on a specific thread, if that thread is not > the one where the core logic happens, you will have to do a lot of > ping-pong to do all the debugging operations (read/write > registers/memory, set breakpoints, etc.). Of all the use cases, the > one where this matters most may be actually yours -- I'm not sure I > understand it fully but if the goal is to have as little impact on the > traced process, then this is going to be a problem, because every > microsecond you spend context-switching between these two threads is a > microsecond when the target process is not executing. In lldb-server > we avoid these context switches (and race conditions!) by being single > threaded. It think it would be good to keep things this way by having > the new api (the lowest layers of it?) accessible in a single-threaded > manner, at least on platforms where this is possible (everything > except windows, I guess). >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20180627/18849310/attachment.html>
On Wed, 27 Jun 2018 at 14:11, Zachary Turner <zturner at google.com> wrote:> > suppose process A (single threaded) is tracing process B (2 threads). If trace events happen on both threads of B, then the second thread can’t continue until both threads’ trace events have been fully handled, synchronously. If process A has a second thread though, the tracer thread can enqueue work via a lock free queue (or worst case scenario, a mutex), and continue immediately. So it seems less overhead this way.I think that depends a lot on the use case. I can certainly see how multithreading would be beneficial if you have a lot of work to do that can be done asynchronously. However, there isn't a ton of other work in lldb-server. We always either wait for the process to stop (in which case we quickly want to gather information and notify the client about that), or we wait for a command from the client (in which case we want to quickly execute it). Threading doesn't help either of those cases. If your tracer is doing some kind of asynchronous processing of the process events (during which the inferior process can continue running) then offloading that to a background thread makes sense. However, even in that case, I think that the collection of the actual data needed for the background processing would be best done synchronously on the ptrace thread because: a) there will be no context switching involved; b) It allows the client to specify exactly the kind of data it wants to collect. Then the collected data can be enqueued to the background thread or whatever, but this does not need to be too tightly integrated with the core APIs.