llvm dev - Aug 2018 - DExTer: Debugging Experience Tester

I'm happy to announce that DExTer (Debugging Experience Tester) is now
available on github at https://github.com/SNSystems/dexter .  To quote from
the project readme:

"DExTer is a suite of tools used to evaluate the "User Debugging
Experience". DExTer drives an external debugger, running on small test
programs, and collects information on the behavior at each debugger step to
provide quantitative values that indicate the quality of the debugging
experience."

My hope is that it will be a useful tool to help find both existing cases
of poor debug experience as well as spot any regressions in that experience
as they occur.  As I've previously stated, I think that this should be
nicely complementary to other utlities such as debugify
<http://llvm.org/docs/SourceLevelDebugging.html#testing-debug-info-preservation-in-optimizations>
and
DIVA <https://github.com/SNSystems/DIVA>. We also intend to contribute
continuous DExTer testing of LLVM trunk in the future so we'll be able to
immediately track any differences in the debugging experience score down to
individual commits as they happen.

For now we've been using it to find small cases where debugging of clang
compiled programs at "-O2 -g" is significantly worse than with
"-O0 -g" in
ways that can hopefully be resolved through more careful preservation of
debug data.  We're using the following meta-bug to keep track of the issues
that we find (and hopefully fix):
https://bugs.llvm.org/show_bug.cgi?id=38768

The original slides and poster that I presented at EuroLLVM earlier this
year can be found here:

slides:
http://llvm.org/devmtg/2018-04/slides/Bedwell-Measuring_the_User_Debugging_Experience.pdf
poster:
http://llvm.org/devmtg/2018-04/slides/Bedwell-Measuring_the_User_Debugging_Experience_poster.png

Thanks to everyone who's helped out with the development of DExTer so far,
especially to Stephen Wilks who worked closely with me on the initial
development of the project.

Thanks,
-Greg
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20180901/a4d9c657/attachment.html>

Really sounds great & the bugs so far look pretty good.

One possible enhancement/request: It might be worth being able to
prioritize certain kinds of debug info quality problems:

* "incorrect stepping" (especially if it can be shown to step into
code
that is /never/ executed) is a pretty high priority for any sample based
profiling
* It's possible that we could prioritize some optimized debug info quality
over others - like "can you print the parameters to a function on
function-entry" (especially without inlining - it'd be good to
prioritize
all those "missing parameters on entry to a non-inlined function"
perhaps)

- Dave

On Fri, Aug 31, 2018 at 4:26 PM Greg Bedwell via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> I'm happy to announce that DExTer (Debugging Experience Tester) is now
> available on github at https://github.com/SNSystems/dexter .  To quote
> from the project readme:
>
> "DExTer is a suite of tools used to evaluate the "User Debugging
> Experience". DExTer drives an external debugger, running on small test
> programs, and collects information on the behavior at each debugger step to
> provide quantitative values that indicate the quality of the debugging
> experience."
>
> My hope is that it will be a useful tool to help find both existing cases
> of poor debug experience as well as spot any regressions in that experience
> as they occur.  As I've previously stated, I think that this should be
> nicely complementary to other utlities such as debugify
>
<http://llvm.org/docs/SourceLevelDebugging.html#testing-debug-info-preservation-in-optimizations>
and
> DIVA <https://github.com/SNSystems/DIVA>. We also intend to
contribute
> continuous DExTer testing of LLVM trunk in the future so we'll be able
to
> immediately track any differences in the debugging experience score down to
> individual commits as they happen.
>
> For now we've been using it to find small cases where debugging of
clang
> compiled programs at "-O2 -g" is significantly worse than with
"-O0 -g" in
> ways that can hopefully be resolved through more careful preservation of
> debug data.  We're using the following meta-bug to keep track of the
issues
> that we find (and hopefully fix):
> https://bugs.llvm.org/show_bug.cgi?id=38768
>
> The original slides and poster that I presented at EuroLLVM earlier this
> year can be found here:
>
> slides:
>
http://llvm.org/devmtg/2018-04/slides/Bedwell-Measuring_the_User_Debugging_Experience.pdf
> poster:
>
http://llvm.org/devmtg/2018-04/slides/Bedwell-Measuring_the_User_Debugging_Experience_poster.png
>
> Thanks to everyone who's helped out with the development of DExTer so
far,
> especially to Stephen Wilks who worked closely with me on the initial
> development of the project.
>
> Thanks,
> -Greg
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20180904/87ebe7ea/attachment.html>