Hey all,
I wanted to share a proposal
<https://docs.google.com/document/d/171ecPTeXw68fbCghdGw_NPBouWvmvUX8vePlbhhHEdA/edit?usp=sharing>
to revamp the current go-to IR debugging tool: Bugpoint. i'd love to hear
any feedback or general thoughts.
Here's the markdown version of the doc:
---
# Bugpoint Redesign
Author: Diego Treviño (diegotf at google.com)
Date: 2016-06-05
Status: Draft
## Introduction
As use of bugpoint has grown several areas of improvement have been
identified through years of use: confusing to use, slow, it doesn’t always
produce high quality test cases, etc. This document proposes a new approach
with a narrower focus: minimization of IR test cases.
## Proposed New Design
### Narrow focus: test-case reduction
The main focus will be a code reduction strategy to obtain much smaller
test cases that still have the same property as the original one. This will
be done via classic delta debugging and by adding some IR-specific
reductions (e.g. replacing globals, removing unused instructions, etc),
similar to what already exists, but with more in-depth minimization.
Granted, if the community differs on this proposal, the legacy code could
still be present in the tool, but with the caveat of still being documented
and designed towards delta reduction.
### Command-Line Options
We are proposing to reduce the plethora of bugpoint’s options to just two:
an interesting-ness test and the arguments for said test, similar to other
delta reduction tools such as CReduce, Delta, and Lithium; the tool should
feel less cluttered, and there should also be no uncertainty about how to
operate it.
The interesting-ness test that’s going to be run to reduce the code is
given by name:
`--test=<test_name>`
If a `--test` option is not given, the program exits; this option is
similar to bugpoint’s current `-compile-custom` option, which lets the user
run a custom script.
The interesting-ness test would be defined as a script that returns 0 when
the IR achieves a user-defined behaviour (e.g. failure to compile on clang)
and a nonzero value when otherwise. Leaving the user the freedom to
determine what is and isn’t interesting to the tool, and thus, streamlining
the process of reducing a test-case.
If the test accepts any arguments (excluding the input ll/bc file), they
are given via the following flag:
`--test_args=<test_arguments>`
If unspecified, the test is run as given. It’s worth noting that the input
file would be passed as a parameter to the test, similar how
`-compile-custom` currently operates.
### Implementation
The tool would behave similar to CReduce’s functionality in that it would
have a list of passes that try to minimize the given test-case. We should
be able to modularize the tool’s behavior, as well as making it easier to
maintain and expand.
The first version of this redesign would try to:
* Split the code into chunks and discard those that fail the given test
* Discard functions, instructions and metadata that don’t influence the
interesting-ness test
* Remove unused parameters from functions
* Eliminate unvisited conditional paths
* Rename variables to more regular ones (such as “a”, “b”, “c”, etc.)
Once these passes are implemented, more meaningful reductions (such as type
reduction) would be added to the tool, to even further reduce IR.
## Background on historical bugpoint issues
### Root Cause Analysis
Presently, bugpoint takes a long time to find the source problem in a given
IR file, mainly due to the fact that it tries to debug the input by running
various strategies to classify the bug, which in turn run multiple
optimizer and compilation passes over the input, taking up a lot of time.
Furthermore, when the IR crashes, it tries to reduce it by performing some
sub-optimal passes (e.g. a lot of unreachable blocks), and sometimes even
fails to minimize at all.
### "Quirky" Interface
Bugpoint’s current interface overwhelms and confuses the user, the help
screen alone ends up confusing rather providing guidance, as seen below:
And, not only are there numerous features and options, but some of them
also work in unexpected ways and most of the time the user ends up using a
custom script. Pruning and simplifying the interface will be worth
considering in order to make the tool more useful in the general case and
easier to maintain.
--
Cheers,
Diego
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20190607/9e12d6ca/attachment.html>
On 6/7/19 4:19 PM, Diego Treviño via llvm-dev wrote:
Hey all,
I wanted to share a
proposal<https://docs.google.com/document/d/171ecPTeXw68fbCghdGw_NPBouWvmvUX8vePlbhhHEdA/edit?usp=sharing>
to revamp the current go-to IR debugging tool: Bugpoint. i'd love to hear
any feedback or general thoughts.
Here's the markdown version of the doc:
---
# Bugpoint Redesign
Author: Diego Treviño (diegotf at google.com<mailto:diegotf at
google.com>)
Date: 2016-06-05
Status: Draft
## Introduction
As use of bugpoint has grown several areas of improvement have been identified
through years of use: confusing to use, slow, it doesn’t always produce high
quality test cases, etc. This document proposes a new approach with a narrower
focus: minimization of IR test cases.
## Proposed New Design
### Narrow focus: test-case reduction
The main focus will be a code reduction strategy to obtain much smaller test
cases that still have the same property as the original one. This will be done
via classic delta debugging and by adding some IR-specific reductions (e.g.
replacing globals, removing unused instructions, etc), similar to what already
exists, but with more in-depth minimization.
Granted, if the community differs on this proposal, the legacy code could still
be present in the tool, but with the caveat of still being documented and
designed towards delta reduction.
### Command-Line Options
We are proposing to reduce the plethora of bugpoint’s options to just two: an
interesting-ness test and the arguments for said test, similar to other delta
reduction tools such as CReduce, Delta, and Lithium; the tool should feel less
cluttered, and there should also be no uncertainty about how to operate it.
The interesting-ness test that’s going to be run to reduce the code is given by
name:
`--test=<test_name>`
If a `--test` option is not given, the program exits; this option is similar to
bugpoint’s current `-compile-custom` option, which lets the user run a custom
script.
The interesting-ness test would be defined as a script that returns 0 when the
IR achieves a user-defined behaviour (e.g. failure to compile on clang) and a
nonzero value when otherwise. Leaving the user the freedom to determine what is
and isn’t interesting to the tool, and thus, streamlining the process of
reducing a test-case.
If the test accepts any arguments (excluding the input ll/bc file), they are
given via the following flag:
`--test_args=<test_arguments>`
If unspecified, the test is run as given. It’s worth noting that the input file
would be passed as a parameter to the test, similar how `-compile-custom`
currently operates.
### Implementation
The tool would behave similar to CReduce’s functionality in that it would have a
list of passes that try to minimize the given test-case. We should be able to
modularize the tool’s behavior, as well as making it easier to maintain and
expand.
The first version of this redesign would try to:
* Split the code into chunks and discard those that fail the given test
* Discard functions, instructions and metadata that don’t influence the
interesting-ness test
* Remove unused parameters from functions
* Eliminate unvisited conditional paths
Do you plan on doing this by running with instrumentation?
* Rename variables to more regular ones (such as “a”, “b”, “c”, etc.)
Great.
Once these passes are implemented, more meaningful reductions (such as type
reduction) would be added to the tool, to even further reduce IR.
Thanks for taking this on. We will all benefit from improvements to bugpoint.
One concern that I have is that, from personal experience, the ability for
bugpoint to reduce the set of optimization passes applied in order to reproduce
a bug is extremely helpful. I understand your desire to decouple the logic
somewhat, and maybe there's some way to generalize that functionality by
enabling simultaneous delta reduction on some secondary inputs (some of which
may happen to be a pass list), but I'd like to see us somehow retain that
capability to isolate the problematic set of transformations.
bugpoint currently has the ability to debug miscompiles by splitting the code
into a "known good" set of functions (which is puts into a separate
library) and the remainder of the code (which, in theory, is the smallest part
of the code necessary to reproduce the bug). This has also proven useful in the
past. Is this something you intend to keep?
My largest set of problems with bugpoint has been that bugpoint's logic for
doing things like loop extraction will themselves often crash, and also,
there's no easy way to start with multiple input files (e.g., what you start
with from a program with multiple source files).
-Hal
## Background on historical bugpoint issues
### Root Cause Analysis
Presently, bugpoint takes a long time to find the source problem in a given IR
file, mainly due to the fact that it tries to debug the input by running various
strategies to classify the bug, which in turn run multiple optimizer and
compilation passes over the input, taking up a lot of time. Furthermore, when
the IR crashes, it tries to reduce it by performing some sub-optimal passes
(e.g. a lot of unreachable blocks), and sometimes even fails to minimize at all.
### "Quirky" Interface
Bugpoint’s current interface overwhelms and confuses the user, the help screen
alone ends up confusing rather providing guidance, as seen below:
And, not only are there numerous features and options, but some of them also
work in unexpected ways and most of the time the user ends up using a custom
script. Pruning and simplifying the interface will be worth considering in order
to make the tool more useful in the general case and easier to maintain.
--
Cheers,
Diego
_______________________________________________
LLVM Developers mailing list
llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org>
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
--
Hal Finkel
Lead, Compiler Technology and Programming Languages
Leadership Computing Facility
Argonne National Laboratory
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20190607/668db684/attachment.html>
On 6/7/2019 5:19 PM, Diego Treviño via llvm-dev wrote:> ### Narrow focus: test-case reduction > The main focus will be a code reduction strategy to obtain much > smaller test cases that still have the same property as the original > one. This will be done via classic delta debugging and by adding some > IR-specific reductions (e.g. replacing globals, removing unused > instructions, etc), similar to what already exists, but with more > in-depth minimization. > > > Granted, if the community differs on this proposal, the legacy code > could still be present in the tool, but with the caveat of still being > documented and designed towards delta reduction.As Hal points out, there are really two dimensions of reduction you can do with bugpoint. One is delta debugging of passes, to figure out which pass is causing the problem, and another is regular delta debugging of the program itself. Supporting both use cases is important. My personal experience, however, has been to only use bugpoint for delta debugging of code, as I'm trying to work out what features of the input programming is causing the pass I'm working on to crash. I can't say what the relative balance of these two use cases are, and it may be that we can solve this by having two versions of bugpoint to solve the two different problems.> ### Command-Line Options > We are proposing to reduce the plethora of bugpoint’s options to just > two: an interesting-ness test and the arguments for said test, similar > to other delta reduction tools such as CReduce, Delta, and Lithium; > the tool should feel less cluttered, and there should also be no > uncertainty about how to operate it.I am /strongly/ in favor of going to an interesting-ness test approach in lieu of the current "try and guess what kind of bug you're looking for" in the current approach. Writing correct test detection scripts is definitely a challenge, but you can usually crib from a preexisting script. A second note is that we can provide much of the current "automatic" functionality in bugpoint via a set of useful test scripts, one for each mode. -- Joshua Cranmer Thunderbird and DXR developer Source code archæologist -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20190608/fefd0350/attachment.html>
"Finkel, Hal J. via llvm-dev" <llvm-dev at lists.llvm.org> writes:> One concern that I have is that, from personal experience, the ability > for bugpoint to reduce the set of optimization passes applied in order > to reproduce a bug is extremely helpful. I understand your desire to > decouple the logic somewhat, and maybe there's some way to generalize > that functionality by enabling simultaneous delta reduction on some > secondary inputs (some of which may happen to be a pass list), but I'd > like to see us somehow retain that capability to isolate the > problematic set of transformations.I wonder if this might be better as a separate tool. The functionality is defintely useful. In fact I'd like to see it enhanced by using DebugCounters when available. This will require some smarts for DebugCounters to either report themselves to the tool or for passes to report their DebugCounter-ness.> bugpoint currently has the ability to debug miscompiles by splitting > the code into a "known good" set of functions (which is puts into a > separate library) and the remainder of the code (which, in theory, is > the smallest part of the code necessary to reproduce the bug). This > has also proven useful in the past. Is this something you intend to > keep?Yes, this is also very useful and would fit in with whatever tool does the pass reduction mentioned above. The first step would be to find the subset of code that is miscompiled. The second would be to do pass reduction (with use of DebugCounters) over that miscompiled piece of code. Isn't this how bugpoint basically operates in miscompile mode now?> My largest set of problems with bugpoint has been that bugpoint's > logic for doing things like loop extraction will themselves often > crash, and also, there's no easy way to start with multiple input > files (e.g., what you start with from a program with multiple source > files).I've always just generated .ll files and linked them manually before starting the bugpoint process. I'd think it would be straightforward to have bugpoint/whatever take a set of input files and do the link step itself. One other aspect of bugpoint/whatever that will become more important since flang is now an official project is the ability to specify a linker to use. Analyzing Fortran codes will require the tool to know which Fortran compiler to link with since there is no standard Fortran ABI or runtime interface. The copiler used to link must be the same one used to generate the IR. When mixing Fortran and C/C++, the tool will need to know to use the Fortran compiler to do the link (and also link in the C/C++ runtimes). -David
Diego Treviño via llvm-dev <llvm-dev at lists.llvm.org> writes:> If the test accepts any arguments (excluding the input ll/bc file), they are given via the following flag: > `--test_args=<test_arguments>`I worry a little bit about oddball test arguments, for example arguments with spaces. Maybe that's a corner case we don't care about but to handle it bugpoint could also support something like: --test-arg <arg1> --test-arg <arg2> ... -David
At the moment, bugpoint has three major use cases: crash reduction, miscompile reduction, and mutation fuzzing. Out of these, a huge proportion of the interface complexity comes from the miscompile handling. I generally agree with removing the auto-detection logic. I've found it to be extraordinarily error prone and confusing. Interface wise, I might suggest something in the spirit of sub-tools (i.e. git or svn). As possible example: bugpoint crash-reduce bugpoint miscompile-reduce bugpoint mutate In addition to these high-level commands, it may also be useful to expose individual reduction steps. I find myself frequently wanting to run only individual reduction steps (and have hacked up my local bugpoint to allow this) via a wrapper script. Having first class support for "bugpoint reduce-step functions <input.ll>" would be awesome. Another idea would be to move all of the complexity of test formation into a separate command. Rather than having the tool detect which opt to use as part of reduction, instead have a generate command which generates a script which is then used for reduction. (i.e. make everything use the custom mode, while still proving helpers to generate). This is probably more natural for crash reduction instead of miscompile reduction, but maybe we could make it work for both? Or maybe if we split the two commands (and thus their interface) it doesn't really matter. Philip p.s. Bugpoint is a fairly critical tool. If we start rewriting it, making sure it continues to work through the process will be critical. We don't have much in the way of testing for it today, and that would need to change. On 6/7/19 2:19 PM, Diego Treviño via llvm-dev wrote:> Hey all, > > I wanted to share a proposal > <https://docs.google.com/document/d/171ecPTeXw68fbCghdGw_NPBouWvmvUX8vePlbhhHEdA/edit?usp=sharing> > to revamp the current go-to IR debugging tool: Bugpoint. i'd love to > hear any feedback or general thoughts. > > Here's the markdown version of the doc: > --- > # Bugpoint Redesign > Author: Diego Treviño (diegotf at google.com <mailto:diegotf at google.com>) > > Date: 2016-06-05 > > Status: Draft > > > ## Introduction > As use of bugpoint has grown several areas of improvement have been > identified through years of use: confusing to use, slow, it doesn’t > always produce high quality test cases, etc. This document proposes a > new approach with a narrower focus: minimization of IR test cases. > > > ## Proposed New Design > > > ### Narrow focus: test-case reduction > The main focus will be a code reduction strategy to obtain much > smaller test cases that still have the same property as the original > one. This will be done via classic delta debugging and by adding some > IR-specific reductions (e.g. replacing globals, removing unused > instructions, etc), similar to what already exists, but with more > in-depth minimization. > > > Granted, if the community differs on this proposal, the legacy code > could still be present in the tool, but with the caveat of still being > documented and designed towards delta reduction. > > > ### Command-Line Options > We are proposing to reduce the plethora of bugpoint’s options to just > two: an interesting-ness test and the arguments for said test, similar > to other delta reduction tools such as CReduce, Delta, and Lithium; > the tool should feel less cluttered, and there should also be no > uncertainty about how to operate it. > > > The interesting-ness test that’s going to be run to reduce the code is > given by name: > `--test=<test_name>` > If a `--test` option is not given, the program exits; this option is > similar to bugpoint’s current `-compile-custom` option, which lets the > user run a custom script. > > > The interesting-ness test would be defined as a script that returns 0 > when the IR achieves a user-defined behaviour (e.g. failure to compile > on clang) and a nonzero value when otherwise. Leaving the user the > freedom to determine what is and isn’t interesting to the tool, and > thus, streamlining the process of reducing a test-case. > > > If the test accepts any arguments (excluding the input ll/bc file), > they are given via the following flag: > `--test_args=<test_arguments>` > If unspecified, the test is run as given. It’s worth noting that the > input file would be passed as a parameter to the test, similar how > `-compile-custom` currently operates. > > > ### Implementation > The tool would behave similar to CReduce’s functionality in that it > would have a list of passes that try to minimize the given test-case. > We should be able to modularize the tool’s behavior, as well as making > it easier to maintain and expand. > > > The first version of this redesign would try to: > > > * Split the code into chunks and discard those that fail the given test > * Discard functions, instructions and metadata that don’t influence > the interesting-ness test > * Remove unused parameters from functions > * Eliminate unvisited conditional paths > * Rename variables to more regular ones (such as “a”, “b”, “c”, etc.) > > > Once these passes are implemented, more meaningful reductions (such as > type reduction) would be added to the tool, to even further reduce IR. > > > ## Background on historical bugpoint issues > > > ### Root Cause Analysis > Presently, bugpoint takes a long time to find the source problem in a > given IR file, mainly due to the fact that it tries to debug the input > by running various strategies to classify the bug, which in turn run > multiple optimizer and compilation passes over the input, taking up a > lot of time. Furthermore, when the IR crashes, it tries to reduce it > by performing some sub-optimal passes (e.g. a lot of unreachable > blocks), and sometimes even fails to minimize at all. > > > ### "Quirky" Interface > Bugpoint’s current interface overwhelms and confuses the user, the > help screen alone ends up confusing rather providing guidance, as seen > below: > >  > > And, not only are there numerous features and options, but some of > them also work in unexpected ways and most of the time the user ends > up using a custom script. Pruning and simplifying the interface will > be worth considering in order to make the tool more useful in the > general case and easier to maintain. > > > -- > Cheers, > Diego > > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20190611/c5a8e7d8/attachment.html>
On 6/11/19 12:25 PM, Philip Reames via llvm-dev wrote:
At the moment, bugpoint has three major use cases: crash reduction, miscompile
reduction, and mutation fuzzing. Out of these, a huge proportion of the
interface complexity comes from the miscompile handling.
I generally agree with removing the auto-detection logic. I've found it to
be extraordinarily error prone and confusing.
I'm not sure if I said this previously, but +1 to this. I don't recall a
situation where I wasn't sure whether the problem was the compiler crashing
or whether the problem was that the code was miscompiled. Even in a CI-type
setup, a failure in the compiler step and in the run step can be distinguished
by the relevant scripts.
-Hal
Interface wise, I might suggest something in the spirit of sub-tools (i.e. git
or svn). As possible example:
bugpoint crash-reduce
bugpoint miscompile-reduce
bugpoint mutate
In addition to these high-level commands, it may also be useful to expose
individual reduction steps. I find myself frequently wanting to run only
individual reduction steps (and have hacked up my local bugpoint to allow this)
via a wrapper script. Having first class support for "bugpoint reduce-step
functions <input.ll>" would be awesome.
Another idea would be to move all of the complexity of test formation into a
separate command. Rather than having the tool detect which opt to use as part
of reduction, instead have a generate command which generates a script which is
then used for reduction. (i.e. make everything use the custom mode, while still
proving helpers to generate). This is probably more natural for crash reduction
instead of miscompile reduction, but maybe we could make it work for both? Or
maybe if we split the two commands (and thus their interface) it doesn't
really matter.
Philip
p.s. Bugpoint is a fairly critical tool. If we start rewriting it, making sure
it continues to work through the process will be critical. We don't have
much in the way of testing for it today, and that would need to change.
On 6/7/19 2:19 PM, Diego Treviño via llvm-dev wrote:
Hey all,
I wanted to share a
proposal<https://docs.google.com/document/d/171ecPTeXw68fbCghdGw_NPBouWvmvUX8vePlbhhHEdA/edit?usp=sharing>
to revamp the current go-to IR debugging tool: Bugpoint. i'd love to hear
any feedback or general thoughts.
Here's the markdown version of the doc:
---
# Bugpoint Redesign
Author: Diego Treviño (diegotf at google.com<mailto:diegotf at
google.com>)
Date: 2016-06-05
Status: Draft
## Introduction
As use of bugpoint has grown several areas of improvement have been identified
through years of use: confusing to use, slow, it doesn’t always produce high
quality test cases, etc. This document proposes a new approach with a narrower
focus: minimization of IR test cases.
## Proposed New Design
### Narrow focus: test-case reduction
The main focus will be a code reduction strategy to obtain much smaller test
cases that still have the same property as the original one. This will be done
via classic delta debugging and by adding some IR-specific reductions (e.g.
replacing globals, removing unused instructions, etc), similar to what already
exists, but with more in-depth minimization.
Granted, if the community differs on this proposal, the legacy code could still
be present in the tool, but with the caveat of still being documented and
designed towards delta reduction.
### Command-Line Options
We are proposing to reduce the plethora of bugpoint’s options to just two: an
interesting-ness test and the arguments for said test, similar to other delta
reduction tools such as CReduce, Delta, and Lithium; the tool should feel less
cluttered, and there should also be no uncertainty about how to operate it.
The interesting-ness test that’s going to be run to reduce the code is given by
name:
`--test=<test_name>`
If a `--test` option is not given, the program exits; this option is similar to
bugpoint’s current `-compile-custom` option, which lets the user run a custom
script.
The interesting-ness test would be defined as a script that returns 0 when the
IR achieves a user-defined behaviour (e.g. failure to compile on clang) and a
nonzero value when otherwise. Leaving the user the freedom to determine what is
and isn’t interesting to the tool, and thus, streamlining the process of
reducing a test-case.
If the test accepts any arguments (excluding the input ll/bc file), they are
given via the following flag:
`--test_args=<test_arguments>`
If unspecified, the test is run as given. It’s worth noting that the input file
would be passed as a parameter to the test, similar how `-compile-custom`
currently operates.
### Implementation
The tool would behave similar to CReduce’s functionality in that it would have a
list of passes that try to minimize the given test-case. We should be able to
modularize the tool’s behavior, as well as making it easier to maintain and
expand.
The first version of this redesign would try to:
* Split the code into chunks and discard those that fail the given test
* Discard functions, instructions and metadata that don’t influence the
interesting-ness test
* Remove unused parameters from functions
* Eliminate unvisited conditional paths
* Rename variables to more regular ones (such as “a”, “b”, “c”, etc.)
Once these passes are implemented, more meaningful reductions (such as type
reduction) would be added to the tool, to even further reduce IR.
## Background on historical bugpoint issues
### Root Cause Analysis
Presently, bugpoint takes a long time to find the source problem in a given IR
file, mainly due to the fact that it tries to debug the input by running various
strategies to classify the bug, which in turn run multiple optimizer and
compilation passes over the input, taking up a lot of time. Furthermore, when
the IR crashes, it tries to reduce it by performing some sub-optimal passes
(e.g. a lot of unreachable blocks), and sometimes even fails to minimize at all.
### "Quirky" Interface
Bugpoint’s current interface overwhelms and confuses the user, the help screen
alone ends up confusing rather providing guidance, as seen below:
And, not only are there numerous features and options, but some of them also
work in unexpected ways and most of the time the user ends up using a custom
script. Pruning and simplifying the interface will be worth considering in order
to make the tool more useful in the general case and easier to maintain.
--
Cheers,
Diego
_______________________________________________
LLVM Developers mailing list
llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org>
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
_______________________________________________
LLVM Developers mailing list
llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org>
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
--
Hal Finkel
Lead, Compiler Technology and Programming Languages
Leadership Computing Facility
Argonne National Laboratory
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20190611/23d471f6/attachment.html>
> On Jun 7, 2019, at 2:19 PM, Diego Treviño via llvm-dev <llvm-dev at lists.llvm.org> wrote: > > Hey all, > > I wanted to share a proposal <https://docs.google.com/document/d/171ecPTeXw68fbCghdGw_NPBouWvmvUX8vePlbhhHEdA/edit?usp=sharing> to revamp the current go-to IR debugging tool: Bugpoint. i'd love to hear any feedback or general thoughts.Hi Diego, This sounds super awesome, I’d love to see Bugpoint get a thorough rethink. It sounds like you’re proposing two things though: 1) you’re making the internal design more principled and putting the various interesting-ness tests into a better structure, 2) You’re proposing removing of the ‘automatic’ stuff (which I agree is not great). You’re likely to get pushback on this from people who like or use #2. Would it make sense to just deemphasize it by moving the auto feature under an explicit flag or something? That way, it is a strict improvement over what we have now, instead of regressing on a bit of functionality. -Chris> > Here's the markdown version of the doc: > --- > # Bugpoint Redesign > Author: Diego Treviño (diegotf at google.com <mailto:diegotf at google.com>) > > Date: 2016-06-05 > > Status: Draft > > > ## Introduction > As use of bugpoint has grown several areas of improvement have been identified through years of use: confusing to use, slow, it doesn’t always produce high quality test cases, etc. This document proposes a new approach with a narrower focus: minimization of IR test cases. > > > ## Proposed New Design > > > ### Narrow focus: test-case reduction > The main focus will be a code reduction strategy to obtain much smaller test cases that still have the same property as the original one. This will be done via classic delta debugging and by adding some IR-specific reductions (e.g. replacing globals, removing unused instructions, etc), similar to what already exists, but with more in-depth minimization. > > > Granted, if the community differs on this proposal, the legacy code could still be present in the tool, but with the caveat of still being documented and designed towards delta reduction. > > > ### Command-Line Options > We are proposing to reduce the plethora of bugpoint’s options to just two: an interesting-ness test and the arguments for said test, similar to other delta reduction tools such as CReduce, Delta, and Lithium; the tool should feel less cluttered, and there should also be no uncertainty about how to operate it. > > > The interesting-ness test that’s going to be run to reduce the code is given by name: > `--test=<test_name>` > If a `--test` option is not given, the program exits; this option is similar to bugpoint’s current `-compile-custom` option, which lets the user run a custom script. > > > The interesting-ness test would be defined as a script that returns 0 when the IR achieves a user-defined behaviour (e.g. failure to compile on clang) and a nonzero value when otherwise. Leaving the user the freedom to determine what is and isn’t interesting to the tool, and thus, streamlining the process of reducing a test-case. > > > If the test accepts any arguments (excluding the input ll/bc file), they are given via the following flag: > `--test_args=<test_arguments>` > If unspecified, the test is run as given. It’s worth noting that the input file would be passed as a parameter to the test, similar how `-compile-custom` currently operates. > > > ### Implementation > The tool would behave similar to CReduce’s functionality in that it would have a list of passes that try to minimize the given test-case. We should be able to modularize the tool’s behavior, as well as making it easier to maintain and expand. > > > The first version of this redesign would try to: > > > * Split the code into chunks and discard those that fail the given test > * Discard functions, instructions and metadata that don’t influence the interesting-ness test > * Remove unused parameters from functions > * Eliminate unvisited conditional paths > * Rename variables to more regular ones (such as “a”, “b”, “c”, etc.) > > > Once these passes are implemented, more meaningful reductions (such as type reduction) would be added to the tool, to even further reduce IR. > > > ## Background on historical bugpoint issues > > > ### Root Cause Analysis > Presently, bugpoint takes a long time to find the source problem in a given IR file, mainly due to the fact that it tries to debug the input by running various strategies to classify the bug, which in turn run multiple optimizer and compilation passes over the input, taking up a lot of time. Furthermore, when the IR crashes, it tries to reduce it by performing some sub-optimal passes (e.g. a lot of unreachable blocks), and sometimes even fails to minimize at all. > > > ### "Quirky" Interface > Bugpoint’s current interface overwhelms and confuses the user, the help screen alone ends up confusing rather providing guidance, as seen below: > >  > > And, not only are there numerous features and options, but some of them also work in unexpected ways and most of the time the user ends up using a custom script. Pruning and simplifying the interface will be worth considering in order to make the tool more useful in the general case and easier to maintain. > > > -- > Cheers, > Diego > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20190614/44f83d6f/attachment.html>