llvm dev - Oct 2019 - Small improvements for the lit tool

Hi All,

I am interested in upstreaming a few small improvements for the LLVM Integrated
Tester.  The main goal of this effort is improving the existing code base and
features to make them more approachable for future contributors and a bit more
useful for developers.  It is a non-goal to change the existing behavior in any
significant way.  Ideally, most people will not even notice.

If there are no general objections then I will start creating review-sized
patches from the changes that I have accumulated [1].

[1] https://github.com/yln/llvm-project/commits/lit-refactor


Rough list of improvements:

* Use "single process mode" when executing with `-j 1` or when
executing a
  single test.  In this mode we do not create additional processes, which can
  help with debugging.  Make sure timeouts are supported in this mode.  In this
  mode CTRL+C prints a run summary instead of a Python process termination
  exception.

* Greatly improve argument validation.  Prevent incompatible options from being
  used together.  Add argument aliases that improve consistency of existing
  options which have accumulated over time.  All existing CL arguments remain
  functional!  Deprecating/retiring any existing options is a non-goal of this
  effort.

* Make it an error when no tests are discovered or all discovered tests are
  filtered out.

* In "no progress bar" mode test results are now printed in order. 
This should
  help with things like diffing two CI runs.

* Test result categories (e.g., PASS, FAIL, XFAIL, ...). Not all
"discovered"
  tests are accounted for.  Add "unexecuted" (e.g., due to REQUIRES in
test) to
  SKIPPED category. Add new category, FILTERED that contain filtered-out tests
  (due to `--filter <regex>` or sharding feature).  This means each test
starts
  out as "discovered" and will in one of the categories at the end of
the run.
  Now all tests are accounted for.

* In the run summary tests that belong to a "failing" category (e.g.,
FAIL,
  XPASS, ...) are always printed. In addition,  we already have
  `--show-unsupported` and `--show-xfail` to show those specific
"success"
  categories.  Introduced a consistent `--show=<list of categories>`
option that
  works with all categories, to configure which tests will be printed.  Default
  is `--show=<all failing>`, i.e., no change in behavior.

* Make run summary more consistent.  Remove redundant word "Test" from
  categories. Print summary counts in the same order as test categories.

* Neatly align test counts as decimal numbers.  Align counts to longest label in
  run summary.
```
    Passing With Retry: 18
    Failing           :  8
```

* Progress bar now changes color to red on first test failure.

* Fix bug related to `--max-failures`.

* Improve formatting of error messages.

* Use smart defaults for command line arguments to streamline code/avoid
  checking for None.

* Factor out main.py into smaller, separate source files, e.g., cl_arguments.py.

* A few additional tests for new and refactored features.

* Lots of smaller refactorings to make codebase more approachable to future
  contributions: extract small functions, remove unnecessary code, remove
"out
  parameters", try to assign good names.  Use Python-style function and
variable
  names.

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20191004/a1da73a1/attachment.html>

> On Oct 4, 2019, at 14:34, Julian Lettner <julian.lettner at
apple.com> wrote:
> 
> Hi All,
> 
> I am interested in upstreaming a few small improvements for the LLVM
Integrated Tester.  The main goal of this effort is improving the existing code
base and features to make them more approachable for future contributors and a
bit more useful for developers.  It is a non-goal to change the existing
behavior in any significant way.  Ideally, most people will not even notice.
> 
> If there are no general objections then I will start creating review-sized
patches from the changes that I have accumulated [1].
> 
> [1] https://github.com/yln/llvm-project/commits/lit-refactor
<https://github.com/yln/llvm-project/commits/lit-refactor>
> 
> 
> Rough list of improvements:
> 
> * Use "single process mode" when executing with `-j 1` or when
executing a
>   single test.  In this mode we do not create additional processes, which
can
>   help with debugging.  Make sure timeouts are supported in this mode.  In
this
>   mode CTRL+C prints a run summary instead of a Python process termination
>   exception.
Mild preference for -j1 to not mean "single-process mode". -j1 is a
nice way to say "multiprocess mode but deterministic".
> 
> * Greatly improve argument validation.  Prevent incompatible options from
being
>   used together.  Add argument aliases that improve consistency of existing
>   options which have accumulated over time.  All existing CL arguments
remain
>   functional!  Deprecating/retiring any existing options is a non-goal of
this
>   effort.
> 
> * Make it an error when no tests are discovered or all discovered tests are
>   filtered out.
> 
> * In "no progress bar" mode test results are now printed in
order.  This should
>   help with things like diffing two CI runs.
Minorly against this one. I can watch the tests pass or fail on a bot right now,
and waiting for them all to finish so they can be printed sorted doesn't
seem worth it over `pbpaste | sort`.
> 
> * Test result categories (e.g., PASS, FAIL, XFAIL, ...). Not all
"discovered"
>   tests are accounted for.  Add "unexecuted" (e.g., due to
REQUIRES in test) to
>   SKIPPED category. Add new category, FILTERED that contain filtered-out
tests
>   (due to `--filter <regex>` or sharding feature).  This means each
test starts
>   out as "discovered" and will in one of the categories at the
end of the run.
>   Now all tests are accounted for.
> 
> * In the run summary tests that belong to a "failing" category
(e.g., FAIL,
>   XPASS, ...) are always printed. In addition,  we already have
>   `--show-unsupported` and `--show-xfail` to show those specific
"success"
>   categories.  Introduced a consistent `--show=<list of categories>`
option that
>   works with all categories, to configure which tests will be printed. 
Default
>   is `--show=<all failing>`, i.e., no change in behavior.
> 
> * Make run summary more consistent.  Remove redundant word "Test"
from
>   categories. Print summary counts in the same order as test categories.
> 
> * Neatly align test counts as decimal numbers.  Align counts to longest
label in
>   run summary.
> ```
>     Passing With Retry: 18
>     Failing           :  8
> ```
> 
> * Progress bar now changes color to red on first test failure.
Cute. :-)
> 
> * Fix bug related to `--max-failures`.
> 
> * Improve formatting of error messages.
> 
> * Use smart defaults for command line arguments to streamline code/avoid
>   checking for None.
> 
> * Factor out main.py into smaller, separate source files, e.g.,
cl_arguments.py.
> 
> * A few additional tests for new and refactored features.
> 
> * Lots of smaller refactorings to make codebase more approachable to future
>   contributions: extract small functions, remove unnecessary code, remove
"out
>   parameters", try to assign good names.  Use Python-style function
and variable
>   names.
> 
Jordan

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20191004/8431d6c7/attachment.html>