llvm dev - Jan 2022 - Automatic Regression Finding Tool

Dear LLVM Developers,

in the last few months Theodoros (theodort at inf.ethz.ch) and I have 
created a tool which automatically finds regressions related to dead 
code elimination.
We have collected more than 200 regressions with respect to trunk and 
would like to report them.

However, as we do not wish to spam the GitHub issues, we would like to 
ask how to best hand over these findings.

An example of how a current issue/report looks like can be found at 
https://github.com/llvm/llvm-project/issues/52965 .
Any feedback on the report template is appreciated.

Yours sincerely,
Yann Girsberger

Hi Yann, this is a great initiative.

On the template I had a few ideas.

Could you put a dividing line between each pair of command/output? Or
perhaps just one after the "can not eliminate foo but..." line. Then
it's easier to see that each command and output is paired starting
from that point.

You have the output of -v, what is the use case for that? The most
obvious thing is the target. I primarily work on Arm so it is good to
see what the default target should be but I wonder if there's a better
way to do that. Not saying you have to remove the -v output, I just
wonder if an explicit --target option to clang would give the reader
the same info and help those of us not running on x86. (I'm not
suggesting you go back and redo 200 reports though, maybe for new
ones)

Having both assembly outputs is great. We (Linaro) just added this to
one of our regression loops and a lot of cases can be triaged with
just that.

However the phrasing is a bit imprecise on first read. "Started with"
I think, did the bisect start with this rev or did the problem start
there? After a few seconds you realise of course the latter is the
only thing that makes sense but it could be clearer. Maybe "Regression
started with".
(you can probably spend ages naming that bit. Some people prefer last
good/first bad, sometimes I want to put it the other way around as
before/after)

Would it be possible to attach the test case and the before/after as
an archive to the issue? (I found out recently github doesn't allow .c
files but an archive containing them is fine) Then it's quick to
download and diff locally if it can't be eyeballed.

Last thing, that I'm not sure is possible automatically. Tagging the
user who wrote the patch. Easy enough to do after the fact but perhaps
github has some way to map commit to github username.

On Thu, 13 Jan 2022 at 18:49, Yann Girsberger via llvm-dev
<llvm-dev at lists.llvm.org> wrote:
>
> Dear LLVM Developers,
>
> in the last few months Theodoros (theodort at inf.ethz.ch) and I have
> created a tool which automatically finds regressions related to dead
> code elimination.
> We have collected more than 200 regressions with respect to trunk and
> would like to report them.
>
> However, as we do not wish to spam the GitHub issues, we would like to
> ask how to best hand over these findings.
>
> An example of how a current issue/report looks like can be found at
> https://github.com/llvm/llvm-project/issues/52965 .
> Any feedback on the report template is appreciated.
>
> Yours sincerely,
> Yann Girsberger
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev