llvm dev - Jan 2022 - Automatic Regression Finding Tool

Hi Yann, this is a great initiative.

On the template I had a few ideas.

Could you put a dividing line between each pair of command/output? Or
perhaps just one after the "can not eliminate foo but..." line. Then
it's easier to see that each command and output is paired starting
from that point.

You have the output of -v, what is the use case for that? The most
obvious thing is the target. I primarily work on Arm so it is good to
see what the default target should be but I wonder if there's a better
way to do that. Not saying you have to remove the -v output, I just
wonder if an explicit --target option to clang would give the reader
the same info and help those of us not running on x86. (I'm not
suggesting you go back and redo 200 reports though, maybe for new
ones)

Having both assembly outputs is great. We (Linaro) just added this to
one of our regression loops and a lot of cases can be triaged with
just that.

However the phrasing is a bit imprecise on first read. "Started with"
I think, did the bisect start with this rev or did the problem start
there? After a few seconds you realise of course the latter is the
only thing that makes sense but it could be clearer. Maybe "Regression
started with".
(you can probably spend ages naming that bit. Some people prefer last
good/first bad, sometimes I want to put it the other way around as
before/after)

Would it be possible to attach the test case and the before/after as
an archive to the issue? (I found out recently github doesn't allow .c
files but an archive containing them is fine) Then it's quick to
download and diff locally if it can't be eyeballed.

Last thing, that I'm not sure is possible automatically. Tagging the
user who wrote the patch. Easy enough to do after the fact but perhaps
github has some way to map commit to github username.

On Thu, 13 Jan 2022 at 18:49, Yann Girsberger via llvm-dev
<llvm-dev at lists.llvm.org> wrote:
>
> Dear LLVM Developers,
>
> in the last few months Theodoros (theodort at inf.ethz.ch) and I have
> created a tool which automatically finds regressions related to dead
> code elimination.
> We have collected more than 200 regressions with respect to trunk and
> would like to report them.
>
> However, as we do not wish to spam the GitHub issues, we would like to
> ask how to best hand over these findings.
>
> An example of how a current issue/report looks like can be found at
> https://github.com/llvm/llvm-project/issues/52965 .
> Any feedback on the report template is appreciated.
>
> Yours sincerely,
> Yann Girsberger
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

Hi David,

thank you very much for your feedback!

I integrated it and other feedback into new reports.
(https://github.com/llvm/llvm-project/issues/53322)

Is there any interest from Linaro (or the ARM community in general) in 
similar regressions for ARM targets?

If you have any further feedback on the reports, please let us know.

Best,
Yann

On 14/01/2022 10:47, David Spickett wrote:
> Hi Yann, this is a great initiative.
> 
> On the template I had a few ideas.
> 
> Could you put a dividing line between each pair of command/output? Or
> perhaps just one after the "can not eliminate foo but..." line.
Then
> it's easier to see that each command and output is paired starting
> from that point.
> 
> You have the output of -v, what is the use case for that? The most
> obvious thing is the target. I primarily work on Arm so it is good to
> see what the default target should be but I wonder if there's a better
> way to do that. Not saying you have to remove the -v output, I just
> wonder if an explicit --target option to clang would give the reader
> the same info and help those of us not running on x86. (I'm not
> suggesting you go back and redo 200 reports though, maybe for new
> ones)
> 
> Having both assembly outputs is great. We (Linaro) just added this to
> one of our regression loops and a lot of cases can be triaged with
> just that.
> 
> However the phrasing is a bit imprecise on first read. "Started
with"
> I think, did the bisect start with this rev or did the problem start
> there? After a few seconds you realise of course the latter is the
> only thing that makes sense but it could be clearer. Maybe "Regression
> started with".
> (you can probably spend ages naming that bit. Some people prefer last
> good/first bad, sometimes I want to put it the other way around as
> before/after)
> 
> Would it be possible to attach the test case and the before/after as
> an archive to the issue? (I found out recently github doesn't allow .c
> files but an archive containing them is fine) Then it's quick to
> download and diff locally if it can't be eyeballed.
> 
> Last thing, that I'm not sure is possible automatically. Tagging the
> user who wrote the patch. Easy enough to do after the fact but perhaps
> github has some way to map commit to github username.
> 
> On Thu, 13 Jan 2022 at 18:49, Yann Girsberger via llvm-dev
> <llvm-dev at lists.llvm.org> wrote:
>>
>> Dear LLVM Developers,
>>
>> in the last few months Theodoros (theodort at inf.ethz.ch) and I have
>> created a tool which automatically finds regressions related to dead
>> code elimination.
>> We have collected more than 200 regressions with respect to trunk and
>> would like to report them.
>>
>> However, as we do not wish to spam the GitHub issues, we would like to
>> ask how to best hand over these findings.
>>
>> An example of how a current issue/report looks like can be found at
>> https://github.com/llvm/llvm-project/issues/52965 .
>> Any feedback on the report template is appreciated.
>>
>> Yours sincerely,
>> Yann Girsberger
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev