llvm dev - Apr 2018 - [RFC] [lit] Report line numbers for failed RUN commands

When some lit tests fail, I find it time-consuming to locate the source
line of the failing RUN command.  Running lit with -vv (or just -v in the
case of the internal shell) makes it easier because it prints an expanded
version of each RUN command as it executes, but it can still be challenging
because lit doesn't provide the RUN command's source line number.

While I haven't personally seen failures in the following tests, grep
helped me discover them as extreme examples where finding the RUN source
line should prove challenging:

1. clang's test/Driver/linux-ld.c has 892 total RUN lines

2. clang's test/Driver/arm-cortex-cpus.c has 424 RUN lines after
concatenation for line continuations

To address this issue, I wrote the following patch to introduce RUN source
line numbers into lit's -vv output (or -v for the internal shell), but the
patch needs a review:

https://reviews.llvm.org/D44598
I've been using this patch for a couple of weeks, and it's a big
improvement for me.  If there's some existing feature that people already
use to handle this issue, please let me know so I can improve my usage of
lit.

Thanks.

Joel
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20180402/34813c1f/attachment.html>

Seems useful, but the current output is quite verbose.
Wouldn’t just

RUN:1; ls

be sufficient in place of

'RUN: at line 1’; ls

?

I’m also curious about the implementation, I recall there was a discussion about
forcing all tests to use internal shell?
In that case a trick with a no-op “:” command would not be necessary.

George
> On Apr 2, 2018, at 1:25 PM, Joel E. Denny via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> 
> When some lit tests fail, I find it time-consuming to locate the source
line of the failing RUN command.  Running lit with -vv (or just -v in the case
of the internal shell) makes it easier because it prints an expanded version of
each RUN command as it executes, but it can still be challenging because lit
doesn't provide the RUN command's source line number.
> 
> While I haven't personally seen failures in the following tests, grep
helped me discover them as extreme examples where finding the RUN source line
should prove challenging:
> 
> 1. clang's test/Driver/linux-ld.c has 892 total RUN lines
> 
> 2. clang's test/Driver/arm-cortex-cpus.c has 424 RUN lines after
concatenation for line continuations
> 
> To address this issue, I wrote the following patch to introduce RUN source
line numbers into lit's -vv output (or -v for the internal shell), but the
patch needs a review:
> 
> https://reviews.llvm.org/D44598
<https://reviews.llvm.org/D44598>I've been using this patch for a
couple of weeks, and it's a big improvement for me.  If there's some
existing feature that people already use to handle this issue, please let me
know so I can improve my usage of lit.
> 
> Thanks.
> 
> Joel
> 
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20180402/8a116a72/attachment.html>

Hi George,

On Mon, Apr 2, 2018 at 5:02 PM, George Karpenkov <ekarpenkov at apple.com>
wrote:
> Seems useful, but the current output is quite verbose.
> Wouldn’t just
>
> RUN:1; ls
>
> be sufficient in place of
>
> 'RUN: at line 1’; ls
>
> ?
>
My only argument against your shorter version is that its meaning might not
be obvious to someone seeing it for the first time.  Nevertheless, it would
certainly suffice for me.

>
> I’m also curious about the implementation, I recall there was a discussion
> about forcing all tests to use internal shell?
>
Do you have a link?

> In that case a trick with a no-op “:” command would not be necessary.
>
Agreed.  Without having seen the discussion, my guess it that usage of the
external shell isn't going away immediately, and this patch seems
immediately useful.  It's also fairly small, so it should be easy to
migrate to an implementation tailored to the internal shell if usage of the
external shell goes away.

Thanks.

Joel

> George
>
> On Apr 2, 2018, at 1:25 PM, Joel E. Denny via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
> When some lit tests fail, I find it time-consuming to locate the source
> line of the failing RUN command.  Running lit with -vv (or just -v in the
> case of the internal shell) makes it easier because it prints an expanded
> version of each RUN command as it executes, but it can still be challenging
> because lit doesn't provide the RUN command's source line number.
>
> While I haven't personally seen failures in the following tests, grep
> helped me discover them as extreme examples where finding the RUN source
> line should prove challenging:
>
> 1. clang's test/Driver/linux-ld.c has 892 total RUN lines
>
> 2. clang's test/Driver/arm-cortex-cpus.c has 424 RUN lines after
> concatenation for line continuations
>
> To address this issue, I wrote the following patch to introduce RUN source
> line numbers into lit's -vv output (or -v for the internal shell), but
the
> patch needs a review:
>
> https://reviews.llvm.org/D44598
> I've been using this patch for a couple of weeks, and it's a big
> improvement for me.  If there's some existing feature that people
already
> use to handle this issue, please let me know so I can improve my usage of
> lit.
>
> Thanks.
>
> Joel
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20180402/0ece7bc9/attachment.html>