llvm dev - May 2016 - [RFC] Changing `lit` default output

Hi all,

I have some patches up in:
  http://reviews.llvm.org/D20721
to make `lit` output more readable when commands fail. I'd like to get a
little more feedback on this before landing, since it impacts how test
failures appear when using the internal shell script runner (which is not
the default so this will currently mostly impact Windows or test frameworks
that intentionally set this behavior).

These patches change the output of a script like `true && echo hi
&& false`
to something like:

```
$ true
$ echo hi
hi

$ false
note: command had no output on stdout or stderr
error: command failed with exit status 1
```

instead of the old:

```
Command 0: "true"
Command 0 Result: 0
Command 0 Output:


Command 0 Stderr:


Command 1: "echo" "hi"
Command 1 Result: 0
Command 1 Output:
hi


Command 1 Stderr:


Command 2: "wc" "missing-file"
Command 2 Result: 1
Command 2 Output:
None

Command 2 Stderr:
None
```

which I think is significantly more readable.

One change in the patches is that it will also start including, inline, the
output of files which were redirected. This solves a common problem where a
 command fails when it wasn't expected to, and its output was captured. In
such cases the test log just showed the command failure, but since the
output was redirected you couldn't tell from the log why the common failed.

Please let me know if you have any objections to the new format.

 - Daniel
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160531/d96ed55d/attachment.html>

> On May 31, 2016, at 4:48 PM, Daniel Dunbar via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> 
> Hi all,
> 
> I have some patches up in:
>   http://reviews.llvm.org/D20721 <http://reviews.llvm.org/D20721>
> to make `lit` output more readable when commands fail. I'd like to get
a little more feedback on this before landing, since it impacts how test
failures appear when using the internal shell script runner (which is not the
default so this will currently mostly impact Windows or test frameworks that
intentionally set this behavior).
> 
> These patches change the output of a script like `true && echo hi
&& false` to something like:
What would the output of "echo '$ echo hi ' " be and how do
you differentiate from '$ echo hi' ?

-- 
Mehdi
> 
> ```
> $ true
> $ echo hi
> hi
>     
> $ false
> note: command had no output on stdout or stderr
> error: command failed with exit status 1
> ```
> 
> instead of the old:
> 
> ```
> Command 0: "true"
> Command 0 Result: 0
> Command 0 Output:
> 
> 
> Command 0 Stderr:
> 
> 
> Command 1: "echo" "hi"
> Command 1 Result: 0
> Command 1 Output:
> hi
> 
> 
> Command 1 Stderr:
> 
> 
> Command 2: "wc" "missing-file"
> Command 2 Result: 1
> Command 2 Output:
> None
> 
> Command 2 Stderr:
> None
> ```
> 
> which I think is significantly more readable.
> 
> One change in the patches is that it will also start including, inline, the
output of files which were redirected. This solves a common problem where a 
command fails when it wasn't expected to, and its output was captured. In
such cases the test log just showed the command failure, but since the output
was redirected you couldn't tell from the log why the common failed.
> 
> Please let me know if you have any objections to the new format.
> 
>  - Daniel
> 
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160531/e74bb1e1/attachment.html>

(+ the real cfe-dev)

On Tue, May 31, 2016 at 4:48 PM, Daniel Dunbar <daniel at zuster.org>
wrote:
> Hi all,
>
> I have some patches up in:
>   http://reviews.llvm.org/D20721
> to make `lit` output more readable when commands fail. I'd like to get
a
> little more feedback on this before landing, since it impacts how test
> failures appear when using the internal shell script runner (which is not
> the default so this will currently mostly impact Windows or test frameworks
> that intentionally set this behavior).
>
> These patches change the output of a script like `true && echo hi
&&
> false` to something like:
>
> ```
> $ true
> $ echo hi
> hi
>
> $ false
> note: command had no output on stdout or stderr
> error: command failed with exit status 1
> ```
>
> instead of the old:
>
> ```
> Command 0: "true"
> Command 0 Result: 0
> Command 0 Output:
>
>
> Command 0 Stderr:
>
>
> Command 1: "echo" "hi"
> Command 1 Result: 0
> Command 1 Output:
> hi
>
>
> Command 1 Stderr:
>
>
> Command 2: "wc" "missing-file"
> Command 2 Result: 1
> Command 2 Output:
> None
>
> Command 2 Stderr:
> None
> ```
>
> which I think is significantly more readable.
>
> One change in the patches is that it will also start including, inline,
> the output of files which were redirected. This solves a common problem
> where a  command fails when it wasn't expected to, and its output was
> captured. In such cases the test log just showed the command failure, but
> since the output was redirected you couldn't tell from the log why the
> common failed.
>
> Please let me know if you have any objections to the new format.
>
>  - Daniel
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160531/49d93764/attachment.html>

As long as the individual commands are still copy-pastable into both cmd
and bash, I'm happy. I think practice this means the output should look
more like this:

some test:
// RUN: %clang_cc1 %s -verify

output:
$ "C:\\src\\llvm\\build\\bin\\clang.exe" -cc1
"C:\\....\\test.cpp" -verify

Essentially every program used in the LLVM test suite understands forward
slashes, so we could rewrite the path to test.cpp to use forward slashes,
but I think cmd wants to see backslashes in argv0. Then our output would
look like:
$ "C:\\src\\llvm\\build\\bin\\clang.exe" -cc1 C:/src/llvm/.../test.cpp
-verify

On Tue, May 31, 2016 at 4:53 PM, Daniel Dunbar via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> (+ the real cfe-dev)
>
>
> On Tue, May 31, 2016 at 4:48 PM, Daniel Dunbar <daniel at zuster.org>
wrote:
>
>> Hi all,
>>
>> I have some patches up in:
>>   http://reviews.llvm.org/D20721
>> to make `lit` output more readable when commands fail. I'd like to
get a
>> little more feedback on this before landing, since it impacts how test
>> failures appear when using the internal shell script runner (which is
not
>> the default so this will currently mostly impact Windows or test
frameworks
>> that intentionally set this behavior).
>>
>> These patches change the output of a script like `true && echo
hi &&
>> false` to something like:
>>
>> ```
>> $ true
>> $ echo hi
>> hi
>>
>> $ false
>> note: command had no output on stdout or stderr
>> error: command failed with exit status 1
>> ```
>>
>> instead of the old:
>>
>> ```
>> Command 0: "true"
>> Command 0 Result: 0
>> Command 0 Output:
>>
>>
>> Command 0 Stderr:
>>
>>
>> Command 1: "echo" "hi"
>> Command 1 Result: 0
>> Command 1 Output:
>> hi
>>
>>
>> Command 1 Stderr:
>>
>>
>> Command 2: "wc" "missing-file"
>> Command 2 Result: 1
>> Command 2 Output:
>> None
>>
>> Command 2 Stderr:
>> None
>> ```
>>
>> which I think is significantly more readable.
>>
>> One change in the patches is that it will also start including, inline,
>> the output of files which were redirected. This solves a common problem
>> where a  command fails when it wasn't expected to, and its output
was
>> captured. In such cases the test log just showed the command failure,
but
>> since the output was redirected you couldn't tell from the log why
the
>> common failed.
>>
>> Please let me know if you have any objections to the new format.
>>
>>  - Daniel
>>
>>
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160531/1badb803/attachment.html>

On Tue, May 31, 2016 at 4:52 PM, Mehdi Amini <mehdi.amini at apple.com>
wrote:
>
> On May 31, 2016, at 4:48 PM, Daniel Dunbar via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
> Hi all,
>
> I have some patches up in:
>   http://reviews.llvm.org/D20721
> to make `lit` output more readable when commands fail. I'd like to get
a
> little more feedback on this before landing, since it impacts how test
> failures appear when using the internal shell script runner (which is not
> the default so this will currently mostly impact Windows or test frameworks
> that intentionally set this behavior).
>
> These patches change the output of a script like `true && echo hi
&&
> false` to something like:
>
>
> What would the output of "echo '$ echo hi ' " be and how
do you
> differentiate from '$ echo hi' ?
>
"Don't do that"?

Actually, the output I gave wasn't complete, what we actually currently
show is:
```
$ "echo" "hi"
# command output:
hi
```
but of course your comment still applies. However, the goal is to make for
a readable output for the majority of use cases, not necessarily to be
unambiguous.

 - Daniel

>
> --
> Mehdi
>
>
> ```
> $ true
> $ echo hi
> hi
>
> $ false
> note: command had no output on stdout or stderr
> error: command failed with exit status 1
> ```
>
> instead of the old:
>
> ```
> Command 0: "true"
> Command 0 Result: 0
> Command 0 Output:
>
>
> Command 0 Stderr:
>
>
> Command 1: "echo" "hi"
> Command 1 Result: 0
> Command 1 Output:
> hi
>
>
> Command 1 Stderr:
>
>
> Command 2: "wc" "missing-file"
> Command 2 Result: 1
> Command 2 Output:
> None
>
> Command 2 Stderr:
> None
> ```
>
> which I think is significantly more readable.
>
> One change in the patches is that it will also start including, inline,
> the output of files which were redirected. This solves a common problem
> where a  command fails when it wasn't expected to, and its output was
> captured. In such cases the test log just showed the command failure, but
> since the output was redirected you couldn't tell from the log why the
> common failed.
>
> Please let me know if you have any objections to the new format.
>
>  - Daniel
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160531/0cf6a61a/attachment.html>