Hello, I am not sold on FileCheck's new behaviour. For failing tests in verbose mode, it first dump the actual error messages, followed by the annotated input file to FileCheck. The result is I can't immediately see error messages if the input is more than just a few lines long, so I have to scroll all the way up to see the errors, then down again, etc. I do see some advantages of dumping the input to FileCheck, but an improvement for me would be: * to dump the input first, then followed by the error message, so that I can the errors first, and then decide to scroll up if I am interested to do so. * dump it to a separate file (controlled with an option). I am interested in changing the behaviour, because I think I find setting environment varibale "FILECHECK_OPTS="--dump-input never"" inconvenient. My 2 pennies. Sjoerd. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20200618/17fa984e/attachment.html>
> On Jun 18, 2020, at 12:15, Sjoerd Meijer via llvm-dev <llvm-dev at lists.llvm.org> wrote: > > Hello, > > I am not sold on FileCheck's new behaviour. For failing tests in verbose mode, it first dump the actual error messages, followed by the annotated input file to FileCheck. The result is I can't immediately see error messages if the input is more than just a few lines long, so I have to scroll all the way up to see the errors, then down again, etc. > > I do see some advantages of dumping the input to FileCheck, but an improvement for me would be: > to dump the input first, then followed by the error message, so that I can the errors first, and then decide to scroll up if I am interested to do so. > dump it to a separate file (controlled with an option). > I am interested in changing the behaviour, because I think I find setting environment varibale "FILECHECK_OPTS="--dump-input never"" inconvenient. > > My 2 pennies. > Sjoerd. > _______________________________________________+1. The new behavior makes sense if you break 1 test with 100 lines of output. It stops making sense if you break 20 tests with 1000s of lines of output. Some more aggressive dumping by default on buildbots may make sense -Matt -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20200618/767ff640/attachment.html>
For anybody viewing these failures through some sort of CI system, showing the error first then the input file is more useful for the same reasons you mentioned. Personally, I rarely run filecheck by hand from the command prompt, so your change would make my life worse. Granted, I'm just one person. The point I'm trying to make is that I don't think it's clear-cut which order is better, so maybe we shouldn't change it. I think it might be fine to add an option to swap the order, but I'd be very sad if it started dumping to some random file by default. Thanks, Christopher Tetreault From: llvm-dev <llvm-dev-bounces at lists.llvm.org> On Behalf Of Sjoerd Meijer via llvm-dev Sent: Thursday, June 18, 2020 9:16 AM To: llvm-dev at lists.llvm.org Subject: [EXT] [llvm-dev] FileCheck Hello, I am not sold on FileCheck's new behaviour. For failing tests in verbose mode, it first dump the actual error messages, followed by the annotated input file to FileCheck. The result is I can't immediately see error messages if the input is more than just a few lines long, so I have to scroll all the way up to see the errors, then down again, etc. I do see some advantages of dumping the input to FileCheck, but an improvement for me would be: * to dump the input first, then followed by the error message, so that I can the errors first, and then decide to scroll up if I am interested to do so. * dump it to a separate file (controlled with an option). I am interested in changing the behaviour, because I think I find setting environment varibale "FILECHECK_OPTS="--dump-input never"" inconvenient. My 2 pennies. Sjoerd. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20200618/c412f97e/attachment.html>
On 6/18/20 11:34 AM, Chris Tetreault via llvm-dev wrote:> For anybody viewing these failures through some sort of CI system, showing the error first then the input file is more useful for the same reasons you mentioned. Personally, I rarely run filecheck by hand from the command prompt, so your change would make my life worse. Granted, I'm just one person.I think a lot of people run filecheck by hand, at least everyone that does not have a proper* CI system set up.> The point I'm trying to make is that I don't think it's clear-cut which order is better, so maybe we shouldn't change it. I think it might be fine to add an option to swap the order, but I'd be very sad if it started dumping to some random file by default.The order can be chosen with a flag and environment variable. We could even check if the output is a file or not to select a sensible default. That said, I think the default should make sense for people that just download LLVM(, modify something,) and run the tests. If you set up your CI you can add proper configuration to change this. Cheers, Johannes> Thanks, > Christopher Tetreault > > From: llvm-dev <llvm-dev-bounces at lists.llvm.org> On Behalf Of Sjoerd Meijer via llvm-dev > Sent: Thursday, June 18, 2020 9:16 AM > To: llvm-dev at lists.llvm.org > Subject: [EXT] [llvm-dev] FileCheck > > Hello, > > I am not sold on FileCheck's new behaviour. For failing tests in verbose mode, it first dump the actual error messages, followed by the annotated input file to FileCheck. The result is I can't immediately see error messages if the input is more than just a few lines long, so I have to scroll all the way up to see the errors, then down again, etc. > > I do see some advantages of dumping the input to FileCheck, but an improvement for me would be: > > * to dump the input first, then followed by the error message, so that I can the errors first, and then decide to scroll up if I am interested to do so. > * dump it to a separate file (controlled with an option). > I am interested in changing the behaviour, because I think I find setting environment varibale "FILECHECK_OPTS="--dump-input never"" inconvenient. > > My 2 pennies. > Sjoerd. > > > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20200618/75216768/attachment.html>
I would guess that in a CI system the order doesn't matter much because you look at a webpage? I looked at some build bots today/yesterday that now also show this, and yeah, it's fine either way, I was guessing. My primary use-case is usage in a terminal, and displaying the errors first followed by all input makes this pretty unusable. ________________________________ From: Chris Tetreault <ctetreau at quicinc.com> Sent: 18 June 2020 17:34 To: Sjoerd Meijer <Sjoerd.Meijer at arm.com> Cc: llvm-dev at lists.llvm.org <llvm-dev at lists.llvm.org> Subject: RE: [llvm-dev] FileCheck For anybody viewing these failures through some sort of CI system, showing the error first then the input file is more useful for the same reasons you mentioned. Personally, I rarely run filecheck by hand from the command prompt, so your change would make my life worse. Granted, I’m just one person. The point I’m trying to make is that I don’t think it’s clear-cut which order is better, so maybe we shouldn’t change it. I think it might be fine to add an option to swap the order, but I’d be very sad if it started dumping to some random file by default. Thanks, Christopher Tetreault From: llvm-dev <llvm-dev-bounces at lists.llvm.org> On Behalf Of Sjoerd Meijer via llvm-dev Sent: Thursday, June 18, 2020 9:16 AM To: llvm-dev at lists.llvm.org Subject: [EXT] [llvm-dev] FileCheck Hello, I am not sold on FileCheck's new behaviour. For failing tests in verbose mode, it first dump the actual error messages, followed by the annotated input file to FileCheck. The result is I can't immediately see error messages if the input is more than just a few lines long, so I have to scroll all the way up to see the errors, then down again, etc. I do see some advantages of dumping the input to FileCheck, but an improvement for me would be: * to dump the input first, then followed by the error message, so that I can the errors first, and then decide to scroll up if I am interested to do so. * dump it to a separate file (controlled with an option). I am interested in changing the behaviour, because I think I find setting environment varibale "FILECHECK_OPTS="--dump-input never"" inconvenient. My 2 pennies. Sjoerd. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20200618/c9f49a9d/attachment.html>