> I don't know how you proceed to debug FileCheck failures, but for me most of the time I'll have to figure out which "RUN" line fail and try to execute it manually and then remove the FileCheck pipe to get the raw input and then painfully tried to match the FileCheck error to the actual input.Yeah, not very different from what you described here. If I 'm creating or editing a test file, just reading the error message catches a lot of cases for me. But yes, if it is more complicated I do exactly what you said, which is definitely a bit labour intensive, not ideal, and that's why I also think dumping the input to FileCheck is very useful, but for me not in its current form. For that, first, I think we only need to dump the context, and not the entire file, as also remarked earlier in this thread. Second, we can discuss the order of the output (or if is dumped to a file/stdout/stderr).> Matt mentioned that some codegen tests are very long: reducing the output by being contextual would likely help, but I'm also not convinced that this is a good practice to have so huge test in single files in the first place, it seems to me almost like anti-pattern and if it gets hard to read it is a good sign that it could benefit from some sharding.While I agree with this, I think the structure of the tests is a separate discussion. That is, I think you can have 1 test-case in 1 file, and the current behaviour is already inconvenient, so don't see that as a solution. Cheers. ________________________________ From: Mehdi AMINI <joker.eph at gmail.com> Sent: 19 June 2020 09:32 To: Sjoerd Meijer <Sjoerd.Meijer at arm.com> Cc: Chris Tetreault <ctetreau at quicinc.com>; Joel E. Denny <jdenny.ornl at gmail.com>; llvm-dev at lists.llvm.org <llvm-dev at lists.llvm.org> Subject: Re: [llvm-dev] FileCheck On Fri, Jun 19, 2020 at 12:56 AM Sjoerd Meijer via llvm-dev <llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org>> wrote: Sorry if I wasn't clear about my use case. In my daily dev work, I do many local "ninja check"s, or "llvm-lit" on a subdirectory as a quick(er) smoke test if I am making changes in that area (e.g. "llvm-lit ../llvm/test/CodeGen"). Nothing wrong here, as indeed nothing changed here. But in case of a test failure, I want to run just that test: bin/llvm-lit ../llvm/test/CodeGen/my_test.ll This only reports success/failure, and doesn't show any cause for failure , so I run it in verbose mode with: bin/llvm-lit -a ../llvm/test/CodeGen/my_test.ll In a terminal, the new default behaviour of FileCheck has become pretty unusable IMHO. I don't know how you proceed to debug FileCheck failures, but for me most of the time I'll have to figure out which "RUN" line fail and try to execute it manually and then remove the FileCheck pipe to get the raw input and then painfully tried to match the FileCheck error to the actual input. At least that's what I used to do before check-input=fail: this changed my relationship with debugging Filecheck failures! Matt mentioned that some codegen tests are very long: reducing the output by being contextual would likely help, but I'm also not convinced that this is a good practice to have so huge test in single files in the first place, it seems to me almost like anti-pattern and if it gets hard to read it is a good sign that it could benefit from some sharding. -- Mehdi ________________________________ From: Chris Tetreault <ctetreau at quicinc.com<mailto:ctetreau at quicinc.com>> Sent: 18 June 2020 20:49 To: Joel E. Denny <jdenny.ornl at gmail.com<mailto:jdenny.ornl at gmail.com>> Cc: Sjoerd Meijer <Sjoerd.Meijer at arm.com<mailto:Sjoerd.Meijer at arm.com>>; llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org> <llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org>> Subject: RE: [llvm-dev] FileCheck Sjoerd specifically said “in verbose mode”, which I interpret to mean “when passing -v or -vv”. If we’re discussing the default behavior, then that’s a separate issue. Regardless, my other points stand independent of that. From: Joel E. Denny <jdenny.ornl at gmail.com<mailto:jdenny.ornl at gmail.com>> Sent: Thursday, June 18, 2020 12:43 PM To: Chris Tetreault <ctetreau at quicinc.com<mailto:ctetreau at quicinc.com>> Cc: Sjoerd Meijer <Sjoerd.Meijer at arm.com<mailto:Sjoerd.Meijer at arm.com>>; llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org> Subject: [EXT] Re: [llvm-dev] FileCheck On Thu, Jun 18, 2020 at 3:37 PM Chris Tetreault <ctetreau at quicinc.com<mailto:ctetreau at quicinc.com>> wrote: We’re talking about verbose output right? Verbose isn’t the default. I'm fairly certain the issue in this thread is just the verbosity of -dump-input=fail. Yes, -vv makes it even more verbose by annotating input lines with good matches, etc., but that's not part of the "new behaviour" Sjoerd meant, I believe. Joel From: Joel E. Denny <jdenny.ornl at gmail.com<mailto:jdenny.ornl at gmail.com>> Sent: Thursday, June 18, 2020 10:54 AM To: Chris Tetreault <ctetreau at quicinc.com<mailto:ctetreau at quicinc.com>> Cc: Sjoerd Meijer <Sjoerd.Meijer at arm.com<mailto:Sjoerd.Meijer at arm.com>>; llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org> Subject: [EXT] Re: [llvm-dev] FileCheck Hi Chris, On Thu, Jun 18, 2020 at 1:37 PM Chris Tetreault via llvm-dev <llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org>> wrote: The thing I use normally only shows the first N lines by default (I don’t know off hand what N is). Honestly, I don’t feel very strongly about the specific order, but it’s not useful when somebody proposes something on the list, and nobody voices any dissent (choosing instead to silently oppose the change). My requests would be: 1. The order should be customizable via command line. 2. By default, it should not dump things to multiple locations. If I ask for verbose output, I want to get blasted with all the stuff. 3. The most important thing for me personally is to see the input to filecheck (I realize that this is in conflict with my earlier point. It’s early and I hadn’t had my coffee 😊 ). When I get a failure I want to be able to reproduce it in an IDE to use a debugger. Any change should not make this use case harder. Personally, I do not find the argument that the defaults should be setup to be best for newcomers to be very compelling; we are talking about changing the behavior of a non-default option after all. What do you mean by a "non-default option"? The default of -dump-input=never was recently changed to -dump-input=fail. Joel If just a bare filecheck invocation doesn’t tell a newcomer what they need to know, then they have to do filecheck -help or google the documentation anyways. At that point, they are going to customize it however they want. I assume anybody using filecheck to debug an issue is tech savvy enough to be able to configure the options, given reasonable documentation. Thanks, Christopher Tetreault From: Sjoerd Meijer <Sjoerd.Meijer at arm.com<mailto:Sjoerd.Meijer at arm.com>> Sent: Thursday, June 18, 2020 9:45 AM To: Chris Tetreault <ctetreau at quicinc.com<mailto:ctetreau at quicinc.com>> Cc: llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org> Subject: [EXT] Re: [llvm-dev] FileCheck I would guess that in a CI system the order doesn't matter much because you look at a webpage? I looked at some build bots today/yesterday that now also show this, and yeah, it's fine either way, I was guessing. My primary use-case is usage in a terminal, and displaying the errors first followed by all input makes this pretty unusable. ________________________________ From: Chris Tetreault <ctetreau at quicinc.com<mailto:ctetreau at quicinc.com>> Sent: 18 June 2020 17:34 To: Sjoerd Meijer <Sjoerd.Meijer at arm.com<mailto:Sjoerd.Meijer at arm.com>> Cc: llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org> <llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org>> Subject: RE: [llvm-dev] FileCheck For anybody viewing these failures through some sort of CI system, showing the error first then the input file is more useful for the same reasons you mentioned. Personally, I rarely run filecheck by hand from the command prompt, so your change would make my life worse. Granted, I’m just one person. The point I’m trying to make is that I don’t think it’s clear-cut which order is better, so maybe we shouldn’t change it. I think it might be fine to add an option to swap the order, but I’d be very sad if it started dumping to some random file by default. Thanks, Christopher Tetreault From: llvm-dev <llvm-dev-bounces at lists.llvm.org<mailto:llvm-dev-bounces at lists.llvm.org>> On Behalf Of Sjoerd Meijer via llvm-dev Sent: Thursday, June 18, 2020 9:16 AM To: llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org> Subject: [EXT] [llvm-dev] FileCheck Hello, I am not sold on FileCheck's new behaviour. For failing tests in verbose mode, it first dump the actual error messages, followed by the annotated input file to FileCheck. The result is I can't immediately see error messages if the input is more than just a few lines long, so I have to scroll all the way up to see the errors, then down again, etc. I do see some advantages of dumping the input to FileCheck, but an improvement for me would be: * to dump the input first, then followed by the error message, so that I can the errors first, and then decide to scroll up if I am interested to do so. * dump it to a separate file (controlled with an option). I am interested in changing the behaviour, because I think I find setting environment varibale "FILECHECK_OPTS="--dump-input never"" inconvenient. My 2 pennies. Sjoerd. _______________________________________________ LLVM Developers mailing list llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev _______________________________________________ LLVM Developers mailing list llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20200619/f4de009a/attachment.html>
Ah, I forgot to ask this:> At least that's what I used to do before check-input=fail: this changed my relationship with debugging Filecheck failures!What do you mean by this? Curious if I am missing a neat trick here? ________________________________ From: Sjoerd Meijer <Sjoerd.Meijer at arm.com> Sent: 19 June 2020 09:56 To: Mehdi AMINI <joker.eph at gmail.com> Cc: Chris Tetreault <ctetreau at quicinc.com>; Joel E. Denny <jdenny.ornl at gmail.com>; llvm-dev at lists.llvm.org <llvm-dev at lists.llvm.org> Subject: Re: [llvm-dev] FileCheck> I don't know how you proceed to debug FileCheck failures, but for me most of the time I'll have to figure out which "RUN" line fail and try to execute it manually and then remove the FileCheck pipe to get the raw input and then painfully tried to match the FileCheck error to the actual input.Yeah, not very different from what you described here. If I 'm creating or editing a test file, just reading the error message catches a lot of cases for me. But yes, if it is more complicated I do exactly what you said, which is definitely a bit labour intensive, not ideal, and that's why I also think dumping the input to FileCheck is very useful, but for me not in its current form. For that, first, I think we only need to dump the context, and not the entire file, as also remarked earlier in this thread. Second, we can discuss the order of the output (or if is dumped to a file/stdout/stderr).> Matt mentioned that some codegen tests are very long: reducing the output by being contextual would likely help, but I'm also not convinced that this is a good practice to have so huge test in single files in the first place, it seems to me almost like anti-pattern and if it gets hard to read it is a good sign that it could benefit from some sharding.While I agree with this, I think the structure of the tests is a separate discussion. That is, I think you can have 1 test-case in 1 file, and the current behaviour is already inconvenient, so don't see that as a solution. Cheers. ________________________________ From: Mehdi AMINI <joker.eph at gmail.com> Sent: 19 June 2020 09:32 To: Sjoerd Meijer <Sjoerd.Meijer at arm.com> Cc: Chris Tetreault <ctetreau at quicinc.com>; Joel E. Denny <jdenny.ornl at gmail.com>; llvm-dev at lists.llvm.org <llvm-dev at lists.llvm.org> Subject: Re: [llvm-dev] FileCheck On Fri, Jun 19, 2020 at 12:56 AM Sjoerd Meijer via llvm-dev <llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org>> wrote: Sorry if I wasn't clear about my use case. In my daily dev work, I do many local "ninja check"s, or "llvm-lit" on a subdirectory as a quick(er) smoke test if I am making changes in that area (e.g. "llvm-lit ../llvm/test/CodeGen"). Nothing wrong here, as indeed nothing changed here. But in case of a test failure, I want to run just that test: bin/llvm-lit ../llvm/test/CodeGen/my_test.ll This only reports success/failure, and doesn't show any cause for failure , so I run it in verbose mode with: bin/llvm-lit -a ../llvm/test/CodeGen/my_test.ll In a terminal, the new default behaviour of FileCheck has become pretty unusable IMHO. I don't know how you proceed to debug FileCheck failures, but for me most of the time I'll have to figure out which "RUN" line fail and try to execute it manually and then remove the FileCheck pipe to get the raw input and then painfully tried to match the FileCheck error to the actual input. At least that's what I used to do before check-input=fail: this changed my relationship with debugging Filecheck failures! Matt mentioned that some codegen tests are very long: reducing the output by being contextual would likely help, but I'm also not convinced that this is a good practice to have so huge test in single files in the first place, it seems to me almost like anti-pattern and if it gets hard to read it is a good sign that it could benefit from some sharding. -- Mehdi ________________________________ From: Chris Tetreault <ctetreau at quicinc.com<mailto:ctetreau at quicinc.com>> Sent: 18 June 2020 20:49 To: Joel E. Denny <jdenny.ornl at gmail.com<mailto:jdenny.ornl at gmail.com>> Cc: Sjoerd Meijer <Sjoerd.Meijer at arm.com<mailto:Sjoerd.Meijer at arm.com>>; llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org> <llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org>> Subject: RE: [llvm-dev] FileCheck Sjoerd specifically said “in verbose mode”, which I interpret to mean “when passing -v or -vv”. If we’re discussing the default behavior, then that’s a separate issue. Regardless, my other points stand independent of that. From: Joel E. Denny <jdenny.ornl at gmail.com<mailto:jdenny.ornl at gmail.com>> Sent: Thursday, June 18, 2020 12:43 PM To: Chris Tetreault <ctetreau at quicinc.com<mailto:ctetreau at quicinc.com>> Cc: Sjoerd Meijer <Sjoerd.Meijer at arm.com<mailto:Sjoerd.Meijer at arm.com>>; llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org> Subject: [EXT] Re: [llvm-dev] FileCheck On Thu, Jun 18, 2020 at 3:37 PM Chris Tetreault <ctetreau at quicinc.com<mailto:ctetreau at quicinc.com>> wrote: We’re talking about verbose output right? Verbose isn’t the default. I'm fairly certain the issue in this thread is just the verbosity of -dump-input=fail. Yes, -vv makes it even more verbose by annotating input lines with good matches, etc., but that's not part of the "new behaviour" Sjoerd meant, I believe. Joel From: Joel E. Denny <jdenny.ornl at gmail.com<mailto:jdenny.ornl at gmail.com>> Sent: Thursday, June 18, 2020 10:54 AM To: Chris Tetreault <ctetreau at quicinc.com<mailto:ctetreau at quicinc.com>> Cc: Sjoerd Meijer <Sjoerd.Meijer at arm.com<mailto:Sjoerd.Meijer at arm.com>>; llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org> Subject: [EXT] Re: [llvm-dev] FileCheck Hi Chris, On Thu, Jun 18, 2020 at 1:37 PM Chris Tetreault via llvm-dev <llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org>> wrote: The thing I use normally only shows the first N lines by default (I don’t know off hand what N is). Honestly, I don’t feel very strongly about the specific order, but it’s not useful when somebody proposes something on the list, and nobody voices any dissent (choosing instead to silently oppose the change). My requests would be: 1. The order should be customizable via command line. 2. By default, it should not dump things to multiple locations. If I ask for verbose output, I want to get blasted with all the stuff. 3. The most important thing for me personally is to see the input to filecheck (I realize that this is in conflict with my earlier point. It’s early and I hadn’t had my coffee 😊 ). When I get a failure I want to be able to reproduce it in an IDE to use a debugger. Any change should not make this use case harder. Personally, I do not find the argument that the defaults should be setup to be best for newcomers to be very compelling; we are talking about changing the behavior of a non-default option after all. What do you mean by a "non-default option"? The default of -dump-input=never was recently changed to -dump-input=fail. Joel If just a bare filecheck invocation doesn’t tell a newcomer what they need to know, then they have to do filecheck -help or google the documentation anyways. At that point, they are going to customize it however they want. I assume anybody using filecheck to debug an issue is tech savvy enough to be able to configure the options, given reasonable documentation. Thanks, Christopher Tetreault From: Sjoerd Meijer <Sjoerd.Meijer at arm.com<mailto:Sjoerd.Meijer at arm.com>> Sent: Thursday, June 18, 2020 9:45 AM To: Chris Tetreault <ctetreau at quicinc.com<mailto:ctetreau at quicinc.com>> Cc: llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org> Subject: [EXT] Re: [llvm-dev] FileCheck I would guess that in a CI system the order doesn't matter much because you look at a webpage? I looked at some build bots today/yesterday that now also show this, and yeah, it's fine either way, I was guessing. My primary use-case is usage in a terminal, and displaying the errors first followed by all input makes this pretty unusable. ________________________________ From: Chris Tetreault <ctetreau at quicinc.com<mailto:ctetreau at quicinc.com>> Sent: 18 June 2020 17:34 To: Sjoerd Meijer <Sjoerd.Meijer at arm.com<mailto:Sjoerd.Meijer at arm.com>> Cc: llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org> <llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org>> Subject: RE: [llvm-dev] FileCheck For anybody viewing these failures through some sort of CI system, showing the error first then the input file is more useful for the same reasons you mentioned. Personally, I rarely run filecheck by hand from the command prompt, so your change would make my life worse. Granted, I’m just one person. The point I’m trying to make is that I don’t think it’s clear-cut which order is better, so maybe we shouldn’t change it. I think it might be fine to add an option to swap the order, but I’d be very sad if it started dumping to some random file by default. Thanks, Christopher Tetreault From: llvm-dev <llvm-dev-bounces at lists.llvm.org<mailto:llvm-dev-bounces at lists.llvm.org>> On Behalf Of Sjoerd Meijer via llvm-dev Sent: Thursday, June 18, 2020 9:16 AM To: llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org> Subject: [EXT] [llvm-dev] FileCheck Hello, I am not sold on FileCheck's new behaviour. For failing tests in verbose mode, it first dump the actual error messages, followed by the annotated input file to FileCheck. The result is I can't immediately see error messages if the input is more than just a few lines long, so I have to scroll all the way up to see the errors, then down again, etc. I do see some advantages of dumping the input to FileCheck, but an improvement for me would be: * to dump the input first, then followed by the error message, so that I can the errors first, and then decide to scroll up if I am interested to do so. * dump it to a separate file (controlled with an option). I am interested in changing the behaviour, because I think I find setting environment varibale "FILECHECK_OPTS="--dump-input never"" inconvenient. My 2 pennies. Sjoerd. _______________________________________________ LLVM Developers mailing list llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev _______________________________________________ LLVM Developers mailing list llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20200619/79723f5c/attachment-0001.html>
On Fri, Jun 19, 2020 at 1:59 AM Sjoerd Meijer <Sjoerd.Meijer at arm.com> wrote:> Ah, I forgot to ask this: > > > At least that's what I used to do before check-input=fail: this changed > my relationship with debugging Filecheck failures! > > What do you mean by this? Curious if I am missing a neat trick here? >Nothing super tricky. For example I swapped two lines in a test: // CHECK: [[VAL_16:%.*]] = muli [[NEW_I1]], [[C13]] : index // CHECK: [[I0:%.*]] = remi_signed [[NEW_I0]], [[C2]] : index And ran it with `./bin/llvm-lit -sv ../mlir/test/Transforms/parallel-loop-collapsing.mlir`. The output is unsurprisingly: llvm-project/mlir/test/Transforms/parallel-loop-collapsing.mlir:40:11: error: CHECK: expected string not found in input // CHECK: [[I0:%.*]] = remi_signed [[NEW_I0]], [[C2]] : index ^ <stdin>:18:2: note: scanning from here %2 = addi %1, %c12 : index ^ There isn't enough information to do much here, while dump-input=fail adds: Full input was: <<<<<< 1: 2: 3: module { 4: func @parallel_many_dims() { 5: %c6 = constant 6 : index 6: %c7 = constant 7 : index 7: %c9 = constant 9 : index 8: %c10 = constant 10 : index 9: %c12 = constant 12 : index 10: %c13 = constant 13 : index 11: %c3 = constant 3 : index 12: %c0 = constant 0 : index 13: %c1 = constant 1 : index 14: %c2 = constant 2 : index 15: scf.parallel (%arg0, %arg1, %arg2) = (%c0, %c0, %c0) to (%c2, %c1, %c1) step (%c1, %c1, %c1) { 16: %0 = remi_signed %arg0, %c2 : index 17: %1 = muli %arg1, %c13 : index 18: %2 = addi %1, %c12 : index check:40 X~~~~~~~~~~~~~~~~~~~~~~~~~ error: no match found 19: %3 = muli %arg0, %c10 : index check:40 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 20: %4 = addi %3, %c9 : index check:40 ~~~~~~~~~~~~~~~~~~~~~~~~~~ 21: %5 = muli %arg2, %c7 : index check:40 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 22: %6 = addi %5, %c6 : index check:40 ~~~~~~~~~~~~~~~~~~~~~~~~~~ 23: %7 = "magic.op"(%0, %c3, %6, %4, %2) : (index, index, index, index, index) -> index check:40 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 24: scf.yield check:40 ~~~~~~~~~~ 25: } check:40 ~~ 26: return check:40 ~~~~~~~ 27: } check:40 ~~ 28: } check:40 ~>>>>>>Which is all I need to understand where/how FileCheck got stuck. -- Mehdi> ------------------------------ > *From:* Sjoerd Meijer <Sjoerd.Meijer at arm.com> > *Sent:* 19 June 2020 09:56 > *To:* Mehdi AMINI <joker.eph at gmail.com> > *Cc:* Chris Tetreault <ctetreau at quicinc.com>; Joel E. Denny < > jdenny.ornl at gmail.com>; llvm-dev at lists.llvm.org <llvm-dev at lists.llvm.org> > *Subject:* Re: [llvm-dev] FileCheck > > > I don't know how you proceed to debug FileCheck failures, but for me > most of the time I'll have to figure out which "RUN" line fail and try to > execute it manually and then remove the FileCheck pipe to get the raw input > and then painfully tried to match the FileCheck error to the actual input. > > Yeah, not very different from what you described here. If I 'm creating or > editing a test file, just reading the error message catches a lot of cases > for me. But yes, if it is more complicated I do exactly what you said, > which is definitely a bit labour intensive, not ideal, and that's why I > also think dumping the input to FileCheck is very useful, but for me not in > its current form. For that, first, I think we only need to dump the > context, and not the entire file, as also remarked earlier in this thread. > Second, we can discuss the order of the output (or if is dumped to a > file/stdout/stderr). > > > Matt mentioned that some codegen tests are very long: reducing the > output by being contextual would likely help, but I'm also not convinced > that this is a good practice to have so huge test in single files in the > first place, it seems to me almost like anti-pattern and if it gets hard to > read it is a good sign that it could benefit from some sharding. > > While I agree with this, I think the structure of the tests is a separate > discussion. That is, I think you can have 1 test-case in 1 file, and the > current behaviour is already inconvenient, so don't see that as a solution. > > Cheers. > > > ------------------------------ > *From:* Mehdi AMINI <joker.eph at gmail.com> > *Sent:* 19 June 2020 09:32 > *To:* Sjoerd Meijer <Sjoerd.Meijer at arm.com> > *Cc:* Chris Tetreault <ctetreau at quicinc.com>; Joel E. Denny < > jdenny.ornl at gmail.com>; llvm-dev at lists.llvm.org <llvm-dev at lists.llvm.org> > *Subject:* Re: [llvm-dev] FileCheck > > > > On Fri, Jun 19, 2020 at 12:56 AM Sjoerd Meijer via llvm-dev < > llvm-dev at lists.llvm.org> wrote: > > Sorry if I wasn't clear about my use case. In my daily dev work, I do many > local "ninja check"s, or "llvm-lit" on a subdirectory as a quick(er) smoke > test if I am making changes in that area (e.g. "llvm-lit > ../llvm/test/CodeGen"). Nothing wrong here, as indeed nothing changed here. > But in case of a test failure, I want to run just that test: > > bin/llvm-lit ../llvm/test/CodeGen/my_test.ll > > This only reports success/failure, and doesn't show any cause for failure > , so I run it in verbose mode with: > > bin/llvm-lit -a ../llvm/test/CodeGen/my_test.ll > > In a terminal, the new default behaviour of FileCheck has become pretty > unusable IMHO. > > > I don't know how you proceed to debug FileCheck failures, but for me most > of the time I'll have to figure out which "RUN" line fail and try to > execute it manually and then remove the FileCheck pipe to get the raw input > and then painfully tried to match the FileCheck error to the actual input. > At least that's what I used to do before check-input=fail: this changed my > relationship with debugging Filecheck failures! > > Matt mentioned that some codegen tests are very long: reducing the output > by being contextual would likely help, but I'm also not convinced that this > is a good practice to have so huge test in single files in the first place, > it seems to me almost like anti-pattern and if it gets hard to read it is a > good sign that it could benefit from some sharding. > > > -- > Mehdi > > > > > > ------------------------------ > *From:* Chris Tetreault <ctetreau at quicinc.com> > *Sent:* 18 June 2020 20:49 > *To:* Joel E. Denny <jdenny.ornl at gmail.com> > *Cc:* Sjoerd Meijer <Sjoerd.Meijer at arm.com>; llvm-dev at lists.llvm.org < > llvm-dev at lists.llvm.org> > *Subject:* RE: [llvm-dev] FileCheck > > > Sjoerd specifically said “in verbose mode”, which I interpret to mean > “when passing -v or -vv”. If we’re discussing the default behavior, then > that’s a separate issue. Regardless, my other points stand independent of > that. > > > > *From:* Joel E. Denny <jdenny.ornl at gmail.com> > *Sent:* Thursday, June 18, 2020 12:43 PM > *To:* Chris Tetreault <ctetreau at quicinc.com> > *Cc:* Sjoerd Meijer <Sjoerd.Meijer at arm.com>; llvm-dev at lists.llvm.org > *Subject:* [EXT] Re: [llvm-dev] FileCheck > > > > On Thu, Jun 18, 2020 at 3:37 PM Chris Tetreault <ctetreau at quicinc.com> > wrote: > > We’re talking about verbose output right? Verbose isn’t the default. > > > > I'm fairly certain the issue in this thread is just the verbosity of > -dump-input=fail. Yes, -vv makes it even more verbose by annotating input > lines with good matches, etc., but that's not part of the "new behaviour" > Sjoerd meant, I believe. > > > > Joel > > > > > > *From:* Joel E. Denny <jdenny.ornl at gmail.com> > *Sent:* Thursday, June 18, 2020 10:54 AM > *To:* Chris Tetreault <ctetreau at quicinc.com> > *Cc:* Sjoerd Meijer <Sjoerd.Meijer at arm.com>; llvm-dev at lists.llvm.org > *Subject:* [EXT] Re: [llvm-dev] FileCheck > > > > Hi Chris, > > > > On Thu, Jun 18, 2020 at 1:37 PM Chris Tetreault via llvm-dev < > llvm-dev at lists.llvm.org> wrote: > > The thing I use normally only shows the first N lines by default (I don’t > know off hand what N is). Honestly, I don’t feel very strongly about the > specific order, but it’s not useful when somebody proposes something on the > list, and nobody voices any dissent (choosing instead to silently oppose > the change). My requests would be: > > > > 1. The order should be customizable via command line. > 2. By default, it should not dump things to multiple locations. If I > ask for verbose output, I want to get blasted with all the stuff. > 3. The most important thing for me personally is to see the input to > filecheck (I realize that this is in conflict with my earlier point. It’s > early and I hadn’t had my coffee 😊 ). When I get a failure I want to > be able to reproduce it in an IDE to use a debugger. Any change should not > make this use case harder. > > > > Personally, I do not find the argument that the defaults should be setup > to be best for newcomers to be very compelling; we are talking about > changing the behavior of a non-default option after all. > > > > What do you mean by a "non-default option"? The default of > -dump-input=never was recently changed to -dump-input=fail. > > > > Joel > > > > If just a bare filecheck invocation doesn’t tell a newcomer what they need > to know, then they have to do filecheck -help or google the documentation > anyways. At that point, they are going to customize it however they want. I > assume anybody using filecheck to debug an issue is tech savvy enough to be > able to configure the options, given reasonable documentation. > > > > Thanks, > > Christopher Tetreault > > > > *From:* Sjoerd Meijer <Sjoerd.Meijer at arm.com> > *Sent:* Thursday, June 18, 2020 9:45 AM > *To:* Chris Tetreault <ctetreau at quicinc.com> > *Cc:* llvm-dev at lists.llvm.org > *Subject:* [EXT] Re: [llvm-dev] FileCheck > > > > > > I would guess that in a CI system the order doesn't matter much because > you look at a webpage? I looked at some build bots today/yesterday that now > also show this, and yeah, it's fine either way, I was guessing. > > > > My primary use-case is usage in a terminal, and displaying the errors > first followed by all input makes this pretty unusable. > ------------------------------ > > *From:* Chris Tetreault <ctetreau at quicinc.com> > *Sent:* 18 June 2020 17:34 > *To:* Sjoerd Meijer <Sjoerd.Meijer at arm.com> > *Cc:* llvm-dev at lists.llvm.org <llvm-dev at lists.llvm.org> > *Subject:* RE: [llvm-dev] FileCheck > > > > For anybody viewing these failures through some sort of CI system, showing > the error first then the input file is more useful for the same reasons you > mentioned. Personally, I rarely run filecheck by hand from the command > prompt, so your change would make my life worse. Granted, I’m just one > person. > > > > The point I’m trying to make is that I don’t think it’s clear-cut which > order is better, so maybe we shouldn’t change it. I think it might be fine > to add an option to swap the order, but I’d be very sad if it started > dumping to some random file by default. > > > > Thanks, > > Christopher Tetreault > > > > *From:* llvm-dev <llvm-dev-bounces at lists.llvm.org> *On Behalf Of *Sjoerd > Meijer via llvm-dev > *Sent:* Thursday, June 18, 2020 9:16 AM > *To:* llvm-dev at lists.llvm.org > *Subject:* [EXT] [llvm-dev] FileCheck > > > > Hello, > > > > I am not sold on FileCheck's new behaviour. For failing tests in verbose > mode, it first dump the actual error messages, followed by the annotated > input file to FileCheck. The result is I can't immediately see error > messages if the input is more than just a few lines long, so I have to > scroll all the way up to see the errors, then down again, etc. > > > > I do see some advantages of dumping the input to FileCheck, but an > improvement for me would be: > > - to dump the input first, then followed by the error message, so that > I can the errors first, and then decide to scroll up if I am interested to > do so. > - dump it to a separate file (controlled with an option). > > I am interested in changing the behaviour, because I think I find setting > environment varibale "FILECHECK_OPTS="--dump-input never"" inconvenient. > > > > My 2 pennies. > > Sjoerd. > > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev > > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20200619/6440fcd2/attachment.html>