similar to: [LLVMdev] Getting FileCheck's colored output through lit (& possibly ninja)

I'm just wondering if anyone's using Ninja (which buffers output from subcommands, such as lit/FileCheck/etc) & has thought about/found a way to force FileCheck to use colour in spite of Ninja's buffering. Thanks, - David

Nope. An easy solution (both here and for clang) is for ninja to set an environment variable (`NINJA_ISATTY`?), and add some magic to LLVM's "isatty" to return true if it sees it. (for matters of correctness, it would actually be best if ninja sets the environment variable only when *it's* stdout/stderr isatty (also taking into account this environment variable), so that

I was curious how widespread bad usage of FileCheck prefixes is in the LLVM/Clang test suite. Running the attached script yields over 200 tests (~1.5%) which are essentially broken because they contain checks that are never checked. Most widespread is the usage of the "CHECK" prefix while all FileCheck invocations have an explicit check prefix. Then there are quite a few prefixes that

On further thought, I assume this issue actually hits everyone - it's not just Ninja getting in the way, but lit as well. (which also means we'd probably need to fix both Ninja and lit to resolve this issue - but everyone should at least be a little invested in fixing lit). On Sun, Jan 6, 2013 at 12:18 AM, Sean Silva <silvas at purdue.edu> wrote: > Nope. An easy solution (both

We currently don't use pipefail when running test under make check. This has the undesirable property that it is really easy for tests to bitrot. For example, something like llc %s | FileCheck %s will still pass if llc crashes after printing what FileCheck was looking for. It is also easy to break the tests when refactoring. I have fixed tests that were doing %clang_cc1 -a-driver-options

On 23.01.2013 00:45, Eli Bendersky wrote: > I do mean semantic extraction which provides a representation that's > meaningful to a user and hence can be effectively compared in a test. > But really, I gave up arguing on this topic a few messages (and heated > IRC discussions) ago. RFC retracted. I actually hit the same predicament for COFF. The few tests that are there usually use

On 10/12/2013 18:03, Jim Grosbach wrote: >> That causes dissonance between what the compiler sees and what lit.py >> sees for no particularly good reason. One of the nice properties of >> lit tests is that they're also valid compiler inputs, so trailing >> slash is a bit unfortunate. >> > > How does the backslash break this in any way? The backslash is

Hello, FileCheck allows us to define match variables and use them on subsequent lines. This is quite useful, but could be even more useful if it was possible to use the variable later on the same line it matched. For example, I would want to write this: ; CHECK: bic [[REG:r[0-9]+]], [[REG]], #3 But I currently can't because [[REG]] will only match a REG variable defined on a _previous_

If I understand correctly, the desire is to change the current behavior in a kind of subtle way. Is there some way you could instrument trunk's FileCheck to die if the old behavior is encountered, and use that to definitively find all tests which rely on the current behavior and migrate them (possibly to an interim solution) in preparation for the change in semantics? How much is the old

On 2019-01-24 20:17, David Greene via llvm-dev wrote: > alexp via llvm-dev <llvm-dev at lists.llvm.org> writes: > >> I have a project which uses llvm, but is (at least not yet) a >> contribution / not in the source tree of llvm. >> Is it a good idea (e.g. instead of using boost test framework) to use >> the llvm testsuite related tools in this case? > >

llvm-lit is already put into <output-tree>/bin by CMake. There is a CMake target called `llvm-lit` that will generate the <output-dir>/bin/llvm-lit script. If you built only a few specific llvm targets such as FileCheck, etc then you may need to run this lit target manually. If you just run "ninja" for example, or "ninja check-llvm", it's automatically

Hi all, quite a few tests use the pattern "2>&1 | FileCheck %s". AFAIK how stdout and stderr are merged into a single character stream is undefined and depends e.g. on whether stdout is buffered. I think we are often saved by the fact that standard output is written only at the end of the program and stderr is unbuffered, i.e. always written before stdout. A lot of tests disable

Hi All, FileCheck is a great tool for writing tests. However, it can be also a great tool for hiding the error cause from you, especially if a very common idiom RUN: ./myprog | FileCheck %s is used. If you need to know the output of “myprog” to figure out why your test is failing — then well, good luck, you would need to manually copy’n paste that line, run it, and then note the output.

When writing MachineFunction unit tests, I find it quite tedious to verify correctness in C++. I would like to use FileCheck in UnitTests because FileCheck is extremely convenient/robust to verify correctness. In order to do so, I moved most of FileCheck’s implementation into a header (Support/FileCheck.h) and updated FileCheck.cpp to use this. I ran into this while writing some target agnostic

Hello all, I am writing test cases which are dedicated to be executed by llvm-lit. Most of my test cases have the same logic but different types. For example: // RUN: %clang_cc1 -fsyntax-only func (int a) { a = 3; } // RUN: %clang_cc1 -fsyntax-only func (char a) { a = 3; } // RUN: %clang_cc1 -fsyntax-only func (unsigned a) { a = 3; } Now I put them in three different test cases but it

> I don't know how you proceed to debug FileCheck failures, but for me most of the time I'll have to figure out which "RUN" line fail and try to execute it manually and then remove the FileCheck pipe to get the raw input and then painfully tried to match the FileCheck error to the actual input. Yeah, not very different from what you described here. If I 'm creating or

Sorry if I wasn't clear about my use case. In my daily dev work, I do many local "ninja check"s, or "llvm-lit" on a subdirectory as a quick(er) smoke test if I am making changes in that area (e.g. "llvm-lit ../llvm/test/CodeGen"). Nothing wrong here, as indeed nothing changed here. But in case of a test failure, I want to run just that test: bin/llvm-lit

On Sep 10, 2016, at 3:33 AM, Steve Canon <scanon at apple.com> wrote: >>> >>> Pretty much. In particular, imagine a user trying to debug an unexpected floating point result caused by conversion of a*b + c into fma(a, b, c). >> >> I think that’s unavoidable, because of the way the optimization levels work. Even fma contraction is on by default (something I’d

Hi, I was reading the FileCheck tool documentation [1] this morning and a two parts of the CHECK-DAG: directive don't quite seem right to me. 1. ``` With captured variables, CHECK-DAG: is able to match valid topological orderings of a DAG with edges from the definition of a variable to its use. ``` This is a little confusing. It is not explained what "captured variables" are. My

On 08/12/2013 16:43, Daniel Dunbar wrote: > > With one-to-one mapping, it becomes possible to use simple tools > like grep to validate common mistakes like %clang / %clang_cc1 > mixups, a missing -o flag and so on. > > Right now there's no obvious way to do those checks and we've > ended up without an easy way to lint for broken tests as a result.