similar to: [RFC] [lit] Report line numbers for failed RUN commands

Seems useful, but the current output is quite verbose. Wouldn’t just RUN:1; ls be sufficient in place of 'RUN: at line 1’; ls ? I’m also curious about the implementation, I recall there was a discussion about forcing all tests to use internal shell? In that case a trick with a no-op “:” command would not be necessary. George > On Apr 2, 2018, at 1:25 PM, Joel E. Denny via llvm-dev

Dear all, Recently I've considering using LIT for my benchmark testing framework, and the only reference for LLVM-LIT is the man page and some READMEs. I don't find any documentations on config, which seems to be quite important to the tool. If I use lit outside LLVM source tree and use on my own test files, LIT marks them as 'unresolved'. So are there any documentations I can

On 10/12/2013 18:03, Jim Grosbach wrote: >> That causes dissonance between what the compiler sees and what lit.py >> sees for no particularly good reason. One of the nice properties of >> lit tests is that they're also valid compiler inputs, so trailing >> slash is a bit unfortunate. >> > > How does the backslash break this in any way? The backslash is

On Dec 10, 2013, at 11:26 AM, Alp Toker <alp at nuanti.com> wrote: > > On 10/12/2013 18:03, Jim Grosbach wrote: >>> That causes dissonance between what the compiler sees and what lit.py sees for no particularly good reason. One of the nice properties of lit tests is that they're also valid compiler inputs, so trailing slash is a bit unfortunate. >>> >>

Hi, Lit utility has valgrind tests already implemented, that are called using --vg parameter. What does "LLVM style valgrind" mean in lit's TODO? What is LLVM style? Thanks, Sergey

On 10/12/2013 19:47, Jim Grosbach wrote: > > On Dec 10, 2013, at 11:26 AM, Alp Toker <alp at nuanti.com > <mailto:alp at nuanti.com>> wrote: > >> >> On 10/12/2013 18:03, Jim Grosbach wrote: >>>> That causes dissonance between what the compiler sees and what >>>> lit.py sees for no particularly good reason. One of the nice

On 10/12/2013 04:20, Sean Silva wrote: > The classic way to do this sort of checking is by hacking into the > tool that actually interprets it (i.e. lit in this case). Considering > that lit is Python, it should be pretty easy to insert an ad-hoc regex > check (or even something substantially more sophisticated). E.g. > insert code into TestRunner.py's

Hi, Me and my supervisor are interested in improving llvm's lit a little bit. Currently, there are few things, that seem to be not very convenient about the tool: * Having several RUN commands in a failing test, it's impossible to figure out, which of them actually caused the test to fail. We can output the number of the failing RUN line. * It would be nice to

Hi, A feature I've wanted in lit for a while is a having a timeout per test. Attached are patches that implement this idea. I'm e-mailing llvm-dev rather than llvm-commits because I want to gather more feedback on my initial implementation and hopefully some answers to some unresolved issues with my implementation. Currently in lit you can set a global timeout for all of the tests but

On Dec 8, 2013, at 4:24 AM, Alp Toker <alp at nuanti.com> wrote: > > On 08/12/2013 11:22, Chandler Carruth wrote: >> On Sun, Dec 8, 2013 at 2:11 AM, Alp Toker <alp at nuanti.com> wrote: >> I'd like to propose deprecating and shortly thereafter removing the lit test runner feature that concatenates RUN lines ending in a trailing \ >> >> I'm

Hi, > Cool, I hope this succeeds. I tried implementing per-test timeouts before, and couldn't get it to work in all cases. The review eventually fizzled out, and I abandoned it. > > Here's that old review: http://reviews.llvm.org/D6584 Perhaps you can cannibalize testcases from it. Thanks for that. I'll take a look. > >> >> I'm e-mailing llvm-dev rather

On 08/12/2013 13:12, Chandler Carruth wrote: > >> * Removing trailing \ will introduce the neat property that >> one RUN line corresponds precisely to one command that's >> executed. This is good for humans and will enable >> simplifications in the test runner. >> >> FWIW, I've never really had a

So I manage to get clang's colored output through ninja simply by force (CMAKE_CXX_FLAGS=-fcolor-diagnostics), which isn't ideal (if I were to pipe ninja's output to a file it'd still have color escapes, etc) but it works. But I haven't found a similar solution for FileCheck & I'm wondering has anyone already solved this problem for themselves - if so, how? if not,

Ok, that makes sense. I don't see this as a good enough argument to remove backslash support though. For problems like the clang_cc1 substitution mistakes, it would be much better to just improve the substitution support so that those cause immediate test failures. I would definitely support a move to make lit's substitution machinery more strict. For problems like your first search, I

On 08/12/2013 11:22, Chandler Carruth wrote: > On Sun, Dec 8, 2013 at 2:11 AM, Alp Toker <alp at nuanti.com > <mailto:alp at nuanti.com>> wrote: > > I'd like to propose deprecating and shortly thereafter removing > the lit test runner feature that concatenates RUN lines ending in > a trailing \ > > > I'm really opposed to this. Especially

Looking, thanks for the report. On Fri, Sep 22, 2017 at 11:22 AM Vedant Kumar <vsk at apple.com> wrote: > As of r313998, this workflow no longer works: > > cd <build-dir> > ./bin/llvm-lit <src>/llvm/tools/clang/test/CoverageMapping > > I get: > > llvm-lit: /Users/vk/src/llvm.org-coverage-braces/llvm/tools/clang/test/ > lit.cfg.py:97: note: using

As of r313998, this workflow no longer works: cd <build-dir> ./bin/llvm-lit <src>/llvm/tools/clang/test/CoverageMapping I get: llvm-lit: /Users/vk/src/llvm.org-coverage-braces/llvm/tools/clang/test/lit.cfg.py:97: note: using clang: '/Volumes/Builds/llvm.org-coverage-braces-RA/bin/clang' llvm-lit:

Ah, the problem goes away once I build clang-func-mapping. I stripped some stuff out, but here's pretty much what clang/test/lit.cfg.py says my PATH is: ** PATH **: /Volumes/Builds/llvm.org-coverage-braces-RA/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin I wonder how this ever worked before, when I didn't have clang-func-mapping built. Anyway, thanks for your help! vedant >

On 08/12/2013 16:43, Daniel Dunbar wrote: > > With one-to-one mapping, it becomes possible to use simple tools > like grep to validate common mistakes like %clang / %clang_cc1 > mixups, a missing -o flag and so on. > > Right now there's no obvious way to do those checks and we've > ended up without an easy way to lint for broken tests as a result.

llvm-lit is already put into <output-tree>/bin by CMake. There is a CMake target called `llvm-lit` that will generate the <output-dir>/bin/llvm-lit script. If you built only a few specific llvm targets such as FileCheck, etc then you may need to run this lit target manually. If you just run "ninja" for example, or "ninja check-llvm", it's automatically