Pavel Labath via llvm-dev
2020-Jul-14 06:58 UTC
[llvm-dev] Multiple documents in one test file
On 14/07/2020 03:27, David Blaikie via llvm-dev wrote:> (+Richard - it's handy to include folks from previous discussions > explicitly so everyone can more easily keep track of the conversation) > > On Mon, Jul 13, 2020 at 6:17 PM Fangrui Song via llvm-dev > <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote: > > Sometimes it is convenient if we can specify multiple independent tests > in one file. To give an example, let's discuss > test/MC/ELF/debug-md5.s and > test/MC/ELF/debug-md5-err.s (.file directive in the assembler). > > a) An invalid .file makes the whole file invalid. Because errors > lead to a > non-zero exit code, We have to use `RUN: not llvm-mc %s` for the > whole file. > This often lead to users placing good (`RUN: llvm-mc %s`) and bad > tests (`RUN: > not llvm-mc %s`) separately. For some features, having both good and > bad tests > in one file may improve readability. > b) .debug_line is a global resource. Whenever we add a (valid) .file, we > contribute an entry to the global resource. If we want to test some > characteristics when include_directories[0] is A, and other > characteristics > when include_directories[0] is B, we have to use another test file. > > The arguments apply to many other types of tests (opt on .ll, llc on > .ll and .mir, clang on .c, yaml2obj on .yaml, etc). > > I have a patch teaching llvm-mc about an option to split input: > https://reviews.llvm.org/D83725 > (30+ lines) > > In a comment, Richard Smith mentioned whether we can add a separate > extractor utility: > > ``` > # RUN: extract bb %s | llvm-mc - 2>&1 | FileCheck %s --check-prefix=BB > > or > > # RUN: extract bb %s -o %t.bb <http://t.bb> > # RUN: llvm-mc %t.bb <http://t.bb> 2>&1 | FileCheck %t.bb <http://t.bb> > ``` > > > Could make "extract" work a bit like "tee" so it can still be one line: > > # RUN: extract bb %s -o %t.bb <http://t.bb> | llvm-mc - 2>&1 | FileCheck > %t.bb <http://t.bb> > > (could even make it a bit shorter for convenience - 'ex' or something) > > > The advantage is its versatility. The downside is somewhat verbose > syntax. > > > Some questoms: > > 1. What do people think of the two approaches? An in-utility option > vs a standalone utility. > 2. For llvm-mc, if we go with an option, is there a better name than > --doc-id? David Blaikie proposed --asm-id > (This is my personal preference, trading 30+ lines in a utility > for simpler syntax)FWIW, the way I've done this in llvm-mc so far is via a combination of "--defsym CASE<N>" command line argument and ".ifdef" asm directives. This has the advantage that individual "documents" don't need to be fully standalone (though they can be), so you can put the common parts of the tests into an unconditionally compiled block. That said, I was using this technique for constructing test cases for other tools via llvm-mc. Things might get a bit awkward if you try to test .ifdef processing itself this way... pl
Robinson, Paul via llvm-dev
2020-Jul-14 13:12 UTC
[llvm-dev] Multiple documents in one test file
> -----Original Message----- > From: llvm-dev <llvm-dev-bounces at lists.llvm.org> On Behalf Of Pavel Labath > via llvm-dev > Sent: Tuesday, July 14, 2020 2:58 AM > To: David Blaikie <dblaikie at gmail.com>; Fangrui Song <maskray at google.com>; > Richard Smith <richard at metafoo.co.uk> > Cc: llvm-dev <llvm-dev at lists.llvm.org> > Subject: Re: [llvm-dev] Multiple documents in one test file > > On 14/07/2020 03:27, David Blaikie via llvm-dev wrote: > > (+Richard - it's handy to include folks from previous discussions > > explicitly so everyone can more easily keep track of the conversation) > > > > On Mon, Jul 13, 2020 at 6:17 PM Fangrui Song via llvm-dev > > <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote: > > > > Sometimes it is convenient if we can specify multiple independent > tests > > in one file. To give an example, let's discuss > > test/MC/ELF/debug-md5.s and > > test/MC/ELF/debug-md5-err.s (.file directive in the assembler). > > > > a) An invalid .file makes the whole file invalid. Because errors > > lead to a > > non-zero exit code, We have to use `RUN: not llvm-mc %s` for the > > whole file. > > This often lead to users placing good (`RUN: llvm-mc %s`) and bad > > tests (`RUN: > > not llvm-mc %s`) separately. For some features, having both good and > > bad tests > > in one file may improve readability. > > b) .debug_line is a global resource. Whenever we add a (valid) > .file, we > > contribute an entry to the global resource. If we want to test some > > characteristics when include_directories[0] is A, and other > > characteristics > > when include_directories[0] is B, we have to use another test file. > > > > The arguments apply to many other types of tests (opt on .ll, llc on > > .ll and .mir, clang on .c, yaml2obj on .yaml, etc). > > > > I have a patch teaching llvm-mc about an option to split input: > > https://reviews.llvm.org/D83725 > > (30+ lines) > > > > In a comment, Richard Smith mentioned whether we can add a separate > > extractor utility: > > > > ``` > > # RUN: extract bb %s | llvm-mc - 2>&1 | FileCheck %s --check- > prefix=BB > > > > or > > > > # RUN: extract bb %s -o %t.bb <http://t.bb> > > # RUN: llvm-mc %t.bb <http://t.bb> 2>&1 | FileCheck %t.bb > <http://t.bb> > > ``` > > > > > > Could make "extract" work a bit like "tee" so it can still be one line: > > > > # RUN: extract bb %s -o %t.bb <http://t.bb> | llvm-mc - 2>&1 | FileCheck > > %t.bb <http://t.bb> > > > > (could even make it a bit shorter for convenience - 'ex' or something) > > > > > > The advantage is its versatility. The downside is somewhat verbose > > syntax. > > > > > > Some questoms: > > > > 1. What do people think of the two approaches? An in-utility option > > vs a standalone utility. > > 2. For llvm-mc, if we go with an option, is there a better name than > > --doc-id? David Blaikie proposed --asm-id > > (This is my personal preference, trading 30+ lines in a utility > > for simpler syntax) > > FWIW, the way I've done this in llvm-mc so far is via a combination of > "--defsym CASE<N>" command line argument and ".ifdef" asm directives. > This has the advantage that individual "documents" don't need to be > fully standalone (though they can be), so you can put the common parts > of the tests into an unconditionally compiled block. > > That said, I was using this technique for constructing test cases for > other tools via llvm-mc. Things might get a bit awkward if you try to > test .ifdef processing itself this way... > > pl+1 for --defsym. I'm sure I've written pairs of tests that could have been done more simply this way. Regarding maskray's original post, which is about valid/invalid .file directives in the same .s file, --defsym and .ifdef work perfectly: .ifdef CASE1 .file 1 "./case1.c" .endif .ifdef CASE2 .file 1 "./case2.c" .endif nop llvm-mc --defsym=CASE1=1 will emit a .debug_line with "case1.c" and --defsym=CASE2=1 will emit it with "case2.c". So this would be ideal for verifying the MD5 cases (good and bad) in one test file. --paulr
Fangrui Song via llvm-dev
2020-Jul-14 17:01 UTC
[llvm-dev] Multiple documents in one test file
On 2020-07-14, Robinson, Paul wrote:> > >> -----Original Message----- >> From: llvm-dev <llvm-dev-bounces at lists.llvm.org> On Behalf Of Pavel Labath >> via llvm-dev >> Sent: Tuesday, July 14, 2020 2:58 AM >> To: David Blaikie <dblaikie at gmail.com>; Fangrui Song <maskray at google.com>; >> Richard Smith <richard at metafoo.co.uk> >> Cc: llvm-dev <llvm-dev at lists.llvm.org> >> Subject: Re: [llvm-dev] Multiple documents in one test file >> >> On 14/07/2020 03:27, David Blaikie via llvm-dev wrote: >> > (+Richard - it's handy to include folks from previous discussions >> > explicitly so everyone can more easily keep track of the conversation) >> > >> > On Mon, Jul 13, 2020 at 6:17 PM Fangrui Song via llvm-dev >> > <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote: >> > >> > Sometimes it is convenient if we can specify multiple independent >> tests >> > in one file. To give an example, let's discuss >> > test/MC/ELF/debug-md5.s and >> > test/MC/ELF/debug-md5-err.s (.file directive in the assembler). >> > >> > a) An invalid .file makes the whole file invalid. Because errors >> > lead to a >> > non-zero exit code, We have to use `RUN: not llvm-mc %s` for the >> > whole file. >> > This often lead to users placing good (`RUN: llvm-mc %s`) and bad >> > tests (`RUN: >> > not llvm-mc %s`) separately. For some features, having both good and >> > bad tests >> > in one file may improve readability. >> > b) .debug_line is a global resource. Whenever we add a (valid) >> .file, we >> > contribute an entry to the global resource. If we want to test some >> > characteristics when include_directories[0] is A, and other >> > characteristics >> > when include_directories[0] is B, we have to use another test file. >> > >> > The arguments apply to many other types of tests (opt on .ll, llc on >> > .ll and .mir, clang on .c, yaml2obj on .yaml, etc). >> > >> > I have a patch teaching llvm-mc about an option to split input: >> > https://reviews.llvm.org/D83725 >> > (30+ lines) >> > >> > In a comment, Richard Smith mentioned whether we can add a separate >> > extractor utility: >> > >> > ``` >> > # RUN: extract bb %s | llvm-mc - 2>&1 | FileCheck %s --check- >> prefix=BB >> > >> > or >> > >> > # RUN: extract bb %s -o %t.bb <http://t.bb> >> > # RUN: llvm-mc %t.bb <http://t.bb> 2>&1 | FileCheck %t.bb >> <http://t.bb> >> > ``` >> > >> > >> > Could make "extract" work a bit like "tee" so it can still be one line: >> > >> > # RUN: extract bb %s -o %t.bb <http://t.bb> | llvm-mc - 2>&1 | FileCheck >> > %t.bb <http://t.bb> >> > >> > (could even make it a bit shorter for convenience - 'ex' or something) >> > >> > >> > The advantage is its versatility. The downside is somewhat verbose >> > syntax. >> > >> > >> > Some questoms: >> > >> > 1. What do people think of the two approaches? An in-utility option >> > vs a standalone utility. >> > 2. For llvm-mc, if we go with an option, is there a better name than >> > --doc-id? David Blaikie proposed --asm-id >> > (This is my personal preference, trading 30+ lines in a utility >> > for simpler syntax) >> >> FWIW, the way I've done this in llvm-mc so far is via a combination of >> "--defsym CASE<N>" command line argument and ".ifdef" asm directives. >> This has the advantage that individual "documents" don't need to be >> fully standalone (though they can be), so you can put the common parts >> of the tests into an unconditionally compiled block. >> >> That said, I was using this technique for constructing test cases for >> other tools via llvm-mc. Things might get a bit awkward if you try to >> test .ifdef processing itself this way... >> >> pl > >+1 for --defsym. I'm sure I've written pairs of tests that could have been >done more simply this way. > >Regarding maskray's original post, which is about valid/invalid .file >directives in the same .s file, --defsym and .ifdef work perfectly: > >.ifdef CASE1 >.file 1 "./case1.c" >.endif >.ifdef CASE2 >.file 1 "./case2.c" >.endif >nop > >llvm-mc --defsym=CASE1=1 will emit a .debug_line with "case1.c" >and --defsym=CASE2=1 will emit it with "case2.c". So this would be >ideal for verifying the MD5 cases (good and bad) in one test file. >--paulr >My bad memory that I should have considered .ifdef (I just saw it last week https://sourceware.org/pipermail/binutils/2020-July/112193.html ). The syntax appears to be good enoguh. An advantage other than being a built-in feature is that line numbers are retained. A note about the proposed external tool 'extract': we probably should insert '\n' to retain the original line numbers, so that the following can work: ``` #--- aa [[#@LINE+1]]: error: ..... ..... ```