llvm dev - Jul 2020 - Multiple documents in one test file

Sometimes it is convenient if we can specify multiple independent tests
in one file. To give an example, let's discuss test/MC/ELF/debug-md5.s and
test/MC/ELF/debug-md5-err.s (.file directive in the assembler).

a) An invalid .file makes the whole file invalid. Because errors lead to a
non-zero exit code, We have to use `RUN: not llvm-mc %s` for the whole file.
This often lead to users placing good (`RUN: llvm-mc %s`) and bad tests (`RUN:
not llvm-mc %s`) separately. For some features, having both good and bad tests
in one file may improve readability.
b) .debug_line is a global resource. Whenever we add a (valid) .file, we
contribute an entry to the global resource. If we want to test some
characteristics when include_directories[0] is A, and other characteristics
when include_directories[0] is B, we have to use another test file.

The arguments apply to many other types of tests (opt on .ll, llc on .ll and
.mir, clang on .c, yaml2obj on .yaml, etc).

I have a patch teaching llvm-mc about an option to split input:
https://reviews.llvm.org/D83725
(30+ lines)

In a comment, Richard Smith mentioned whether we can add a separate extractor
utility:

```
# RUN: extract bb %s | llvm-mc - 2>&1 | FileCheck %s --check-prefix=BB

or

# RUN: extract bb %s -o %t.bb
# RUN: llvm-mc %t.bb 2>&1 | FileCheck %t.bb
```

The advantage is its versatility. The downside is somewhat verbose syntax.


Some questoms:

1. What do people think of the two approaches? An in-utility option vs a
standalone utility.
2. For llvm-mc, if we go with an option, is there a better name than --doc-id?
David Blaikie proposed --asm-id
   (This is my personal preference, trading 30+ lines in a utility for simpler
syntax)
3. If we add a standalone utility, how shall we name it? (Note that llvm-extract
exists, but people can probably distinguish 'extract' from llvm-extract

(+Richard - it's handy to include folks from previous discussions
explicitly so everyone can more easily keep track of the conversation)

On Mon, Jul 13, 2020 at 6:17 PM Fangrui Song via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> Sometimes it is convenient if we can specify multiple independent tests
> in one file. To give an example, let's discuss test/MC/ELF/debug-md5.s
and
> test/MC/ELF/debug-md5-err.s (.file directive in the assembler).
>
> a) An invalid .file makes the whole file invalid. Because errors lead to a
> non-zero exit code, We have to use `RUN: not llvm-mc %s` for the whole
> file.
> This often lead to users placing good (`RUN: llvm-mc %s`) and bad tests
> (`RUN:
> not llvm-mc %s`) separately. For some features, having both good and bad
> tests
> in one file may improve readability.
> b) .debug_line is a global resource. Whenever we add a (valid) .file, we
> contribute an entry to the global resource. If we want to test some
> characteristics when include_directories[0] is A, and other characteristics
> when include_directories[0] is B, we have to use another test file.
>
> The arguments apply to many other types of tests (opt on .ll, llc on .ll
> and .mir, clang on .c, yaml2obj on .yaml, etc).
>
> I have a patch teaching llvm-mc about an option to split input:
> https://reviews.llvm.org/D83725
> (30+ lines)
>
> In a comment, Richard Smith mentioned whether we can add a separate
> extractor utility:
>
> ```
> # RUN: extract bb %s | llvm-mc - 2>&1 | FileCheck %s
--check-prefix=BB
>
> or
>
> # RUN: extract bb %s -o %t.bb
> # RUN: llvm-mc %t.bb 2>&1 | FileCheck %t.bb
> ```
>
Could make "extract" work a bit like "tee" so it can still
be one line:

# RUN: extract bb %s -o %t.bb | llvm-mc - 2>&1 | FileCheck %t.bb

(could even make it a bit shorter for convenience - 'ex' or something)

> The advantage is its versatility. The downside is somewhat verbose syntax.
>
>
> Some questoms:
>
> 1. What do people think of the two approaches? An in-utility option vs a
> standalone utility.
> 2. For llvm-mc, if we go with an option, is there a better name than
> --doc-id? David Blaikie proposed --asm-id
>    (This is my personal preference, trading 30+ lines in a utility for
> simpler syntax)
> 3. If we add a standalone utility, how shall we name it? (Note that
> llvm-extract exists, but people can probably distinguish 'extract'
from
> llvm-extract
>
I think some of the truly internal utilities are named without the llvm
prefix - isn't stuff like "not" actually implemented as a local
tool? hmm,
guess not, maybe that's a built-in inside lit.
Only risk I can think of with the name is the auto-name expansion of lit
replacing any token 'ex' with the full path to the tool, so you might
have
to be careful about not using that character sequence as a standalone
argument on a RUN line - but that seems OK.

> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20200713/f6e7dc12/attachment.html>

`extract` +1 for consistency.

________________________________________
From: llvm-dev <llvm-dev-bounces at lists.llvm.org> on behalf of Fangrui
Song via llvm-dev <llvm-dev at lists.llvm.org>
Sent: Monday, July 13, 2020 6:17 PM
To: llvm-dev
Subject: [llvm-dev] Multiple documents in one test file

Sometimes it is convenient if we can specify multiple independent tests
in one file. To give an example, let's discuss test/MC/ELF/debug-md5.s and
test/MC/ELF/debug-md5-err.s (.file directive in the assembler).

a) An invalid .file makes the whole file invalid. Because errors lead to a
non-zero exit code, We have to use `RUN: not llvm-mc %s` for the whole file.
This often lead to users placing good (`RUN: llvm-mc %s`) and bad tests (`RUN:
not llvm-mc %s`) separately. For some features, having both good and bad tests
in one file may improve readability.
b) .debug_line is a global resource. Whenever we add a (valid) .file, we
contribute an entry to the global resource. If we want to test some
characteristics when include_directories[0] is A, and other characteristics
when include_directories[0] is B, we have to use another test file.

The arguments apply to many other types of tests (opt on .ll, llc on .ll and
.mir, clang on .c, yaml2obj on .yaml, etc).

I have a patch teaching llvm-mc about an option to split input:
https://reviews.llvm.org/D83725
(30+ lines)

In a comment, Richard Smith mentioned whether we can add a separate extractor
utility:

```
# RUN: extract bb %s | llvm-mc - 2>&1 | FileCheck %s --check-prefix=BB

or

# RUN: extract bb %s -o %t.bb
# RUN: llvm-mc %t.bb 2>&1 | FileCheck %t.bb
```

The advantage is its versatility. The downside is somewhat verbose syntax.


Some questoms:

1. What do people think of the two approaches? An in-utility option vs a
standalone utility.
2. For llvm-mc, if we go with an option, is there a better name than --doc-id?
David Blaikie proposed --asm-id
   (This is my personal preference, trading 30+ lines in a utility for simpler
syntax)
3. If we add a standalone utility, how shall we name it? (Note that llvm-extract
exists, but people can probably distinguish 'extract' from llvm-extract
_______________________________________________
LLVM Developers mailing list
llvm-dev at lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

We have a similar option (-split-input-file) in `mlir-opt`:
https://github.com/llvm/llvm-project/blob/master/mlir/test/Dialect/Affine/invalid.mlir

With a single `RUN:` lit invocation the tool itself will loop over all the
split sections in the file. This is convenient to test error cases where
the tool would abort at the first error otherwise. I don't think we can
easily achieve this with a single pipe and a separate `extract` command
though?

-- 
Mehdi



On Mon, Jul 13, 2020 at 6:43 PM Chen, Yuanfang via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> `extract` +1 for consistency.
>
> ________________________________________
> From: llvm-dev <llvm-dev-bounces at lists.llvm.org> on behalf of
Fangrui
> Song via llvm-dev <llvm-dev at lists.llvm.org>
> Sent: Monday, July 13, 2020 6:17 PM
> To: llvm-dev
> Subject: [llvm-dev] Multiple documents in one test file
>
> Sometimes it is convenient if we can specify multiple independent tests
> in one file. To give an example, let's discuss test/MC/ELF/debug-md5.s
and
> test/MC/ELF/debug-md5-err.s (.file directive in the assembler).
>
> a) An invalid .file makes the whole file invalid. Because errors lead to a
> non-zero exit code, We have to use `RUN: not llvm-mc %s` for the whole
> file.
> This often lead to users placing good (`RUN: llvm-mc %s`) and bad tests
> (`RUN:
> not llvm-mc %s`) separately. For some features, having both good and bad
> tests
> in one file may improve readability.
> b) .debug_line is a global resource. Whenever we add a (valid) .file, we
> contribute an entry to the global resource. If we want to test some
> characteristics when include_directories[0] is A, and other characteristics
> when include_directories[0] is B, we have to use another test file.
>
> The arguments apply to many other types of tests (opt on .ll, llc on .ll
> and .mir, clang on .c, yaml2obj on .yaml, etc).
>
> I have a patch teaching llvm-mc about an option to split input:
> https://reviews.llvm.org/D83725
> (30+ lines)
>
> In a comment, Richard Smith mentioned whether we can add a separate
> extractor utility:
>
> ```
> # RUN: extract bb %s | llvm-mc - 2>&1 | FileCheck %s
--check-prefix=BB
>
> or
>
> # RUN: extract bb %s -o %t.bb
> # RUN: llvm-mc %t.bb 2>&1 | FileCheck %t.bb
> ```
>
> The advantage is its versatility. The downside is somewhat verbose syntax.
>
>
> Some questoms:
>
> 1. What do people think of the two approaches? An in-utility option vs a
> standalone utility.
> 2. For llvm-mc, if we go with an option, is there a better name than
> --doc-id? David Blaikie proposed --asm-id
>    (This is my personal preference, trading 30+ lines in a utility for
> simpler syntax)
> 3. If we add a standalone utility, how shall we name it? (Note that
> llvm-extract exists, but people can probably distinguish 'extract'
from
> llvm-extract
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20200713/c2634bdb/attachment.html>

On 14/07/2020 03:27, David Blaikie via llvm-dev wrote:
> (+Richard - it's handy to include folks from previous discussions
> explicitly so everyone can more easily keep track of the conversation)
> 
> On Mon, Jul 13, 2020 at 6:17 PM Fangrui Song via llvm-dev
> <llvm-dev at lists.llvm.org <mailto:llvm-dev at
lists.llvm.org>> wrote:
> 
>     Sometimes it is convenient if we can specify multiple independent tests
>     in one file. To give an example, let's discuss
>     test/MC/ELF/debug-md5.s and
>     test/MC/ELF/debug-md5-err.s (.file directive in the assembler).
> 
>     a) An invalid .file makes the whole file invalid. Because errors
>     lead to a
>     non-zero exit code, We have to use `RUN: not llvm-mc %s` for the
>     whole file.
>     This often lead to users placing good (`RUN: llvm-mc %s`) and bad
>     tests (`RUN:
>     not llvm-mc %s`) separately. For some features, having both good and
>     bad tests
>     in one file may improve readability.
>     b) .debug_line is a global resource. Whenever we add a (valid) .file,
we
>     contribute an entry to the global resource. If we want to test some
>     characteristics when include_directories[0] is A, and other
>     characteristics
>     when include_directories[0] is B, we have to use another test file.
> 
>     The arguments apply to many other types of tests (opt on .ll, llc on
>     .ll and .mir, clang on .c, yaml2obj on .yaml, etc).
> 
>     I have a patch teaching llvm-mc about an option to split input:
>     https://reviews.llvm.org/D83725
>     (30+ lines)
> 
>     In a comment, Richard Smith mentioned whether we can add a separate
>     extractor utility:
> 
>     ```
>     # RUN: extract bb %s | llvm-mc - 2>&1 | FileCheck %s
--check-prefix=BB
> 
>     or
> 
>     # RUN: extract bb %s -o %t.bb <http://t.bb>
>     # RUN: llvm-mc %t.bb <http://t.bb> 2>&1 | FileCheck %t.bb
<http://t.bb>
>     ```
> 
> 
> Could make "extract" work a bit like "tee" so it can
still be one line:
> 
> # RUN: extract bb %s -o %t.bb <http://t.bb> | llvm-mc - 2>&1 |
FileCheck
> %t.bb <http://t.bb>
> 
> (could even make it a bit shorter for convenience - 'ex' or
something)
>  
> 
>     The advantage is its versatility. The downside is somewhat verbose
>     syntax.
> 
> 
>     Some questoms:
> 
>     1. What do people think of the two approaches? An in-utility option
>     vs a standalone utility.
>     2. For llvm-mc, if we go with an option, is there a better name than
>     --doc-id? David Blaikie proposed --asm-id
>        (This is my personal preference, trading 30+ lines in a utility
>     for simpler syntax)
FWIW, the way I've done this in llvm-mc so far is via a combination of
"--defsym CASE<N>" command line argument and ".ifdef"
asm directives.
This has the advantage that individual "documents" don't need to
be
fully standalone (though they can be), so you can put the common parts
of the tests into an unconditionally compiled block.

That said, I was using this technique for constructing test cases for
other tools via llvm-mc. Things might get a bit awkward if you try to
test .ifdef processing itself this way...

pl

The clang has a `clang-offload-bundle` tool that can combine and
extract multiple IR modules in/from a single file. Implementation is
here:
https://github.com/llvm/llvm-project/blob/master/clang/tools/clang-offload-bundler/ClangOffloadBundler.cpp

Michael


Am Mo., 13. Juli 2020 um 20:17 Uhr schrieb Fangrui Song via llvm-dev
<llvm-dev at lists.llvm.org>:
>
> Sometimes it is convenient if we can specify multiple independent tests
> in one file. To give an example, let's discuss test/MC/ELF/debug-md5.s
and
> test/MC/ELF/debug-md5-err.s (.file directive in the assembler).
>
> a) An invalid .file makes the whole file invalid. Because errors lead to a
> non-zero exit code, We have to use `RUN: not llvm-mc %s` for the whole
file.
> This often lead to users placing good (`RUN: llvm-mc %s`) and bad tests
(`RUN:
> not llvm-mc %s`) separately. For some features, having both good and bad
tests
> in one file may improve readability.
> b) .debug_line is a global resource. Whenever we add a (valid) .file, we
> contribute an entry to the global resource. If we want to test some
> characteristics when include_directories[0] is A, and other characteristics
> when include_directories[0] is B, we have to use another test file.
>
> The arguments apply to many other types of tests (opt on .ll, llc on .ll
and .mir, clang on .c, yaml2obj on .yaml, etc).
>
> I have a patch teaching llvm-mc about an option to split input:
https://reviews.llvm.org/D83725
> (30+ lines)
>
> In a comment, Richard Smith mentioned whether we can add a separate
extractor utility:
>
> ```
> # RUN: extract bb %s | llvm-mc - 2>&1 | FileCheck %s
--check-prefix=BB
>
> or
>
> # RUN: extract bb %s -o %t.bb
> # RUN: llvm-mc %t.bb 2>&1 | FileCheck %t.bb
> ```
>
> The advantage is its versatility. The downside is somewhat verbose syntax.
>
>
> Some questoms:
>
> 1. What do people think of the two approaches? An in-utility option vs a
standalone utility.
> 2. For llvm-mc, if we go with an option, is there a better name than
--doc-id? David Blaikie proposed --asm-id
>    (This is my personal preference, trading 30+ lines in a utility for
simpler syntax)
> 3. If we add a standalone utility, how shall we name it? (Note that
llvm-extract exists, but people can probably distinguish 'extract' from
llvm-extract
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev