David Blaikie via llvm-dev
2021-Feb-05 18:50 UTC
[llvm-dev] [RFC] Cross-project lit test suite
Reid and Amy might have some context for Windows (though I don't know if any Windows folks have done much with this test suite). On Fri, Feb 5, 2021 at 7:38 AM James Henderson via llvm-dev <llvm-dev at lists.llvm.org> wrote:> > Given that the debuginfo tests already have cross-project dependencies, I figured I'd try adapting them instead. I've updated https://reviews.llvm.org/D95339 accordingly. Ideally, I think making the existing debug-info tests a subdirectory, and renaming the top-level directory, might be a good idea, but I haven't really come to any conclusions about that yet. > > I also found that several of the existing debuginfo-test tests fail for me. Are these tests expected to work on Windows? If so, are there any slightly more unusual prerequisites that I might be missing? > > What do people think? > > James > > On Wed, 27 Jan 2021 at 15:40, <paul.robinson at sony.com> wrote: >> >> >> >> > -----Original Message----- >> > From: llvm-dev <llvm-dev-bounces at lists.llvm.org> On Behalf Of David Greene >> > via llvm-dev >> > Sent: Wednesday, January 27, 2021 10:29 AM >> > To: jh7370.2008 at my.bristol.ac.uk; llvm-dev <llvm-dev at lists.llvm.org> >> > Subject: Re: [llvm-dev] [RFC] Cross-project lit test suite >> > >> > James Henderson via llvm-dev <llvm-dev at lists.llvm.org> writes: >> > >> > > Currently, there is no location where lit tests that use both clang and >> > LLD >> > > can be put, whilst the llvm-symbolizer cases I’ve hit are testing >> > > llvm-symbolizer (and not LLD), so don’t really fit in the LLD test >> > suite. I >> > > therefore have prototyped a lit test suite that would be part of the >> > > monorepo, and which can support tests that use elements from multiple >> > > projects - see >> > https://urldefense.com/v3/__https://reviews.llvm.org/D95339__;!!JmoZiZGBv3 >> > RvKRSx!vWidWrbKJid6-eIKVUT-dGDzcG-65TMZMzhyd33jgyBwi7p-JRSgFVZkxqKCvkqW4A$ >> > . Tests could be added to >> > > this suite as needed. The suite is modelled as an additional top-level >> > > directory, and is enabled by enabling the “cross-project-tests” project >> > in >> > > CMake. >> > >> > This is fantastic! >> > >> > > Back in October 2019, there was an extensive discussion on end-to-end >> > > testing and how to write them (starting from >> > > https://urldefense.com/v3/__https://lists.llvm.org/pipermail/cfe- >> > dev/2019-October/063509.html__;!!JmoZiZGBv3RvKRSx!vWidWrbKJid6-eIKVUT- >> > dGDzcG-65TMZMzhyd33jgyBwi7p-JRSgFVZkxqJU8k_M_Q$ ). >> > > The suggestion was that these tests would be lit-based and run as part >> > > of check-all, and would not be inside the clang tree, although there >> > > was some opposition. This concluded with a round table. Unfortunately, >> > > I am unaware of what the conclusion of that round table conversation >> > > was, so it’s possible that what I am proposing is redundant/being >> > > worked on by someone else. >> > >> > I started that thread and IIRC we ended up with the suggestion that such >> > tests should live in test-suite. As you noted having tests separated >> > from the monorepo is less than ideal. I haven't done anything with this >> > conclusion yet, mostly due to lack of time. If your proposal gains >> > traction I would like to see if we could build end-to-end testing on top >> > of it. >> > >> > > Additionally, I don’t consider all classes of tests that the proposed >> > > lit suite would be useful for to be “end-to-end” testing. >> > >> > Agreed. There are various classes of tests that could make use of your >> > proposed layout, one of which is "end-to-end." Your proposal doesn't >> > provide end-to-end testing per se, but it does make adding end-to-end >> > tests later on more straightforward. >> > >> > -David >> >> I think a useful distinction here is that lit tests are generally very >> focused on a specific feature/function, where test-suite has a much >> broader scope. Another slice at it would be that lit tests tend to be >> "regression" tests, while test-suite is more of an "integration" suite. >> >> I am not a QA person so I may be abusing some of these terms, but that's >> how I look at it. Sometimes writing that focused lit test ends up >> depending on multiple tools, and the cross-project lit suite would be a >> good place to drop those; debuginfo-tests is a prime example. >> --paulr >> > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
Alexandre Ganea via llvm-dev
2021-Feb-05 20:47 UTC
[llvm-dev] [RFC] Cross-project lit test suite
The debuginfo-test tests are failing for a long time for us too. I won't
have much time to fix them in the short term, but here's the errors I'm
seeing (James do you see the same thing on your end?):
Failed Tests (7):
debuginfo-tests ::
dexter/feature_tests/commands/penalty/expect_program_state.cpp
debuginfo-tests :: dexter/feature_tests/commands/penalty/expect_step_kinds.cpp
debuginfo-tests :: dexter/feature_tests/commands/penalty/expect_step_order.cpp
debuginfo-tests :: dexter/feature_tests/commands/penalty/expect_watch_type.cpp
debuginfo-tests ::
dexter/feature_tests/commands/penalty/expect_watch_value.cpp
debuginfo-tests :: dexter/feature_tests/commands/penalty/unreachable.cpp
debuginfo-tests :: dexter/feature_tests/unittests/run.test
The first ones seem to be related to a missing Python library in my
installation:
F:\aganea\llvm-project>"C:/Program Files/Python39/python.exe"
"F:/aganea/llvm-project/debuginfo-tests\dexter\dexter.py"
list-debuggers
dbgeng [dbgeng] YES (1)
lldb [lldb] NO (The system cannot find the file specified
["lldb.exe"])
vs2015 [Visual Studio 2015] NO (No module named 'win32com')
vs2017 [Visual Studio 2017] NO (No module named 'win32com')
vs2019 [Visual Studio 2019] NO (No module named 'win32com')
Which in turns generates this error:
F:\aganea\llvm-project>"C:/Program Files/Python39/python.exe"
F:/aganea/llvm-project/debuginfo-tests\dexter\dexter.py test --fail-lt 1.0 -w
--builder clang-cl_vs2015 --debugger dbgeng --cflags "/Zi /Od"
--ldflags "/Zi" --
F:\aganea\llvm-project\debuginfo-tests\dexter\feature_tests\commands\penalty\expect_watch_type.cpp
expect_watch_type.cpp: nan/nan (nan)
[F:\aganea\llvm-project\debuginfo-tests\dexter\dex\builder\scripts\windows\clang-cl_vs2015.bat:
failed with returncode 1. : The system cannot find the path specified.]
The last failure (dexter/feature_tests/unittests/run.test) is related to slash
vs. backslash:
F:\aganea\llvm-project>"C:/Program Files/Python39/python.exe"
"F:/aganea/llvm-project/debuginfo-tests\dexter\dexter.py"
"--unittest=show-all"
test_get_script_environment (builder.Builder.TestBuilder) ... ok
test_parse_bad_whitespace (command.ParseCommand.TestParseCommand)
Throw exception when parsing badly formed whitespace. ... ok
test_parse_empty (command.ParseCommand.TestParseCommand)
Empty files are silently ignored. ... ok
test_parse_escaped (command.ParseCommand.TestParseCommand)
Escaped commands are ignored. ... ok
test_parse_good_whitespace (command.ParseCommand.TestParseCommand)
Try to emulate python whitespace rules ... ok
test_parse_inline (command.ParseCommand.TestParseCommand)
Commands can be embedded in other text. ... ok
test_parse_multi_line_comment (command.ParseCommand.TestParseCommand)
Multi-line commands can embed comments. ... ok
test_parse_share_line (command.ParseCommand.TestParseCommand)
More than one command can appear on one line. ... ok
test_add_breakpoint_no_source_root_dir (debugger.DebuggerBase.TestDebuggerBase)
... ok
test_add_breakpoint_with_source_root_dir
(debugger.DebuggerBase.TestDebuggerBase) ... FAIL
test_add_breakpoint_with_source_root_dir_slash_suffix
(debugger.DebuggerBase.TestDebuggerBase) ... ok
test_get_step_info (debugger.DebuggerBase.TestDebuggerBase) ... FAIL
test_get_step_info_no_frames (debugger.DebuggerBase.TestDebuggerBase) ... ok
test_get_step_info_no_source_root_dir (debugger.DebuggerBase.TestDebuggerBase)
... FAIL
test_did_you_mean (utils.ExtArgParse.TestExtArgumentParser) ... ok
test_PreserveAutoColors (utils.PrettyOutputBase.TestPrettyOutput) ... ok
test_auto (utils.PrettyOutputBase.TestPrettyOutput) ... ok
test_blue (utils.PrettyOutputBase.TestPrettyOutput) ... ok
test_default (utils.PrettyOutputBase.TestPrettyOutput) ... ok
test_green (utils.PrettyOutputBase.TestPrettyOutput) ... ok
test_red (utils.PrettyOutputBase.TestPrettyOutput) ... ok
test_tags (utils.PrettyOutputBase.TestPrettyOutput) ... ok
test_yellow (utils.PrettyOutputBase.TestPrettyOutput) ... ok
test_sanity (utils.UnitTests.TestUnitTests) ... ok
=====================================================================FAIL:
test_add_breakpoint_with_source_root_dir
(debugger.DebuggerBase.TestDebuggerBase)
----------------------------------------------------------------------
Traceback (most recent call last):
File
"F:\aganea\llvm-project\debuginfo-tests\dexter\dex\debugger\DebuggerBase.py",
line 245, in test_add_breakpoint_with_source_root_dir
self.assertEqual('some_file', self.dbg.breakpoint_file)
AssertionError: 'some_file' != '/some_file'
- some_file
+ /some_file
? +
=====================================================================FAIL:
test_get_step_info (debugger.DebuggerBase.TestDebuggerBase)
----------------------------------------------------------------------
Traceback (most recent call last):
File
"F:\aganea\llvm-project\debuginfo-tests\dexter\dex\debugger\DebuggerBase.py",
line 268, in test_get_step_info
self.assertEqual([None, '/other/file',
'/my_root/some_file'],
AssertionError: Lists differ: [None, '/other/file',
'/my_root/some_file'] != [None, '\\other\\file',
'\\dbg\\some_file']
First differing element 1:
'/other/file'
'\\other\\file'
- [None, '/other/file', '/my_root/some_file']
+ [None, '\\other\\file', '\\dbg\\some_file']
=====================================================================FAIL:
test_get_step_info_no_source_root_dir (debugger.DebuggerBase.TestDebuggerBase)
----------------------------------------------------------------------
Traceback (most recent call last):
File
"F:\aganea\llvm-project\debuginfo-tests\dexter\dex\debugger\DebuggerBase.py",
line 254, in test_get_step_info_no_source_root_dir
self.assertEqual(['/root/some_file'],
AssertionError: Lists differ: ['/root/some_file'] !=
['\\root\\some_file']
First differing element 0:
'/root/some_file'
'\\root\\some_file'
- ['/root/some_file']
? ^ ^
+ ['\\root\\some_file']
? ^^ ^^
----------------------------------------------------------------------
Ran 24 tests in 0.005s
FAILED (failures=3)
error: unit test failures
-----Message d'origine-----
De : llvm-dev <llvm-dev-bounces at lists.llvm.org> De la part de David
Blaikie via llvm-dev
Envoyé : February 5, 2021 1:50 PM
À : James Henderson <jh7370.2008 at my.bristol.ac.uk>; Reid Kleckner
<rnk at google.com>; Amy Huang <akhuang at google.com>
Cc : llvm-dev <llvm-dev at lists.llvm.org>; David Greene <greened at
obbligato.org>
Objet : Re: [llvm-dev] [RFC] Cross-project lit test suite
Reid and Amy might have some context for Windows (though I don't know if any
Windows folks have done much with this test suite).
On Fri, Feb 5, 2021 at 7:38 AM James Henderson via llvm-dev <llvm-dev at
lists.llvm.org> wrote:>
> Given that the debuginfo tests already have cross-project dependencies, I
figured I'd try adapting them instead. I've updated
https://reviews.llvm.org/D95339 accordingly. Ideally, I think making the
existing debug-info tests a subdirectory, and renaming the top-level directory,
might be a good idea, but I haven't really come to any conclusions about
that yet.
>
> I also found that several of the existing debuginfo-test tests fail for me.
Are these tests expected to work on Windows? If so, are there any slightly more
unusual prerequisites that I might be missing?
>
> What do people think?
>
> James
>
> On Wed, 27 Jan 2021 at 15:40, <paul.robinson at sony.com> wrote:
>>
>>
>>
>> > -----Original Message-----
>> > From: llvm-dev <llvm-dev-bounces at lists.llvm.org> On
Behalf Of David
>> > Greene via llvm-dev
>> > Sent: Wednesday, January 27, 2021 10:29 AM
>> > To: jh7370.2008 at my.bristol.ac.uk; llvm-dev
>> > <llvm-dev at lists.llvm.org>
>> > Subject: Re: [llvm-dev] [RFC] Cross-project lit test suite
>> >
>> > James Henderson via llvm-dev <llvm-dev at lists.llvm.org>
writes:
>> >
>> > > Currently, there is no location where lit tests that use both
>> > > clang and
>> > LLD
>> > > can be put, whilst the llvm-symbolizer cases I’ve hit are
testing
>> > > llvm-symbolizer (and not LLD), so don’t really fit in the LLD
>> > > test
>> > suite. I
>> > > therefore have prototyped a lit test suite that would be part
of
>> > > the monorepo, and which can support tests that use elements
from
>> > > multiple projects - see
>> >
https://urldefense.com/v3/__https://reviews.llvm.org/D95339__;!!Jmo
>> > ZiZGBv3
>> >
RvKRSx!vWidWrbKJid6-eIKVUT-dGDzcG-65TMZMzhyd33jgyBwi7p-JRSgFVZkxqKC
>> > vkqW4A$
>> > . Tests could be added to
>> > > this suite as needed. The suite is modelled as an additional
>> > > top-level directory, and is enabled by enabling the
>> > > “cross-project-tests” project
>> > in
>> > > CMake.
>> >
>> > This is fantastic!
>> >
>> > > Back in October 2019, there was an extensive discussion on
>> > > end-to-end testing and how to write them (starting from
>> > >
https://urldefense.com/v3/__https://lists.llvm.org/pipermail/cfe-
>> >
dev/2019-October/063509.html__;!!JmoZiZGBv3RvKRSx!vWidWrbKJid6-eIKV
>> > UT- dGDzcG-65TMZMzhyd33jgyBwi7p-JRSgFVZkxqJU8k_M_Q$ ).
>> > > The suggestion was that these tests would be lit-based and
run as
>> > > part of check-all, and would not be inside the clang tree,
>> > > although there was some opposition. This concluded with a
round
>> > > table. Unfortunately, I am unaware of what the conclusion of
that
>> > > round table conversation was, so it’s possible that what I am
>> > > proposing is redundant/being worked on by someone else.
>> >
>> > I started that thread and IIRC we ended up with the suggestion
that
>> > such tests should live in test-suite. As you noted having tests
>> > separated from the monorepo is less than ideal. I haven't
done
>> > anything with this conclusion yet, mostly due to lack of time. If
>> > your proposal gains traction I would like to see if we could build
>> > end-to-end testing on top of it.
>> >
>> > > Additionally, I don’t consider all classes of tests that the
>> > > proposed lit suite would be useful for to be “end-to-end”
testing.
>> >
>> > Agreed. There are various classes of tests that could make use of
>> > your proposed layout, one of which is "end-to-end."
Your proposal
>> > doesn't provide end-to-end testing per se, but it does make
adding
>> > end-to-end tests later on more straightforward.
>> >
>> > -David
>>
>> I think a useful distinction here is that lit tests are generally
>> very focused on a specific feature/function, where test-suite has a
>> much broader scope. Another slice at it would be that lit tests tend
>> to be "regression" tests, while test-suite is more of an
"integration" suite.
>>
>> I am not a QA person so I may be abusing some of these terms, but
>> that's how I look at it. Sometimes writing that focused lit test
>> ends up depending on multiple tools, and the cross-project lit suite
>> would be a good place to drop those; debuginfo-tests is a prime
example.
>> --paulr
>>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
_______________________________________________
LLVM Developers mailing list
llvm-dev at lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
Reid Kleckner via llvm-dev
2021-Feb-08 17:31 UTC
[llvm-dev] [RFC] Cross-project lit test suite
I haven't done any Windows work with debuginfo-tests on Windows since the dexter transition. I had to disable the tests on my bot because they didn't pass out of the box, and I never found time to follow up. It seems like other people have more leads as to what exactly is going wrong. I'd love to re-enable them, but I probably don't have time to debug them. On Fri, Feb 5, 2021 at 10:50 AM David Blaikie <dblaikie at gmail.com> wrote:> Reid and Amy might have some context for Windows (though I don't know > if any Windows folks have done much with this test suite). > > On Fri, Feb 5, 2021 at 7:38 AM James Henderson via llvm-dev > <llvm-dev at lists.llvm.org> wrote: > > > > Given that the debuginfo tests already have cross-project dependencies, > I figured I'd try adapting them instead. I've updated > https://reviews.llvm.org/D95339 accordingly. Ideally, I think making the > existing debug-info tests a subdirectory, and renaming the top-level > directory, might be a good idea, but I haven't really come to any > conclusions about that yet. > > > > I also found that several of the existing debuginfo-test tests fail for > me. Are these tests expected to work on Windows? If so, are there any > slightly more unusual prerequisites that I might be missing? > > > > What do people think? > > > > James > > > > On Wed, 27 Jan 2021 at 15:40, <paul.robinson at sony.com> wrote: > >> > >> > >> > >> > -----Original Message----- > >> > From: llvm-dev <llvm-dev-bounces at lists.llvm.org> On Behalf Of David > Greene > >> > via llvm-dev > >> > Sent: Wednesday, January 27, 2021 10:29 AM > >> > To: jh7370.2008 at my.bristol.ac.uk; llvm-dev <llvm-dev at lists.llvm.org> > >> > Subject: Re: [llvm-dev] [RFC] Cross-project lit test suite > >> > > >> > James Henderson via llvm-dev <llvm-dev at lists.llvm.org> writes: > >> > > >> > > Currently, there is no location where lit tests that use both clang > and > >> > LLD > >> > > can be put, whilst the llvm-symbolizer cases I’ve hit are testing > >> > > llvm-symbolizer (and not LLD), so don’t really fit in the LLD test > >> > suite. I > >> > > therefore have prototyped a lit test suite that would be part of the > >> > > monorepo, and which can support tests that use elements from > multiple > >> > > projects - see > >> > > https://urldefense.com/v3/__https://reviews.llvm.org/D95339__;!!JmoZiZGBv3 > >> > > RvKRSx!vWidWrbKJid6-eIKVUT-dGDzcG-65TMZMzhyd33jgyBwi7p-JRSgFVZkxqKCvkqW4A$ > >> > . Tests could be added to > >> > > this suite as needed. The suite is modelled as an additional > top-level > >> > > directory, and is enabled by enabling the “cross-project-tests” > project > >> > in > >> > > CMake. > >> > > >> > This is fantastic! > >> > > >> > > Back in October 2019, there was an extensive discussion on > end-to-end > >> > > testing and how to write them (starting from > >> > > https://urldefense.com/v3/__https://lists.llvm.org/pipermail/cfe- > >> > dev/2019-October/063509.html__;!!JmoZiZGBv3RvKRSx!vWidWrbKJid6-eIKVUT- > >> > dGDzcG-65TMZMzhyd33jgyBwi7p-JRSgFVZkxqJU8k_M_Q$ ). > >> > > The suggestion was that these tests would be lit-based and run as > part > >> > > of check-all, and would not be inside the clang tree, although there > >> > > was some opposition. This concluded with a round table. > Unfortunately, > >> > > I am unaware of what the conclusion of that round table conversation > >> > > was, so it’s possible that what I am proposing is redundant/being > >> > > worked on by someone else. > >> > > >> > I started that thread and IIRC we ended up with the suggestion that > such > >> > tests should live in test-suite. As you noted having tests separated > >> > from the monorepo is less than ideal. I haven't done anything with > this > >> > conclusion yet, mostly due to lack of time. If your proposal gains > >> > traction I would like to see if we could build end-to-end testing on > top > >> > of it. > >> > > >> > > Additionally, I don’t consider all classes of tests that the > proposed > >> > > lit suite would be useful for to be “end-to-end” testing. > >> > > >> > Agreed. There are various classes of tests that could make use of > your > >> > proposed layout, one of which is "end-to-end." Your proposal doesn't > >> > provide end-to-end testing per se, but it does make adding end-to-end > >> > tests later on more straightforward. > >> > > >> > -David > >> > >> I think a useful distinction here is that lit tests are generally very > >> focused on a specific feature/function, where test-suite has a much > >> broader scope. Another slice at it would be that lit tests tend to be > >> "regression" tests, while test-suite is more of an "integration" suite. > >> > >> I am not a QA person so I may be abusing some of these terms, but that's > >> how I look at it. Sometimes writing that focused lit test ends up > >> depending on multiple tools, and the cross-project lit suite would be a > >> good place to drop those; debuginfo-tests is a prime example. > >> --paulr > >> > > _______________________________________________ > > LLVM Developers mailing list > > llvm-dev at lists.llvm.org > > https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20210208/5335ae89/attachment.html>