thr3ads.net - llvm dev - [llvm-dev] [lit] check-all hanging [Jan 2019]

If this information is useful, please help other people find it:
Share via:

Joel E. Denny via llvm-dev

2019-Jan-02 22:05 UTC

[llvm-dev] [lit] check-all hanging

Hi David, Chandler,

I see lldb tests hang often, and then I kill the dotest process.

I'd like to stop running check-all too, but I feel it's important when I
modify FileCheck.  The flakiness that Chandler mentioned makes it
time-consuming to verify test results.

Joel

On Wed, Jan 2, 2019 at 4:41 PM Chandler Carruth via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> What you're seeing is just the fact that lit is waiting on subprocesses
> (select is waiting on the pipes i suspect).
>
> Anyways, you'll need to dig into what it is waiting on, and what *that*
> process is doing that is stuck to make progress.
>
> I've not seen anything like this, but I basically never run `check-all`
> these days because LLDB and sanitizer tests are too flaky. =[ I've not
been
> able to interest anyone in fixing this either sadly.
>
> On Wed, Jan 2, 2019 at 10:09 AM David Greene via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>> Hi,
>>
>> From time to time, I see check-all hang during running of lit tests.
>> The hang always happens at the > 90% completion stage and I'm
pretty
>> sure all tests have been run and check-all is just waiting for
>> lit/python to exit.  I see a single python processing running, taking
>> very little CPU time.  An strace of that process shows this:
>>
>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>> select(0, NULL, NULL, NULL, {0, 32168}) = 0 (Timeout)
>> select(0, NULL, NULL, NULL, {0, 1000})  = 0 (Timeout)
>> select(0, NULL, NULL, NULL, {0, 2000})  = 0 (Timeout)
>> select(0, NULL, NULL, NULL, {0, 4000})  = 0 (Timeout)
>> select(0, NULL, NULL, NULL, {0, 8000})  = 0 (Timeout)
>> select(0, NULL, NULL, NULL, {0, 16000}) = 0 (Timeout)
>> select(0, NULL, NULL, NULL, {0, 32000}) = 0 (Timeout)
>> futex(0x3bcc8c0, FUTEX_WAKE_PRIVATE, 1) = 1
>> futex(0x3bcc8c0, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0,
NULL,
>> ffffffff) = 0
>> futex(0x3bcc8c0, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0,
NULL,
>> ffffffff) = -1 EAGAIN (Resourc
>> e temporarily unavailable)
>>
>> futex(0x3bcc8c0, FUTEX_WAKE_PRIVATE, 1) = 1
>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>> futex(0x3bcc8c0, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0,
NULL,
>> ffffffff) = -1 EAGAIN (Resourc
>> e temporarily unavailable)
>>
>> futex(0x3bcc8c0, FUTEX_WAKE_PRIVATE, 1) = 1
>> futex(0x3bcc8c0, FUTEX_WAKE_PRIVATE, 1) = 1
>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>>
>> It appears that python is waiting for some I/O or something which never
>> appears.
>>
>> Has anyone else seen this before?  Any ideas of what is going on or how
>> to fix it?
>>
>>                               -David
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20190102/90aaa7af/attachment.html>

Chandler Carruth via llvm-dev

2019-Jan-02 22:50 UTC

head link

[llvm-dev] [lit] check-all hanging

Might be worth reporting this on the lldb list?

On Wed, Jan 2, 2019 at 2:05 PM Joel E. Denny via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> Hi David, Chandler,
>
> I see lldb tests hang often, and then I kill the dotest process.
>
> I'd like to stop running check-all too, but I feel it's important
when I
> modify FileCheck.  The flakiness that Chandler mentioned makes it
> time-consuming to verify test results.
>
> Joel
>
> On Wed, Jan 2, 2019 at 4:41 PM Chandler Carruth via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>> What you're seeing is just the fact that lit is waiting on
subprocesses
>> (select is waiting on the pipes i suspect).
>>
>> Anyways, you'll need to dig into what it is waiting on, and what
*that*
>> process is doing that is stuck to make progress.
>>
>> I've not seen anything like this, but I basically never run
`check-all`
>> these days because LLDB and sanitizer tests are too flaky. =[ I've
not been
>> able to interest anyone in fixing this either sadly.
>>
>> On Wed, Jan 2, 2019 at 10:09 AM David Greene via llvm-dev <
>> llvm-dev at lists.llvm.org> wrote:
>>
>>> Hi,
>>>
>>> From time to time, I see check-all hang during running of lit
tests.
>>> The hang always happens at the > 90% completion stage and
I'm pretty
>>> sure all tests have been run and check-all is just waiting for
>>> lit/python to exit.  I see a single python processing running,
taking
>>> very little CPU time.  An strace of that process shows this:
>>>
>>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>>> select(0, NULL, NULL, NULL, {0, 32168}) = 0 (Timeout)
>>> select(0, NULL, NULL, NULL, {0, 1000})  = 0 (Timeout)
>>> select(0, NULL, NULL, NULL, {0, 2000})  = 0 (Timeout)
>>> select(0, NULL, NULL, NULL, {0, 4000})  = 0 (Timeout)
>>> select(0, NULL, NULL, NULL, {0, 8000})  = 0 (Timeout)
>>> select(0, NULL, NULL, NULL, {0, 16000}) = 0 (Timeout)
>>> select(0, NULL, NULL, NULL, {0, 32000}) = 0 (Timeout)
>>> futex(0x3bcc8c0, FUTEX_WAKE_PRIVATE, 1) = 1
>>> futex(0x3bcc8c0, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0,
>>> NULL, ffffffff) = 0
>>> futex(0x3bcc8c0, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0,
>>> NULL, ffffffff) = -1 EAGAIN (Resourc
>>> e temporarily unavailable)
>>>
>>> futex(0x3bcc8c0, FUTEX_WAKE_PRIVATE, 1) = 1
>>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>>> futex(0x3bcc8c0, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0,
>>> NULL, ffffffff) = -1 EAGAIN (Resourc
>>> e temporarily unavailable)
>>>
>>> futex(0x3bcc8c0, FUTEX_WAKE_PRIVATE, 1) = 1
>>> futex(0x3bcc8c0, FUTEX_WAKE_PRIVATE, 1) = 1
>>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>>>
>>> It appears that python is waiting for some I/O or something which
never
>>> appears.
>>>
>>> Has anyone else seen this before?  Any ideas of what is going on or
how
>>> to fix it?
>>>
>>>                               -David
>>> _______________________________________________
>>> LLVM Developers mailing list
>>> llvm-dev at lists.llvm.org
>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20190102/dd37532b/attachment.html>

Kuba Mracek via llvm-dev

2019-Jan-02 22:51 UTC

head link

[llvm-dev] [lit] check-all hanging

+Fred, +me

For LLDB tests: I believe this got much much better recently. Are you still
seeing flaky LLDB tests? Any details you can share?
For sanitizer tests: I'm very much interesting in removing flakiness as
well. Any specific tests you see as flaky?

Kuba
> On Jan 2, 2019, at 2:05 PM, Joel E. Denny via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> 
> Hi David, Chandler,
> 
> I see lldb tests hang often, and then I kill the dotest process.
> 
> I'd like to stop running check-all too, but I feel it's important
when I modify FileCheck.  The flakiness that Chandler mentioned makes it
time-consuming to verify test results.
> 
> Joel
> 
> On Wed, Jan 2, 2019 at 4:41 PM Chandler Carruth via llvm-dev <llvm-dev
at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote:
> What you're seeing is just the fact that lit is waiting on subprocesses
(select is waiting on the pipes i suspect).
> 
> Anyways, you'll need to dig into what it is waiting on, and what *that*
process is doing that is stuck to make progress.
> 
> I've not seen anything like this, but I basically never run `check-all`
these days because LLDB and sanitizer tests are too flaky. =[ I've not been
able to interest anyone in fixing this either sadly.
> 
> On Wed, Jan 2, 2019 at 10:09 AM David Greene via llvm-dev <llvm-dev at
lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote:
> Hi,
> 
> From time to time, I see check-all hang during running of lit tests.
> The hang always happens at the > 90% completion stage and I'm pretty
> sure all tests have been run and check-all is just waiting for
> lit/python to exit.  I see a single python processing running, taking
> very little CPU time.  An strace of that process shows this:
> 
> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
> select(0, NULL, NULL, NULL, {0, 32168}) = 0 (Timeout)
> select(0, NULL, NULL, NULL, {0, 1000})  = 0 (Timeout)
> select(0, NULL, NULL, NULL, {0, 2000})  = 0 (Timeout)
> select(0, NULL, NULL, NULL, {0, 4000})  = 0 (Timeout)
> select(0, NULL, NULL, NULL, {0, 8000})  = 0 (Timeout)
> select(0, NULL, NULL, NULL, {0, 16000}) = 0 (Timeout)
> select(0, NULL, NULL, NULL, {0, 32000}) = 0 (Timeout)
> futex(0x3bcc8c0, FUTEX_WAKE_PRIVATE, 1) = 1
> futex(0x3bcc8c0, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL,
ffffffff) = 0
> futex(0x3bcc8c0, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL,
ffffffff) = -1 EAGAIN (Resourc
> e temporarily unavailable)
> futex(0x3bcc8c0, FUTEX_WAKE_PRIVATE, 1) = 1
> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
> futex(0x3bcc8c0, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL,
ffffffff) = -1 EAGAIN (Resourc
> e temporarily unavailable)
> futex(0x3bcc8c0, FUTEX_WAKE_PRIVATE, 1) = 1
> futex(0x3bcc8c0, FUTEX_WAKE_PRIVATE, 1) = 1
> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
> 
> It appears that python is waiting for some I/O or something which never
> appears.
> 
> Has anyone else seen this before?  Any ideas of what is going on or how
> to fix it?
> 
>                               -David
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
<http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
<http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20190102/031842b1/attachment.html>

Joel E. Denny via llvm-dev

2019-Jan-03 15:33 UTC

head link

[llvm-dev] [lit] check-all hanging

All,

Thanks for the replies.  Kuba: For LLDB, when were things expected to have
improved?  It's possible things improved for me at some point, but this
isn't something I've found time to track carefully, and I still see
problems.

I ran check-all a couple of times last night at r350238, which I pulled
yesterday.  Here are the results:

```
********************
Testing Time: 5043.24s
********************
Unexpected Passing Tests (2):
    lldb-Suite :: functionalities/asan/TestMemoryHistory.py
    lldb-Suite :: functionalities/asan/TestReportData.py

********************
Failing Tests (54):
    Clang :: CXX/modules-ts/basic/basic.link/p2/module.cpp
    Clang :: Modules/ExtDebugInfo.cpp
    Clang :: Modules/using-directive-redecl.cpp
    Clang :: Modules/using-directive.cpp
    Clang :: PCH/chain-late-anonymous-namespace.cpp
    Clang :: PCH/cxx-namespaces.cpp
    Clang :: PCH/namespaces.cpp
    LLDB :: ExecControl/StopHook/stop-hook-threads.test
    LeakSanitizer-AddressSanitizer-x86_64 ::
TestCases/Linux/use_tls_dynamic.cc
    LeakSanitizer-Standalone-x86_64 :: TestCases/Linux/use_tls_dynamic.cc
    MemorySanitizer-X86_64 :: dtls_test.c
    MemorySanitizer-lld-X86_64 :: dtls_test.c
    lldb-Suite :: functionalities/register/register_command/TestRegisters.py
    lldb-Suite :: tools/lldb-server/TestGdbRemoteRegisterState.py
    lldb-Unit ::
ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestBorrowedReferences
    lldb-Unit ::
ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestDictionaryResolutionWithDot
    lldb-Unit ::
ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestExtractingUInt64ThroughStructuredData
    lldb-Unit ::
ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestGlobalNameResolutionNoDot
    lldb-Unit ::
ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestGlobalNameResolutionWithDot
    lldb-Unit ::
ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestInstanceNameResolutionNoDot
    lldb-Unit ::
ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestModuleNameResolutionNoDot
    lldb-Unit ::
ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestObjectAttributes
    lldb-Unit ::
ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestOwnedReferences
    lldb-Unit ::
ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonByteArray
    lldb-Unit ::
ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonBytes
    lldb-Unit ::
ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonCallableCheck
    lldb-Unit ::
ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonCallableInvoke
    lldb-Unit ::
ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonDictionaryManipulation
    lldb-Unit ::
ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonDictionaryToStructuredDictionary
    lldb-Unit ::
ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonDictionaryValueEquality
    lldb-Unit ::
ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonFile
    lldb-Unit ::
ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonInteger
    lldb-Unit ::
ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonIntegerToStr
    lldb-Unit ::
ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonIntegerToStructuredInteger
    lldb-Unit ::
ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonListManipulation
    lldb-Unit ::
ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonListToStructuredList
    lldb-Unit ::
ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonListValueEquality
    lldb-Unit ::
ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonString
    lldb-Unit ::
ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonStringToStr
    lldb-Unit ::
ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonStringToStructuredString
    lldb-Unit ::
ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonTupleInitializerList
    lldb-Unit ::
ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonTupleInitializerList2
    lldb-Unit ::
ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonTupleSize
    lldb-Unit ::
ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonTupleToStructuredList
    lldb-Unit ::
ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonTupleValues
    lldb-Unit ::
ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestResetting
    lldb-Unit ::
ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestTypeNameResolutionNoDot
    lldb-Unit ::
ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonExceptionStateTest.TestAcquisitionSemantics
    lldb-Unit ::
ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonExceptionStateTest.TestAutoRestoreChanged
    lldb-Unit ::
ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonExceptionStateTest.TestAutoRestoreSemantics
    lldb-Unit ::
ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonExceptionStateTest.TestDiscardSemantics
    lldb-Unit ::
ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonExceptionStateTest.TestExceptionStateChecking
    lldb-Unit ::
ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonExceptionStateTest.TestManualRestoreSemantics
    lldb-Unit ::
ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonExceptionStateTest.TestResetSemantics

  Expected Passes    : 57489
  Expected Failures  : 276
  Unsupported Tests  : 1883
  Unexpected Passes  : 2
  Unexpected Failures: 54

14 warning(s) in tests.
FAILED: CMakeFiles/check-all
```

I immediately ran it again and saw one new unexpected fail:

```
    lldb-Suite :: tools/lldb-mi/syntax/TestMiSyntax.py
```

and one new unresolved test:

```
    lldb-Suite :: tools/lldb-vscode/breakpoint/TestVSCode_setBreakpoints.py
```

On the second run but not the first, it hung all night long waiting for
TestVSCode_setBreakpoints.py to terminate.  I killed dotest.py to get the
final results.

I currently clone <https://github.com/llvm-project/llvm-project-20170507>.
I configure with `BUILD_SHARED_LIBS=true` and
`-DLLVM_ENABLE_PROJECTS='clang;openmp;libcxx;libcxxabi;lldb;compiler-rt;lld;polly'`,
among other options.  I have to run check-all with LD_LIBRARY_PATH pointing
at my build's lib directory, or there are many more LLDB failures.  I
believe that's not true for most test suites.  I'm building and testing
under Ubuntu 18.04.1.

Hope that helps.  I'm happy to provide more details.  Just tell me where
you'd like to start.

Thanks.

Joel

On Wed, Jan 2, 2019 at 5:51 PM Kuba Mracek <mracek at apple.com> wrote:
> +Fred, +me
>
> For LLDB tests: I believe this got much much better recently. Are you
> still seeing flaky LLDB tests? Any details you can share?
> For sanitizer tests: I'm very much interesting in removing flakiness as
> well. Any specific tests you see as flaky?
>
> Kuba
>
> On Jan 2, 2019, at 2:05 PM, Joel E. Denny via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
> Hi David, Chandler,
>
> I see lldb tests hang often, and then I kill the dotest process.
>
> I'd like to stop running check-all too, but I feel it's important
when I
> modify FileCheck.  The flakiness that Chandler mentioned makes it
> time-consuming to verify test results.
>
> Joel
>
> On Wed, Jan 2, 2019 at 4:41 PM Chandler Carruth via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>> What you're seeing is just the fact that lit is waiting on
subprocesses
>> (select is waiting on the pipes i suspect).
>>
>> Anyways, you'll need to dig into what it is waiting on, and what
*that*
>> process is doing that is stuck to make progress.
>>
>> I've not seen anything like this, but I basically never run
`check-all`
>> these days because LLDB and sanitizer tests are too flaky. =[ I've
not been
>> able to interest anyone in fixing this either sadly.
>>
>> On Wed, Jan 2, 2019 at 10:09 AM David Greene via llvm-dev <
>> llvm-dev at lists.llvm.org> wrote:
>>
>>> Hi,
>>>
>>> From time to time, I see check-all hang during running of lit
tests.
>>> The hang always happens at the > 90% completion stage and
I'm pretty
>>> sure all tests have been run and check-all is just waiting for
>>> lit/python to exit.  I see a single python processing running,
taking
>>> very little CPU time.  An strace of that process shows this:
>>>
>>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>>> select(0, NULL, NULL, NULL, {0, 32168}) = 0 (Timeout)
>>> select(0, NULL, NULL, NULL, {0, 1000})  = 0 (Timeout)
>>> select(0, NULL, NULL, NULL, {0, 2000})  = 0 (Timeout)
>>> select(0, NULL, NULL, NULL, {0, 4000})  = 0 (Timeout)
>>> select(0, NULL, NULL, NULL, {0, 8000})  = 0 (Timeout)
>>> select(0, NULL, NULL, NULL, {0, 16000}) = 0 (Timeout)
>>> select(0, NULL, NULL, NULL, {0, 32000}) = 0 (Timeout)
>>> futex(0x3bcc8c0, FUTEX_WAKE_PRIVATE, 1) = 1
>>> futex(0x3bcc8c0, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0,
>>> NULL, ffffffff) = 0
>>> futex(0x3bcc8c0, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0,
>>> NULL, ffffffff) = -1 EAGAIN (Resourc
>>> e temporarily unavailable)
>>>
>>> futex(0x3bcc8c0, FUTEX_WAKE_PRIVATE, 1) = 1
>>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>>> futex(0x3bcc8c0, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0,
>>> NULL, ffffffff) = -1 EAGAIN (Resourc
>>> e temporarily unavailable)
>>>
>>> futex(0x3bcc8c0, FUTEX_WAKE_PRIVATE, 1) = 1
>>> futex(0x3bcc8c0, FUTEX_WAKE_PRIVATE, 1) = 1
>>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>>> select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>>>
>>> It appears that python is waiting for some I/O or something which
never
>>> appears.
>>>
>>> Has anyone else seen this before?  Any ideas of what is going on or
how
>>> to fix it?
>>>
>>>                               -David
>>> _______________________________________________
>>> LLVM Developers mailing list
>>> llvm-dev at lists.llvm.org
>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
>-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20190103/e5c5e7d8/attachment.html>

David Greene via llvm-dev

2019-Jan-03 21:01 UTC

head link

[llvm-dev] [lit] check-all hanging

We're not running lldb tests, so something else is going on.  I'll dig
into it.

                         -David

"Joel E. Denny" <jdenny.ornl at gmail.com> writes:
> Hi David, Chandler,
>
> I see lldb tests hang often, and then I kill the dotest process.
>
> I'd like to stop running check-all too, but I feel it's important
when
> I modify FileCheck. The flakiness that Chandler mentioned makes it
> time-consuming to verify test results.
>
> Joel
>
> On Wed, Jan 2, 2019 at 4:41 PM Chandler Carruth via llvm-dev
> <llvm-dev at lists.llvm.org> wrote:
>
>     What you're seeing is just the fact that lit is waiting on
>     subprocesses (select is waiting on the pipes i suspect).
>     
>     
>     Anyways, you'll need to dig into what it is waiting on, and what
>     *that* process is doing that is stuck to make progress.
>     
>     
>     I've not seen anything like this, but I basically never run
>     `check-all` these days because LLDB and sanitizer tests are too
>     flaky. =[ I've not been able to interest anyone in fixing this
>     either sadly.
>
>     
>     
>     On Wed, Jan 2, 2019 at 10:09 AM David Greene via llvm-dev
>     <llvm-dev at lists.llvm.org> wrote:
>     
>     Hi,
>         
>         From time to time, I see check-all hang during running of lit
>         tests.
>         The hang always happens at the > 90% completion stage and
I'm
>         pretty
>         sure all tests have been run and check-all is just waiting for
>         lit/python to exit. I see a single python processing running,
>         taking
>         very little CPU time. An strace of that process shows this:
>         
>         select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>         select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>         select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>         select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>         select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>         select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>         select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>         select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>         select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>         select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>         select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>         select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>         select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>         select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>         select(0, NULL, NULL, NULL, {0, 32168}) = 0 (Timeout)
>         select(0, NULL, NULL, NULL, {0, 1000}) = 0 (Timeout)
>         select(0, NULL, NULL, NULL, {0, 2000}) = 0 (Timeout)
>         select(0, NULL, NULL, NULL, {0, 4000}) = 0 (Timeout)
>         select(0, NULL, NULL, NULL, {0, 8000}) = 0 (Timeout)
>         select(0, NULL, NULL, NULL, {0, 16000}) = 0 (Timeout)
>         select(0, NULL, NULL, NULL, {0, 32000}) = 0 (Timeout)
>         futex(0x3bcc8c0, FUTEX_WAKE_PRIVATE, 1) = 1
>         futex(0x3bcc8c0,
>         FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL,
>         ffffffff) = 0
>         futex(0x3bcc8c0,
>         FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL,
>         ffffffff) = -1 EAGAIN (Resourc
>         e temporarily unavailable) 
>         futex(0x3bcc8c0, FUTEX_WAKE_PRIVATE, 1) = 1
>         select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>         select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>         futex(0x3bcc8c0,
>         FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL,
>         ffffffff) = -1 EAGAIN (Resourc
>         e temporarily unavailable) 
>         futex(0x3bcc8c0, FUTEX_WAKE_PRIVATE, 1) = 1
>         futex(0x3bcc8c0, FUTEX_WAKE_PRIVATE, 1) = 1
>         select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>         select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>         select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>         select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>         select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>         select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>         select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>         select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>         select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>         select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
>         
>         It appears that python is waiting for some I/O or something
>         which never
>         appears.
>         
>         Has anyone else seen this before? Any ideas of what is going
>         on or how
>         to fix it?
>         
>         -David
>         _______________________________________________
>         LLVM Developers mailing list
>         llvm-dev at lists.llvm.org
>         http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>     _______________________________________________
>     LLVM Developers mailing list
>     llvm-dev at lists.llvm.org
>     http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

llvm dev - Jan 2019 - [lit] check-all hanging

[llvm-dev] [lit] check-all hanging

[llvm-dev] [lit] check-all hanging

[llvm-dev] [lit] check-all hanging

[llvm-dev] [lit] check-all hanging

[llvm-dev] [lit] check-all hanging