llvm dev - May 2015 - [LLVMdev] WinEH work to be done (in progress and otherwise)

Hi,

Have any decisions been made on where to put executable WinEH tests?  Wherever
they go, would it be helpful if I were to add some C++EH and/or SEH tests pulled
from MSVC's test suite, since compatibility is the goal here?

Thanks
-Joseph

From: llvmdev-bounces at cs.uiuc.edu [mailto:llvmdev-bounces at cs.uiuc.edu] On
Behalf Of Reid Kleckner
Sent: Tuesday, May 19, 2015 5:56 PM
To: Kaylor, Andrew
Cc: llvmdev at cs.uiuc.edu
Subject: Re: [LLVMdev] WinEH work to be done (in progress and otherwise)

On Tue, May 19, 2015 at 2:16 PM, Kaylor, Andrew <andrew.kaylor at
intel.com<mailto:andrew.kaylor at intel.com>> wrote:
Apart from this, the other WinEH-related task that I have on my radar (but as of
yet haven’t done anything about) is getting “nightly” tests running on Windows. 
I did look into the instructions for running the “test-suite” tests, and it
looks like that test suite is assuming a Unix-type platform to some extent.  I
tried to re-interpret the instruction and run this suite on Windows, but it
didn’t work and I haven’t looked into trying to fix whatever the problems were. 
I saw that someone submitted a patch last November that claimed to improve the
running of these tests on Windows, but the patch was never reviewed or
committed.

So I was wondering, is anyone running compile-and-execute type tests with clang
on a native (i.e. non-cygwin/mingw) Windows platform?  It seems like this will
be an important thing to do in general, but particularly so for Windows EH,
dependent as it is on interaction with the runtime library.  That is, having
tests which verify that we’re producing the xdata entries we intend to produce
doesn’t seem like a very thorough method of testing and is no guarantee at all
that we are producing working executables.  In addition to the existing C++ EH
tests in the “test-suite” tests, I have a suite of tests (not currently in any
public repository) that were specifically developed to exercise Windows-specific
problem cases.  I’d eventually like to see these tests run on some regular basis
against clang on Windows.  Is there any work currently under way to make
something like this happen?

I'd like to have a place where we can put small, general purpose execution
tests for clang, but we don't have a good place. test-suite is just too big
and non-portable. The results it generates are also not suitable for consumption
by developers. If someone upstream breaks a C++ EH execution test, what should
they do, how should they debug? A portable assembly-emission test works because
they can see the before and after and evaluate whether their change was correct
or not.

For the rest of the MS C++ ABI, we've gotten by self-hosting Clang and then
building Chromium, and that's enough coverage. Neither project uses C++
exceptions, so that's not going to cut it for this. Long ago Timur used to
maintain a small repo in Google Code for this:
https://code.google.com/p/smoke-cpp-tests/
Nobody runs it though and we haven't updated as we encountered and solved
problems.

I think the best we can do is add a subdirectory of test-suite that's
outside it's makefile system and go from there. Or we could add a new repo
similar to the ABI test suite that Sony contributed. However, you may see that
as a cautionary tale. Sony added the test suite, and nobody else runs it and the
bot has been broken for a month.

I guess my conclusion is that this will probably be a lot of work, and I think
it'll provide less value than you expect from it. But, if you think it's
worth it and want to push it through, then it costs LLVM very little check a few
test cases in outside the main clang/llvm repos.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20150528/ed1974bc/attachment.html>

I think we should make a new svn repository for these. test-suite is the
logical place, but it's way too bloated and has too much Unix specific
baggage. I'd really like to have something that just uses lit, can be
checked out into projects, and can be run as part of check-all, like the
compiler-rt execution test suite.

My suggestion is to call it 'execution-tests', and we can arrange to
auto-detect it if it gets checked out under llvm/projects/ and set up a
test suite.

On Thu, May 28, 2015 at 1:35 PM, Joseph Tremoulet <jotrem at
microsoft.com>
wrote:
>  Hi,
>
>
>
> Have any decisions been made on where to put executable WinEH tests?
> Wherever they go, would it be helpful if I were to add some C++EH and/or
> SEH tests pulled from MSVC's test suite, since compatibility is the
goal
> here?
>
>
>
> Thanks
>
> -Joseph
>
>
>
> *From:* llvmdev-bounces at cs.uiuc.edu [mailto:llvmdev-bounces at
cs.uiuc.edu] *On
> Behalf Of *Reid Kleckner
> *Sent:* Tuesday, May 19, 2015 5:56 PM
> *To:* Kaylor, Andrew
> *Cc:* llvmdev at cs.uiuc.edu
> *Subject:* Re: [LLVMdev] WinEH work to be done (in progress and otherwise)
>
>
>
> On Tue, May 19, 2015 at 2:16 PM, Kaylor, Andrew <andrew.kaylor at
intel.com>
> wrote:
>
>  Apart from this, the other WinEH-related task that I have on my radar
> (but as of yet haven’t done anything about) is getting “nightly” tests
> running on Windows.  I did look into the instructions for running the
> “test-suite” tests, and it looks like that test suite is assuming a
> Unix-type platform to some extent.  I tried to re-interpret the instruction
> and run this suite on Windows, but it didn’t work and I haven’t looked into
> trying to fix whatever the problems were.  I saw that someone submitted a
> patch last November that claimed to improve the running of these tests on
> Windows, but the patch was never reviewed or committed.
>
>
>
>  So I was wondering, is anyone running compile-and-execute type tests
> with clang on a native (i.e. non-cygwin/mingw) Windows platform?  It seems
> like this will be an important thing to do in general, but particularly so
> for Windows EH, dependent as it is on interaction with the runtime
> library.  That is, having tests which verify that we’re producing the xdata
> entries we intend to produce doesn’t seem like a very thorough method of
> testing and is no guarantee at all that we are producing working
> executables.  In addition to the existing C++ EH tests in the “test-suite”
> tests, I have a suite of tests (not currently in any public repository)
> that were specifically developed to exercise Windows-specific problem
> cases.  I’d eventually like to see these tests run on some regular basis
> against clang on Windows.  Is there any work currently under way to make
> something like this happen?
>
>
>
> I'd like to have a place where we can put small, general purpose
execution
> tests for clang, but we don't have a good place. test-suite is just too
big
> and non-portable. The results it generates are also not suitable for
> consumption by developers. If someone upstream breaks a C++ EH execution
> test, what should they do, how should they debug? A portable
> assembly-emission test works because they can see the before and after and
> evaluate whether their change was correct or not.
>
>
>
> For the rest of the MS C++ ABI, we've gotten by self-hosting Clang and
> then building Chromium, and that's enough coverage. Neither project
uses
> C++ exceptions, so that's not going to cut it for this. Long ago Timur
used
> to maintain a small repo in Google Code for this:
>
> https://code.google.com/p/smoke-cpp-tests/
>
> Nobody runs it though and we haven't updated as we encountered and
solved
> problems.
>
>
>
> I think the best we can do is add a subdirectory of test-suite that's
> outside it's makefile system and go from there. Or we could add a new
repo
> similar to the ABI test suite that Sony contributed. However, you may see
> that as a cautionary tale. Sony added the test suite, and nobody else runs
> it and the bot has been broken for a month.
>
>
>
> I guess my conclusion is that this will probably be a lot of work, and I
> think it'll provide less value than you expect from it. But, if you
think
> it's worth it and want to push it through, then it costs LLVM very
little
> check a few test cases in outside the main clang/llvm repos.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20150528/2ebf9d08/attachment.html>

We already have a "unittests" tree in the test-suite.

If you want to create a new repository, you need to widely discuss it and
have a good plan for integrating it into build bots etc.

On Thu, May 28, 2015 at 2:36 PM Reid Kleckner <rnk at google.com> wrote:
> I think we should make a new svn repository for these. test-suite is the
> logical place, but it's way too bloated and has too much Unix specific
> baggage. I'd really like to have something that just uses lit, can be
> checked out into projects, and can be run as part of check-all, like the
> compiler-rt execution test suite.
>
> My suggestion is to call it 'execution-tests', and we can arrange
to
> auto-detect it if it gets checked out under llvm/projects/ and set up a
> test suite.
>
> On Thu, May 28, 2015 at 1:35 PM, Joseph Tremoulet <jotrem at
microsoft.com>
> wrote:
>
>>  Hi,
>>
>>
>>
>> Have any decisions been made on where to put executable WinEH tests?
>> Wherever they go, would it be helpful if I were to add some C++EH
and/or
>> SEH tests pulled from MSVC's test suite, since compatibility is the
goal
>> here?
>>
>>
>>
>> Thanks
>>
>> -Joseph
>>
>>
>>
>> *From:* llvmdev-bounces at cs.uiuc.edu [mailto:llvmdev-bounces at
cs.uiuc.edu] *On
>> Behalf Of *Reid Kleckner
>> *Sent:* Tuesday, May 19, 2015 5:56 PM
>> *To:* Kaylor, Andrew
>> *Cc:* llvmdev at cs.uiuc.edu
>> *Subject:* Re: [LLVMdev] WinEH work to be done (in progress and
>> otherwise)
>>
>>
>>
>> On Tue, May 19, 2015 at 2:16 PM, Kaylor, Andrew <andrew.kaylor at
intel.com>
>> wrote:
>>
>>  Apart from this, the other WinEH-related task that I have on my radar
>> (but as of yet haven’t done anything about) is getting “nightly” tests
>> running on Windows.  I did look into the instructions for running the
>> “test-suite” tests, and it looks like that test suite is assuming a
>> Unix-type platform to some extent.  I tried to re-interpret the
instruction
>> and run this suite on Windows, but it didn’t work and I haven’t looked
into
>> trying to fix whatever the problems were.  I saw that someone submitted
a
>> patch last November that claimed to improve the running of these tests
on
>> Windows, but the patch was never reviewed or committed.
>>
>>
>>
>>  So I was wondering, is anyone running compile-and-execute type tests
>> with clang on a native (i.e. non-cygwin/mingw) Windows platform?  It
seems
>> like this will be an important thing to do in general, but particularly
so
>> for Windows EH, dependent as it is on interaction with the runtime
>> library.  That is, having tests which verify that we’re producing the
xdata
>> entries we intend to produce doesn’t seem like a very thorough method
of
>> testing and is no guarantee at all that we are producing working
>> executables.  In addition to the existing C++ EH tests in the
“test-suite”
>> tests, I have a suite of tests (not currently in any public repository)
>> that were specifically developed to exercise Windows-specific problem
>> cases.  I’d eventually like to see these tests run on some regular
basis
>> against clang on Windows.  Is there any work currently under way to
make
>> something like this happen?
>>
>>
>>
>> I'd like to have a place where we can put small, general purpose
>> execution tests for clang, but we don't have a good place.
test-suite is
>> just too big and non-portable. The results it generates are also not
>> suitable for consumption by developers. If someone upstream breaks a
C++ EH
>> execution test, what should they do, how should they debug? A portable
>> assembly-emission test works because they can see the before and after
and
>> evaluate whether their change was correct or not.
>>
>>
>>
>> For the rest of the MS C++ ABI, we've gotten by self-hosting Clang
and
>> then building Chromium, and that's enough coverage. Neither project
uses
>> C++ exceptions, so that's not going to cut it for this. Long ago
Timur used
>> to maintain a small repo in Google Code for this:
>>
>> https://code.google.com/p/smoke-cpp-tests/
>>
<https://urldefense.proofpoint.com/v2/url?u=https-3A__code.google.com_p_smoke-2Dcpp-2Dtests_&d=AwMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=Mfk2qtn1LTDThVkh6-oGglNfMADXfJdty4_bhmuhMHA&m=nSJRS9-5NwSLdiLng62PIvrk3rcZRDmpF2oboUMhpgE&s=4QsKoKOgmJ6EUy3biMinr91bQc7pSbgjbvRibkg6lIA&e=>
>>
>> Nobody runs it though and we haven't updated as we encountered and
solved
>> problems.
>>
>>
>>
>> I think the best we can do is add a subdirectory of test-suite
that's
>> outside it's makefile system and go from there. Or we could add a
new repo
>> similar to the ABI test suite that Sony contributed. However, you may
see
>> that as a cautionary tale. Sony added the test suite, and nobody else
runs
>> it and the bot has been broken for a month.
>>
>>
>>
>> I guess my conclusion is that this will probably be a lot of work, and
I
>> think it'll provide less value than you expect from it. But, if you
think
>> it's worth it and want to push it through, then it costs LLVM very
little
>> check a few test cases in outside the main clang/llvm repos.
>>
>
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20150528/2df3f699/attachment.html>