llvm dev - Nov 2019 - libunwind is not configured with -funwind-tables when building it for ARM Linux?

Hi Peter,

Thanks for your response.
> On 18 Nov 2019, at 17:44, Peter Smith <peter.smith at linaro.org>
wrote:
> 
> On Mon, 18 Nov 2019 at 12:32, Sergej Jaskiewicz via llvm-dev
> <llvm-dev at lists.llvm.org <mailto:llvm-dev at
lists.llvm.org>> wrote:
>> 
>> There’s this bug: https://bugs.llvm.org/show_bug.cgi?id=38468
<https://bugs.llvm.org/show_bug.cgi?id=38468>.
>> 
>> I’ve managed to track it down to a configuration issue. The thing is
that in order for libunwind to be usable on ARM Linux, it should be built with
the -funwind-tables flag. This flag is conditionally set here:
https://github.com/llvm/llvm-project/blob/master/libunwind/CMakeLists.txt#L294
<https://github.com/llvm/llvm-project/blob/master/libunwind/CMakeLists.txt#L294>,
if the compiler “supports” it.
>> 
> 
> Are you sure that libunwind being built without -funwind-tables is the
> cause of the bug?
Yep, pretty sure, I’ve found that the cause of the problem is the
_Unwind_Backtrace function not executing the provided callback. It isn’t doing
so because, since libunwind is compiled without the flag, the information about
the stack frame is lost, so, when _Unwind_Backtrace looks for it, it can’t find
it (since we’ve entered the _Unwind_Backtrace stack frame, which lives in
libunwind, where no unwind info is present).

I’ve looked at the generated assembly of libunwind and found the .cantunwind
directives all over the place.
> I would only expect that to be a problem if an
> exception were being propagated through a libunwind function, and that
> shouldn't happen unless something has gone badly wrong.
Can you explain what you mean?
> I tried the
> example with the armv7l release of clang 8.0 which I happened to have
> installed on an Armv8l machine and the program worked. I was able to
> reproduce the problem with the PR with the default Ubuntu16.04 clang
> (3.8) and libc++-dev package.
> 
> I also note that when looking at the link line for the example in the
> PR, clang was linking libgcc_s and not libunwind so I think the
> problem may be somewhere else. There have been quite a lot of fixes
> since clang 3.8 and its libc++ so it may be worth trying a more recent
> clang.
I’m using mainline just-built clang to build libunwind. Actually, I’m
cross-compiling on Windows for Linux.
Also, I’m using compiler-rt instead of libgcc.
And yes, the problem is not with libgcc_s, because, as I’ve said, force-setting
the -funwind-tables flag in libunwind configuration makes the problem go away.

So, the main question remains: when we’re configuring libunwind build, CMake
checks the -funwind-tables flag and that check fails because the
__aeabi_unwind_cpp_pr0 symbol is absent. This symbol should be defined in
libunwind, which is not build yet. I’m having a hard time understanding how can
this be even possible. Can it be that you are not experiencing the problem
because your clang uses libgcc and not compiler-rt?
> 
> Can I ask that you try and narrow down what the problem is, what your
> environment is, and comment on the PR if it helps reproduce? It is
> definitely worth looking at the output of clang -v to see which
> unwinder it is using, by default it will be libgcc_s on Linux.
> 
> Peter
> 
>> However, the CMake check fails with the following error:
>> 
>> ```
>> ld.lld: error: undefined symbol: __aeabi_unwind_cpp_pr0
>> 
>>>>> referenced by src.cxx
>> 
>>>>>             
CMakeFiles/cmTC_e9739.dir/src.cxx.o:(.ARM.exidx.text.main+0x0)
>> 
>> clang++: error: linker command failed with exit code 1 (use -v to see
invocation)
>> 
>> ninja: build stopped: subcommand failed.
>> 
>> Source file was:
>> int main() { return 0; }
>> ```
>> 
>> No wonder! __aeabi_unwind_cpp_pr0 is defined in libunwind itself, which
we haven’t built yet.
>> 
>> If I instead set the -funwind-tables flag unconditionally using
`add_compile_flags(-funwind-tables)` instead of
`add_cxx_compile_flags_if_supported(-funwind-tables)`, everything is fine and
the aforementioned bug is gone.
>> 
>> I’ve found a PR which seemed to address this:
https://reviews.llvm.org/D31858 (cc’ing @phosek, @compnerd and @beanz as
participants of this PR). It mentions that the __aeabi_unwind_cpp_pr0 symbol is
provided by the compiler runtime. However, as I’ve already said, it lives in
libunwind, so the problem doesn’t seem to be solved.
>> 
>> I’m very tempted to just set the -funwind-tables flag unconditionally,
but I’m afraid it’ll break something. What would be the right solution for
building libunwind with this flag for ARM Linux?
>> 
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
<https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20191118/d39ae63d/attachment.html>

On Mon, 18 Nov 2019 at 15:23, Sergej Jaskiewicz <jaskiewiczs at
icloud.com> wrote:
>
> Hi Peter,
>
> Thanks for your response.
>
> On 18 Nov 2019, at 17:44, Peter Smith <peter.smith at linaro.org>
wrote:
>
> On Mon, 18 Nov 2019 at 12:32, Sergej Jaskiewicz via llvm-dev
> <llvm-dev at lists.llvm.org> wrote:
>
>
> There’s this bug: https://bugs.llvm.org/show_bug.cgi?id=38468.
>
> I’ve managed to track it down to a configuration issue. The thing is that
in order for libunwind to be usable on ARM Linux, it should be built with the
-funwind-tables flag. This flag is conditionally set here:
https://github.com/llvm/llvm-project/blob/master/libunwind/CMakeLists.txt#L294,
if the compiler “supports” it.
>
>
> Are you sure that libunwind being built without -funwind-tables is the
> cause of the bug?
>
>
> Yep, pretty sure, I’ve found that the cause of the problem is the
_Unwind_Backtrace function not executing the provided callback. It isn’t doing
so because, since libunwind is compiled without the flag, the information about
the stack frame is lost, so, when _Unwind_Backtrace looks for it, it can’t find
it (since we’ve entered the _Unwind_Backtrace stack frame, which lives in
libunwind, where no unwind info is present).
>
> I’ve looked at the generated assembly of libunwind and found the
.cantunwind directives all over the place.
>
> I would only expect that to be a problem if an
> exception were being propagated through a libunwind function, and that
> shouldn't happen unless something has gone badly wrong.
>
>
> Can you explain what you mean?
>
I'm going from the example provided in the PR:

int main(int argc, char* argv[])
{
    try
    {
        throw 1;
    }
    catch(...)
    {
        std::cout << "Exception is caught\n";
    }
}

When the exception is thrown via a call to __cxa_throw, what I would
expect is for the unwinder to look up the PC value at the callsite in
the .ARM.exidx table. This should contain information for unwinding
the stack should the exception propagate outside of main. This
propagation of the exception out of main shouldn't go through a
function in libunwind as libunwind should not be in the call stack for
main. This is the case if the search in the table starts from the
callsite at main, and in this case it shouldn't matter whether
libunwind has unwind tables or not.

One thing I've not done is go back to the source and check if the
search does not start in main, but instead starts in a function in
libunwind, which via a further unwind gets to main where the exception
is caught. If that were the case then yes we would need unwinding
information. We'd also need unwinding information in libunwind if it
called a function that could throw an exception.
> I tried the
> example with the armv7l release of clang 8.0 which I happened to have
> installed on an Armv8l machine and the program worked. I was able to
> reproduce the problem with the PR with the default Ubuntu16.04 clang
> (3.8) and libc++-dev package.
>
> I also note that when looking at the link line for the example in the
> PR, clang was linking libgcc_s and not libunwind so I think the
> problem may be somewhere else. There have been quite a lot of fixes
> since clang 3.8 and its libc++ so it may be worth trying a more recent
> clang.
>
>
> I’m using mainline just-built clang to build libunwind. Actually, I’m
cross-compiling on Windows for Linux.
> Also, I’m using compiler-rt instead of libgcc.
> And yes, the problem is not with libgcc_s, because, as I’ve said,
force-setting the -funwind-tables flag in libunwind configuration makes the
problem go away.
>
I've just checked a native build on Arm v8l, apologies I've not got a
Windows machine to cross-compile on. In my case I am getting unwind
information generated for libunwind. I'm guessing you are cross
compiling with a standalone build? In the past I have cross-compiled
libunwind, in summary:
build compiler-rt builtins
build libuwind (statically, depending on compiler-rt)
build libcxxabi depending on libunwind and compiler-rt
build libcxx depending on libcxxabi and compiler-rt

Are you trying to build libunwind into a shared library? I don't think
that is a supported configuration due to the cyclical dependency.

Peter
> So, the main question remains: when we’re configuring libunwind build,
CMake checks the -funwind-tables flag and that check fails because the
__aeabi_unwind_cpp_pr0 symbol is absent. This symbol should be defined in
libunwind, which is not build yet. I’m having a hard time understanding how can
this be even possible. Can it be that you are not experiencing the problem
because your clang uses libgcc and not compiler-rt?
>
>
> Can I ask that you try and narrow down what the problem is, what your
> environment is, and comment on the PR if it helps reproduce? It is
> definitely worth looking at the output of clang -v to see which
> unwinder it is using, by default it will be libgcc_s on Linux.
>
>
> Peter
>
>
> However, the CMake check fails with the following error:
>
> ```
> ld.lld: error: undefined symbol: __aeabi_unwind_cpp_pr0
>
> referenced by src.cxx
>
>
>              CMakeFiles/cmTC_e9739.dir/src.cxx.o:(.ARM.exidx.text.main+0x0)
>
>
> clang++: error: linker command failed with exit code 1 (use -v to see
invocation)
>
> ninja: build stopped: subcommand failed.
>
> Source file was:
> int main() { return 0; }
> ```
>
> No wonder! __aeabi_unwind_cpp_pr0 is defined in libunwind itself, which we
haven’t built yet.
>
> If I instead set the -funwind-tables flag unconditionally using
`add_compile_flags(-funwind-tables)` instead of
`add_cxx_compile_flags_if_supported(-funwind-tables)`, everything is fine and
the aforementioned bug is gone.
>
> I’ve found a PR which seemed to address this:
https://reviews.llvm.org/D31858 (cc’ing @phosek, @compnerd and @beanz as
participants of this PR). It mentions that the __aeabi_unwind_cpp_pr0 symbol is
provided by the compiler runtime. However, as I’ve already said, it lives in
libunwind, so the problem doesn’t seem to be solved.
>
> I’m very tempted to just set the -funwind-tables flag unconditionally, but
I’m afraid it’ll break something. What would be the right solution for building
libunwind with this flag for ARM Linux?
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
>
>

> On 18 Nov 2019, at 19:55, Peter Smith <peter.smith at linaro.org>
wrote:
> 
> On Mon, 18 Nov 2019 at 15:23, Sergej Jaskiewicz <jaskiewiczs at
icloud.com <mailto:jaskiewiczs at icloud.com>> wrote:
>> 
>> Hi Peter,
>> 
>> Thanks for your response.
>> 
>> On 18 Nov 2019, at 17:44, Peter Smith <peter.smith at linaro.org>
wrote:
>> 
>> On Mon, 18 Nov 2019 at 12:32, Sergej Jaskiewicz via llvm-dev
>> <llvm-dev at lists.llvm.org> wrote:
>> 
>> 
>> There’s this bug: https://bugs.llvm.org/show_bug.cgi?id=38468.
>> 
>> I’ve managed to track it down to a configuration issue. The thing is
that in order for libunwind to be usable on ARM Linux, it should be built with
the -funwind-tables flag. This flag is conditionally set here:
https://github.com/llvm/llvm-project/blob/master/libunwind/CMakeLists.txt#L294,
if the compiler “supports” it.
>> 
>> 
>> Are you sure that libunwind being built without -funwind-tables is the
>> cause of the bug?
>> 
>> 
>> Yep, pretty sure, I’ve found that the cause of the problem is the
_Unwind_Backtrace function not executing the provided callback. It isn’t doing
so because, since libunwind is compiled without the flag, the information about
the stack frame is lost, so, when _Unwind_Backtrace looks for it, it can’t find
it (since we’ve entered the _Unwind_Backtrace stack frame, which lives in
libunwind, where no unwind info is present).
>> 
>> I’ve looked at the generated assembly of libunwind and found the
.cantunwind directives all over the place.
>> 
>> I would only expect that to be a problem if an
>> exception were being propagated through a libunwind function, and that
>> shouldn't happen unless something has gone badly wrong.
>> 
>> 
>> Can you explain what you mean?
>> 
> I'm going from the example provided in the PR:
> 
> int main(int argc, char* argv[])
> {
>    try
>    {
>        throw 1;
>    }
>    catch(...)
>    {
>        std::cout << "Exception is caught\n";
>    }
> }
> 
> When the exception is thrown via a call to __cxa_throw, what I would
> expect is for the unwinder to look up the PC value at the callsite in
> the .ARM.exidx table. This should contain information for unwinding
> the stack should the exception propagate outside of main. This
> propagation of the exception out of main shouldn't go through a
> function in libunwind as libunwind should not be in the call stack for
> main. This is the case if the search in the table starts from the
> callsite at main, and in this case it shouldn't matter whether
> libunwind has unwind tables or not.
> 
> One thing I've not done is go back to the source and check if the
> search does not start in main, but instead starts in a function in
> libunwind, which via a further unwind gets to main where the exception
> is caught. If that were the case then yes we would need unwinding
> information. We'd also need unwinding information in libunwind if it
> called a function that could throw an exception.
> 
>> I tried the
>> example with the armv7l release of clang 8.0 which I happened to have
>> installed on an Armv8l machine and the program worked. I was able to
>> reproduce the problem with the PR with the default Ubuntu16.04 clang
>> (3.8) and libc++-dev package.
>> 
>> I also note that when looking at the link line for the example in the
>> PR, clang was linking libgcc_s and not libunwind so I think the
>> problem may be somewhere else. There have been quite a lot of fixes
>> since clang 3.8 and its libc++ so it may be worth trying a more recent
>> clang.
>> 
>> 
>> I’m using mainline just-built clang to build libunwind. Actually, I’m
cross-compiling on Windows for Linux.
>> Also, I’m using compiler-rt instead of libgcc.
>> And yes, the problem is not with libgcc_s, because, as I’ve said,
force-setting the -funwind-tables flag in libunwind configuration makes the
problem go away.
>> 
> 
> I've just checked a native build on Arm v8l, apologies I've not got
a
> Windows machine to cross-compile on. In my case I am getting unwind
> information generated for libunwind. I'm guessing you are cross
> compiling with a standalone build?
No, I’m building libunwind as part of LLVM. Can you please provide the options
that you’re using to build libunwind?
> In the past I have cross-compiled
> libunwind, in summary:
> build compiler-rt builtins
> build libuwind (statically, depending on compiler-rt)
> build libcxxabi depending on libunwind and compiler-rt
> build libcxx depending on libcxxabi and compiler-rt
That’s what I’m doing as well.
> 
> Are you trying to build libunwind into a shared library? I don't think
> that is a supported configuration due to the cyclical dependency.
I’m building a static library.

I’m building llvm & clang & lld first using the native compiler (MSVC),
and then the just-built clang acts as a cross-compiler for the runtime libraries
that you’ve mentioned.
> 
> Peter
> 
>> So, the main question remains: when we’re configuring libunwind build,
CMake checks the -funwind-tables flag and that check fails because the
__aeabi_unwind_cpp_pr0 symbol is absent. This symbol should be defined in
libunwind, which is not build yet. I’m having a hard time understanding how can
this be even possible. Can it be that you are not experiencing the problem
because your clang uses libgcc and not compiler-rt?
>> 
>> 
>> Can I ask that you try and narrow down what the problem is, what your
>> environment is, and comment on the PR if it helps reproduce? It is
>> definitely worth looking at the output of clang -v to see which
>> unwinder it is using, by default it will be libgcc_s on Linux.
>> 
>> 
>> Peter
>> 
>> 
>> However, the CMake check fails with the following error:
>> 
>> ```
>> ld.lld: error: undefined symbol: __aeabi_unwind_cpp_pr0
>> 
>> referenced by src.cxx
>> 
>> 
>>            
CMakeFiles/cmTC_e9739.dir/src.cxx.o:(.ARM.exidx.text.main+0x0)
>> 
>> 
>> clang++: error: linker command failed with exit code 1 (use -v to see
invocation)
>> 
>> ninja: build stopped: subcommand failed.
>> 
>> Source file was:
>> int main() { return 0; }
>> ```
>> 
>> No wonder! __aeabi_unwind_cpp_pr0 is defined in libunwind itself, which
we haven’t built yet.
>> 
>> If I instead set the -funwind-tables flag unconditionally using
`add_compile_flags(-funwind-tables)` instead of
`add_cxx_compile_flags_if_supported(-funwind-tables)`, everything is fine and
the aforementioned bug is gone.
>> 
>> I’ve found a PR which seemed to address this:
https://reviews.llvm.org/D31858 (cc’ing @phosek, @compnerd and @beanz as
participants of this PR). It mentions that the __aeabi_unwind_cpp_pr0 symbol is
provided by the compiler runtime. However, as I’ve already said, it lives in
libunwind, so the problem doesn’t seem to be solved.
>> 
>> I’m very tempted to just set the -funwind-tables flag unconditionally,
but I’m afraid it’ll break something. What would be the right solution for
building libunwind with this flag for ARM Linux?
>> 
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20191118/03554980/attachment.html>