llvm dev - Jan 2014 - [LLVMdev] Sanitizers libs in Compiler-RT

On 30 January 2014 21:50, Reid Kleckner <rnk at google.com>
wrote:
>
> I don't see any compelling reason to split the sanitizers out today.
>
Clear, next.


I think the sanitizer guys would probably take a patch to CMakeLists.txt
to
> disable the sanitizer RTLs.
>
Right. I'll first make it build and test, than I'll disable via a flag,
since other people can work in parallel to make it work on ARM.


I'm told that ASan supports ARM, but it's maybe not production quality,
so
> I don't think disabling all sanitizers on non-x86 architectures will
work.
>
I hear Android folks use it, this is why I want to enable them by default.
Even if the tests don't mean much, at least there's a seed from where
they
can pick that up and complete the port.

I'm fine on building them on my own, but buildbots and release builds
should not need to, if it's not suitable.

Thanks, that answers all my questions. I'll proceed with my changes.

--renato
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20140130/eb901121/attachment.html>

Renato Golin wrote:
>> I'm told that ASan supports ARM, but it's maybe not production
>> quality, so I don't think disabling all sanitizers on non-x86
>> architectures will work.
>
> I hear Android folks use it, this is why I want to enable them by
> default. Even if the tests don't mean much, at least there's a seed
from
> where they can pick that up and complete the port.
We have been using ASan on ARM for last several months and we haven't 
seen any serious problem in runtime library.
We are using GCC's ASan though.

-Y

On Thu, Jan 30, 2014 at 1:54 PM, Renato Golin <renato.golin at
linaro.org>wrote:
> On 30 January 2014 21:50, Reid Kleckner <rnk at google.com> wrote:
>>
>> I don't see any compelling reason to split the sanitizers out
today.
>>
>
> Clear, next.
>
Just as a side note, I had some thoughts about the organization of this
stuff a while back that I wanted to replay here.

Fundamentally, I feel like we're in particular getting a few things
conflated. Maybe separating them out helps:

- There is the repository "compiler-rt" that holds all of the runtime
libraries that (if desired) need to be shipped along side the compiler.
They're separated so that they can be omitted when they aren't needed,
and
potentially to isolate an unusual desired property: we would really like to
(eventually) build them with the just-built compiler rather than the host
compiler (where possible, clearly this requires the target compiler to be
executable on the host).

- There is the core runtime library. Historically this was called
'compiler-rt' informally, but perhaps better called
'libclang_rt', which
provides the core necessary runtime library facilities to compile C or C++
applications. It's analogous to libgcc but without some of the unwinding
code (as I understand it, there may be details I'm wrong about here or
glossing over, but it's not relevant to the organization of things).

- There are several optional runtime libraries to support specific compiler
/ toolchain features. These include the sanitizers and profiling libraries.

I think all of the libraries here make sense in the same repository because
of the shared concerns of building runtime libraries. For example, it would
be useful to compile them with the just-built-clang when target is
executable on the host, and it would be useful to automatically cross
compile versions of them for as many targets as are possible on the given
host and supported by the host compiler (if used) and the just-built-clang.

However, the organization of the tree is ... very hard to understand.
originally, there was only the one 'libclang_rt' library that had its
generic C99 implementation in 'lib', and architecture-specific assembly
routines when desirable in architecture subdirectories of lib. When we
added new libraries, we put them in subdirectories of lib, making the whole
thing kind of a mess. My suggestion to fix this was to create a "core"
subdirectory of lib to contain the code used for 'libclang_rt'.
"core" is a
terrible name, but i've no better. suggestions welcome there. Then we would
have a more sensible organization of the 'lib' tree.

I also think it might be useful to have a single large test tree (much like
with llvm or clang) that has subdirectories for the various tests rather
than test directories under lib/asan/ and friends, but maybe the sanitizer
folks have objections to that. consistency seems a compelling reason here,
but there might be other compelling concerns.


Now, the build system has always been problematic because the build system
for this tree is *hard* and no one who has worked on it has really had the
time to do an extremely thorough job and finish all aspects. It's a huge
project. Right now, the makefile system does a good job of using the
just-built-clang and can do a limited amount of cross-building runtimes for
other targets. But the makefile system of compiler-rt is also terribly,
terribly complex, doesn't follow the conventions of LLVM's makefiles,
and
is generally painful to maintain and update, so folks have been reluctant
to flesh out its support for new libraries and other new things.

The CMake system is much cleaner in some respects (a bit less opaque to the
folks trying to maintain it), but CMake makes it much harder to use the
just-built-clang, especially for the C++ runtime code. The consequence is
that we've never finished either the cross-building or
just-built-clang-hosting features that are desirable. Any work toward these
would be really awesome to see, but is a huge pile of work. Finally, when I
was originally doing the CMake build for this I didn't understand what
really needed to be done to build and use the core 'libclang_rt'
library,
and so I don't think I got it right. Some folks have sent patches to
improve it, but I suspect it still really needs more work to be a solid
system to use instead of libgcc. So I'm really excited about your emails.
=] Having a more self-contained stack would be a significant improvement.


I think the obvious incremental steps are to disable building any parts of
compiler-rt that don't build cleanly. There should never be a requirement
for you to port an optional runtime library unless you need it. =] I think
having good ports is important, but that should never block progress
getting other thinsg ported and working well.

If it helps to reorganize things, I'm happy to even help there as much as I
can. I agree that the organization isn't great, but I *really* didn't
want
to fight the makefile build system to do the reorganization myself, so its
something that has lingered too long.


Sorry for the long ramble, but hopefully this gives you some of tho context.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20140131/1505c51a/attachment.html>

On 31 Jan 2014, at 08:12, Chandler Carruth <chandlerc at google.com>
wrote:
> - There is the core runtime library. Historically this was called
'compiler-rt' informally, but perhaps better called
'libclang_rt', which provides the core necessary runtime library
facilities to compile C or C++ applications. It's analogous to libgcc but
without some of the unwinding code (as I understand it, there may be details
I'm wrong about here or glossing over, but it's not relevant to the
organization of things).
For some reason, the (generic, language-agnostic) unwind code is in libcxxabi. 
There was some discussion about moving it into the compiler-rt repository, where
it would make sense.  No one objected, but I'd rather not move it without a
'yes' from someone who is actually working on the code currently
(I'd like to start factoring out the #ifdefs into a cleaner platform /
architecture layer).

On FreeBSD, the 'libgcc' that we ship is now mostly code from the
compiler-rt repository and it would be nice to have libgcc_s from the same
source.

David

Chandler,

Thanks for the great overview.

On Fri, Jan 31, 2014 at 12:12 PM, Chandler Carruth <chandlerc at
google.com>wrote:
>
> On Thu, Jan 30, 2014 at 1:54 PM, Renato Golin <renato.golin at
linaro.org>wrote:
>
>> On 30 January 2014 21:50, Reid Kleckner <rnk at google.com>
wrote:
>>>
>>> I don't see any compelling reason to split the sanitizers out
today.
>>>
>>
>> Clear, next.
>>
>
> Just as a side note, I had some thoughts about the organization of this
> stuff a while back that I wanted to replay here.
>
> Fundamentally, I feel like we're in particular getting a few things
> conflated. Maybe separating them out helps:
>
> - There is the repository "compiler-rt" that holds all of the
runtime
> libraries that (if desired) need to be shipped along side the compiler.
> They're separated so that they can be omitted when they aren't
needed, and
> potentially to isolate an unusual desired property: we would really like to
> (eventually) build them with the just-built compiler rather than the host
> compiler (where possible, clearly this requires the target compiler to be
> executable on the host).
>
> - There is the core runtime library. Historically this was called
> 'compiler-rt' informally, but perhaps better called
'libclang_rt', which
> provides the core necessary runtime library facilities to compile C or C++
> applications. It's analogous to libgcc but without some of the
unwinding
> code (as I understand it, there may be details I'm wrong about here or
> glossing over, but it's not relevant to the organization of things).
>
> - There are several optional runtime libraries to support specific
> compiler / toolchain features. These include the sanitizers and profiling
> libraries.
>
> I think all of the libraries here make sense in the same repository
> because of the shared concerns of building runtime libraries. For example,
> it would be useful to compile them with the just-built-clang when target is
> executable on the host, and it would be useful to automatically cross
> compile versions of them for as many targets as are possible on the given
> host and supported by the host compiler (if used) and the just-built-clang.
>
> However, the organization of the tree is ... very hard to understand.
> originally, there was only the one 'libclang_rt' library that had
its
> generic C99 implementation in 'lib', and architecture-specific
assembly
> routines when desirable in architecture subdirectories of lib. When we
> added new libraries, we put them in subdirectories of lib, making the whole
> thing kind of a mess. My suggestion to fix this was to create a
"core"
> subdirectory of lib to contain the code used for 'libclang_rt'.
"core" is a
> terrible name, but i've no better. suggestions welcome there. Then we
would
> have a more sensible organization of the 'lib' tree.
>
I can't suggest a better name than "core" (maybe,
"runtime"?), but would
*love* to put 'libclang_rt' into a separate directory under
/compiler-rt/lib :) We should've done it long ago, IMO.
>
> I also think it might be useful to have a single large test tree (much
> like with llvm or clang) that has subdirectories for the various tests
> rather than test directories under lib/asan/ and friends, but maybe the
> sanitizer folks have objections to that. consistency seems a compelling
> reason here, but there might be other compelling concerns.
>
Yes, I don't like the way testing is organized either. Originally it made
sense to put sanitizer lit tests under lib/xsan/, but now we've got a lot
of sanitizers, and a lot of duplicating configuration code. As Renato
mentioned, browsing
through lit configs is a bit painful. I would like to proceed with
re-organizing the tests for sanitizers, unless anyone objects. Kostya?

> Now, the build system has always been problematic because the build system
> for this tree is *hard* and no one who has worked on it has really had the
> time to do an extremely thorough job and finish all aspects. It's a
huge
> project. Right now, the makefile system does a good job of using the
> just-built-clang and can do a limited amount of cross-building runtimes for
> other targets. But the makefile system of compiler-rt is also terribly,
> terribly complex, doesn't follow the conventions of LLVM's
makefiles, and
> is generally painful to maintain and update, so folks have been reluctant
> to flesh out its support for new libraries and other new things.
>
Note that it would be really hard to add support for running test suites
into Makefile build system. It's also good for the one-time build only, not
for continuous development - I don't think it respects dependencies
correctly.

>
> The CMake system is much cleaner in some respects (a bit less opaque to
> the folks trying to maintain it), but CMake makes it much harder to use the
> just-built-clang, especially for the C++ runtime code. The consequence is
> that we've never finished either the cross-building or
> just-built-clang-hosting features that are desirable.
>
I once tried to implement a pseudo-build-system for compiler-rt on top of
CMake, so that we use "just-built" Clang instead of a host compiler,
but
failed miserably. Maybe I was doing wrong things, but I got the impression
that CMake isn't suited for switching the compiler on the fly :)

> Any work toward these would be really awesome to see, but is a huge pile
> of work. Finally, when I was originally doing the CMake build for this I
> didn't understand what really needed to be done to build and use the
core
> 'libclang_rt' library, and so I don't think I got it right.
Some folks have
> sent patches to improve it, but I suspect it still really needs more work
> to be a solid system to use instead of libgcc. So I'm really excited
about
> your emails. =] Having a more self-contained stack would be a significant
> improvement.
>
>
> I think the obvious incremental steps are to disable building any parts of
> compiler-rt that don't build cleanly.
>
As I mentioned above, currently in CMake build system we try to avoid
building anything if we're not sure we can produce a working and correct
library on the host platform. That is, I still don't see what the problem
is - it's relatively easy to enable building just the compiler-rt library
on ARM and not enable building sanitizers on ARM.

> There should never be a requirement for you to port an optional runtime
> library unless you need it. =] I think having good ports is important, but
> that should never block progress getting other thinsg ported and working
> well.
>
> If it helps to reorganize things, I'm happy to even help there as much
as
> I can. I agree that the organization isn't great, but I *really*
didn't
> want to fight the makefile build system to do the reorganization myself, so
> its something that has lingered too long.
>
>
> Sorry for the long ramble, but hopefully this gives you some of tho
> context.
>
> _______________________________________________
> cfe-dev mailing list
> cfe-dev at cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
>
>

-- 
Alexey Samsonov, MSK
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20140131/c987d752/attachment.html>