llvm dev - Mar 2017 - Use of the C++ standard library in XRay compiler-rt

So I stumbled across an issue that I think is a bit fundamental:

The xray runtime uses the C++ standard library.

This seems like a problem because whatever C++ standard library is used to
compile the XRay runtime may not be the same as the C++ standard library
(if any) that is used to build the target application and link XRay into.

Does this make sense? Is this a problem?

Talking to Chandler over lunch it sounds like there's a couple of options -
either remove the dependency (much like, I believe, the sanitizer runtimes
- use nothing from the C++ standard library, replace everything with custom
data structures, etc) or, perhaps more drastically, change the way the
runtimes are built such that they statically link a private version of,
say, libc++.

Chandler seemed to think maybe we could do this state-side (Tim? Might be
something you could handle) rather than pushing it back on to Dean, if that
sounds reasonable?

(this came up for me due to what's probably a bug in the way compiler-rt is
built - where the lib itself is built with the host compiler but the tests
are built/linked with the just-bulit clang. My host compiler uses libstdc++
6, whereas the just-built clang will use libstdc++ 4.8. So it fails to link
due to this mismatch)
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20170308/986f3472/attachment.html>

On Wed, Mar 8, 2017 at 1:49 PM David Blaikie <dblaikie at gmail.com>
wrote:
> So I stumbled across an issue that I think is a bit fundamental:
>
> The xray runtime uses the C++ standard library.
>
> This seems like a problem because whatever C++ standard library is used to
> compile the XRay runtime may not be the same as the C++ standard library
> (if any) that is used to build the target application and link XRay into.
>
> Does this make sense? Is this a problem?
>
> Talking to Chandler over lunch it sounds like there's a couple of
options
> - either remove the dependency (much like, I believe, the sanitizer
> runtimes - use nothing from the C++ standard library, replace everything
> with custom data structures, etc) or, perhaps more drastically, change the
> way the runtimes are built such that they statically link a private version
> of, say, libc++.
>
What's the reason of not static-linking a C++ standard library for
sanitizer runtimes back to when it was created?

>
> Chandler seemed to think maybe we could do this state-side (Tim? Might be
> something you could handle) rather than pushing it back on to Dean, if that
> sounds reasonable?
>
I believe that "state-side" is LLVM team side? I agree that we should
clean
up the standard library usage even just for consistency.

Searching the xray directory for dependencies:
...compiler-rt/lib/xray % grep '#include <[^>.]*>' -oh `find .
-type f|grep
-v 'tests'` | sort | uniq -c
      1 #include <algorithm>
     10 #include <atomic>
      1 #include <bitset>
      6 #include <cassert>
      1 #include <cerrno>
      1 #include <cstddef>
      7 #include <cstdint>
      2 #include <cstdio>
      1 #include <cstdlib>
      2 #include <cstring>
      1 #include <deque>
      2 #include <iterator>
      2 #include <limits>
      2 #include <memory>
      4 #include <mutex>
      1 #include <system_error>
      1 #include <thread>
      2 #include <tuple>
      1 #include <unordered_map>
      1 #include <unordered_set>
      3 #include <utility>
I think the biggest part is containers, and they are mostly
in ./xray_buffer_queue.h and ./xray_fdr_logging.cc.

dependencies without buffer queue and fdr logging:
...compiler-rt/lib/xray % grep '#include <[^>.]*>' -oh `find .
-type
f|egrep -v 'tests|buffer|fdr'` | sort | uniq -c
      9 #include <atomic>
      4 #include <cassert>
      1 #include <cerrno>
      1 #include <cstddef>
      6 #include <cstdint>
      2 #include <cstdio>
      1 #include <cstring>
      2 #include <iterator>
      2 #include <limits>
      1 #include <memory>
      3 #include <mutex>
      1 #include <thread>
      2 #include <tuple>
      2 #include <utility>
I believe that this is relatively easy to cleanup. I can do that.

I don't know how hard it is to rewrite buffer queue and fdr logging using
compiler_rt infrastructure.

>
> (this came up for me due to what's probably a bug in the way
compiler-rt
> is built - where the lib itself is built with the host compiler but the
> tests are built/linked with the just-bulit clang. My host compiler uses
> libstdc++ 6, whereas the just-built clang will use libstdc++ 4.8. So it
> fails to link due to this mismatch)
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20170308/1fdb24dc/attachment.html>

On Wed, Mar 8, 2017 at 2:28 PM Tim Shen <timshen at google.com> wrote:
> On Wed, Mar 8, 2017 at 1:49 PM David Blaikie <dblaikie at gmail.com>
wrote:
>
> So I stumbled across an issue that I think is a bit fundamental:
>
> The xray runtime uses the C++ standard library.
>
> This seems like a problem because whatever C++ standard library is used to
> compile the XRay runtime may not be the same as the C++ standard library
> (if any) that is used to build the target application and link XRay into.
>
> Does this make sense? Is this a problem?
>
> Talking to Chandler over lunch it sounds like there's a couple of
options
> - either remove the dependency (much like, I believe, the sanitizer
> runtimes - use nothing from the C++ standard library, replace everything
> with custom data structures, etc) or, perhaps more drastically, change the
> way the runtimes are built such that they statically link a private version
> of, say, libc++.
>
>
> What's the reason of not static-linking a C++ standard library for
> sanitizer runtimes back to when it was created?
>
Not sure - Evgeniy (cc'd) might know. Partly perhaps the development cost
of having to isolate that statically linked library from colliding with any
other (some kind of mangling scheme would have to be used, I think? to
avoid such a collision).

>
>
>
> Chandler seemed to think maybe we could do this state-side (Tim? Might be
> something you could handle) rather than pushing it back on to Dean, if that
> sounds reasonable?
>
>
> I believe that "state-side" is LLVM team side?
>
Right, yes, sorry.

> I agree that we should clean up the standard library usage even just for
> consistency.
>
> Searching the xray directory for dependencies:
> ...compiler-rt/lib/xray % grep '#include <[^>.]*>' -oh
`find . -type
> f|grep -v 'tests'` | sort | uniq -c
>       1 #include <algorithm>
>      10 #include <atomic>
>       1 #include <bitset>
>       6 #include <cassert>
>       1 #include <cerrno>
>       1 #include <cstddef>
>       7 #include <cstdint>
>       2 #include <cstdio>
>       1 #include <cstdlib>
>       2 #include <cstring>
>       1 #include <deque>
>       2 #include <iterator>
>       2 #include <limits>
>       2 #include <memory>
>       4 #include <mutex>
>       1 #include <system_error>
>       1 #include <thread>
>       2 #include <tuple>
>       1 #include <unordered_map>
>       1 #include <unordered_set>
>       3 #include <utility>
> I think the biggest part is containers, and they are mostly
> in ./xray_buffer_queue.h and ./xray_fdr_logging.cc.
>
> dependencies without buffer queue and fdr logging:
> ...compiler-rt/lib/xray % grep '#include <[^>.]*>' -oh
`find . -type
> f|egrep -v 'tests|buffer|fdr'` | sort | uniq -c
>       9 #include <atomic>
>       4 #include <cassert>
>       1 #include <cerrno>
>       1 #include <cstddef>
>       6 #include <cstdint>
>       2 #include <cstdio>
>       1 #include <cstring>
>       2 #include <iterator>
>       2 #include <limits>
>       1 #include <memory>
>       3 #include <mutex>
>       1 #include <thread>
>       2 #include <tuple>
>       2 #include <utility>
> I believe that this is relatively easy to cleanup. I can do that.
>
> I don't know how hard it is to rewrite buffer queue and fdr logging
using
> compiler_rt infrastructure.
>
I think buffer_queue's probably sufficiently well bounded that it
shouldn't
be drastically hard to replace it with a custom implementation. Haven't
looked at fdr_logging.

Maps/dictionary-like things might be a bit of a pain in particular. Not
sure if the sanitizers already have some reusable idioms/libraries for that.

I'm also not really clear on where the boundary is - which headers or
language features ('new'?) can be used, and which can't. Can't
say I've
ever tried to make code library agnostic.

>
>
>
> (this came up for me due to what's probably a bug in the way
compiler-rt
> is built - where the lib itself is built with the host compiler but the
> tests are built/linked with the just-bulit clang. My host compiler uses
> libstdc++ 6, whereas the just-built clang will use libstdc++ 4.8. So it
> fails to link due to this mismatch)
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20170308/987e468a/attachment.html>