llvm dev - Aug 2012 - [LLVMdev] RFC: How can AddressSanitizer, ThreadSanitizer, and similar runtime libraries leverage shared library code?

On Thu, Jun 21, 2012 at 1:34 PM, Dmitry Vyukov <dvyukov at google.com>
wrote:
> On Thu, Jun 21, 2012 at 1:30 PM, Chandler Carruth <chandlerc at
google.com>wrote:
>
>>  Can we alter the build system so that when building a run-time library
>>>>>> it modifies all .cpp files like this:
>>>>>>    namespace FOO {
>>>>>>    <file body>
>>>>>>    }
>>>>>> This will give us essentially the same thing, but w/o
system
>>>>>> dependent object file hackery.
>>>>>> Maybe we can add a Clang flag to add such a namespace
for us?
>>>>>>
>>>>>
>>>>> I think this is essentially what Dmitry was talking about
w/ past
>>>>> STLport experience. It has lots of limitations:
>>>>>
>>>>
>>>> Patching object files still sounds much scarier and harder to
port.
>>>> I'd prefer to find a solution that involves only source
files and maybe
>>>> clang.
>>>> Pondering...
>>>>
>>>>
>>>>> - You can't use the normal system standard library
>>>>>
>>>> - You have to build the standard library from source
>>>>> - You can't wrap certain parts of it (operator new,
delete, a few
>>>>> other things)
>>>>> - You can't re-use any C libraries (zlib for example)
>>>>>
>>>>
>>>
>>> Perhaps you are solving a broader problem. But as for asan/tsan, we
>>> currently need only symbolizer, it's separable from everything
else, and
>>> can be made to not use STL.
>>>
>>
>> If you want to share LLVM code for the object and dwarf reading, I do
not
>> believe this to be true at all.
>>
>
> I've already removed code for the object reading for exactly that
reason,
> so now it's just dwarf parsing :) There are some CTL containers
involved,
> but I think they can be replaced.
>
Agree here. I hope to modify/extend this code soon anyway.

-- 
Alexey Samsonov, MSK
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20120621/914e9502/attachment.html>

On Thu, Jun 21, 2012 at 2:39 AM, Alexey Samsonov <samsonov at
google.com>wrote:
>
>
> On Thu, Jun 21, 2012 at 1:34 PM, Dmitry Vyukov <dvyukov at
google.com> wrote:
>
>> On Thu, Jun 21, 2012 at 1:30 PM, Chandler Carruth <chandlerc at
google.com>wrote:
>>
>>>  Can we alter the build system so that when building a run-time
library
>>>>>>> it modifies all .cpp files like this:
>>>>>>>    namespace FOO {
>>>>>>>    <file body>
>>>>>>>    }
>>>>>>> This will give us essentially the same thing, but
w/o system
>>>>>>> dependent object file hackery.
>>>>>>> Maybe we can add a Clang flag to add such a
namespace for us?
>>>>>>>
>>>>>>
>>>>>> I think this is essentially what Dmitry was talking
about w/ past
>>>>>> STLport experience. It has lots of limitations:
>>>>>>
>>>>>
>>>>> Patching object files still sounds much scarier and harder
to port.
>>>>> I'd prefer to find a solution that involves only source
files and
>>>>> maybe clang.
>>>>> Pondering...
>>>>>
>>>>>
>>>>>> - You can't use the normal system standard library
>>>>>>
>>>>> - You have to build the standard library from source
>>>>>> - You can't wrap certain parts of it (operator new,
delete, a few
>>>>>> other things)
>>>>>> - You can't re-use any C libraries (zlib for
example)
>>>>>>
>>>>>
>>>>
>>>> Perhaps you are solving a broader problem. But as for
asan/tsan, we
>>>> currently need only symbolizer, it's separable from
everything else, and
>>>> can be made to not use STL.
>>>>
>>>
>>> If you want to share LLVM code for the object and dwarf reading, I
do
>>> not believe this to be true at all.
>>>
>>
>> I've already removed code for the object reading for exactly that
reason,
>> so now it's just dwarf parsing :) There are some CTL containers
involved,
>> but I think they can be replaced.
>>
>
> Agree here. I hope to modify/extend this code soon anyway.
>
Folks, this is not the path to sharing code. This is the path to forking
code.

Let's go back to the very premise: I think it is highly desirable to be
capable of building runtimes such as ASan and TSan and *share* code rather
than forking it.

I have reasons: I have seen the creation of at least three separate ELF
and/or DWARF parsing libraries thus far. I have seen a long series of bugs
found and fixed in them over the course of years, often the same bug, often
with great expense in debugging to understand why. I don't want us to keep
paying this cost. I don't think these pieces of code are likely to be alone
in this.


Now, perhaps I am wrong, and it is not worth it. Thus far, I don't hear any
convincing arguments to that effect, but I'm very willing to believe I'm
wrong as I don't work on one of these runtimes, and so don't have a
direct
appreciation for all of the costs involved.

But let's be extremely clear on what you are suggesting: you are
specifically doing away with the very idea of sharing code with the rest of
the LLVM project, and instead deciding to fork and write custom code in the
runtime for all functionality.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20120621/82dab42b/attachment.html>

On Thu, Jun 21, 2012 at 1:44 PM, Chandler Carruth <chandlerc at
google.com>wrote:
> Can we alter the build system so that when building a run-time library it
>>>>>>>> modifies all .cpp files like this:
>>>>>>>>    namespace FOO {
>>>>>>>>    <file body>
>>>>>>>>    }
>>>>>>>> This will give us essentially the same thing,
but w/o system
>>>>>>>> dependent object file hackery.
>>>>>>>> Maybe we can add a Clang flag to add such a
namespace for us?
>>>>>>>>
>>>>>>>
>>>>>>> I think this is essentially what Dmitry was talking
about w/ past
>>>>>>> STLport experience. It has lots of limitations:
>>>>>>>
>>>>>>
>>>>>> Patching object files still sounds much scarier and
harder to port.
>>>>>> I'd prefer to find a solution that involves only
source files and
>>>>>> maybe clang.
>>>>>> Pondering...
>>>>>>
>>>>>>
>>>>>>> - You can't use the normal system standard
library
>>>>>>>
>>>>>> - You have to build the standard library from source
>>>>>>> - You can't wrap certain parts of it (operator
new, delete, a few
>>>>>>> other things)
>>>>>>> - You can't re-use any C libraries (zlib for
example)
>>>>>>>
>>>>>>
>>>>>
>>>>> Perhaps you are solving a broader problem. But as for
asan/tsan, we
>>>>> currently need only symbolizer, it's separable from
everything else, and
>>>>> can be made to not use STL.
>>>>>
>>>>
>>>> If you want to share LLVM code for the object and dwarf
reading, I do
>>>> not believe this to be true at all.
>>>>
>>>
>>> I've already removed code for the object reading for exactly
that
>>> reason, so now it's just dwarf parsing :) There are some CTL
containers
>>> involved, but I think they can be replaced.
>>>
>>
>> Agree here. I hope to modify/extend this code soon anyway.
>>
>
> Folks, this is not the path to sharing code. This is the path to forking
> code.
>
> Let's go back to the very premise: I think it is highly desirable to be
> capable of building runtimes such as ASan and TSan and *share* code rather
> than forking it.
>
> I have reasons: I have seen the creation of at least three separate ELF
> and/or DWARF parsing libraries thus far. I have seen a long series of bugs
> found and fixed in them over the course of years, often the same bug, often
> with great expense in debugging to understand why. I don't want us to
keep
> paying this cost. I don't think these pieces of code are likely to be
alone
> in this.
>
>
> Now, perhaps I am wrong, and it is not worth it. Thus far, I don't hear
> any convincing arguments to that effect, but I'm very willing to
believe
> I'm wrong as I don't work on one of these runtimes, and so
don't have a
> direct appreciation for all of the costs involved.
>
> But let's be extremely clear on what you are suggesting: you are
> specifically doing away with the very idea of sharing code with the rest of
> the LLVM project, and instead deciding to fork and write custom code in the
> runtime for all functionality.
>

No, we do not want to fork any code.
My ObjectFile replacement is 20 lines of code including error handling
(open file, get size, mmap).
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20120621/95d3e7a9/attachment.html>

(resurrecting the thread, as much is discussed here already)

Formulating Kostya's suggestion:
What do you think of compiling LLVM sources into ASan/TSan runtime by just
taking the library sources, providing custom
compiler (target) flags *and* a flag "-Dllvm=__sanitizer_llvm"?
Yeah, it's hacky and applicable to LLVM libs, but OTOH we don't plan to
use
smth else for now (in the short-term).
And it's trivial to implement and portable as well :)

See http://codereview.appspot.com/6458066/ (or attached patch).

On Thu, Jun 21, 2012 at 1:44 PM, Chandler Carruth <chandlerc at
google.com>wrote:
> On Thu, Jun 21, 2012 at 2:39 AM, Alexey Samsonov <samsonov at
google.com>wrote:
>
>>
>>
>> On Thu, Jun 21, 2012 at 1:34 PM, Dmitry Vyukov <dvyukov at
google.com>wrote:
>>
>>> On Thu, Jun 21, 2012 at 1:30 PM, Chandler Carruth <chandlerc at
google.com>wrote:
>>>
>>>>  Can we alter the build system so that when building a run-time
>>>>>>>> library it modifies all .cpp files like this:
>>>>>>>>    namespace FOO {
>>>>>>>>    <file body>
>>>>>>>>    }
>>>>>>>> This will give us essentially the same thing,
but w/o system
>>>>>>>> dependent object file hackery.
>>>>>>>> Maybe we can add a Clang flag to add such a
namespace for us?
>>>>>>>>
>>>>>>>
>>>>>>> I think this is essentially what Dmitry was talking
about w/ past
>>>>>>> STLport experience. It has lots of limitations:
>>>>>>>
>>>>>>
>>>>>> Patching object files still sounds much scarier and
harder to port.
>>>>>> I'd prefer to find a solution that involves only
source files and
>>>>>> maybe clang.
>>>>>> Pondering...
>>>>>>
>>>>>>
>>>>>>> - You can't use the normal system standard
library
>>>>>>>
>>>>>> - You have to build the standard library from source
>>>>>>> - You can't wrap certain parts of it (operator
new, delete, a few
>>>>>>> other things)
>>>>>>> - You can't re-use any C libraries (zlib for
example)
>>>>>>>
>>>>>>
>>>>>
>>>>> Perhaps you are solving a broader problem. But as for
asan/tsan, we
>>>>> currently need only symbolizer, it's separable from
everything else, and
>>>>> can be made to not use STL.
>>>>>
>>>>
>>>> If you want to share LLVM code for the object and dwarf
reading, I do
>>>> not believe this to be true at all.
>>>>
>>>
>>> I've already removed code for the object reading for exactly
that
>>> reason, so now it's just dwarf parsing :) There are some CTL
containers
>>> involved, but I think they can be replaced.
>>>
>>
>> Agree here. I hope to modify/extend this code soon anyway.
>>
>
> Folks, this is not the path to sharing code. This is the path to forking
> code.
>
> Let's go back to the very premise: I think it is highly desirable to be
> capable of building runtimes such as ASan and TSan and *share* code rather
> than forking it.
>
> I have reasons: I have seen the creation of at least three separate ELF
> and/or DWARF parsing libraries thus far. I have seen a long series of bugs
> found and fixed in them over the course of years, often the same bug, often
> with great expense in debugging to understand why. I don't want us to
keep
> paying this cost. I don't think these pieces of code are likely to be
alone
> in this.
>
>
> Now, perhaps I am wrong, and it is not worth it. Thus far, I don't hear
> any convincing arguments to that effect, but I'm very willing to
believe
> I'm wrong as I don't work on one of these runtimes, and so
don't have a
> direct appreciation for all of the costs involved.
>
> But let's be extremely clear on what you are suggesting: you are
> specifically doing away with the very idea of sharing code with the rest of
> the LLVM project, and instead deciding to fork and write custom code in the
> runtime for all functionality.
>


-- 
Alexey Samsonov, MSK
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20120813/4e9acdb9/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: issue6458066_4001.diff
Type: application/octet-stream
Size: 2121 bytes
Desc: not available
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20120813/4e9acdb9/attachment.obj>