Evgeniy Stepanov
2013-May-07 09:24 UTC
[LLVMdev] RFC: Using zlib to decompress debug info sections.
This might be a bit late, but I've got another argument for bundling zlib source with LLVM. Sanitizer tools need to symbolize stack traces in the reports. We've been using standalone symbolizer binary until now; sanitizer runtime spawns a new process as soon as an error is found, and communicates with it over a pipe. This is very cumbersome to deploy, because we need to keep another binary around, specify a path to it at runtime, etc. LLVM lit.cfg already carries some of this burden. A much better solution would be to statically link symbolization code into the user application, the same as sanitizer runtime library. Unfortunately, symbolizer depends on several LLVM libraries, C++ runtime, zlib, etc. Statically linking all that stuff with user code results in symbol name conflicts. We've come up with what seems to be a perfect solution (thanks to a Chandler's advice at the recent developer meeting). We build everything down to (but not including) libc into LLVM bitcode. This includes LLVMSupport, LLVMObject, LLVMDebugInfo, libc++, libc++abi, zlib (!). Then we bundle it all together and internalize all non-interface symbols: llvm-link && opt -internalize. Then compile down to a single object file. This results in a perfect isolation of symbolizer internals. One drawback is that this requires source for all the things that I mentioned - and at the moment we've got everything but zlib. We'd like this to be a part of the normal LLVM build, but that requires zlib source available somewhere. We could add a cmake/configure option to point to an externally available source, but that sounds like a complication we would like to avoid. WDYT? On Wed, Apr 17, 2013 at 5:02 PM, Alexey Samsonov <samsonov at google.com> wrote:> > On Wed, Apr 17, 2013 at 12:37 AM, Chris Lattner <clattner at apple.com> wrote: >> >> On Apr 16, 2013, at 11:53 AM, Eric Christopher <echristo at gmail.com> wrote: >> > Historically we've done the former. The latter would require Chris >> > wanting to do that. >> >> This case isn't so clearcut. We like to include libraries in the source >> to make it easy to get up and running without having to install a ton of >> dependencies. However, this has license implications and is generally >> annoying. > > > Looks like zlib license is good enough to avoid implications, but I can't > really judge. >> >> >> Given that zlib is so widely available by default, and that the compiler >> can generate correct (albeit uncompressed) debug info, I think the best >> thing is to *not* include a copy in llvm. Just detect and use it if we can >> find it, but otherwise generate uncompressed output. > > > Sure, I'll go this way then. Thanks! > > -- > Alexey Samsonov, MSK > > _______________________________________________ > LLVM Developers mailing list > LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev >
Chandler Carruth
2013-May-07 09:26 UTC
[LLVMdev] RFC: Using zlib to decompress debug info sections.
Lemme chat with Danny off list about the best way to do this, and I'll post an update. On Tue, May 7, 2013 at 11:24 AM, Evgeniy Stepanov <eugeni.stepanov at gmail.com> wrote:> This might be a bit late, but I've got another argument for bundling > zlib source with LLVM. > > Sanitizer tools need to symbolize stack traces in the reports. We've > been using standalone symbolizer binary until now; sanitizer runtime > spawns a new process as soon as an error is found, and communicates > with it over a pipe. This is very cumbersome to deploy, because we > need to keep another binary around, specify a path to it at runtime, > etc. LLVM lit.cfg already carries some of this burden. > > A much better solution would be to statically link symbolization code > into the user application, the same as sanitizer runtime library. > Unfortunately, symbolizer depends on several LLVM libraries, C++ > runtime, zlib, etc. Statically linking all that stuff with user code > results in symbol name conflicts. > > We've come up with what seems to be a perfect solution (thanks to a > Chandler's advice at the recent developer meeting). We build > everything down to (but not including) libc into LLVM bitcode. This > includes LLVMSupport, LLVMObject, LLVMDebugInfo, libc++, libc++abi, > zlib (!). Then we bundle it all together and internalize all > non-interface symbols: llvm-link && opt -internalize. Then compile > down to a single object file. > > This results in a perfect isolation of symbolizer internals. One > drawback is that this requires source for all the things that I > mentioned - and at the moment we've got everything but zlib. > > We'd like this to be a part of the normal LLVM build, but that > requires zlib source available somewhere. We could add a > cmake/configure option to point to an externally available source, but > that sounds like a complication we would like to avoid. > > WDYT? > > > On Wed, Apr 17, 2013 at 5:02 PM, Alexey Samsonov <samsonov at google.com> > wrote: > > > > On Wed, Apr 17, 2013 at 12:37 AM, Chris Lattner <clattner at apple.com> > wrote: > >> > >> On Apr 16, 2013, at 11:53 AM, Eric Christopher <echristo at gmail.com> > wrote: > >> > Historically we've done the former. The latter would require Chris > >> > wanting to do that. > >> > >> This case isn't so clearcut. We like to include libraries in the source > >> to make it easy to get up and running without having to install a ton of > >> dependencies. However, this has license implications and is generally > >> annoying. > > > > > > Looks like zlib license is good enough to avoid implications, but I can't > > really judge. > >> > >> > >> Given that zlib is so widely available by default, and that the compiler > >> can generate correct (albeit uncompressed) debug info, I think the best > >> thing is to *not* include a copy in llvm. Just detect and use it if we > can > >> find it, but otherwise generate uncompressed output. > > > > > > Sure, I'll go this way then. Thanks! > > > > -- > > Alexey Samsonov, MSK > > > > _______________________________________________ > > LLVM Developers mailing list > > LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu > > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev > > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20130507/68da2c38/attachment.html>
James Courtier-Dutton
2013-May-07 10:10 UTC
[LLVMdev] RFC: Using zlib to decompress debug info sections.
On May 7, 2013 10:27 AM, "Evgeniy Stepanov" <eugeni.stepanov at gmail.com> wrote:> > This might be a bit late, but I've got another argument for bundling > zlib source with LLVM. > > Sanitizer tools need to symbolize stack traces in the reports. We've > been using standalone symbolizer binary until now; sanitizer runtime > spawns a new process as soon as an error is found, and communicates > with it over a pipe. This is very cumbersome to deploy, because we > need to keep another binary around, specify a path to it at runtime, > etc. LLVM lit.cfg already carries some of this burden. > > A much better solution would be to statically link symbolization code > into the user application, the same as sanitizer runtime library. > Unfortunately, symbolizer depends on several LLVM libraries, C++ > runtime, zlib, etc. Statically linking all that stuff with user code > results in symbol name conflicts. > > We've come up with what seems to be a perfect solution (thanks to a > Chandler's advice at the recent developer meeting). We build > everything down to (but not including) libc into LLVM bitcode. This > includes LLVMSupport, LLVMObject, LLVMDebugInfo, libc++, libc++abi, > zlib (!). Then we bundle it all together and internalize all > non-interface symbols: llvm-link && opt -internalize. Then compile > down to a single object file. > > This results in a perfect isolation of symbolizer internals. One > drawback is that this requires source for all the things that I > mentioned - and at the moment we've got everything but zlib. > > We'd like this to be a part of the normal LLVM build, but that > requires zlib source available somewhere. We could add a > cmake/configure option to point to an externally available source, but > that sounds like a complication we would like to avoid. > > WDYT? > > > On Wed, Apr 17, 2013 at 5:02 PM, Alexey Samsonov <samsonov at google.com>wrote:> > > > On Wed, Apr 17, 2013 at 12:37 AM, Chris Lattner <clattner at apple.com>wrote:> >> > >> On Apr 16, 2013, at 11:53 AM, Eric Christopher <echristo at gmail.com>wrote:> >> > Historically we've done the former. The latter would require Chris > >> > wanting to do that. > >> > >> This case isn't so clearcut. We like to include libraries in thesource> >> to make it easy to get up and running without having to install a tonof> >> dependencies. However, this has license implications and is generally > >> annoying. > > > > > > Looks like zlib license is good enough to avoid implications, but Ican't> > really judge. > >> > >> > >> Given that zlib is so widely available by default, and that thecompiler> >> can generate correct (albeit uncompressed) debug info, I think the best > >> thing is to *not* include a copy in llvm. Just detect and use it ifwe can> >> find it, but otherwise generate uncompressed output. > > > > > > Sure, I'll go this way then. Thanks! > > > > --It is possible to do both. Include an internal one and also link to and external one, and make it a compile time option which to use. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20130507/1683a105/attachment.html>
Reid Kleckner
2013-May-07 13:06 UTC
[LLVMdev] RFC: Using zlib to decompress debug info sections.
You shouldn't need to use bitcode and opt -internalize to hide the symbols. You can do it with objcopy --localize-hidden like we did for DynamoRIO, but I assume you prefer this route because it ports nicely to Mac. :) On Tue, May 7, 2013 at 5:24 AM, Evgeniy Stepanov <eugeni.stepanov at gmail.com> wrote:> This might be a bit late, but I've got another argument for bundling > zlib source with LLVM. > > Sanitizer tools need to symbolize stack traces in the reports. We've > been using standalone symbolizer binary until now; sanitizer runtime > spawns a new process as soon as an error is found, and communicates > with it over a pipe. This is very cumbersome to deploy, because we > need to keep another binary around, specify a path to it at runtime, > etc. LLVM lit.cfg already carries some of this burden. > > A much better solution would be to statically link symbolization code > into the user application, the same as sanitizer runtime library. > Unfortunately, symbolizer depends on several LLVM libraries, C++ > runtime, zlib, etc. Statically linking all that stuff with user code > results in symbol name conflicts. > > We've come up with what seems to be a perfect solution (thanks to a > Chandler's advice at the recent developer meeting). We build > everything down to (but not including) libc into LLVM bitcode. This > includes LLVMSupport, LLVMObject, LLVMDebugInfo, libc++, libc++abi, > zlib (!). Then we bundle it all together and internalize all > non-interface symbols: llvm-link && opt -internalize. Then compile > down to a single object file. > > This results in a perfect isolation of symbolizer internals. One > drawback is that this requires source for all the things that I > mentioned - and at the moment we've got everything but zlib. > > We'd like this to be a part of the normal LLVM build, but that > requires zlib source available somewhere. We could add a > cmake/configure option to point to an externally available source, but > that sounds like a complication we would like to avoid. > > WDYT? > > > On Wed, Apr 17, 2013 at 5:02 PM, Alexey Samsonov <samsonov at google.com> wrote: >> >> On Wed, Apr 17, 2013 at 12:37 AM, Chris Lattner <clattner at apple.com> wrote: >>> >>> On Apr 16, 2013, at 11:53 AM, Eric Christopher <echristo at gmail.com> wrote: >>> > Historically we've done the former. The latter would require Chris >>> > wanting to do that. >>> >>> This case isn't so clearcut. We like to include libraries in the source >>> to make it easy to get up and running without having to install a ton of >>> dependencies. However, this has license implications and is generally >>> annoying. >> >> >> Looks like zlib license is good enough to avoid implications, but I can't >> really judge. >>> >>> >>> Given that zlib is so widely available by default, and that the compiler >>> can generate correct (albeit uncompressed) debug info, I think the best >>> thing is to *not* include a copy in llvm. Just detect and use it if we can >>> find it, but otherwise generate uncompressed output. >> >> >> Sure, I'll go this way then. Thanks! >> >> -- >> Alexey Samsonov, MSK >> >> _______________________________________________ >> LLVM Developers mailing list >> LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu >> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev >> > _______________________________________________ > LLVM Developers mailing list > LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Evgeniy Stepanov
2013-May-07 14:50 UTC
[LLVMdev] RFC: Using zlib to decompress debug info sections.
Portability is always good. But objdump method does not seem to work well when there is code we don't fully control. Hidden visibility is overridable, and there is enough cases of that in libcxx and libcxxabi to cause problems. Entire exception interface, for example. On Tue, May 7, 2013 at 5:06 PM, Reid Kleckner <rnk at google.com> wrote:> You shouldn't need to use bitcode and opt -internalize to hide the > symbols. You can do it with objcopy --localize-hidden like we did for > DynamoRIO, but I assume you prefer this route because it ports nicely > to Mac. :) > > On Tue, May 7, 2013 at 5:24 AM, Evgeniy Stepanov > <eugeni.stepanov at gmail.com> wrote: >> This might be a bit late, but I've got another argument for bundling >> zlib source with LLVM. >> >> Sanitizer tools need to symbolize stack traces in the reports. We've >> been using standalone symbolizer binary until now; sanitizer runtime >> spawns a new process as soon as an error is found, and communicates >> with it over a pipe. This is very cumbersome to deploy, because we >> need to keep another binary around, specify a path to it at runtime, >> etc. LLVM lit.cfg already carries some of this burden. >> >> A much better solution would be to statically link symbolization code >> into the user application, the same as sanitizer runtime library. >> Unfortunately, symbolizer depends on several LLVM libraries, C++ >> runtime, zlib, etc. Statically linking all that stuff with user code >> results in symbol name conflicts. >> >> We've come up with what seems to be a perfect solution (thanks to a >> Chandler's advice at the recent developer meeting). We build >> everything down to (but not including) libc into LLVM bitcode. This >> includes LLVMSupport, LLVMObject, LLVMDebugInfo, libc++, libc++abi, >> zlib (!). Then we bundle it all together and internalize all >> non-interface symbols: llvm-link && opt -internalize. Then compile >> down to a single object file. >> >> This results in a perfect isolation of symbolizer internals. One >> drawback is that this requires source for all the things that I >> mentioned - and at the moment we've got everything but zlib. >> >> We'd like this to be a part of the normal LLVM build, but that >> requires zlib source available somewhere. We could add a >> cmake/configure option to point to an externally available source, but >> that sounds like a complication we would like to avoid. >> >> WDYT? >> >> >> On Wed, Apr 17, 2013 at 5:02 PM, Alexey Samsonov <samsonov at google.com> wrote: >>> >>> On Wed, Apr 17, 2013 at 12:37 AM, Chris Lattner <clattner at apple.com> wrote: >>>> >>>> On Apr 16, 2013, at 11:53 AM, Eric Christopher <echristo at gmail.com> wrote: >>>> > Historically we've done the former. The latter would require Chris >>>> > wanting to do that. >>>> >>>> This case isn't so clearcut. We like to include libraries in the source >>>> to make it easy to get up and running without having to install a ton of >>>> dependencies. However, this has license implications and is generally >>>> annoying. >>> >>> >>> Looks like zlib license is good enough to avoid implications, but I can't >>> really judge. >>>> >>>> >>>> Given that zlib is so widely available by default, and that the compiler >>>> can generate correct (albeit uncompressed) debug info, I think the best >>>> thing is to *not* include a copy in llvm. Just detect and use it if we can >>>> find it, but otherwise generate uncompressed output. >>> >>> >>> Sure, I'll go this way then. Thanks! >>> >>> -- >>> Alexey Samsonov, MSK >>> >>> _______________________________________________ >>> LLVM Developers mailing list >>> LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu >>> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev >>> >> _______________________________________________ >> LLVM Developers mailing list >> LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu >> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Maybe Matching Threads
- [LLVMdev] RFC: Using zlib to decompress debug info sections.
- [LLVMdev] RFC: Using zlib to decompress debug info sections.
- [LLVMdev] RFC: Using zlib to decompress debug info sections.
- [LLVMdev] RFC: Using zlib to decompress debug info sections.
- [LLVMdev] RFC: Using zlib to decompress debug info sections.