Matthias Braun
2015-Jul-02 22:50 UTC
[LLVMdev] Load MachineFunctionPass plugin from library in llc?
> On Jul 2, 2015, at 2:05 PM, Jim Grosbach <grosbach at apple.com> wrote: > >> >> On Jul 2, 2015, at 1:17 PM, Ethan J. Johnson <ejohns48 at cs.rochester.edu <mailto:ejohns48 at cs.rochester.edu>> wrote: >> >> Hi all, >> >> I am working on creating a MachineFunctionPass to perform an analysis on X86 code. After a bit of trouble, I was able to get my pass compiling and running correctly in llc. However, since a machine pass is compiled directly into the code generator, rerunning “make” across the LLVM build tree involves re-linking most of the major libraries and executables. This is a rather intense process, both in terms of CPU usage and memory (some of the bigger binaries, especially clang, use as much as 5 GB peak memory during linking). With a powerful multicore machine and plenty of RAM, this is doable, but still very time-consuming, and seems wastefully repetitive. >> >> For a front-end pass, I can load the pass from a dynamic shared library using “opt –l path_to_library”; thus I only need to rebuild the pass itself to test a code change. There does not appear to be such an option in llc for dynamically linking a machine pass as a plugin. Does anyone know of a way to achieve something similar? > > There is unfortunately no way to do that currently. > >> >> At the very least, it would be nice to be able to just re-compile the parts of the LLVM tree I need (perhaps just llc). It seems particularly odd that clang, which is a front-end, would need to be relinked for a code change in a back-end pass. (clang, and some of its related components, are responsible for the lion’s share of the time-consuming CPU/memory-intensive linking phase.) >> > > clang the project is the front end, but clang the executable program is the full compiler, including all of optimization, codegen, object emission, etc.. > >> FYI, I am using the Autotools (./configure, make) process to build LLVM, but have also successfully built with CMake and would be willing to try suggestions involving either. > > This, however, can be done, at least w/ CMake+ninja (probably other permutations, too, but that’s the one I use these days). Just do “ninja llc” and it will build just what’s required for llc. Same for any of the other tools.Also enabling shared libraries "cmake -DBUILD_SHARED_LIBS=ON" reduces link times dramatically in my experience. - Matthias -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20150702/963b421b/attachment.html>
Ethan J. Johnson
2015-Jul-03 03:15 UTC
[LLVMdev] Load MachineFunctionPass plugin from library in llc?
From: Matthias Braun [mailto:mbraun at apple.com] Sent: Thursday, July 2, 2015 6:51 PM To: Jim Grosbach Cc: Ethan J. Johnson; llvmdev at cs.uiuc.edu Subject: Re: [LLVMdev] Load MachineFunctionPass plugin from library in llc? On Jul 2, 2015, at 2:05 PM, Jim Grosbach <grosbach at apple.com <mailto:grosbach at apple.com> > wrote: On Jul 2, 2015, at 1:17 PM, Ethan J. Johnson <ejohns48 at cs.rochester.edu <mailto:ejohns48 at cs.rochester.edu> > wrote: Hi all, I am working on creating a MachineFunctionPass to perform an analysis on X86 code. After a bit of trouble, I was able to get my pass compiling and running correctly in llc. However, since a machine pass is compiled directly into the code generator, rerunning “make” across the LLVM build tree involves re-linking most of the major libraries and executables. This is a rather intense process, both in terms of CPU usage and memory (some of the bigger binaries, especially clang, use as much as 5 GB peak memory during linking). With a powerful multicore machine and plenty of RAM, this is doable, but still very time-consuming, and seems wastefully repetitive. For a front-end pass, I can load the pass from a dynamic shared library using “opt –l path_to_library”; thus I only need to rebuild the pass itself to test a code change. There does not appear to be such an option in llc for dynamically linking a machine pass as a plugin. Does anyone know of a way to achieve something similar? There is unfortunately no way to do that currently. At the very least, it would be nice to be able to just re-compile the parts of the LLVM tree I need (perhaps just llc). It seems particularly odd that clang, which is a front-end, would need to be relinked for a code change in a back-end pass. (clang, and some of its related components, are responsible for the lion’s share of the time-consuming CPU/memory-intensive linking phase.) clang the project is the front end, but clang the executable program is the full compiler, including all of optimization, codegen, object emission, etc.. FYI, I am using the Autotools (./configure, make) process to build LLVM, but have also successfully built with CMake and would be willing to try suggestions involving either. This, however, can be done, at least w/ CMake+ninja (probably other permutations, too, but that’s the one I use these days). Just do “ninja llc” and it will build just what’s required for llc. Same for any of the other tools. Also enabling shared libraries "cmake -DBUILD_SHARED_LIBS=ON" reduces link times dramatically in my experience. - Matthias Thanks for the tips! I will definitely try building just llc, since that is all I need for testing/debugging the pass. Out of curiosity, what exactly does BUILD_SHARED_LIBS do? Does it move most of the shared code to .so files and create binaries that load them at run time? If I go that route, is there anything special I need to do to make sure the binaries can “find” the shared libraries at run time? (I do not have root access on my development machine, so “make install” to put the shared libs in a standard location isn’t a really good option.) Thanks, Ethan Johnson -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20150702/e16290cc/attachment.html>
Jim Grosbach
2015-Jul-06 17:18 UTC
[LLVMdev] Load MachineFunctionPass plugin from library in llc?
> On Jul 2, 2015, at 8:15 PM, Ethan J. Johnson <ejohns48 at cs.rochester.edu> wrote: > > From: Matthias Braun [mailto:mbraun at apple.com] > Sent: Thursday, July 2, 2015 6:51 PM > To: Jim Grosbach > Cc: Ethan J. Johnson; llvmdev at cs.uiuc.edu > Subject: Re: [LLVMdev] Load MachineFunctionPass plugin from library in llc? > > >> On Jul 2, 2015, at 2:05 PM, Jim Grosbach <grosbach at apple.com <mailto:grosbach at apple.com>> wrote: >> >>> >>> On Jul 2, 2015, at 1:17 PM, Ethan J. Johnson <ejohns48 at cs.rochester.edu <mailto:ejohns48 at cs.rochester.edu>> wrote: >>> >>> Hi all, >>> >>> I am working on creating a MachineFunctionPass to perform an analysis on X86 code. After a bit of trouble, I was able to get my pass compiling and running correctly in llc. However, since a machine pass is compiled directly into the code generator, rerunning “make” across the LLVM build tree involves re-linking most of the major libraries and executables. This is a rather intense process, both in terms of CPU usage and memory (some of the bigger binaries, especially clang, use as much as 5 GB peak memory during linking). With a powerful multicore machine and plenty of RAM, this is doable, but still very time-consuming, and seems wastefully repetitive. >>> >>> For a front-end pass, I can load the pass from a dynamic shared library using “opt –l path_to_library”; thus I only need to rebuild the pass itself to test a code change. There does not appear to be such an option in llc for dynamically linking a machine pass as a plugin. Does anyone know of a way to achieve something similar? >> >> There is unfortunately no way to do that currently. >> >> >>> >>> At the very least, it would be nice to be able to just re-compile the parts of the LLVM tree I need (perhaps just llc). It seems particularly odd that clang, which is a front-end, would need to be relinked for a code change in a back-end pass. (clang, and some of its related components, are responsible for the lion’s share of the time-consuming CPU/memory-intensive linking phase.) >>> >> >> clang the project is the front end, but clang the executable program is the full compiler, including all of optimization, codegen, object emission, etc.. >> >> >>> FYI, I am using the Autotools (./configure, make) process to build LLVM, but have also successfully built with CMake and would be willing to try suggestions involving either. >> >> This, however, can be done, at least w/ CMake+ninja (probably other permutations, too, but that’s the one I use these days). Just do “ninja llc” and it will build just what’s required for llc. Same for any of the other tools. > > Also enabling shared libraries "cmake -DBUILD_SHARED_LIBS=ON" reduces link times dramatically in my experience. > > - Matthias > > Thanks for the tips! I will definitely try building just llc, since that is all I need for testing/debugging the pass. > > Out of curiosity, what exactly does BUILD_SHARED_LIBS do? Does it move most of the shared code to .so files and create binaries that load them at run time? If I go that route, is there anything special I need to do to make sure the binaries can “find” the shared libraries at run time? (I do not have root access on my development machine, so “make install” to put the shared libs in a standard location isn’t a really good option.)Yep, that’s exactly what it does. The load paths are relative to the main executable, so everything should “just work” without having to do any path munging or installation tricks. -Jim> > Thanks, > Ethan Johnson-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20150706/f55e60d5/attachment.html>