llvm dev - Sep 2011 - [LLVMdev] running a module pass via opt on multiple bitcode files

Thanks John. Your approach worked fine for my small toy program. Now, I would
like to run my module pass on a huge project (still a single executable)
consisting of a few thousand bit code files scattered in many different
directories. And some of them are static libraries. Do you still think it's
a good idea to manually gather and link them with llvm-ld? I've seen other
module pass implementations where they directly patch a specific version of LLVM
or clang, instead of running a shared object implementation with opt. Which
would be the ideal approach to analyze a program with a module pass? Thanks.

From: John Criswell <criswell at illinois.edu<mailto:criswell at
illinois.edu>>
Organization: University of Illinois
Date: Tue, 12 Jul 2011 16:19:06 -0500
To: Jinwook Shin <Jinwook.Shin at microsoft.com<mailto:Jinwook.Shin at
microsoft.com>>
Cc: "llvmdev at cs.uiuc.edu<mailto:llvmdev at cs.uiuc.edu>"
<llvmdev at cs.uiuc.edu<mailto:llvmdev at cs.uiuc.edu>>
Subject: Re: [LLVMdev] running a module pass via opt on multiple bitcode files

On 7/12/11 4:11 PM, Jinwook Shin (HOTWING) wrote:
Hi,

I'm trying to implement a module pass as a share object to process an entire
program as a unit. The target program is built from multiple object files, hence
multiple bitcode files. However, it seems that opt does not take the whole
program, but just one bitcode file, "test.bc" in the example run shown
below.

$ opt -load mypass.dylib -mymodulepass < test.bc > /dev/null

How do I run my module pass in a library on a whole program?

Link the bitcode files together into a single bitcode file with llvm-ld.  You
can then run the single bitcode file through opt.

-- John T.



Thanks.







_______________________________________________

LLVM Developers mailing list

LLVMdev at cs.uiuc.edu<mailto:LLVMdev at cs.uiuc.edu>        
http://llvm.cs.uiuc.eduhttp://lists.cs.uiuc.edu/mailman/listinfo/llvmdev

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20110819/0b167c23/attachment.html>

On 8/19/11 4:36 PM, Jinwook Shin (HOTWING) wrote:
>
> Thanks John. Your approach worked fine for my small toy program. Now, 
> I would like to run my module pass on a huge project (still a single 
> executable) consisting of a few thousand bit code files scattered in 
> many different directories. And some of them are static libraries. Do 
> you still think it's a good idea to manually gather and link them with 
> llvm-ld? I've seen other module pass implementations where they 
> directly patch a specific version of LLVM or clang, instead of running 
> a shared object implementation with opt. Which would be the ideal 
> approach to analyze a program with a module pass? Thanks.
>
The question is whether your pass requires having the whole program to 
work (or to be effective) or whether it can be done on each compilation 
unit separately.

If you can do it on separate compilation units, integrate it into 
Clang.  We did this for some of the SAFECode passes in mainline 
SAFECode; you can use it as an example.

If your pass must do whole program analysis (or simply works better if 
it does), then you want to add your pass to libLTO and enable the use of 
link-time optimization using the -emit-llvm option on the llmv-gcc/clang 
command line.

Note that all libLTO passes are enabled when compiling using link-time 
optimization.  If you replace libLTO on your system, your pass is going 
to be run for every program compiled using link-time optimization 
(unless you can make your pass conditional on some optimization level 
like -O4 or -O5 that gets communicated from the compiler front-end to 
the linker to the plugin).  If you're doing instrumentation, you might 
want to do what we did in SAFECode: perform instrumentation in the 
front-end and then use whole-program analysis to change/optimize the 
instrumentation in libLTO.  That way, our libLTO transforms can always 
be turned on; for uninstrumented code, they become a no-op.

-- John T.

> *From: *John Criswell <criswell at illinois.edu 
> <mailto:criswell at illinois.edu>>
> *Organization: *University of Illinois
> *Date: *Tue, 12 Jul 2011 16:19:06 -0500
> *To: *Jinwook Shin <Jinwook.Shin at microsoft.com 
> <mailto:Jinwook.Shin at microsoft.com>>
> *Cc: *"llvmdev at cs.uiuc.edu <mailto:llvmdev at
cs.uiuc.edu>"
> <llvmdev at cs.uiuc.edu <mailto:llvmdev at cs.uiuc.edu>>
> *Subject: *Re: [LLVMdev] running a module pass via opt on multiple 
> bitcode files
>
> On 7/12/11 4:11 PM, Jinwook Shin (HOTWING) wrote:
>
> Hi,
>
> I'm trying to implement a module pass as a share object to process an 
> entire program as a unit. The target program is built from multiple 
> object files, hence multiple bitcode files. However, it seems that opt 
> does not take the whole program, but just one bitcode file,
"test.bc"
> in the example run shown below.
>
> $ opt -load mypass.dylib -mymodulepass < test.bc > /dev/null
>
> How do I run my module pass in a library on a whole program?
>
>
> Link the bitcode files together into a single bitcode file with 
> llvm-ld.  You can then run the single bitcode file through opt.
>
> -- John T.
>
>
> Thanks.
>
>
>
>
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu  <mailto:LLVMdev at cs.uiuc.edu>         
http://llvm.cs.uiuc.eduhttp://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20110819/5ccb4b2a/attachment.html>

Thanks John. I appreciate your help.

I have taken the libLTO approach and it worked like a charm :) For those who
wanted to achieve the same thing, below are the steps I've taken.

John --  a small question: I thought I wouldn't need to touch Makefiles at
all, but that wasn't the case. Could you take a look at Step 4) and tell me
what you think? Thanks!

1) Create your own libLTO and add your pass to it. Do a code diff between
/safecode/tools/LTO/LTOCodeGenerator.cpp and
/llvm/tools/lto/LTOCodeGenerator.cpp to figure out what changes are needed to do
this.
2) Make a backup copy of the original linker  /usr/lib/libLTO.dylib.
3) Copy your libLTO.dylib to /usr/lib.
4) Open Makefiles in your target project and change CC and CFLAGS as follows:
CC=llvm-gcc
CFLAGS=-emit-llvm …
5) Build the project with LLVM.

From: John Criswell <criswell at illinois.edu<mailto:criswell at
illinois.edu>>
Organization: University of Illinois
Date: Fri, 19 Aug 2011 17:00:14 -0500
To: Jinwook Shin <Jinwook.Shin at microsoft.com<mailto:Jinwook.Shin at
microsoft.com>>
Cc: "llvmdev at cs.uiuc.edu<mailto:llvmdev at cs.uiuc.edu>"
<llvmdev at cs.uiuc.edu<mailto:llvmdev at cs.uiuc.edu>>
Subject: Re: [LLVMdev] running a module pass via opt on multiple bitcode files

On 8/19/11 4:36 PM, Jinwook Shin (HOTWING) wrote:
Thanks John. Your approach worked fine for my small toy program. Now, I would
like to run my module pass on a huge project (still a single executable)
consisting of a few thousand bit code files scattered in many different
directories. And some of them are static libraries. Do you still think it's
a good idea to manually gather and link them with llvm-ld? I've seen other
module pass implementations where they directly patch a specific version of LLVM
or clang, instead of running a shared object implementation with opt. Which
would be the ideal approach to analyze a program with a module pass? Thanks.

The question is whether your pass requires having the whole program to work (or
to be effective) or whether it can be done on each compilation unit separately.

If you can do it on separate compilation units, integrate it into Clang.  We did
this for some of the SAFECode passes in mainline SAFECode; you can use it as an
example.

If your pass must do whole program analysis (or simply works better if it does),
then you want to add your pass to libLTO and enable the use of link-time
optimization using the -emit-llvm option on the llmv-gcc/clang command line.

Note that all libLTO passes are enabled when compiling using link-time
optimization.  If you replace libLTO on your system, your pass is going to be
run for every program compiled using link-time optimization (unless you can make
your pass conditional on some optimization level like -O4 or -O5 that gets
communicated from the compiler front-end to the linker to the plugin).  If
you're doing instrumentation, you might want to do what we did in SAFECode:
perform instrumentation in the front-end and then use whole-program analysis to
change/optimize the instrumentation in libLTO.  That way, our libLTO transforms
can always be turned on; for uninstrumented code, they become a no-op.

-- John T.



From: John Criswell <criswell at illinois.edu<mailto:criswell at
illinois.edu>>
Organization: University of Illinois
Date: Tue, 12 Jul 2011 16:19:06 -0500
To: Jinwook Shin <Jinwook.Shin at microsoft.com<mailto:Jinwook.Shin at
microsoft.com>>
Cc: "llvmdev at cs.uiuc.edu<mailto:llvmdev at cs.uiuc.edu>"
<llvmdev at cs.uiuc.edu<mailto:llvmdev at cs.uiuc.edu>>
Subject: Re: [LLVMdev] running a module pass via opt on multiple bitcode files

On 7/12/11 4:11 PM, Jinwook Shin (HOTWING) wrote:
Hi,

I'm trying to implement a module pass as a share object to process an entire
program as a unit. The target program is built from multiple object files, hence
multiple bitcode files. However, it seems that opt does not take the whole
program, but just one bitcode file, "test.bc" in the example run shown
below.

$ opt -load mypass.dylib -mymodulepass < test.bc > /dev/null

How do I run my module pass in a library on a whole program?

Link the bitcode files together into a single bitcode file with llvm-ld.  You
can then run the single bitcode file through opt.

-- John T.



Thanks.







_______________________________________________

LLVM Developers mailing list

LLVMdev at cs.uiuc.edu<mailto:LLVMdev at cs.uiuc.edu>        
http://llvm.cs.uiuc.eduhttp://lists.cs.uiuc.edu/mailman/listinfo/llvmdev


-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20110922/33dc065e/attachment.html>