llvm dev - Apr 2011 - [LLVMdev] Registering a custom opt pass as a default one

Hi all,

we're working on compile-time instrumentation for ThreadSanitizer (a
data race detector, see http://code.google.com/p/data-race-test and
http://code.google.com/p/data-race-test/wiki/CompileTimeInstrumentation),
which is implemented as an opt plugin that is ran for each client
C/C++ module we compile.

To build a binary consisting of several modules the following steps
are performed:

1. each filename.cc module is translated into LLVM internal
representation (filename.ll file) using llvm-gcc
2. the instrumentation pass is ran on each filename.ll file using opt
to produce filename-instr.ll
3. filename-instr.ll is translated into assembly file filename.S
4. filename.S is compiled into filename.o using llvm-gcc
5. all .o files are linked together with the runtime library (actually
ThreadSanitizer) into the
resulting binary

In order to compile a large program using GNU make, we had to write
wrappers for gcc and g++ that parse the command-line args and perform
either steps 1-4 or step 5, and put those wrappers into $PATH.
On the other hand, if our opt pass was a default one, we could just do:
  $ llvm-gcc filename.cc <-opt_pass_name> <additional opt args> -o
filename.o
instead of steps 1-4.

The question is: does llvm-gcc (or Clang, or whatever) allow to
register a custom optimization pass as a default one (by means of env
vars, configuration files or anything besides rebuilding the
compiler)?

Thanks in advance,
Alexander Potapenko

On Apr 18, 2011, at 6:46 AM, Alexander Potapenko wrote:
> Hi all,
> 
> we're working on compile-time instrumentation for ThreadSanitizer (a
> data race detector, see http://code.google.com/p/data-race-test and
> http://code.google.com/p/data-race-test/wiki/CompileTimeInstrumentation),
> which is implemented as an opt plugin that is ran for each client
> C/C++ module we compile.
> 
> To build a binary consisting of several modules the following steps
> are performed:
> 
> 1. each filename.cc module is translated into LLVM internal
> representation (filename.ll file) using llvm-gcc
> 2. the instrumentation pass is ran on each filename.ll file using opt
> to produce filename-instr.ll
> 3. filename-instr.ll is translated into assembly file filename.S
> 4. filename.S is compiled into filename.o using llvm-gcc
> 5. all .o files are linked together with the runtime library (actually
> ThreadSanitizer) into the
> resulting binary
> 
> In order to compile a large program using GNU make, we had to write
> wrappers for gcc and g++ that parse the command-line args and perform
> either steps 1-4 or step 5, and put those wrappers into $PATH.
> On the other hand, if our opt pass was a default one, we could just do:
>  $ llvm-gcc filename.cc <-opt_pass_name> <additional opt args>
-o filename.o
> instead of steps 1-4.
> 
> The question is: does llvm-gcc (or Clang, or whatever) allow to
> register a custom optimization pass as a default one (by means of env
> vars, configuration files or anything besides rebuilding the
> compiler)?
You can modify llvm/Support/StandardPasses.h to include your pass by default one
appropriate optimization level is used. However, llvm-gcc or clang works on one
source file at a time, so it won't do your step 2) automatically for you.

What you want is link time optimization. See
	http://llvm.org/docs/LinkTimeOptimization.html
and
	http://llvm.org/docs/GoldPlugin.html

If you add your pass in gold plugin by modifying createStandardLTOPasses() then
I believe it will do what you intend to do.

-
Devang

Hi Devang,
> You can modify llvm/Support/StandardPasses.h to include your pass by
default one appropriate optimization level is used. However, llvm-gcc or clang
works on one source file at a time, so it won't do your step 2)
automatically for you.
My wrapper is working on one source file at a time as well, so it's
not a problem IIUYC.
Do you mean that adding a pass to llvm/Support/StandardPasses.h makes
llvm-gcc automatically handle it without recompilation of the
compiler?
>
> What you want is link time optimization. See
>        http://llvm.org/docs/LinkTimeOptimization.html
> and
>        http://llvm.org/docs/GoldPlugin.html
>
> If you add your pass in gold plugin by modifying createStandardLTOPasses()
then I believe it will do what you intend to do.
What we're doing is somewhat orthogonal to link time optimization.
Indeed running the instrumentation on the single module after
everything has been linked is a promising idea (although I suspect
that large projects, e.g. Chromium, may have problems with LLVM LTO at
this time -- it just hadn't been tested by anyone yet). But running
per-file instrumentation has its advantages as well: for example, we
can easily recompile a piece of the project with instrumentation
disabled.

Alex