llvm dev - Sep 2014 - [LLVMdev] How to translate library functions into LLVM IR bitcode?

Hi Johannes,

By following your directions, I can use your script as is to produce the
.bc file now. Here's my command line for compiling s_sin.c into s_sin.bc
file and the output:
command line:
~/Downloads/newlib-2.1.0/newlib/libm/mathfp » python ~/llvm_link.py s_sin.c
-I../common/ -I../../libc/include/ -o s_sin.bc

output:
Initiate CLANG (/path-to-clang):
     Options: 's_sin.c -I../common/ -I../../libc/include/ -o s_sin.bc
-emit-llvm -c'
In file included from s_sin.c:18:
./zmath.h:7:9: warning: 'NAN' macro redefined
#define NAN 2
        ^
../../libc/include/math.h:57:11: note: previous definition is here
#  define NAN (__builtin_nanf(""))
          ^
1 warning generated.
     Retcode: 0

I don't know why this warning gets generated since in
../../libc/include/math.h:57 the macro NAN is wrapped by ifndef as shown
below, but that's not important.
# ifndef NAN
#  define NAN (__builtin_nanf(""))
# endif


I llvm-dis this s_sin.bc file and get a s_sin.ll file as follows.
; Function Attrs: nounwind uwtable
define double @sin(double %x) #0 {
entry:
  %x.addr = alloca double, align 8
  store double %x, double* %x.addr, align 8
  %0 = load double* %x.addr, align 8
  %call = call double @sine(double %0, i32 0)
  ret double %call
}

declare double @sine(double, i32) #1

Now here comes the point. In this bitcode file, the callee of sin()
function, i.e. sine(), does not have a function body. I know that the
definition of sine() is in another .c file, which is s_sine.c in the same
folder. So essentially, I'm wondering if the script can help to
automatically compile and link all the callees recursively required by
sin() without manually specifying every source file of the callees.

Sorry for coming back late on this topic and thank you so much Johannes,
for sharing your helper script.


On Tue, Sep 16, 2014 at 9:00 PM, Johannes Doerfert <
doerfert at cs.uni-saarland.de> wrote:
> Hey Liwei,
>
> I attached a script I used some time back to compile multiple source
> files (of a benchmark) into one bitcode file (instead of an executable).
> The script is very simple but worked surprisingly well. It checks the
> command line options for indicators what kind of output is expected and
> produces bitcode files with the appropriate names. In order to use it
> you just need to put/link the script on your path and use it as your
> standard compiler (e.g., export CC=<script_name>) instead of clang.
> However, clang and llvm-link need to be on the path. If the name of the
> executed script is <script_name>++ (note the ++ in the end) then
clang++
> will be used to compile the source files, otherwise clang. Also note that
> the script will remove some command line options as they are not supported
> or desired when creating bitcode instead of object code files.
>
> It should also be able to pass a usual autoconf configure run by
> detecting it and simply redirecting the complete command line to clang(++).
> But I never used it to link libraries though.
>
> I'm not sure if you can use this script as is or as a starting point to
> create your own but maybe you can.
>
>
> Best regards,
>   Johannes
>
>
>
> On 09/15, Liwei Yang wrote:
> > Good tips. Although I have used llvm-link to merge .bc files together,
I
> > guess -flto could optimize the resultant .bc file further.
> >
> > As for the assembly, yes it is an issue. Anyway, I'll try to
address
> those
> > sources which are available for being translated into .bc first.
> >
> > Thanks for your advice, Tim.
> >
> > On Mon, Sep 15, 2014 at 2:55 PM, Tim Northover <t.p.northover at
gmail.com>
> > wrote:
> >
> > > > If there's any automated way to infer about all the
subroutines that
> one
> > > > function needs, clang them into .bc file and link them into
a
> stand-alone
> > > > .bc library, that will be more than appreciated:-)
> > >
> > > If manually compiling the files is working for you, you could try
> > > building the entire library with "-flto" for Link-time
optimisation.
> > > The output of that will be LLVM IR (if you can convince the build
> > > system to do it for you).
> > >
> > > The issue is that parts of the standard library are
> > > performance-critical and often written in assembly. If the entire
file
> > > is assembly you won't be able to produce IR very easily at
all.
> > >
> > > Cheers.
> > >
> > > Tim.
> > >
> >
> >
> >
> > --
> > Best Regards
> > Liwei
>
> > _______________________________________________
> > LLVM Developers mailing list
> > LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>
>
> --
>
> Johannes Doerfert
> Researcher / PhD Student
>
> Compiler Design Lab (Prof. Hack)
> Saarland University, Computer Science
> Building E1.3, Room 4.26
>
> Tel. +49 (0)681 302-57521 : doerfert at cs.uni-saarland.de
> Fax. +49 (0)681 302-3065  : http://www.cdl.uni-saarland.de/people/doerfert
>


-- 
Best Regards
Liwei
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20140920/91fd10fe/attachment.html>

Hey Liwei,

the script was "designed" to act as a compile during "the
usual"
configure + make of a benchmark/program/library. If you have such a setup
(a configure script which will produce the Makefile file), you can do
something similar to:

ln -s ~/llvm_link.py ~/llvm_link
ln -s ~/llvm_link.py ~/llvm_link++

export CC=~/llvm_link
export CXX=~/llvm_link++

cd <SRC>
./configure
make

and it should result in a .bc file where otherwise an e.g., an
executable would have been produced.

If you just have a bunch of .c files which can be compiled into an
executable with:

clang -I <HEADER_DIR> a.c b.c c.c d.c -o ex

you can replace clang by the script and it should give you a ex.bc in
the end.

Does that help?

Best regards,
  Johannes

On 09/20, Liwei Yang wrote:
> Hi Johannes,
> 
> By following your directions, I can use your script as is to produce the
> .bc file now. Here's my command line for compiling s_sin.c into
s_sin.bc
> file and the output:
> command line:
> ~/Downloads/newlib-2.1.0/newlib/libm/mathfp » python ~/llvm_link.py s_sin.c
> -I../common/ -I../../libc/include/ -o s_sin.bc
> 
> output:
> Initiate CLANG (/path-to-clang):
>      Options: 's_sin.c -I../common/ -I../../libc/include/ -o s_sin.bc
> -emit-llvm -c'
> In file included from s_sin.c:18:
> ./zmath.h:7:9: warning: 'NAN' macro redefined
> #define NAN 2
>         ^
> ../../libc/include/math.h:57:11: note: previous definition is here
> #  define NAN (__builtin_nanf(""))
>           ^
> 1 warning generated.
>      Retcode: 0
> 
> I don't know why this warning gets generated since in
> ../../libc/include/math.h:57 the macro NAN is wrapped by ifndef as shown
> below, but that's not important.
> # ifndef NAN
> #  define NAN (__builtin_nanf(""))
> # endif
> 
> 
> I llvm-dis this s_sin.bc file and get a s_sin.ll file as follows.
> ; Function Attrs: nounwind uwtable
> define double @sin(double %x) #0 {
> entry:
>   %x.addr = alloca double, align 8
>   store double %x, double* %x.addr, align 8
>   %0 = load double* %x.addr, align 8
>   %call = call double @sine(double %0, i32 0)
>   ret double %call
> }
> 
> declare double @sine(double, i32) #1
> 
> Now here comes the point. In this bitcode file, the callee of sin()
> function, i.e. sine(), does not have a function body. I know that the
> definition of sine() is in another .c file, which is s_sine.c in the same
> folder. So essentially, I'm wondering if the script can help to
> automatically compile and link all the callees recursively required by
> sin() without manually specifying every source file of the callees.
> 
> Sorry for coming back late on this topic and thank you so much Johannes,
> for sharing your helper script.
> 
> 
> On Tue, Sep 16, 2014 at 9:00 PM, Johannes Doerfert <
> doerfert at cs.uni-saarland.de> wrote:
> 
> > Hey Liwei,
> >
> > I attached a script I used some time back to compile multiple source
> > files (of a benchmark) into one bitcode file (instead of an
executable).
> > The script is very simple but worked surprisingly well. It checks the
> > command line options for indicators what kind of output is expected
and
> > produces bitcode files with the appropriate names. In order to use it
> > you just need to put/link the script on your path and use it as your
> > standard compiler (e.g., export CC=<script_name>) instead of
clang.
> > However, clang and llvm-link need to be on the path. If the name of
the
> > executed script is <script_name>++ (note the ++ in the end) then
clang++
> > will be used to compile the source files, otherwise clang. Also note
that
> > the script will remove some command line options as they are not
supported
> > or desired when creating bitcode instead of object code files.
> >
> > It should also be able to pass a usual autoconf configure run by
> > detecting it and simply redirecting the complete command line to
clang(++).
> > But I never used it to link libraries though.
> >
> > I'm not sure if you can use this script as is or as a starting
point to
> > create your own but maybe you can.
> >
> >
> > Best regards,
> >   Johannes
> >
> >
> >
> > On 09/15, Liwei Yang wrote:
> > > Good tips. Although I have used llvm-link to merge .bc files
together, I
> > > guess -flto could optimize the resultant .bc file further.
> > >
> > > As for the assembly, yes it is an issue. Anyway, I'll try to
address
> > those
> > > sources which are available for being translated into .bc first.
> > >
> > > Thanks for your advice, Tim.
> > >
> > > On Mon, Sep 15, 2014 at 2:55 PM, Tim Northover <t.p.northover
at gmail.com>
> > > wrote:
> > >
> > > > > If there's any automated way to infer about all the
subroutines that
> > one
> > > > > function needs, clang them into .bc file and link them
into a
> > stand-alone
> > > > > .bc library, that will be more than appreciated:-)
> > > >
> > > > If manually compiling the files is working for you, you
could try
> > > > building the entire library with "-flto" for
Link-time optimisation.
> > > > The output of that will be LLVM IR (if you can convince the
build
> > > > system to do it for you).
> > > >
> > > > The issue is that parts of the standard library are
> > > > performance-critical and often written in assembly. If the
entire file
> > > > is assembly you won't be able to produce IR very easily
at all.
> > > >
> > > > Cheers.
> > > >
> > > > Tim.
> > > >
> > >
> > >
> > >
> > > --
> > > Best Regards
> > > Liwei
> >
> > > _______________________________________________
> > > LLVM Developers mailing list
> > > LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> > > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
> >
> >
> > --
> >
> > Johannes Doerfert
> > Researcher / PhD Student
> >
> > Compiler Design Lab (Prof. Hack)
> > Saarland University, Computer Science
> > Building E1.3, Room 4.26
> >
> > Tel. +49 (0)681 302-57521 : doerfert at cs.uni-saarland.de
> > Fax. +49 (0)681 302-3065  :
http://www.cdl.uni-saarland.de/people/doerfert
> >
> 
> 
> 
> -- 
> Best Regards
> Liwei
-- 

Johannes Doerfert
Researcher / PhD Student

Compiler Design Lab (Prof. Hack)
Saarland University, Computer Science
Building E1.3, Room 4.26

Tel. +49 (0)681 302-57521 : doerfert at cs.uni-saarland.de
Fax. +49 (0)681 302-3065  : http://www.cdl.uni-saarland.de/people/doerfert
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 213 bytes
Desc: not available
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20140920/85a8394b/attachment.sig>

Hi Johannes,

Actually, I'm working in the same scenario, i.e. configure + make of a
benchmark/program/library like you said. I've got your point of using this
script as a replacement to generate .bc files instead of a executable.
That's truly helpful and has already answered my original question.

Now I'm actually moving a step further. Take the same example in your
reply, say, if I have a bunch of .c files, where a.c calls functions
defined in b.c, b.c calls functions defined in c.c and so on. I know that I can
compile them into a .bc file by using this command:
~/llvm_link -I <HEADER_DIR> a.c b.c c.c d.c -emit-llvm -c -o linked.bc

But the problem is all of the files which define the callees need to be
specified manually. Is it possible that the source files in which the
callees are defined can be automatically inferred, compiled and linked? If
so, then we can just simply use this command below to include all the
callees required by in a.c:
~/llvm_link -I <HEADER_DIR> a.c -emit-llvm -c -o linked.bc

Though we can define the dependency for the source files defining callees
in makefile, it's still a manual specification there. So that's why
I'm
looking into this further step.


On Sat, Sep 20, 2014 at 10:53 AM, Johannes Doerfert <
doerfert at cs.uni-saarland.de> wrote:
> Hey Liwei,
>
> the script was "designed" to act as a compile during "the
usual"
> configure + make of a benchmark/program/library. If you have such a setup
> (a configure script which will produce the Makefile file), you can do
> something similar to:
>
> ln -s ~/llvm_link.py ~/llvm_link
> ln -s ~/llvm_link.py ~/llvm_link++
>
> export CC=~/llvm_link
> export CXX=~/llvm_link++
>
> cd <SRC>
> ./configure
> make
>
> and it should result in a .bc file where otherwise an e.g., an
> executable would have been produced.
>
> If you just have a bunch of .c files which can be compiled into an
> executable with:
>
> clang -I <HEADER_DIR> a.c b.c c.c d.c -o ex
>
> you can replace clang by the script and it should give you a ex.bc in
> the end.
>
> Does that help?
>
> Best regards,
>   Johannes
>
> On 09/20, Liwei Yang wrote:
> > Hi Johannes,
> >
> > By following your directions, I can use your script as is to produce
the
> > .bc file now. Here's my command line for compiling s_sin.c into
s_sin.bc
> > file and the output:
> > command line:
> > ~/Downloads/newlib-2.1.0/newlib/libm/mathfp » python ~/llvm_link.py
> s_sin.c
> > -I../common/ -I../../libc/include/ -o s_sin.bc
> >
> > output:
> > Initiate CLANG (/path-to-clang):
> >      Options: 's_sin.c -I../common/ -I../../libc/include/ -o
s_sin.bc
> > -emit-llvm -c'
> > In file included from s_sin.c:18:
> > ./zmath.h:7:9: warning: 'NAN' macro redefined
> > #define NAN 2
> >         ^
> > ../../libc/include/math.h:57:11: note: previous definition is here
> > #  define NAN (__builtin_nanf(""))
> >           ^
> > 1 warning generated.
> >      Retcode: 0
> >
> > I don't know why this warning gets generated since in
> > ../../libc/include/math.h:57 the macro NAN is wrapped by ifndef as
shown
> > below, but that's not important.
> > # ifndef NAN
> > #  define NAN (__builtin_nanf(""))
> > # endif
> >
> >
> > I llvm-dis this s_sin.bc file and get a s_sin.ll file as follows.
> > ; Function Attrs: nounwind uwtable
> > define double @sin(double %x) #0 {
> > entry:
> >   %x.addr = alloca double, align 8
> >   store double %x, double* %x.addr, align 8
> >   %0 = load double* %x.addr, align 8
> >   %call = call double @sine(double %0, i32 0)
> >   ret double %call
> > }
> >
> > declare double @sine(double, i32) #1
> >
> > Now here comes the point. In this bitcode file, the callee of sin()
> > function, i.e. sine(), does not have a function body. I know that the
> > definition of sine() is in another .c file, which is s_sine.c in the
same
> > folder. So essentially, I'm wondering if the script can help to
> > automatically compile and link all the callees recursively required by
> > sin() without manually specifying every source file of the callees.
> >
> > Sorry for coming back late on this topic and thank you so much
Johannes,
> > for sharing your helper script.
> >
> >
> > On Tue, Sep 16, 2014 at 9:00 PM, Johannes Doerfert <
> > doerfert at cs.uni-saarland.de> wrote:
> >
> > > Hey Liwei,
> > >
> > > I attached a script I used some time back to compile multiple
source
> > > files (of a benchmark) into one bitcode file (instead of an
> executable).
> > > The script is very simple but worked surprisingly well. It checks
the
> > > command line options for indicators what kind of output is
expected and
> > > produces bitcode files with the appropriate names. In order to
use it
> > > you just need to put/link the script on your path and use it as
your
> > > standard compiler (e.g., export CC=<script_name>) instead
of clang.
> > > However, clang and llvm-link need to be on the path. If the name
of the
> > > executed script is <script_name>++ (note the ++ in the end)
then
> clang++
> > > will be used to compile the source files, otherwise clang. Also
note
> that
> > > the script will remove some command line options as they are not
> supported
> > > or desired when creating bitcode instead of object code files.
> > >
> > > It should also be able to pass a usual autoconf configure run by
> > > detecting it and simply redirecting the complete command line to
> clang(++).
> > > But I never used it to link libraries though.
> > >
> > > I'm not sure if you can use this script as is or as a
starting point to
> > > create your own but maybe you can.
> > >
> > >
> > > Best regards,
> > >   Johannes
> > >
> > >
> > >
> > > On 09/15, Liwei Yang wrote:
> > > > Good tips. Although I have used llvm-link to merge .bc files
> together, I
> > > > guess -flto could optimize the resultant .bc file further.
> > > >
> > > > As for the assembly, yes it is an issue. Anyway, I'll
try to address
> > > those
> > > > sources which are available for being translated into .bc
first.
> > > >
> > > > Thanks for your advice, Tim.
> > > >
> > > > On Mon, Sep 15, 2014 at 2:55 PM, Tim Northover <
> t.p.northover at gmail.com>
> > > > wrote:
> > > >
> > > > > > If there's any automated way to infer about
all the subroutines
> that
> > > one
> > > > > > function needs, clang them into .bc file and link
them into a
> > > stand-alone
> > > > > > .bc library, that will be more than appreciated:-)
> > > > >
> > > > > If manually compiling the files is working for you, you
could try
> > > > > building the entire library with "-flto" for
Link-time
> optimisation.
> > > > > The output of that will be LLVM IR (if you can convince
the build
> > > > > system to do it for you).
> > > > >
> > > > > The issue is that parts of the standard library are
> > > > > performance-critical and often written in assembly. If
the entire
> file
> > > > > is assembly you won't be able to produce IR very
easily at all.
> > > > >
> > > > > Cheers.
> > > > >
> > > > > Tim.
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Best Regards
> > > > Liwei
> > >
> > > > _______________________________________________
> > > > LLVM Developers mailing list
> > > > LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> > > > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
> > >
> > >
> > > --
> > >
> > > Johannes Doerfert
> > > Researcher / PhD Student
> > >
> > > Compiler Design Lab (Prof. Hack)
> > > Saarland University, Computer Science
> > > Building E1.3, Room 4.26
> > >
> > > Tel. +49 (0)681 302-57521 : doerfert at cs.uni-saarland.de
> > > Fax. +49 (0)681 302-3065  :
> http://www.cdl.uni-saarland.de/people/doerfert
> > >
> >
> >
> >
> > --
> > Best Regards
> > Liwei
>
> --
>
> Johannes Doerfert
> Researcher / PhD Student
>
> Compiler Design Lab (Prof. Hack)
> Saarland University, Computer Science
> Building E1.3, Room 4.26
>
> Tel. +49 (0)681 302-57521 : doerfert at cs.uni-saarland.de
> Fax. +49 (0)681 302-3065  : http://www.cdl.uni-saarland.de/people/doerfert
>


-- 
Best Regards
Liwei
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20140920/c3a3c3a2/attachment.html>