thr3ads.net - llvm dev - [LLVMdev] How to translate library functions into LLVM IR bitcode? [Sep 2014]

If this information is useful, please help other people find it:
Share via:

Liwei Yang

2014-Sep-15 12:56 UTC

[LLVMdev] How to translate library functions into LLVM IR bitcode?

Good tips. Although I have used llvm-link to merge .bc files together, I
guess -flto could optimize the resultant .bc file further.

As for the assembly, yes it is an issue. Anyway, I'll try to address those
sources which are available for being translated into .bc first.

Thanks for your advice, Tim.

On Mon, Sep 15, 2014 at 2:55 PM, Tim Northover <t.p.northover at
gmail.com>
wrote:
> > If there's any automated way to infer about all the subroutines
that one
> > function needs, clang them into .bc file and link them into a
stand-alone
> > .bc library, that will be more than appreciated:-)
>
> If manually compiling the files is working for you, you could try
> building the entire library with "-flto" for Link-time
optimisation.
> The output of that will be LLVM IR (if you can convince the build
> system to do it for you).
>
> The issue is that parts of the standard library are
> performance-critical and often written in assembly. If the entire file
> is assembly you won't be able to produce IR very easily at all.
>
> Cheers.
>
> Tim.
>


-- 
Best Regards
Liwei
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20140915/785580e2/attachment.html>

Johannes Doerfert

2014-Sep-16 13:00 UTC

head link

[LLVMdev] How to translate library functions into LLVM IR bitcode?

Hey Liwei,

I attached a script I used some time back to compile multiple source
files (of a benchmark) into one bitcode file (instead of an executable).
The script is very simple but worked surprisingly well. It checks the
command line options for indicators what kind of output is expected and
produces bitcode files with the appropriate names. In order to use it
you just need to put/link the script on your path and use it as your
standard compiler (e.g., export CC=<script_name>) instead of clang.
However, clang and llvm-link need to be on the path. If the name of the
executed script is <script_name>++ (note the ++ in the end) then clang++
will be used to compile the source files, otherwise clang. Also note that
the script will remove some command line options as they are not supported
or desired when creating bitcode instead of object code files.

It should also be able to pass a usual autoconf configure run by
detecting it and simply redirecting the complete command line to clang(++).
But I never used it to link libraries though.

I'm not sure if you can use this script as is or as a starting point to
create your own but maybe you can. 


Best regards,
  Johannes



On 09/15, Liwei Yang wrote:> Good tips. Although I have used llvm-link to merge .bc files together, I
> guess -flto could optimize the resultant .bc file further.
> 
> As for the assembly, yes it is an issue. Anyway, I'll try to address
those
> sources which are available for being translated into .bc first.
> 
> Thanks for your advice, Tim.
> 
> On Mon, Sep 15, 2014 at 2:55 PM, Tim Northover <t.p.northover at
gmail.com>
> wrote:
> 
> > > If there's any automated way to infer about all the
subroutines that one
> > > function needs, clang them into .bc file and link them into a
stand-alone
> > > .bc library, that will be more than appreciated:-)
> >
> > If manually compiling the files is working for you, you could try
> > building the entire library with "-flto" for Link-time
optimisation.
> > The output of that will be LLVM IR (if you can convince the build
> > system to do it for you).
> >
> > The issue is that parts of the standard library are
> > performance-critical and often written in assembly. If the entire file
> > is assembly you won't be able to produce IR very easily at all.
> >
> > Cheers.
> >
> > Tim.
> >
> 
> 
> 
> -- 
> Best Regards
> Liwei
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev

-- 

Johannes Doerfert
Researcher / PhD Student

Compiler Design Lab (Prof. Hack)
Saarland University, Computer Science
Building E1.3, Room 4.26

Tel. +49 (0)681 302-57521 : doerfert at cs.uni-saarland.de
Fax. +49 (0)681 302-3065  : http://www.cdl.uni-saarland.de/people/doerfert
-------------- next part --------------
#!/usr/bin/python
import sys, os, subprocess

###############################################################################
############################# Helper functions ################################
###############################################################################
def center(string, columns = 80):
  length = len(string)
  return "%s%s%s" % (" " * int((columns + 1 - length) / 2),
                     string,
                     " " * int((columns + 0 - length) / 2))

def exists(full_path):
  return full_path and os.access(full_path, os.X_OK)

def no_ext(filename):
  return os.path.splitext(filename)[0]

def which(program):
  path = os.getenv('PATH')
  if not path:
    return None
  for sub_path in path.split(':'):
    full_path = os.path.join(sub_path, program)
    if exists(full_path):
      return full_path
  return None

def call(program, arguments):
  try:
    proc = subprocess.Popen([program] + arguments, stdout=sys.stdout,
stderr=sys.stderr)
    retcode = proc.wait()
    if retcode:
      sys.stderr.write("\nERROR:\n------\n")
      sys.stderr.write("Got return code %s while executing:\n  %s
%s\n\n" % (retcode, program, " ".join(arguments)))
      sys.exit(1)
    return retcode
  except Exception as e:
    sys.stderr.write("\nERROR:\n------\n")
    sys.stderr.write("Got an exception while executing:\n  %s
%s\n%s\n\n" % (program, " ".join(arguments), repr(e)))
    sys.exit(1)

###############################################################################
############################# Global constants ################################
###############################################################################

# Enable DEBUG mode
DEBUG = True

# The name of this 'compiler'
NAME = "LLVM-Link-All"


# The environment variables with functions to derive their default values
env_variables = {
    'CLANG': ((lambda : which('clang')), True),
    'CLANG++': ((lambda : which('clang++')), True),
    'LLVM_LINK': ((lambda : which('llvm-link')), True),
  }

# Flag replacements
flag_replacements = {
    #'-O2' : '-O0',
    #'-O3' : '-O0',
    #'-g' : '',
    }


###############################################################################


if DEBUG:
  print ("\n%s" % (center(NAME)))

if DEBUG:
  print ("\n Check environment variables:")

variables = {}
for variable, default_pair in env_variables.items():
  val_inv, def_inv = "", ""
  default_fn, is_path = default_pair
  default = default_fn()
  value = os.getenv(variable, default)

  if is_path:
    if not exists(default):
      def_inv = " INVALID!"
    if not exists(value):
      val_inv = " INVALID!"

  variables[variable] = value
  if DEBUG:
    print ("   %-25s := %s%s \t\t(default: %s%s)" % (variable, value,
val_inv, default, def_inv))

COMPILER = 'CLANG++' if sys.argv[0].endswith('++') else
'CLANG'

for variable, value in variables.items():
  if env_variables[variable][1] and not exists(value):
    sys.stderr.write("\nERROR:\n------\n")
    sys.stderr.write("The executable '%s' was not found! " %
variable.lower())
    sys.stderr.write("The determined value was '%s'\n" %
value)
    sys.stderr.write("Either put it on the 'PATH' or set the
environment ")
    sys.stderr.write("variable '%s' pointing to the
executable.\n\n" % variable)
    sys.exit(1)

arguments = sys.argv[1:]
if DEBUG:
  print ("\n Start parsing the command line:")
  print ("   '%s'" % (" ".join(arguments)))


if 'conftest.c' in arguments or '--version' in arguments:
  sys.stderr.write("\nCONFIGURE IS RUNNING:\n------\n")
  sys.stderr.write("Call %s (%s) %s\n" % (COMPILER,
variables[COMPILER], ' '.join(arguments)))
  retcode = call(variables[COMPILER], arguments)
  if DEBUG:
    sys.stderr.write("     Retcode: %i\n" % retcode)
  sys.exit(retcode)

output_name = None
input_names = []
output_name_add_ending = False
output_kind = None


if DEBUG:
  print ("\n   Test for input files:")

skip_next   = False
for argument in arguments:
  if skip_next:
    skip_next = False
  elif '-o' == argument:
    skip_next = True
  elif '-I' == argument:
    skip_next = True
  elif not argument.startswith('-'):
    input_names.append(argument)

if DEBUG:
  print ("     Input files are '%s'" % ('
'.join(input_names)))

if not input_names:
  sys.stderr.write("\nERROR:\n------\n")
  sys.stderr.write("No input files found\n\n")
  sys.stderr.write("Call %s (%s)\n" % (COMPILER, variables[COMPILER]))
  retcode = call(variables[COMPILER], arguments)
  if DEBUG:
    print ("     Retcode: %i" % retcode)
  sys.exit(retcode)


if DEBUG:
  print ("\n   Test for output file:")

assert(arguments.count('-o') < 2 and "Multiple occurrences of
'-o'!")
if '-o' in arguments:
  index = arguments.index('-o')
  assert(len(arguments) > index + 1 and "-o was not followed by any
value!")
  output_name = arguments[index + 1]
else:
  if len(input_names) > 1:
    output_name = 'a.out'
  else:
    output_name = input_names[0]
    output_name_add_ending = True

if DEBUG:
  print ("     Output file is '%s'" % (output_name))
  if output_name_add_ending:
    print ("     -- but the ending might need adjustment!")

if not output_name:
  sys.stderr.write("\nERROR:\n------\n")
  sys.stderr.write("No output file found\n\n")
  sys.stderr.write("Call %s (%s)\n" % (COMPILER, variables[COMPILER]))
  retcode = call(variables[COMPILER], arguments)
  if DEBUG:
    print ("     Retcode: %i" % retcode)
  sys.exit(retcode)


if DEBUG:
  print ("\n   Test for output kind:")

if '-c' in arguments:
  if DEBUG:
    print ("     An intermediate should be emitted!")
  if '-emit-llvm' in arguments:
    if DEBUG:
      print ("     It is already LLVM-IR ('-emit-llvm' is
used)!")
  else:
    arguments.append('-emit-llvm')
    if DEBUG:
      print ("     Add '-emit-llvm' to emit LLVM-IR!")
  output_kind = 'ir'
else:
  if '-emit-llvm' in arguments:
    if DEBUG:
      print ("     It is already LLVM-IR ('-emit-llvm' is
used)!")
    output_kind = 'ir'
  else:
    if DEBUG:
      print ("     An executable is emitted!")
    output_kind = 'ex'
    if output_name_add_ending:
      new_output_name = 'a.out'
      arguments[arguments.index(output_name)] = new_output_name
      output_name = new_output_name
      if DEBUG:
        print ("       Change output name to '%s'!" %
(new_output_name))


if DEBUG:
  print ("\n   Replace common flags:")

no_replacements = 0
for index in range(len(arguments)):
  argument = arguments[index]
  if argument in flag_replacements:
    new_argument = flag_replacements[argument]
    arguments[index] = new_argument
    no_replacements += 1
    if DEBUG:
      print ("     Replace '%s' by '%s'!" % (argument,
new_argument))

if DEBUG and no_replacements == 0:
  print ("     Nothing found to replace!")

if output_kind == 'ir':

  clang_arguments = arguments
  if DEBUG:
    print ("\n   Initiate CLANG (%s):" % variables['CLANG'])
    print ("     Options: '%s'" % '
'.join(clang_arguments))
  retcode = call(variables['CLANG'], clang_arguments)
  if DEBUG:
    print ("     Retcode: %i" % retcode)

elif output_kind == 'ex' and len(input_names) == 1:

  clang_output_name = no_ext(output_name) + '.bc'
  clang_arguments = arguments + ['-emit-llvm', '-c']
  clang_arguments[clang_arguments.index(output_name)] = clang_output_name
  if DEBUG:
    print ("\n   Initiate CLANG (%s):" % variables['CLANG'])
    print ("     Options: '%s'" % '
'.join(clang_arguments))
  retcode = call(variables['CLANG'], clang_arguments)
  if DEBUG:
    print ("     Retcode: %i" % retcode)

elif output_kind == 'ex' and len(input_names) > 1:

  linked_output_name = no_ext(output_name) + '-linked.bc'
  link_arguments = input_names + ['-o', linked_output_name]
  if DEBUG:
    print ("\n   Initiate LLVM_LINK (%s):" %
variables['LLVM_LINK'])
    print ("     Options: '%s'" % '
'.join(link_arguments))
  retcode = call(variables['LLVM_LINK'], link_arguments)
  if DEBUG:
    print ("     Retcode: %i" % retcode)

else:
  assert(False and "Unknown output kind" and output_kind)


# vim: set ft=python
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 213 bytes
Desc: not available
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20140916/8e2ea61d/attachment.sig>

Liwei Yang

2014-Sep-20 02:41 UTC

head link

[LLVMdev] How to translate library functions into LLVM IR bitcode?

Hi Johannes,

By following your directions, I can use your script as is to produce the
.bc file now. Here's my command line for compiling s_sin.c into s_sin.bc
file and the output:
command line:
~/Downloads/newlib-2.1.0/newlib/libm/mathfp » python ~/llvm_link.py s_sin.c
-I../common/ -I../../libc/include/ -o s_sin.bc

output:
Initiate CLANG (/path-to-clang):
     Options: 's_sin.c -I../common/ -I../../libc/include/ -o s_sin.bc
-emit-llvm -c'
In file included from s_sin.c:18:
./zmath.h:7:9: warning: 'NAN' macro redefined
#define NAN 2
        ^
../../libc/include/math.h:57:11: note: previous definition is here
#  define NAN (__builtin_nanf(""))
          ^
1 warning generated.
     Retcode: 0

I don't know why this warning gets generated since in
../../libc/include/math.h:57 the macro NAN is wrapped by ifndef as shown
below, but that's not important.
# ifndef NAN
#  define NAN (__builtin_nanf(""))
# endif


I llvm-dis this s_sin.bc file and get a s_sin.ll file as follows.
; Function Attrs: nounwind uwtable
define double @sin(double %x) #0 {
entry:
  %x.addr = alloca double, align 8
  store double %x, double* %x.addr, align 8
  %0 = load double* %x.addr, align 8
  %call = call double @sine(double %0, i32 0)
  ret double %call
}

declare double @sine(double, i32) #1

Now here comes the point. In this bitcode file, the callee of sin()
function, i.e. sine(), does not have a function body. I know that the
definition of sine() is in another .c file, which is s_sine.c in the same
folder. So essentially, I'm wondering if the script can help to
automatically compile and link all the callees recursively required by
sin() without manually specifying every source file of the callees.

Sorry for coming back late on this topic and thank you so much Johannes,
for sharing your helper script.


On Tue, Sep 16, 2014 at 9:00 PM, Johannes Doerfert <
doerfert at cs.uni-saarland.de> wrote:
> Hey Liwei,
>
> I attached a script I used some time back to compile multiple source
> files (of a benchmark) into one bitcode file (instead of an executable).
> The script is very simple but worked surprisingly well. It checks the
> command line options for indicators what kind of output is expected and
> produces bitcode files with the appropriate names. In order to use it
> you just need to put/link the script on your path and use it as your
> standard compiler (e.g., export CC=<script_name>) instead of clang.
> However, clang and llvm-link need to be on the path. If the name of the
> executed script is <script_name>++ (note the ++ in the end) then
clang++
> will be used to compile the source files, otherwise clang. Also note that
> the script will remove some command line options as they are not supported
> or desired when creating bitcode instead of object code files.
>
> It should also be able to pass a usual autoconf configure run by
> detecting it and simply redirecting the complete command line to clang(++).
> But I never used it to link libraries though.
>
> I'm not sure if you can use this script as is or as a starting point to
> create your own but maybe you can.
>
>
> Best regards,
>   Johannes
>
>
>
> On 09/15, Liwei Yang wrote:
> > Good tips. Although I have used llvm-link to merge .bc files together,
I
> > guess -flto could optimize the resultant .bc file further.
> >
> > As for the assembly, yes it is an issue. Anyway, I'll try to
address
> those
> > sources which are available for being translated into .bc first.
> >
> > Thanks for your advice, Tim.
> >
> > On Mon, Sep 15, 2014 at 2:55 PM, Tim Northover <t.p.northover at
gmail.com>
> > wrote:
> >
> > > > If there's any automated way to infer about all the
subroutines that
> one
> > > > function needs, clang them into .bc file and link them into
a
> stand-alone
> > > > .bc library, that will be more than appreciated:-)
> > >
> > > If manually compiling the files is working for you, you could try
> > > building the entire library with "-flto" for Link-time
optimisation.
> > > The output of that will be LLVM IR (if you can convince the build
> > > system to do it for you).
> > >
> > > The issue is that parts of the standard library are
> > > performance-critical and often written in assembly. If the entire
file
> > > is assembly you won't be able to produce IR very easily at
all.
> > >
> > > Cheers.
> > >
> > > Tim.
> > >
> >
> >
> >
> > --
> > Best Regards
> > Liwei
>
> > _______________________________________________
> > LLVM Developers mailing list
> > LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>
>
> --
>
> Johannes Doerfert
> Researcher / PhD Student
>
> Compiler Design Lab (Prof. Hack)
> Saarland University, Computer Science
> Building E1.3, Room 4.26
>
> Tel. +49 (0)681 302-57521 : doerfert at cs.uni-saarland.de
> Fax. +49 (0)681 302-3065  : http://www.cdl.uni-saarland.de/people/doerfert
>


-- 
Best Regards
Liwei
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20140920/91fd10fe/attachment.html>

llvm dev - Sep 2014 - [LLVMdev] How to translate library functions into LLVM IR bitcode?

[LLVMdev] How to translate library functions into LLVM IR bitcode?

[LLVMdev] How to translate library functions into LLVM IR bitcode?

[LLVMdev] How to translate library functions into LLVM IR bitcode?