llvm dev - Sep 2009 - [LLVMdev] accessing a bitcode library exported from C++ using the JIT

----- Original Message ----
> From: Eli Friedman <eli.friedman at gmail.com>
> To: Samuel Crow <samuraileumas at yahoo.com>
> Cc: LLVM Developers Mailing List <llvmdev at cs.uiuc.edu>
> Sent: Monday, August 31, 2009 3:49:01 PM
> Subject: Re: [LLVMdev] accessing a bitcode library exported from C++ using
the  JIT
> 
> On Mon, Aug 31, 2009 at 12:17 PM, Samuel Crowwrote:
> > Hello,
> >
> > My partner and I am making a small app needs to access a C++ library
from the
> LLVM 2.5 JIT.  We've made sure that there are no classes and have put
'extern
> "c"' in front of the functions we need to access.  In order
to make this work,
> we seem to need to have a bitcode version of libstdc++ so we can avoid
writing
> our own implementations of std::string and std::vector.  I've been
trying to get
> the version of libstdc++ that came with llvm-gcc-4.2-2.5 to configure and 
> compile.  I've set the environment variables to use llvm-gcc and
llvm-g++
> respectively and placed the --emit-llvm flag in the CCFLAGS and CXXFLAGS 
> environment variables.  Configure works but make fails with:
> >
> > from 
>
/Users/samuraicrow/Documents/llvm-gcc4.2-2.5.source/libstdc++-v3/include/precompiled/extc++.h:60:
> > 
>
/Users/samuraicrow/Documents/llvm-gcc4.2-2.5.source/libstdc++-v3/include/ext/pb_ds/detail/priority_queue_base_dispatch.hpp:79:
> error: ISO C++ forbids declaration of ‘rc_binomial_heap_’ with no type
> > 
>
/Users/samuraicrow/Documents/llvm-gcc4.2-2.5.source/libstdc++-v3/include/ext/pb_ds/detail/priority_queue_base_dispatch.hpp:79:
> error: expected ‘;’ before ‘<’ token
> >
> > I'm building on MacOSX 10.5.8 and my parnter will be building on
MinGW under
> Windows XP.
> >
> > Is this a problem with the source or with my configuration?  Has this
been
> solved in version 2.6 prerelease or ToT?
> 
> It's probably a problem with your configuration, since llvm-gcc can
> definitely build libstdc++ as part of the normal build.
> 
> This seems like the wrong approach, though... can't you just make your
> C APIs take char*/T* instead?
> 
> -Eli
Hello Eli,

We're using the LLVM Value * class functions to box and unbox values and
functions for our stack.  The stack needs to be able to take indexing without
changing the stack pointer so we're using a std::vector for that.  The
std::string section would be a lot easier to replace than the two I just
mentioned since we store the length separately.

This library is serving as part of an interpreted parser generator so it needs
to be able to handle all types and functions that we choose to feed into it. 
For now it is i32, Float and String and in the future, Doubles and functions of
various sorts need to go onto the stack.  The only thing I doubt about our
approach is that we might be able to get away with using an interpreter instead
of a JIT since the code is fed through the LLVM assembly parser just as it gets
executed.

If you're wondering why we're doing an interpreted PEG parser generator
rather than Boost Spirit 2.x, it's because we need it to be easier to debug
the parser.  Once the parser is debugged it can be fed into a compiled parser
generator and "frozen" into stand-alone parser code.

--Sam

On Tue, Sep 1, 2009 at 4:53 AM, Samuel Crow<samuraileumas at yahoo.com>
wrote:
> We're using the LLVM Value * class functions to box and unbox values
and functions for our stack.  The stack needs to be able to take indexing
without changing the stack pointer so we're using a std::vector for that.
 The std::string section would be a lot easier to replace than the two I just
mentioned since we store the length separately.
If you really want to stick with std::string and std::vector, I'd
suggest just writing some wrappers for the relevant functions.  For
example:

extern "C" char* String_cstr(std::string*);
extern "C" void Stack_pushback(std::vector*, void* value);

You can even compile these to bitcode with llvm-g++ and inline them
into your other IR.

I don't think compiling libstdc++ to bitcode actually helps you here;
you can call the relevant functions in libstdc++ whether or not
they're bitcode.

-Eli

----- Original Message ----
> From: Eli Friedman <eli.friedman at gmail.com>
> To: Samuel Crow <samuraileumas at yahoo.com>
> Cc: LLVM Developers Mailing List <llvmdev at cs.uiuc.edu>
> Sent: Monday, August 31, 2009 7:36:10 PM
> Subject: Re: [LLVMdev] accessing a bitcode library exported from C++ using
the  JIT
> 
> On Tue, Sep 1, 2009 at 4:53 AM, Samuel Crowwrote:
> > We're using the LLVM Value * class functions to box and unbox
values and
> functions for our stack.  The stack needs to be able to take indexing
without
> changing the stack pointer so we're using a std::vector for that.  The 
> std::string section would be a lot easier to replace than the two I just 
> mentioned since we store the length separately.
> 
> If you really want to stick with std::string and std::vector, I'd
> suggest just writing some wrappers for the relevant functions.  For
> example:
> 
> extern "C" char* String_cstr(std::string*);
> extern "C" void Stack_pushback(std::vector*, void* value);
> 
> You can even compile these to bitcode with llvm-g++ and inline them
> into your other IR.
> 
> I don't think compiling libstdc++ to bitcode actually helps you here;
> you can call the relevant functions in libstdc++ whether or not
> they're bitcode.
> 
> -Eli
Hi Eli,

That's exactly what we are trying to do.  All of the parameters are passed
in as text and parsed using StringStreams into their relavent types in the Value
stack.  You can view the current code at
http://mattathias.cvs.sourceforge.net/viewvc/mattathias/mattathias/modules/front_ends/Amos/BasicLib.cpp?view=markup
and the BasicLib.h included in it only has a bunch of LLVM includes and an
irrelevant previous attempt at code abstraction and a C struct to return the
length-terminated string in.  I think my mistake is that I'm using Value *
as a return code in one place instead of a void *.  I think that's causing
the name mangling to kick in.  I'll see what I can do to fix it.

Thanks for talking me through the problem,

--Sam

On Mon, Aug 31, 2009 at 6:23 PM, Samuel Crow<samuraileumas at yahoo.com>
wrote:
> If you're wondering why we're doing an interpreted PEG parser
generator rather than Boost Spirit 2.x, it's because we need it to be easier
to debug the parser.  Once the parser is debugged it can be fed into a compiled
parser generator and "frozen" into stand-alone parser code.
You do know that Spirit2.1 has very nice debugging facilities,
everything from error handlers so you can handle parsing failures to
getting detailed info reports on the entire processing process (of
which you can do something like dump it to cout or something and you
get this nice tree of everything that is tried, what passes, what
fails, etc...).  It is exceedingly detailed, so I am not sure what
your homegrown one would have over it, if you have an example then I
could add that functionality to Boost.Spirit2.1.

Hello OvermindDL1,

We are implementing an extensible language.  That's one where you can add
commands and constructs to the language without having to recompile the parser. 
We want compilation of the parser in order to "freeze" it but only as
an option.  One goal is to eventually get the macro functions of our language to
the point where they are equivalent to the template functions of C++ so we can
make a self-hosting langauge.

Also, all of the action nodes in our example PEG parser are written in LLVM
Assembly rather than only C++ (although they call functions from a C++ library)
so I doubt it would work in Spirit2.1 without writing a syntax converter.  The
syntax converter may come in time but it's only optional.  If you're
really interested in a partially finished example parser, it is at
http://mattathias.cvs.sourceforge.net/viewvc/mattathias/mattathias/modules/front_ends/Amos/
.  It contains a PEG parser file and the source to a C++ library included in
that PEG file.


Thanks for your interest,

--Sam


----- Original Message ----
> From: OvermindDL1 <overminddl1 at gmail.com>
> To: LLVM Developers Mailing List <llvmdev at cs.uiuc.edu>
> Sent: Monday, August 31, 2009 8:47:05 PM
> Subject: Re: [LLVMdev] accessing a bitcode library exported from C++ using
the JIT
> 
> On Mon, Aug 31, 2009 at 6:23 PM, Samuel Crowwrote:
> > If you're wondering why we're doing an interpreted PEG parser
generator rather
> than Boost Spirit 2.x, it's because we need it to be easier to debug
the parser.
>  Once the parser is debugged it can be fed into a compiled parser generator
and
> "frozen" into stand-alone parser code.
> 
> You do know that Spirit2.1 has very nice debugging facilities,
> everything from error handlers so you can handle parsing failures to
> getting detailed info reports on the entire processing process (of
> which you can do something like dump it to cout or something and you
> get this nice tree of everything that is tried, what passes, what
> fails, etc...).  It is exceedingly detailed, so I am not sure what
> your homegrown one would have over it, if you have an example then I
> could add that functionality to Boost.Spirit2.1.
> 
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev