llvm dev - Oct 2015 - The future of LLVM's C APIs: Notes and BoF.

Yup that is quite similar to my use case, but in addition I use LLVM to JIT
as well from C using MCJIT.

I have a patch for a DIBuilder binding, but it does change the API a bit,
so depending on how this discussion goes, it can or cannot be added. It was
necessary as the current binding wrap all metadata in values, which is very
wasteful when dealing with a large amount of metadata. The breakage is
small (all my existing code still compile and work even using the old
binding API) but very real. I obviously think the change is good or I
wouldn't have made it, but that may be too much for others.

Back on point, the importants things for me to do through the C API are :
 - Write IR, including debug infos and other metadata (it is mostly good,
some IR specificities are not covered, like ordered loads/stores, but these
are edges cases that ca be added. I can prepare a patch if people want
these).
 - Read IR, so I can do transformations.
 - Codegen to object files
 - JIT (using MCJIT is fine, Orca would be better but making a good C api
for it may prove challenging).

I preferably would like that the ABI is not broken, as typecheking won't
work cross language boundary. That being said, I'm ok with this being a
"best practice" or whatever people want to call it rather than a
strong,
enforced, requirement. However, if it is not a strong, enforced
requirement, we need a way to test what versions of LLVM is currently in
use as to choose the right version of the function to call. As long as ABI
breakage are not too frequent, this is workable as far as I'm concerned,
and put less constraint on the evolution of the whole thing.

Last but not least, I started working on a test that would read IR and
regenerate it. This would enforce that IR is readable and writable from the
C API, and that breakage are not done unknowingly like it happened in the
past several times. I really would like this to move forward whatever the
policy is chosen. Random breakage + detective work to figure out what
happened are the worst when we could know things in an automated manner.

As a final note, I'd like for us to focus on resolving this soon enough so
that some work can be done related to the C API before 3.8 is out. Nothing
is on fire yet, but let's not let this slip too much.


2015-10-20 13:16 GMT-07:00 Rodney M. Bates via llvm-dev <
llvm-dev at lists.llvm.org>:
>
>
> On 10/17/2015 01:20 PM, Justin Bogner via llvm-dev wrote:
>
>> (Moving this to llvm-dev)
>>
>>
>> On Friday, October 16, 2015, Justin Bogner <mail at justinbogner.com
>> <mailto:mail at justinbogner.com>> wrote:
>>
>>     Some users of llvm-c want stable API interfaces into various parts
of
>>     the LLVM infrasture, others want further ABI guarantees about this
>>     usage, and still others simply want a way to bind to LLVM through
>> their
>>     language frontend’s existing FFI support for C.
>>
>>     If we want to improve the situation for any of these users, we need
to
>>     properly understand how these APIs are being used (or abused)
>>     today. Juergen and I will be hosting a BoF at the dev meeting where
we
>>     can discuss what the requirements of a sustainable C API are, and
how
>> we
>>     can organize things in LLVM to support this.
>>
>>     There's been a fair amount of discussion about this lately, and
it's
>>     pretty clear that the "hopefully stable bindings to whatever
APIs
>>     somebody needed" approach isn't really good enough for
anybody. There
>>     are a couple of points that I think are fairly non-controversial,
but
>>     will more or less drive the discussion at the BoF:
>>
>>     1. It isn't practical to keep a bindings API stable, unless the
>>         underlying API is also stable.
>>
>>     2. Handrolling bindings as they're needed tends to leave
conspicuous
>>         gaps where some API is inaccessible for no good reason.
>>
>>     So based on (1), we'll really want to create some purpose built
APIs
>>     that we can keep stable for various tasks. What's needed here?
People
>>     want to do things like building a pass manager, setting up a canned
>> JIT
>>     config, and to some degree even emit IR. We'll discuss
what's
>> practical
>>     and what people want, and hopefully strike a good balance.
>>
>>     Similarly, (2) implies that if we really need a *full* bindings API
>>     we'll want to automate it. But what is a full bindings API? Who
uses
>> it,
>>     and what do they want from it? If it's automated, should
installing
>> LLVM
>>     install this API, or should we simply provide an easy way to
generate
>>     it?
>>
>>     In the end, I hope to have a good idea of what people are actually
>> using
>>     these APIs for, and both to support stable API users less
haphazardly
>>     and to make unstable API more thorough and/or easier to create.
>>
>>
> One actual use:
>
> A couple of us in the Modula3 support community are working on splicing a
> llvm back end onto our Modula3 compiler front end.  We have zero-thickness
> bindings written in Modula3 which match C bindings like core.h.  In our
> front
> end executable, we only build llvm IR in memory, then use
> LLVMPrintModuleToFile
> and/or LLVMWriteBitcodeToFile.  We only link in the needed parts of the
> llvm
> infrastructure to build and write the llvm IR.  Then we run stock llc on
> the
> IR code in the emitted file.
>
> We have had to write an additional binding to DIBuilder for this purpose,
> as well as to a few odd other C++ functions here and there.  The latest
> version of this is for llvm 3.6.1, after the separation of metadata from
> values.  It has had minimal testing, but is fairly complete.  Older
> bindings
> to an older DIBuilder exist, and are at least partly working, but only
> contain specifically needed functions.
>
> This scheme is passing a majority of preexisting compiler tests of code
> function, but there is not much testing of emitted debug info.  Speaking
> for myself, better debugger support is one of the primary motivations for
> using llvm.  Our older back ends use a really cobbled-up extension of
> stabs+.  It's a mess, and lots of debugger function is difficult to
> provide.
>
> The DIBuilder binding includes .h and .cpp files similar to core, that
> provide
> a C binding, with lots of [un]wrapping, etc.  Like core, it loses the type
> hierarchy.  Modula3 has a type hierarchy, but machine-level representation
> is undoubtedly not ABI compatible with any C++ compiler.  The current
> handling
> of this is not thoroughly thought out, just using lots of unsafe casting of
> pointers.  I would like to think of something better, but haven't had
much
> time
> for that, so far.
>
>
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
>>
> --
> Rodney Bates
> rodney.m.bates at acm.org
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20151021/359e9a7e/attachment.html>

On 22 Oct 2015, at 02:52, deadal nix via llvm-dev <llvm-dev at
lists.llvm.org> wrote:
> 
> Back on point, the importants things for me to do through the C API are :
>  - Write IR, including debug infos and other metadata (it is mostly good,
some IR specificities are not covered, like ordered loads/stores, but these are
edges cases that ca be added. I can prepare a patch if people want these).
>  - Read IR, so I can do transformations.
>  - Codegen to object files
>  - JIT (using MCJIT is fine, Orca would be better but making a good C api
for it may prove challenging).
Just to clarify, are your requirements for:

- A C API?
- A stable C API?
- An ABI-stable C API?
- An API?
- A stable API?
- An ABI-stable API?

For most of these things, I’d be happy with a stable C++ API (ABI-stable within
major releases, recompile - but nothing else - needed between), which could
potentially be wrapped in a machine-generated C API for bindings in other
languages.

David

A C API (the code is not written in C or C++, and C is kind of the
Esperanto of programming). Preferably a ABI-stable C API, as no
typechecking is going to be done at language boundary. That being said, I
would put the ABI condition as a *should* rather than as a *must*.

2015-10-22 2:42 GMT-07:00 David Chisnall <David.Chisnall at cl.cam.ac.uk>:
> On 22 Oct 2015, at 02:52, deadal nix via llvm-dev <llvm-dev at
lists.llvm.org>
> wrote:
> >
> > Back on point, the importants things for me to do through the C API
are :
> >  - Write IR, including debug infos and other metadata (it is mostly
> good, some IR specificities are not covered, like ordered loads/stores, but
> these are edges cases that ca be added. I can prepare a patch if people
> want these).
> >  - Read IR, so I can do transformations.
> >  - Codegen to object files
> >  - JIT (using MCJIT is fine, Orca would be better but making a good C
> api for it may prove challenging).
>
> Just to clarify, are your requirements for:
>
> - A C API?
> - A stable C API?
> - An ABI-stable C API?
> - An API?
> - A stable API?
> - An ABI-stable API?
>
> For most of these things, I’d be happy with a stable C++ API (ABI-stable
> within major releases, recompile - but nothing else - needed between),
> which could potentially be wrapped in a machine-generated C API for
> bindings in other languages.
>
> David
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20151022/e78a239b/attachment.html>