llvm dev - Aug 2015 - [LLVMdev] [RFC] Developer Policy for LLVM C API

On Sun, Aug 16, 2015 at 10:34 PM, deadal nix via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
>
> 2015-08-16 21:51 GMT-07:00 Eric Christopher <echristo at gmail.com>:
>>
>> The promise of stability. We don't promise that the C++ API will
stay
>> stable.
>>
>
>
> Why was that promise be made in the first place ? Has it been made in the
> first place ?
>
It sounds like you're in favor of dropping C API stability then, if it's
holding us back? That feedback is actually really helpful. :)

There are really three goals here: flexibility to change LLVM IR,
completeness of the C API, and stability of the C API. Pick two.

The goals are mutually incompatible and we have to trade off one for the
other. Most of the LLVM core developers value goal #1, the ability to
change the IR. Look at the pointee type changes that David Blaikie is
working on, and the new EH representation. If we promise both stability and
completeness, these things are impossible.

One way forward is to guarantee stability, but limit completeness. This
would mean limiting the C API to constructs that will always and forever be
easily represented in LLVM.

The other choice is to forget stability and wrap the C++ API completely,
potentially with something auto-generated. We could make a more limited
stability promise along the lines of "these APIs will be updated to reflect
changes to the IR, and are otherwise stable." I think everyone would be
fine with that.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20150817/3c3a7f1f/attachment.html>

On 08/17/2015 08:34 AM, Reid Kleckner via llvm-dev
wrote:
> On Sun, Aug 16, 2015 at 10:34 PM, deadal nix via llvm-dev 
> <llvm-dev at lists.llvm.org <mailto:llvm-dev at
lists.llvm.org>> wrote:
>
>
>     2015-08-16 21:51 GMT-07:00 Eric Christopher <echristo at gmail.com
>     <mailto:echristo at gmail.com>>:
>
>         The promise of stability. We don't promise that the C++ API
>         will stay stable.
>
>
>
>     Why was that promise be made in the first place ? Has it been made
>     in the first place ?
>
>
> It sounds like you're in favor of dropping C API stability then, if 
> it's holding us back? That feedback is actually really helpful. :)
>
> There are really three goals here: flexibility to change LLVM IR, 
> completeness of the C API, and stability of the C API. Pick two.
>
> The goals are mutually incompatible and we have to trade off one for 
> the other. Most of the LLVM core developers value goal #1, the ability 
> to change the IR. Look at the pointee type changes that David Blaikie 
> is working on, and the new EH representation. If we promise both 
> stability and completeness, these things are impossible.
>
> One way forward is to guarantee stability, but limit completeness. 
> This would mean limiting the C API to constructs that will always and 
> forever be easily represented in LLVM.
>
> The other choice is to forget stability and wrap the C++ API 
> completely, potentially with something auto-generated. We could make a 
> more limited stability promise along the lines of "these APIs will be 
> updated to reflect changes to the IR, and are otherwise stable." I 
> think everyone would be fine with that.
In particular, such an API would be a good start to the "unstable" API
which has been discussed.  If you wanted to use the opportunity to 
introduce such a thing, I'd be in support.  Put it in a different 
directory for now, but if we decide to merge the current C API into the 
unstable grouping, we'll combine them at some point.
>
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org         http://llvm.cs.uiuc.edu
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20150817/365964bf/attachment.html>

2015-08-17 8:34 GMT-07:00 Reid Kleckner <rnk at google.com>:
> On Sun, Aug 16, 2015 at 10:34 PM, deadal nix via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>>
>> 2015-08-16 21:51 GMT-07:00 Eric Christopher <echristo at
gmail.com>:
>>>
>>> The promise of stability. We don't promise that the C++ API
will stay
>>> stable.
>>>
>>
>>
>> Why was that promise be made in the first place ? Has it been made in
the
>> first place ?
>>
>
> It sounds like you're in favor of dropping C API stability then, if
it's
> holding us back? That feedback is actually really helpful. :)
>
> There are really three goals here: flexibility to change LLVM IR,
> completeness of the C API, and stability of the C API. Pick two.
>
>
Yes but these aren't black and white IMO. These are need in tension, sure,
but all API have these kind of problem not specifically the C API.


> The goals are mutually incompatible and we have to trade off one for the
> other. Most of the LLVM core developers value goal #1, the ability to
> change the IR. Look at the pointee type changes that David Blaikie is
> working on, and the new EH representation. If we promise both stability and
> completeness, these things are impossible.
>
>
Yes I was actually thinking about that. This is the kind of change that
justify a change in the C API IMO. We should try our best to keep the C API
stable, but if somethign fundamental changes in LLVM, and that there is no
nice way to map it to the old API, then


> One way forward is to guarantee stability, but limit completeness. This
> would mean limiting the C API to constructs that will always and forever be
> easily represented in LLVM.
>
>
That would be a problem for me. It seems like other in this thread
mentioned the incompleteness of the C API as a problem for them and
mentioned the use of various hacks to work around it. On my side I tried to
propose diff to increase completeness of the C API in the past, but the
review process was frankly too painful and I mostly give up. I'd be up for
being a maintainer of the C API if that is needed.


> The other choice is to forget stability and wrap the C++ API completely,
> potentially with something auto-generated. We could make a more limited
> stability promise along the lines of "these APIs will be updated to
reflect
> changes to the IR, and are otherwise stable." I think everyone would
be
> fine with that.
>
I do not think this is a good idea. The C API is used a lot to use LLVM
from other languages than C++ . When used directly from C, you get function
signature, but when used from other languages, changing method randomly
cause undefined behavior. For instance, the landing pad instruction was
changed recently and the C API to create a landing pad was updated. As a
result, 2 thing happened :
 - It was not possible to set the personality function from C, which is a
problem for many frontends.
 - In other languages, as mangling remained the same, the thing compiles
and fail randomly at runtime.

This need to be avoided as much as possible. I'd propose the following
policy :
1/ Do change the IR as per need of LLVM development. The C API must not
prevent LLVM to move forward.
2/ If the change can be mapped nicely without changing the C API, then do
that.
3/ If there is no way to map it nicely, then update the C API.

For instance, the pointer update work would probably be a valid reason to
go with 3/ .

Right now, as far as I'm concerned, the main limitation of the C API are
the capability to set attributes and the capability to use atomic
loads/stores.

In any ways, I do think the patch adding tests for existing API should be
merged. The need for test for the C API was mentioned various times in this
thread and I do think that whatever the road taken, having test is a good
thing.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20150817/b99a433d/attachment.html>

As someone who used the LLVM C API for an experiment back in 2009ish
(porting the SBCL lisp compiler to target LLVM as a backend -- never
finished), I thought it was great that LLVM provided the C API. I was sad
that it wasn't properly updated to include support for all the newly
introduced IR features, though. (E.g. atomics). I tried to send patches for
some of that stuff a while later, but didn't manage to get it in. (I
wasn't
active in llvm at that point, I was just trying to contribute a patch. I
gave up.).

It doesn't make sense to me to have the C API be less than a full wrapping
of the IR building functions. Having less than that, you almost might as
well not have it at all. (Which would be sad.)

I'd like suggest that the most important thing for the C API is NOT a 100%
promise of eternal compatibility, but to _strive_ for compatibility, as
much as is realistically possible. If it's not possible, oh well. We're
not
talking about libc here, so the costs of changing it incompatibly -- with
careful deliberation -- are not impossibly huge. I mean, even libstdc++
changes its ABI occassionally! LLVM's C API certainly shouldn't be
considered more important to keep stable than libstdc++!

That said, incompatible changes ought not be made just for trivial reasons:
they should be made when there's a true need, e.g. when an existing LLVM-C
function cannot sensibly maintain its existing behavior anymore because the
underlying functionality of LLVM was removed, or drastically changed.

I'd propose that the only 100% strict rule should be that if the ABI/API
changes, it is done in a way that *loudly* breaks old programs -- e.g. they
fail to compile, link, or run (depending on how the other-lang wrappers are
accessing the API functions) -- not that you get some random weird
misbehavior because a function's argument types or return type has been
changed.

Ideally, if you're going to remove an API, you'd deprecate it first, if
you
can see the problem coming. Otherwise, oh well -- it's not reasonable to
hold up LLVM development to adhere to deprecation policy on the C API.

That is:
1) Adding new functions for new LLVM functionality: great.
2) Removing old functions: if it's required -- after careful deliberation.
3) Modifying the signature of existing functions: no way.

A couple of concrete examples:

- Let's say you run out of space in an existing enum type (ahem,
LLVMAttribute...).

You should introduce new function names, taking a better type (perhaps some
sort of set type, instead of a fixed-width integer...), mark the old ones
deprecated so that users will update to the new functions. But, keep the
old ones around so that old code can keep working. There's no real need to
remove the old deprecated functions -- they'll keep working as well as they
ever did with the smaller-valued attributes.

- Let's say you remove types from pointers.

This is a change that can be seen coming up. So, one should fairly early on
add an LLVMBuildGEP2 function that takes the extra type argument. Leave the
old function name around for now (deprecated), because with the current
state of LLVM, it can continue to work just fine.

When LLVM is changed to *actually* remove all pointer types in its core,
then the LLVMBuildGEP function cannot reasonably continue to exist. It
simply cannot work, if pointers don't have types, and the type isn't
provided. So, remove it.

Preferably, there would be a release of LLVM in between with both functions
available to let people have some time to adjust, but if not, oh well.


On Mon, Aug 17, 2015 at 1:28 PM, deadal nix via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
>
>
> 2015-08-17 8:34 GMT-07:00 Reid Kleckner <rnk at google.com>:
>
>> On Sun, Aug 16, 2015 at 10:34 PM, deadal nix via llvm-dev <
>> llvm-dev at lists.llvm.org> wrote:
>>
>>>
>>> 2015-08-16 21:51 GMT-07:00 Eric Christopher <echristo at
gmail.com>:
>>>>
>>>> The promise of stability. We don't promise that the C++ API
will stay
>>>> stable.
>>>>
>>>
>>>
>>> Why was that promise be made in the first place ? Has it been made
in
>>> the first place ?
>>>
>>
>> It sounds like you're in favor of dropping C API stability then, if
it's
>> holding us back? That feedback is actually really helpful. :)
>>
>> There are really three goals here: flexibility to change LLVM IR,
>> completeness of the C API, and stability of the C API. Pick two.
>>
>>
> Yes but these aren't black and white IMO. These are need in tension,
sure,
> but all API have these kind of problem not specifically the C API.
>
>
>
>> The goals are mutually incompatible and we have to trade off one for
the
>> other. Most of the LLVM core developers value goal #1, the ability to
>> change the IR. Look at the pointee type changes that David Blaikie is
>> working on, and the new EH representation. If we promise both stability
and
>> completeness, these things are impossible.
>>
>>
> Yes I was actually thinking about that. This is the kind of change that
> justify a change in the C API IMO. We should try our best to keep the C API
> stable, but if somethign fundamental changes in LLVM, and that there is no
> nice way to map it to the old API, then
>
>
>
>> One way forward is to guarantee stability, but limit completeness. This
>> would mean limiting the C API to constructs that will always and
forever be
>> easily represented in LLVM.
>>
>>
> That would be a problem for me. It seems like other in this thread
> mentioned the incompleteness of the C API as a problem for them and
> mentioned the use of various hacks to work around it. On my side I tried to
> propose diff to increase completeness of the C API in the past, but the
> review process was frankly too painful and I mostly give up. I'd be up
for
> being a maintainer of the C API if that is needed.
>
>
>
>> The other choice is to forget stability and wrap the C++ API
completely,
>> potentially with something auto-generated. We could make a more limited
>> stability promise along the lines of "these APIs will be updated
to reflect
>> changes to the IR, and are otherwise stable." I think everyone
would be
>> fine with that.
>>
>
> I do not think this is a good idea. The C API is used a lot to use LLVM
> from other languages than C++ . When used directly from C, you get function
> signature, but when used from other languages, changing method randomly
> cause undefined behavior. For instance, the landing pad instruction was
> changed recently and the C API to create a landing pad was updated. As a
> result, 2 thing happened :
>  - It was not possible to set the personality function from C, which is a
> problem for many frontends.
>  - In other languages, as mangling remained the same, the thing compiles
> and fail randomly at runtime.
>
> This need to be avoided as much as possible. I'd propose the following
> policy :
> 1/ Do change the IR as per need of LLVM development. The C API must not
> prevent LLVM to move forward.
> 2/ If the change can be mapped nicely without changing the C API, then do
> that.
> 3/ If there is no way to map it nicely, then update the C API.
>
> For instance, the pointer update work would probably be a valid reason to
> go with 3/ .
>
> Right now, as far as I'm concerned, the main limitation of the C API
are
> the capability to set attributes and the capability to use atomic
> loads/stores.
>
> In any ways, I do think the patch adding tests for existing API should be
> merged. The need for test for the C API was mentioned various times in this
> thread and I do think that whatever the road taken, having test is a good
> thing.
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org         http://llvm.cs.uiuc.edu
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20150817/bb688af5/attachment.html>