llvm dev - Jun 2015 - [LLVMdev] [cfe-dev] llvm-abi: A library for generating ABI-compliant LLVM IR

Hi Reid,

Thanks for your response.

The issue is that every LLVM frontend needing ABI compliance has to
re-implement the same target-dependent logic, which is a significant
burden; the ABI compliance code inside Clang isn't really usable for
other frontends as-is. We haven't got many good options here :-). I
think a lot of people would've hoped that LLVM would provide the means
for achieving ABI compliance (at least for C), though I'm well aware
of the complexity involved and understand the decisions taken thus
far.

Within this context, llvm-abi is an immediately actionable way of
sharing code between frontends, that should lead to a higher quality
codebase for a significantly reduced effort on everyone's part. It
seems like this would further build on LLVM's success because I think
many people would like to generate code that can interact with C APIs.

Ultimately it would be great to see this functionality be provided in
an accessible form inside LLVM and hence for Clang to use that
functionality. This would move much of the target-dependent logic out
of Clang while at the same time making this functionality available to
other LLVM-based tools.

I am sure this is a substantial and complex long term project, but I
think it is a worthwhile aim. Do you (and the LLVM community as a
whole) agree? (Acknowledging that some of the details would need to be
worked out.)

I have a long term and substantial interest in this (as I expect do
other frontend developers), so I'm willing to contribute significant
effort to move this forward. Hopefully the llvm-abi library can
provide a better understanding of what needs to be represented and
help non-Clang frontends :-).

Implementation-wise the changes to LLVM could involve encoding ABI
information into LLVM IR or simply providing C++ APIs for generating
ABI-compliant code inside LLVM (much like llvm-abi). Each solution has
advantages and disadvantages and I'm planning to make a proposal about
this on the mailing list later (hence I'd suggest leaving that
discussion until I make the proposal).

In terms of the mentioned deficiencies of the Type structure, these
are all problems that I intend to address (it would be great if you
could provide details on any of the more difficult areas).

Thanks again,
Stephen

On Mon, Jun 29, 2015 at 7:32 PM, Reid Kleckner <rnk at google.com>
wrote:
> I agree it would be really nice to build a library for ABI lowering, but
any
> solution that isn't clang or isn't ultimately picked up by clang
will
> necessarily be incomplete. Perhaps that is OK for your frontend's uses,
but
> I think it's the main reason that we haven't done something like
this in
> LLVM already.
>
> Any solution that doesn't involve actual Clang ASTs is unlikely to be
able
> to represent all C-with-extensions types (unions, bitfields, alignment
> attributes, transparent_union attribute). I took a look at Type.hpp in your
> project, and it seems to be missing some of these things. Keeping such a
> library up to date with new extensions is going to be a maintenance burden.
>
> That said, I wish you luck, and I hope the project eases some of the
> difficulties for new frontends. It is very possible that the corner cases
> that keep me up at night are not the problems that users actually face. :)
>
> On Sun, Jun 28, 2015 at 3:22 PM, Stephen Cross <scross at
scross.co.uk> wrote:
>>
>> Hi everyone,
>>
>> (Also CC'ed cfe-dev since this seems relevant to Clang,
particularly
>> the questions at the end.)
>>
>> I've been working on a library to generate LLVM IR that complies
with
>> platform ABIs (the current focus is on C but I'm also interested in
>> ABIs for other languages).
>>
>> You can find it here: https://github.com/scross99/llvm-abi
>>
>> To explain further (for those who are unfamiliar), LLVM frontends have
>> to modify function argument types, attributes etc. in order to ensure
>> the backend generates code that satisfies the ABI; this is needed
>> because LLVM's type system can't encode all the necessary
information.
>> This is a complex task and involves substantial amounts of
>> target-dependent logic. Clang performs this encoding and indeed much
>> of the current functionality is derived from Clang's source.
>>
>> This project originated as a necessary piece of functionality for the
>> Loci compiler frontend [1] and is now an external dependency; my aim
>> is to make this usable for other LLVM frontends that also need to
>> generate ABI-compliant IR (I assume this is a fairly large subset of
>> the frontends).
>>
>> I'd be very interested in any suggestions/queries/comments.
>>
>> I made a few interesting discoveries while working on this:
>>
>> * Clang generates 8 byte alignment for 16+ byte arrays on x86-64, even
>> though the AMD64 ABI seems to require that arrays of 16+ bytes are
>> aligned to 16 bytes. Is this a bug or am I missing something obvious?
>>
>> * Clang determines the features for a CPU (e.g. whether we have AVX
>> support on x86-64 CPUs), even though this functionality is already
>> available in LLVM (but appears to be very difficult to query). Would
>> it be possible to expose the information from LLVM and hence eliminate
>> the duplication in Clang?
>>
>> * Clang determines a 'generic CPU' if the user doesn't
specify a CPU.
>> My understanding is that we don't usually generate code for the
native
>> CPU because it may have features unavailable on other similar CPUs.
>> LLVM can provide full details of the native CPU but can't determine
a
>> generic CPU; could this functionality be added to LLVM?
>>
>> (Separately I've also been considering a proposal to add ABI
>> information directly inside LLVM IR in a language-independent way and
>> I'll discuss this in a later email.)
>>
>> Thanks,
>> Stephen
>>
>> [1] https://github.com/scross99/locic
>> _______________________________________________
>> cfe-dev mailing list
>> cfe-dev at cs.uiuc.edu
>> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
>
>

----- Original Message -----
> From: "Stephen Cross" <scross at scross.co.uk>
> To: "Reid Kleckner" <rnk at google.com>
> Cc: "Clang Developers List" <cfe-dev at cs.uiuc.edu>,
"LLVM Developers Mailing List" <llvmdev at cs.uiuc.edu>
> Sent: Monday, June 29, 2015 3:03:40 PM
> Subject: Re: [cfe-dev] llvm-abi: A library for generating ABI-compliant
LLVM	IR
> 
> Hi Reid,
> 
> Thanks for your response.
> 
> The issue is that every LLVM frontend needing ABI compliance has to
> re-implement the same target-dependent logic, which is a significant
> burden; the ABI compliance code inside Clang isn't really usable for
> other frontends as-is. We haven't got many good options here :-). I
> think a lot of people would've hoped that LLVM would provide the
> means
> for achieving ABI compliance (at least for C), though I'm well aware
> of the complexity involved and understand the decisions taken thus
> far.
> 
> Within this context, llvm-abi is an immediately actionable way of
> sharing code between frontends, that should lead to a higher quality
> codebase for a significantly reduced effort on everyone's part. It
> seems like this would further build on LLVM's success because I think
> many people would like to generate code that can interact with C
> APIs.
> 
> Ultimately it would be great to see this functionality be provided in
> an accessible form inside LLVM and hence for Clang to use that
> functionality. This would move much of the target-dependent logic out
> of Clang while at the same time making this functionality available
> to
> other LLVM-based tools.
> 
> I am sure this is a substantial and complex long term project, but I
> think it is a worthwhile aim. Do you (and the LLVM community as a
> whole) agree? (Acknowledging that some of the details would need to
> be
> worked out.)
> 
> I have a long term and substantial interest in this (as I expect do
> other frontend developers), so I'm willing to contribute significant
> effort to move this forward. Hopefully the llvm-abi library can
> provide a better understanding of what needs to be represented and
> help non-Clang frontends :-).
> 
> Implementation-wise the changes to LLVM could involve encoding ABI
> information into LLVM IR or simply providing C++ APIs for generating
> ABI-compliant code inside LLVM (much like llvm-abi). Each solution
> has
> advantages and disadvantages and I'm planning to make a proposal
> about
> this on the mailing list later (hence I'd suggest leaving that
> discussion until I make the proposal).
> 
> In terms of the mentioned deficiencies of the Type structure, these
> are all problems that I intend to address (it would be great if you
> could provide details on any of the more difficult areas).
First, let me say that I also think this is an incredibly important area in
definite need of a solution. However, I also think that we really need to work
from Clang's logic here: We don't want to have two different
implementations of the ABI rules. They're complicated (and subtle), and it
is hard enough to have one correct implementation in our C/C++ compiler. Given
that we're talking about the C/C++ ABI, this is a perfectly appropriate
place for the logic to live. The issue, as you well know, is that other
languages need to interact with C/C++ functions, and so need to understand the
C/C++ ABI. My recommended strategy is this:

 1. Refactor the logic in Clang's lib/CodeGen/TargetInfo.cpp so that it can
be used by external clients (just as Clang's AST headers are available to
external clients).

 2. Write your library as a wrapper around that logic so that simple cases can
be handled without burdening other frontends with the subtleties of Clang's
AST data structures.

 -Hal
> 
> Thanks again,
> Stephen
> 
> On Mon, Jun 29, 2015 at 7:32 PM, Reid Kleckner <rnk at google.com>
> wrote:
> > I agree it would be really nice to build a library for ABI
> > lowering, but any
> > solution that isn't clang or isn't ultimately picked up by
clang
> > will
> > necessarily be incomplete. Perhaps that is OK for your frontend's
> > uses, but
> > I think it's the main reason that we haven't done something
like
> > this in
> > LLVM already.
> >
> > Any solution that doesn't involve actual Clang ASTs is unlikely to
> > be able
> > to represent all C-with-extensions types (unions, bitfields,
> > alignment
> > attributes, transparent_union attribute). I took a look at Type.hpp
> > in your
> > project, and it seems to be missing some of these things. Keeping
> > such a
> > library up to date with new extensions is going to be a maintenance
> > burden.
> >
> > That said, I wish you luck, and I hope the project eases some of
> > the
> > difficulties for new frontends. It is very possible that the corner
> > cases
> > that keep me up at night are not the problems that users actually
> > face. :)
> >
> > On Sun, Jun 28, 2015 at 3:22 PM, Stephen Cross
> > <scross at scross.co.uk> wrote:
> >>
> >> Hi everyone,
> >>
> >> (Also CC'ed cfe-dev since this seems relevant to Clang,
> >> particularly
> >> the questions at the end.)
> >>
> >> I've been working on a library to generate LLVM IR that
complies
> >> with
> >> platform ABIs (the current focus is on C but I'm also
interested
> >> in
> >> ABIs for other languages).
> >>
> >> You can find it here: https://github.com/scross99/llvm-abi
> >>
> >> To explain further (for those who are unfamiliar), LLVM frontends
> >> have
> >> to modify function argument types, attributes etc. in order to
> >> ensure
> >> the backend generates code that satisfies the ABI; this is needed
> >> because LLVM's type system can't encode all the necessary
> >> information.
> >> This is a complex task and involves substantial amounts of
> >> target-dependent logic. Clang performs this encoding and indeed
> >> much
> >> of the current functionality is derived from Clang's source.
> >>
> >> This project originated as a necessary piece of functionality for
> >> the
> >> Loci compiler frontend [1] and is now an external dependency; my
> >> aim
> >> is to make this usable for other LLVM frontends that also need to
> >> generate ABI-compliant IR (I assume this is a fairly large subset
> >> of
> >> the frontends).
> >>
> >> I'd be very interested in any suggestions/queries/comments.
> >>
> >> I made a few interesting discoveries while working on this:
> >>
> >> * Clang generates 8 byte alignment for 16+ byte arrays on x86-64,
> >> even
> >> though the AMD64 ABI seems to require that arrays of 16+ bytes are
> >> aligned to 16 bytes. Is this a bug or am I missing something
> >> obvious?
> >>
> >> * Clang determines the features for a CPU (e.g. whether we have
> >> AVX
> >> support on x86-64 CPUs), even though this functionality is already
> >> available in LLVM (but appears to be very difficult to query).
> >> Would
> >> it be possible to expose the information from LLVM and hence
> >> eliminate
> >> the duplication in Clang?
> >>
> >> * Clang determines a 'generic CPU' if the user doesn't
specify a
> >> CPU.
> >> My understanding is that we don't usually generate code for
the
> >> native
> >> CPU because it may have features unavailable on other similar
> >> CPUs.
> >> LLVM can provide full details of the native CPU but can't
> >> determine a
> >> generic CPU; could this functionality be added to LLVM?
> >>
> >> (Separately I've also been considering a proposal to add ABI
> >> information directly inside LLVM IR in a language-independent way
> >> and
> >> I'll discuss this in a later email.)
> >>
> >> Thanks,
> >> Stephen
> >>
> >> [1] https://github.com/scross99/locic
> >> _______________________________________________
> >> cfe-dev mailing list
> >> cfe-dev at cs.uiuc.edu
> >> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
> >
> >
> _______________________________________________
> cfe-dev mailing list
> cfe-dev at cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
> 
-- 
Hal Finkel
Assistant Computational Scientist
Leadership Computing Facility
Argonne National Laboratory

On Tue, 30 Jun 2015 at 06:02 Hal Finkel <hfinkel at anl.gov> wrote:
> ----- Original Message -----
> > From: "Stephen Cross" <scross at scross.co.uk>
> > To: "Reid Kleckner" <rnk at google.com>
> > Cc: "Clang Developers List" <cfe-dev at cs.uiuc.edu>,
"LLVM Developers
> Mailing List" <llvmdev at cs.uiuc.edu>
> > Sent: Monday, June 29, 2015 3:03:40 PM
> > Subject: Re: [cfe-dev] llvm-abi: A library for generating
ABI-compliant
> LLVM  IR
> >
> > Hi Reid,
> >
> > Thanks for your response.
> >
> > The issue is that every LLVM frontend needing ABI compliance has to
> > re-implement the same target-dependent logic, which is a significant
> > burden; the ABI compliance code inside Clang isn't really usable
for
> > other frontends as-is. We haven't got many good options here :-).
I
> > think a lot of people would've hoped that LLVM would provide the
> > means
> > for achieving ABI compliance (at least for C), though I'm well
aware
> > of the complexity involved and understand the decisions taken thus
> > far.
> >
> > Within this context, llvm-abi is an immediately actionable way of
> > sharing code between frontends, that should lead to a higher quality
> > codebase for a significantly reduced effort on everyone's part. It
> > seems like this would further build on LLVM's success because I
think
> > many people would like to generate code that can interact with C
> > APIs.
> >
> > Ultimately it would be great to see this functionality be provided in
> > an accessible form inside LLVM and hence for Clang to use that
> > functionality. This would move much of the target-dependent logic out
> > of Clang while at the same time making this functionality available
> > to
> > other LLVM-based tools.
> >
> > I am sure this is a substantial and complex long term project, but I
> > think it is a worthwhile aim. Do you (and the LLVM community as a
> > whole) agree? (Acknowledging that some of the details would need to
> > be
> > worked out.)
> >
> > I have a long term and substantial interest in this (as I expect do
> > other frontend developers), so I'm willing to contribute
significant
> > effort to move this forward. Hopefully the llvm-abi library can
> > provide a better understanding of what needs to be represented and
> > help non-Clang frontends :-).
> >
> > Implementation-wise the changes to LLVM could involve encoding ABI
> > information into LLVM IR or simply providing C++ APIs for generating
> > ABI-compliant code inside LLVM (much like llvm-abi). Each solution
> > has
> > advantages and disadvantages and I'm planning to make a proposal
> > about
> > this on the mailing list later (hence I'd suggest leaving that
> > discussion until I make the proposal).
> >
> > In terms of the mentioned deficiencies of the Type structure, these
> > are all problems that I intend to address (it would be great if you
> > could provide details on any of the more difficult areas).
>
> First, let me say that I also think this is an incredibly important area
> in definite need of a solution. However, I also think that we really need
> to work from Clang's logic here: We don't want to have two
different
> implementations of the ABI rules. They're complicated (and subtle), and
it
> is hard enough to have one correct implementation in our C/C++ compiler.
> Given that we're talking about the C/C++ ABI, this is a perfectly
> appropriate place for the logic to live. The issue, as you well know, is
> that other languages need to interact with C/C++ functions, and so need to
> understand the C/C++ ABI. My recommended strategy is this:
>
>  1. Refactor the logic in Clang's lib/CodeGen/TargetInfo.cpp so that it
> can be used by external clients (just as Clang's AST headers are
available
> to external clients).
>
>  2. Write your library as a wrapper around that logic so that simple cases
> can be handled without burdening other frontends with the subtleties of
> Clang's AST data structures.
>
This is what I was thinking of doing for llgo in the long term. At the
moment we have support for x86-64 only:
http://llvm.org/klaus/llgo/blob/master/irgen/cabi.go. It would be nice to
have a simplified API to TargetInfo, ideally with a C API that other
languages to can easily bind to. Stephen, if you do go down that route, I'd
be keen to hear about it.

Cheers,
Andrew

 -Hal
>
> >
> > Thanks again,
> > Stephen
> >
> > On Mon, Jun 29, 2015 at 7:32 PM, Reid Kleckner <rnk at
google.com>
> > wrote:
> > > I agree it would be really nice to build a library for ABI
> > > lowering, but any
> > > solution that isn't clang or isn't ultimately picked up
by clang
> > > will
> > > necessarily be incomplete. Perhaps that is OK for your
frontend's
> > > uses, but
> > > I think it's the main reason that we haven't done
something like
> > > this in
> > > LLVM already.
> > >
> > > Any solution that doesn't involve actual Clang ASTs is
unlikely to
> > > be able
> > > to represent all C-with-extensions types (unions, bitfields,
> > > alignment
> > > attributes, transparent_union attribute). I took a look at
Type.hpp
> > > in your
> > > project, and it seems to be missing some of these things. Keeping
> > > such a
> > > library up to date with new extensions is going to be a
maintenance
> > > burden.
> > >
> > > That said, I wish you luck, and I hope the project eases some of
> > > the
> > > difficulties for new frontends. It is very possible that the
corner
> > > cases
> > > that keep me up at night are not the problems that users actually
> > > face. :)
> > >
> > > On Sun, Jun 28, 2015 at 3:22 PM, Stephen Cross
> > > <scross at scross.co.uk> wrote:
> > >>
> > >> Hi everyone,
> > >>
> > >> (Also CC'ed cfe-dev since this seems relevant to Clang,
> > >> particularly
> > >> the questions at the end.)
> > >>
> > >> I've been working on a library to generate LLVM IR that
complies
> > >> with
> > >> platform ABIs (the current focus is on C but I'm also
interested
> > >> in
> > >> ABIs for other languages).
> > >>
> > >> You can find it here: https://github.com/scross99/llvm-abi
> > >>
> > >> To explain further (for those who are unfamiliar), LLVM
frontends
> > >> have
> > >> to modify function argument types, attributes etc. in order
to
> > >> ensure
> > >> the backend generates code that satisfies the ABI; this is
needed
> > >> because LLVM's type system can't encode all the
necessary
> > >> information.
> > >> This is a complex task and involves substantial amounts of
> > >> target-dependent logic. Clang performs this encoding and
indeed
> > >> much
> > >> of the current functionality is derived from Clang's
source.
> > >>
> > >> This project originated as a necessary piece of functionality
for
> > >> the
> > >> Loci compiler frontend [1] and is now an external dependency;
my
> > >> aim
> > >> is to make this usable for other LLVM frontends that also
need to
> > >> generate ABI-compliant IR (I assume this is a fairly large
subset
> > >> of
> > >> the frontends).
> > >>
> > >> I'd be very interested in any
suggestions/queries/comments.
> > >>
> > >> I made a few interesting discoveries while working on this:
> > >>
> > >> * Clang generates 8 byte alignment for 16+ byte arrays on
x86-64,
> > >> even
> > >> though the AMD64 ABI seems to require that arrays of 16+
bytes are
> > >> aligned to 16 bytes. Is this a bug or am I missing something
> > >> obvious?
> > >>
> > >> * Clang determines the features for a CPU (e.g. whether we
have
> > >> AVX
> > >> support on x86-64 CPUs), even though this functionality is
already
> > >> available in LLVM (but appears to be very difficult to
query).
> > >> Would
> > >> it be possible to expose the information from LLVM and hence
> > >> eliminate
> > >> the duplication in Clang?
> > >>
> > >> * Clang determines a 'generic CPU' if the user
doesn't specify a
> > >> CPU.
> > >> My understanding is that we don't usually generate code
for the
> > >> native
> > >> CPU because it may have features unavailable on other similar
> > >> CPUs.
> > >> LLVM can provide full details of the native CPU but can't
> > >> determine a
> > >> generic CPU; could this functionality be added to LLVM?
> > >>
> > >> (Separately I've also been considering a proposal to add
ABI
> > >> information directly inside LLVM IR in a language-independent
way
> > >> and
> > >> I'll discuss this in a later email.)
> > >>
> > >> Thanks,
> > >> Stephen
> > >>
> > >> [1] https://github.com/scross99/locic
> > >> _______________________________________________
> > >> cfe-dev mailing list
> > >> cfe-dev at cs.uiuc.edu
> > >> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
> > >
> > >
> > _______________________________________________
> > cfe-dev mailing list
> > cfe-dev at cs.uiuc.edu
> > http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
> >
>
> --
> Hal Finkel
> Assistant Computational Scientist
> Leadership Computing Facility
> Argonne National Laboratory
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20150629/1f01d210/attachment.html>