llvm dev - Apr 2016 - [cfe-dev] [Openmp-dev] RFC: Proposing an LLVM subproject for parallelism runtime and support libraries

Chandler,

It seems that the general consensus is to save further discussions for
later and go ahead with your proposal. I can add my +1 to this as well.

Could you / Jason prepare a proposal on the new project, with all the usual
questions covered? One interesting thing is where to put SE and
libomptarget in the project's tree. I would be happy to review it from
Intel / OpenMP side.

Also, as Carlo noted, libomptarget is currently under review (a bit stalled
one, if I may say so). Do we expect SE to undergo through a similar review
-- piece by piece -- as well?

Yours,
Andrey
=====Software Engineer
Intel Compiler Team


On Sat, Apr 23, 2016 at 1:24 AM, Chandler Carruth via Openmp-dev <
openmp-dev at lists.llvm.org> wrote:
> On Fri, Apr 22, 2016 at 3:05 PM Mehdi Amini <mehdi.amini at
apple.com> wrote:
>
>>
>> > On Apr 22, 2016, at 3:01 PM, Chandler Carruth <chandlerc at
gmail.com>
>> wrote:
>> >
>> > I feel like this thread got a bit stalled. I'd like to pick it
up and
>> try to suggest a path forward.
>> >
>> > I don't hear any real objections to the overall idea of having
an LLVM
>> subproject for parallelism runtimes and support libraries. I think we
>> should get that created.
>>
>> I think it should be clarified if "parallelism runtimes and
support
>> libraries" are intended to expose user-level APIs or if these are
intended
>> to expose APIs for the compiler generated code (this may be part of
your
>> point about "writing up its charter, scope" but I also think
it shouldn't
>> be underestimated as a task so I called it out).
>>
>
> Absolutely. I think that needs to be clearly spelled out.
>
> Personally, I'd like to see the subproject open to *both*. Here are
some
> libraries I would love to see (but don't necessarily have concrete
plans
> around):
> - A nice vectorized math library
> - Linear algebra libraries like BLAS implementations or such
> - Highly tuned FFT or other domain specific libraries for GPUs.
> Essentially the same is the vectorized math libraries but for GPUs and
> slightly higher level.
> - Stream executor
> - Any generic components of the OpenMP libraries.
>
> Clearly each of these would need to be discussed on a case by case basis,
> but there seems to be a healthy mixture of both user-level APIs and
> compiler-level APIs. I would suggest criteria for being here along the
> lines of:
>
> - Includes compiler-targeted APIs (maybe in addition to user-level APIs,
> maybe even with overlap), or
> - Leverages compiler details for its implementation (for example, using
> vector extensions we know LLVM supports), or
> - Wants to use compiler-specific packaging techniques or other integration
> techniques (for example shipping as bitcode), or
> - Helps support compiler or programming language functionality
>
> The first three here seem clear cut to me. If any part of the library is
> intended to be callable by the compiler, its a good fit. SE has such
> interfaces. Vectorized math libraries do too, etc. If the implementation of
> th elibrary really wants to use compiler internals like our vector math
> extensions, again, I think it makes sense to keep it reasonably co-located
> with the compiler.
>
> The last seems a bit tricky, but I think its really important. Currently,
> CUDA provides a pretty big programming surface, and having a well tuned
> BLAS or FFT implementation for example that integrates with CUDA is pretty
> important. Similarly in the future, we expect C++ to get lots of parallel
> standard library interfaces, potentially even BLAS-looking ones and we
> might want a good parallel BLAS implementation or other very fundamental
> parallel library implementation to use when implementing it.
>
> But at the same time, I think its really important to have a clear place
> where any library here ties back into the compiler ecosystem and/or the
> programming language ecosystem that are the core of LLVM.
>
> Does this seem like its going in the right direction? (Jason can probably
> take on the non-trivial task of writing this up more formally and make sure
> it is clearly documented.)
>
>
>> Otherwise you plan sounds good to me.
>>
>> --
>> Mehdi
>>
>>
>>
>> >
>> > I don't actually see any real objections to StreamExecutor
being one of
>> the runtimes. There are some interesting questions however:
>> > - Is there common code in the OpenMP runtime that could be unified
with
>> this?
>> > - Could OpenMP end up using SE or some common shared library
between
>> them as a basis for offloading?
>> > - Would instead it make more sense to have the OpenMP offload
library
>> be a plugin for StreamExecutor?
>> >
>> > I don't know the answer to any of these really, but I also
don't think
>> that they should prevent us from making progress here. And I think if
>> anything, they'll become easier to answer if we do.
>> >
>> > So my suggestion would be:
>> > 1) Create the broader scoped LLVM subproject, including writing up
its
>> charter, scope, plans, etc.
>> >
>> > 2) Add stream executor to it
>> >
>> > 3) Initially, leave the OpenMP offloading stuff targeted at
OpenMP.
>> Then, as it evolves, consider moving it to be another runtime in the
broad
>> project if and when it makes sense.
>> >
>> > 4) As both OpenMP and SE evolve and are used some in the project,
>> evaluate whether there is a common core that makes sense to extract. If
so,
>> do it and rebase them appropriately.
>> >
>> >
>> > Does this make sense? Are there objections to moving forward here?
>>
>>
> _______________________________________________
> Openmp-dev mailing list
> Openmp-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/openmp-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160426/292781a9/attachment.html>

Ahh, I just noticed that Chandler's proposal is to put SE only into this
new project, and to keep libomptarget separately, in OpenMP project. I
wonder why so? Why SE (a library serving only one PPM so far) is different
from libomptarget (a library also serving only one PPM so far)?

Are people have opinion on this?

Yours,
Andrey
====Software Engineer
Intel Compiler Team

On Tue, Apr 26, 2016 at 6:06 PM, Andrey Bokhanko <andreybokhanko at
gmail.com>
wrote:
> Chandler,
>
> It seems that the general consensus is to save further discussions for
> later and go ahead with your proposal. I can add my +1 to this as well.
>
> Could you / Jason prepare a proposal on the new project, with all the
> usual questions covered? One interesting thing is where to put SE and
> libomptarget in the project's tree. I would be happy to review it from
> Intel / OpenMP side.
>
> Also, as Carlo noted, libomptarget is currently under review (a bit
> stalled one, if I may say so). Do we expect SE to undergo through a similar
> review -- piece by piece -- as well?
>
> Yours,
> Andrey
> =====> Software Engineer
> Intel Compiler Team
>
>
> On Sat, Apr 23, 2016 at 1:24 AM, Chandler Carruth via Openmp-dev <
> openmp-dev at lists.llvm.org> wrote:
>
>> On Fri, Apr 22, 2016 at 3:05 PM Mehdi Amini <mehdi.amini at
apple.com>
>> wrote:
>>
>>>
>>> > On Apr 22, 2016, at 3:01 PM, Chandler Carruth <chandlerc at
gmail.com>
>>> wrote:
>>> >
>>> > I feel like this thread got a bit stalled. I'd like to
pick it up and
>>> try to suggest a path forward.
>>> >
>>> > I don't hear any real objections to the overall idea of
having an LLVM
>>> subproject for parallelism runtimes and support libraries. I think
we
>>> should get that created.
>>>
>>> I think it should be clarified if "parallelism runtimes and
support
>>> libraries" are intended to expose user-level APIs or if these
are intended
>>> to expose APIs for the compiler generated code (this may be part of
your
>>> point about "writing up its charter, scope" but I also
think it shouldn't
>>> be underestimated as a task so I called it out).
>>>
>>
>> Absolutely. I think that needs to be clearly spelled out.
>>
>> Personally, I'd like to see the subproject open to *both*. Here are
some
>> libraries I would love to see (but don't necessarily have concrete
plans
>> around):
>> - A nice vectorized math library
>> - Linear algebra libraries like BLAS implementations or such
>> - Highly tuned FFT or other domain specific libraries for GPUs.
>> Essentially the same is the vectorized math libraries but for GPUs and
>> slightly higher level.
>> - Stream executor
>> - Any generic components of the OpenMP libraries.
>>
>> Clearly each of these would need to be discussed on a case by case
basis,
>> but there seems to be a healthy mixture of both user-level APIs and
>> compiler-level APIs. I would suggest criteria for being here along the
>> lines of:
>>
>> - Includes compiler-targeted APIs (maybe in addition to user-level
APIs,
>> maybe even with overlap), or
>> - Leverages compiler details for its implementation (for example, using
>> vector extensions we know LLVM supports), or
>> - Wants to use compiler-specific packaging techniques or other
>> integration techniques (for example shipping as bitcode), or
>> - Helps support compiler or programming language functionality
>>
>> The first three here seem clear cut to me. If any part of the library
is
>> intended to be callable by the compiler, its a good fit. SE has such
>> interfaces. Vectorized math libraries do too, etc. If the
implementation of
>> th elibrary really wants to use compiler internals like our vector math
>> extensions, again, I think it makes sense to keep it reasonably
co-located
>> with the compiler.
>>
>> The last seems a bit tricky, but I think its really important.
Currently,
>> CUDA provides a pretty big programming surface, and having a well tuned
>> BLAS or FFT implementation for example that integrates with CUDA is
pretty
>> important. Similarly in the future, we expect C++ to get lots of
parallel
>> standard library interfaces, potentially even BLAS-looking ones and we
>> might want a good parallel BLAS implementation or other very
fundamental
>> parallel library implementation to use when implementing it.
>>
>> But at the same time, I think its really important to have a clear
place
>> where any library here ties back into the compiler ecosystem and/or the
>> programming language ecosystem that are the core of LLVM.
>>
>> Does this seem like its going in the right direction? (Jason can
probably
>> take on the non-trivial task of writing this up more formally and make
sure
>> it is clearly documented.)
>>
>>
>>> Otherwise you plan sounds good to me.
>>>
>>> --
>>> Mehdi
>>>
>>>
>>>
>>> >
>>> > I don't actually see any real objections to StreamExecutor
being one
>>> of the runtimes. There are some interesting questions however:
>>> > - Is there common code in the OpenMP runtime that could be
unified
>>> with this?
>>> > - Could OpenMP end up using SE or some common shared library
between
>>> them as a basis for offloading?
>>> > - Would instead it make more sense to have the OpenMP offload
library
>>> be a plugin for StreamExecutor?
>>> >
>>> > I don't know the answer to any of these really, but I also
don't think
>>> that they should prevent us from making progress here. And I think
if
>>> anything, they'll become easier to answer if we do.
>>> >
>>> > So my suggestion would be:
>>> > 1) Create the broader scoped LLVM subproject, including
writing up its
>>> charter, scope, plans, etc.
>>> >
>>> > 2) Add stream executor to it
>>> >
>>> > 3) Initially, leave the OpenMP offloading stuff targeted at
OpenMP.
>>> Then, as it evolves, consider moving it to be another runtime in
the broad
>>> project if and when it makes sense.
>>> >
>>> > 4) As both OpenMP and SE evolve and are used some in the
project,
>>> evaluate whether there is a common core that makes sense to
extract. If so,
>>> do it and rebase them appropriately.
>>> >
>>> >
>>> > Does this make sense? Are there objections to moving forward
here?
>>>
>>>
>> _______________________________________________
>> Openmp-dev mailing list
>> Openmp-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/openmp-dev
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160427/991c5baa/attachment-0001.html>

I respect Hal's more tactful approach and response..

Let me play devils advocate for a minute

1) Yet another programming model - Is the advantage spelled out
somewhere? (I know there are reasons, but I'd like to see a FAQ or
this clearly documented. Examples pretty please.. More for long term
than my own selfish benefit)
2) Is this an "open standard" - If I wanted to propose a major change
to SE - how would I or someone else go about it? OpenMP/ACC have more
or less clearly defined paths for new features.. What's the governing
policy here.. Bug fixes are easy to deal with, but does Google have
final say on the roadmap..
3) When the project is created - will it include lots of good tests?
4) It's probably used internally @google - who else will be using
this? Is the target HPC, Android.. etc

Lastly - sorry, but I don't like this kick-the-can approach to what
should be proper engineering and planning upfront. Can someone @google
gentleman's promise to actively work on playing nice with other
projects, specifically OpenMP and Intel. From my perspective nothing
stops Google from tossing it up on github or google code and letting
it stay there until all the pieces are in the correct place. Why it
*must* be an llvm project now doesn't make sense to me. When the shoe
was on the other foot (OpenMP) there was all sorts of shit and redtape
Intel (and others) had to jump around to get it included. Google has a
lot of good karma in the llvm community and maybe that's the
difference..

----- Original Message -----
> From: "Andrey Bokhanko via Openmp-dev" <openmp-dev at
lists.llvm.org>
> To: "Chandler Carruth" <chandlerc at gmail.com>
> Cc: "llvm-dev" <llvm-dev at lists.llvm.org>, "Mehdi
Amini"
> <mehdi.amini at apple.com>, "cfe-dev" <cfe-dev at
lists.llvm.org>,
> openmp-dev at lists.llvm.org
> Sent: Wednesday, April 27, 2016 11:12:17 AM
> Subject: Re: [Openmp-dev] [llvm-dev] [cfe-dev] RFC: Proposing an LLVM
> subproject for parallelism runtime and support libraries
> Ahh, I just noticed that Chandler's proposal is to put SE only into
> this new project, and to keep libomptarget separately, in OpenMP
> project. I wonder why so? Why SE (a library serving only one PPM so
> far) is different from libomptarget (a library also serving only one
> PPM so far)?
> Are people have opinion on this?
I just clarified with Chandler that he proposes both to go into the new project.
I agree that this is what we should do.

-Hal 
> Yours,
> Andrey
> ====
> Software Engineer
> Intel Compiler Team
> On Tue, Apr 26, 2016 at 6:06 PM, Andrey Bokhanko <
> andreybokhanko at gmail.com > wrote:
> > Chandler,
> 
> > It seems that the general consensus is to save further discussions
> > for later and go ahead with your proposal. I can add my +1 to this
> > as well.
> 
> > Could you / Jason prepare a proposal on the new project, with all
> > the
> > usual questions covered? One interesting thing is where to put SE
> > and libomptarget in the project's tree. I would be happy to review
> > it from Intel / OpenMP side.
> 
> > Also, as Carlo noted, libomptarget is currently under review (a bit
> > stalled one, if I may say so). Do we expect SE to undergo through a
> > similar review -- piece by piece -- as well?
> 
> > Yours,
> 
> > Andrey
> 
> > =====> 
> > Software Engineer
> 
> > Intel Compiler Team
> 
> > On Sat, Apr 23, 2016 at 1:24 AM, Chandler Carruth via Openmp-dev <
> > openmp-dev at lists.llvm.org > wrote:
> 
> > > On Fri, Apr 22, 2016 at 3:05 PM Mehdi Amini <
> > > mehdi.amini at apple.com
> > > >
> > > wrote:
> > 
> 
> > > > > On Apr 22, 2016, at 3:01 PM, Chandler Carruth <
> > > > > chandlerc at gmail.com
> > > > > > wrote:
> > > 
> > 
> 
> > > > >
> > > 
> > 
> 
> > > > > I feel like this thread got a bit stalled. I'd like
to pick
> > > > > it
> > > > > up
> > > > > and try to suggest a path forward.
> > > 
> > 
> 
> > > > >
> > > 
> > 
> 
> > > > > I don't hear any real objections to the overall
idea of
> > > > > having
> > > > > an
> > > > > LLVM subproject for parallelism runtimes and support
> > > > > libraries.
> > > > > I
> > > > > think we should get that created.
> > > 
> > 
> 
> > > > I think it should be clarified if "parallelism runtimes
and
> > > > support
> > > > libraries" are intended to expose user-level APIs or if
these
> > > > are
> > > > intended to expose APIs for the compiler generated code
(this
> > > > may
> > > > be
> > > > part of your point about "writing up its charter,
scope" but I
> > > > also
> > > > think it shouldn't be underestimated as a task so I
called it
> > > > out).
> > > 
> > 
> 
> > > Absolutely. I think that needs to be clearly spelled out.
> > 
> 
> > > Personally, I'd like to see the subproject open to *both*.
Here
> > > are
> > > some libraries I would love to see (but don't necessarily
have
> > > concrete plans around):
> > 
> 
> > > - A nice vectorized math library
> > 
> 
> > > - Linear algebra libraries like BLAS implementations or such
> > 
> 
> > > - Highly tuned FFT or other domain specific libraries for GPUs.
> > > Essentially the same is the vectorized math libraries but for
> > > GPUs
> > > and slightly higher level.
> > 
> 
> > > - Stream executor
> > 
> 
> > > - Any generic components of the OpenMP libraries.
> > 
> 
> > > Clearly each of these would need to be discussed on a case by
> > > case
> > > basis, but there seems to be a healthy mixture of both user-level
> > > APIs and compiler-level APIs. I would suggest criteria for being
> > > here along the lines of:
> > 
> 
> > > - Includes compiler-targeted APIs (maybe in addition to
> > > user-level
> > > APIs, maybe even with overlap), or
> > 
> 
> > > - Leverages compiler details for its implementation (for example,
> > > using vector extensions we know LLVM supports), or
> > 
> 
> > > - Wants to use compiler-specific packaging techniques or other
> > > integration techniques (for example shipping as bitcode), or
> > 
> 
> > > - Helps support compiler or programming language functionality
> > 
> 
> > > The first three here seem clear cut to me. If any part of the
> > > library
> > > is intended to be callable by the compiler, its a good fit. SE
> > > has
> > > such interfaces. Vectorized math libraries do too, etc. If the
> > > implementation of th elibrary really wants to use compiler
> > > internals
> > > like our vector math extensions, again, I think it makes sense to
> > > keep it reasonably co-located with the compiler.
> > 
> 
> > > The last seems a bit tricky, but I think its really important.
> > > Currently, CUDA provides a pretty big programming surface, and
> > > having a well tuned BLAS or FFT implementation for example that
> > > integrates with CUDA is pretty important. Similarly in the
> > > future,
> > > we expect C++ to get lots of parallel standard library
> > > interfaces,
> > > potentially even BLAS-looking ones and we might want a good
> > > parallel
> > > BLAS implementation or other very fundamental parallel library
> > > implementation to use when implementing it.
> > 
> 
> > > But at the same time, I think its really important to have a
> > > clear
> > > place where any library here ties back into the compiler
> > > ecosystem
> > > and/or the programming language ecosystem that are the core of
> > > LLVM.
> > 
> 
> > > Does this seem like its going in the right direction? (Jason can
> > > probably take on the non-trivial task of writing this up more
> > > formally and make sure it is clearly documented.)
> > 
> 
> > > > Otherwise you plan sounds good to me.
> > > 
> > 
> 
> > > > --
> > > 
> > 
> 
> > > > Mehdi
> > > 
> > 
> 
> > > > >
> > > 
> > 
> 
> > > > > I don't actually see any real objections to
StreamExecutor
> > > > > being
> > > > > one of the runtimes. There are some interesting
questions
> > > > > however:
> > > 
> > 
> 
> > > > > - Is there common code in the OpenMP runtime that could
be
> > > > > unified
> > > > > with this?
> > > 
> > 
> 
> > > > > - Could OpenMP end up using SE or some common shared
library
> > > > > between them as a basis for offloading?
> > > 
> > 
> 
> > > > > - Would instead it make more sense to have the OpenMP
offload
> > > > > library be a plugin for StreamExecutor?
> > > 
> > 
> 
> > > > >
> > > 
> > 
> 
> > > > > I don't know the answer to any of these really, but
I also
> > > > > don't
> > > > > think that they should prevent us from making progress
here.
> > > > > And
> > > > > I
> > > > > think if anything, they'll become easier to answer
if we do.
> > > 
> > 
> 
> > > > >
> > > 
> > 
> 
> > > > > So my suggestion would be:
> > > 
> > 
> 
> > > > > 1) Create the broader scoped LLVM subproject, including
> > > > > writing
> > > > > up
> > > > > its charter, scope, plans, etc.
> > > 
> > 
> 
> > > > >
> > > 
> > 
> 
> > > > > 2) Add stream executor to it
> > > 
> > 
> 
> > > > >
> > > 
> > 
> 
> > > > > 3) Initially, leave the OpenMP offloading stuff
targeted at
> > > > > OpenMP.
> > > > > Then, as it evolves, consider moving it to be another
runtime
> > > > > in
> > > > > the broad project if and when it makes sense.
> > > 
> > 
> 
> > > > >
> > > 
> > 
> 
> > > > > 4) As both OpenMP and SE evolve and are used some in
the
> > > > > project,
> > > > > evaluate whether there is a common core that makes
sense to
> > > > > extract. If so, do it and rebase them appropriately.
> > > 
> > 
> 
> > > > >
> > > 
> > 
> 
> > > > >
> > > 
> > 
> 
> > > > > Does this make sense? Are there objections to moving
forward
> > > > > here?
> > > 
> > 
> 
> > > _______________________________________________
> > 
> 
> > > Openmp-dev mailing list
> > 
> 
> > > Openmp-dev at lists.llvm.org
> > 
> 
> > > http://lists.llvm.org/cgi-bin/mailman/listinfo/openmp-dev
> > 
> 
> _______________________________________________
> Openmp-dev mailing list
> Openmp-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/openmp-dev
-- 

Hal Finkel 
Assistant Computational Scientist 
Leadership Computing Facility 
Argonne National Laboratory 
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20160531/7c398b1c/attachment.html>