llvm dev - Jun 2016 - [Openmp-dev] [cfe-dev] RFC: Proposing an LLVM subproject for parallelism runtime and support libraries

On Wed, Jun 1, 2016 at 7:22 PM, Hal Finkel <hfinkel at anl.gov>
wrote:
>
> ________________________________
>
> From: "C Bergström" <cbergstrom at pathscale.com>
> To: "Hal Finkel" <hfinkel at anl.gov>
> Cc: "llvm-dev" <llvm-dev at lists.llvm.org>,
"cfe-dev"
> <cfe-dev at lists.llvm.org>, "openmp-dev" <openmp-dev at
lists.llvm.org>,
> "Chandler Carruth" <chandlerc at gmail.com>, "Carlo
Bertolli"
> <cbertol at us.ibm.com>, "Andrey Bokhanko"
<andreybokhanko at gmail.com>
> Sent: Wednesday, June 1, 2016 12:19:19 AM
> Subject: Re: [Openmp-dev] [llvm-dev] [cfe-dev] RFC: Proposing an LLVM
> subproject for parallelism runtime and support libraries
>
> The thread has lost focus and cherry picking replies..
>
> To restate things since maybe you missed my points
> ----
> 1) SE is a programming model and needs a home of it's own. Having a
> programming model with it's headers and all other stuff glued into a
runtime
> project which intends to be universal and PM agnostic doesn't make
sense.
>
> They'd start in separate subdirectories.
>
>
> 1.1) The more I look, the most it seems SE is just a step-child project and
> stuffing it in llvm while still not having any users or strong backing
> doesn't make sense. We have enough PM already and my gut feeling is
this
> isn't going in a direction to bring in other stakeholders.
Point by point because I don't agree with what you write below entirely
>
> I think this is the core of my reply. OpenMP has a strong user community,
I'd agree here
> but OpenMP 4 offloading is still young.
ditto
> OpenMP 4 offloading does not yet
> have a real user community
agreement
> yet because the first implementations just
> started shipping very recently.
Partially strongly disagree - Maybe you meant on Power8?
Intel has had their support for OMP4 for quite some time. It may have
been buggy and focused on simd directive, but as best I can tell they
have worked quite hard to fix bugs and improve it.

(Shameless self promotion) We have had some degree of OMP4 offloading
for like 1.5 years now.. It's mostly targeting the GPU and x86, but
also more recently working on Power8/AArch64.

I'm quite certain these companies all have varying degrees of OMP4
done: Cray, IBM, PGI
> Furthermore, our implementation is certainly
> quite new, and OpenMP 4 offloading is really quite akin to SE in that
> regard.
Strongly disagree - Intel has been working with the LLVM community on
the parsing/sema and other parts of OMP4 for like a year or more..
This has been a long and well vetted process backed back a well
defined standard. Compared to SE which is just some thing that popped
up out of nowhere and has a couple people from Google throwing their
weight around.
> I view them both as experimental projects, and both have strong
> backing with significant investment, so I expect both to mature over time.
I'd agree both are experimental and probably need some work. I'd
strongly disagree the amount of investment of both projects is equal.
Again - OMP4 may have zero real world users, but there's a lot of
stubborn people trying to make it work. Compared to SE which the
people involved have yet to even answer basic things - like what's the
future of SE and how to contribute to it.
> Our non-subtle strategic goal as a community should be to encourage the
> various teams to take advantage of each others expertise in the most
> practical way.
I think we need to pushback against more programming models, but I
agree we should encourage lots of great patches around things like the
pass manager.

IMHO it's more productive to spend time trying to improve the
programming models and standards we have than experimental fun ideas.
---------
Can you point me at something concrete which SE would allow you to do
which an existing model can't?

I'll shutup and apologize if real side by side comparisons for the
beauty of SE come out.

----- Original Message -----
> From: "C Bergström" <cbergstrom at pathscale.com>
> To: "Hal Finkel" <hfinkel at anl.gov>
> Cc: "llvm-dev" <llvm-dev at lists.llvm.org>,
"cfe-dev" <cfe-dev at lists.llvm.org>, "openmp-dev"
> <openmp-dev at lists.llvm.org>, "Chandler Carruth"
<chandlerc at gmail.com>, "Carlo Bertolli" <cbertol at
us.ibm.com>,
> "Andrey Bokhanko" <andreybokhanko at gmail.com>
> Sent: Wednesday, June 1, 2016 6:46:57 AM
> Subject: Re: [Openmp-dev] [llvm-dev] [cfe-dev] RFC: Proposing an LLVM
subproject for parallelism runtime and support
> libraries
> 
> On Wed, Jun 1, 2016 at 7:22 PM, Hal Finkel <hfinkel at anl.gov>
wrote:
> >
> > ________________________________
> >
> > From: "C Bergström" <cbergstrom at pathscale.com>
> > To: "Hal Finkel" <hfinkel at anl.gov>
> > Cc: "llvm-dev" <llvm-dev at lists.llvm.org>,
"cfe-dev"
> > <cfe-dev at lists.llvm.org>, "openmp-dev"
<openmp-dev at lists.llvm.org>,
> > "Chandler Carruth" <chandlerc at gmail.com>,
"Carlo Bertolli"
> > <cbertol at us.ibm.com>, "Andrey Bokhanko"
<andreybokhanko at gmail.com>
> > Sent: Wednesday, June 1, 2016 12:19:19 AM
> > Subject: Re: [Openmp-dev] [llvm-dev] [cfe-dev] RFC: Proposing an
> > LLVM
> > subproject for parallelism runtime and support libraries
> >
> > The thread has lost focus and cherry picking replies..
> >
> > To restate things since maybe you missed my points
> > ----
> > 1) SE is a programming model and needs a home of it's own. Having
a
> > programming model with it's headers and all other stuff glued into
> > a runtime
> > project which intends to be universal and PM agnostic doesn't make
> > sense.
> >
> > They'd start in separate subdirectories.
> >
> >
> > 1.1) The more I look, the most it seems SE is just a step-child
> > project and
> > stuffing it in llvm while still not having any users or strong
> > backing
> > doesn't make sense. We have enough PM already and my gut feeling
is
> > this
> > isn't going in a direction to bring in other stakeholders.
> 
> Point by point because I don't agree with what you write below
> entirely
> 
> >
> > I think this is the core of my reply. OpenMP has a strong user
> > community,
> 
> I'd agree here
> 
> > but OpenMP 4 offloading is still young.
> 
> ditto
> 
> > OpenMP 4 offloading does not yet
> > have a real user community
> 
> agreement
> 
> > yet because the first implementations just
> > started shipping very recently.
> 
> Partially strongly disagree - Maybe you meant on Power8?
> Intel has had their support for OMP4 for quite some time. It may have
> been buggy and focused on simd directive, but as best I can tell they
> have worked quite hard to fix bugs and improve it.
I meant across many platforms. Intel had preliminary support for OpenMP 4
offloading directives even before OpenMP 4 was standardized (in 2013). This did
not include all of what ended up in the standard, and the offloading part of the
standard needed to be fixed in a breaking way for OpenMP 4.1/4.5 regardless, so
this is certainly the exception rather than the rule.
> 
> (Shameless self promotion) We have had some degree of OMP4 offloading
> for like 1.5 years now.. It's mostly targeting the GPU and x86, but
> also more recently working on Power8/AArch64.
> 
> I'm quite certain these companies all have varying degrees of OMP4
> done: Cray, IBM, PGI
Yes, many are working on implementations, and some have shipped. There's a
list here: http://openmp.org/wp/openmp-compilers/ - no one here really lists any
OpenMP 4 offloading implementations before 2015. PGI does not currently list
OpenMP 4 at all (although they've certainly done a lot of work on OpenACC).
> 
> > Furthermore, our implementation is certainly
> > quite new, and OpenMP 4 offloading is really quite akin to SE in
> > that
> > regard.
> 
> Strongly disagree - Intel has been working with the LLVM community on
> the parsing/sema and other parts of OMP4 for like a year or more..
> This has been a long and well vetted process backed back a well
> defined standard.
Yes, both Intel and IBM have contributed significantly to Clang in this regard.
> Compared to SE which is just some thing that popped
> up out of nowhere and 
It popped up in TensorFlow, which itself popped up out of nowhere, but is
already a popular open-source project for machine learning.
> has a couple people from Google throwing their
> weight around.
Google is a trusted member of the LLVM community and a major contributor. So I
agree with you in the following sense: When Google promises to update the code
to follow LLVM's coding conventions and to manage its future development in
accordance with our community norms, I believe them. I think they've earned
a place in the "trust but verify" category in this regard, and I think
this is a positive thing.
 
> > I view them both as experimental projects, and both have strong
> > backing with significant investment, so I expect both to mature
> > over time.
> 
> I'd agree both are experimental and probably need some work. I'd
> strongly disagree the amount of investment of both projects is equal.
> Again - OMP4 may have zero real world users, but there's a lot of
> stubborn people trying to make it work. Compared to SE which the
> people involved have yet to even answer basic things - like what's
> the
> future of SE and how to contribute to it.
They might not know what the future is. They have code that is useful to them,
and they feel would be useful to others. There's strong potential for
compiler integration in the future, but it is just a library now. They've
talked about extending it to cover different backends (OpenCL, etc.), and
investigating host-based-task dependencies, etc. My impression is that, in
general, they'd like the code's users, present and future, to shape its
future. All of this seems perfectly healthy. If it becomes part of LLVM, how to
contribute to it will become clear.
> 
> > Our non-subtle strategic goal as a community should be to encourage
> > the
> > various teams to take advantage of each others expertise in the
> > most
> > practical way.
> 
> I think we need to pushback against more programming models, but I
> agree we should encourage lots of great patches around things like
> the
> pass manager.
> 
> IMHO it's more productive to spend time trying to improve the
> programming models and standards we have than experimental fun ideas.
Perhaps this is our fundamental disagreement. I still view experimentation in
the accelerator offloading space as appropriate and healthy, even though some
standards have emerged.
> ---------
> Can you point me at something concrete which SE would allow you to do
> which an existing model can't?
OpenMP has problems integrating cleanly with C++ abstractions, and having been
involved in some of the discussions on the OpenMP side of things, the path
forward here is yet unclear. DOE itself is developing several C++ parallelism
abstractions which support offloading (Kokkos, RAJA, etc.), which can backend on
OpenMP 4, but can be (and some have been) retargeted to other programming models
as well.

I still don't think we've figured out the right answer here.

Thanks again,
Hal
 
> I'll shutup and apologize if real side by side comparisons for the
> beauty of SE come out.
> 
-- 
Hal Finkel
Assistant Computational Scientist
Leadership Computing Facility
Argonne National Laboratory

On Wed, Jun 1, 2016 at 8:52 PM, Hal Finkel <hfinkel at anl.gov>
wrote:
> ----- Original Message -----
>> From: "C Bergström" <cbergstrom at pathscale.com>
>> To: "Hal Finkel" <hfinkel at anl.gov>
>> Cc: "llvm-dev" <llvm-dev at lists.llvm.org>,
"cfe-dev" <cfe-dev at lists.llvm.org>, "openmp-dev"
>> <openmp-dev at lists.llvm.org>, "Chandler Carruth"
<chandlerc at gmail.com>, "Carlo Bertolli" <cbertol at
us.ibm.com>,
>> "Andrey Bokhanko" <andreybokhanko at gmail.com>
>> Sent: Wednesday, June 1, 2016 6:46:57 AM
>> Subject: Re: [Openmp-dev] [llvm-dev] [cfe-dev] RFC: Proposing an LLVM
subproject for parallelism runtime and support
>> libraries
>>
>> On Wed, Jun 1, 2016 at 7:22 PM, Hal Finkel <hfinkel at anl.gov>
wrote:
>> >
>> > ________________________________
>> >
>> > From: "C Bergström" <cbergstrom at pathscale.com>
>> > To: "Hal Finkel" <hfinkel at anl.gov>
>> > Cc: "llvm-dev" <llvm-dev at lists.llvm.org>,
"cfe-dev"
>> > <cfe-dev at lists.llvm.org>, "openmp-dev"
<openmp-dev at lists.llvm.org>,
>> > "Chandler Carruth" <chandlerc at gmail.com>,
"Carlo Bertolli"
>> > <cbertol at us.ibm.com>, "Andrey Bokhanko"
<andreybokhanko at gmail.com>
>> > Sent: Wednesday, June 1, 2016 12:19:19 AM
>> > Subject: Re: [Openmp-dev] [llvm-dev] [cfe-dev] RFC: Proposing an
>> > LLVM
>> > subproject for parallelism runtime and support libraries
>> >
>> > The thread has lost focus and cherry picking replies..
>> >
>> > To restate things since maybe you missed my points
>> > ----
>> > 1) SE is a programming model and needs a home of it's own.
Having a
>> > programming model with it's headers and all other stuff glued
into
>> > a runtime
>> > project which intends to be universal and PM agnostic doesn't
make
>> > sense.
>> >
>> > They'd start in separate subdirectories.
>> >
>> >
>> > 1.1) The more I look, the most it seems SE is just a step-child
>> > project and
>> > stuffing it in llvm while still not having any users or strong
>> > backing
>> > doesn't make sense. We have enough PM already and my gut
feeling is
>> > this
>> > isn't going in a direction to bring in other stakeholders.
>>
>> Point by point because I don't agree with what you write below
>> entirely
>>
>> >
>> > I think this is the core of my reply. OpenMP has a strong user
>> > community,
>>
>> I'd agree here
>>
>> > but OpenMP 4 offloading is still young.
>>
>> ditto
>>
>> > OpenMP 4 offloading does not yet
>> > have a real user community
>>
>> agreement
>>
>> > yet because the first implementations just
>> > started shipping very recently.
>>
>> Partially strongly disagree - Maybe you meant on Power8?
>> Intel has had their support for OMP4 for quite some time. It may have
>> been buggy and focused on simd directive, but as best I can tell they
>> have worked quite hard to fix bugs and improve it.
>
> I meant across many platforms. Intel had preliminary support for OpenMP 4
offloading directives even before OpenMP 4 was standardized (in 2013). This did
not include all of what ended up in the standard, and the offloading part of the
standard needed to be fixed in a breaking way for OpenMP 4.1/4.5 regardless, so
this is certainly the exception rather than the rule.
>
>>
>> (Shameless self promotion) We have had some degree of OMP4 offloading
>> for like 1.5 years now.. It's mostly targeting the GPU and x86, but
>> also more recently working on Power8/AArch64.
>>
>> I'm quite certain these companies all have varying degrees of OMP4
>> done: Cray, IBM, PGI
>
> Yes, many are working on implementations, and some have shipped.
There's a list here: http://openmp.org/wp/openmp-compilers/ - no one here
really lists any OpenMP 4 offloading implementations before 2015. PGI does not
currently list OpenMP 4 at all (although they've certainly done a lot of
work on OpenACC).
>
>>
>> > Furthermore, our implementation is certainly
>> > quite new, and OpenMP 4 offloading is really quite akin to SE in
>> > that
>> > regard.
>>
>> Strongly disagree - Intel has been working with the LLVM community on
>> the parsing/sema and other parts of OMP4 for like a year or more..
>> This has been a long and well vetted process backed back a well
>> defined standard.
>
> Yes, both Intel and IBM have contributed significantly to Clang in this
regard.
>
>> Compared to SE which is just some thing that popped
>> up out of nowhere and
>
> It popped up in TensorFlow, which itself popped up out of nowhere, but is
already a popular open-source project for machine learning.
>
>> has a couple people from Google throwing their
>> weight around.
>
> Google is a trusted member of the LLVM community and a major contributor.
So I agree with you in the following sense: When Google promises to update the
code to follow LLVM's coding conventions and to manage its future
development in accordance with our community norms, I believe them. I think
they've earned a place in the "trust but verify" category in this
regard, and I think this is a positive thing.
>
yes, but not every project that is started is finished and who would
be the maintainer if the person leaves Google? I still don't see why
they can't fork it on github or create a project to let it bake and
get some users or traction.
>> > I view them both as experimental projects, and both have strong
>> > backing with significant investment, so I expect both to mature
>> > over time.
>>
>> I'd agree both are experimental and probably need some work.
I'd
>> strongly disagree the amount of investment of both projects is equal.
>> Again - OMP4 may have zero real world users, but there's a lot of
>> stubborn people trying to make it work. Compared to SE which the
>> people involved have yet to even answer basic things - like what's
>> the
>> future of SE and how to contribute to it.
>
> They might not know what the future is. They have code that is useful to
them, and they feel would be useful to others. There's strong potential for
compiler integration in the future, but it is just a library now. They've
talked about extending it to cover different backends (OpenCL, etc.), and
investigating host-based-task dependencies, etc. My impression is that, in
general, they'd like the code's users, present and future, to shape its
future. All of this seems perfectly healthy. If it becomes part of LLVM, how to
contribute to it will become clear.
I love your optimistic attitude, but the reality is that successful
programming models take a significant amount of work. I'm well aware
that they can handle at least part of that, but so far I'm not sure
who else will invest in it..

The industry has : CUDA, OpenCL, OpenMP4, OpenACC, C++AMP, Parallel
STL(??), some research projects from DOE and dead stuff like HMPP...
Are all these so broken that they can't be fixed to fill the same
necessity as SE?

How similar is it to the failed AMD Stream Computing (sorry AMD)
http://www.ele.uri.edu/courses/ele408/StreamGPU.pdf

Intel's work had to live and grow outside for quite a while - Google
may have more good karma, but I don't see this as being fair. Nothing
stops them from posting the code online somewhere.