llvm dev - May 2015 - [LLVMdev] libiomp, not libgomp as default library linked with -fopenmp

Chandler:
 1) I completely agree with the comments some others have made about
 us needing to make it clear that this isn't some Intel-only thing,
 it’s the LLVM OpenMP runtime. Some suggestions that I think would
 make sense to help here:

… code owner discussion elided since Chris has endorsed Andrey…

- Clearly updating the readme and such would be appropriate.
Certainly, no problem with that at all. As I ponted out before, people from
Intel are not the right ones to be inserting compatiblity tables for the runtime
when it is running on IBM Power or ARM processors. The people who know about
that should contribute…

 - I suspect we should change the name of the installed library.
 'libiomp' is pretty clearly the Intel library. We could continue in
 the grand tradition of LLVM naming conventions and use 'libllomp'?
 Of course, we should install symlinks under the name 'libiomp' if
 needed for existing users to not be broken.

Changing the file name (at least when targeting X86 processors) is a more
interesting issue, and not as trivial as it seems.
The problem here is related to inter-operation with existing (Intel and gcc)
compilers, and backwards binary compatibility.
As Hal pointed out, if you intend ever to mix code compiled by multiple,
different, OpenMP compilers into the same process, even if they all use an ABI
to the runtime that it supports (remember that the LLVM runtime supports both
the LLVM and libgomp interfaces), you must ensure that there is only one copy of
the runtime (dynamically) linked into the process. This is because (unlike most
libraries) the runtime contains state (for instance, the thread pool), therefore
if you have multiple instances even of the identical runtime linked into the
process, bad things will happen. (Over-subscription and associated poor
performance, critical sections not providing mutual exclusion, …). This is why
we don’t normally provide a static library; it tempts people into building their
own dynamic libraries that include a static copy of the OpenMP runtime, then
when their DLL is linked into an OpenMP program you instantly have two copies of
the runtime and a hard to diagnose disaster.

The problem this causes is that unless there is an agreed name for the runtime,
the dynamic linker will load it twice if it is demanded under different names.
(On Linux, you can use soname to help here, but doesn’t completely solve the
problem).

Therefore, if we do not want to make it unnecessarily hard for people who use
Clang/OpenMP on X86 to link against libraries (such as Intel MKL) which use
OpenMP and were compiled with a different compiler, it is helpful to use the
same file name for the runtime library no matter which compiler generated the
code.

Since there are many libraries and executables which have already been built by
the Intel compiler, so already have the library name built into them (in the
DT_NEEDED ELF tag, or equivalent), changing the library name does not come for
free.

I am therefore loath to change the library name simply for political/publicity
reasons when that will cause unnecessary additional complexity to the users of
one of our main target platforms. (Before anyone else says it, yes, of course
X86 is the platform I care most about, but the issue isn’t about compiler
competition, but about making things easy for the users of LLVM/OpenMP so that
things they expect will work do “just work” rather than adding extra complexity
and more manuals to explain what has to be done to make them work. Like moving
into your partner’s appartment, you’re welll advised not to move all the
furniture around immediately ☺).

 Just some bikeshed-painting: if we're expecting each compiler that
 uses the library to distribute a separate copy as part of that
 compiler's runtime, then I think the best name for clang to use for
 the library would probably be libclang_rt.omp-<arch> or
 libclang_rt.openmp-<arch> (as we do for all our other runtime
 libraries).
Aside from the argument above that we shouldn’t change the name (at least on
X86), note that if Hal’s efforts on Flang work out, we’ll also have a
Fortran/OpenMP implementation that will use the same runtime library. Therefore
putting “clang” into the name seems short-sighted.

If we expect this to be installed somewhere central on
 the system and shared by different compilers and different versions
 of the same compiler, then libllomp or similar seems reasonable to
 me. What's the intended distribution model here?
Whether OS vendors choose to install the runtime in a standard place (like
/usr/lib64) is not something we can control. (Though doing so is a reasonable
thing to do).

How stable is the ABI?
Stable; we add new interfaces when required by the evolution of the OpenMP
language specification, but maintain backwards compatibility, so code compiled
by older compilers can be run against newer runtime libraries.

On the benchmarks side of things, since everyone is so keen on this not only
being about Intel, surely it’d be good to have some comparisons of performance
vs gcc on Power or Arm as well as on X86.

-- Jim

James Cownie <james.h.cownie at intel.com>
SSG/DPD/TCAR (Technical Computing, Analyzers and Runtimes)
Tel: +44 117 9071438

From: Hal Finkel [mailto:hfinkel at anl.gov]
Sent: Tuesday, May 5, 2015 5:39 PM
To: Richard Smith
Cc: Andrey Bokhanko; cfe-dev; LLVM Developers Mailing List; Douglas Gregor; C
Bergström; Michael Wong; Alexey Bataev; Chandler Carruth; Cownie, James H
Subject: Re: [LLVMdev] libiomp, not libgomp as default library linked with
-fopenmp

________________________________


From: "Richard Smith" <richard at metafoo.co.uk<mailto:richard
at metafoo.co.uk>>
 To: "Chandler Carruth" <chandlerc at
google.com<mailto:chandlerc at google.com>>
 Cc: "Andrey Bokhanko" <andreybokhanko at
gmail.com<mailto:andreybokhanko at gmail.com>>, "cfe-dev"
<cfe-dev at cs.uiuc.edu<mailto:cfe-dev at cs.uiuc.edu>>, "LLVM
Developers Mailing List"
 <llvmdev at cs.uiuc.edu<mailto:llvmdev at cs.uiuc.edu>>,
"Douglas Gregor" <dgregor at apple.com<mailto:dgregor at
apple.com>>, "Hal Finkel" <hfinkel at
anl.gov<mailto:hfinkel at anl.gov>>, "C Bergström"
 <cbergstrom at pathscale.com<mailto:cbergstrom at pathscale.com>>,
"Michael Wong" <fraggamuffin at gmail.com<mailto:fraggamuffin at
gmail.com>>, "Alexey Bataev" <a.bataev at
gmx.com<mailto:a.bataev at gmx.com>>
 Sent: Friday, May 1, 2015 1:31:06 PM
 Subject: Re: [LLVMdev] libiomp, not libgomp as default library linked with
-fopenmp




 On Thu, Apr 30, 2015 at 2:51 PM, Chandler Carruth <
 chandlerc at google.com<mailto:chandlerc at google.com> > wrote:




 On Thu, Apr 30, 2015 at 6:52 AM Andrey Bokhanko <
 andreybokhanko at gmail.com<mailto:andreybokhanko at gmail.com> >
wrote:



 All,


 I'd like to resurrect the discussion on replacing libgomp with
 libiomp as the default OpenMP runtime library linked with -fopenmp.


 Just for the record, I'm really excited to see this. =]




 We are very close to getting *full* OpenMP 3.1 specification
 supported in clang (only one (!) clause is not implemented yet, and
 the patch is already sent for review today:
 http://reviews.llvm.org/D9370 ). This implementation generates Intel
 API library calls; thus, it can't be used with libgomp and it is
 simply logical to link a compatible runtime (libiomp) instead.



 Is there no way to support libgomp here as well? I don't say this to
 hold up changing the defaults in any way, just curious. =]


 Also, for the record, I'm really excited to see the progress here as
 well.




 Chandler?



 Hi! ;]


 I totally agree, I think things are way better now. I generally
 support the direction. I think there are a few things I'd suggest we
 do as part of the process, but I think these are really small and
 just about "how" we switch.


 1) I completely agree with the comments some others have made about
 us needing to make it clear that this isn't some Intel-only thing,
 its the LLVM OpenMP runtime. Some suggestions that I think would
 make sense to help here:
 - I agree with finding some non-Intel folks to add as explicit code
 owners. I don't know who has been sufficiently involved, but if Hal
 makes sense, awesome.
 - Clearly updating the readme and such would be appropriate.
 - I suspect we should change the name of the installed library.
 'libiomp' is pretty clearly the Intel library. We could continue in
 the grand tradition of LLVM naming conventions and use 'libllomp'?
 Of course, we should install symlinks under the name 'libiomp' if
 needed for existing users to not be broken.


 Just some bikeshed-painting: if we're expecting each compiler that
 uses the library to distribute a separate copy as part of that
 compiler's runtime, then I think the best name for clang to use for
 the library would probably be libclang_rt.omp-<arch> or
 libclang_rt.openmp-<arch> (as we do for all our other runtime
 libraries). If we expect this to be installed somewhere central on
 the system and shared by different compilers and different versions
 of the same compiler, then libllomp or similar seems reasonable to
 me. What's the intended distribution model here? How stable is the
 ABI?


I agree. However, for dynamic executables we need a dynamic OpenMP runtime
library (there can be only one in the process because it has to keep
process-wide state). The ABI is stable, and has a fairly-extensive history (on
x86, specifically). cc'ing Jim so he can also comment directly.

 -Hal






 - Any other changes?


 2) I think we need to update the instructions for checking out LLVM
 and all the tools to include checking out the openmp project. I'm
 planning to try it out in a bit.


 3) It would be nice to get at least one boring benchmark into the
 test-suite that uses OpenMP just so there's more coverage that the
 basic stuff all works. In particular, if we could get the benchmark
 that Phoronix and others keep pointing at, that'd be nice.




 Speaking of which, have you checked the performance of some of the
 basic benchmarks using OpenMP with the two runtimes? Or looked at
 Clang vs GCC there? I'd be interested to see the numbers.







 Yours,
 Andrey Bokhanko
 ============= Software Engineer
 Intel Compiler Team
 Intel

 _______________________________________________
 LLVM Developers mailing list
 LLVMdev at cs.uiuc.edu<mailto:LLVMdev at cs.uiuc.edu>
http://llvm.cs.uiuc.edu
 http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev

 _______________________________________________
 LLVM Developers mailing list
 LLVMdev at cs.uiuc.edu<mailto:LLVMdev at cs.uiuc.edu>
http://llvm.cs.uiuc.edu
 http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev




--

Hal Finkel
Assistant Computational Scientist
Leadership Computing Facility
Argonne National Laboratory
---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20150506/a1cc53cc/attachment.html>

On 6 May 2015, at 14:39, Cownie, James H <james.h.cownie at intel.com>
wrote:
> 
> I am therefore loath to change the library name simply for
political/publicity reasons when that will cause unnecessary additional
complexity to the users of one of our main target platforms. (Before anyone else
says it, yes, of course X86 is the platform I care most about, but the issue
isn’t about compiler competition, but about making things easy for the users of
LLVM/OpenMP so that things they expect will work do “just work” rather than
adding extra complexity and more manuals to explain what has to be done to make
them work. Like moving into your partner’s appartment, you’re welll advised not
to move all the furniture around immediately J).
For what it’s worth, on FreeBSD we’re likely to install a symlink from
libgomp.so to lib[il]omp.so when we import the LLVM runtime (which we’re
currently in the process of doing), because we intend to replace the GNU runtime
that ships in the base system.

David

----- Original Message -----
> From: "James H Cownie" <james.h.cownie at intel.com>
> To: "Hal Finkel" <hfinkel at anl.gov>, "Richard
Smith"
> <richard at metafoo.co.uk>
> Cc: "Andrey Bokhanko" <andreybokhanko at gmail.com>,
"cfe-dev"
> <cfe-dev at cs.uiuc.edu>, "LLVM Developers Mailing List"
> <llvmdev at cs.uiuc.edu>, "Douglas Gregor" <dgregor at
apple.com>, "C
> Bergström" <cbergstrom at pathscale.com>, "Michael
Wong"
> <fraggamuffin at gmail.com>, "Alexey Bataev" <a.bataev
at gmx.com>,
> "Chandler Carruth" <chandlerc at google.com>
> Sent: Wednesday, May 6, 2015 8:39:16 AM
> Subject: RE: [LLVMdev] libiomp, not libgomp as default library linked
> with -fopenmp
> Chandler:
> 1) I completely agree with the comments some others have made about
> us needing to make it clear that this isn't some Intel-only thing,
> it’s the LLVM OpenMP runtime. Some suggestions that I think would
> make sense to help here:
> … code owner discussion elided since Chris has endorsed Andrey…
> - Clearly updating the readme and such would be appropriate.
> Certainly, no problem with that at all. As I ponted out before,
> people from Intel are not the right ones to be inserting
> compatiblity tables for the runtime when it is running on IBM Power
> or ARM processors. The people who know about that should contribute…
> - I suspect we should change the name of the installed library.
> 'libiomp' is pretty clearly the Intel library. We could continue in
> the grand tradition of LLVM naming conventions and use 'libllomp'?
> Of course, we should install symlinks under the name 'libiomp' if
> needed for existing users to not be broken.
> Changing the file name (at least when targeting X86 processors) is a
> more interesting issue, and not as trivial as it seems.
> The problem here is related to inter-operation with existing (Intel
> and gcc) compilers, and backwards binary compatibility.
> As Hal pointed out, if you intend ever to mix code compiled by
> multiple, different, OpenMP compilers into the same process, even if
> they all use an ABI to the runtime that it supports (remember that
> the LLVM runtime supports both the LLVM and libgomp interfaces), you
> must ensure that there is only one copy of the runtime (dynamically)
> linked into the process. This is because (unlike most libraries) the
> runtime contains state (for instance, the thread pool), therefore if
> you have multiple instances even of the identical runtime linked
> into the process, bad things will happen. (Over-subscription and
> associated poor performance, critical sections not providing mutual
> exclusion, …). This is why we don’t normally provide a static
> library; it tempts people into building their own dynamic libraries
> that include a static copy of the OpenMP runtime, then when their
> DLL is linked into an OpenMP program you instantly have two copies
> of the runtime and a hard to diagnose disaster.
> The problem this causes is that unless there is an agreed name for
> the runtime, the dynamic linker will load it twice if it is demanded
> under different names. (On Linux, you can use soname to help here,
> but doesn’t completely solve the problem).
> Therefore, if we do not want to make it unnecessarily hard for people
> who use Clang/OpenMP on X86 to link against libraries (such as Intel
> MKL) which use OpenMP and were compiled with a different compiler,
> it is helpful to use the same file name for the runtime library no
> matter which compiler generated the code.
> Since there are many libraries and executables which have already
> been built by the Intel compiler, so already have the library name
> built into them (in the DT_NEEDED ELF tag, or equivalent), changing
> the library name does not come for free.
> I am therefore loath to change the library name simply for
> political/publicity reasons when that will cause unnecessary
> additional complexity to the users of one of our main target
> platforms. (Before anyone else says it, yes, of course X86 is the
> platform I care most about, but the issue isn’t about compiler
> competition, but about making things easy for the users of
> LLVM/OpenMP so that things they expect will work do “just work”
> rather than adding extra complexity and more manuals to explain what
> has to be done to make them work. Like moving into your partner’s
> appartment, you’re welll advised not to move all the furniture
> around immediately J ).
Okay, but I don't understand any of this. 

1. We really do need to change the name. 

2. We really should install a symlink with the libiomp5 name so that existing
applications will pick up the same library (or maybe hard links on windows?)

Having LLVM ship a library called libiomp5 by default would be very misleading
on non-X86 platforms, and even on x86 platforms, where it will likely not match
the binaries that you're shipping as part of supported products. In
addition, as a matter of branding for LLVM, is highly suboptimal. We can,
however, change the name without any of the negative side effects mentioned so
long as proper symlinks are installed. Am I missing something?

-Hal 
> Just some bikeshed-painting: if we're expecting each compiler that
> uses the library to distribute a separate copy as part of that
> compiler's runtime, then I think the best name for clang to use for
> the library would probably be libclang_rt.omp-<arch> or
> libclang_rt.openmp-<arch> (as we do for all our other runtime
> libraries).
> Aside from the argument above that we shouldn’t change the name (at
> least on X86), note that if Hal’s efforts on Flang work out, we’ll
> also have a Fortran/OpenMP implementation that will use the same
> runtime library. Therefore putting “clang” into the name seems
> short-sighted.
> If we expect this to be installed somewhere central on
> the system and shared by different compilers and different versions
> of the same compiler, then libllomp or similar seems reasonable to
> me. What's the intended distribution model here?
> Whether OS vendors choose to install the runtime in a standard place
> (like /usr/lib64) is not something we can control. (Though doing so
> is a reasonable thing to do).
> How stable is the ABI?
> Stable; we add new interfaces when required by the evolution of the
> OpenMP language specification, but maintain backwards compatibility,
> so code compiled by older compilers can be run against newer runtime
> libraries.
> On the benchmarks side of things, since everyone is so keen on this
> not only being about Intel, surely it’d be good to have some
> comparisons of performance vs gcc on Power or Arm as well as on X86.
> -- Jim
> James Cownie <james.h.cownie at intel.com>
> SSG/DPD/TCAR (Technical Computing, Analyzers and Runtimes)
> Tel: +44 117 9071438
> From: Hal Finkel [mailto:hfinkel at anl.gov]
> Sent: Tuesday, May 5, 2015 5:39 PM
> To: Richard Smith
> Cc: Andrey Bokhanko; cfe-dev; LLVM Developers Mailing List; Douglas
> Gregor; C Bergström; Michael Wong; Alexey Bataev; Chandler Carruth;
> Cownie, James H
> Subject: Re: [LLVMdev] libiomp, not libgomp as default library linked
> with -fopenmp
> ----- Original Message -----
> > From: "Richard Smith" < richard at metafoo.co.uk >
> 
> > To: "Chandler Carruth" < chandlerc at google.com >
> 
> > Cc: "Andrey Bokhanko" < andreybokhanko at gmail.com >,
"cfe-dev" <
> > cfe-dev at cs.uiuc.edu >, "LLVM Developers Mailing List"
> 
> > < llvmdev at cs.uiuc.edu >, "Douglas Gregor" <
dgregor at apple.com >,
> > "Hal
> > Finkel" < hfinkel at anl.gov >, "C Bergström"
> 
> > < cbergstrom at pathscale.com >, "Michael Wong" <
> > fraggamuffin at gmail.com
> > >, "Alexey Bataev" < a.bataev at gmx.com >
> 
> > Sent: Friday, May 1, 2015 1:31:06 PM
> 
> > Subject: Re: [LLVMdev] libiomp, not libgomp as default library
> > linked
> > with -fopenmp
> 
> > On Thu, Apr 30, 2015 at 2:51 PM, Chandler Carruth <
> 
> > chandlerc at google.com > wrote:
> 
> > On Thu, Apr 30, 2015 at 6:52 AM Andrey Bokhanko <
> 
> > andreybokhanko at gmail.com > wrote:
> 
> > All,
> 
> > I'd like to resurrect the discussion on replacing libgomp with
> 
> > libiomp as the default OpenMP runtime library linked with -fopenmp.
> 
> > Just for the record, I'm really excited to see this. =]
> 
> > We are very close to getting *full* OpenMP 3.1 specification
> 
> > supported in clang (only one (!) clause is not implemented yet, and
> 
> > the patch is already sent for review today:
> 
> > http://reviews.llvm.org/D9370 ). This implementation generates
> > Intel
> 
> > API library calls; thus, it can't be used with libgomp and it is
> 
> > simply logical to link a compatible runtime (libiomp) instead.
> 
> > Is there no way to support libgomp here as well? I don't say this
> > to
> 
> > hold up changing the defaults in any way, just curious. =]
> 
> > Also, for the record, I'm really excited to see the progress here
> > as
> 
> > well.
> 
> > Chandler?
> 
> > Hi! ;]
> 
> > I totally agree, I think things are way better now. I generally
> 
> > support the direction. I think there are a few things I'd suggest
> > we
> 
> > do as part of the process, but I think these are really small and
> 
> > just about "how" we switch.
> 
> > 1) I completely agree with the comments some others have made about
> 
> > us needing to make it clear that this isn't some Intel-only thing,
> 
> > its the LLVM OpenMP runtime. Some suggestions that I think would
> 
> > make sense to help here:
> 
> > - I agree with finding some non-Intel folks to add as explicit code
> 
> > owners. I don't know who has been sufficiently involved, but if
Hal
> 
> > makes sense, awesome.
> 
> > - Clearly updating the readme and such would be appropriate.
> 
> > - I suspect we should change the name of the installed library.
> 
> > 'libiomp' is pretty clearly the Intel library. We could
continue in
> 
> > the grand tradition of LLVM naming conventions and use
'libllomp'?
> 
> > Of course, we should install symlinks under the name 'libiomp'
if
> 
> > needed for existing users to not be broken.
> 
> > Just some bikeshed-painting: if we're expecting each compiler that
> 
> > uses the library to distribute a separate copy as part of that
> 
> > compiler's runtime, then I think the best name for clang to use
for
> 
> > the library would probably be libclang_rt.omp-<arch> or
> 
> > libclang_rt.openmp-<arch> (as we do for all our other runtime
> 
> > libraries). If we expect this to be installed somewhere central on
> 
> > the system and shared by different compilers and different versions
> 
> > of the same compiler, then libllomp or similar seems reasonable to
> 
> > me. What's the intended distribution model here? How stable is the
> 
> > ABI?
> 
> I agree. However, for dynamic executables we need a dynamic OpenMP
> runtime library (there can be only one in the process because it has
> to keep process-wide state). The ABI is stable, and has a
> fairly-extensive history (on x86, specifically). cc'ing Jim so he
> can also comment directly.
> -Hal
> > - Any other changes?
> 
> > 2) I think we need to update the instructions for checking out LLVM
> 
> > and all the tools to include checking out the openmp project. I'm
> 
> > planning to try it out in a bit.
> 
> > 3) It would be nice to get at least one boring benchmark into the
> 
> > test-suite that uses OpenMP just so there's more coverage that the
> 
> > basic stuff all works. In particular, if we could get the benchmark
> 
> > that Phoronix and others keep pointing at, that'd be nice.
> 
> > Speaking of which, have you checked the performance of some of the
> 
> > basic benchmarks using OpenMP with the two runtimes? Or looked at
> 
> > Clang vs GCC there? I'd be interested to see the numbers.
> 
> > Yours,
> 
> > Andrey Bokhanko
> 
> > =============> 
> > Software Engineer
> 
> > Intel Compiler Team
> 
> > Intel
> 
> > _______________________________________________
> 
> > LLVM Developers mailing list
> 
> > LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu
> 
> > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
> 
> > _______________________________________________
> 
> > LLVM Developers mailing list
> 
> > LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu
> 
> > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
> 
> > --
> 
> Hal Finkel
> Assistant Computational Scientist
> Leadership Computing Facility
> Argonne National Laboratory
> ---------------------------------------------------------------------
> Intel Corporation (UK) Limited
> Registered No. 1134945 (England)
> Registered Office: Pipers Way, Swindon SN3 1RJ
> VAT No: 860 2173 47
> This e-mail and any attachments may contain confidential material for
> the sole use of the intended recipient(s). Any review or distribution
> by others is strictly prohibited. If you are not the intended
> recipient, please contact the sender and delete all copies.
-- 

Hal Finkel 
Assistant Computational Scientist 
Leadership Computing Facility 
Argonne National Laboratory 
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20150506/875c930a/attachment.html>

Why is this thread still going? Isn't the most pragmatic choice to
just make it a build configuration option and be done? Then whoever is
actually packaging it can make the most sensible choice for their
needs..

Should I send a patch so this bikeshed thread can die?