llvm dev - Sep 2017 - Process of integrating Polly into LLVM

Dear all,

The proposal to move Polly into core-LLVM has generated an interesting
conversation and I’d like to supplement the discussion with a proposal
for initial, concrete integration steps. Please provide feedback both on
the goals and also on the process proposed to reach these goals.

First, I propose a set of goals a good integration process should
achieve:

- Start off from a well-tested baseline configuration
- Changes should be incremental or mechanical
- Preserve version history as much as possible 
- Broad community review on changes that lead to closer integration into
LLVM and clang
- Closer community integration

A process that aims to match the above requirements is the following:

      0)   Update Polly website to announce integration plan
      1) Close Polly mailing list and instead have Polly discussions on
      [llvm-dev] possibly using a [Polly] subject prefix

       2) svn copy Polly from its own repo into the LLVM main repository

	svn mv <polly>/          to            <llvm>/tools/polly

update buildbots

After this step an LLVM + clang checkout will contain a fully functional
build of LLVM + clang + Polly. The default behavior of LLVM is
unchanged, no user-visible flags are exposed, but Polly can be enabled
with “-mllvm -polly”. It can be removed from all LLVM tools by setting
LLVM_POLLY_LINK_INTO_TOOLS=OFF and LLVM_POLLY_BUILD=OFF. The
configuration
with Polly linked into LLVM is the configuration that is shipped as the
LLVM 5.0 release and has been tested in the LLVM release process to work
on all LLVM platforms without negatively impacting anybody.

Already after the very first step, Polly is available to the broader
LLVM community, such that all subsequent steps of the integration
process can be performed incrementally with separate mailing list
discussions or patch reviews that have full community visibility. 

The following discussions and changes should follow then:
      3) Update Polly website and documentation and mark Polly as
      experimental feature 
          (similar to an experimental backend)
      4) Remove unneeded parts of Polly
           Do not use PM callbacks, but place the Polly passes directly
           into the pass pipeline
           Drop support for loading Polly as a plugin
     5) Move polly passes and helper libraries such as isl into
     appropriate directories. E.g.,
           lib/Analysis/Polly, lib/Transforms/Polly and tests into
           tests/…
     6) Introduce a user visible clang flag to enable Polly
	e.g., -floop-optimize-aggressive

     7) Cross-reference Polly documentation and LLVM website
     8) Continue testing, fuzzing, and profiling

At this point, Polly is available to the wider community, but nobody is
affected in the default configuration as long as no explicit options
have been chosen. I expect now a longer period of extensive development
to further improve robustness of Polly, ensure it does not regress
existing codes, adapt the LLVM passes to Polly, and obviously also
introduce more optimizations based on Polly (or related tools).

When considered stable, regression free, and tested to a level that LLVM
considers sufficient, we can consider turning on Polly by default in
some user facing optimization levels. When this is considered, community
feedback will be asked for once again.

Best,
Tobias

On 23 Sep 2017, at 06:19, Tobias Grosser via llvm-dev <llvm-dev at
lists.llvm.org<mailto:llvm-dev at lists.llvm.org>> wrote:

      2) svn copy Polly from its own repo into the LLVM main repository

svn mv <polly>/          to            <llvm>/tools/polly

Hi Tobias,

Just a minor point remark: IIUC, in SVN, the different subprojects are all part
of the same repository, but in the official git mirrors, each subproject is its
own separate top-level git project. Have you checked that doing an "svn mv”
like above will also retain the full history in the git mirror? The “svn mv”
will move Polly from one top-level git repository (polly) to another (llvm). I’m
wondering since I wonder how out-of-repository history (i.e. the Polly history
before it was moved to <llvm>/tools/polly) would be represented in git.

Thanks,

Kristof
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20170929/54ec04da/attachment.html>

On Fri, Sep 29, 2017, at 12:03, Kristof Beyls via llvm-dev
wrote:
> 
> On 23 Sep 2017, at 06:19, Tobias Grosser via llvm-dev
> <llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org>>
wrote:
> 
>       2) svn copy Polly from its own repo into the LLVM main repository
> 
> svn mv <polly>/          to            <llvm>/tools/polly
> 
> Hi Tobias,
> 
> Just a minor point remark: IIUC, in SVN, the different subprojects are
> all part of the same repository, but in the official git mirrors, each
> subproject is its own separate top-level git project. 
> Have you checked
> that doing an "svn mv” like above will also retain the full history in
> the git mirror? The “svn mv” will move Polly from one top-level git
> repository (polly) to another (llvm). I’m wondering since I wonder how
> out-of-repository history (i.e. the Polly history before it was moved to
> <llvm>/tools/polly) would be represented in git.
In the per-subproject git mirrors history would not be visible AFAIU,
but the llvm-project repositories will show the full history. When
moving to github (and git in general) we might even be able to get the
pre-polly-in-llvm-history into the official repository, as Mehdi
explained to me.

Best,
Tobias
> 
> Thanks,
> 
> Kristof
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev