llvm dev - Jul 2012 - [LLVMdev] RFC: Staging area proposal for new backends

On 07/27/2012 01:01 PM, Tom Stellard wrote:
> Hi,
>
> I would like to try to keep the staging area discussion going.  There
> seems to be a general consensus that a staging area for backends and also
> new features would be acceptable for the LLVM project.  What actions
> are required to make the staging area a reality?  Is more discussion
> needed? Is anyone willing to volunteer to be the "Code Owner" for
the
> staging area, to help move the process forward?
>
> Thanks,
> Tom
The main issue I see is defining exactly how to incorporate the
"staged"
back-ends into the build system.  I see a couple of possibilities.

 1. Fully-integrate the "staged" back-ends into the CMake/Autotools
    build systems, but exclude them from the "all" pseudo-target.
 2. Maintain a separate list of "staged" back-ends, and create an
    additional option that must be set in order to build them (with an
    appropriate warning).

We also need to come up with a plan regarding cutting releases. When 3.2 
is branched, will all "staged" back-ends be removed?  Or will they be 
left in the distribution so interested parties can build them?

Beyond that, I don't believe we have established the criteria for 
back-end promotion.  Then again, it may make more sense to use R600 as a 
guinea pig and address issues as they come up.
>
> On Fri, Jul 20, 2012 at 11:51:03AM -0400, Tom Stellard wrote:
>> Hi,
>>
>> I would like to follow up on the recent discussion on the mailing list
>> about requirements for new backends[1] by submitting the following
>> proposal for a staging area for new LLVM backends.  This proposal
>> incorporates ideas from Owen, Chandler, and others who chimed in on
>> the original thread, and I hope the LLVM developers will be able to
>> come to a consensus on this proposal or a modified version, so the
>> project is able to accept new backends.
>>
>> The goals of the staging area will be:
>>    1. Facilitate communication between the LLVM project and backend
>>       developers
>>    2. Ensure that new backends meet LLVM standards
>>    3. Give the backend more exposure to users and prospective
developers
>>
>> ++ Staging Area:
>>
>> Similar to the Linux kernel, the staging area for new backends will
>> be in the main LLVM tree, with building of the backend being disabled
>> by default.  There will also be a TODO file in the backend's root
>> directory, that contains a list of improvements that are required to
>> promote the backend out of the staging area.  The backend will be
>> assigned a steward who's role will be to guide the backend through
>> the staging process and help solicit feedback from other developers.
>>
>> There are several advantages to having the staging area be in the main
>> tree as opposed to a separate branch:
>>
>> 1. It will be easier for LLVM developers to become familiar with the
>>     new backend and identify areas for improvement.
>>
>>    If the new backend is in the main tree, LLVM developers are more
>>    likely to encounter it in their day to day development.  Imagine a
>>    scenario where a developer makes a change to LLVM core that impacts
>>    several backends.  The developer may grep the code looking for
>>    backends that make use of the feature that they have added or
>>    changed.  If the new backend is in tree and uses that feature,
>>    the developer will see the code and might take a few moments to
>>    read through.  While doing this the developer may notice an area for
>>    improvement for the backend and can update the backends's TODO
file.
>>    The end result of this is that the LLVM developer has been able to
>>    provide some feedback with a minimal time commitment on their part.
>>
>>    If the backend were staged in a separate tree, this kind of
>>    simple review would not be possible, and I would be concerned that
>>    developers would be too busy to ever get around to checking out
>>    the staging tree.
>>
>> 2. It will allow the backend developers to always develop against TOT.
>>
>>    Developing against TOT is the recommended development procedure for
>>    anyone working on LLVM, and this is regularly reiterated on the
>>    mailing list.  If the new backend is included in the main tree,
>>    the backend developers will have no choice but to work against TOT.
>>
>> 3. It will make it easier for end users and distributions to test and
>>     also make it easier for new contributors.
>>
>>    New backends will be more visible to the public if they are in the
>>    main tree.  This will mean more users, an expanded testing base, and
>>    more potential developers which will lead to a higher quality
backend.
>>
>>
>> ++ Promotion/Demotion from staging area:
>>
>>    After a period of time, or when the tasks in the TODO file have been
>>    completed, the backend developers or the steward can initiate the
>>    review process.  The review process will be conducted by either the
>>    steward, a committee, or some select developers, who will decide
>>    (maybe by vote in the case of a committee) whether the backend
>>    should be:
>>
>>    - Promoted = Build of backend will be enabled by default.
>>    - Extended = Backend remains in the staging area.
>>    - Demoted  = Removed from the main tree
>>      (I can't really think of any disadvantages to having a backend
be
>>      in the main tree as long as its not being built by default, so
maybe
>>      demotion would be reserved for cases of long term absence
>>      of maintainership)
>>
>>    The Promoted/Extended/Demoted decision will be made using the
>>    following criteria (These won't necessarily all be absolutely
>>    required, they merely serve as a way for a backend's progress to
>>    be measured) :
>>
>>    - Progress towards completion of TODO tasks
>>    - Active maintainership
>>    - Use of incremental development techniques
>>    - Adherence to LLVM coding style
>>    - Usage of modern LLVM features
>>    - Quality and quantity of regression tests
>>    - Availability of buildbots
>>    - Size of user base
>>    - Other criteria deemed important by LLVM developers
>>
>>    - Contributions to core LLVM
>>      In the previous mailing list discussions there were differing
>>      opinions of how important contributing to the core LLVM code is
>>      for having a backend accepted.  It seems like a good middle ground
>>      would be that backends should be free of code that works around
>>      bugs or deficiency in core LLVM and instead fix the problem in
>>      shared code, and also should make an effort to push optimization
>>      passes that may be useful to other targets into the shared parts
>>      of the code.
>>
>>
>> ++ What is needed from the LLVM developers:
>>
>>    In order to make this staging program successful, the LLVM project
>>    will need to appoint a "code owner" for the staging
process, who
>>    backend developers can contact when they are interested in getting
>>    the backend included in the main tree.  An LLVM developer will also
>>    be needed to act as a steward for the new backend and help guide
>>    the backend developers through the process.
>>
>> Looking forward to comments on this proposal.
>>
>> Thanks,
>> Tom Stellard
>>
>> [1]
http://lists.cs.uiuc.edu/pipermail/llvm-commits/Week-of-Mon-20120716/146560.html
>>
>> _______________________________________________
>> LLVM Developers mailing list
>> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
>> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>>
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
-- 
Thanks,

Justin Holewinski

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20120727/e24f5cfc/attachment.html>

On Fri, Jul 27, 2012 at 01:30:49PM -0400, Justin Holewinski
wrote:
> On 07/27/2012 01:01 PM, Tom Stellard wrote:
> >Hi,
> >
> >I would like to try to keep the staging area discussion going.  There
> >seems to be a general consensus that a staging area for backends and
also
> >new features would be acceptable for the LLVM project.  What actions
> >are required to make the staging area a reality?  Is more discussion
> >needed? Is anyone willing to volunteer to be the "Code Owner"
for the
> >staging area, to help move the process forward?
> >
> >Thanks,
> >Tom
> 
> The main issue I see is defining exactly how to incorporate the
> "staged" back-ends into the build system.  I see a couple of
> possibilities.
> 
> 1. Fully-integrate the "staged" back-ends into the
CMake/Autotools
>    build systems, but exclude them from the "all" pseudo-target.
> 2. Maintain a separate list of "staged" back-ends, and create an
>    additional option that must be set in order to build them (with an
>    appropriate warning).
>
I think it would be good if each backend / feature its own enable flag
rather than lumping them all together into one.  This way if one feature
or backend in the staging area was being neglected, then it wouldn't
affect all the others.  Do you have any particular preference?
> We also need to come up with a plan regarding cutting releases. When
> 3.2 is branched, will all "staged" back-ends be removed?  Or will
> they be left in the distribution so interested parties can build
> them?
>
I can't really think of any disadvantages to keeping staged backends
in releases.  Being in a release would help a backend get more exposure
and increase the number of users/testers it would get.
> Beyond that, I don't believe we have established the criteria for
> back-end promotion.  Then again, it may make more sense to use R600
> as a guinea pig and address issues as they come up.
Here are a few objective criteria that have been mention to me.
- Test Cases
- Assembly Printer
- No pre-MC AsmPrinter or CodeEmitter

There are several subjective ones as well, like coding style and
robustnesses, but those are a little harder to define.  Can you think
of any others?

-Tom
> 
> >
> >On Fri, Jul 20, 2012 at 11:51:03AM -0400, Tom Stellard wrote:
> >>Hi,
> >>
> >>I would like to follow up on the recent discussion on the mailing
list
> >>about requirements for new backends[1] by submitting the following
> >>proposal for a staging area for new LLVM backends.  This proposal
> >>incorporates ideas from Owen, Chandler, and others who chimed in on
> >>the original thread, and I hope the LLVM developers will be able to
> >>come to a consensus on this proposal or a modified version, so the
> >>project is able to accept new backends.
> >>
> >>The goals of the staging area will be:
> >>   1. Facilitate communication between the LLVM project and backend
> >>      developers
> >>   2. Ensure that new backends meet LLVM standards
> >>   3. Give the backend more exposure to users and prospective
developers
> >>
> >>++ Staging Area:
> >>
> >>Similar to the Linux kernel, the staging area for new backends will
> >>be in the main LLVM tree, with building of the backend being
disabled
> >>by default.  There will also be a TODO file in the backend's
root
> >>directory, that contains a list of improvements that are required
to
> >>promote the backend out of the staging area.  The backend will be
> >>assigned a steward who's role will be to guide the backend
through
> >>the staging process and help solicit feedback from other
developers.
> >>
> >>There are several advantages to having the staging area be in the
main
> >>tree as opposed to a separate branch:
> >>
> >>1. It will be easier for LLVM developers to become familiar with
the
> >>    new backend and identify areas for improvement.
> >>
> >>   If the new backend is in the main tree, LLVM developers are more
> >>   likely to encounter it in their day to day development.  Imagine
a
> >>   scenario where a developer makes a change to LLVM core that
impacts
> >>   several backends.  The developer may grep the code looking for
> >>   backends that make use of the feature that they have added or
> >>   changed.  If the new backend is in tree and uses that feature,
> >>   the developer will see the code and might take a few moments to
> >>   read through.  While doing this the developer may notice an area
for
> >>   improvement for the backend and can update the backends's
TODO file.
> >>   The end result of this is that the LLVM developer has been able
to
> >>   provide some feedback with a minimal time commitment on their
part.
> >>
> >>   If the backend were staged in a separate tree, this kind of
> >>   simple review would not be possible, and I would be concerned
that
> >>   developers would be too busy to ever get around to checking out
> >>   the staging tree.
> >>
> >>2. It will allow the backend developers to always develop against
TOT.
> >>
> >>   Developing against TOT is the recommended development procedure
for
> >>   anyone working on LLVM, and this is regularly reiterated on the
> >>   mailing list.  If the new backend is included in the main tree,
> >>   the backend developers will have no choice but to work against
TOT.
> >>
> >>3. It will make it easier for end users and distributions to test
and
> >>    also make it easier for new contributors.
> >>
> >>   New backends will be more visible to the public if they are in
the
> >>   main tree.  This will mean more users, an expanded testing base,
and
> >>   more potential developers which will lead to a higher quality
backend.
> >>
> >>
> >>++ Promotion/Demotion from staging area:
> >>
> >>   After a period of time, or when the tasks in the TODO file have
been
> >>   completed, the backend developers or the steward can initiate
the
> >>   review process.  The review process will be conducted by either
the
> >>   steward, a committee, or some select developers, who will decide
> >>   (maybe by vote in the case of a committee) whether the backend
> >>   should be:
> >>
> >>   - Promoted = Build of backend will be enabled by default.
> >>   - Extended = Backend remains in the staging area.
> >>   - Demoted  = Removed from the main tree
> >>     (I can't really think of any disadvantages to having a
backend be
> >>     in the main tree as long as its not being built by default, so
maybe
> >>     demotion would be reserved for cases of long term absence
> >>     of maintainership)
> >>
> >>   The Promoted/Extended/Demoted decision will be made using the
> >>   following criteria (These won't necessarily all be
absolutely
> >>   required, they merely serve as a way for a backend's
progress to
> >>   be measured) :
> >>
> >>   - Progress towards completion of TODO tasks
> >>   - Active maintainership
> >>   - Use of incremental development techniques
> >>   - Adherence to LLVM coding style
> >>   - Usage of modern LLVM features
> >>   - Quality and quantity of regression tests
> >>   - Availability of buildbots
> >>   - Size of user base
> >>   - Other criteria deemed important by LLVM developers
> >>
> >>   - Contributions to core LLVM
> >>     In the previous mailing list discussions there were differing
> >>     opinions of how important contributing to the core LLVM code
is
> >>     for having a backend accepted.  It seems like a good middle
ground
> >>     would be that backends should be free of code that works
around
> >>     bugs or deficiency in core LLVM and instead fix the problem in
> >>     shared code, and also should make an effort to push
optimization
> >>     passes that may be useful to other targets into the shared
parts
> >>     of the code.
> >>
> >>
> >>++ What is needed from the LLVM developers:
> >>
> >>   In order to make this staging program successful, the LLVM
project
> >>   will need to appoint a "code owner" for the staging
process, who
> >>   backend developers can contact when they are interested in
getting
> >>   the backend included in the main tree.  An LLVM developer will
also
> >>   be needed to act as a steward for the new backend and help guide
> >>   the backend developers through the process.
> >>
> >>Looking forward to comments on this proposal.
> >>
> >>Thanks,
> >>Tom Stellard
> >>
> >>[1]
http://lists.cs.uiuc.edu/pipermail/llvm-commits/Week-of-Mon-20120716/146560.html
> >>
> >>_______________________________________________
> >>LLVM Developers mailing list
> >>LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> >>http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
> >>
> >_______________________________________________
> >LLVM Developers mailing list
> >LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> >http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
> 
> -- 
> Thanks,
> 
> Justin Holewinski
>

On Jul 27, 2012, at 1:39 PM, Tom Stellard wrote:
> 
>> We also need to come up with a plan regarding cutting releases. When
>> 3.2 is branched, will all "staged" back-ends be removed?  Or
will
>> they be left in the distribution so interested parties can build
>> them?
>> 
> 
> I can't really think of any disadvantages to keeping staged backends
> in releases.  Being in a release would help a backend get more exposure
> and increase the number of users/testers it would get.
I agree, there is no reason to remove it from the source drop of the release. 
The binaries produced for each release shouldn't include them though.

-Chris

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20120727/097773fc/attachment.html>