llvm dev - Aug 2010 - [LLVMdev] [RFC] Moving to Sphinx for LLVM and friends documentation (with partial implementation (in both 10pt and 12pt font)).

Moving the LLVM Documentation to Sphinx
======================================
As a few of you that are on IRC already know, I have experimented with moving
the LLVM documentation over to `Sphinx
<http://sphinx.pocoo.org/index.html>`__
from the current html form. I have moved almost all of the content over and have
begun "Sphinxifying" the documentation to correct links and make use
of the many
features that Sphinx provides.

What is Sphinx?
---------------

To quote from the Sphinx website:

  Sphinx is a tool that makes it easy to create intelligent and beautiful
  documentation, written by Georg Brandl and licensed under the BSD license.

  It was originally created for the new Python documentation, and it has
  excellent facilities for the documentation of Python projects, but C/C++ is
  already supported as well.

  Sphinx uses reStructuredText as its markup language, and many of its strengths
  come from the power and straightforwardness of reStructuredText and its
  parsing  and translating suite, the Docutils.


What Does Sphinx Do For LLVM?
-----------------------------

* Creates a unified documentation infrastructure. Sphinx can create man pages,
  which we are currently using a seperate format to do. There is also work on
  integrating doxygen.
* Is pretty (both the source and output).
* Is easier to edit and maintain than html.
* Is extensible via Python.
* Uses `Pygments <http://pygments.org/>`__ for syntax highlighting, which
  supports LLVM IR.
* Provides a rather good search engine.


Current Status
--------------

Currently I have simply moved the html and pod docs over to reStructuredText
using a modified version of `html2rst
<https://www.dropbox.com/s/pqzwlxvwjbvm4xq/html2rest.py#>`__ and `pod2rst
<http://search.cpan.org/~dowens/Pod-POM-View-Restructured-0.02/bin/pod2rst>`__
and fixed minor porting issues.

Here are the links to the current documentation and associated files.

Current Docs
  https://dl.dropbox.com/u/9889801/llvm-sphinx-docs/_build/html/index.html

  Click on ``Show Source`` on the righthand side of any page to view the
  reStructuredText that generated that page.

Configuration File
  https://www.dropbox.com/s/vdgl8hhkojujv19/conf.py

Source + Build Directory
  https://www.dropbox.com/s/jlo7adms1890e56


What's Left?
------------

#. Finish moving over the docs.
#. Finish Sphinxifying them.
#. Pick the paint color for the bike shed. I have done no work on this other
   than copying over the default ``sphinxdoc`` theme to the source dir as
   ``llvm-theme``. Creating Sphinx themes is actually rather easy, so anyone is
   welcome to take a stab at this if they wish.
#. Integrate building the docs into the autoconf and CMake build systems.


What Do You Think?
------------------

I realize that changing the documentation format is non-trivial, but I believe
that the benefits are worth the effort. If we go forward with this I will finish
the first two points above and work to integrate doxygen and keep everything
running smoothly.


BTW
---

This email is a valid Sphinx document and can be seen `here
<http://dl.dropbox.com/u/9889801/llvm-sphinx-docs/_build/html/meta-sphinx-
documentation.html>`__.



- Michael Spencer

Hi Michael,


Awesome work!

I'm a strong supporter of using Sphinx. I've been using it for the LNT
docs (http://llvm.org/docs/lnt/) and it is quite nice to work with. +1
for migrating over.

 - Daniel

On Sun, Aug 8, 2010 at 11:29 PM, Michael Spencer <bigcheesegs at
gmail.com> wrote:
> Moving the LLVM Documentation to Sphinx
> ======================================>
> As a few of you that are on IRC already know, I have experimented with
moving
> the LLVM documentation over to `Sphinx
<http://sphinx.pocoo.org/index.html>`__
> from the current html form. I have moved almost all of the content over and
have
> begun "Sphinxifying" the documentation to correct links and make
use of the many
> features that Sphinx provides.
>
> What is Sphinx?
> ---------------
>
> To quote from the Sphinx website:
>
>  Sphinx is a tool that makes it easy to create intelligent and beautiful
>  documentation, written by Georg Brandl and licensed under the BSD license.
>
>  It was originally created for the new Python documentation, and it has
>  excellent facilities for the documentation of Python projects, but C/C++
is
>  already supported as well.
>
>  Sphinx uses reStructuredText as its markup language, and many of its
strengths
>  come from the power and straightforwardness of reStructuredText and its
>  parsing  and translating suite, the Docutils.
>
>
> What Does Sphinx Do For LLVM?
> -----------------------------
>
> * Creates a unified documentation infrastructure. Sphinx can create man
pages,
>  which we are currently using a seperate format to do. There is also work
on
>  integrating doxygen.
> * Is pretty (both the source and output).
> * Is easier to edit and maintain than html.
> * Is extensible via Python.
> * Uses `Pygments <http://pygments.org/>`__ for syntax highlighting,
which
>  supports LLVM IR.
> * Provides a rather good search engine.
>
>
> Current Status
> --------------
>
> Currently I have simply moved the html and pod docs over to
reStructuredText
> using a modified version of `html2rst
> <https://www.dropbox.com/s/pqzwlxvwjbvm4xq/html2rest.py#>`__ and
`pod2rst
>
<http://search.cpan.org/~dowens/Pod-POM-View-Restructured-0.02/bin/pod2rst>`__
> and fixed minor porting issues.
>
> Here are the links to the current documentation and associated files.
>
> Current Docs
>  https://dl.dropbox.com/u/9889801/llvm-sphinx-docs/_build/html/index.html
>
>  Click on ``Show Source`` on the righthand side of any page to view the
>  reStructuredText that generated that page.
>
> Configuration File
>  https://www.dropbox.com/s/vdgl8hhkojujv19/conf.py
>
> Source + Build Directory
>  https://www.dropbox.com/s/jlo7adms1890e56
>
>
> What's Left?
> ------------
>
> #. Finish moving over the docs.
> #. Finish Sphinxifying them.
> #. Pick the paint color for the bike shed. I have done no work on this
other
>   than copying over the default ``sphinxdoc`` theme to the source dir as
>   ``llvm-theme``. Creating Sphinx themes is actually rather easy, so anyone
is
>   welcome to take a stab at this if they wish.
> #. Integrate building the docs into the autoconf and CMake build systems.
>
>
> What Do You Think?
> ------------------
>
> I realize that changing the documentation format is non-trivial, but I
believe
> that the benefits are worth the effort. If we go forward with this I will
finish
> the first two points above and work to integrate doxygen and keep
everything
> running smoothly.
>
>
> BTW
> ---
>
> This email is a valid Sphinx document and can be seen `here
>
<http://dl.dropbox.com/u/9889801/llvm-sphinx-docs/_build/html/meta-sphinx-
> documentation.html>`__.
>
>
>
> - Michael Spencer
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>

On Aug 9, 2010, at 1:06 PM, Daniel Dunbar wrote:
> Hi Michael,
> 
> 
> Awesome work!
> 
> I'm a strong supporter of using Sphinx. I've been using it for the
LNT
> docs (http://llvm.org/docs/lnt/) and it is quite nice to work with. +1
> for migrating over.
> 
I'll give another +1.  It looks pretty decent and I particularly like the
formatting.


-eric

On Aug 8, 2010, at 11:29 PM, Michael Spencer wrote:
> 
> What Do You Think?
> ------------------
> 
> I realize that changing the documentation format is non-trivial, but I
believe
> that the benefits are worth the effort. If we go forward with this I will
finish
> the first two points above and work to integrate doxygen and keep
everything
> running smoothly.
I strongly support this.  I've been interested in moving the documentation
over to a better system for a long time, but the only decent one I've known
about was docbook, which is ... a bit more complicated than I hoped we'd
need.  :-)
> What's Left?
> ------------
> 
> #. Pick the paint color for the bike shed. I have done no work on this
other
>   than copying over the default ``sphinxdoc`` theme to the source dir as
>   ``llvm-theme``. Creating Sphinx themes is actually rather easy, so anyone
is
>   welcome to take a stab at this if they wish.
I suggest you just dictate what you think the right thing is, asking for and
ignoring or responding to feedback as you think is best.  Don't try to make
everyone happy.  I agree with Vikram that there are some major formatting issues
that should be fixed before this is rolled out.
> #. Integrate building the docs into the autoconf and CMake build systems.
Yes, this seems important.  Also, the web page needs to auto-update the docs in
response to commits.
> #. Finish moving over the docs.
> #. Finish Sphinxifying them.
Can this be done incrementally?  It would be great to start with one doc (like
LangRef) and make it really really good and get the infrastructure right, then
move on to other docs (which others might help out with).

-Chris

On Mon, Aug 9, 2010 at 6:51 PM, Chris Lattner <clattner at apple.com>
wrote:
>
> On Aug 8, 2010, at 11:29 PM, Michael Spencer wrote:
>>
>> What Do You Think?
>> ------------------
>>
>> I realize that changing the documentation format is non-trivial, but I
believe
>> that the benefits are worth the effort. If we go forward with this I
will finish
>> the first two points above and work to integrate doxygen and keep
everything
>> running smoothly.
>
> I strongly support this.  I've been interested in moving the
documentation over to a better system for a long time, but the only decent one
I've known about was docbook, which is ... a bit more complicated than I
hoped we'd need.  :-)
>
>> What's Left?
>> ------------
>>
>> #. Pick the paint color for the bike shed. I have done no work on this
other
>>   than copying over the default ``sphinxdoc`` theme to the source dir
as
>>   ``llvm-theme``. Creating Sphinx themes is actually rather easy, so
anyone is
>>   welcome to take a stab at this if they wish.
>
> I suggest you just dictate what you think the right thing is, asking for
and ignoring or responding to feedback as you think is best.  Don't try to
make everyone happy.  I agree with Vikram that there are some major formatting
issues that should be fixed before this is rolled out.
>
>> #. Integrate building the docs into the autoconf and CMake build
systems.
>
> Yes, this seems important.  Also, the web page needs to auto-update the
docs in response to commits.
>
>> #. Finish moving over the docs.
>> #. Finish Sphinxifying them.
>
> Can this be done incrementally?  It would be great to start with one doc
(like LangRef) and make it really really good and get the infrastructure right,
then move on to other docs (which others might help out with).
BoostBook is a simplification of DocBook, with many things specific
for C/C++ source.  it can pull in parts of a source file so the doc
examples always show examples of code that is actually tested, it has
proper links to source code files, can integrate with doxygen with a
bit of work, can export to html, pdf, and a few other things...  I
like Sphinx, a lot actually, but BoostBook is more designed for this
kind of project.

On Mon, Aug 9, 2010 at 8:51 PM, Chris Lattner <clattner at apple.com>
wrote:
>
> On Aug 8, 2010, at 11:29 PM, Michael Spencer wrote:
>>
>> What Do You Think?
>> ------------------
>>
>> I realize that changing the documentation format is non-trivial, but I
believe
>> that the benefits are worth the effort. If we go forward with this I
will finish
>> the first two points above and work to integrate doxygen and keep
everything
>> running smoothly.
>
> I strongly support this.  I've been interested in moving the
documentation over to a better system for a long time, but the only decent one
I've known about was docbook, which is ... a bit more complicated than I
hoped we'd need.  :-)
That and BoostBook are all I had heard of either until Daniel popped
into the conversation and mentioned Sphinx.


==================================================================================================I
will also use this space to reply to OvermindDL1 instead of writing
two emails.

You are right that BoostBook is targeted directly for large C++
projects, and I seriously considered BoostBook for this project, but
ran into a few road blocks.

* It's tightly integrated into boost and makes quite a few assumptions
about that.
* It requires boost-build, which is a rather complex dependency and
yet another full build system.
* It just feels really complex all around.

In the end I felt that there would be too much resistance to
boost-build and the complexity. I then discovered Sphinx and found
that it did everything I wanted and just felt clean and simple.
==================================================================================================

>
>> What's Left?
>> ------------
>>
>> #. Pick the paint color for the bike shed. I have done no work on this
other
>>   than copying over the default ``sphinxdoc`` theme to the source dir
as
>>   ``llvm-theme``. Creating Sphinx themes is actually rather easy, so
anyone is
>>   welcome to take a stab at this if they wish.
>
> I suggest you just dictate what you think the right thing is, asking for
and ignoring or responding to feedback as you think is best.  Don't try to
make everyone happy.  I agree with Vikram that there are some major formatting
issues that should be fixed before this is rolled out.
I was planning on stealing Daniel's layout from the LNT docs for now
(which are Sphinx based (I actually didn't know this until _after_ I
had done a bunch of work :P)), and mirror the colors from the llvm
logo. That's as far as my artistic skills go. I just want to make sure
people know I'm completely open to any suggestions.
>
>> #. Integrate building the docs into the autoconf and CMake build
systems.
>
> Yes, this seems important.  Also, the web page needs to auto-update the
docs in response to commits.
I'll need help integrating it into the build system. As for
auto-update, I think Daniel (there's that name again) already has this
somewhat setup. Again I don't know how to do this.
>
>> #. Finish moving over the docs.
>> #. Finish Sphinxifying them.
>
> Can this be done incrementally?  It would be great to start with one doc
(like LangRef) and make it really really good and get the infrastructure right,
then move on to other docs (which others might help out with).
>
> -Chris
Of course. I agree that it should be incremental. This really goes
with how this project should make its way into trunk. I was thinking
of making a docs/sphinx directory and putting {conf.py, index.rst, <a
few core docs>, _templates} in there. This will have no effect other
than adding the files. From there we can work on build system
integration and infrastructure.

The thing with "sphinxifying" is that a lot of it has to do with how
all of the documentation fits together, so it may be hard to get a
feel with only one doc. However, I think it would be a good idea to
start with just a few core docs (LangRef, GettingStarted{,VS}, ???)
and go from there.

- Michael Spencer