llvm dev - Aug 2010 - [LLVMdev] [RFC] Moving to Sphinx for LLVM and friends documentation (with partial implementation (in both 10pt and 12pt font)).

On Mon, Aug 9, 2010 at 8:51 PM, Chris Lattner <clattner at apple.com>
wrote:
>
> On Aug 8, 2010, at 11:29 PM, Michael Spencer wrote:
>>
>> What Do You Think?
>> ------------------
>>
>> I realize that changing the documentation format is non-trivial, but I
believe
>> that the benefits are worth the effort. If we go forward with this I
will finish
>> the first two points above and work to integrate doxygen and keep
everything
>> running smoothly.
>
> I strongly support this.  I've been interested in moving the
documentation over to a better system for a long time, but the only decent one
I've known about was docbook, which is ... a bit more complicated than I
hoped we'd need.  :-)
That and BoostBook are all I had heard of either until Daniel popped
into the conversation and mentioned Sphinx.


==================================================================================================I
will also use this space to reply to OvermindDL1 instead of writing
two emails.

You are right that BoostBook is targeted directly for large C++
projects, and I seriously considered BoostBook for this project, but
ran into a few road blocks.

* It's tightly integrated into boost and makes quite a few assumptions
about that.
* It requires boost-build, which is a rather complex dependency and
yet another full build system.
* It just feels really complex all around.

In the end I felt that there would be too much resistance to
boost-build and the complexity. I then discovered Sphinx and found
that it did everything I wanted and just felt clean and simple.
==================================================================================================

>
>> What's Left?
>> ------------
>>
>> #. Pick the paint color for the bike shed. I have done no work on this
other
>>   than copying over the default ``sphinxdoc`` theme to the source dir
as
>>   ``llvm-theme``. Creating Sphinx themes is actually rather easy, so
anyone is
>>   welcome to take a stab at this if they wish.
>
> I suggest you just dictate what you think the right thing is, asking for
and ignoring or responding to feedback as you think is best.  Don't try to
make everyone happy.  I agree with Vikram that there are some major formatting
issues that should be fixed before this is rolled out.
I was planning on stealing Daniel's layout from the LNT docs for now
(which are Sphinx based (I actually didn't know this until _after_ I
had done a bunch of work :P)), and mirror the colors from the llvm
logo. That's as far as my artistic skills go. I just want to make sure
people know I'm completely open to any suggestions.
>
>> #. Integrate building the docs into the autoconf and CMake build
systems.
>
> Yes, this seems important.  Also, the web page needs to auto-update the
docs in response to commits.
I'll need help integrating it into the build system. As for
auto-update, I think Daniel (there's that name again) already has this
somewhat setup. Again I don't know how to do this.
>
>> #. Finish moving over the docs.
>> #. Finish Sphinxifying them.
>
> Can this be done incrementally?  It would be great to start with one doc
(like LangRef) and make it really really good and get the infrastructure right,
then move on to other docs (which others might help out with).
>
> -Chris
Of course. I agree that it should be incremental. This really goes
with how this project should make its way into trunk. I was thinking
of making a docs/sphinx directory and putting {conf.py, index.rst, <a
few core docs>, _templates} in there. This will have no effect other
than adding the files. From there we can work on build system
integration and infrastructure.

The thing with "sphinxifying" is that a lot of it has to do with how
all of the documentation fits together, so it may be hard to get a
feel with only one doc. However, I think it would be a good idea to
start with just a few core docs (LangRef, GettingStarted{,VS}, ???)
and go from there.

- Michael Spencer

Michael Spencer <bigcheesegs at gmail.com> writes:

[snip]
> I'll need help integrating it into the build system.
I can help with the CMake build.

[snip]

On Mon, Aug 9, 2010 at 11:29 PM, Michael Spencer <bigcheesegs at
gmail.com> wrote:
> You are right that BoostBook is targeted directly for large C++
> projects, and I seriously considered BoostBook for this project, but
> ran into a few road blocks.
>
> * It's tightly integrated into boost and makes quite a few assumptions
> about that.
What assumptions is that?


On Mon, Aug 9, 2010 at 11:29 PM, Michael Spencer <bigcheesegs at
gmail.com> wrote:
> * It requires boost-build, which is a rather complex dependency and
> yet another full build system.
How does it require it?  I have used it directly before, without
boost-build (even if you wanted to simplify its use by calling it
using boost-build, it is not like it is CMake; boost-build, once
compiled, is a single executable, it requires no other files).


On Mon, Aug 9, 2010 at 11:29 PM, Michael Spencer <bigcheesegs at
gmail.com> wrote:
> * It just feels really complex all around.
Complex how so?  It is 'interesting' to set up, but once set up it is
extremely simple.  As well as the fact that to do some things in it
would be extremely simple compared to Sphinx which would not even
support the ability, such as the simple example of pulling in part of
some source code into the documentation straight from a compilable
source file, how would you do that with Sphinx?  Plenty of other
examples too.


On Mon, Aug 9, 2010 at 11:29 PM, Michael Spencer <bigcheesegs at
gmail.com> wrote:
> In the end I felt that there would be too much resistance to
> boost-build and the complexity. I then discovered Sphinx and found
> that it did everything I wanted and just felt clean and simple.

On Tue, Aug 10, 2010 at 1:09 PM, David A. Greene <greened at
obbligato.org> wrote:
> OvermindDL1 <overminddl1 at gmail.com> writes:
>
>> BoostBook is a simplification of DocBook, with many things specific
>> for C/C++ source.  it can pull in parts of a source file so the doc
>> examples always show examples of code that is actually tested, it has
>> proper links to source code files, can integrate with doxygen with a
>> bit of work, can export to html, pdf, and a few other things...  I
>> like Sphinx, a lot actually, but BoostBook is more designed for this
>> kind of project.
>
> BoostBook/QuickBook is really cool but I found it nearly impossible to
> use outside of the Boost tree.  In other words, a few years ago, at
> least, it was intimately tied to the Boost project.  That may have
> changed recently, I don't know.
What of it is too tied to Boost?  Once compiled it is completely standalone.


On Tue, Aug 10, 2010 at 1:09 PM, David A. Greene <greened at
obbligato.org> wrote:
> I also really hate the way QuickBook chops up and represents pages.  The
> Table-of-Contents + prev/next links format is really terrible.  It's
> very difficult to navigate non-linearly.
Actually that is an option when you define your documentation, a
simple single-line change can make it so it chops it up, or introduces
it as a single mass page properly linked together by anchors, and
there are many other patterns that can be done.  The Boosty way is
just to chop it up, but if you notice, not all of the library's
documentation does that.


On Tue, Aug 10, 2010 at 1:09 PM, David A. Greene <greened at
obbligato.org> wrote:
> The QuickBook language is really nice for specifying documentation so if
> we can fix the output format and make it work outside of Boost I'd be
> all for using it.
What output format issues do you have?  It is completely configurable,
a single file can define a specific look for all of the documentation.


On Tue, Aug 10, 2010 at 2:10 PM, Douglas Gregor <dgregor at apple.com>
wrote:
> The BoostBook tool-chain is a total nightmare to configure and maintain. It
produces decent, standards-style documentation and has a lot of great features,
but the setup and learning curve is far too steep for me to recommend its use
for LLVM.
>
>        - Original author of BoostBook
It is difficult to set up, but once setup it never needs to be touched
again, adding new pages are very simple, should not make it sound more
difficult then it is.

On Thu, Aug 12, 2010 at 5:28 PM, Aaron Gray <aaronngray.lists at
gmail.com> wrote:
> Can you do a quickish test and example, of LLVM doc using BoostBook ?
> I don't want to vote on group quite yet for political reasons.
> +1
> Aaron
I could, but it may take a little bit, time is something I do not have
currently, but as long as someone keeps poking/reminding me daily then
I shall.

On Thu, Aug 12, 2010 at 5:52 PM, Aaron Gray <aaronngray.lists at
gmail.com> wrote:
> poke !
I whipped up a quick example, this is a total of about an hour of work
due to finding parameters to set it up, once that is finished though
the actual documentation is simple.

Look in the html subdirectory to see the generated html as separate
files, but it can also generate a single large html file, pdf, man
pages, docbook, whatever.

The source qbk files and Jamroot is included, but it would be a cinch
to have CMake build it instead.  If you do:
  bjam -n html
then it will not build it, but it outputs all of the commands and
everything it does, which is surprisingly little, so yes, it could be
integrated with ease.

I ported the first two chapters of the C++ Kaleidoscope tutorial to
it.  I used the standard boost-style setup, except as an example I
changed the section headers to be LLVM-doc style, was simple.  The
boost standard DTD and XSL and such files can be edited to change
style as necessary as well, but CSS should work for the bulk.  I did
not bother changing style of much, but to have autogenerated TOC at
the top of those would be a single option change, etc...  Many things
can be done, and as you can see the documentation style of the qbk
files is simple, and C++ and such is auto color coded, although that
could be disabled if wanted.

To show how the documentation can stay in sync with the source, I
included the Kaleidoscope source in the example and I edited the
Chapter 2 source to add a few lines of markup to let it pull in the
source automatically so the documentation is always in sync with the
source as they are one and the same.  I also showed how to link to the
cpp source instead of embedding it at the bottom of Chapter 2, but
that could be done as well if needed.

I have all the chapter's in there, but the rest are empty, I just
included them to let all the links work.

If there is a specific example that you want demonstrated, just say.

To build the example yourself, just put the two qbk files and the
Jamroot in a directory, make sure you have boost's quickbook and
boostbook built, then just do:
  /path/to/bjam
and it will build the multi-part html by default, or of course do:
  /path/to/bjam -n
to see what it is doing instead, which basically involves just
creating a system-customized link to the dtd and xsl and such
directories in boostbook (if LLVM used this then you would include
those in LLVM itself, probably customized, and the manifest would not
need to be system-generated-specific each time, it would be static),
then calling quickbook, then calling xsltproc twice, quite simple, and
I have ran the steps manually myself to confirm they work, and they
do.

Correction, even the zip by itself is too big, here is the 7z, if
someone wants a giant zip, I can host it somewhere...

On Sat, Aug 14, 2010 at 6:49 PM, OvermindDL1 <overminddl1 at gmail.com>
wrote:
> The zip format was too big to fit through the moderator queue, so here
> is another with just the zip by itself (although the 7z is only
> 29kb...).
>
> On Sat, Aug 14, 2010 at 6:29 PM, OvermindDL1 <overminddl1 at
gmail.com> wrote:
>> Helps if I attach the files...
>> Attached a zip for those that like the huge format, and a 7z for those
>> that like tiny formats.
>>
>> On Sat, Aug 14, 2010 at 6:21 PM, OvermindDL1 <overminddl1 at
gmail.com> wrote:
>>> On Thu, Aug 12, 2010 at 5:52 PM, Aaron Gray <aaronngray.lists at
gmail.com> wrote:
>>>> poke !
>>>
>>> I whipped up a quick example, this is a total of about an hour of
work
>>> due to finding parameters to set it up, once that is finished
though
>>> the actual documentation is simple.
>>>
>>> Look in the html subdirectory to see the generated html as separate
>>> files, but it can also generate a single large html file, pdf, man
>>> pages, docbook, whatever.
>>>
>>> The source qbk files and Jamroot is included, but it would be a
cinch
>>> to have CMake build it instead.  If you do:
>>>  bjam -n html
>>> then it will not build it, but it outputs all of the commands and
>>> everything it does, which is surprisingly little, so yes, it could
be
>>> integrated with ease.
>>>
>>> I ported the first two chapters of the C++ Kaleidoscope tutorial to
>>> it.  I used the standard boost-style setup, except as an example I
>>> changed the section headers to be LLVM-doc style, was simple.  The
>>> boost standard DTD and XSL and such files can be edited to change
>>> style as necessary as well, but CSS should work for the bulk.  I
did
>>> not bother changing style of much, but to have autogenerated TOC at
>>> the top of those would be a single option change, etc...  Many
things
>>> can be done, and as you can see the documentation style of the qbk
>>> files is simple, and C++ and such is auto color coded, although
that
>>> could be disabled if wanted.
>>>
>>> To show how the documentation can stay in sync with the source, I
>>> included the Kaleidoscope source in the example and I edited the
>>> Chapter 2 source to add a few lines of markup to let it pull in the
>>> source automatically so the documentation is always in sync with
the
>>> source as they are one and the same.  I also showed how to link to
the
>>> cpp source instead of embedding it at the bottom of Chapter 2, but
>>> that could be done as well if needed.
>>>
>>> I have all the chapter's in there, but the rest are empty, I
just
>>> included them to let all the links work.
>>>
>>> If there is a specific example that you want demonstrated, just
say.
>>>
>>> To build the example yourself, just put the two qbk files and the
>>> Jamroot in a directory, make sure you have boost's quickbook
and
>>> boostbook built, then just do:
>>>  /path/to/bjam
>>> and it will build the multi-part html by default, or of course do:
>>>  /path/to/bjam -n
>>> to see what it is doing instead, which basically involves just
>>> creating a system-customized link to the dtd and xsl and such
>>> directories in boostbook (if LLVM used this then you would include
>>> those in LLVM itself, probably customized, and the manifest would
not
>>> need to be system-generated-specific each time, it would be
static),
>>> then calling quickbook, then calling xsltproc twice, quite simple,
and
>>> I have ran the steps manually myself to confirm they work, and they
>>> do.
>>>
>>
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: llvm_doc.7z
Type: application/octet-stream
Size: 29309 bytes
Desc: not available
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20100814/e839ce00/attachment.obj>

OvermindDL1 <overminddl1 at gmail.com> writes:
> On Tue, Aug 10, 2010 at 1:09 PM, David A. Greene <greened at
obbligato.org> wrote:
>> BoostBook/QuickBook is really cool but I found it nearly impossible to
>> use outside of the Boost tree.  In other words, a few years ago, at
>> least, it was intimately tied to the Boost project.  That may have
>> changed recently, I don't know.
>
> What of it is too tied to Boost?  Once compiled it is completely
standalone.
It's been a while, but IIRC the doxygen integration relied on bjam
somehow, finding the tools perhaps?  The BoostBook XML files also
hard-code some assumptions about the project layout, IIRC.

If building the tool requires Boost.Build that is a pretty heavy burden.
> On Tue, Aug 10, 2010 at 1:09 PM, David A. Greene <greened at
obbligato.org> wrote:
>> I also really hate the way QuickBook chops up and represents pages. 
The
>> Table-of-Contents + prev/next links format is really terrible. 
It's
>> very difficult to navigate non-linearly.
>
> Actually that is an option when you define your documentation, a
> simple single-line change can make it so it chops it up, or introduces
> it as a single mass page properly linked together by anchors, and
> there are many other patterns that can be done.  The Boosty way is
> just to chop it up, but if you notice, not all of the library's
> documentation does that.
Well, that's good news!  I know that various Boost libraries do it
different ways (compare MPL to Proto), but I've never found a scheme I
really liked.  Of course, that's personal preference, but ease of
navigation does matter.
> On Tue, Aug 10, 2010 at 1:09 PM, David A. Greene <greened at
obbligato.org> wrote:
>> The QuickBook language is really nice for specifying documentation so
if
>> we can fix the output format and make it work outside of Boost I'd
be
>> all for using it.
>
> What output format issues do you have?  It is completely configurable,
> a single file can define a specific look for all of the documentation.
If that's true and we can make it do what we want, I'm all for
QuickBook.
> On Tue, Aug 10, 2010 at 2:10 PM, Douglas Gregor <dgregor at
apple.com> wrote:
>> The BoostBook tool-chain is a total nightmare to configure and
maintain. It produces decent, standards-style documentation and has a lot of
great features, but the setup and learning curve is far too steep for me to
recommend its use for LLVM.
>>
>>        - Original author of BoostBook
>
> It is difficult to set up, but once setup it never needs to be touched
> again, adding new pages are very simple, should not make it sound more
> difficult then it is.
But it should be possible for all of us to build LLVM's documentation
tools.  I really don't want to see bjam code in LLVM *shudder*.

It would be a great contribution if someone could port the
BoostBook/QuickBook build to make and completely divorce it from Boost
proper.  It's a tool a lot of projects could really use, IMHO.

                               -Dave

OvermindDL1 <overminddl1 at gmail.com> writes:
> Correction, even the zip by itself is too big, here is the 7z, if
> someone wants a giant zip, I can host it somewhere...
Please do.  7z is not supported on Linux.  I would love to take a look
at this, but I can't.  :(

                            -Dave

On Sat, Aug 14, 2010 at 8:21 PM, OvermindDL1 <overminddl1 at gmail.com>
wrote:
> On Thu, Aug 12, 2010 at 5:52 PM, Aaron Gray <aaronngray.lists at
gmail.com> wrote:
>> poke !
>
> I whipped up a quick example, this is a total of about an hour of work
> due to finding parameters to set it up, once that is finished though
> the actual documentation is simple.
>
> Look in the html subdirectory to see the generated html as separate
> files, but it can also generate a single large html file, pdf, man
> pages, docbook, whatever.
>
> The source qbk files and Jamroot is included, but it would be a cinch
> to have CMake build it instead.  If you do:
>  bjam -n html
> then it will not build it, but it outputs all of the commands and
> everything it does, which is surprisingly little, so yes, it could be
> integrated with ease.
>
> I ported the first two chapters of the C++ Kaleidoscope tutorial to
> it.  I used the standard boost-style setup, except as an example I
> changed the section headers to be LLVM-doc style, was simple.  The
> boost standard DTD and XSL and such files can be edited to change
> style as necessary as well, but CSS should work for the bulk.  I did
> not bother changing style of much, but to have autogenerated TOC at
> the top of those would be a single option change, etc...  Many things
> can be done, and as you can see the documentation style of the qbk
> files is simple, and C++ and such is auto color coded, although that
> could be disabled if wanted.
>
> To show how the documentation can stay in sync with the source, I
> included the Kaleidoscope source in the example and I edited the
> Chapter 2 source to add a few lines of markup to let it pull in the
> source automatically so the documentation is always in sync with the
> source as they are one and the same.  I also showed how to link to the
> cpp source instead of embedding it at the bottom of Chapter 2, but
> that could be done as well if needed.
>
> I have all the chapter's in there, but the rest are empty, I just
> included them to let all the links work.
>
> If there is a specific example that you want demonstrated, just say.
>
> To build the example yourself, just put the two qbk files and the
> Jamroot in a directory, make sure you have boost's quickbook and
> boostbook built, then just do:
>  /path/to/bjam
> and it will build the multi-part html by default, or of course do:
>  /path/to/bjam -n
> to see what it is doing instead, which basically involves just
> creating a system-customized link to the dtd and xsl and such
> directories in boostbook (if LLVM used this then you would include
> those in LLVM itself, probably customized, and the manifest would not
> need to be system-generated-specific each time, it would be static),
> then calling quickbook, then calling xsltproc twice, quite simple, and
> I have ran the steps manually myself to confirm they work, and they
> do.
Thanks for setting this up, I didn't even attempt to use boostbook. I
like the quickbook syntax.

I finally got around to trying this out on my windows machine. The
problem is that you have to have a full boost installation so that you
can compile quickbook and boostbook as there are no packaged binaries.
There are also 3 other packages you have to install.

Sphinx can include a file as source code. It is also extensible in
python by simply adding a python file and adding a reference to it in
the config file.

- Michael Spencer