llvm dev - Jun 2019 - [docs] Updating the sphinx build bots

Hi,

TL;DR - We should update the bot(s) responsible for building the LLVM
documentation so that they use recommonmark 0.5.0, and apply
https://reviews.llvm.org/D63211.

Full context:

I recently started trying to build the sphinx docs. However, a change by
Jordan (r363100) resulted in me no longer being able to build them, even
though the build bots were green and he could build them himself. After
some digging, it looks like it's an issue with the latest version of the
recommonmark python module. I verified that I don't see the issue with
v0.4.0, but do with v0.5.0. The sphinx version doesn't seem to make any
difference.

I could downgrade my module version (and have temporarily done so).
However, that leads to another issue: the links to llvm-symbolizer's page
in https://llvm.org/docs/CommandGuide/llvm-addr2line.html are broken when
building with 0.4.0. This can be fixed by changing the markdown doc to
explicitly specify ".html" in the link, but this precise change is
what
breaks 0.5.0 (which otherwise produces working links in the llvm-addr2line
doc). I think we should update the build bots version of recommonmark.
Since this would cause the build to break, we should simultaneously (or
more exactly slightly before) apply the patch in
https://reviews.llvm.org/D63211, which fixes the warning that I am seeing
with 0.5.0.

I've included a couple of people I know to be involved with either the
website or the build bots. Sorry if it's misdirected!

Thoughts?

James
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20190613/bfb80f16/attachment.html>

(+cc gribozavr, maintainer of gribozavr3 buildslave)

I played around with this yesterday and today (I had to head out before I
could reply on D63211).

The recommonmark version is what matters most, but the Sphinx version isn't
completely irrelevant. For instance, the current way we install the
recommonmark module is deprecated in >1.8 and removed in 3.0 (trunk, seems
to be unreleased): http://www.sphinx-doc.org/en/master/usage/markdown.html.
However, when I change registration that way, I get errors that "extension
'recommonmark' has no setup() function".

(This is tangential to the original problem, but I note it because if you
pull Sphinx from trunk and try to build llvm docs, it will not work because
of this. If possible, we should try to deal with it now so we only have to
reconfigure buildbots once)

My suggested approach (no buildbot breakages):
- Change broken links between md<->rst files to plain html links
(annoying,
but temporary)
- Updated recommonmark on buildbots to 0.5.0 (presumably it's at <= 0.4.0
on build bots, but I can't find a way to verify this)
- Change html links back to normal markdown syntax
- Update Sphinx to >1.8 (and < 3.0) (possibly waiting until it hits
whatever stable repo the buildbot installs from)
- Switch the registration method of recommonmark in conf.py to the
non-deprecated way
- Upgrading to Sphinx >= 3.0 will not be a problem now

On Thu, Jun 13, 2019 at 6:18 AM James Henderson <
jh7370.2008 at my.bristol.ac.uk> wrote:
> Hi,
>
> TL;DR - We should update the bot(s) responsible for building the LLVM
> documentation so that they use recommonmark 0.5.0, and apply
> https://reviews.llvm.org/D63211.
>
> Full context:
>
> I recently started trying to build the sphinx docs. However, a change by
> Jordan (r363100) resulted in me no longer being able to build them, even
> though the build bots were green and he could build them himself. After
> some digging, it looks like it's an issue with the latest version of
the
> recommonmark python module. I verified that I don't see the issue with
> v0.4.0, but do with v0.5.0. The sphinx version doesn't seem to make any
> difference.
>
> I could downgrade my module version (and have temporarily done so).
> However, that leads to another issue: the links to llvm-symbolizer's
page
> in https://llvm.org/docs/CommandGuide/llvm-addr2line.html are broken when
> building with 0.4.0. This can be fixed by changing the markdown doc to
> explicitly specify ".html" in the link, but this precise change
is what
> breaks 0.5.0 (which otherwise produces working links in the llvm-addr2line
> doc). I think we should update the build bots version of recommonmark.
> Since this would cause the build to break, we should simultaneously (or
> more exactly slightly before) apply the patch in
> https://reviews.llvm.org/D63211, which fixes the warning that I am seeing
> with 0.5.0.
>
> I've included a couple of people I know to be involved with either the
> website or the build bots. Sorry if it's misdirected!
>
> Thoughts?
>
> James
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20190613/45a67cc2/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4849 bytes
Desc: S/MIME Cryptographic Signature
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20190613/45a67cc2/attachment.bin>

On Thu, Jun 13, 2019 at 11:16 AM Jordan Rupprecht <rupprecht at
google.com>
wrote:
> (+cc gribozavr, maintainer of gribozavr3 buildslave)
>
> I played around with this yesterday and today (I had to head out before I
> could reply on D63211).
>
> The recommonmark version is what matters most, but the Sphinx version
> isn't completely irrelevant. For instance, the current way we install
the
> recommonmark module is deprecated in >1.8 and removed in 3.0 (trunk,
seems
> to be unreleased): http://www.sphinx-doc.org/en/master/usage/markdown.html.
> However, when I change registration that way, I get errors that
"extension
> 'recommonmark' has no setup() function".
>
To be clear: this error is w/ Sphinx 1.7.5 + recommonmark 0.5.0. There is
no error here w/ Sphinx >= 1.8.

> (This is tangential to the original problem, but I note it because if you
> pull Sphinx from trunk and try to build llvm docs, it will not work because
> of this. If possible, we should try to deal with it now so we only have to
> reconfigure buildbots once)
>
> My suggested approach (no buildbot breakages):
> - Change broken links between md<->rst files to plain html links
> (annoying, but temporary)
>
Mailed https://reviews.llvm.org/D63292
(btw, sphinx also logs some unrelated warnings, fixed w/
https://reviews.llvm.org/rL363294)
> - Updated recommonmark on buildbots to 0.5.0 (presumably it's at <=
0.4.0
> on build bots, but I can't find a way to verify this)
> - Change html links back to normal markdown syntax
> - Update Sphinx to >1.8 (and < 3.0) (possibly waiting until it hits
> whatever stable repo the buildbot installs from)
> - Switch the registration method of recommonmark in conf.py to the
> non-deprecated way
> - Upgrading to Sphinx >= 3.0 will not be a problem now
>
> On Thu, Jun 13, 2019 at 6:18 AM James Henderson <
> jh7370.2008 at my.bristol.ac.uk> wrote:
>
>> Hi,
>>
>> TL;DR - We should update the bot(s) responsible for building the LLVM
>> documentation so that they use recommonmark 0.5.0, and apply
>> https://reviews.llvm.org/D63211.
>>
>> Full context:
>>
>> I recently started trying to build the sphinx docs. However, a change
by
>> Jordan (r363100) resulted in me no longer being able to build them,
even
>> though the build bots were green and he could build them himself. After
>> some digging, it looks like it's an issue with the latest version
of the
>> recommonmark python module. I verified that I don't see the issue
with
>> v0.4.0, but do with v0.5.0. The sphinx version doesn't seem to make
any
>> difference.
>>
>> I could downgrade my module version (and have temporarily done so).
>> However, that leads to another issue: the links to
llvm-symbolizer's page
>> in https://llvm.org/docs/CommandGuide/llvm-addr2line.html are broken
>> when building with 0.4.0. This can be fixed by changing the markdown
doc to
>> explicitly specify ".html" in the link, but this precise
change is what
>> breaks 0.5.0 (which otherwise produces working links in the
llvm-addr2line
>> doc). I think we should update the build bots version of recommonmark.
>> Since this would cause the build to break, we should simultaneously (or
>> more exactly slightly before) apply the patch in
>> https://reviews.llvm.org/D63211, which fixes the warning that I am
>> seeing with 0.5.0.
>>
>> I've included a couple of people I know to be involved with either
the
>> website or the build bots. Sorry if it's misdirected!
>>
>> Thoughts?
>>
>> James
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20190613/39769df6/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4849 bytes
Desc: S/MIME Cryptographic Signature
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20190613/39769df6/attachment-0001.bin>

Hi,

Thanks for bringing this up. I am reluctant to update to
recommonmark
>= 0.4.0, because it is not available in any Ubuntu version, Debian
Testing or even Debian Unstable.

While we could do something custom on the buildbot, I think it would
be a disservice for our users to use a library that is not packaged in
modern operating systems yet.

Dmitri