Xen devel - Jul 2012 - [PATCH DOCDAY] use lynx to produce better formatted text documentation from markdown

Markdown, while easy to read and write, isn''t the most consumable
format for users reading documentation on a terminal. This patch uses
lynx to format markdown produced HTML into text files.

Signed-off-by: Matt Wilson <msw@amazon.com>

diff -r c3a6e679bdfa -r ef1271aef866 docs/Docs.mk
--- a/docs/Docs.mk	Mon Jul 30 19:04:59 2012 +0000
+++ b/docs/Docs.mk	Mon Jul 30 19:33:41 2012 +0000
@@ -10,3 +10,4 @@ POD2TEXT	:= pod2text
 DOT		:= dot
 NEATO		:= neato
 MARKDOWN	:= markdown
+LYNX		:= lynx
diff -r c3a6e679bdfa -r ef1271aef866 docs/Makefile
--- a/docs/Makefile	Mon Jul 30 19:04:59 2012 +0000
+++ b/docs/Makefile	Mon Jul 30 19:33:41 2012 +0000
@@ -103,7 +103,16 @@ html/%.html: %.markdown
 
 html/%.txt: %.txt
 	@$(INSTALL_DIR) $(@D)
-	cp $< $@
+	@set -e ; \
+	if which $(MARKDOWN) >/dev/null 2>&1 && \
+		which $(LYNX) >/dev/null 2>&1 ; then \
+		echo "Running markdown to generate $*.txt ... "; \
+		$(MARKDOWN) $< | lynx -dump -stdin > $@.tmp ; \
+		$(call move-if-changed,$@.tmp,$@) ; \
+	else \
+		echo "markdown or lynx not installed; just copying $<."; \
+		cp $< $@; \
+	fi
 
 html/man/%.1.html: man/%.pod.1 Makefile
 	$(INSTALL_DIR) $(@D)
@@ -131,9 +140,17 @@ txt/%.txt: %.txt
 	$(call move-if-changed,$@.tmp,$@)
 
 txt/%.txt: %.markdown
-	$(INSTALL_DIR) $(@D)
-	cp $< $@.tmp
-	$(call move-if-changed,$@.tmp,$@)
+	@$(INSTALL_DIR) $(@D)
+	@set -e ; \
+	if which $(MARKDOWN) >/dev/null 2>&1 && \
+		which $(LYNX) >/dev/null 2>&1 ; then \
+		echo "Running markdown to generate $*.txt ... "; \
+		$(MARKDOWN) $< | lynx -dump -stdin > $@.tmp ; \
+		$(call move-if-changed,$@.tmp,$@) ; \
+	else \
+		echo "markdown or lynx not installed; just copying $<."; \
+		cp $< $@; \
+	fi
 
 txt/man/%.1.txt: man/%.pod.1 Makefile
 	$(INSTALL_DIR) $(@D)

On Mon, 2012-07-30 at 20:33 +0100, Matt Wilson wrote:
> Markdown, while easy to read and write, isn''t the most consumable
> format for users reading documentation on a terminal. This patch uses
> lynx to format markdown produced HTML into text files.
The markdown syntax is supposed to be readable as plain text too, if
there are particular instances where this is not the case perhaps we can
tidy them up with that in mind?

Why wouldn''t you just run lynx on the generated .html instead of less
on
the generated .txt if you wanted something a bit better formatted?
> 
> Signed-off-by: Matt Wilson <msw@amazon.com>
> 
> diff -r c3a6e679bdfa -r ef1271aef866 docs/Docs.mk
> --- a/docs/Docs.mk	Mon Jul 30 19:04:59 2012 +0000
> +++ b/docs/Docs.mk	Mon Jul 30 19:33:41 2012 +0000
> @@ -10,3 +10,4 @@ POD2TEXT	:= pod2text
>  DOT		:= dot
>  NEATO		:= neato
>  MARKDOWN	:= markdown
> +LYNX		:= lynx
> diff -r c3a6e679bdfa -r ef1271aef866 docs/Makefile
> --- a/docs/Makefile	Mon Jul 30 19:04:59 2012 +0000
> +++ b/docs/Makefile	Mon Jul 30 19:33:41 2012 +0000
> @@ -103,7 +103,16 @@ html/%.html: %.markdown
>  
>  html/%.txt: %.txt
>  	@$(INSTALL_DIR) $(@D)
> -	cp $< $@
> +	@set -e ; \
> +	if which $(MARKDOWN) >/dev/null 2>&1 && \
> +		which $(LYNX) >/dev/null 2>&1 ; then \
> +		echo "Running markdown to generate $*.txt ... "; \
> +		$(MARKDOWN) $< | lynx -dump -stdin > $@.tmp ; \
> +		$(call move-if-changed,$@.tmp,$@) ; \
> +	else \
> +		echo "markdown or lynx not installed; just copying $<."; \
> +		cp $< $@; \
> +	fi
Does formatting a non-markdown .txt file like this produce reasonable
results for all the random ASCII formatting used under misc?
 
>  html/man/%.1.html: man/%.pod.1 Makefile
>  	$(INSTALL_DIR) $(@D)
> @@ -131,9 +140,17 @@ txt/%.txt: %.txt
>  	$(call move-if-changed,$@.tmp,$@)
>  
>  txt/%.txt: %.markdown
> -	$(INSTALL_DIR) $(@D)
> -	cp $< $@.tmp
> -	$(call move-if-changed,$@.tmp,$@)
> +	@$(INSTALL_DIR) $(@D)
> +	@set -e ; \
> +	if which $(MARKDOWN) >/dev/null 2>&1 && \
> +		which $(LYNX) >/dev/null 2>&1 ; then \
> +		echo "Running markdown to generate $*.txt ... "; \
> +		$(MARKDOWN) $< | lynx -dump -stdin > $@.tmp ; \
> +		$(call move-if-changed,$@.tmp,$@) ; \
> +	else \
> +		echo "markdown or lynx not installed; just copying $<."; \
> +		cp $< $@; \
> +	fi
>  
>  txt/man/%.1.txt: man/%.pod.1 Makefile
>  	$(INSTALL_DIR) $(@D)
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

On Tue, Jul 31, 2012 at 01:29:03AM -0700, Ian Campbell
wrote:
> On Mon, 2012-07-30 at 20:33 +0100, Matt Wilson wrote:
> > Markdown, while easy to read and write, isn''t the most
consumable
> > format for users reading documentation on a terminal. This patch uses
> > lynx to format markdown produced HTML into text files.
> 
> The markdown syntax is supposed to be readable as plain text too, if
> there are particular instances where this is not the case perhaps we can
> tidy them up with that in mind?
I''m not sure how much the markdown can be tidied for constructs like:

### apic
> `= summit | bigsmp | default`
Override Xen''s logic for choosing the APIC driver.  By default, if
there are more than 8 CPUs, Xen will switch to `bigsmp` over
`default`.

### allow\_unsafe
> `= <boolean>`
> Default: `false`
Force boot on potentially unsafe systems. By default Xen will refuse
to boot on systems with the following errata:

* AMD Erratum 121. Processors with this erratum are subject to a guest
  triggerable Denial of Service. Override only if you trust all of
  your PV guests.

When processed as I propose, it looks like:

  apic

     = summit | bigsmp | default

   Override Xen''s logic for choosing the APIC driver. By default, if
there
   are more than 8 CPUs, Xen will switch to bigsmp over default.

  allow_unsafe

     = <boolean>

     Default: false

   Force boot on potentially unsafe systems. By default Xen will refuse to
   boot on systems with the following errata:
     * AMD Erratum 121. Processors with this erratum are subject to a
       guest triggerable Denial of Service. Override only if you trust all
       of your PV guests.

> Why wouldn''t you just run lynx on the generated .html instead of
less on
> the generated .txt if you wanted something a bit better formatted?
I generally don''t have lynx installed on my production machines.
> > Signed-off-by: Matt Wilson <msw@amazon.com>
> > 
> > diff -r c3a6e679bdfa -r ef1271aef866 docs/Docs.mk
> > --- a/docs/Docs.mk	Mon Jul 30 19:04:59 2012 +0000
> > +++ b/docs/Docs.mk	Mon Jul 30 19:33:41 2012 +0000
> > @@ -10,3 +10,4 @@ POD2TEXT	:= pod2text
> >  DOT		:= dot
> >  NEATO		:= neato
> >  MARKDOWN	:= markdown
> > +LYNX		:= lynx
> > diff -r c3a6e679bdfa -r ef1271aef866 docs/Makefile
> > --- a/docs/Makefile	Mon Jul 30 19:04:59 2012 +0000
> > +++ b/docs/Makefile	Mon Jul 30 19:33:41 2012 +0000
> > @@ -103,7 +103,16 @@ html/%.html: %.markdown
> >  
> >  html/%.txt: %.txt
> >  	@$(INSTALL_DIR) $(@D)
> > -	cp $< $@
> > +	@set -e ; \
> > +	if which $(MARKDOWN) >/dev/null 2>&1 && \
> > +		which $(LYNX) >/dev/null 2>&1 ; then \
> > +		echo "Running markdown to generate $*.txt ... "; \
> > +		$(MARKDOWN) $< | lynx -dump -stdin > $@.tmp ; \
> > +		$(call move-if-changed,$@.tmp,$@) ; \
> > +	else \
> > +		echo "markdown or lynx not installed; just copying
$<."; \
> > +		cp $< $@; \
> > +	fi
> 
> Does formatting a non-markdown .txt file like this produce reasonable
> results for all the random ASCII formatting used under misc?
Oops, sorry. This is bogus. I''ll resubmit with it removed.

Matt
> >  html/man/%.1.html: man/%.pod.1 Makefile
> >  	$(INSTALL_DIR) $(@D)
> > @@ -131,9 +140,17 @@ txt/%.txt: %.txt
> >  	$(call move-if-changed,$@.tmp,$@)
> >  
> >  txt/%.txt: %.markdown
> > -	$(INSTALL_DIR) $(@D)
> > -	cp $< $@.tmp
> > -	$(call move-if-changed,$@.tmp,$@)
> > +	@$(INSTALL_DIR) $(@D)
> > +	@set -e ; \
> > +	if which $(MARKDOWN) >/dev/null 2>&1 && \
> > +		which $(LYNX) >/dev/null 2>&1 ; then \
> > +		echo "Running markdown to generate $*.txt ... "; \
> > +		$(MARKDOWN) $< | lynx -dump -stdin > $@.tmp ; \
> > +		$(call move-if-changed,$@.tmp,$@) ; \
> > +	else \
> > +		echo "markdown or lynx not installed; just copying
$<."; \
> > +		cp $< $@; \
> > +	fi
> >  
> >  txt/man/%.1.txt: man/%.pod.1 Makefile
> >  	$(INSTALL_DIR) $(@D)
> > 
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xen.org
> > http://lists.xen.org/xen-devel
> 
>

On Tue, Jul 31, 2012 at 08:34:59AM -0700, Matt Wilson
wrote:
> On Tue, Jul 31, 2012 at 01:29:03AM -0700, Ian Campbell wrote:
> > On Mon, 2012-07-30 at 20:33 +0100, Matt Wilson wrote:
> > > Markdown, while easy to read and write, isn''t the most
consumable
> > > format for users reading documentation on a terminal. This patch
uses
> > > lynx to format markdown produced HTML into text files.
> > 
> > The markdown syntax is supposed to be readable as plain text too, if
> > there are particular instances where this is not the case perhaps we
can
> > tidy them up with that in mind?
> 
> I''m not sure how much the markdown can be tidied for constructs
like:
> 
> ### apic
> > `= summit | bigsmp | default`
> 
> Override Xen''s logic for choosing the APIC driver.  By default, if
> there are more than 8 CPUs, Xen will switch to `bigsmp` over
> `default`.
> 
> ### allow\_unsafe
> > `= <boolean>`
> 
> > Default: `false`
> 
> Force boot on potentially unsafe systems. By default Xen will refuse
> to boot on systems with the following errata:
> 
> * AMD Erratum 121. Processors with this erratum are subject to a guest
>   triggerable Denial of Service. Override only if you trust all of
>   your PV guests.
> 
> When processed as I propose, it looks like:
> 
>   apic
> 
>      = summit | bigsmp | default
> 
>    Override Xen''s logic for choosing the APIC driver. By default,
if there
>    are more than 8 CPUs, Xen will switch to bigsmp over default.
> 
>   allow_unsafe
> 
>      = <boolean>
> 
>      Default: false
> 
>    Force boot on potentially unsafe systems. By default Xen will refuse to
>    boot on systems with the following errata:
>      * AMD Erratum 121. Processors with this erratum are subject to a
>        guest triggerable Denial of Service. Override only if you trust all
>        of your PV guests.
> 
> 
> > Why wouldn''t you just run lynx on the generated .html instead
of less on
> > the generated .txt if you wanted something a bit better formatted?
> 
> I generally don''t have lynx installed on my production machines.
Ian,

Any further concerns?

Matt
> > > Signed-off-by: Matt Wilson <msw@amazon.com>
> > > 
> > > diff -r c3a6e679bdfa -r ef1271aef866 docs/Docs.mk
> > > --- a/docs/Docs.mk	Mon Jul 30 19:04:59 2012 +0000
> > > +++ b/docs/Docs.mk	Mon Jul 30 19:33:41 2012 +0000
> > > @@ -10,3 +10,4 @@ POD2TEXT	:= pod2text
> > >  DOT		:= dot
> > >  NEATO		:= neato
> > >  MARKDOWN	:= markdown
> > > +LYNX		:= lynx
> > > diff -r c3a6e679bdfa -r ef1271aef866 docs/Makefile
> > > --- a/docs/Makefile	Mon Jul 30 19:04:59 2012 +0000
> > > +++ b/docs/Makefile	Mon Jul 30 19:33:41 2012 +0000
> > > @@ -103,7 +103,16 @@ html/%.html: %.markdown
> > >  
> > >  html/%.txt: %.txt
> > >  	@$(INSTALL_DIR) $(@D)
> > > -	cp $< $@
> > > +	@set -e ; \
> > > +	if which $(MARKDOWN) >/dev/null 2>&1 && \
> > > +		which $(LYNX) >/dev/null 2>&1 ; then \
> > > +		echo "Running markdown to generate $*.txt ... "; \
> > > +		$(MARKDOWN) $< | lynx -dump -stdin > $@.tmp ; \
> > > +		$(call move-if-changed,$@.tmp,$@) ; \
> > > +	else \
> > > +		echo "markdown or lynx not installed; just copying
$<."; \
> > > +		cp $< $@; \
> > > +	fi
> > 
> > Does formatting a non-markdown .txt file like this produce reasonable
> > results for all the random ASCII formatting used under misc?
> 
> Oops, sorry. This is bogus. I''ll resubmit with it removed.
> 
> Matt
> 
> > >  html/man/%.1.html: man/%.pod.1 Makefile
> > >  	$(INSTALL_DIR) $(@D)
> > > @@ -131,9 +140,17 @@ txt/%.txt: %.txt
> > >  	$(call move-if-changed,$@.tmp,$@)
> > >  
> > >  txt/%.txt: %.markdown
> > > -	$(INSTALL_DIR) $(@D)
> > > -	cp $< $@.tmp
> > > -	$(call move-if-changed,$@.tmp,$@)
> > > +	@$(INSTALL_DIR) $(@D)
> > > +	@set -e ; \
> > > +	if which $(MARKDOWN) >/dev/null 2>&1 && \
> > > +		which $(LYNX) >/dev/null 2>&1 ; then \
> > > +		echo "Running markdown to generate $*.txt ... "; \
> > > +		$(MARKDOWN) $< | lynx -dump -stdin > $@.tmp ; \
> > > +		$(call move-if-changed,$@.tmp,$@) ; \
> > > +	else \
> > > +		echo "markdown or lynx not installed; just copying
$<."; \
> > > +		cp $< $@; \
> > > +	fi
> > >  
> > >  txt/man/%.1.txt: man/%.pod.1 Makefile
> > >  	$(INSTALL_DIR) $(@D)
> > > 
> > > _______________________________________________
> > > Xen-devel mailing list
> > > Xen-devel@lists.xen.org
> > > http://lists.xen.org/xen-devel
> > 
> > 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

On Tue, 2012-08-07 at 04:22 +0100, Matt Wilson wrote:
> On Tue, Jul 31, 2012 at 08:34:59AM -0700, Matt Wilson wrote:
> > > Why wouldn''t you just run lynx on the generated .html
instead of less on
> > > the generated .txt if you wanted something a bit better
formatted?
> > 
> > I generally don''t have lynx installed on my production
machines.
You can read it on your workstation or on line at
http://xenbits.xen.org/docs/unstable/misc/xen-command-line.html
> Any further concerns?
I''m afraid I just don''t see the point. The plain markdown docs
are
serviceable enough, if not great, and there are plenty of ways to view
the html docs both on and off line if you want something prettier,
without adding a web browser to our build dependencies.

Ian.

On Tue, Aug 07, 2012 at 01:23:02AM -0700, Ian Campbell
wrote:
> On Tue, 2012-08-07 at 04:22 +0100, Matt Wilson wrote:
> > On Tue, Jul 31, 2012 at 08:34:59AM -0700, Matt Wilson wrote:
> > > > Why wouldn''t you just run lynx on the generated
.html instead of less on
> > > > the generated .txt if you wanted something a bit better
formatted?
> > > 
> > > I generally don''t have lynx installed on my production
machines.
> 
> You can read it on your workstation or on line at
> http://xenbits.xen.org/docs/unstable/misc/xen-command-line.html
> 
> > Any further concerns?
> 
> I''m afraid I just don''t see the point. The plain markdown
docs are
> serviceable enough, if not great, and there are plenty of ways to view
> the html docs both on and off line if you want something prettier,
> without adding a web browser to our build dependencies.
I''m certainly not suggesting that we add a web browser to the build
dependencies. If lynx isn''t installed, the current behavior of copying
the markdown file is maintained.

If a packager wishes to produce prettier text documentation, they can
elect to add lynx to their build dependencies. Today doing this
post-build from build control files is a bit tricky, since we drop the
semantic information conveyed by the .markdown suffix by calling the
final file .txt.

4.2 will be the first release with markdown documentation. I think
that making it well formatted, just as the previous .txt
documentation, will be a better experience for the user.

Matt

Matt Wilson writes ("Re: [Xen-devel] [PATCH DOCDAY] use lynx to produce
better formatted text documentation from
markdown"):
> I''m certainly not suggesting that we add a web browser to the
build
> dependencies. If lynx isn''t installed, the current behavior of
copying
> the markdown file is maintained.
> 
> If a packager wishes to produce prettier text documentation, they can
> elect to add lynx to their build dependencies. Today doing this
> post-build from build control files is a bit tricky, since we drop the
> semantic information conveyed by the .markdown suffix by calling the
> final file .txt.
> 
> 4.2 will be the first release with markdown documentation. I think
> that making it well formatted, just as the previous .txt
> documentation, will be a better experience for the user.
I tend to agree with this line of argument.  But I have orthogonal
queries:

Firstly, why lynx and not w3m ?  Is lynx even maintained upstream any
more ?  Secondly, shouldn''t we do the testing for the presence of
markdown and the html formatter in configure ?

Ian.

On Mon, Aug 13, 2012 at 07:16:54AM -0700, Ian Jackson
wrote:
> Matt Wilson writes ("Re: [Xen-devel] [PATCH DOCDAY] use lynx to
produce better formatted text documentation from markdown"):
> > I''m certainly not suggesting that we add a web browser to the
build
> > dependencies. If lynx isn''t installed, the current behavior
of copying
> > the markdown file is maintained.
> > 
> > If a packager wishes to produce prettier text documentation, they can
> > elect to add lynx to their build dependencies. Today doing this
> > post-build from build control files is a bit tricky, since we drop the
> > semantic information conveyed by the .markdown suffix by calling the
> > final file .txt.
> > 
> > 4.2 will be the first release with markdown documentation. I think
> > that making it well formatted, just as the previous .txt
> > documentation, will be a better experience for the user.
> 
> I tend to agree with this line of argument.  But I have orthogonal
> queries:
> 
> Firstly, why lynx and not w3m ?  Is lynx even maintained upstream any
> more ?  Secondly, shouldn''t we do the testing for the presence of
> markdown and the html formatter in configure ?
I suppose links or w3m work equally well. I tend to personally like
links'' formatting over w3m. The last lynx development release was in
2010, whereas the latest links release was June 26, 2012. The latest
w3m release was in January 2011.

Adding these checks in configure is a good idea. It''s a good method of
communicating the tools that can be or must be present for a
successful build of the auxiliary pieces of Xen.

Unfortunately I lost the build environment I set up that has the
correct version of autoconf, etc. so it might take a few days before I
can cook up a new version that adds autoconf checks.

Thanks for the suggestions.

Matt