Lustre devel - Nov 2012 - Seeking contributors for Lustre User Manual

In hopes of improving the quality, coverage, and currency of the Lustre
User Manual, I''m putting out a call to the Lustre community for
contributors to this important resource.

The user manual is especially important for new users to Lustre, but has
fallen into some disrepair now that there is no longer a dedicated
documentation writer for it.  As well, there are many sections of the
manual that have become outdated over the years (such as example output,
command descriptions, etc) that need to be refreshed.

The Lustre User Manual is also a component of Lustre that has a much
larger pool of potential contributors than the code itself.  If you want
to contribute to Lustre, but are not able to contribute with patches to
the code, then this is a great opportunity to help out.  If you benefit
from the open source development of Lustre, then contributing to the
manual is a chance to return something back to the community.  The Lustre
Manual is released under a Creative Commons license, so it is open to all
of us to improve.


While there is not currently a "todo" list for the areas of the manual
that need updating, looking through open LUDOC tickets is one option:

    
http://bugs.whamcloud.com/secure/QuickSearch.jspa?searchString=LUDOC%20open

There are a number of existing documentation tickets for features that are
under development for the Lustre 2.4 release (for which we expect to be
completed internally), but there are also some tickets from users pointing
out errors in the document that need to be fixed.

Another way to improve the manual is to simply fetch the manual and read
some section at random that you are either interested in, or have some
knowledge about, and see if any of the text is confusing, outdated, or
incorrect and needs to be updated.


The PDF and HTML versions of the current manual are available at:

    http://wiki.whamcloud.com/display/PUB/Documentation

The manual source is hosted in a Git/Gerrit repository in Docbook XML
format and can be downloaded at:

    git clone http://git.whamcloud.com/doc/manual lustre-manual

An account in Gerrit is required to download the manual and submit patches.

There are some wiki pages that describe the process and packages needed to
modify, build, and submit patches to the manual:

    
http://wiki.whamcloud.com/display/PUB/Making+changes+to+the+Lustre+Manual

Cheers, Andreas
--
Andreas Dilger
Lustre Software Architect
Intel Corporation

On Sat, Nov 10, 2012 at 12:09:09AM +0000, Dilger, Andreas
wrote:
> The manual source is hosted in a Git/Gerrit repository in Docbook XML
> format and can be downloaded at:
> 
>     git clone http://git.whamcloud.com/doc/manual lustre-manual
That doesn''t work for me:

  % git clone http://review.whamcloud.com/doc/manual  lustre-manual
  Cloning into lustre-manual...
  fatal: http://review.whamcloud.com/doc/manual/info/refs not found: did
    you run git update-server-info on the server?

I''ve only been able to check out the manual source directly from
gerrit:

  git clone http://review.whamcloud.com/p/doc/manual lustre-manual

or,

  git clone ssh://nedbass at review.whamcloud.com:29418/doc/manual lustre-manual


Ned

Would it be easier to move the manual back to a Wiki?  The low hassle factor of
wikis has always been a draw for contribution.  The openSFS site is up and
running with MediaWiki now (wiki.opensfs.org).


On Nov 9, 2012, at 4:09 PM, "Dilger, Andreas" <andreas.dilger at
intel.com> wrote:
> In hopes of improving the quality, coverage, and currency of the Lustre
> User Manual, I''m putting out a call to the Lustre community for
> contributors to this important resource.
> 
> The user manual is especially important for new users to Lustre, but has
> fallen into some disrepair now that there is no longer a dedicated
> documentation writer for it.  As well, there are many sections of the
> manual that have become outdated over the years (such as example output,
> command descriptions, etc) that need to be refreshed.
> 
> The Lustre User Manual is also a component of Lustre that has a much
> larger pool of potential contributors than the code itself.  If you want
> to contribute to Lustre, but are not able to contribute with patches to
> the code, then this is a great opportunity to help out.  If you benefit
> from the open source development of Lustre, then contributing to the
> manual is a chance to return something back to the community.  The Lustre
> Manual is released under a Creative Commons license, so it is open to all
> of us to improve.
> 
> 
> While there is not currently a "todo" list for the areas of the
manual
> that need updating, looking through open LUDOC tickets is one option:
> 
> 
> http://bugs.whamcloud.com/secure/QuickSearch.jspa?searchString=LUDOC%20open
> 
> There are a number of existing documentation tickets for features that are
> under development for the Lustre 2.4 release (for which we expect to be
> completed internally), but there are also some tickets from users pointing
> out errors in the document that need to be fixed.
> 
> Another way to improve the manual is to simply fetch the manual and read
> some section at random that you are either interested in, or have some
> knowledge about, and see if any of the text is confusing, outdated, or
> incorrect and needs to be updated.
> 
> 
> The PDF and HTML versions of the current manual are available at:
> 
>    http://wiki.whamcloud.com/display/PUB/Documentation
> 
> The manual source is hosted in a Git/Gerrit repository in Docbook XML
> format and can be downloaded at:
> 
>    git clone http://git.whamcloud.com/doc/manual lustre-manual
> 
> An account in Gerrit is required to download the manual and submit patches.
> 
> There are some wiki pages that describe the process and packages needed to
> modify, build, and submit patches to the manual:
> 
> 
> http://wiki.whamcloud.com/display/PUB/Making+changes+to+the+Lustre+Manual
> 
> Cheers, Andreas
> --
> Andreas Dilger
> Lustre Software Architect
> Intel Corporation
>

On Tue, Nov 13, 2012 at 11:48:35AM -0800, Nathan Rutman
wrote:
> Would it be easier to move the manual back to a Wiki?  The low hassle
> factor of wikis has always been a draw for contribution.  The openSFS
> site is up and running with MediaWiki now (wiki.opensfs.org).
Easier? Yes, probably. Better? I personally don''t think so.  Wikis are
great collaboration tools for informally sharing information, but I
don''t think the paradigm scales well for documents of this size and
complexity. And a wiki isn''t the right tool for producing a formal
professional-quality document, which is what I think the Lustre manual
should strive to be.

True, we would lower the bar for contributions, but for that we would
sacrifice the following features that I consider essential.

- Ability to export to multiple formats (pdf, html, epub) from one source
- Consistency of formatting and navigation elements
- A review process for proposed changes that assures a high standard of quality

However, there are some short articles that probably do belong in the
wiki that could be poached from the manual, i.e. installation and
configuration procedures, etc.

Ned

+1

Steven

On Tue, Nov 13, 2012 at 4:26 PM, Ned Bass <bass6 at llnl.gov>
wrote:
> On Tue, Nov 13, 2012 at 11:48:35AM -0800, Nathan Rutman wrote:
>> Would it be easier to move the manual back to a Wiki?  The low hassle
>> factor of wikis has always been a draw for contribution.  The openSFS
>> site is up and running with MediaWiki now (wiki.opensfs.org).
>
> Easier? Yes, probably. Better? I personally don''t think so.  Wikis
are
> great collaboration tools for informally sharing information, but I
> don''t think the paradigm scales well for documents of this size
and
> complexity. And a wiki isn''t the right tool for producing a formal
> professional-quality document, which is what I think the Lustre manual
> should strive to be.
>
> True, we would lower the bar for contributions, but for that we would
> sacrifice the following features that I consider essential.
>
> - Ability to export to multiple formats (pdf, html, epub) from one source
> - Consistency of formatting and navigation elements
> - A review process for proposed changes that assures a high standard of
quality
>
> However, there are some short articles that probably do belong in the
> wiki that could be poached from the manual, i.e. installation and
> configuration procedures, etc.
>
> Ned

On 2012-11-13, at 14:26, Ned Bass <bass6 at llnl.gov> wrote:
> On Tue, Nov 13, 2012 at 11:48:35AM -0800, Nathan Rutman wrote:
>> Would it be easier to move the manual back to a Wiki?  The low hassle
>> factor of wikis has always been a draw for contribution.  The openSFS
>> site is up and running with MediaWiki now (wiki.opensfs.org).
> 
> Easier? Yes, probably. Better? I personally don''t think so.  Wikis
are
> great collaboration tools for informally sharing information, but I
> don''t think the paradigm scales well for documents of this size
and
> complexity. And a wiki isn''t the right tool for producing a formal
> professional-quality document, which is what I think the Lustre manual
> should strive to be.
> 
> True, we would lower the bar for contributions, but for that we would
> sacrifice the following features that I consider essential.
> 
> - Ability to export to multiple formats (pdf, html, epub) from one source
> - Consistency of formatting and navigation elements
> - A review process for proposed changes that assures a high standard of
quality
Ned, these are the reasons we never moved the manual to a wiki in the first
place.  The original manual at Oracle was in FrameMaker format and only two
people could edit it. When we regenerated the manual again, it was converted
into Docbook XML.

Since Docbook is a relatively lightweight plain text markup language, and we
have good tools for validating the syntax along with WYSIWYG editing, so it is
hopefully a good tradeoff between ease of editing and the feature set needed to
produce the manual.

The other major drawback of a wiki is that it only works online, so it
can''t (easily?) be saved to read on a plane or in a secure facility.
> However, there are some short articles that probably do belong in the
> wiki that could be poached from the manual, i.e. installation and
> configuration procedures, etc.
Sure, and some of these exist (Richard''s "Getting Started With
Lustre" page). Definitely there could be a lot more effort out into this,
however.
> Ned

On Nov 13, 2012, at 3:26 PM, Ned Bass wrote:
> On Tue, Nov 13, 2012 at 11:48:35AM -0800, Nathan Rutman wrote:
>> Would it be easier to move the manual back to a Wiki?  The low hassle
>> factor of wikis has always been a draw for contribution.  The openSFS
>> site is up and running with MediaWiki now (wiki.opensfs.org).
> 
> Easier? Yes, probably. Better? I personally don''t think so.  Wikis
are
> great collaboration tools for informally sharing information, but I
> don''t think the paradigm scales well for documents of this size
and
> complexity. And a wiki isn''t the right tool for producing a formal
> professional-quality document, which is what I think the Lustre manual
> should strive to be.
> 
> True, we would lower the bar for contributions, but for that we would
> sacrifice the following features that I consider essential.
> 
> - Ability to export to multiple formats (pdf, html, epub) from one source
http://www.docbook.org ?
> - Consistency of formatting and navigation elements
> - A review process for proposed changes that assures a high standard of
quality
- ability to track changes between document versions to incrementally update
''higher level'' documents
> 
> However, there are some short articles that probably do belong in the
> wiki that could be poached from the manual, i.e. installation and
> configuration procedures, etc.
Right. 
And also the other way around: detailed articles on wiki written by developers
can be later ''harvested'' by professional writer into manual
chapter, referencing to wiki for details.
Lowering entry bar is vital to encourage developers to write or update
documentation.

DB:
In addition to wiki and "manual" it will be nice to have Document
Database, where conference reports, RFCs, RFP, HLD, DLD, ... can be committed,
updated and later searched.
Something like DocDB
	http://sourceforge.net/projects/docdb-v/
Document format can be any.
DocDB has been created to keep track of documentation in large collaboration -
BTeV experiment -  and then used by several others. DocDB has ability to manage
access rights to some documents.

I think we need all three - wiki, DocDB and manuals, they serve different
purpose.

KB:
Right now lustre support tips and hints are living on lustre-discuss list. It is
tedious to search emails (no tags,no links), and when the answer found, there is
no guarantee it is still relevant.
It can be useful to accumulate tips and best practices in Knowledge Base and
have mechanisms to update it, e.g. instead of answering directly to the list
create entry in KB and post the ref. to the list.

Alex.
> 
> Ned
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss

________________________________________
From: Alex Kulyavtsev [aik at fnal.gov]
Sent: Wednesday, November 14, 2012 11:47 AM
To: Ned Bass
Cc: Nathan Rutman; <wc-discuss at whamcloud.com>; <lustre-discuss at
lists.lustre.org>; lustre-devel at lists.lustre.org
Subject: Document Database Re: [Lustre-discuss] [wc-discuss] Seeking
contributors for Lustre User Manual

On Nov 13, 2012, at 3:26 PM, Ned Bass wrote:
> On Tue, Nov 13, 2012 at 11:48:35AM -0800, Nathan Rutman wrote:
>> Would it be easier to move the manual back to a Wiki?  The low hassle
>> factor of wikis has always been a draw for contribution.  The openSFS
>> site is up and running with MediaWiki now (wiki.opensfs.org).
>
> Easier? Yes, probably. Better? I personally don''t think so.  Wikis
are
> great collaboration tools for informally sharing information, but I
> don''t think the paradigm scales well for documents of this size
and
> complexity. And a wiki isn''t the right tool for producing a formal
> professional-quality document, which is what I think the Lustre manual
> should strive to be.
>
> True, we would lower the bar for contributions, but for that we would
> sacrifice the following features that I consider essential.
>
> - Ability to export to multiple formats (pdf, html, epub) from one source
http://www.docbook.org ?

Sphinx http://sphinx-doc.org/?

All the multiple format benefits, with none of the XML. And I agree
wholeheartedly that Wikis are not the best home for complex documentation.

--Rick
> - Consistency of formatting and navigation elements
> - A review process for proposed changes that assures a high standard of
quality
- ability to track changes between document versions to incrementally update
''higher level'' documents
>
> However, there are some short articles that probably do belong in the
> wiki that could be poached from the manual, i.e. installation and
> configuration procedures, etc.
Right.
And also the other way around: detailed articles on wiki written by developers
can be later ''harvested'' by professional writer into manual
chapter, referencing to wiki for details.
Lowering entry bar is vital to encourage developers to write or update
documentation.

DB:
In addition to wiki and "manual" it will be nice to have Document
Database, where conference reports, RFCs, RFP, HLD, DLD, ... can be committed,
updated and later searched.
Something like DocDB
        http://sourceforge.net/projects/docdb-v/
Document format can be any.
DocDB has been created to keep track of documentation in large collaboration -
BTeV experiment -  and then used by several others. DocDB has ability to manage
access rights to some documents.

I think we need all three - wiki, DocDB and manuals, they serve different
purpose.

KB:
Right now lustre support tips and hints are living on lustre-discuss list. It is
tedious to search emails (no tags,no links), and when the answer found, there is
no guarantee it is still relevant.
It can be useful to accumulate tips and best practices in Knowledge Base and
have mechanisms to update it, e.g. instead of answering directly to the list
create entry in KB and post the ref. to the list.

Alex.
>
> Ned
> _______________________________________________
> Lustre-discuss mailing list
> Lustre-discuss at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-discuss

On 13/11/12 21:26, Ned Bass wrote:
> - A review process for proposed changes that assures a high standard of
quality
That''s important. I''ve spotted cases where it''s clear
there''s an issue,
but am not sure what the correct answer is.

A quick way of reporting issues in the manual would be useful too - 
whilst I did go to the effort of registering in order to submit typos, 
that''s rather heavyweight. Just being able to send an e-mail would be 
easier - but it''s true that someone needs to sort them.

Chris

On 11/14/12 4:51 PM, "Christopher J. Walker" <C.J.Walker at
qmul.ac.uk> wrote:
>On 13/11/12 21:26, Ned Bass wrote:
>> - A review process for proposed changes that assures a high standard of
>>quality
>
>That''s important. I''ve spotted cases where it''s
clear there''s an issue,
>but am not sure what the correct answer is.
>
>A quick way of reporting issues in the manual would be useful too -
>whilst I did go to the effort of registering in order to submit typos,
>that''s rather heavyweight. Just being able to send an e-mail would
be
>easier - but it''s true that someone needs to sort them.
I think that while email may be easier for the initial report to be
submitted, it places more burden on the recipient to track the issue (or
risk losing/forgetting the email over time), and the issue is only visible
to the recipient (and cannot easily be seen by others or transferred to
someone else for resolution).  If we really want to scale the process of
updating the manual, then pushing the work to turn the emails into
tickets/fixes is not moving in the right direction.

I would prefer to see a fix immediately rather than someone filing a
ticket to describe the fix, since the documentation fix should be
self-describing.  However, if there is a problem that isn''t immediately
resolved then a Jira ticket should be submitted in order to track the
defect and allow assigning the work to someone.  The effort of setting up
Jira and/or Gerrit accounts is a one-time thing, and is shared with normal
bug reporting, so hopefully not a huge burden (no worse than having to get
an account for the wiki, which would also be a requirement to submit, to
avoid wiki spam).

Hopefully this will not be a huge obstacle for contributions in the end.

Cheers, Andreas

On Thu, Nov 15, 2012 at 12:12:40AM +0000, Dilger, Andreas
wrote:
>
> I would prefer to see a fix immediately rather than someone filing a
> ticket to describe the fix, since the documentation fix should be
> self-describing.  However, if there is a problem that isn''t
immediately
> resolved then a Jira ticket should be submitted in order to track the
> defect and allow assigning the work to someone.
LUDOC-11 seems to be a catch-all issue for submitting fixes to minor
problems like typos.  However it sounds like you''re saying we can
bypass
Jira altogether for such patches.  That would be nice; linking to a
ticket with no useful content doesn''t serve any purpose that I can see.

The "Making changes to the Lustre Manual source" article currently
instructs the reader to "file an LUDOC bug for change tracking in
Jira"
as the first step.  To avoid discouraging submission of minor fixes,
perhaps a more lightweight process for that case should be covered
first.  In particular, say either that minor fixes should reference
LUDOC-11 in the summary, or just omit the Jira reference altogether,
whichever is appropriate.

Ned

On Wed, 2012-11-14 at 17:26 -0800, Ned Bass wrote:
> On Thu, Nov 15, 2012 at 12:12:40AM +0000, Dilger, Andreas wrote:
> >
> > I would prefer to see a fix immediately rather than someone filing a
> > ticket to describe the fix, since the documentation fix should be
> > self-describing.  However, if there is a problem that isn''t
immediately
> > resolved then a Jira ticket should be submitted in order to track the
> > defect and allow assigning the work to someone.
> 
> LUDOC-11 seems to be a catch-all issue for submitting fixes to minor
> problems like typos.  However it sounds like you''re saying we can
bypass
> Jira altogether for such patches.  That would be nice; linking to a
> ticket with no useful content doesn''t serve any purpose that I can
see.
> 
> The "Making changes to the Lustre Manual source" article
currently
> instructs the reader to "file an LUDOC bug for change tracking in
Jira"
> as the first step.  To avoid discouraging submission of minor fixes,
> perhaps a more lightweight process for that case should be covered
> first.  In particular, say either that minor fixes should reference
> LUDOC-11 in the summary, or just omit the Jira reference altogether,
> whichever is appropriate.
> 
I''ve introduced a procedure for minor changes on the page:
http://wiki.whamcloud.com/display/PUB/Making+changes+to+the+Lustre
+Manual+source

This includes a cut-and-paste commit message to get people started.

My reasons including an arguably redundant issue (LUDOC-11) are: 
- it creates consistent submission policy across the different projects.
- it gives an opportunity to easily measure minor changes

cheers,
Richard
-- 
Richard.Henwood at intel.com
High Performance Data Division