CentOS docs - Jun 2015 - Finished a prototype of GSoC project: Implement a new doc toolchain

Hi,

Kunaal and I have completed a prototype of the first part of our GSoC project
(implement a new doc toolchain). Our goal is to implement a doc toolchain for
short how-to docs and lower the barrier for new comers to contribute.

This is the workflow of the prototype:
1. Contributor creates a new pull request on GitHub
2. A corresponding issue is automatically created on Pagure
3. Staff and contributor discuss on Pagure
4. When the pull request can be accepted, it is marked as ?fixed?, and the pull
request on github is automatically merged
5. New change is synced to Pagure (doc website will update according to Pagure
repo, to be implemented later)

We have made a prototype performing the steps above, and is now running ;) Next
we plan to integrate a CI so the doc can be built in real time and ready for
staff to review. Could you please give us some suggestions about the workflow
above and about the toolchain? The original GSoC idea can be found here:
http://wiki.centos.org/GSoC/2015/Ideas#docs-toolchain
<http://wiki.centos.org/GSoC/2015/Ideas#docs-toolchain>

We use Pagure as a part of the toolchain. Pagure is an excellent git-centered
forge created by Pierre-Yves Chibon <pingou at pingoured.fr <mailto:pingou
at pingoured.fr>>. Currently Pagure is in its early stage and the APIs and
web hooks are changing and improving. We are working closely with Pierre to
better integrate Pagure into the toolchain. In the future we plan to sync
complete pull request so that discussion can happen either on github or Paugre,
and contributor doesn?t need to leave github in the whole process, which lowers
the barrier of contributing a lot.

To test it, you may create a PR on github. A corresponding issue will be created
automatically on Pagure, so you may go to Pagure.io to check it. If you want to
merge the PR, change the status of issue from Open to Fixed on Pagure (need
admin access to the repo, if you want to test this part, please email me or
Kunaal <kunaalus at gmail.com <mailto:kunaalus at gmail.com>> so we
can add you as admin). Once the issue is marked as Fixed, the PR on github will
be merged.

GitHub repo link: https://github.com/yangl1996/doc-test
<https://github.com/yangl1996/doc-test>
Pagure repo link: https://pagure.io/docs-test
<https://pagure.io/docs-test>

The syncing tool is a prototype and is a basic part of the toolchain. Please
tell us what you think about the workflow, the syncing tool and the idea. Thank
you in advance :)

Also, we are not very sure whether running a Pagure instance will add much
workload to sys admin team. What?s your idea?

Thanks!


Regards,
Lei

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.centos.org/pipermail/centos-docs/attachments/20150620/89405a59/attachment-0002.html>

Hi Lei,
> On Jun 20, 2015, at 11:21 AM, Lei Yang <yltt1234512 at gmail.com>
wrote:
I am sorry it has taken me so long to test this workflow.
> Hi,
> 
> Kunaal and I have completed a prototype of the first part of our GSoC
project (implement a new doc toolchain). Our goal is to implement a doc
toolchain for short how-to docs and lower the barrier for new comers to
contribute.
I am not trying to be dense, but I want to make sure I understand the goals
here:
> This is the workflow of the prototype:
> 1. Contributor creates a new pull request on GitHub
A fairly well understand platform with lots of potential users - cool.  Github
also has some social cachet.
> 2. A corresponding issue is automatically created on Pagure
> 3. Staff and contributor discuss on Pagure
This part is a bit odd to me, today.  I believe your goal is to make sure the
discussion happens in a CentOS controlled space in case github ever becomes an
issue.  However, it seems odd that we don't just mirror the whole PR over
(see below for where I read the paragraph where you said this was coming ...). 
Right now it is weird to have to visit one site to see the change and another to
discuss it.  Furthermore I feel like we are creating an educational challenge
(which could be resolved with some great docs and a bot that posted in the PR on
github) to get people to understand what is going on.
> 4. When the pull request can be accepted, it is marked as ?fixed?, and the
pull request on github is automatically merged
OK
> 5. New change is synced to Pagure (doc website will update according to
Pagure repo, to be implemented later)
While this makes sense, it seems like we are barely using github and I am not
sure it is helping us in this equation completely.  Can you help me understand
what I am missing?

Openstack does something sort of similar, but backwards from our workflow.

They have both a private git (with gerrit) and a github repo.  If you open a PR
on github it is closed by a bot with instructions to submit the PR via gerrit to
their private repo.  When the PR is resolved and merged in the private repo, the
entire commit is pushed to github awarding the social "status" that
comes from a commit. I am not suggesting we use gerrit.
> We have made a prototype performing the steps above, and is now running ;)
Next we plan to integrate a CI so the doc can be built in real time and ready
for staff to review. Could you please give us some suggestions about the
workflow above and about the toolchain? The original GSoC idea can be found
here: http://wiki.centos.org/GSoC/2015/Ideas#docs-toolchain
<http://wiki.centos.org/GSoC/2015/Ideas#docs-toolchain>
I like the CI.  I'd love to have us consider including something like
emender in a future release that could do CI on the PR and post a response with
any automated integration failures.
> We use Pagure as a part of the toolchain. Pagure is an excellent
git-centered forge created by Pierre-Yves Chibon <pingou at pingoured.fr
<mailto:pingou at pingoured.fr>>. Currently Pagure is in its early
stage and the APIs and web hooks are changing and improving. We are working
closely with Pierre to better integrate Pagure into the toolchain. In the future
we plan to sync complete pull request so that discussion can happen either on
github or Paugre, and contributor doesn?t need to leave github in the whole
process, which lowers the barrier of contributing a lot.
Do you have a timeline for the full PR mirroring?

I agree that using github will definitely lower the barrier to contribution. My
experience with sync problems has shown that two-way is problematic.  Do you
have a plan for that?

Can we achieve our goals by having a one-way sync from github to pagure?  This
would seem to simplify the problem and provide us the protection - unless I am
missing something.
> To test it, you may create a PR on github. A corresponding issue will be
created automatically on Pagure, so you may go to Pagure.io to check it. If you
want to merge the PR, change the status of issue from Open to Fixed on Pagure
(need admin access to the repo, if you want to test this part, please email me
or Kunaal <kunaalus at gmail.com <mailto:kunaalus at gmail.com>> so
we can add you as admin). Once the issue is marked as Fixed, the PR on github
will be merged.
Feel free to merge or reject my PR as you see fit :)
> Also, we are not very sure whether running a Pagure instance will add much
workload to sys admin team. What?s your idea?
I can't speak to the overhead, but I can say that I believe having Pagure
running is a good idea.

regards,

bex
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.centos.org/pipermail/centos-docs/attachments/20150708/8d369448/attachment.html>

On Thu, Jul 9, 2015 at 1:19 AM, Brian (bex) Exelbierd <bex at pobox.com>
wrote:
> Hi Lei,
>
>
Hi, I am Kunaal the other GSoC student working on this toolchain.
> On Jun 20, 2015, at 11:21 AM, Lei Yang <yltt1234512 at gmail.com>
wrote:
>
>
> I am sorry it has taken me so long to test this workflow.
>
Thanks a lot! We need testers and input from community.
>
> Hi,
>
> Kunaal and I have completed a prototype of the first part of our GSoC
> project (implement a new doc toolchain). Our goal is to implement a doc
> toolchain for short how-to docs and lower the barrier for new comers to
> contribute.
>
>
> I am not trying to be dense, but I want to make sure I understand the
> goals here:
>
> This is the workflow of the prototype:
> 1. Contributor creates a new pull request on GitHub
>
>
> A fairly well understand platform with lots of potential users - cool.
> Github also has some social cachet.
>
That's the aim. Exploit this social cachet.
>
> 2. A corresponding issue is automatically created on Pagure
> 3. Staff and contributor discuss on Pagure
>
>
> This part is a bit odd to me, today.  I believe your goal is to make sure
> the discussion happens in a CentOS controlled space in case github ever
> becomes an issue.  However, it seems odd that we don't just mirror the
> whole PR over (see below for where I read the paragraph where you said this
> was coming ...).  Right now it is weird to have to visit one site to see
> the change and another to discuss it.  Furthermore I feel like we are
> creating an educational challenge (which could be resolved with some great
> docs and a bot that posted in the PR on github) to get people to understand
> what is going on.
>
I agree! That time pagure was not having such webhooks and API. We are
working with pingou(pagure creator). Fortunately it has now.

Yes that's what bot's aim was, to point out to the link.
>
> 4. When the pull request can be accepted, it is marked as ?fixed?, and the
> pull request on github is automatically merged
>
>
> OK
>
> 5. New change is synced to Pagure (doc website will update according to
> Pagure repo, to be implemented later)
>
>
> While this makes sense, it seems like we are barely using github and I am
> not sure it is helping us in this equation completely.  Can you help me
> understand what I am missing?
>
> Openstack does something sort of similar, but backwards from our workflow.
>
> They have both a private git (with gerrit) and a github repo.  If you open
> a PR on github it is closed by a bot with instructions to submit the PR via
> gerrit to their private repo.  When the PR is resolved and merged in the
> private repo, the entire commit is pushed to github awarding the social
> "status" that comes from a commit. I am not suggesting we use
gerrit.
>
I agree. Main reason is because of two way sync of pagure and github. Once
that done, user even doesn't need to be aware that pagure exists. I agree
right now we are doing something like openstack which you described, but we
dont want that.
>
> We have made a prototype performing the steps above, and is now running ;)
> Next we plan to integrate a CI so the doc can be built in real time and
> ready for staff to review. Could you please give us some suggestions about
> the workflow above and about the toolchain? The original GSoC idea can be
> found here: http://wiki.centos.org/GSoC/2015/Ideas#docs-toolchain
>
>
> I like the CI.  I'd love to have us consider including something like
> emender in a future release that could do CI on the PR and post a response
> with any automated integration failures.
>
Thanks! We don't have much knowledge about that, once we get there, maybe
you can help us. I made a sample CI for listening to github repo changes,
and automatically generating new site and deploying it.
https://github.com/kunaaljain/test-centos-docs
>
> We use Pagure as a part of the toolchain. Pagure is an excellent
> git-centered forge created by Pierre-Yves Chibon <pingou at
pingoured.fr>.
> Currently Pagure is in its early stage and the APIs and web hooks are
> changing and improving. We are working closely with Pierre to better
> integrate Pagure into the toolchain. In the future we plan to sync complete
> pull request so that discussion can happen either on github or Paugre, and
> contributor doesn?t need to leave github in the whole process, which lowers
> the barrier of contributing a lot.
>
>
> Do you have a timeline for the full PR mirroring?
>
> I agree that using github will definitely lower the barrier to
> contribution. My experience with sync problems has shown that two-way is
> problematic.  Do you have a plan for that?
>
> Can we achieve our goals by having a one-way sync from github to pagure?
> This would seem to simplify the problem and provide us the protection -
> unless I am missing something.
>
That is our priority. There was a couple of roadblocks, which I think are
now cleared. So we will be again working on full sync, and by this weekend
we hope we can achieve this.

Yes two way sync is very problematic. A mess of GitHub and pagure webhooks
and API, listening to each other.

Our aim is two way sync, if we fail, we can fall back to one way sync from
github to pagure. This will still achieve our goal, but then maybe not
every CentOS docs maintainer be open to idea of using Github. We had some
sort of these ideas in a discussion.

>
> To test it, you may create a PR on github. A corresponding issue will be
> created automatically on Pagure, so you may go to Pagure.io to check it. If
> you want to merge the PR, change the status of issue from Open to Fixed on
> Pagure (need admin access to the repo, if you want to test this part,
> please email me or Kunaal <kunaalus at gmail.com> so we can add you
as
> admin). Once the issue is marked as Fixed, the PR on github will be merged.
>
>
> Feel free to merge or reject my PR as you see fit :)
>
> Also, we are not very sure whether running a Pagure instance will add much
> workload to sys admin team. What?s your idea?
>
>
> I can't speak to the overhead, but I can say that I believe having
Pagure
> running is a good idea.
>
>
The idea of having another platform was met with some resistance due to
overhead and hence the concern.

> regards,
>
> bex
>
Please let us know what you think of whole workflow? We have many exciting
things ahead, like API for upstream projects to take the docs. But first
aim is get basic toolchain running
>
> _______________________________________________
> CentOS-docs mailing list
> CentOS-docs at centos.org
> http://lists.centos.org/mailman/listinfo/centos-docs
>
>
--
Regards,
Kunaal Jain
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.centos.org/pipermail/centos-docs/attachments/20150709/05ef1cbf/attachment-0002.html>