Nouveau - Aug 2021 - Proposal for allowing more Nouveau contributors to merge patches

Hey everybody,

so, here is a proposal of what we could change in order to allow
patches to land faster, more reliably and to increase the overall bus
factor in terms of nouveau kernel maintenance.

But let's start with the current situation:

At the moment contributors have to send patches to the nouveau mailing
list in order to submit them for inclusion into Bens nouveau tree.
Then Ben has to look at them and either respond with a comment on what
needs to change or merge it. At some point Ben submits all new changes
to nouveau for inclusion into the drm-next tree.

Practically there are a few problems here:
1. Not all patch submissions get a response
2. Patch submitters usually don't get to know if their patches are
accepted unless they appear in drm-next or linus' tree.
3. A lot of trivial patches never make it into the tree.
4. Depending on how overloaded Ben is, even bigger patch series adding
new features never get any responses at all.
5. Patch series might get stale for any other reason and there is no
good tool to track open patch submissions (no, patchwork isn't good
and it sucks and not even in the mood to explain this to people
thinking otherwise as this should just be obvious)

This usually ends up discouraging new contributors from making more
contributions to nouveau as they see that some or all of their patches
never get any reaction and it's even annoying to current contributors
if they see their patches being stuck as well.

And here I want to stress that none of this is Ben's fault and has his
own things to work on and none of this should just depend on one
person finding enough time. So the solution shouldn't be a simple
"let's find a different tool and everything should be perfect" but
the
solution should be "how can we change the process so it's less time
consuming for Ben to accept patches". And while working on this, I'd
also want to tackle the problem that contributing to nouveau shouldn't
be frustrating for new contributors and the process should be more
lively overall.

So for this I want to propose a new workflow, which would also spread
some responsibilities around active members of the nouveau community:

1. We should have a public nouveau tree (which could be
https://gitlab.freedesktop.org/drm/nouveau or something else) with a
nouveau-next branch collecting all patches for the next kernel major
kernel release.
At the moment the "official" nouveau tree is kind of
https://github.com/skeggsb/nouveau but there is
https://github.com/skeggsb/linux as well. The main problem is, it's a
repository tight to a person. We already have
https://gitlab.freedesktop.org/drm/nouveau as a bug tracker, but it
also contains a full linux git tree which is updated automatically
through a CI job.

2. A chosen group of people should have push rights to this repository
in order to accept patches sent in via emails to the nouveau Mailing
List or fancy UIs (like gitlabs MRs) or other ways.
Trusted contributors should be allowed to review and accept submitted
patches in order to reduce the workload on Ben. Those patches will be
collected on the nouveau-next branch. Patches from a mailing list
could e.g. be submitted through a MR on gitlab or just pushed to the
branch directly.

3. We should ensure through CI that submitted patches are at least
passing basic quality control (checkpatch, compile testing, sparse,
etc...)
I already worked on this and one example can be seen here:
https://gitlab.freedesktop.org/drm/nouveau/-/merge_requests/3
There are more things we could add to it, like checking if sparse is
happy or add additional checks. Adding tests against hw is something
we want to target for the future as well.

4. patches from nouveau-next should be included into a "higher" git
tree (drm-next or drm-misc-next) by authorized people after getting a
final review by Ben or somebody else.
The idea is to post all collected patches for final review to the
mailing list as Ben wanted to have a final possibility to nack it in
case something stands out. We could then depend on Ben including those
patches, but we could also use commit rights to drm-next or
drm-misc-next from other people as well. I have commit access to the
drm-misc repository, and I would assume that Lyude would be able to
get it as well or already has it.

5. Urgent fixes should land via drm-fixes or drm-misc-fixes.
We kind of already do this already though. We could spin up a
nouveau-fixes branch in the future, but the amount of such fixes is so
low it's not really worth the effort at the moment. And we have access
to drm-misc.

Once we decide to start and agree to some process, we should try it
out with trivial patches, we already get enough of. Like Typo fixes,
patches adding docs or removing dead code and can expand this to more
"serious" work once we agree that it actually works and does have
benefits to nouveau overall.

Any thoughts on this?

On Fri, Aug 06, 2021 at 06:53:06PM +0200, Karol Herbst
wrote:
> Hey everybody,
> 
> so, here is a proposal of what we could change in order to allow
> patches to land faster, more reliably and to increase the overall bus
> factor in terms of nouveau kernel maintenance.
Yay!
> But let's start with the current situation:
> 
> At the moment contributors have to send patches to the nouveau mailing
> list in order to submit them for inclusion into Bens nouveau tree.
> Then Ben has to look at them and either respond with a comment on what
> needs to change or merge it. At some point Ben submits all new changes
> to nouveau for inclusion into the drm-next tree.
> 
> Practically there are a few problems here:
> 1. Not all patch submissions get a response
> 2. Patch submitters usually don't get to know if their patches are
> accepted unless they appear in drm-next or linus' tree.
> 3. A lot of trivial patches never make it into the tree.
> 4. Depending on how overloaded Ben is, even bigger patch series adding
> new features never get any responses at all.
> 5. Patch series might get stale for any other reason and there is no
> good tool to track open patch submissions (no, patchwork isn't good
> and it sucks and not even in the mood to explain this to people
> thinking otherwise as this should just be obvious)
> 
> This usually ends up discouraging new contributors from making more
> contributions to nouveau as they see that some or all of their patches
> never get any reaction and it's even annoying to current contributors
> if they see their patches being stuck as well.
> 
> And here I want to stress that none of this is Ben's fault and has his
> own things to work on and none of this should just depend on one
> person finding enough time. So the solution shouldn't be a simple
> "let's find a different tool and everything should be
perfect" but the
> solution should be "how can we change the process so it's less
time
> consuming for Ben to accept patches". And while working on this,
I'd
> also want to tackle the problem that contributing to nouveau shouldn't
> be frustrating for new contributors and the process should be more
> lively overall.
> 
> So for this I want to propose a new workflow, which would also spread
> some responsibilities around active members of the nouveau community:
> 
> 1. We should have a public nouveau tree (which could be
> https://gitlab.freedesktop.org/drm/nouveau or something else) with a
> nouveau-next branch collecting all patches for the next kernel major
> kernel release.
> At the moment the "official" nouveau tree is kind of
> https://github.com/skeggsb/nouveau but there is
> https://github.com/skeggsb/linux as well. The main problem is, it's a
> repository tight to a person. We already have
> https://gitlab.freedesktop.org/drm/nouveau as a bug tracker, but it
> also contains a full linux git tree which is updated automatically
> through a CI job.
> 
> 2. A chosen group of people should have push rights to this repository
> in order to accept patches sent in via emails to the nouveau Mailing
> List or fancy UIs (like gitlabs MRs) or other ways.
> Trusted contributors should be allowed to review and accept submitted
> patches in order to reduce the workload on Ben. Those patches will be
> collected on the nouveau-next branch. Patches from a mailing list
> could e.g. be submitted through a MR on gitlab or just pushed to the
> branch directly.
> 
> 3. We should ensure through CI that submitted patches are at least
> passing basic quality control (checkpatch, compile testing, sparse,
> etc...)
> I already worked on this and one example can be seen here:
> https://gitlab.freedesktop.org/drm/nouveau/-/merge_requests/3
> There are more things we could add to it, like checking if sparse is
> happy or add additional checks. Adding tests against hw is something
> we want to target for the future as well.
> 
> 4. patches from nouveau-next should be included into a "higher"
git
> tree (drm-next or drm-misc-next) by authorized people after getting a
> final review by Ben or somebody else.
> The idea is to post all collected patches for final review to the
> mailing list as Ben wanted to have a final possibility to nack it in
> case something stands out. We could then depend on Ben including those
> patches, but we could also use commit rights to drm-next or
> drm-misc-next from other people as well. I have commit access to the
> drm-misc repository, and I would assume that Lyude would be able to
> get it as well or already has it.
Hm I'm not sure that double-indirect makes a lot of sense. There's imo
two
reasonable models here:

- commit rights on drm/nouveau, Ben sends out the pull requests to
  drm-next for those, which gives him a good point to check over and do
  any last touches

- directly push to drm-misc-next, defacto having a group commit right
  thing going on there directly

Doing first commit rights and then resending the entire pile is way too
much beaurocracy and wont solve much in my experience.

This does mean that there's a lot of trust required for committers, since
if the "drop/revert a patch" part becomes a regularity none of this
works
out. Otoh it also doesn't work out if committers aren't trusted, because
then you're stuck in the "not really my problem" space.

This is both from plenty of experience from other kernel teams trying to
do something like this. If you don't have upfront trust, then commit
rights/group maintainership model will simply not work, or at least be
very painful. And yes this means occasionally things go wrong and need to
be fixed up, and more importantly, a discussion about tooling, testing and
review standards is likely required.

For drm-misc/intel we handle those by asking for improvements to the dim
tooling/docs anytime something goes wrong. This way hopefully the entire
team learns.
> 5. Urgent fixes should land via drm-fixes or drm-misc-fixes.
> We kind of already do this already though. We could spin up a
> nouveau-fixes branch in the future, but the amount of such fixes is so
> low it's not really worth the effort at the moment. And we have access
> to drm-misc.
drm.git trees pretty much only take pull requests because Dave&me are
lazy. We do push the occasionally hotfix if someone managed to set the
tree on fire, but that's it (usually because we still suck at CI).

Imo for fixes just make drm-misc the official place.
> Once we decide to start and agree to some process, we should try it
> out with trivial patches, we already get enough of. Like Typo fixes,
> patches adding docs or removing dead code and can expand this to more
> "serious" work once we agree that it actually works and does have
> benefits to nouveau overall.
> 
> Any thoughts on this?
Imo start out with fixes in drm-misc-fixes, and leave feature work and
anything substantial to the current skeggsb/nouveau process. Then reasses
how that's going and what needs to be improved (like get the CI infra up
and running meanwhile, maybe move the repo to drm/nouveau if you feel
like, that kind of stuff).

At least a get a bit a feeling for big stuff nouveau aint ready yet for
group maintainership, judging from what you're proposing here. This also
allows you to experiment around more with gitlab CI and maybe MR, perhaps
nouveau will got directly to that for group maintainership (with Ben and
Marge bot as sole committers)?

Plus I think offloading and speeding up the simple fixes should help a lot
already in making nouveau more alive and perhaps attracting new
contributors.

Cheers, Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch