Nouveau - Mar 2009 - DRM changes

Hi,

As part of aiming at upstreaming our code, I suppose we have to
discuss the DRM situation a little. In order to aim for merging, I
think we'd better be working on a linux kernel tree layout. And
considering we're technically the only ones still working in drm.git,
it doesn't really make sense to keep doing that for the sake of
sharing code with other drivers.

As we discussed on irc, there would be multiple changes related to this:
- we move to a linux kernel-like tree, that should make it easier when
upstreaming the code.
- this new tree is hosted in freedekstop.org in the nouveau/ git so we
don't need additional accounts for everyone all around and people can
keep pushing things (even better, all nouveau people can push to the
drm, which used to require mesa rights before)
- we keep a(some) branch(es) in the tree for backwards compat with
older kernels. Either in the form of separate kernel versions
including nouveau, or in the form of an out-of-tree-compilable
drm/nouveau module.

So, does that plan sound sane? Do you have better plans?

Stephane

Questions:

A tree based on what? (linus, drm-next, etc)

What happens to shared-core?

Opinions (assuming this happens):

I'd do a branch for each kernel version (or less if needed), and let
people who care cherry-pick fixes.

Maarten.

Stephane Marchesin wrote:
> Hi,
> 
> As part of aiming at upstreaming our code, I suppose we have to
> discuss the DRM situation a little. In order to aim for merging, I
> think we'd better be working on a linux kernel tree layout. And
> considering we're technically the only ones still working in drm.git,
> it doesn't really make sense to keep doing that for the sake of
> sharing code with other drivers.
> 
> As we discussed on irc, there would be multiple changes related to this:
> - we move to a linux kernel-like tree, that should make it easier when
> upstreaming the code.
> - this new tree is hosted in freedekstop.org in the nouveau/ git so we
> don't need additional accounts for everyone all around and people can
> keep pushing things (even better, all nouveau people can push to the
> drm, which used to require mesa rights before)
> - we keep a(some) branch(es) in the tree for backwards compat with
> older kernels. Either in the form of separate kernel versions
> including nouveau, or in the form of an out-of-tree-compilable
> drm/nouveau module.
> 
> So, does that plan sound sane? Do you have better plans?
> 
> Stephane
(apologies for the crappy mailman archive->email forgery)

I have reservations about a full Linux kernel tree, if that's the proposal.

Assuming it is a relatively important goal that people should test 
Nouveau, then it makes sense to make it as easy as possible for them to 
do so.  Certainly as long as Nouveau has not merged to the Linux kernel, 
our repository is the only (non-packaged) source for those wishing to 
test.  Post merge, some repository other than Linus' kernel is still 
going to be the place to point people at for new development and fixes 
which are not in a kernel.org stable release.

I'd suggest that a) requiring 0.5 - 1 hour to git clone the half-gig or 
so of our new kernel-based tree (longer if the fd.o machinery is having 
a bad day), and b) needing people to then checkout some random branch in 
order for it to work with their kernel ( c) would be rebuild their 
kernel, if the out-of-tree compilable thing doesn't happen ), are all 
barriers to testers that are not currently there.  Although my usage may 
be atypical (or git outdated), the repository size issue also bites 
developers; cold-cache git diff and git status on the linux kernel takes 
a noticeable time, and gitk, even if regularly invoked, eats disk and 
processor for ages on each execution.

If the goal of these changes is to have a more Linux kernel-like layout, 
could we not just change drm/linux-core, or make a new repo, so that it 
mirrored everything necessary under drivers/drm?  I believe airlied 
suggested this ages ago, and it would make moving code from Linux to 
Nouveau or the other way round simpler.  Admittedly this doesn't work if 
for some reason your development demands changes elsewhere in the Linux 
tree, but in that case you may well have already failed the back-compat 
aspect.

As for branching and back-compat, I'd have thought it more useful to 
keep compat on the main branch (again, helping testers), and split 
specific kernel-merge-aimed branches off from that, as automating ifdef 
removal on a branch ought to be easier than manually adding compat to a 
bunch of branches, but I guess in some way the solutions here are all 
equivalent, and it's just which branch you bless as being called master.

Stuart

m?n 2009-03-09 klockan 22:49 +0100 skrev Stephane
Marchesin:
> Hi,
> 
> So, does that plan sound sane? Do you have better plans?
> 
> Stephane
I must say that for me it does not give that big problems, since I
already are using my own hem-brewed kernel. But I think it will be
harder getting testers if they need to not just compile som modules for
their kernel, but also try to configure the beast on their own.

Just my 2-cents.

On Mon, 9 Mar 2009 22:49:32 +0100
Stephane Marchesin <marchesin at icps.u-strasbg.fr> wrote:
> As we discussed on irc, there would be multiple changes related to this:
> - we move to a linux kernel-like tree, that should make it easier when
> upstreaming the code.
> - this new tree is hosted in freedekstop.org in the nouveau/ git so we
> don't need additional accounts for everyone all around and people can
> keep pushing things (even better, all nouveau people can push to the
> drm, which used to require mesa rights before)
> - we keep a(some) branch(es) in the tree for backwards compat with
> older kernels. Either in the form of separate kernel versions
> including nouveau, or in the form of an out-of-tree-compilable
> drm/nouveau module.
Hi,

I took airlied's drm-rawhide kernel tree, applied darktama's patch
that adds Nouveau, wrote/copied my own compat-patches on top of that,
and compiled the DRM modules (drm.ko and nouveau.ko) against my
currently running kernel 2.6.24.

It works, with the caveat that I needed a tiny patch to 2.6.24 in
addition to reconfiguring it. The details will follow. I've been
running Nouveau for two hours now without any hickups.

I'm CC'ing also dri-devel@ just in case people working on other
drivers might be interested in offering a "drm.git-like" version
of their drivers for people with older, custom or otherwise
not-this-driver's-development-tree kernels. And because my
compat patches probably suck.

Feel free to drop the dri-devel@ CC when you discuss Nouveau-only
matters.


This raises some questions for Nouveau:
- like marcheu proposed, shall Nouveau DRM move into a kernel tree?
- do we abandon drm.git kernel directories completely?
- is the out-of-tree DRM build sketched below feasible for users?
- how do we maintain and distribute the compat patches and DRM code?

I'd expect the compat patches would be maintained by whoever happens
to run older kernels. Developers do not consider compat patching.


Building DRM out-of-tree for 2.6.24:

Prepare the "upstream" DRM and Nouveau code:

- Get airlied's linux kernel tree, the drm-rawhide branch:
http://git.kernel.org/?p=linux/kernel/git/airlied/drm-2.6.git;a=shortlog;h=drm-rawhide
(the version I used is probably a week old now)

- Get darktama's nouveau patch and apply it on drm-rawhide:
http://cvs.fedoraproject.org/viewvc/rpms/kernel/devel/drm-nouveau.patch?revision=1.25&view=markup
(this is probably outdated by now)

- apply the included nouveau-drm-compat-24.patch
This patch is combined from the individual patches listed in its
description, diffstat is included.

- run 'make clean' unless your tree is already clean.


Prepare your 2.6.24 kernel:

- apply the included kernel-compat-24.patch to export shmem_file_setup()

- configure the following as modules:
	CONFIG_DRM
	CONFIG_FB
	CONFIG_BACKLIGHT_CLASS_DEVICE
	CONFIG_FB_SAVAGE
The savage driver is abused to get FB_CFB_FILLRECT, FB_CFB_COPYAREA and
FB_CFB_IMAGEBLIT enabled, since they do not have menuconfig entries.
If you actually have a Savage card in your computer, maybe you should
pick another fb driver.

- build and install as usual, remove the drm.ko generated.


Building the DRM and Nouveau modules as external modules
using the 2.6.24 kernel tree:

- save the included mkbuild-nouveau.bash script

- edit the script, and fix the paths to the 2.6.24 and the
drm-rawhide kernel trees

- create a new empty directory and cd into it

- run '../mkbuild-nouveau.bash init', it should not whine
about anything, just print good stuff.

- run '../mkbuild-nouveau.bash build', it should build drm.ko and
nouveau.ko. These are probably created in the drm-rawhide kernel
tree.

- find the kernel modules, and load these:
	fb.ko
	drm.ko
	backlight.ko
	cfbcopyarea.ko
	cfbfillrect.ko
	cfbimgblt.ko
	nouveau.ko
Now the Nouveau DDX should run just fine using the new DRM modules.


Ben, I still don't see any comment from you on this thread :-)

-- 
Pekka Paalanen
http://www.iki.fi/pq/
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: mkbuild-nouveau.bash
Url:
http://lists.freedesktop.org/archives/nouveau/attachments/20090321/c4643f0c/attachment-0001.asc
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: nouveau-drm-compat-24.patch
Url:
http://lists.freedesktop.org/archives/nouveau/attachments/20090321/c4643f0c/attachment-0001.txt
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: kernel-compat-24.patch
Url:
http://lists.freedesktop.org/archives/nouveau/attachments/20090321/c4643f0c/attachment-0002.asc

On Mon, 9 Mar 2009 22:49:32 +0100
Stephane Marchesin <marchesin at icps.u-strasbg.fr> wrote:
> Hi,
> 
> As part of aiming at upstreaming our code, I suppose we have to
> discuss the DRM situation a little. In order to aim for merging, I
> think we'd better be working on a linux kernel tree layout. And
> considering we're technically the only ones still working in drm.git,
> it doesn't really make sense to keep doing that for the sake of
> sharing code with other drivers.
> 
> As we discussed on irc, there would be multiple changes related to this:
> - we move to a linux kernel-like tree, that should make it easier when
> upstreaming the code.
> - this new tree is hosted in freedekstop.org in the nouveau/ git so we
> don't need additional accounts for everyone all around and people can
> keep pushing things (even better, all nouveau people can push to the
> drm, which used to require mesa rights before)
> - we keep a(some) branch(es) in the tree for backwards compat with
> older kernels. Either in the form of separate kernel versions
> including nouveau, or in the form of an out-of-tree-compilable
> drm/nouveau module.
> 
> So, does that plan sound sane? Do you have better plans?
I had a discussion with darktama (Ben) about this.
Dave, there are questions specifically for you.
Marcheu, we need your verdict after Dave's comments.

Using a kernel tree would free Ben and others from backporting
things to drm.git. My "Building rawhide+darktama on 2.6.24"
experiment shows, that it is possible to reach practically the
same level of ease as drm.git offers to users. More on this later.
One major point here is to keep the Nouveau DRM development in
one place.

The downside is that BSD devels will probably get hurt. The
backporting work Ben and others are doing now will fall onto them.
Yet we cannot really stick to drm.git if we want upstream, since
de facto upstream is Linux. Someone has to do the porting and it's
pretty much a trade-off between helping BSDs or concentrating on
core development.

Why have a full kernel tree, and not just drivers/gpu/?
Because then it is easy to merge from upstream back to our tree.

My proposition follows:

Step 1: Nouveau development tree

This will be a complete kernel tree hosted at freedesktop.org.
Ben is ready to request it from fd.o. This tree will be free for all
developers to commit, adhering to kernel coding style. This is
NOT a tree that will be sent upstream, but all (future) patches
must be as upstream-quality as possible. This tree will
completely replace drm.git's shared-core and linux-core directories.

This tree is never rebased. Ever. Period. Upstream kernel
development will be marged back here, but we do not rebase.
Whatever gets pushed into the repo, stays there. This offers
a solid ground for devels to base their private work on.
Somewhat related, read also this:
http://sourceforge.net/mailarchive/message.php?msg_name=alpine.LFD.2.00.0903291409030.3994%40localhost.localdomain

The $1M question: which kernel tree will this be based on?
Ben tells me Dave's drm-next[1] gets rebased sometimes.

Dave, what are your plans on drm-next now after Linus'
comments? Are you able to offer a DRM kernel tree that will
not be rebased, and that we could use as the base?

Since this is not an upstreaming tree, it should not matter
too much when we pull updates from upstream, but we probably
need a policy for it.

Compatibility for older kernels can be implemented as a
branch in this tree, let's call it compat. Compat starts as a
fork of master, gets the basic compatibility patches from me,
and then we periodically just merge master into compat to keep
it up-to-date. If compatibility needs some fixes, they are just
committed into the branch. This keeps the main development and
backwards compat separate. I can maintain the compat stuff for
now, and I will also have personal interest as long as I use
2.6.24. The single compat branch will serve "all" old kernels
using #ifdefs. The compat branch is used with out-of-tree compiling.

There are several ways to offer the branches to users:
- cgit.fd.o can produce tar-balls of the complete tree, i.e. snapshots
- someone could set up an automatic snapshot service, that packages
only the drivers/gpu/ directory and other relevant files instead of
the full kernel tree
- users can clone the kernel tree without history, and still later
use git-fetch or git-pull to stay up-to-date. (See git-clone --depth)

It is possible to build drm.ko and nouveau.ko out-of-tree. This means
the modules are built from the Nouveau kernel tree sources, but
for (with) whatever kernel the user is already running (provided the
configuration supports Nouveau, and we have compat stuff for the
kernel version). The commands are a bit long,
but we can offer a script that nicely automates it, and also checks
the kernel config. I think we can get pretty close to how easy
drm.git is for building kernel modules. Currently there is one
nasty obstacle: some older kernels are missing
EXPORT_SYMBOL_GPL(shmem_file_setup), which means the user needs to
patch that in. Ben says we should be able to work around that in near
future.


Step 2: Nouveau upstreaming tree

Because the development tree is a development tree, and from the
kernel's perspective is a private tree, it will not be used for
upstreaming. As you know, accidents happen in development trees, and
the patches do not get reviewed before being pushed. A separate tree
is made for upstreaming.

This tree will live in kernel.org and have a dedicated maintainer,
who will cherry-pick patches from the Nouveau development tree for
upstreaming. The upstreaming tree is based on whatever Dave wants,
as the patches (or maybe pull-requests, when the practices have
stabilized) will be sent to Dave.

There is no need to create the upstreaming tree until Nouveau is
in the shape that it could start pushing for upstream. Hopefully
by then we have also found the maintainer (or two). Personally
I cannot promise to take this task, since I am not convinced I
have the time to do it well.

In the meanwhile, we should rehearse making upstream-quality
patches in the development tree.


Marcheu, what do you think about this? Let's give Ben the green light?
Dave, do you think the git use policy I described is feasible?

Comments, objections?
I have read the earlier comments on this thread and hope to have
answered all concerns.

A hint for kernel git trees: gitk -n 1000


Thanks.


[1] http://git.kernel.org/?p=linux/kernel/git/airlied/drm-2.6.git;a=summary

-- 
Pekka Paalanen
http://www.iki.fi/pq/