Nouveau - Oct 2021 - Advice about debugging nouveau driver suspend issue (init_on_alloc=1 and init_on_free=1)

Hello,

Thanks for the answer.

On Wed, Oct 13, 2021 at 01:43:56PM -0400, Ilia Mirkin
wrote:
> The most straightforward thing, if you can reproduce at will, would be
> to do a bisect to figure out which change this happened with. Once we
> know which change caused the problem, it will hopefully provide with
> more ability to sort out where we're going wrong. You can also play
> with things like KASAN, although it's not clear to me that they would
> necessarily catch this problem.
The patch is "mm: security: introduce init_on_alloc=1 and init_on_free=1 
boot options" (commit 6471384af2a6530696fc0203bafe4de41a23c9ef) [1]. 

It was introduced in kernel version 5.2:
$ git describe --tags 6471384af2a6530696fc0203bafe4de41a23c9ef
v5.2-5754-g6471384af2a6

This patch does not specifically change the source code of the nouveau 
module. I suppose it interacts with the memory management of nouveau 
kernel module during kernel suspend to ram and resume from ram (logs 
recorded during malfunction are available in [2]).

I activated the kasan support in the kernel (recompiling it, for example 
with kernel version 5.14.10), but I wasn't able to collect a kasan 
report/dump triggered by the issue.

A hypotesis could be that the issue is caused by an extra latency in 
buffer eviction during suspend to ram (due to memory init), that causes 
the nouveau kernel not responding to suspend, causing a "forced"
suspend
that leaves the nouveau module in an inconsistent state when the system 
is resumed. I'm not a kernel expert, therefore I don't know how to test 
this hypotesis.

Many other patches has been written over time about init_on_alloc and 
init_on_free kernel features, but the issue with nouveau driver is still 
there, up to now.

For the previous reasons, it don't know how to investigate it.

Thanks for any suggestions.

[1]
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?h=v5.14.12&id=6471384af2a6530696fc0203bafe4de41a23c9ef
[2]
https://gitlab.freedesktop.org/xorg/driver/xf86-video-nouveau/uploads/4294703a356e4c00814f581a42988999/suspend-errors-5.14.9.log.zip
> On Wed, Oct 13, 2021 at 1:31 PM Computer Enthusiastic
> <computer.enthusiastic at gmail.com> wrote:
> >
> > Hello,
> >
> > Greeting to all list members.
> >
> > I hope this mailing list is the right place to ask about it, otherwise
> > I apologize in advance and, please, direct me to the right mailing
> > list.
> >
> > I would like to receive advice on how to debug (or help to debug) the
> > noveau kernel module suspend issue reported in [1][2][3]. The issue is
> > constantly reproducible and it consists in a  screen corruption with
> > system lockup on resume after suspend to ram or suspend to disk. It
> > seems to affect NV50 nvidia graphic cards at least from kernel 5.10.0
> > (probably from 5.2 or 5.3) up to 5.14.10 (I have not tested 5.15 in rc
> > state, actually).
> >
> > Thanks in advance.
> >
> > [1]
https://gitlab.freedesktop.org/xorg/driver/xf86-video-nouveau/-/issues/547
> > [2] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=989705
> > [3] https://bugzilla.kernel.org/show_bug.cgi?id=213617

On Thu, Oct 14, 2021 at 12:51 PM Computer Enthusiastic
<computer.enthusiastic at gmail.com> wrote:
>
> Hello,
>
> Thanks for the answer.
>
> On Wed, Oct 13, 2021 at 01:43:56PM -0400, Ilia Mirkin wrote:
> > The most straightforward thing, if you can reproduce at will, would be
> > to do a bisect to figure out which change this happened with. Once we
> > know which change caused the problem, it will hopefully provide with
> > more ability to sort out where we're going wrong. You can also
play
> > with things like KASAN, although it's not clear to me that they
would
> > necessarily catch this problem.
>
> The patch is "mm: security: introduce init_on_alloc=1 and
init_on_free=1
> boot options" (commit 6471384af2a6530696fc0203bafe4de41a23c9ef) [1].
>
> It was introduced in kernel version 5.2:
> $ git describe --tags 6471384af2a6530696fc0203bafe4de41a23c9ef
> v5.2-5754-g6471384af2a6
Uff... so you've verified that a kernel at that commit +
init_on_alloc=1 and init_on_free=1 (or the CONFIG_* equivalents) will
cause the problems?

That means the problem has "always been there" (or at least we have no
great way of knowing when it might have been introduced).

I think there's some sort of SLUB debug options that help find
use-after-frees and such. What if you disable the init_on_*, enable
SLUB, and boot with like "slub_debug=FZP"?

  -ilia