Nouveau - Sep 2013 - [Bug 69928] New: [NVAA] Boot of linux kernel 3.12-rc2 hangs

https://bugs.freedesktop.org/show_bug.cgi?id=69928

          Priority: medium
            Bug ID: 69928
          Assignee: nouveau at lists.freedesktop.org
           Summary: [NVAA] Boot of linux kernel 3.12-rc2 hangs
        QA Contact: xorg-team at lists.x.org
          Severity: normal
    Classification: Unclassified
                OS: Linux (All)
          Reporter: dirkneukirchen at web.de
          Hardware: x86 (IA32)
            Status: NEW
           Version: unspecified
         Component: Driver/nouveau
           Product: xorg

Created attachment 86798
  --> https://bugs.freedesktop.org/attachment.cgi?id=86798&action=edit
3.12-rc2 fails to boot with NVAA

Steps to reproduce:
- Compiling Kernel 3.12-rc2 
- boot
- boot hangs at black screen/no signal

System:
- NVAA onboard (Jetway JNC62K)
- Linux Mint 15 (Ubuntu 13.04)

No error with 3.11.0 Kernel.

I compiled kernel 3.12-rc2 once as a dpkg and once with vanilla methods (make
install). Both same behaviour.

Attached are a netconsole log of 3.12-rc2 and a dmesg of a "working"
3.11.0


Example error message of 3.12-rc2:
[   22.081905] nouveau E[     PFB][0000:02:00.0] trapped write at 0x01002b75b4
on channel 0x00007ee0 [unknown] BAR/PFIFO_WRITE/IN reason: PAGE_NOT_PRESENT

-- 
You are receiving this mail because:
You are the assignee for the bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.freedesktop.org/archives/nouveau/attachments/20130929/22ea62ee/attachment.html>

https://bugs.freedesktop.org/show_bug.cgi?id=69928

--- Comment #1 from dirkneukirchen at web.de ---
Created attachment 86799
  --> https://bugs.freedesktop.org/attachment.cgi?id=86799&action=edit
3.11.0 boots fine

-- 
You are receiving this mail because:
You are the assignee for the bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.freedesktop.org/archives/nouveau/attachments/20130929/9c7b0aba/attachment.html>

https://bugs.freedesktop.org/show_bug.cgi?id=69928

Emil Velikov <emil.l.velikov at gmail.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |NEEDINFO

--- Comment #2 from Emil Velikov <emil.l.velikov at gmail.com> ---
Your 3.12-rc2 lod looks incomplete - note the timestamps.

AFAICS the only commit that should be affecting you is
commit a27e56996687e79416d69a7e6dc26f9d8fe06059
Author: Lucas Stach <dev at lynxeye.de>
Date:   Wed Aug 28 02:00:50 2013 +0200

    drm/nouveau: use MSI interrupts

    MSIs were only problematic on some old, broken chipsets. But now that we
    already see systems where PCI legacy interrupts are somewhat flaky, it's
    really time to move to MSIs.

    v2 (Ben Skeggs): blacklist BR02 boards


You should be able to toggle MSI off using "nouveau.config=NvMSI=0"
appended to
your kernel command line. Can you give it a try and attach the resulting dmesg

-- 
You are receiving this mail because:
You are the assignee for the bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.freedesktop.org/archives/nouveau/attachments/20130929/727cd99b/attachment.html>

https://bugs.freedesktop.org/show_bug.cgi?id=69928

--- Comment #3 from dirkneukirchen at web.de ---
Created attachment 86806
  --> https://bugs.freedesktop.org/attachment.cgi?id=86806&action=edit
dmesg 3.12-rc2 boots successful with nouveau.config=NvMSI=0
> note the timestamps
I noticed but that was my 1st usage of netconsole - maybe it was some issue
with target system (a VM), router or something else.
> You should be able to toggle MSI off using
"nouveau.config=NvMSI=0" appended to > your kernel command line.
Can you give it a try and attach the resulting dmesg
see new attachment

-- 
You are receiving this mail because:
You are the assignee for the bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.freedesktop.org/archives/nouveau/attachments/20130929/e4d7fe8a/attachment.html>

https://bugs.freedesktop.org/show_bug.cgi?id=69928

Emil Velikov <emil.l.velikov at gmail.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEEDINFO                    |NEW
           See Also|                            |https://bugs.freedesktop.or
                   |                            |g/show_bug.cgi?id=69982

--- Comment #4 from Emil Velikov <emil.l.velikov at gmail.com> ---
Thanks for confirming. The author of the patch has requested information wrt
MSI on nvidia boards. I'm hoping that we can get reply soon(ish) otherwise
we
may have to disable MSI default on all boards.

-- 
You are receiving this mail because:
You are the assignee for the bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.freedesktop.org/archives/nouveau/attachments/20131001/5edb3f33/attachment.html>

https://bugs.freedesktop.org/show_bug.cgi?id=69928

--- Comment #5 from Dmitry Chichkov <dchichkov at gmail.com> ---
Confirmed with 3.12-rc4. 

System:
 ASUS P6T SE, i7
 Nvidia GF100 [GeForce GTX 470] (rev a3).
 Ubuntu 13.04.

Details: No error with 3.10 kernel.   Boots / hangs in the text mode before
unity-greater with 3.12-rc4.     Boots succesfully / works fine with 3.12-rc4
with nouveau.config=NvMSI=0 in the boot command line.


I would suggest rising a priority on this one.

-- 
You are receiving this mail because:
You are the assignee for the bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.freedesktop.org/archives/nouveau/attachments/20131011/1d0469b3/attachment.html>

https://bugs.freedesktop.org/show_bug.cgi?id=69928

--- Comment #6 from Ben Skeggs <skeggsb at gmail.com> ---
(In reply to comment #5)
> Confirmed with 3.12-rc4. 
> 
> System:
>  ASUS P6T SE, i7
>  Nvidia GF100 [GeForce GTX 470] (rev a3).
>  Ubuntu 13.04.
> 
> Details: No error with 3.10 kernel.   Boots / hangs in the text mode before
> unity-greater with 3.12-rc4.     Boots succesfully / works fine with
> 3.12-rc4 with nouveau.config=NvMSI=0 in the boot command line.
> 
> 
> I would suggest rising a priority on this one.
MSI will be disabled by default for 3.12 as a result of this and a few other
related regressions.

To help fix this in the future, would you be able to get me a trace of the
nvidia binary driver?

Instructions of how to do so are here:
http://nouveau.freedesktop.org/wiki/MmioTrace/

Thanks!

-- 
You are receiving this mail because:
You are the assignee for the bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.freedesktop.org/archives/nouveau/attachments/20131014/01d04db6/attachment.html>

https://bugs.freedesktop.org/show_bug.cgi?id=69928

--- Comment #7 from dirkneukirchen at web.de ---
Created attachment 87677
  --> https://bugs.freedesktop.org/attachment.cgi?id=87677&action=edit
mmiotrace loading module nvidia 304

i hope kernel version doesn't play a role in mmio tracing

trace was done from fresh ubuntu 13.04 install (on usb stick)
Linux vm1304 3.8.0-19-generic #30-Ubuntu SMP Wed May 1 16:36:13 UTC 2013 i686
athlon i686 GNU/Linux

with nvidia blobs from ubuntu repos (nvidia-304)

-- 
You are receiving this mail because:
You are the assignee for the bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.freedesktop.org/archives/nouveau/attachments/20131015/bb01c6ee/attachment.html>

https://bugs.freedesktop.org/show_bug.cgi?id=69928

--- Comment #8 from dirkneukirchen at web.de ---
additional mmio traces:

simple xinit with sleep:
mmiotrace_xinit.log.xz (3.1 MB)
https://mega.co.nz/#!u4RwFLjK!Yzi4UVujupmRdEIMFZtbQBOtjyoJFR1jrXheO1RUpew

starting glxgears:
mmiotrace_glxgears.log.xz (3.4 MB)
https://mega.co.nz/#!744wlQ7b!CBnbnEYfsIwOuXKFojgy3kuecVkC6U43Bye4nZkKMUk

starting nvidia-settings - since dual screen is active already just browse
through options and then quit:
mmiotrace_dualsettings.log.xz (2.7 MB)
https://mega.co.nz/#!L1ZR3ApZ!XEuYymYmvLyQSQdpwQ0HaiBEipp2CBHz9y3PHNBNn6Q


glxgears posted because: https://bugs.freedesktop.org/show_bug.cgi?id=69952

-- 
You are receiving this mail because:
You are the assignee for the bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.freedesktop.org/archives/nouveau/attachments/20131015/a56c8758/attachment.html>

https://bugs.freedesktop.org/show_bug.cgi?id=69928

--- Comment #9 from Ben Skeggs <skeggsb at gmail.com> ---
Can you also show the contents of /proc/interrupts while X is running with the
NVIDIA binary driver?

Thanks!

-- 
You are receiving this mail because:
You are the assignee for the bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.freedesktop.org/archives/nouveau/attachments/20131015/4172bef0/attachment.html>

https://bugs.freedesktop.org/show_bug.cgi?id=69928

--- Comment #10 from dirkneukirchen at web.de ---
Created attachment 87714
  --> https://bugs.freedesktop.org/attachment.cgi?id=87714&action=edit
output /proc/interrupts every 1 second with nvidia-304

/proc/interrupts is attached from one "test" run of 
with a parallel mmiotrace

xinit nvidia-settings
(browse through some menus, then quit)
(somehow xinit "sleep 20" wasnt really working this time because some
shell
issue )

from the recovery console
- Kernel was 3.8.0-19 default provided by Ubuntu 13.04 (like above)
like before

Do you need proc/interrupts from a normal user desktop too ?

-- 
You are receiving this mail because:
You are the assignee for the bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.freedesktop.org/archives/nouveau/attachments/20131016/58f3b8ae/attachment.html>

https://bugs.freedesktop.org/show_bug.cgi?id=69928

--- Comment #11 from dirkneukirchen at web.de ---
Created attachment 87715
  --> https://bugs.freedesktop.org/attachment.cgi?id=87715&action=edit
mmiotrace of proc/interrupts run

mmiotrace of 
xinit nvidia-settings of the proc/interrupts run



with nvidia-304, Kernel 3.8.0-19 Ubuntu 13.04 default (32 bit)

-- 
You are receiving this mail because:
You are the assignee for the bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.freedesktop.org/archives/nouveau/attachments/20131016/e0168dd5/attachment.html>

https://bugs.freedesktop.org/show_bug.cgi?id=69928

--- Comment #12 from Ilia Mirkin <imirkin at alum.mit.edu> ---
The request to see /proc/interrupts was just to see if it was using MSI or not,
not for the actual interrupt counts.

 20:       5187   IO-APIC-fasteoi   ehci_hcd:usb2, nvidia

Sadly it's not using MSI. (If it were, it'd say PCI-MSI-edge or
something like
that.) Would it be possible for you to get a version that enables MSI? I
believe 325 and newer do this. If it does enable MSI, Ben could use a trace.

-- 
You are receiving this mail because:
You are the assignee for the bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.freedesktop.org/archives/nouveau/attachments/20131016/ba6c940c/attachment-0001.html>

https://bugs.freedesktop.org/show_bug.cgi?id=69928

--- Comment #13 from dirkneukirchen at web.de ---
(In reply to comment #12)
> The request to see /proc/interrupts was just to see if it was using MSI or
> not, not for the actual interrupt counts.
> 
>  20:       5187   IO-APIC-fasteoi   ehci_hcd:usb2, nvidia
> 
> Sadly it's not using MSI. (If it were, it'd say PCI-MSI-edge or
something
> like that.) Would it be possible for you to get a version that enables MSI?
> I believe 325 and newer do this. If it does enable MSI, Ben could use a
> trace.
MSI is indeed enabled in 325.15

Creating a test system (ubuntu 13.10) took a while my setup is:
- apply a patch to NVIDIA drivers because Kernel 3.11 is used (via google)
- disable modeset (kernel cmdline: nomodeset )
- blacklist vesafb, nouveau (modprobe.d entries and cmdline:
rdblacklist=nouveau  nouveau.blacklist=1  nouveau.modeset=0)
- enable vga console (cmdline: video=vesa:off vga=normal )
- install nvidia custom driver (created by --apply-patch patch311.patch)
- reboot

two mmiotraces are attached

-- 
You are receiving this mail because:
You are the assignee for the bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.freedesktop.org/archives/nouveau/attachments/20131020/1004fb01/attachment.html>

https://bugs.freedesktop.org/show_bug.cgi?id=69928

--- Comment #14 from dirkneukirchen at web.de ---
Created attachment 87876
  --> https://bugs.freedesktop.org/attachment.cgi?id=87876&action=edit
mmiotrace modprobe nvidia

mmiotrace when doing modprobe nvidia

-- 
You are receiving this mail because:
You are the assignee for the bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.freedesktop.org/archives/nouveau/attachments/20131020/995cec3e/attachment.html>

https://bugs.freedesktop.org/show_bug.cgi?id=69928

--- Comment #15 from dirkneukirchen at web.de ---
mmiotrace of xinit "sleep 10" is too big (>3000kB xz compressed)

System is a clean Ubuntu 13.10 with Nvidia 325.15 with the needed small patch


--- a/kernel/nv-linux.h
+++ b/kernel/nv-linux.h
@@ -957,7 +957,11 @@ static inline int nv_execute_on_all_cpus
 #endif

 #if !defined(NV_VMWARE)
+#if LINUX_VERSION_CODE >= KERNEL_VERSION(3, 11, 0)
+#define NV_NUM_PHYSPAGES                get_num_physpages()
+#else
 #define NV_NUM_PHYSPAGES                num_physpages
+#endif
 #define NV_GET_CURRENT_PROCESS()        current->tgid
 #define NV_IN_ATOMIC()                  in_atomic()
 #define NV_LOCAL_BH_DISABLE()           local_bh_disable()


here is the link:
mmiotrace_xinit_sleep10.log.xz (3.0 MB)
https://mega.co.nz/#!2hRgDRqb!BwPChW79v7SdYK0rvxbtT3EPmHVM300Flqfuw65Xpbs

without disabling of vesafb and enabling of vga console (see kernel cmdline in
comment above) there will be lockups after logging into X (there is a NVRM
warning about that in dmesg)

-- 
You are receiving this mail because:
You are the assignee for the bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.freedesktop.org/archives/nouveau/attachments/20131020/384b7004/attachment.html>

https://bugs.freedesktop.org/show_bug.cgi?id=69928

--- Comment #16 from Ilia Mirkin <imirkin at alum.mit.edu> ---
FTR, the trace contains:

[1] 698.683063 MMIO32 R 0x088068 0x00810005 PPCI.MSI_HEAD => { CAP_ID = MSI |
NEXT_CAP_PTR = 0 | ENABLE | QMASK = 0 | QSIZE = 0 | 64BIT }
[1] 698.683072 MMIO8 W 0x088068 0x000000ff PPCI.MSI_HEAD <= { CAP_ID = 0xff |
NEXT_CAP_PTR = 0 | QMASK = 0 | QSIZE = 0 }

and then later

[1] 699.766366 MMIO32 R 0x000100 0x80000000 PMC.INTR_HOST => { SOFTWARE }
[1] 699.766375 MMIO32 W 0x000100 0x00000000 PMC.INTR_HOST <= { 0 }
[1] 699.766434 MMIO8 W 0x088068 0x000000ff PPCI.MSI_HEAD <= { CAP_ID = 0xff |
NEXT_CAP_PTR = 0 | QMASK = 0 | QSIZE = 0 }
[1] 699.766443 MMIO32 W 0x000140 0x00000000 PMC.INTR_EN_HOST <= { 0 }

etc. So the blob does enable MSI on NVAA.

-- 
You are receiving this mail because:
You are the assignee for the bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.freedesktop.org/archives/nouveau/attachments/20131109/66949382/attachment.html>

https://bugs.freedesktop.org/show_bug.cgi?id=69928

--- Comment #17 from Pierre Moreau <pierre.morrow at free.fr> ---
Created attachment 106850
  --> https://bugs.freedesktop.org/attachment.cgi?id=106850&action=edit
Add some writes to 0x100c14

Does this patch help (applying it to a recent kernel code would be better)?
I'm
not sure it will, but well, maybe worth the try.

-- 
You are receiving this mail because:
You are the assignee for the bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.freedesktop.org/archives/nouveau/attachments/20140925/68201fe5/attachment.html>

https://bugs.freedesktop.org/show_bug.cgi?id=69928

Pierre Moreau <pierre.morrow at free.fr> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |NEEDINFO

--- Comment #18 from Pierre Moreau <pierre.morrow at free.fr> ---
Is this still an issue using kernel 3.19-rc4?

-- 
You are receiving this mail because:
You are the assignee for the bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.freedesktop.org/archives/nouveau/attachments/20150118/32c5d2cd/attachment.html>

https://bugs.freedesktop.org/show_bug.cgi?id=69928

Ilia Mirkin <imirkin at alum.mit.edu> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEEDINFO                    |RESOLVED
         Resolution|---                         |INVALID

--- Comment #19 from Ilia Mirkin <imirkin at alum.mit.edu> ---
No retest in over a year. Marking invalid.

-- 
You are receiving this mail because:
You are the assignee for the bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.freedesktop.org/archives/nouveau/attachments/20151022/e4a60cd0/attachment-0001.html>