Nouveau - Feb 2018 - [PATCH 3/3] drm/nouveau/pci: SOR crossbar quirk for 10b0:1b81

On 2018-02-05 02:39, Ben Skeggs wrote:
> On 5 February 2018 at 11:37, Ben Skeggs <skeggsb at gmail.com> wrote:
>> On 5 February 2018 at 11:22, Danilo Krummrich
>> <danilokrummrich at dk-develop.de> wrote:
>>> On Gainward GTX 1070 routing any other SOR than SOR-1 to macro link
>>> 'G' (outp index 7) causes failures:
>>> 
>>> [    6.712111] nouveau 0000:01:00.0: bus: MMIO read of 00000000
FAULT
>>> at 61c880 [ IBUS ]
>>> [    6.724888] nouveau 0000:01:00.0: disp: intr24 80000000
>>> [    8.716668] nouveau 0000:01:00.0: DRM: base-0: timeout
>>> [   10.716679] nouveau 0000:01:00.0: DRM: base-1: timeout
>>> [   63.511862] nouveau 0000:01:00.0: DRM: EVO timeout
>>> 
>>> As I'm not able to spot an issue in the driver, I suppose
it's
>>> firmware related.
>> Are you able to mail me /dev/dri/card0/vbios.rom from that, please?
>> I'd like to look into this some more and be 100% certain this is
>> indeed a quirk, and not some subtle driver bug.
> Err.. /sys/kernel/debug/dri/0/vbios.rom rather ;)
> 
Sure, that makes sense definitely, as I have checked gm200_sor_route_set 
and
gm200_sor_route_get only to conform to the statements in this mail 
thread:
https://lists.freedesktop.org/archives/nouveau/2014-December/019408.html

BTW, I can reproduce the problem with a two monitor setup only, as (of 
course) having one
monitor only at the physical port macro link 'G' is attached to makes to
vbios pick the
working SOR. Therefore the physical port macro link 'G' is attached to 
must not be picked
as primary monitor.

Also, may I ask you a related question: I was a bit confused why 'link' 
is completely unused
in nvkm_outp_init_route() after gm200_sor_route_get() returns. Is this 
just obsolete or
intended to use in the future somehow?

Thanks,
Danilo
>> 
>> Thanks,
>> Ben.
>> 
>>> 
>>> Therefore to work around this issue skip crossbar routing for this
>>> particular macro link and instead use identity mapping.
>>> 
>>> Signed-off-by: Danilo Krummrich <danilokrummrich at
dk-develop.de>
>>> ---
>>>  drivers/gpu/drm/nouveau/nvkm/engine/device/pci.c | 9 ++++++++-
>>>  1 file changed, 8 insertions(+), 1 deletion(-)
>>> 
>>> diff --git a/drivers/gpu/drm/nouveau/nvkm/engine/device/pci.c 
>>> b/drivers/gpu/drm/nouveau/nvkm/engine/device/pci.c
>>> index d2f9664afcf4..29de270f2232 100644
>>> --- a/drivers/gpu/drm/nouveau/nvkm/engine/device/pci.c
>>> +++ b/drivers/gpu/drm/nouveau/nvkm/engine/device/pci.c
>>> @@ -797,6 +797,13 @@ nvkm_device_pci_10de_139b[] = {
>>>         {}
>>>  };
>>> 
>>> +static const struct nvkm_device_pci_vendor
>>> +nvkm_device_pci_10de_1b81[] = {
>>> +       /* Gainward GTX 1070 8192 MB */
>>> +       { 0x10b0, 0x1b81, "GeForce GTX 1070",{
.outp_links_skip =
>>> BIT(7) } },
>>> +       {}
>>> +};
>>> +
>>>  static const struct nvkm_device_pci_device
>>>  nvkm_device_pci_10de[] = {
>>>         { 0x0020, "RIVA TNT" },
>>> @@ -1556,7 +1563,7 @@ nvkm_device_pci_10de[] = {
>>>         { 0x1b06, "GeForce GTX 1080 TI" },
>>>         { 0x1bb7, "Quadro P6000" },
>>>         { 0x1b80, "GeForce GTX 1080" },
>>> -       { 0x1b81, "GeForce GTX 1070" },
>>> +       { 0x1b81, "GeForce GTX 1070",
nvkm_device_pci_10de_1b81 },
>>>         { 0x1b82, "GeForce GTX 1070 TI" },
>>>         { 0x1b84, "GeForce GTX 1060 3GB" },
>>>         { 0x1b87, "P104-100" },
>>> --
>>> 2.14.1
>>> 
>>> _______________________________________________
>>> Nouveau mailing list
>>> Nouveau at lists.freedesktop.org
>>> https://lists.freedesktop.org/mailman/listinfo/nouveau
> _______________________________________________
> dri-devel mailing list
> dri-devel at lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/dri-devel
-------------- next part --------------
A non-text attachment was scrubbed...
Name: vbios.rom
Type: application/octet-stream
Size: 232448 bytes
Desc: not available
URL:
<https://lists.freedesktop.org/archives/nouveau/attachments/20180205/05868fa0/attachment-0001.obj>

On Mon, Feb 5, 2018 at 12:19 PM, Danilo Krummrich
<danilokrummrich at dk-develop.de> wrote:
> On 2018-02-05 02:39, Ben Skeggs wrote:
>>
>> On 5 February 2018 at 11:37, Ben Skeggs <skeggsb at gmail.com>
wrote:
>>>
>>> On 5 February 2018 at 11:22, Danilo Krummrich
>>> <danilokrummrich at dk-develop.de> wrote:
>>>>
>>>> On Gainward GTX 1070 routing any other SOR than SOR-1 to macro
link
>>>> 'G' (outp index 7) causes failures:
>>>>
>>>> [    6.712111] nouveau 0000:01:00.0: bus: MMIO read of 00000000
FAULT at
>>>> 61c880 [ IBUS ]
>>>> [    6.724888] nouveau 0000:01:00.0: disp: intr24 80000000
>>>> [    8.716668] nouveau 0000:01:00.0: DRM: base-0: timeout
>>>> [   10.716679] nouveau 0000:01:00.0: DRM: base-1: timeout
>>>> [   63.511862] nouveau 0000:01:00.0: DRM: EVO timeout
>>>>
>>>> As I'm not able to spot an issue in the driver, I suppose
it's
>>>> firmware related.
>>>
>>> Are you able to mail me /dev/dri/card0/vbios.rom from that, please?
>>> I'd like to look into this some more and be 100% certain this
is
>>> indeed a quirk, and not some subtle driver bug.
>>
>> Err.. /sys/kernel/debug/dri/0/vbios.rom rather ;)
>>
> Sure, that makes sense definitely, as I have checked gm200_sor_route_set
and
> gm200_sor_route_get only to conform to the statements in this mail thread:
> https://lists.freedesktop.org/archives/nouveau/2014-December/019408.html
>
> BTW, I can reproduce the problem with a two monitor setup only, as (of
> course) having one
> monitor only at the physical port macro link 'G' is attached to
makes to
> vbios pick the
> working SOR. Therefore the physical port macro link 'G' is attached
to must
> not be picked
> as primary monitor.
Thanks for that.  I've only had a quick look so far, but I'm going to
guess the is a driver bug already.  The DCB specifies two different
outputs on pad macro 1 (which, would be SOR1 if identity-mapped) that
can apparently be used together.  If used at the same time though,
they both can't be driven by the same SOR, and would need routing.

I guess it'd be interesting to see if NVIDIA can manage to drive those
two outputs together, which would be a big hint as to whether the
board is buggy, or we are.  I'm going to guess the latter ;)
>
> Also, may I ask you a related question: I was a bit confused why
'link' is
> completely unused
> in nvkm_outp_init_route() after gm200_sor_route_get() returns. Is this just
> obsolete or
> intended to use in the future somehow?
I suspect it's a left-over from an earlier revision of that code, or
perhaps I intended to validate it against what we discovered?  Not
sure now!

Thanks,
Ben.
>
> Thanks,
> Danilo
>
>>>
>>> Thanks,
>>> Ben.
>>>
>>>>
>>>> Therefore to work around this issue skip crossbar routing for
this
>>>> particular macro link and instead use identity mapping.
>>>>
>>>> Signed-off-by: Danilo Krummrich <danilokrummrich at
dk-develop.de>
>>>> ---
>>>>  drivers/gpu/drm/nouveau/nvkm/engine/device/pci.c | 9 ++++++++-
>>>>  1 file changed, 8 insertions(+), 1 deletion(-)
>>>>
>>>> diff --git a/drivers/gpu/drm/nouveau/nvkm/engine/device/pci.c
>>>> b/drivers/gpu/drm/nouveau/nvkm/engine/device/pci.c
>>>> index d2f9664afcf4..29de270f2232 100644
>>>> --- a/drivers/gpu/drm/nouveau/nvkm/engine/device/pci.c
>>>> +++ b/drivers/gpu/drm/nouveau/nvkm/engine/device/pci.c
>>>> @@ -797,6 +797,13 @@ nvkm_device_pci_10de_139b[] = {
>>>>         {}
>>>>  };
>>>>
>>>> +static const struct nvkm_device_pci_vendor
>>>> +nvkm_device_pci_10de_1b81[] = {
>>>> +       /* Gainward GTX 1070 8192 MB */
>>>> +       { 0x10b0, 0x1b81, "GeForce GTX 1070",{
.outp_links_skip = BIT(7)
>>>> } },
>>>> +       {}
>>>> +};
>>>> +
>>>>  static const struct nvkm_device_pci_device
>>>>  nvkm_device_pci_10de[] = {
>>>>         { 0x0020, "RIVA TNT" },
>>>> @@ -1556,7 +1563,7 @@ nvkm_device_pci_10de[] = {
>>>>         { 0x1b06, "GeForce GTX 1080 TI" },
>>>>         { 0x1bb7, "Quadro P6000" },
>>>>         { 0x1b80, "GeForce GTX 1080" },
>>>> -       { 0x1b81, "GeForce GTX 1070" },
>>>> +       { 0x1b81, "GeForce GTX 1070",
nvkm_device_pci_10de_1b81 },
>>>>         { 0x1b82, "GeForce GTX 1070 TI" },
>>>>         { 0x1b84, "GeForce GTX 1060 3GB" },
>>>>         { 0x1b87, "P104-100" },
>>>> --
>>>> 2.14.1
>>>>
>>>> _______________________________________________
>>>> Nouveau mailing list
>>>> Nouveau at lists.freedesktop.org
>>>> https://lists.freedesktop.org/mailman/listinfo/nouveau
>>
>> _______________________________________________
>> dri-devel mailing list
>> dri-devel at lists.freedesktop.org
>> https://lists.freedesktop.org/mailman/listinfo/dri-devel

On 2018-02-05 03:47, Ben Skeggs wrote:
> On Mon, Feb 5, 2018 at 12:19 PM, Danilo Krummrich
> <danilokrummrich at dk-develop.de> wrote:
>> On 2018-02-05 02:39, Ben Skeggs wrote:
>>> 
>>> On 5 February 2018 at 11:37, Ben Skeggs <skeggsb at
gmail.com> wrote:
>>>> 
>>>> On 5 February 2018 at 11:22, Danilo Krummrich
>>>> <danilokrummrich at dk-develop.de> wrote:
>>>>> 
>>>>> On Gainward GTX 1070 routing any other SOR than SOR-1 to
macro link
>>>>> 'G' (outp index 7) causes failures:
>>>>> 
>>>>> [    6.712111] nouveau 0000:01:00.0: bus: MMIO read of
00000000
>>>>> FAULT at
>>>>> 61c880 [ IBUS ]
>>>>> [    6.724888] nouveau 0000:01:00.0: disp: intr24 80000000
>>>>> [    8.716668] nouveau 0000:01:00.0: DRM: base-0: timeout
>>>>> [   10.716679] nouveau 0000:01:00.0: DRM: base-1: timeout
>>>>> [   63.511862] nouveau 0000:01:00.0: DRM: EVO timeout
>>>>> 
>>>>> As I'm not able to spot an issue in the driver, I
suppose it's
>>>>> firmware related.
>>>> 
>>>> Are you able to mail me /dev/dri/card0/vbios.rom from that,
please?
>>>> I'd like to look into this some more and be 100% certain
this is
>>>> indeed a quirk, and not some subtle driver bug.
>>> 
>>> Err.. /sys/kernel/debug/dri/0/vbios.rom rather ;)
>>> 
>> Sure, that makes sense definitely, as I have checked 
>> gm200_sor_route_set and
>> gm200_sor_route_get only to conform to the statements in this mail 
>> thread:
>>
https://lists.freedesktop.org/archives/nouveau/2014-December/019408.html
>> 
>> BTW, I can reproduce the problem with a two monitor setup only, as (of
>> course) having one
>> monitor only at the physical port macro link 'G' is attached to
makes
>> to
>> vbios pick the
>> working SOR. Therefore the physical port macro link 'G' is
attached to
>> must
>> not be picked
>> as primary monitor.
> Thanks for that.  I've only had a quick look so far, but I'm going
to
> guess the is a driver bug already.  The DCB specifies two different
> outputs on pad macro 1 (which, would be SOR1 if identity-mapped) that
First of all, sorry for confusing with the macro link numeration.
Of course, I am talking about pad macro 1, link 2 (which is also called
macro link D). I was wrongly looking at outp->index.
I will send an updated patch series correcting this, just in
case.
> can apparently be used together.  If used at the same time though,
> they both can't be driven by the same SOR, and would need routing.
I agree that there's definitely the need of routing here. Anyway, it
can still be it's just buggy in a way that only macro link D cannot be
routed to a different SOR than SOR-1. Probably you know better, if such
a case can be possible.
> I guess it'd be interesting to see if NVIDIA can manage to drive those
> two outputs together, which would be a big hint as to whether the
> board is buggy, or we are.  I'm going to guess the latter ;)
> 
I tested that - even nouveau is able to drive macro link C and D 
together.
Macro link C corresponds to connector 3 (which is HDMI) and macro link D
corresponds to connector 4 (which is DP). And it actually works because 
VBIOS
serves macro link D as primary OR and routes SOR-1 to it already.
Nouveau picks (of course) SOR-0 for macro link C then.
BTW, any other combination works well, as long as macro link D (if used) 
is
not routed to another SOR than SOR-1. E.g. macro link A on SOR-2 and 
macro
link E on either SOR-0 or SOR-3.
>> 
>> Also, may I ask you a related question: I was a bit confused why 
>> 'link' is
>> completely unused
>> in nvkm_outp_init_route() after gm200_sor_route_get() returns. Is this 
>> just
>> obsolete or
>> intended to use in the future somehow?
> I suspect it's a left-over from an earlier revision of that code, or
> perhaps I intended to validate it against what we discovered?  Not
> sure now!
> 
> Thanks,
> Ben.
> 
>> 
>> Thanks,
>> Danilo
>> 
>>>> 
>>>> Thanks,
>>>> Ben.
>>>> 
>>>>> 
>>>>> Therefore to work around this issue skip crossbar routing
for this
>>>>> particular macro link and instead use identity mapping.
>>>>> 
>>>>> Signed-off-by: Danilo Krummrich <danilokrummrich at
dk-develop.de>
>>>>> ---
>>>>>  drivers/gpu/drm/nouveau/nvkm/engine/device/pci.c | 9
++++++++-
>>>>>  1 file changed, 8 insertions(+), 1 deletion(-)
>>>>> 
>>>>> diff --git
a/drivers/gpu/drm/nouveau/nvkm/engine/device/pci.c
>>>>> b/drivers/gpu/drm/nouveau/nvkm/engine/device/pci.c
>>>>> index d2f9664afcf4..29de270f2232 100644
>>>>> --- a/drivers/gpu/drm/nouveau/nvkm/engine/device/pci.c
>>>>> +++ b/drivers/gpu/drm/nouveau/nvkm/engine/device/pci.c
>>>>> @@ -797,6 +797,13 @@ nvkm_device_pci_10de_139b[] = {
>>>>>         {}
>>>>>  };
>>>>> 
>>>>> +static const struct nvkm_device_pci_vendor
>>>>> +nvkm_device_pci_10de_1b81[] = {
>>>>> +       /* Gainward GTX 1070 8192 MB */
>>>>> +       { 0x10b0, 0x1b81, "GeForce GTX 1070",{
.outp_links_skip =
>>>>> BIT(7)
>>>>> } },
>>>>> +       {}
>>>>> +};
>>>>> +
>>>>>  static const struct nvkm_device_pci_device
>>>>>  nvkm_device_pci_10de[] = {
>>>>>         { 0x0020, "RIVA TNT" },
>>>>> @@ -1556,7 +1563,7 @@ nvkm_device_pci_10de[] = {
>>>>>         { 0x1b06, "GeForce GTX 1080 TI" },
>>>>>         { 0x1bb7, "Quadro P6000" },
>>>>>         { 0x1b80, "GeForce GTX 1080" },
>>>>> -       { 0x1b81, "GeForce GTX 1070" },
>>>>> +       { 0x1b81, "GeForce GTX 1070",
nvkm_device_pci_10de_1b81 },
>>>>>         { 0x1b82, "GeForce GTX 1070 TI" },
>>>>>         { 0x1b84, "GeForce GTX 1060 3GB" },
>>>>>         { 0x1b87, "P104-100" },
>>>>> --
>>>>> 2.14.1
>>>>> 
>>>>> _______________________________________________
>>>>> Nouveau mailing list
>>>>> Nouveau at lists.freedesktop.org
>>>>> https://lists.freedesktop.org/mailman/listinfo/nouveau
>>> 
>>> _______________________________________________
>>> dri-devel mailing list
>>> dri-devel at lists.freedesktop.org
>>> https://lists.freedesktop.org/mailman/listinfo/dri-devel
> _______________________________________________
> dri-devel mailing list
> dri-devel at lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/dri-devel