Nouveau - Nov 2019 - [PATCH v4] pci: prevent putting nvidia GPUs into lower device states on certain intel bridges

On Thu, Nov 21, 2019 at 1:53 PM Karol Herbst <kherbst at redhat.com>
wrote:
>
> On Thu, Nov 21, 2019 at 12:46 PM Mika Westerberg
> <mika.westerberg at intel.com> wrote:
> >
> > On Thu, Nov 21, 2019 at 12:34:22PM +0100, Rafael J. Wysocki wrote:
> > > On Thu, Nov 21, 2019 at 12:28 PM Mika Westerberg
> > > <mika.westerberg at intel.com> wrote:
> > > >
> > > > On Wed, Nov 20, 2019 at 11:29:33PM +0100, Rafael J. Wysocki
wrote:
> > > > > > last week or so I found systems where the GPU was
under the "PCI
> > > > > > Express Root Port" (name from lspci) and on
those systems all of that
> > > > > > seems to work. So I am wondering if it's
indeed just the 0x1901 one,
> > > > > > which also explains Mikas case that Thunderbolt
stuff works as devices
> > > > > > never get populated under this particular bridge
controller, but under
> > > > > > those "Root Port"s
> > > > >
> > > > > It always is a PCIe port, but its location within the
SoC may matter.
> > > >
> > > > Exactly. Intel hardware has PCIe ports on CPU side (these
are called
> > > > PEG, PCI Express Graphics, ports), and the PCH side. I think
the IP is
> > > > still the same.
> > > >
>
> yeah, I meant the bridge controller with the ID 0x1901 is on the CPU
> side. And if the Nvidia GPU is on a port on the PCH side it all seems
> to work just fine.
But that may involve different AML too, may it not?
> > > > > Also some custom AML-based power management is involved
and that may
> > > > > be making specific assumptions on the configuration of
the SoC and the
> > > > > GPU at the time of its invocation which unfortunately
are not known to
> > > > > us.
> > > > >
> > > > > However, it looks like the AML invoked to power down
the GPU from
> > > > > acpi_pci_set_power_state() gets confused if it is not
in PCI D0 at
> > > > > that point, so it looks like that AML tries to access
device memory on
> > > > > the GPU (beyond the PCI config space) or similar which
is not
> > > > > accessible in PCI power states below D0.
> > > >
> > > > Or the PCI config space of the GPU when the parent root port
is in D3hot
> > > > (as it is the case here). Also then the GPU config space is
not
> > > > accessible.
> > >
> > > Why would the parent port be in D3hot at that point? 
Wouldn't that be
> > > a suspend ordering violation?
> >
> > No. We put the GPU into D3hot first, then the root port and then turn
> > off the power resource (which is attached to the root port) resulting
> > the topology entering D3cold.
> >
>
> If the kernel does a D0 -> D3hot -> D0 cycle this works as well, but
> the power savings are way lower, so I kind of prefer skipping D3hot
> instead of D3cold. Skipping D3hot doesn't seem to make any difference
> in power savings in my testing.
OK

What exactly did you do to skip D3cold in your testing?

On Thu, Nov 21, 2019 at 4:47 PM Rafael J. Wysocki <rafael at kernel.org>
wrote:
>
> On Thu, Nov 21, 2019 at 1:53 PM Karol Herbst <kherbst at redhat.com>
wrote:
> >
> > On Thu, Nov 21, 2019 at 12:46 PM Mika Westerberg
> > <mika.westerberg at intel.com> wrote:
> > >
> > > On Thu, Nov 21, 2019 at 12:34:22PM +0100, Rafael J. Wysocki
wrote:
> > > > On Thu, Nov 21, 2019 at 12:28 PM Mika Westerberg
> > > > <mika.westerberg at intel.com> wrote:
> > > > >
> > > > > On Wed, Nov 20, 2019 at 11:29:33PM +0100, Rafael J.
Wysocki wrote:
> > > > > > > last week or so I found systems where the GPU
was under the "PCI
> > > > > > > Express Root Port" (name from lspci) and
on those systems all of that
> > > > > > > seems to work. So I am wondering if it's
indeed just the 0x1901 one,
> > > > > > > which also explains Mikas case that
Thunderbolt stuff works as devices
> > > > > > > never get populated under this particular
bridge controller, but under
> > > > > > > those "Root Port"s
> > > > > >
> > > > > > It always is a PCIe port, but its location within
the SoC may matter.
> > > > >
> > > > > Exactly. Intel hardware has PCIe ports on CPU side
(these are called
> > > > > PEG, PCI Express Graphics, ports), and the PCH side. I
think the IP is
> > > > > still the same.
> > > > >
> >
> > yeah, I meant the bridge controller with the ID 0x1901 is on the CPU
> > side. And if the Nvidia GPU is on a port on the PCH side it all seems
> > to work just fine.
>
> But that may involve different AML too, may it not?
>
> > > > > > Also some custom AML-based power management is
involved and that may
> > > > > > be making specific assumptions on the
configuration of the SoC and the
> > > > > > GPU at the time of its invocation which
unfortunately are not known to
> > > > > > us.
> > > > > >
> > > > > > However, it looks like the AML invoked to power
down the GPU from
> > > > > > acpi_pci_set_power_state() gets confused if it is
not in PCI D0 at
> > > > > > that point, so it looks like that AML tries to
access device memory on
> > > > > > the GPU (beyond the PCI config space) or similar
which is not
> > > > > > accessible in PCI power states below D0.
> > > > >
> > > > > Or the PCI config space of the GPU when the parent root
port is in D3hot
> > > > > (as it is the case here). Also then the GPU config
space is not
> > > > > accessible.
> > > >
> > > > Why would the parent port be in D3hot at that point? 
Wouldn't that be
> > > > a suspend ordering violation?
> > >
> > > No. We put the GPU into D3hot first, then the root port and then
turn
> > > off the power resource (which is attached to the root port)
resulting
> > > the topology entering D3cold.
> > >
> >
> > If the kernel does a D0 -> D3hot -> D0 cycle this works as well,
but
> > the power savings are way lower, so I kind of prefer skipping D3hot
> > instead of D3cold. Skipping D3hot doesn't seem to make any
difference
> > in power savings in my testing.
>
> OK
>
> What exactly did you do to skip D3cold in your testing?
>
For that I poked into the PCI registers directly and skipped doing the
ACPI calls and simply checked for the idle power consumption on my
laptop. But I guess I should retest with calling pci_d3cold_disable
from nouveau instead? Or is there a different preferable way of
testing this?

On Thu, Nov 21, 2019 at 5:06 PM Karol Herbst <kherbst at redhat.com>
wrote:
>
> On Thu, Nov 21, 2019 at 4:47 PM Rafael J. Wysocki <rafael at
kernel.org> wrote:
> >
> > On Thu, Nov 21, 2019 at 1:53 PM Karol Herbst <kherbst at
redhat.com> wrote:
> > >
> > > On Thu, Nov 21, 2019 at 12:46 PM Mika Westerberg
> > > <mika.westerberg at intel.com> wrote:
> > > >
> > > > On Thu, Nov 21, 2019 at 12:34:22PM +0100, Rafael J. Wysocki
wrote:
> > > > > On Thu, Nov 21, 2019 at 12:28 PM Mika Westerberg
> > > > > <mika.westerberg at intel.com> wrote:
> > > > > >
> > > > > > On Wed, Nov 20, 2019 at 11:29:33PM +0100, Rafael
J. Wysocki wrote:
> > > > > > > > last week or so I found systems where
the GPU was under the "PCI
> > > > > > > > Express Root Port" (name from
lspci) and on those systems all of that
> > > > > > > > seems to work. So I am wondering if
it's indeed just the 0x1901 one,
> > > > > > > > which also explains Mikas case that
Thunderbolt stuff works as devices
> > > > > > > > never get populated under this
particular bridge controller, but under
> > > > > > > > those "Root Port"s
> > > > > > >
> > > > > > > It always is a PCIe port, but its location
within the SoC may matter.
> > > > > >
> > > > > > Exactly. Intel hardware has PCIe ports on CPU side
(these are called
> > > > > > PEG, PCI Express Graphics, ports), and the PCH
side. I think the IP is
> > > > > > still the same.
> > > > > >
> > >
> > > yeah, I meant the bridge controller with the ID 0x1901 is on the
CPU
> > > side. And if the Nvidia GPU is on a port on the PCH side it all
seems
> > > to work just fine.
> >
> > But that may involve different AML too, may it not?
> >
> > > > > > > Also some custom AML-based power management
is involved and that may
> > > > > > > be making specific assumptions on the
configuration of the SoC and the
> > > > > > > GPU at the time of its invocation which
unfortunately are not known to
> > > > > > > us.
> > > > > > >
> > > > > > > However, it looks like the AML invoked to
power down the GPU from
> > > > > > > acpi_pci_set_power_state() gets confused if
it is not in PCI D0 at
> > > > > > > that point, so it looks like that AML tries
to access device memory on
> > > > > > > the GPU (beyond the PCI config space) or
similar which is not
> > > > > > > accessible in PCI power states below D0.
> > > > > >
> > > > > > Or the PCI config space of the GPU when the parent
root port is in D3hot
> > > > > > (as it is the case here). Also then the GPU config
space is not
> > > > > > accessible.
> > > > >
> > > > > Why would the parent port be in D3hot at that point? 
Wouldn't that be
> > > > > a suspend ordering violation?
> > > >
> > > > No. We put the GPU into D3hot first, then the root port and
then turn
> > > > off the power resource (which is attached to the root port)
resulting
> > > > the topology entering D3cold.
> > > >
> > >
> > > If the kernel does a D0 -> D3hot -> D0 cycle this works as
well, but
> > > the power savings are way lower, so I kind of prefer skipping
D3hot
> > > instead of D3cold. Skipping D3hot doesn't seem to make any
difference
> > > in power savings in my testing.
> >
> > OK
> >
> > What exactly did you do to skip D3cold in your testing?
> >
>
> For that I poked into the PCI registers directly and skipped doing the
> ACPI calls and simply checked for the idle power consumption on my
> laptop.
That doesn't involve the PCIe port PM, however.
> But I guess I should retest with calling pci_d3cold_disable
> from nouveau instead? Or is there a different preferable way of
> testing this?
There is a sysfs attribute called "d3cold_allowed" which can be used
for "blocking" D3cold, so can you please retest using that?