Nouveau - Nov 2017 - Addressing the problem of noisy GPUs under Nouveau

Hey,

Thanks for your answer, Andy!

On 22/11/17 04:06, Ilia Mirkin wrote:
> On Tue, Nov 21, 2017 at 8:29 PM, Andy Ritger <aritger at nvidia.com>
wrote:
>> Hi Martin,
> 
> Martin should have complete answers,
> 
>>
>> I was asked to clarify a few things:
>>
>> (1) Are all the user reports of loud fans on Fermi-era GPUs?
> 
> Yes. Although I believe some GK208 users are also having trouble,
> including yours truly. (It's been quite a while since I've checked
> though... my memory is weak in that regard.)
We did not hear back from a lot of users about these issues, but I can
see that most GF108 vbios in our vbios repo are problematic, and some
GK106/GT215/GT216/GT218 might be too.
> 
>>
>> (2) When the VBIOS POSTs the card, it loads initial ucode onto the
Falcon
>> processor (PMU), which will do basic fan management on its own.  We
call this
>> init ucode "IFR" (Init From ROM).  nvidia.ko will restore the
IFR ucode when
>> unloaded.  I assume the loud fan symptom occurs after Nouveau is loaded
and
>> running, correct?  I.e., this is a problem in Nouveau's fan control
>> programming, rather than a problem in IFR.
> 
> Correct.
Indeed.
> 
>>
>> (3) IFR will run until something else is loaded on the Falcon processor
(PMU).
>> On Fermi, I assume the Nouveau kernel driver is uploading the
Nouveau-written
>> ucode from here:
>>
>>     drivers/gpu/drm/nouveau/nvkm/subdev/pmu/fuc
>>
>> correct?  I only ask to rule out the possibility that IFR and Nouveau
are both
>> attempting to program fans simultaneously.  The symptoms you describe
don't
>> sound like that, but just double checking...
> 
> Correct.
Indeed.
> 
>>
>> (4) Given the PMU ucode debacle, I'm embarrassed to ask, but at
least on Fermi,
>> how much does Nouveau strictly depend on Nouveau's PMU ucode? 
Would it be an
>> option to just let IFR continue to manage fans?
> 
> Reclocking is still on our horizon, which clearly won't happen without
> nouveau PMU code loaded. Not sure what it's used for until reclocking
> becomes a thing on Fermi.
Yeah, this would hinder our reclocking efforts :s

The best idea I can come up with is to fake the temperature (register
0x20408) to 1°C (minimum the hardware allows us) and read the PWM duty,
then we can get the maximum duty by setting the temperature to the
fan_boost threshold.

Not sure we have a sure-way of computing the fan_boost threshold though,
maybe we can just use of the thermal throttling threshold for this (more
on this later in the email).

In any case, all of these solutions are workarounds. Given that the code
to compute these values is already found in vbioses, why is it a problem
to share the meaning of all the values in the fan calibration table,
and/or the algorithm?
> 
>>
>> (5) Lastly, I was asked how Nouveau determines what fan speed to
(attempt
>> to) program.
Oh, thanks for giving me an idea about what the other values in this
table may be about :D

Anyways, the current code uses the entry id 0x46 of the thermal table
(bit P, offset 0x10) to find out what are the thermal points for
$fan_min and $fan_max. The $fan_min and $fan_max values are found in the
entry id 0x22 of the same table.

If the 0x46 entry is not present in the thermal table (which seems to be
the norm for Fermi), we revert to default values: 40 -> 85°C.

With these 4 values, we get 2 trip points (temp_min, fan_min)
and(temp_max, fan_max), and we merely do linear interpolation between them.
> 
> I'll let Martin answer this, but as you're probably aware,
there's 2
> different ways this can be done - there might be a PWM, we might have
> to toggle it manually. Maybe something else still.
The manual toggle fans are only present on pre-tesla GPUs, let's ignore
them here, because we know what to do there.

All recent (2006+) GPUs use PWM, and anything after the GT215 use this
fan calibration table which took me a while to find, and that is still
mostly a mystery to me :s
> 
> Have a look at drm/nouveau/nvkm/subdev/therm/fan.c and the various
> bits it ends up calling (pre-GF119 fermi's end up with the nv50
> fan_set, I believe).
> 
> The bios stuff is parsed in nvkm/subdev/bios/fan.c and therm.c,
> although I believe Martin's latest analysis is more advanced than
> what's in that code.
Absolutely :) I have not updated Nouveau yet, in fear of setting a value
lower than what the proprietary driver does...
> 
> Martin's question was very long, but it boils down to this:
> 
> How do we compute the correct values to write into the e114/e118 pwm
> registers based on the VBIOS contents and current state of the board
> (like temperature).
Unfortunately, it can also be the e11c/e120 couple, or 0x200d8/dc on
GF119+, or 0x200cd/d0 on Kepler+.

At least, it looks like we know which PWM controler we need to drive, so
I did not want to muddy the water even more by giving register
addresses, rather concentrating on the problem at hand: How to compute
the duty value for the PWM controler.
> 
> We generally do this right, but appear to get it extra-wrong for certain
GPUs.
Yes... So far, we are always safe, but users tend to mind when their
computer sound like a jumbo jet at take off... Who would have thought? :D

Anyway, looking forward to your answer!

Cheers,
Martin

On 11/22/2017 05:07 PM, Martin Peres wrote:
> Hey,
> 
> Thanks for your answer, Andy!
> 
> On 22/11/17 04:06, Ilia Mirkin wrote:
>> On Tue, Nov 21, 2017 at 8:29 PM, Andy Ritger <aritger at
nvidia.com> wrote:
>> Martin's question was very long, but it boils down to this:
>>
>> How do we compute the correct values to write into the e114/e118 pwm
>> registers based on the VBIOS contents and current state of the board
>> (like temperature).
> 
> Unfortunately, it can also be the e11c/e120 couple, or 0x200d8/dc on
> GF119+, or 0x200cd/d0 on Kepler+.
> 
> At least, it looks like we know which PWM controler we need to drive, so
> I did not want to muddy the water even more by giving register
> addresses, rather concentrating on the problem at hand: How to compute
> the duty value for the PWM controler.
> 
>>
>> We generally do this right, but appear to get it extra-wrong for
certain GPUs.
> 
> Yes... So far, we are always safe, but users tend to mind when their
> computer sound like a jumbo jet at take off... Who would have thought? :D
> 
> Anyway, looking forward to your answer!
> 
> Cheers,
> Martin

Hi Martin,

One of our firmware engineers thinks that this looks a lot like PWM inversion.
For some SKUs, the interpretation of the PWM duty cycle is inverted. That 
would probably make it *very* difficult to find a sensible algorithm that 
covered all the SKUs, given that some are inverted and others are not.

For the noisy GPUs, a very useful experiment would be to try inverting it, 
like this:

	pwmDutyCycle = pwmPeriod - pwmDutyCycle;

...and then see if fan control starts behaving closer to how you've actually
programmed it.

Would that be easy enough to try out? It should help narrow down the
problem at least.

thanks,
John Hubbard

On 23/11/17 10:06, John Hubbard wrote:
> On 11/22/2017 05:07 PM, Martin Peres wrote:
>> Hey,
>>
>> Thanks for your answer, Andy!
>>
>> On 22/11/17 04:06, Ilia Mirkin wrote:
>>> On Tue, Nov 21, 2017 at 8:29 PM, Andy Ritger <aritger at
nvidia.com> wrote:
>>> Martin's question was very long, but it boils down to this:
>>>
>>> How do we compute the correct values to write into the e114/e118
pwm
>>> registers based on the VBIOS contents and current state of the
board
>>> (like temperature).
>>
>> Unfortunately, it can also be the e11c/e120 couple, or 0x200d8/dc on
>> GF119+, or 0x200cd/d0 on Kepler+.
>>
>> At least, it looks like we know which PWM controler we need to drive,
so
>> I did not want to muddy the water even more by giving register
>> addresses, rather concentrating on the problem at hand: How to compute
>> the duty value for the PWM controler.
>>
>>>
>>> We generally do this right, but appear to get it extra-wrong for
certain GPUs.
>>
>> Yes... So far, we are always safe, but users tend to mind when their
>> computer sound like a jumbo jet at take off... Who would have thought?
:D
>>
>> Anyway, looking forward to your answer!
>>
>> Cheers,
>> Martin
> 
> 
> Hi Martin,
> 
> One of our firmware engineers thinks that this looks a lot like PWM
inversion.
> For some SKUs, the interpretation of the PWM duty cycle is inverted. That 
> would probably make it *very* difficult to find a sensible algorithm that 
> covered all the SKUs, given that some are inverted and others are not.
> 
> For the noisy GPUs, a very useful experiment would be to try inverting it, 
> like this:
> 
> 	pwmDutyCycle = pwmPeriod - pwmDutyCycle;
> 
> ...and then see if fan control starts behaving closer to how you've
actually
> programmed it.
> 
> Would that be easy enough to try out? It should help narrow down the
> problem at least.
>
Hey John,

Unfortunately, we know about PWM inversion, and one can know which mode
to use based on the GPIO entry associated to the fan (inverted). We have
had support for this in Nouveau for a long time. At the very least, this
is not the problem on my GF108.

I am certain that the problem I am seeing is related to this vbios table
I wrote about (BIT P, offset 0x18). It is used to compute what PWM duty
I should use for both 0 and 100% of the fan speed.

Computing the value for 0% fan speed is difficult because of
non-continuous nature of some of the functions[1], but I can always
over-approximate. However, I failed to accurately compute the duty I
need to write to get the 100% fan speed (I have cases where I greatly
over-estimate it...).

Could you please check out the vbios table I am pointing at? I am quite
sure that your documentation will be clearer than my babbling :D

Thanks,
Martin

[1] http://fs.mupuf.org/nvidia/fan_calib/pwm_offset.png