Looks like my AMD box isn't quite working correctly with regards to P-states. I noted this repeating periodically: hwpstate0: set freq failed, err 6 I noted these errors when setting the hwpstate verbose systctl on: May 17 22:28:32 Alice kernel: hwpstate0: set freq failed, err 6 May 17 22:28:32 Alice kernel: hwpstate0: setting P1-state on cpu0 May 17 22:28:32 Alice kernel: hwpstate0: result P1-state on cpu0 May 17 22:28:32 Alice kernel: hwpstate0: setting P1-state on cpu1 May 17 22:28:32 Alice kernel: hwpstate0: result P1-state on cpu1 May 17 22:28:32 Alice kernel: hwpstate0: setting P1-state on cpu2 May 17 22:28:32 Alice kernel: hwpstate0: result P1-state on cpu2 May 17 22:28:32 Alice kernel: hwpstate0: setting P1-state on cpu3 May 17 22:28:32 Alice kernel: hwpstate0: result P1-state on cpu3 May 17 22:28:32 Alice kernel: hwpstate0: setting P1-state on cpu4 May 17 22:28:32 Alice kernel: hwpstate0: result P1-state on cpu4 May 17 22:28:32 Alice kernel: hwpstate0: setting P1-state on cpu5 May 17 22:28:32 Alice kernel: hwpstate0: result P1-state on cpu5 May 17 22:28:32 Alice kernel: hwpstate0: setting P1-state on cpu6 May 17 22:28:32 Alice kernel: hwpstate0: result P1-state on cpu6 May 17 22:28:32 Alice kernel: hwpstate0: setting P1-state on cpu7 May 17 22:28:32 Alice kernel: hwpstate0: result P1-state on cpu7 Sean ref this machine: http://forums.pcbsd.org/showthread.php?t=16733
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 2012-05-18 01:32:09 -0400, Sean Bruno wrote:> Looks like my AMD box isn't quite working correctly with regards > to P-states. I noted this repeating periodically: hwpstate0: set > freq failed, err 6 > > > I noted these errors when setting the hwpstate verbose systctl on: > > May 17 22:28:32 Alice kernel: hwpstate0: set freq failed, err 6 May > 17 22:28:32 Alice kernel: hwpstate0: setting P1-state on cpu0 May > 17 22:28:32 Alice kernel: hwpstate0: result P1-state on cpu0 May > 17 22:28:32 Alice kernel: hwpstate0: setting P1-state on cpu1 May > 17 22:28:32 Alice kernel: hwpstate0: result P1-state on cpu1 May > 17 22:28:32 Alice kernel: hwpstate0: setting P1-state on cpu2 May > 17 22:28:32 Alice kernel: hwpstate0: result P1-state on cpu2 May > 17 22:28:32 Alice kernel: hwpstate0: setting P1-state on cpu3 May > 17 22:28:32 Alice kernel: hwpstate0: result P1-state on cpu3 May > 17 22:28:32 Alice kernel: hwpstate0: setting P1-state on cpu4 May > 17 22:28:32 Alice kernel: hwpstate0: result P1-state on cpu4 May > 17 22:28:32 Alice kernel: hwpstate0: setting P1-state on cpu5 May > 17 22:28:32 Alice kernel: hwpstate0: result P1-state on cpu5 May > 17 22:28:32 Alice kernel: hwpstate0: setting P1-state on cpu6 May > 17 22:28:32 Alice kernel: hwpstate0: result P1-state on cpu6 May > 17 22:28:32 Alice kernel: hwpstate0: setting P1-state on cpu7 May > 17 22:28:32 Alice kernel: hwpstate0: result P1-state on cpu7 > > > Sean > > ref this machine: http://forums.pcbsd.org/showthread.php?t=16733I briefly looked at the dmesg. You have a Family 15h processor (Bulldozer with Turbo Core) and I believe it isn't supported (yet). It won't be too hard to implement, though. Jung-uk Kim -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (FreeBSD) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk+2dlkACgkQmlay1b9qnVOhzwCfRi7ZqdpQ8yH/N3IHqCFl/586 XdoAnRLeenDvgQKvB6BIjyMsoskXcSNs =hGYz -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 2012-05-25 11:47:53 -0400, Andriy Gapon wrote:> on 25/05/2012 17:36 Yamagi Burmeister said the following: >> Hello, a user at BSDForen.de had the same problem and I helped >> him to track it down. While I was unable to find a solution I >> found the cause of the problems. The problem is (all files and >> line numbers relative to FreeBSD 9.0-RELEASE): >> >> 1. When a new p-state is requested (by powerd or by changing the >> sysctl) in kern/kern_cpu.c the function cf_set_method() is >> invoked. 2. In line 335 the driver depended function is called. >> For newer AMD CPU it's hwpstate_set() in x86/cpufreq/hwpstate.c >> 3. In x86/cpufreq/hwpstate.c line 227 hwpstate_goto_pstate() is >> called which does the actual magic. 4. In line 199 "if (msr !>> id)" triggers, returns ENXIO. The error is send back to >> cf_set_method(), which prints the "hwpstate0: set freq failed". >> powerd translates the error to "Device not configured" >> >> After some further investigation and looking at the linux driver >> [0] I changed the loop at x86/cpufreq/hwpstate.c 188ff from 100 >> iterations to 250. It lessend the problem but didn't solve it. >> The next step was to rewrite the logic between line 183 and 203 >> and adding a lot of debug printf. The patch (non style(9) >> compliant) is attached. With the new logic every 100 usec the new >> p-state is set again, until it's accepted. After 100 tries ENXIO >> is returned. >> >> This lessend the problem even more and showed that - On an old >> Phenom II X4 940 (K10 / Deneb) the new p-state is always accepted >> at the first try. At a test run for about 3 hours there was not a >> single failure. - On the Bulldozer CPU in about 9 in 10 times the >> new p-state is accepted at the first try. At most other times the >> new p-state is accepted after 1 to 10 tries. And there is a >> ~0,25% chance that the new p-state is never accepted, leading to >> "hwpstate0: set freq failed". At the next call to cf_set_method() >> (about 500ms to 1s later) the new p-state it's most likely set >> successfull. This can be seen at the log (full log attached): >> >> # First call, failed after 102 iterations hwpstate0: MSR: 0 ID: >> 1 hwpstate0: Setting failed! hwpstate0: Iterations: 102 >> >> # Second call, successfull hwpstate0: setting P1-state on cpu1 >> hwpstate0: MSR: 1 ID: 1 hwpstate0: Iterations: 1 >> >> So the big question is: Why is the new p-state sometimes not >> accepted? And why does this only happen on Bulldozer CPUs and not >> at the old K10 (Barcelona, Deneb), etc? Reading the "BIOS and >> kernel developer guide" for Bulldozer didn't show anything, but I >> may have overlooked it. One solution may be to change hwpstate >> not to set p-states but "Frequency IDs" (FID) and "Voltage ID" >> (VID) like the linux driver does. > > I think that you misread their code a little bit. The vid/fid > transition is used for K8 processors, for newer ones they do the > same P-state transitioning. The secret of their success seems to be > that they just write the MSR without any post-write checks. > > As to your questions about hardware behavior - yes, they are quite > interesting, but perhaps irrelevant. The BKDG never specifies the > OS P-state transition command sequence and timings, it just says > "write the MSR" (exactly what Linux does). There could be > different reasons why a core could be in a different P-state > (mostly I suspect interactions between cores in a Bulldozer compute > unit). When BKDG does specify the P-state transitioning sequence > (for BIOS) it suggest a different one from what we do - first write > the MSR on al processors, only then check the result (and the way > of checking the result is also a bit different - using MSR xxx71).I just looked through the BKDG and I think you should definitely check MSRC001_0071[18:16]. MSRC001_0063[2:0] is "SharedC" but MSRC001_0062[2:0] and MSRC001_0071[18:15] are "Not-same-for-all". I think this means writing a P-state to MSRC001_0062[2:0] will be reflected in MSRC001_0070[18:16] first, then MSRC001_0071[18:16] gets updated when the P-state transition is complete. MSRC001_0063[2:0] will only change when all cores in a compute unit is in sync., which may be too late.> But as I've said, these details are probably not really useful in > practice. We should just quit worrying and double-checking the > hardware :-)I think we should check. Jung-uk Kim>> 0: >> http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=blob;f=drivers/cpufreq/powernow-k8.c;h=c0e816468e300f242735f4825d09b9d291a9b522;hb=HEAD >> >> >>1:>> http://support.amd.com/us/Processor_TechDocs/42301_15h_Mod_00h-0Fh_BKDG.pdf-----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (FreeBSD) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk+/vTkACgkQmlay1b9qnVPY9QCgqcEvUHKKQ3U0Rec5Kzdlrw3L kSkAnj6ofOf8PVkEHlxNrgGZAHJ2so1p =GQw9 -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 2012-05-25 13:11:21 -0400, Jung-uk Kim wrote:> I just looked through the BKDG and I think you should definitely > check MSRC001_0071[18:16]. MSRC001_0063[2:0] is "SharedC" but > MSRC001_0062[2:0] and MSRC001_0071[18:15] are "Not-same-for-all".^^ 16 Sorry for the typo. Jung-uk Kim -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (FreeBSD) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk+/vb4ACgkQmlay1b9qnVOjPACbBIyx9eHTXqDqJjeAQpamAfPT 9+oAoL99VMf9MAhspIBJGvjFBG2aLynE =4Z9b -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 2012-05-25 15:39:39 -0400, Andriy Gapon wrote:> on 25/05/2012 20:11 Jung-uk Kim said the following: >> I just looked through the BKDG and I think you should definitely >> check MSRC001_0071[18:16]. MSRC001_0063[2:0] is "SharedC" but >> MSRC001_0062[2:0] and MSRC001_0071[18:15] are "Not-same-for-all". >> I think this means writing a P-state to MSRC001_0062[2:0] will be >> reflected in MSRC001_0070[18:16] first, then MSRC001_0071[18:16] >> gets updated when the P-state transition is complete. >> MSRC001_0063[2:0] will only change when all cores in a compute >> unit is in sync., which may be too late. >> > [snip] >> I think we should check. > > Jung-uk, > > if we decide so, then I think that we could still keep the things > "simple". As we currently use the "wholesale" approach (all CPUs > are set to the same P-state regardless of topology), then we could > first make a pass of writing the MSR on all processors with a new > P-state value and then make another pass of checking via MSR > C001_0063 that the P-state is acquired.No, I believe checking MSRC001_0071[18:16] is much simpler if it works. And it does not break current cpufreq(4) design principles. Jung-uk Kim -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (FreeBSD) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk+/5iQACgkQmlay1b9qnVOHLgCfY0ELt5oN1hml8S+bDGSHbOux bj4AoKisSh9DlK46U+LFthaSGicp/+Hc =BYej -----END PGP SIGNATURE-----
On Sat, 26 May 2012 12:34:25 +0300 Andriy Gapon <avg@FreeBSD.org> wrote:> >>> if we decide so, then I think that we could still keep the things > >>> "simple". As we currently use the "wholesale" approach (all CPUs are > >>> set to the same P-state regardless of topology), then we could first > >>> make a pass of writing the MSR on all processors with a new P-state > >>> value and then make another pass of checking via MSR C001_0063 that the > >>> P-state is acquired. > >> > >> No, I believe checking MSRC001_0071[18:16] is much simpler if it works. > >> And it does not break current cpufreq(4) design principles.One potential problem with MSRC001_0071[18:16] is, that it's on older CPUs supported by hwpstate it's the same as C001_0063. Only on newer models with "turbo" it containts the actual "hardware p-state". So additional logic would be required: 1. Set new p-state 2. Check CPUID for support of "hardware p-states" 3.1 If yes, read MSRC001_0071[18:16] and convert the "hardware p-state" into a "software p-state" 3.2 If not, just read MSRC001_0071[18:16] 4. Compare read (and converted) p-state to the requested p-state I don't think that it's worth this additional efford. The solution suggest by Andriy Gapon is trivial, works fine and is supported by all CPUs supported by hwpstate.> I believe the approach that I suggested to be so trivial to implement (and also > correct) that here is a patch: > > diff --git a/sys/x86/cpufreq/hwpstate.c b/sys/x86/cpufreq/hwpstate.c > index 40e1943..9c17a41 100644 > --- a/sys/x86/cpufreq/hwpstate.c > +++ b/sys/x86/cpufreq/hwpstate.c > @@ -186,16 +186,21 @@ hwpstate_goto_pstate(device_t dev, int pstate) > id, PCPU_GET(cpuid)); > /* Go To Px-state */ > wrmsr(MSR_AMD_10H_11H_CONTROL, id); > + } > + CPU_FOREACH(i) { > + /* Bind to each cpu. */ > + thread_lock(curthread); > + sched_bind(curthread, i); > + thread_unlock(curthread); > /* wait loop (100*100 usec is enough ?) */ > for(j = 0; j < 100; j++){ > + /* get the result. not assure msr=id */ > msr = rdmsr(MSR_AMD_10H_11H_STATUS); > if(msr == id){ > break; > } > DELAY(100); > } > - /* get the result. not assure msr=id */ > - msr = rdmsr(MSR_AMD_10H_11H_STATUS); > HWPSTATE_DEBUG(dev, "result P%d-state on cpu%d\n", > (int)msr, PCPU_GET(cpuid)); > if (msr != id) {I can confirm, that this patchs works on a "Bulldozer" CPU and on an old Phenom II "Deneb". -- Homepage: www.yamagi.org XMPP: yamagi@yamagi.org GnuPG/GPG: 0xEFBCCBCB -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 196 bytes Desc: not available Url : http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20120529/f4826c1c/attachment.pgp
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 2012-05-26 05:34:25 -0400, Andriy Gapon wrote:> on 26/05/2012 10:02 Yamagi Burmeister said the following: >> On Fri, 25 May 2012 16:05:56 -0400 Jung-uk Kim <jkim@FreeBSD.org> >> wrote: >> >>>> if we decide so, then I think that we could still keep the >>>> things "simple". As we currently use the "wholesale" approach >>>> (all CPUs are set to the same P-state regardless of >>>> topology), then we could first make a pass of writing the MSR >>>> on all processors with a new P-state value and then make >>>> another pass of checking via MSR C001_0063 that the P-state >>>> is acquired. >>> >>> No, I believe checking MSRC001_0071[18:16] is much simpler if >>> it works. And it does not break current cpufreq(4) design >>> principles. >> >> Okay, thank's for your input. I'll come up with a patch. But it >> won't happen until tuesday or wednesday next week. >> > > I believe the approach that I suggested to be so trivial to > implement (and also correct) that here is a patch:... It is simple but I don't like locking scheduler, binding CPU, and writing the same MSR, multiple times for each core. Besides, it introduces more delay and you may be reading the correct status because of that. :-P If people really think checking MSRC001_0071[18:16] is unworthy for Bulldozer, I prefer skipping status check but I disagree with this patch. Thanks, Jung-uk Kim -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (FreeBSD) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk/H1GQACgkQmlay1b9qnVM+RQCfaYl6LpyARoO2oiyimwrWrhXD BPoAoIna4GHZKlsCRaXV3jqH8ujpzur5 =NYS0 -----END PGP SIGNATURE-----