Maxim Sobolev
2009-Feb-22 22:20 UTC
The machdep.hyperthreading_allowed & ULE weirdness in 7.1
Hi Jeff, I have a single-CPU system with P4 HTT-enabled processor (7.1-RELEASE-p3), kernel compiled with SCHED_ULE. Copyright (c) 1992-2009 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 7.1-RELEASE-p3 #0: Fri Feb 20 09:53:32 UTC 2009 root@foo.com:/usr/obj/usr/src/sys/SSP-PRODUCTION7 Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Intel(R) Pentium(R) 4 CPU 3.00GHz (2992.51-MHz 686-class CPU) Origin = "GenuineIntel" Id = 0xf34 Stepping = 4 Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> Features2=0x441d<SSE3,DTES64,MON,DS_CPL,CNXT-ID,xTPR> Logical CPUs per core: 2 real memory = 3758030848 (3583 MB) avail memory = 3674083328 (3503 MB) ACPI APIC Table: <IntelR AWRDACPI> FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 ioapic0: Changing APIC ID to 2 ioapic0 <Version 2.0> irqs 0-23 on motherboard ioapic1 <Version 2.0> irqs 24-47 on motherboard machdep.cpu_idle_hlt: 1 machdep.hlt_cpus: 0 machdep.hlt_logical_cpus: 0 machdep.logical_cpus_mask: 2 If I flip machdep.hyperthreading_allowed sysctl, I still can see processes scheduled to both logical CPUs, yet, if I check IRQ status in the systat -vm, the CPU1 is not getting any timer interrupts, while CPU0 gets twice the normal amount. I wonder if something is broken - I would expect no processes to be scheduled to the CPU1. machdep.hyperthreading_allowed=1: top: http://sobomax.sippysoft.com/~sobomax/ScreenShot458.png systat -vm: http://sobomax.sippysoft.com/~sobomax/ScreenShot459.png machdep.hyperthreading_allowed=0: top: http://sobomax.sippysoft.com/~sobomax/ScreenShot460.png systat -vm: http://sobomax.sippysoft.com/~sobomax/ScreenShot461.png Please let me know if any other debug information is needed. -Maxim
Robert Watson
2009-Feb-23 02:08 UTC
The machdep.hyperthreading_allowed & ULE weirdness in 7.1
On Sun, 22 Feb 2009, Maxim Sobolev wrote:> Hi Jeff, > > I have a single-CPU system with P4 HTT-enabled processor (7.1-RELEASE-p3), > kernel compiled with SCHED_ULE.This is because machdep.hlt_logical_cpus doesn't do what you think it does. It causes HTT cores to invoke the hlt instruction in their idle loop, causing them to sleep until they receive an interrupt. For 4BSD, which uses a "pull" model (on the whole) to bring work from work queues, this means that CPUs will go to sleep and remain that way unless they're actively receiving interrupts. For ULE, which uses "push" as well as "pull", threads will constantly be being shed from too-busy CPUs to apparently idle ones under load. The only reliable way to disable hyperthreading is to do so using your BIOS setting, or use loader.conf to disable probing of the pics on unwanted cores, which will cause the CPUs not to be enumerated and hence not used. We don't support taking CPUs fully offline with any scheduler, although you can use the cpuset facility to prevent threads from running on specific cores. You could imagine teaching ULE (and presumably 4BSD) about policies such as "Don't use HTT cores", or perhaps just cpuset about those policies. Robert N M Watson Computer Laboratory University of Cambridge> > Copyright (c) 1992-2009 The FreeBSD Project. > Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 > The Regents of the University of California. All rights reserved. > FreeBSD is a registered trademark of The FreeBSD Foundation. > FreeBSD 7.1-RELEASE-p3 #0: Fri Feb 20 09:53:32 UTC 2009 > root@foo.com:/usr/obj/usr/src/sys/SSP-PRODUCTION7 > Timecounter "i8254" frequency 1193182 Hz quality 0 > CPU: Intel(R) Pentium(R) 4 CPU 3.00GHz (2992.51-MHz 686-class CPU) > Origin = "GenuineIntel" Id = 0xf34 Stepping = 4 > > Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> > Features2=0x441d<SSE3,DTES64,MON,DS_CPL,CNXT-ID,xTPR> > Logical CPUs per core: 2 > real memory = 3758030848 (3583 MB) > avail memory = 3674083328 (3503 MB) > ACPI APIC Table: <IntelR AWRDACPI> > FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs > cpu0 (BSP): APIC ID: 0 > cpu1 (AP): APIC ID: 1 > ioapic0: Changing APIC ID to 2 > ioapic0 <Version 2.0> irqs 0-23 on motherboard > ioapic1 <Version 2.0> irqs 24-47 on motherboard > > machdep.cpu_idle_hlt: 1 > machdep.hlt_cpus: 0 > machdep.hlt_logical_cpus: 0 > machdep.logical_cpus_mask: 2 > > If I flip machdep.hyperthreading_allowed sysctl, I still can see processes > scheduled to both logical CPUs, yet, if I check IRQ status in the systat -vm, > the CPU1 is not getting any timer interrupts, while CPU0 gets twice the > normal amount. I wonder if something is broken - I would expect no processes > to be scheduled to the CPU1. > > machdep.hyperthreading_allowed=1: > top: http://sobomax.sippysoft.com/~sobomax/ScreenShot458.png > systat -vm: http://sobomax.sippysoft.com/~sobomax/ScreenShot459.png > > machdep.hyperthreading_allowed=0: > top: http://sobomax.sippysoft.com/~sobomax/ScreenShot460.png > systat -vm: http://sobomax.sippysoft.com/~sobomax/ScreenShot461.png > > Please let me know if any other debug information is needed. > > -Maxim > _______________________________________________ > freebsd-current@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org" >
Maxim Sobolev
2009-Feb-23 10:40 UTC
The machdep.hyperthreading_allowed & ULE weirdness in 7.1
Robert Watson wrote:> In the mean time, it sounds like the sysctl does need to be > reimplemented or removed, but one question is how far to take it -- > caches are shared to varying degrees at varying levels of the topology. > However, I believe the recommendation has generally moved to disabling > hyperthreading using the BIOS, as that uses the vendor's notion of > hyperthreading. The idea of changing the setting at run-time is > currently untenable because we don't have the OS infrastructure to take > CPUs out of service, although growing it would be useful in order to > support virtual machine dynamic CPU reconfiguration.Well, as far as I know, what SCHED_4BSD does is simply stopping scheduling threads to the logical core(s). One doesn't need infrastructure to take CPU off-line for doing the same in SCHED_ULE. Unfortunately access to BIOS is not always an option and also some BIOSes don't even provide a feature to turn HTT off. -Maxim
Apparently Analagous Threads
- cvs commit: src/sys/amd64/amd64 mp_machdep.c src/sys/i386/i386 mp_machdep.c
- FreeBSD Security Advisory FreeBSD-SA-05:09.htt [REVISED]
- Panic : bad pte
- [Bug 1103] New option for ssh(1) to make it failing when remote port forwarding can't be established
- OpenSSH slow on OSX High Sierra (maybe due to libcrypto)?