freebsd stable - Sep 2015 - process scheduling and cpuset

Hello,

I have 32 processor machine (2x CPU E5-2650) running several CPU-bound processes
(ULE scheduler).
3 processes are 32-threaded, and 8 are single threaded.

I bind all 3 32-threaded processes to CPUs 0-24 (cpuset -C -l 0-24 -p XXX).

I expect that the remaining 8 single-threaded processes will (mostly) run on the
remaining 25-31 CPU cores and use (almost) 100% cpu each.

But this is not the case (according to top(1)):  they spend a lot of time on
0-24 CPUs and CPU Idle time is about 10%.

These are all purely computational programs, in idle system single-threaded
programs steadily consume 100% of a core, and 32-threaded programs consume all
32 cores and idle time is zero.

Is it an ULE scheduler feature or am I doing something wrong?

The goal is to give a single-threaded program a chance to run when somebody
started several 32-threaded processes.

Thanks!

On Sun, Sep 13, 2015 at 02:52:08PM +0300, Dmitry Sivachenko wrote:
> Hello,
> 
> I have 32 processor machine (2x CPU E5-2650) running several CPU-bound
processes (ULE scheduler).
> 3 processes are 32-threaded, and 8 are single threaded.
> 
> I bind all 3 32-threaded processes to CPUs 0-24 (cpuset -C -l 0-24 -p XXX).
> 
> I expect that the remaining 8 single-threaded processes will (mostly) run
on the remaining 25-31 CPU cores and use (almost) 100% cpu each.
> 
> But this is not the case (according to top(1)):  they spend a lot of time
on 0-24 CPUs and CPU Idle time is about 10%.
> 
> These are all purely computational programs, in idle system single-threaded
programs steadily consume 100% of a core, and 32-threaded programs consume all
32 cores and idle time is zero.
> 
> Is it an ULE scheduler feature or am I doing something wrong?
> 
> The goal is to give a single-threaded program a chance to run when somebody
started several 32-threaded processes.
You don't have 32 processor machine, you have only 16 processor
machine.
SMT/hyperthreading don't give real processor, SMT "CPU" have
unpredicable power and his load depend on load parent CPU.

For example, for my case I see such condition (simpliy) on CPU 0 and 1
(SMT of one real core) with rise load:

load 0.1  0.1
load 0.2  0.2
load 0.3  0.3
load 0.4  0.4
load 0.45 0.45
load 0.48 0.48
load 1.00 1.00