freebsd stable - Sep 2015 - process scheduling and cpuset

> On 13 ????. 2015 ?., at 16:09, Slawa Olhovchenkov <slw at zxy.spb.ru>
wrote:
> 
> On Sun, Sep 13, 2015 at 02:52:08PM +0300, Dmitry Sivachenko wrote:
> 
>> Hello,
>> 
>> I have 32 processor machine (2x CPU E5-2650) running several CPU-bound
processes (ULE scheduler).
>> 3 processes are 32-threaded, and 8 are single threaded.
>> 
>> I bind all 3 32-threaded processes to CPUs 0-24 (cpuset -C -l 0-24 -p
XXX).
>> 
>> I expect that the remaining 8 single-threaded processes will (mostly)
run on the remaining 25-31 CPU cores and use (almost) 100% cpu each.
>> 
>> But this is not the case (according to top(1)):  they spend a lot of
time on 0-24 CPUs and CPU Idle time is about 10%.
>> 
>> These are all purely computational programs, in idle system
single-threaded programs steadily consume 100% of a core, and 32-threaded
programs consume all 32 cores and idle time is zero.
>> 
>> Is it an ULE scheduler feature or am I doing something wrong?
>> 
>> The goal is to give a single-threaded program a chance to run when
somebody started several 32-threaded processes.
> 
> You don't have 32 processor machine, you have only 16 processor
> machine.
> SMT/hyperthreading don't give real processor, SMT "CPU" have
> unpredicable power and his load depend on load parent CPU.
> 
> For example, for my case I see such condition (simpliy) on CPU 0 and 1
> (SMT of one real core) with rise load:
> 
> load 0.1  0.1
> load 0.2  0.2
> load 0.3  0.3
> load 0.4  0.4
> load 0.45 0.45
> load 0.48 0.48
> load 1.00 1.00\

Yes I know about HT.  But how does this explain why I have 10% of CPU idle?

If I explicitly bind my single-threaded processes to the remaining CPU cores
(25-32), they start to receive expected 100% of CPU and overall Idle decreases.

I just expect scheduler to do the same for me.

On Sun, Sep 13, 2015 at 04:44:40PM +0300, Dmitry Sivachenko wrote:
> 
> > On 13 ????. 2015 ?., at 16:09, Slawa Olhovchenkov <slw at
zxy.spb.ru> wrote:
> > 
> > On Sun, Sep 13, 2015 at 02:52:08PM +0300, Dmitry Sivachenko wrote:
> > 
> >> Hello,
> >> 
> >> I have 32 processor machine (2x CPU E5-2650) running several
CPU-bound processes (ULE scheduler).
> >> 3 processes are 32-threaded, and 8 are single threaded.
> >> 
> >> I bind all 3 32-threaded processes to CPUs 0-24 (cpuset -C -l 0-24
-p XXX).
> >> 
> >> I expect that the remaining 8 single-threaded processes will
(mostly) run on the remaining 25-31 CPU cores and use (almost) 100% cpu each.
> >> 
> >> But this is not the case (according to top(1)):  they spend a lot
of time on 0-24 CPUs and CPU Idle time is about 10%.
> >> 
> >> These are all purely computational programs, in idle system
single-threaded programs steadily consume 100% of a core, and 32-threaded
programs consume all 32 cores and idle time is zero.
> >> 
> >> Is it an ULE scheduler feature or am I doing something wrong?
> >> 
> >> The goal is to give a single-threaded program a chance to run when
somebody started several 32-threaded processes.
> > 
> > You don't have 32 processor machine, you have only 16 processor
> > machine.
> > SMT/hyperthreading don't give real processor, SMT "CPU"
have
> > unpredicable power and his load depend on load parent CPU.
> > 
> > For example, for my case I see such condition (simpliy) on CPU 0 and 1
> > (SMT of one real core) with rise load:
> > 
> > load 0.1  0.1
> > load 0.2  0.2
> > load 0.3  0.3
> > load 0.4  0.4
> > load 0.45 0.45
> > load 0.48 0.48
> > load 1.00 1.00\
> 
> 
> Yes I know about HT.  But how does this explain why I have 10% of CPU idle?
> 
> If I explicitly bind my single-threaded processes to the remaining CPU
cores (25-32), they start to receive expected 100% of CPU and overall Idle
decreases.
> 
> I just expect scheduler to do the same for me.
> 
Idle is not goal, goal is lessing task executing time.