freebsd stable - Oct 2007 - ULE vs. 4BSD in RELENG_7

All,

I mentioned this on another thread, but I think it deserves a separate
thread. Not only so it will get its own attention, but also so I don't
hijack the other thread.

I have noticed some performance discrepancies with ULE and 4BSD in
RELENG_7, specifically with ffmpeg. I have all the kernel debugging
options disabled, and as I understand it, the userland debugging is
all off by default in RELENG_7.

That said, I'm seeing 5% better performance from the 4BSD scheduler
for ffmpeg -threads 8 on a quad-core Intel desktop setup (Q6600 cpu).
I am also comparing as a baseline to RELENG_6_2 (I only have 4BSD
scheduler numbers for that, though). Here are the run times for the
selected benchmark:

RELENG_6_2 (4BSD):   1:32.39
RELENG_7 (4BSD):       1:32.44
RELENG_7 (ULE):         1:37.15

Difference (4BSD vs. ULE on 7):  5.095 %

Please see below for "backup data" on this (vmstat 5 output during
both a 4BSD and ULE run).

As an additional data point, I went ahead and wrote a perl script that
forks 4 simultaneous ffmpeg processes (without -threads, obviously).
Here are the results (much closer, but still 4BSD is faster):

RELENG_7 (4BSD):       7:20.81
RELENG_7 (ULE):         7:22.63

This is only a 1% difference, but 4BSD is still faster. So, am I doing
something wrong here? Are there additional kernel options/settings I
need to tweak ULE properly? Or is this particular workload simply
better on 4BSD?

Thanks!
Josh


vmstat 5 output during ULE/4BSD runs with ffmpeg ... -threads 8:

4BSD:
 procs      memory      page                    disks     faults      cpu
 r b w     avm    fre   flt  re  pi  po    fr  sr ad6 ad8   in   sy
cs us sy id
 1 1 0  412348 1589544   508   0   0   0   507   0   0   0   22  694
643  3  0 97
 3 1 0  754860 1416760  8504   0   0   0    20   0   0   0   15 1339
2092 94  1  5
 6 1 0  757940 1402456   565   0   0   0     1   0   0   0   18 1081
944 98  0  2
 6 1 0  757940 1396564   188   0   0   0     3   0  15   0   45 1136
1075 96  0  4
 7 1 0  757940 1393236    30   0   0   0     1   0   1   0   21  937
851 97  0  3
 6 2 0  760192 1390756    60   0   0   1    43   0   5   0   25 1065
902 96  0  3
 5 2 0  760192 1390304     3   0   0   0   100   0   0   0   19 1365
1240 98  0  2
 4 1 0  758964 1388592    25   0   0   0    16   0   0   0   15  997
869 96  0  4
 7 1 0  758964 1386132     3   0   0   0     0   0   0   0   14  810
766 96  0  4
 5 1 0  758964 1383608     2   0   0   0     0   0   0   0   28  695
713 99  0  1
 4 1 0  758964 1380372    26   0   0   0     1   0   7   0   29  806
842 96  0  3
 7 1 0  759992 1377440    11   0   0   0     1   0   1   0   29  810
802 98  0  2
 3 1 0  759992 1375412     2   0   0   0     2   0   0   0   20  879
785 97  0  3
 4 1 0  759992 1373168     1   0   0   0     1   0   0   0   13 1032
876 95  0  5
 5 1 0  759992 1371204     2   0   0   0     0   0   0   0   21 1197
984 85  0 15
 5 1 0  759992 1367676     3   0   0   0     0   0   0   0   20 1155
897 98  0  1
 4 1 0  759992 1362272    24   0   0   0     0   0   0   0   21  826
719 98  0  2
 6 2 0  762244 1359028    58   0   0   0    43   0   1   0   23 1404
1071 96  0  4
 procs      memory      page                    disks     faults      cpu
 r b w     avm    fre   flt  re  pi  po    fr  sr ad6 ad8   in   sy
cs us sy id
 7 2 0  762244 1356036     7   0   0   0     0   0   0   0   21 1236
959 99  0  1


ULE:
 procs      memory      page                    disks     faults      cpu
 r b w     avm    fre   flt  re  pi  po    fr  sr ad6 ad8   in   sy
cs us sy id
 0 4 0  283788 1781208  2333   7  17   0  1633   0   0   0  211 4057
1702  1  1 99
 6 2 0  631152 1606120  8180   5  23   0    30   0  65   0  157  915
371336 79  0 21
 6 2 0  634228 1583908   906   0   0   0     3   0   5   0   32  566
940216 93  0  7
 6 2 0  634228 1575052   227   0   0   0     0   0   1   0   96  915
694291 94  0  6
 5 2 0  636276 1569768    54   0   0   0     0   0   1   0   79  722
984529 93  0  7
 6 2 0  636276 1565184     3   0   0   0     0   0   0   0   56  749
615641 95  0  5
 6 2 0  633024 1560904   182   0   0   0   224   0  15   0   64  892
554573 95  0  5
 6 2 0  633024 1557648     1   0   0   0     0   0   0   0   21 1046
925070 94  0  6
 8 1 0  631800 1553088    27   0   0   0    16   0   0   0   23  773
421820 96  0  3
 7 1 0  631800 1548752     3   0   0   0     0   0   0   0   27  552
714116 96  0  4
 7 1 0  631800 1543804    20   0   0   0     0   0   0   0   20  541
887702 94  0  6
 6 1 0  632824 1537948    15   0   0   0     1   0   1   0   25  590
691248 95  0  5
 7 1 0  632824 1533896     1   0   0   0     4   0   1   0   21  676
545389 95  0  5
 7 1 0  632824 1528332     2   0   0   0     0   0   0   0   23  787
637327 93  0  7
 5 1 0  632824 1525668     1   0   0   0     0   0   0   0   13  893
722130 90  0 10
 3 1 0  632824 1520008     3   0   0   0     0   0   0   0   25  897
1216075 81  0 19
 6 1 0  632824 1510980    25   0   0   0     0   0   0   0   33  932
1126821 90  0 10
 7 1 0  632824 1503956     1   0   0   0     3   0   0   0   29  804
961332 93  0  7
 procs      memory      page                    disks     faults      cpu
 r b w     avm    fre   flt  re  pi  po    fr  sr ad6 ad8   in   sy
cs us sy id
 4 2 0  635076 1498816    59   0   0   0    43   0   0   0   24 1014
1188872 91  0  9
 7 2 0  635076 1492764     7   0   0   0     0   0   1   0   28 1283
1085336 88  0 11

> I have noticed some performance discrepancies with ULE and 4BSD in
> RELENG_7, specifically with ffmpeg. I have all the kernel debugging
> options disabled, and as I understand it, the userland debugging is
> all off by default in RELENG_7.
Here are a couple of additional benchmarks comparing the schedulers on
my system:

make -j8 -DNOCLEAN buildkernel
4BSD: 3:25.56
ULE: 3:39.20
Difference: -6.6 %

ubench (CPU):
4BSD: 1705258
ULE: 1713510
Difference: +0.48 %

super-smack (select-key 10 10000):
4BSD: 55044.38
ULE: 68085.21
Difference: +23.69 %

super-smack (update-select 10 10000):
4BSD: 16734.15
ULE: 17631.43
Difference: +5.36 %

So at least for the MySQL super-smack benchmark (I know it's a rather
contrived benchmark), ULE is significantly faster for select-key and a
decent improvement for update-select. ubench is about the same, but
building a kernel is also slower with ULE.

Was ULE tuned with MySQL in mind, without considering other workloads?
Are there other benchmarks for "real" workloads I can run to compare
(e.g. Apache benchmarks, etc)? I'd like to help in any way I can, so
folks can choose the right scheduler for their usage model.

Regards,
Josh