Hi, On Sun, 21 Aug 2016 15:21:01 +1000 Kubilay Kocak <koobs at FreeBSD.org> wrote:> On 19/08/2016 9:34 AM, Erich Dollansky wrote: > > > > I am sure that some know of this site: > > > > http://www.phoronix.com/scan.php?page=article&item=2bsd-7linux-bench&num=4 > > > > I wonder about the results for FreeBSD. As I do not have 11 on my > > machines, a stupid question. Are there still some debugging aids > > enabled in 11? > > They're off in those versions, but did note compiler (and compiler > args) differences between within most tests (See attachments) as you > mentioned. >the benchmark then compares the off-the-shelve distributions.> > > I know that some of the results are caused by the use of CLang and > > some of the results test applications/compilers and not operating > > systems. > > gcc/clang tests and defaults in upstream build systems are almost > certainly contributors. At a minimum it would be nice to see an > attempt to standardise (force) compiler args across all OS runs for > the same test, even if this doesn't prove to be perfect.Yes, without, he compares compilers more than operating systems.> > Separating or adding tests for the same tests using non-default > compilers (in particular latest GCC versions from ports) so they match > across OS's would also be valuable. > > At a minimum it would be worth Michael highlighting the differences, > and ideally removing these variables from the tests, even if they > aren't default configurations. Though a test of out of the box > configurations is still valuable, it can serve to muddy the > underlying differences and make them tougher to isolate.He never does this. I have no idea what he wants to achieve with his benchmarking all the while as the results are very difficult to compare with real life workloads. Erich
Unfortunately people (customers, developers, hardward vendors) make decisions on the basis of bang-for-buck. FreeBSD is consistently underperforming on benchmarks. And regardless of real-world similarity, the contrived benchmarks are the best that is used. If clang (v3.4.1 on 10.3 Stable) really is the cause, then there should be an option to use "whatever" will let the source perform at its best and reflect the stability that we have with FreeBSD. We can criticise the test results or the test regime, but unless knowledgable people respond publicly and/or in the phoronix forums, such as https://www.phoronix.com/forums/forum/software/distributions/891660-benchmarks-2-bsds-vs-7-linux-distributions Then this interpretation of reality will be fixed in decision-makers' minds and consequently the uptake (and support) of FreeBSD. As I recall Michael's testing regime has been consistently criticised (by FreeBSD folks) since he started his website, unfortunately when someone is making public statements and comparisons we really need to have a relationship with them to ensure that FreeBSD's best position is put forward.
Il 2016-08-21 08:45 Erich Dollansky ha scritto:> I am sure that some know of this site: > > http://www.phoronix.com/scan.php?page=article&item=2bsd-7linux-bench&num=4 > > I wonder about the results for FreeBSD. As I do not have 11 on my > machines, a stupid question. Are there still some debugging aids > enabled in 11? > They're off in those versions, but did note compiler (and compiler > args) differences between within most tests (See attachments) as you > mentioned.the benchmark then compares the off-the-shelve distributions. Excuse me, as a casual reader of the list, I don't get this "critique". I never recompile my installations, I just use them from the installation CD (as probably 90% of the rest of the world), so I don't get what is wrong with the approach of comparing an out-of-the-box FreeBSD 11 with an out-of-the-box Ubuntu whatever. If FreeBSD 11 "out-the-box" performs slow because the standard compilers options aren't good it's not a problem with the benchmarking platform but with the default CD compiling options. Am I getting it wrong? Thanks :) --- Andrea Brancatelli Schema31 S.p.a. Responsabile IT ROMA - BO - FI - PA ITALY Tel: +39.06.98.358.472 Cell: +39.331.2488468 Fax: +39.055.71.880.466 Societ? del Gruppo SC31 ITALIA