I will need to build several Web caches over the next few months, and just took advantage of the Christmas lull (and a snowy day, when I couldn't work outside) to test FreeBSD 7.0 BETA 4 to see how it will perform at this task. I built up a 4 core FreeBSD box, and asked a friend who's a Linux fanatic to do the same with Linux on identical hardware. I didn't watch closely how he installed everything, but asked him not to tune it beyond setting it up properly for SMP. We then ran a test suite in which a client starts several processes. Each uses wget to fetch a series of objects in rapid succession via the cache. The fetches done by each process are the same batch of URLS, but shuffled differently, so each URL will get a miss the first time and then hits each time it comes up thereafter unless the cache overflows. We're doing all GETs, with no tricky stuff like subranges. As has been reported in some other messages on this list, Linux is currently blowing FreeBSD away. It's taking as much as 20% less time to get through the benchmark, depending on exactly how the random shuffle came out. This is with 4 GB RAM, the GENERIC FreeBSD SMP kernel (using SCHED_ULE), and aufs as the storage schema for Squid. It appears, though I'd need to instrument the code more to be sure, that the slowdown is coming from file I/O. Could it be that there less concurrency or more overhead in FreeBSD file operations than there is in Linux? Even with SoftUpdates turned on, the cache volume mounted with -noatime, and aufs (which uses kqueues -- a FreeBSD invention -- to optimize multithreaded disk access), the benchmark shows FreeBSD losing out. Why? --Brett Glass
> I will need to build several Web caches over the next few months, > and just took advantage of the Christmas lull (and a snowy day, > when I couldn't work outside) to test FreeBSD 7.0 BETA 4 to see how > it will perform at this task. I built up a 4 core FreeBSD box, and > asked a friend who's a Linux fanatic to do the same with Linux on > identical hardware. I didn't watch closely how he installed > everything, but asked him not to tune it beyond setting it up > properly for SMP. > > We then ran a test suite in which a client starts several > processes. Each uses wget to fetch a series of objects in rapid > succession via the cache. The fetches done by each process are the > same batch of URLS, but shuffled differently, so each URL will get > a miss the first time and then hits each time it comes up > thereafter unless the cache overflows. We're doing all GETs, with > no tricky stuff like subranges. > > As has been reported in some other messages on this list, Linux is > currently blowing FreeBSD away. It's taking as much as 20% less > time to get through the benchmark, depending on exactly how the > random shuffle came out. This is with 4 GB RAM, the GENERIC FreeBSD > SMP kernel (using SCHED_ULE), and aufs as the storage schema for Squid. > > It appears, though I'd need to instrument the code more to be sure, > that the slowdown is coming from file I/O. Could it be that there > less concurrency or more overhead in FreeBSD file operations than > there is in Linux? Even with SoftUpdates turned on, the cache > volume mounted with -noatime, and aufs (which uses kqueues -- a > FreeBSD invention -- to optimize multithreaded disk access), the > benchmark shows FreeBSD losing out. Why?I have noticed an entry in GENERIC called device cpufreq Could this have any influence on the performance (on FreeBSD)? I saw this device late in the 7.0 release-process and I since I'm accustomed to comment out any devices and options I don't need I have commented this out as well. So I haven't performed any tests with and without this device. -- regards Claus When lenity and cruelty play for a kingdom, the gentlest gamester is the soonest winner. Shakespeare
On Fri, 21.12.2007 at 22:31:24 -0700, Brett Glass wrote:> As has been reported in some other messages on this list, Linux is currently > blowing FreeBSD away. It's taking as much as 20% less time to get through > the benchmark, depending on exactly how the random shuffle came out. This is > with 4 GB RAM, the GENERIC FreeBSD SMP kernel (using SCHED_ULE), and aufs as > the storage schema for Squid.Apples and Oranges, I know, but if you're building a "simple" reverse cacheing proxy, have you considered Varnish? Would be very interessting how it would compare to a) FreeBSD+Squid b) Linux+Squid and c) Linux+Varnish. Cheers, Ulrich Spoerlein -- It is better to remain silent and be thought a fool, than to speak, and remove all doubt.
On Saturday 22 December 2007, Brett Glass wrote:> I will need to build several Web caches over the next few months, > and just took advantage of the Christmas lull (and a snowy day, > when I couldn't work outside) to test FreeBSD 7.0 BETA 4 to see how > it will perform at this task. I built up a 4 core FreeBSD box, and > asked a friend who's a Linux fanatic to do the same with Linux on > identical hardware. I didn't watch closely how he installed > everything, but asked him not to tune it beyond setting it up > properly for SMP. > > We then ran a test suite in which a client starts several > processes. Each uses wget to fetch a series of objects in rapid > succession via the cache. The fetches done by each process are the > same batch of URLS, but shuffled differently, so each URL will get > a miss the first time and then hits each time it comes up > thereafter unless the cache overflows. We're doing all GETs, with > no tricky stuff like subranges. > > As has been reported in some other messages on this list, Linux is > currently blowing FreeBSD away. It's taking as much as 20% less > time to get through the benchmark, depending on exactly how the > random shuffle came out. This is with 4 GB RAM, the GENERIC FreeBSD > SMP kernel (using SCHED_ULE), and aufs as the storage schema for Squid. > > It appears, though I'd need to instrument the code more to be sure, > that the slowdown is coming from file I/O. Could it be that there > less concurrency or more overhead in FreeBSD file operations than > there is in Linux? Even with SoftUpdates turned on, the cache > volume mounted with -noatime, and aufs (which uses kqueues -- a > FreeBSD invention -- to optimize multithreaded disk access), the > benchmark shows FreeBSD losing out. Why?Since you're using the fs as a cache, I presume it wouldn't be a big problem if the data was lost by a power outage (or crash). If so, you can try the async mount option to seriously increase fs performance. Pieter de Goeje
Brett Glass wrote:> I will need to build several Web caches over the next few months, and > just took advantage of the Christmas lull (and a snowy day, when I > couldn't work outside) to test FreeBSD 7.0 BETA 4 to see how it will > perform at this task. I built up a 4 core FreeBSD box, and asked a > friend who's a Linux fanatic to do the same with Linux on identical > hardware. I didn't watch closely how he installed everything, but asked > him not to tune it beyond setting it up properly for SMP. > > We then ran a test suite in which a client starts several processes. > Each uses wget to fetch a series of objects in rapid succession via the > cache. The fetches done by each process are the same batch of URLS, but > shuffled differently, so each URL will get a miss the first time and > then hits each time it comes up thereafter unless the cache overflows. > We're doing all GETs, with no tricky stuff like subranges. > > As has been reported in some other messages on this list, Linux is > currently blowing FreeBSD away. It's taking as much as 20% less time to > get through the benchmark, depending on exactly how the random shuffle > came out. This is with 4 GB RAM, the GENERIC FreeBSD SMP kernel (using > SCHED_ULE), and aufs as the storage schema for Squid. > > It appears, though I'd need to instrument the code more to be sure, that > the slowdown is coming from file I/O. Could it be that there less > concurrency or more overhead in FreeBSD file operations than there is in > Linux? Even with SoftUpdates turned on, the cache volume mounted with > -noatime, and aufs (which uses kqueues -- a FreeBSD invention -- to > optimize multithreaded disk access), the benchmark shows FreeBSD losing > out. Why? >Brett, There could be several problems here: 1. WITNESS, INVARIANTS, malloc debugging. Are any of these turned on for you? I don't recall if malloc debugging got turned off yet for the 7.0 snapshots. 2. Disk subsystem. What kind of disk controller are you using? Not all drivers work well in FreeBSD. Are linux and freebsd using identical hardware? 3. Directory hashing. If you're using squid, you __must__ tune the DIRHASH, otherwise you'll spend a lot of time doing pathname lookups. What filesystem is linux using? Would you mind if I logged into your test system and looked around to help diagnose the problem? Scott
It's Tue, Dec 25, 2007 at 12:00 . I'm in a small dim room with doors labeled "Dungeon" and "Forbidden". There is noise, the door marked Dungeon flies open and freebsd-stable-request@freebsd.org SHOUTS:> Date: Mon, 24 Dec 2007 08:49:36 -0700 > From: Brett Glass <brett@lariat.net> > Subject: Re: SMP on FreeBSD 6.x and 7.0: Worth doing? > To: Scott Long <scottl@samsco.org> > Cc: stable@freebsd.org> At 07:14 AM 12/24/2007, Scott Long wrote:> >Brett,> >There could be several problems here:> >1. WITNESS, INVARIANTS, malloc debugging. Are any of these > >turned on for you? I don't recall if malloc debugging got > >turned off yet for the 7.0 snapshots.> I nuked debugging when I recompiled the kernel with SCHED_ULE.> >2. Disk subsystem. What kind of disk controller are you using? > >Not all drivers work well in FreeBSD. Are linux and freebsd > >using identical hardware?> They were. The drives are SATA.That still doesn't tell us if the drives are identical in throughput. Unless the drives are the same model/manufacturer, you should go to the manufacturers web-site for both drives, and look for the technical specs. THen you need to look for the speed of data transfer from the platter to the internal memory. I've seen [ in the past ] drives with lower revolutions per minute out-perform faster rotating drives because the slower drive had a better head design and could transfer data much faster. If this task is not up to you perhaps you could post the make/model number of the drives on both systems. Without knowing the sub-system capability you could be misleading yourself with other tests. Bill -- Bill Vermillion - bv @ wjv . com
Brett Glass wrote on Fri, Dec 21, 2007 at 10:31:24PM -0700:> > As has been reported in some other messages on this list, Linux is > currently blowing FreeBSD away. It's taking as much as 20% less > time to get through the benchmark, depending on exactly how the > random shuffle came out. This is with 4 GB RAM, the GENERIC FreeBSD > SMP kernel (using SCHED_ULE), and aufs as the storage schema for Squid.How much CPU load is there on the server during this benchmark? As I have measured and reported since 1997, if you have CPU load and do throughput timing at the same time, then FreeBSD always defaults to satisfy the CPU eating processes and starves network (at least TCP) and Linux is the other way round, keeps the throughput up but CPU needing processes lose out. Kind of opposite of what you'd expect from the ninja macho networker OS (FreeBSD) and the desktop friendly OS (Linux). Anyway, the real question is whether you get anything in return, aka is there anything running on the server that uses CPU but is not directly bound to the network throughput and hence gets work done quicker.> It appears, though I'd need to instrument the code more to be sure, > that the slowdown is coming from file I/O.I could measure the above effects with things coming out of nowhere, aka data made up by a process. If you are observing the same phaenomenon then it has nothing to do with file I/O.> Could it be that there > less concurrency or more overhead in FreeBSD file operations than > there is in Linux? Even with SoftUpdates turned on, the cache > volume mounted with -noatime, and aufs (which uses kqueues -- a > FreeBSD invention -- to optimize multithreaded disk access), the > benchmark shows FreeBSD losing out. Why?All that async and softupdates stuff only matters if you use a lot of small files, which you shouldn't do in the first place if you are building a cache of some sorts. In all likelyness the cache that you are building is satisfying requests out of the RAM, even if the cache is file-backed. You need to verify that before you can blame file I/O. So the real questions right now are: - how much CPU load (post top output) - how much disk activity (post iostat or vmstat output) - can't hurt to have a netstat throughput output, too Martin -- %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% Martin Cracauer <cracauer@cons.org> http://www.cons.org/cracauer/ FreeBSD - where you want to go, today. http://www.freebsd.org/