Scott Long
2005-Jul-22 18:09 UTC
[FreeBSD-Announce] FreeBSD Status Report Second Quarter 2005
March-June 2005 Status Report
Introduction
The second quarter of 2005 has again been very exciting. The BSDCan
and MeetBSD conferences were both very interesting and and the sources
of very good times. I highly recommend attending them again next year.
The Google Summer of Code project has also generated quite a bit of
excitement. FreeBSD has been granted 18 funded mentorship spots, the
fourth most of all of participating organizations. Projects being
worked on range from UFS Journalling to porting the new BSDInstaller
to redesigning the venerable www.FreeBSD.org website. We are quite
pleased to be working with so many talented students, and eagerly
await the results of their work. More information and status can be
found at the Wiki site at
http://wikitest.freebsd.org/moin.cgi/SummerOfCode2005.
The FreeBSD 6.0 release cycle is also starting up. The purpose of
quickly jumping from 5.x to 6.0 is to reduce the amount of transition
pain that most users and developers felt when switching from 4-STABLE
to 5.x. 6.0 will feature improved performance and stability over 5.x,
experimental PowerPC support, and many new WiFi/802.11 features. The
5.x series will continue for at least one more release this fall, and
will then be supported by the security team for at least 2 years after
that. We encourage everyone to give the 6.0-BETA snapshots a try and
help us make it ready for production. We hope to release FreeBSD 6.0
by the end of August.
Thanks again to everyone who submitted reports, and thanks to Max
Laier for running the show and putting the reports together. Enjoy
reading!
_________________________________________________________________
Google summer of code
* FreeBSD Summer of Code
* FreeBSD website improvements
* FreeSBIE toolkit integration
* gjournal
* gvinum 'move', 'rename'
* Improve libalias
* Integrate the BSD Installer into FreeBSD
* launchd(8) for FreeBSD
* Network Interface API Cleanup
* Nsswitch / Caching daemon
* SEBSD
* UFSJ -- Journaling for UFS
Projects
* Fundraising - TCP & IP Routing Optimization
* GEOM Gate rewrite
* TODO list for volunteers
* VFS SMP
Documentation
* The FreeBSD Dutch Documentation Project
Kernel
* Autotuning of the page queue coloring algorithm
* CPU Cache Prefetching
* libmemstat(3), UMA(9) and malloc(9) statistics
* Low-overhead performance monitoring for FreeBSD
* Removable interface improvements
* SMP Network Stack
* Transparent support for superpages in the FreeBSD Kernel
* TrustedBSD Audit
Network infrastructure
* Dingo
* if_bridge
* IPv6 Support for IPFW
* Move ARP out of routing table
* TCP Reassembly Rewrite and Optimization
* TTCPv2: Transactional TCP version 2
* Wireless Networking Support
Userland programs
* OpenBSD dhclient import.
* Removing of old basesystem files and directories
Architectures
* PowerPC Port
Ports
* FreshPorts
* Porting v9 of Intels C/C++ Compiler
* Update of the Linux userland infrastructure
Vendor / 3rd Party Software
* OpenBSD packet filter - pf
Miscellaneous
* BSDCan
* EuroBSDCon 2005 - Basel
* FreeBSD Security Officer and Security Team
* TrustedBSD SEBSD
_________________________________________________________________
Autotuning of the page queue coloring algorithm
URL: http://www.Leidinger.net/FreeBSD/current-patches/pq.diff
Contact: Alexander Leidinger <netchild@FreeBSD.org>
The VM subsystem has code to reduce the amount of cache collisions of
VM pages. Currently this code needs to be tuned with a kernel option.
I have a patch which changes this to auto-tuning at boot time. The
auto-tuning is MI, the cache size detection is MD. Cache size
detection is currently available for x86/amd64 (on other systems it
uses default values).
Open tasks:
1. Add cache-detection code for other arches too (Marius told me how
to do this for sparc64).
2. Analyze why the cache detection on Athlons doesn't work (no
problems on a P4, but it uses a different code-path).
_________________________________________________________________
BSDCan
URL: http://www.bsdcan.org/2005/
Contact: Dan Langille <dan@langille.org>
The second annual BSDCan conference was well presented, well attended,
and everyone went away with good stories to tell. If you know anything
that attended, get them to tell you what they did, who they met with,
and talks they listened to.
We had 197 people from 15 different countries. That's a strong turnout
by any definition.
We'll be adding more people to the program committee for BSDCan 2006.
This job involves prodding and poking people from your respective
projects. You get them to submit papers. There are a lot of very
interesting projects out there and not all of them submit a paper.
If you know someone doing interesting work, please let me know and
urge them to start thinking about BSDCan 2006.
_________________________________________________________________
CPU Cache Prefetching
URL: http://people.freebsd.org/~andre/tcpoptimization.html
URL: http://www.nrg4u.com/freebsd/tcp_reass+prefetch-20041216.patch
Contact: Andre Oppermann <andre@freebsd.org>
Modern CPU's can only perform to their maximum if their working code
is in fast L1-3 cache memory instead of the bulk main memory. All of
today's CPU's support certain L1-3 cache prefetching instructions
which cause data to be retrieved from main memory to the cache ahead
of the time that it is already in place when it is eventually accessed
by the CPU.
CPU Cache Prefetching however is not a silver bullet and has to be
used with extreme care and only in very specific places to be
beneficial. Incorrect usage can lead to massive cache pollution and a
drop in effective performance. Correct and very carefully usage on the
other can lead to drastic performance increases in common operations.
In the linked patch CPU cache prefetching has been used to prefetch
the packet header (OSI layer 2 to 4) into the CPU caches right after
entering into the network stack. This avoids a complete CPU stall on
the first access to the packet header because packets get DMA'd into
main memory and thus never are already pre-cache in the CPU caches. A
second use in the patch is in the TCP input code to prefetch the
entire struct tcpcb which is very large and used with a very high
probability. Use in both of these places show a very significant
performance gain but not yet fully quantified.
The final patch will include documentation and a guide to evaluate and
assess the use of CPU cache prefetch instructions in the kernel.
Open tasks:
1. Need funding, see "Fundraising - TCP & IP Routing
Optimization".
_________________________________________________________________
Dingo
URL: http://www.freebsd.org/projects/dingo/
Contact: Several <>
Currently trying to restart bits of the project. Cleaning up the p4
branch. Recently more people have volunteered to help as well. Brooks
Davis has completed removing the ifnet from the softc.
Open tasks:
1. See the web page.
_________________________________________________________________
EuroBSDCon 2005 - Basel
URL: http://www.eurobsdcon.org/
URL: http://www.eurobsdcon.org/cfp.php
Contact: Information <info@eurobsdcon.org>
The fourth European BSD conference in Basel, Switzerland is a great
opportunity to present new ideas to the community and to meet some of
the developers behind the different BSDs.
The two day conference program (Nov 26 and 27) will be complemented by
a tutorial day preceeding the conference (Nov 25).
The program committee is looking for tutorial and paper submissions.
For details, please see: The call for papers online.
_________________________________________________________________
FreeBSD Security Officer and Security Team
URL: http://www.freebsd.org/security/
URL:
http://www.freebsd.org/doc/en_US.ISO8859-1/articles/contributors/st
aff-listing.html#STAFF-SECTEAM
URL: http://vuxml.freebsd.org/
Contact: Security Officer <security-officer@FreeBSD.org>
Contact: Security Team <security-team@FreeBSD.org>
In May 2005, Remko Lodder joined the FreeBSD Security Team, followed
by Christian S.J. Peron in July 2005. In the same time period, Gregory
Shapiro and Josef El-Rayes resigned from the team in order to devote
their time to other projects. The current Security Team membership is
published on the web site.
In the time since the last FreeBSD status report, twelve security
advisories have been issued concerning problems in the base system of
FreeBSD; of these, six problems were in "contributed" code, while
five
problems were in code maintained within FreeBSD. The Vulnerabilities
and Exposures Markup Language (VuXML) document has continued to be
updated by the Security Team and the Ports Committers documenting new
vulnerabilities in the FreeBSD Ports Collection; since the last status
report, 97 new entries have been added, bringing the total up to 519.
The following FreeBSD releases are supported by the FreeBSD Security
Team: FreeBSD 4.10, FreeBSD 4.11, FreeBSD 5.3, and FreeBSD 5.4. Their
respective End of Life dates are listed on the web site.
_________________________________________________________________
FreeBSD Summer of Code
URL: http://wikitest.freebsd.org/moin.cgi/SummerOfCode2005
Contact: Summer of Code Mentors <soc-mentors@FreeBSD.org>
Google has generously funded 18 students to spend the summer working
on FreeBSD related projects. Each student is working with one or more
mentors to learn about how open source software development is done
with FreeBSD. This development work is happening in the Perforce
repository as //depot/projects/soc2005. This tree will soon be
exported via CVSup -- check the Wiki for more information.
_________________________________________________________________
FreeBSD website improvements
Contact: Emily Boyd <soc-emily@freebsd.org>
As part of the Google Summer of Code, I'm working on improvements to
the FreeBSD website (including a proposed website redesign). My mentor
for this project is Murray Stokely.
_________________________________________________________________
FreeSBIE toolkit integration
URL: http://www.freesbie.org
URL: http://wikitest.freebsd.org/moin.cgi/DarioFreni
Contact: Dario Freni <saturnero@freesbie.org>
My Summer of Code project is reengineering and rewrite of FreeSBIE
toolkit, in order to include it in the source tree. Let's call it
FreeSBIE 2
Before being accepted, I worked hard on the FreeSBIE 1 toolkit to make
it more flexible. It now supports amd64 and powerpc architecture. The
built filesystem can now boot from almost every media, from dvd to
compact flash or hard disk. Also on i386 is possible to include the
BSD Installer on the livefs. We've received reports that our toolkit
is successfully used for install cd of pfSense and PC-BSD projects.
My future goals are to make the toolkit even more flexible, capable to
build embedded images (like nanoBSD) or big Live-DVD systems,
depending on user's choice, to support all the architectures supported
by FreeBSD and to write a set of tools for making a netboot server
with a FreeSBIE image.
_________________________________________________________________
FreshPorts
URL: http://www.freshports.org/
Contact: Dan Langille <dan@langille.org>
The following new features have been added to FreshPorts:
* Deprecated Ports
* Expired Ports
* Ports Set To Expire
* Display relevent entries from ports/UPDATING on your watch list
Open tasks:
1. I've noticed that FreshPorts is incorrectly reporting
vulnerabilities under a ver y specific situation . The fix is
sitting in BETA, waiting to be moved to production.
2. I've been working on added Last-Modified to the headers. At
present, there are none. Most of the pages on the BETA website
have been completed. I need to move this to production soon.
3. Customized news feeds are in the works. You'll be able to create a
news feed for each of your watch lists This work is contingent
upon finishing the Last-Modified headers.
_________________________________________________________________
Fundraising - TCP & IP Routing Optimization
URL: http://people.freebsd.org/~andre/tcpoptimization.html
Contact: Andre Oppermann <andre@freebsd.org>
The TCP code in FreeBSD has evolved significantly since the fork from
4.4BSD-Lite2 in 1994 primarily due to new features and refinements of
the TCP specifications.
The TCP code now needs a general overhaul, streamlining and cleanup to
make it easily comprehensible, maintainable and extensible again. In
addition there are many little optimizations that can be done during
such an operation, propelling FreeBSD back at the top of the best
performing TCP/IP stacks again, a position it has held for the longest
time in the 90's.
This overhaul is a very involved and delicate matter and needs
extensive formal and actual testing to ensure no regressions compared
to the current code. The effort needed for this work is about three
man-month of fully focused and dedicated time. To get it done I need
funding to take time off my day job and to dedicate me to FreeBSD work
much the way PHK did with his buffer cache and vnode rework projects.
I've got the opportunity to work up to three man-month exclusively
full-time on FreeBSD during the second half of 2005. That means up to
720 hours of full-steam coding (at 60 hours/week)! I will work as much
time as the fundraise provides.
I need to raise enough money for each month from donations from the
FreeBSD community to cover my fixed cost of living, office and
associated overhead. These fixed cost amount to US$6,300/month
(EUR5,200 or CHF8,000). Yes, Switzerland is not the cheapest place to
live. :)
A detailed description of the tasks involved and the code I will write
is on my FreeBSD website; Follow the link above.
Open tasks:
1. Raise enough money to get all the almost finished TCP and IP code
into the tree.
_________________________________________________________________
GEOM Gate rewrite
URL: http://cvsweb.freebsd.org/src/sys/geom/gate/
URL: http://cvsweb.freebsd.org/src/sbin/ggate/
Contact: Pawel Jakub Dawidek <pjd@FreeBSD.org>
GGATE is a machinism for exporting storage devices over the network.
It was reimplemented to be much faster and to handle network failures
better. The ggatec uses two threads now: sendtd, which takes I/O
request from the kernel and sends it to ggated; recvtd, which receives
finished requests and forwards them to the kernel. The ggated uses
three threads: recvtd, which receives I/O requests from ggatec;
disktd, which executes I/O requests (reads or writes data); sendtd,
which sends finished requests to ggatec. The new ggate has been
committed to 6.x.
The work was sponsored by Wheel Sp. z o.o.
_________________________________________________________________
gjournal
URL: http://wikitest.freebsd.org/moin.cgi/gjournal
Contact: Ivan Voras <ivoras@gmail.com>
The schedule (as stated on the wiki page) is honoured, which means
that the development has started, but there's not enough code for
testing. Many details have been thought-out and the development is
ongoing.
_________________________________________________________________
gvinum 'move', 'rename'
URL: http://wikitest.freebsd.org/moin.cgi/GvinumMoveRename
Contact: Chris Jones <soc-cjones@freebsd.org>
With the releases of FreeBSD 5.3 and 5.4, FreeBSD has been moving away
from "old-style" vinum towards GEOM-enabled gvinum for logical
volume
management. While gvinum is a mostly feature-complete replacement for
vinum, it does not implement the 'move' or 'rename' verbs
which are
rather useful when reorganizing one's volume layout, the alternative
being a tedious process of deleting and recreating subdisks, plexes,
or volumes. Additionally, gvinum is nearly completely undocumented,
which contributes to the perception of gvinum as an unfinished
project.
I'm working on implementing 'move' (being able to move a subdisk
from
one drive to another) and 'rename' (being able to rename an subdisk,
plex, volume, or drive), as well as on documentation for gvinum.
So far, I've come up with a plan of attack with le@ and phk@, and
implemented the bulk of the userland code for gvinum 'move' and
'rename'. Still to come are the kernel-side code and documentation.
Open tasks:
1. 'move' and 'rename' userland implementation
2. 'move' and 'rename' kernel-side implementation
3. Outline new handbook section and man page
4. Implement new handbook section and man page
_________________________________________________________________
if_bridge
Contact: Andrew Thompson <thompsa@freebsd.org>
This was committed to current on 5 Jun 2005 and will first appear in
the 6.0 release, thanks to everyone who tested. Recent improvements
include:
* IPFW layer2 filtering
* DUMMYNET support
* IP header alignment checking
There is ongoing work to bring in some of the advanced features from
OpenBSD such as IPSec bridging. People are encouraged to use if_bridge
and report any problems to the mailing lists.
_________________________________________________________________
Improve libalias
URL: http://wikitest.freebsd.org/moin.cgi/PaoloPisati
Contact: Paolo Pisati <soc-pisati@freebsd.org>
My SoC project is about improving libalias and integrating it with
ipfw2, adding nat support into the firewall. Till now i ported
libalias (as a kld) and ng_nat to 4.x and 5.x branches, and i've
already a first working patchset that adds 'nat' action into ipfw.
Next step will be to add a complete syntax to ipfw that will let us
manipulate libalias operations, much like we already do with queue and
pipes for dummynet. In the end the entire work will compile and work
out of the box for 4.x, 5.x and 6.x. More details about the project
and its status are available on wiki page.
_________________________________________________________________
Integrate the BSD Installer into FreeBSD
URL: http://www.bsdinstaller.org
URL: http://wikitest.freebsd.org/moin.cgi/BSDInstaller
URL:
http://perforce.freebsd.org/depotTreeBrowser.cgi?FSPC=//depot/projects
/soc2005/bsdinstaller
Contact: Andrew Turner <soc-andrew@FreeBSD.org>
Progress towards integrating the BSD Installer for Google's Summer of
Code is coming along nicely. The installation CD will boot to
multi-user mode and run both the front and back ends. It can then
partition a hard drive, install the base distribution and make the
disk bootable.
Open tasks:
1. Test in non-i386
2. Investigate installing from other media
3. Many more tasks
_________________________________________________________________
IPv6 Support for IPFW
Contact: Max Laier <mlaier@freebsd.org>
Contact: Brooks Davis <brooks@freebsd.org>
At the developer summit before BSDCan it was decided to remove IP6FW
from the tree as it has a couple of problems. The most pressing one is
the lack of synchronization and thus the need for debug.mpsafenet=0.
As a replacement Brooks Davis has imported patches to teach the
existing and well-locked IPFW2 code about IPv6.
Since the initial import I have added some features required to manage
IPv4 and IPv6 in a single ruleset. I have also extended existing
opcodes to work with IPv6. There are, however, still some opcodes that
do not work with IPv6 and most of the more exotic ones haven't been
tested. As long as IPFW2+v6 does not provide enough functionality and
stability to work as a drop-in replacement for IP6FW, we won't remove
IP6FW.
In order to get the new code to that point we really need more testers
with real world IPv6 deployment and interest in IPFW+v6. The lack
thereof (I haven't received a single answer on my requests to various
FreeBSD mailing lists) has made it hard to progress.
Open tasks:
1. Properly implement O_REJECT for IPv6
2. Maybe implement O_LOG
3. Test new(er) IPFW2 opcodes with IPv6
4. Test
5. Test
6. Test
_________________________________________________________________
launchd(8) for FreeBSD
URL: http://wikitest.freebsd.org/moin.cgi/launchd
URL:
http://developer.apple.com/documentation/Darwin/Reference/ManPages/man
8/launchd.8.html
Contact: R. Tyler Ballance <tyler@tamu.edu>
So far progress has been slow, the autoconf build system has been
removed from all of the launchd(8) code, and launchctl(1) is building
and semi-functional on FreeBSD-CURRENT (i.e. CoreFoundation hooks have
been removed)
I'm currently working on porting "liblaunch" which is the core
backend
to both launchd(8) (the actual daemon) and launchctl(1), there are
some mach/xnu specific hooks and calls that need to be remove and
either reimplemented or worked around.
We're also waiting on a response from Apple on a possible BSD-licensed
version of the code (it's currently under the APSL) Progress is slow,
but steady.
_________________________________________________________________
libmemstat(3), UMA(9) and malloc(9) statistics
URL: http://www.watson.org/~robert/freebsd/libmemstat/
Contact: Robert Watson <rwatson@FreeBSD.org>
libmemstat(3) provides a user space library API to monitor kernel
memory allocators, currently uma(9) and malloc(9), with the following
benefits:
* ABI-robust interface making use of accessor functions, in order to
divorce monitoring applications from kernel/user ABI changes.
* Allocator-independent interfaces, allowing monitoring of multiple
allocators using the same interface.
* CPU-cache awareness, allowing tracking of memory use across
multiple CPUs for allocators aware of caches. Unlike previous
interfaces, libmemstat(3) coalesces per-CPU stats in user space
rather than kernel, and exposes per-CPU stats to interested
applications.
* Ability to track memory types over multiple queries, and update
existing structures, allowing easy tracking of statistics over
time.
libmemstat(3) and the the appropriate allocator changes for uma(9) and
malloc(9) are currently in HEAD (7-CURRENT), and MFC has been approved
to RELENG_6 for inclusion in 6.0-RELEASE. These changes may also be
backported to 5.x.
Sample applications include memstat(8), an allocator-independent
statistics viewing tool, memtop(8), which provides a top(1)-like
interface for monitoring kernel memory use and active memory types.
None of these are "pretty".
netstat -mb has also been updated to use libmemstat(3) to track
network memory use using uma(9), rather than the less reliable mbuf
allocator statistics interface. As a result, the statistics are now
more reliable on SMP systems (this corrects the bug in which mbuf
statistics sometimes "leaked", even though memory didn't), and
more
informative (cache information is now displayed, as well as mbuf tag
information).
Open tasks:
1. Teach libmemstat(3) to speak libkvm(3) in order to allow tools
linked -lmemstat to interogate kernel core dumps.
2. Teach libmemstat(3) to interface with user space malloc and track
malloc allocations for user space applications.
3. Update vmstat(8) -m and -z implementations to use libmemstat(3)
instead of the old monitoring interfaces. Code to do this exists
in the sample libmemstat(3) applications.
4. Identify how to make streams or the library endian-aware so that
streams dumped from a kernel of alternative endian could be
processed using libmemstat(3) on another system.
5. Identify any remaining caching allocators in the kernel, such as
the sfbuf allocator, and teach libmemstat(3) how to interface with
them.
_________________________________________________________________
Low-overhead performance monitoring for FreeBSD
URL: http://people.freebsd.org/~jkoshy/projects/perf-measurement
Contact: Joseph Koshy <jkoshy@FreeBSD.org>
Modern CPUs have on-chip performance monitoring counters (PMCs) that
may be used to count low-level hardware events like instruction
retirals, branch mispredictions, and cache misses. PMC architectures
and capabilities vary between CPU vendors and between CPU generations
from the same vendor, making the creation of portable applications
difficult. This project implements a cross-platform PMC management API
for applications, and implements the infrastructure to "virtualize"
and manage these PMCs. The creation of performance analysis tools that
use this infrastructure is also part of the project's goals.
Work since the last status report:
* Sampling mode support for P4 and AMD64 PMCs has been implemented.
* A pmclog(3) API for parsing hwpmc(4) log files has been added.
* A number of bugs in libpmc(3), hwpmc(4) and pmcstat(8) have been
fixed.
Future work:
* Creating user documentation showing a few real-world uses of the
currently available tools.
* Testing, improving the stability of the code, and characterizing
its overheads.
* Implementing P5 PMC support.
_________________________________________________________________
Move ARP out of routing table
URL: http://people.freebsd.org/~qingli/
Contact: Qing Li <qingli@freebsd.org>
I've sent the patch to jinmei@isl.rdc.toshiba.co.jp @KAME for review.
I'm still waiting for feedback from Andre. There hasn't been any
major
change since the last report. I've kept the code in sync with CURRENT.
Gleb has created a separate P4 branch and has been helping out on the
locking side. Gleb is also helping out on the testing front.
Open tasks:
1. I'm waiting for review feedback from my mentor Andre on the
overall design and code. I'm waiting for feedback from Andre on
Gleb's suggested modification.
_________________________________________________________________
Network Interface API Cleanup
URL: http://wikitest.freebsd.org/moin.cgi/CleanupOfNetworkIterfaceApis
Contact: Anders Persson <soc-anders@freebsd.org>
The goal of this project is to review the network interface API and
try to remove references to kernel-only data structures by removing
the use of libkvm and instead rely on other interfaces to provide
information. If there are no adequate interfaces, they would be
created.
Currently netstat is being reviewed and parts of it have been modified
to use sysctl rather than libkvm to provide the information.
A big thank you to Brooks Davis for mentoring :-)
_________________________________________________________________
Nsswitch / Caching daemon
URL:
http://wikitest.freebsd.org/moin.cgi/NsswitchAndCachingTechnicalDetail
s
URL: http://wikitest.freebsd.org/moin.cgi/MichaelBushkov
Contact: Michael Bushkov <soc-bushman@rsu.ru>
The nsswitch / caching daemon project is being developed within the
Google's Summer Of Code program. The first goal of this project is to
implement a set of patches to extend the use of nsswitch subsystem.
The second goal is the development of the caching library and daemon
to add the caching ability to the nsswitch.
Currently services, protocols, rpc and openssh patches are finished.
Support for services, services_compat, rpc, protocols, and
ssh_host_keys databases is added with 'files', 'nis' and
'compat' (for
services) sources possible. The nsswitch-friendly openssh port is
amlost completed.
Open tasks:
1. Implement set of patches to make nsswitch support globus grid
security files , MAC and audit related configuration files
databases.
2. Implement the caching library and the caching daemon and patch
nsdispatch function to support caching.
_________________________________________________________________
OpenBSD dhclient import.
Contact: Brooks Davis <brooks@FreeBSD.org>
Contact: Sam Leffler <sam@FreeBSD.org>
The OpenBSD rewrite of dhclient has been imported, replacing the ISC
dhclient. The OpenBSD client provides better support for roaming on
wireless networks and a simpler model of operation. Instead of a
single dhclient process per system, there is on per network interface.
This instance automatically goes away in the even of link loss and is
restarted via devd when link is reacquired. To support this change,
many aspects of the network interface configuration process were
overhauled.
The current code works well in most circumstances, but more testing
and polishing is needed.
_________________________________________________________________
OpenBSD packet filter - pf
Contact: Max Laier <mlaier@freebsd.org>
We will have pf as of OpenBSD 3.7 for RELENG_6. Import has been
completed in early May and FreeBSD release 6.0 will ship with it.
A few serious issues with pfsync on SMP have been discovered since
CARP is around and more and more people use it on big iron. Everything
that has been discovered is fixed in HEAD and (if applicable) MFCed
back to RELENG_5. Some functional changes are undergoing testing right
now and will be MFCed in the coming days.
With the import of if_bridge from Net/OpenBSD we finally have a bridge
implementation that allows for stateful filtering as well as IPv6
filtering. Please see the respective report.
Open tasks:
1. Shared lock implementation?
_________________________________________________________________
Porting v9 of Intels C/C++ Compiler
Contact: Alexander Leidinger <netchild@FreeBSD.org>
Intel released version 9 of its C/C++ compiler. Work to port the x86
version to FreeBSD is in progress as time permits. Porting the EM64T
(amd64) version is on the TODO list too, but is subject to enough free
time and access to appropriate hardware.
_________________________________________________________________
PowerPC Port
URL: http://www.freebsd.org/platforms/ppc.html
Contact: Peter Grehan <grehan@FreeBSD.org>
Florent Thoumie has updated the massively out-of-date platform page.
Work continues to creating a 6.0 release of the PowerPC port.
_________________________________________________________________
Removable interface improvements
URL: http://people.freebsd.org/~brooks/pubs/eurobsdcon2004/
URL: http://www.freebsd.org/projects/dingo/
Contact: Brooks Davis <brooks@FreeBSD.org>
This project is an attempt to clean up handling of network interfaces
in order to allow interfaces to be removed reliably. Current problems
include panics if Dummynet is delaying packets to an interface when it
is removed.
I have removed struct ifnet's and layer two common structures from
device driver structures. This will eventually allow them to be
managed properly upon device removal. This code has been committed and
will appear in 6.0. Popular drivers have generally been fixed, but
more testing is needed.
_________________________________________________________________
Removing of old basesystem files and directories
URL:
http://www.Leidinger.net/FreeBSD/current-patches/obsolete_removal.diff
Contact: Alexander Leidinger <netchild@FreeBSD.org>
FreeBSD lacks a way to remove old/outdated files and directories in
the basesystem. I have a patch which removes obsolete files in a safe
way (interactively, since only the administrator really knows if
there's a need to keed an old file or not; there's a switch for
batch-processing). This feature may or may not be available for
6.0-RELEASE, depending on the decission from the Release Engineering
team.
Open tasks:
1. Respect the NO_* switches and remove those files too. This is easy
to do with the current implementation, but isn't needed to commit
the removal of obsolete files feature.
_________________________________________________________________
SEBSD
URL: http://wikitest.freebsd.org/moin.cgi/YanjunWu
Contact: Yanjun Wu <yanjun03@ios.cn>
1. Setup a local P4 workspace of sebsd source and Setup lxr for
TrustedBSD source for studying source code.
2. Test a simple policy configuration for vsftpd.
3. Writing a HOWTO document Getting Started with SEBSD HOWTO by
deriving the existing Getting Started with SELinux HOWTO .
Thanks Robert Watson and Scott Long for their kind help.
Open tasks:
1. When writing the document, try to figure out the sebsd userland
utils that need to be ported.
2. Test and edit more policies for BSD environment.
_________________________________________________________________
SMP Network Stack
URL: http://www.FreeBSD.org/projects/netperf/
Contact: Robert Watson <rwatson@FreeBSD.org>
Significant work has occurred over the last few months relating to the
SMP network stack work. A few of the highlights are covered here at a
high level:
* The UMA(9) per-CPU caches have been modified to use critical
sections instead of mutexes. Recent critical section optimizations
make this a performance win for both UP and SMP systems. This
results in a several percent improvement in a number of user space
benchmarks, and larger improvement for kernel-only network
forwarding and processing benchmarks.
* The malloc(9) allocator has been modified to store statistics
per-CPU instead of using a cross-CPU statistics pool, with each
per-CPU pool now using critical sections to synchronize access.
This results in a measurable performance win, especially on SMP
systems
* The netnatm ATM code is now MPSAFE.
* netipx MPSAFEty has been merged to RELENG_5.
* The netperf cluster has now been expanded to include two
additional quad-CPU systems (one dual dual-core AMD system, one
quad-CPU PIII system).
* libmemsetat(3) (see separate report) now corrects SMP-related
races in the measuring of mbuf allocator statistics, as well as
substantially improving kernel memory monitoring capabilities and
tools.
* A range of locking bug fixes, and general network stack bug fixes.
* Significant updates to the SMPng web page (still more to do!).
* Identification of all non-MPSAFE network device drivers, with
ultimatum issued, on freebsd-arch. Quite a bit of new driver
locking work as a result (if_ed, if_de, ...).
* Lots of other stuff.
In most cases, these changes will appear in FreeBSD 6.0-RELEASE; some
have been, or will be, merged to FreeBSD 5.x.
On-going tasks include:
* Review and improvement of ifnet locking, such as address lists and
flags.
* Optimization of interface start hand-off.
* Prototyping of queue-oriented packet hand-off in the stack.
* Performance measurement and analysis.
* Prototype rewrite and simplification of socket locking.
_________________________________________________________________
TCP Reassembly Rewrite and Optimization
URL: http://people.freebsd.org/~andre/tcpoptimization.html
URL: http://www.nrg4u.com/freebsd/tcp_reass-20041213.patch
URL:
http://lists.freebsd.org/pipermail/freebsd-net/2004-December/005918.ht
ml
Contact: Andre Oppermann <andre@freebsd.org>
Currently TCP segment reassembly is implemented as a linked list of
segments. With today's high bandwidth links and large bandwidth*delay
products this doesn't scale and perform well.
The rewrite optimizes a large number of operational aspects of the
segments reassembly process. For example it is very likely that the
just arrived segment attaches to the end of the reassembly queue, so
we check that first. Second we check if it is the missing segment or
alternatively attaches to the start of the reassembly queue. Third
consecutive segments are merged together (logically) and are skipped
over in one jump for linear searches instead of each segment at a
time.
Further optimizations prototyped merge consecutive segments on the
mbuf level instead of only logically. This is expected to give another
significant performance gain. The new reassembly queue is tracking all
holes in the queue and it may be beneficial to integrate this with the
scratch pad of SACK in the future.
Andrew Gallatin was able to get 3.7Gb/sec TCP performance on
dual-2Gbit Myrinet cards with severe packet reordering (due to a
firmware bug) with the new TCP reassembly code. See second link.
Open tasks:
1. Need funding, see "Fundraising - TCP & IP Routing
Optimization".
_________________________________________________________________
The FreeBSD Dutch Documentation Project
URL: http://www.freebsd.org/doc/nl/books/handbook
URL: http://www.evilcoder.org/content/section/6/39/
URL: http://www.evilcoder.org/freebsd_html/
URL: http://www.evilcoder.org/freebsd/flyer.pdf
Contact: Remko Lodder <remko@FreeBSD.org>
Contact: Siebrand Mazeland <siebrand.mazeland@xs4all.nl>
Contact: Rene Ladan <r.c.ladan@student.tue.nl>
The FreeBSD Dutch Documentation Project is a ongoing project in
translating the english documentation to the Dutch language. Currently
we are almost done with the FreeBSD Handbook. Finishing the Handbook
is our first priority, and we could use your help. Please contact
Siebrand or myself if you want to helpout. After the handbook we will
focus on other documents as well, so feel free to help us there as
well
Open tasks:
1. FreeBSD Handbook translation. Finish the translation from English
to Dutch
2. FreeBSD Handbook review. Finish the review of the translated
documents
3. FreeBSD Articles. Start translating the articles from English to
the Dutch Language
4. FreeBSD www. Start translating the website from English to the
Dutch Language
5. The rest of the FreeBSD Documents. Start translating them from
English to the Dutch Language.
_________________________________________________________________
TODO list for volunteers
Contact: Alexander Leidinger <netchild@FreeBSD.org>
Since Google's "Summer of Code" resulted in a lot of interest
in open
projects, I'm in the process of compiling a list of nice projects for
volunteers. Unlike Google's SoC those projects aren't backed with
money (but this doesn't means nobody is allowed to sponsor one of
those projects), so we can only guarantee the social aspects (some
"Thank you!" and "That's great!" messages). So far
the list has
several entries where the difficulty ranges from "someone just has to
sit down and spend some time on it" up to "we need a guru for
this".
Open tasks:
1. Merging untaken entries from the SoC list as soon as the official
participants/tasks in the SoC are announced.
2. Sending the document to some doc people for review.
3. Commit the list.
_________________________________________________________________
Transparent support for superpages in the FreeBSD Kernel
Contact: Alan L. Cox <alc@cs.rice.edu>
Contact: Olivier Crameri <olivier.crameri@epfl.ch>
We are currently working on an updated implementation of Juan
Navarro's transparent support for superpages in FreeBSD
The idea is to take advantage of the architectural support for big
memory pages (superpages) by using a reservation mechanism allowing us
to transparently promote groups of base pages into superpages and
demote superpages into several smaller superpages or base pages.
The advantage of using superpages vs. base pages is to significantly
improve the TLB coverage of the physical memory, thus improving the
peformance by reducing the number of TLB misses.
The modification of the FreeBSD kernel that we are working on involves
the replacement of the current list based page allocation mechanism
with a system using a buddy allocator to reserve groups of pages for a
memory object. The promotion and demotion of the pages occur directly
within the pmap module.
The former implementation was supporting the alpha and IA64
architectures. We are adding the support for amd64. We currently have
an almost complete implementation. Once completed we will make a
performance study with a particular emphasis on TLB and cache misses.
_________________________________________________________________
TrustedBSD Audit
URL: http://www.trustedbsd.org/components.html#audit
Contact: Robert Watson <rwatson@FreeBSD.org>
Contact: Wayne Salamon <wsalmon@FreeBSD.org>
Contact: <trustedbsd-discuss@TrustedBSD.org>
In the past few months, significant work has been done relating to the
TrustedBSD audit implementation, including preparatory work to merge
audit into the FreeBSD CVS repository for FreeBSD 6.x. In particular:
* The user space components, such as libbsm, include files, and
command line utilities have been broken out into an OpenBSM
distribution in Perforce. Improvements in OpenBSM will be made
available separately for use by projects such as Darwin, and
imported into the contrib area of FreeBSD.
* The system call table format has been updated to include an audit
event identifier for each system call across all hardware
platforms and ABIs (merged), and all system calls have been
assigned event identifiers (not yet merged).
* The audit management daemon has been rewritten to run on FreeBSD
(originally derived from Darwin) using /dev/audit to track kernel
events.
* Many system calls now properly audit their arguments.
* The TrustedBSD audit3 branch has been updated to a recent
6.x-CURRENT.
* Significant work has gone into synchronizing the audit event
tables between FreeBSD, Darwin, and OpenSolaris to make sure file
formats and events are portable.
* OpenBSM has been adapted to consume and generate
endian-independent event streams.
* OpenBSM documentation has been created.
The hope is still to provide audit as "experimental" in 6.0; the
primary blocking factor is our awaiting relicensing of the last
remaining audit files from Apple's APSL license to BSDL so that they
can be included in the FreeBSD kernel. This is anticipated to complete
in the near future. Once this is done, the changes can be merged to
CVS, and then MFC'd to RELENG_6. If this is not complete by
6.0-RELEASE, the work will be merged shortly after the release, as all
ABI-sensitive data structures have been updated as needed.
_________________________________________________________________
TrustedBSD SEBSD
URL: http://www.TrustedBSD.org/sebsd.html
Contact: Robert Watson <rwatson@FreeBSD.org>
The TrustedBSD Project has released a new snapshot of "SEBSD", a
port
of NSA's SELinux FLASK and Type Enforcement implementation to FreeBSD
based on a late 2005 FreeBSD 6.x snapshot. The SEBSD distribution has
now been updated in Perforce to a recent 6.x snapshot, and a new
distribution will be made available in the near future.
Work has been performed to merge additional dependencies for SEBSD
back into the base FreeBSD tree, including most recently, changes to
devfs, and System V and POSIX IPC.
Open tasks:
1. Update to new NSA FLASK implementation, which has improved MLS
support.
2. Merge remaining kernel changes to support SEBSD back to the base
FreeBSD CVS repository, including file descriptor labeling and
access control (in contrast to file labeling and access control),
and categorization of kernel privileges.
_________________________________________________________________
TTCPv2: Transactional TCP version 2
URL: http://people.freebsd.org/~andre/tcpoptimization.html
URL:
http://lists.freebsd.org/pipermail/cvs-all/2004-November/089939.html
Contact: Andre Oppermann <andre@freebsd.org>
The old TTCP according to RFC1644 was insecure, intrusive, complicated
and has been removed from FreeBSD >= 5.3. Although the idea and
semantics behind it are still sound and valid.
The rewrite uses a much easier and more secure system with 24bit long
client and server cookies which are transported in the TCP options.
Client cookies protect against various kinds of blind injection
attacks and can be used as well to generally secure TCP sessions (for
BGP for example). Server cookies are only exchanged during the
SYN-SYN/ACK phase and allow a server to ensure that it has
communicated with this particular client before. The first connection
is always performing a 3WHS and assigning a server cookie to a client.
Subsequent connections can send the cookie back to the server and
short-cut the 3WHS to SYN->OPEN on the server.
TTCPv2 is fully configurable per-socket via the setsockopt() system
call. Clients and server not capable of TTCPv2 remain fully compatible
and just continue using the normal 3WHS without any delay or other
complications.
Work on implementing TTCPv2 is done to 90% and expected to be
available by early February 2005. Writing the implementation
specification (RFC Draft) has just started.
Open tasks:
1. Need funding, see "Fundraising - TCP & IP Routing
Optimization".
_________________________________________________________________
UFSJ -- Journaling for UFS
Contact: Brian Wilson <polytopes@gmail.com>
Contact: Scott Long <scottl@FreeBSD.org>
filesystem. Journaling helps ensure the filesystem's integrity should
the system crash. Journaling eliminates the need for fsck'ing a
filesystem, as the filesystem is never in an inconsistent state
(barring hardware failure). This implementation is inspired by
Darwin's HFS+ filesystem and the SGI XFS filesystem. This is a Summer
of Code project, with Scott Long as the mentor and Brian Wilson as the
developer/mentee. Currently this project is still in the early stages,
but will be in a usable state by September 1 (the Google Summer of
Code completion date).
Open tasks:
1. Finish making the file system log metadata updates.
2. Add facilities to replay the log on dirty file systems.
3. Make snapshots work with journaling.
_________________________________________________________________
Update of the Linux userland infrastructure
Contact: Alexander Leidinger <netchild@FreeBSD.org>
Contact: Emulation Mailinglist <emulation@FreeBSD.org>
The cleanup/streamlining and the possibility of overriding the default
Linux base as reported in the last report happened without major
problems. Work on the open tasks hasn't started yet, but is scheduled
to start "soon". If a volunteer wants to spend some hours on one of
the open tasks, he should tell it on the emulation mailinglist.
Open tasks:
1. Refactoring the common RPM code in x11-toolkits/linux-gtk/Makefile
into bsd.rpm.mk.
2. Determining which up-to-date Linux distribution to use as the next
default Linux base. Important criteria:
+ RPM based (to be able to use the existing infrastructure)
+ good track record regarding availability of security fixes
+ packages available from several mirror sites
+ available for several hardware architectures (e.g. i386,
amd64, sparc64; Note: not all architectures have a working
linuxolator for their native bit with, but as long as there
are no userland bits available, no motivation regarding
writing the kernel bits will arise)
3. Moving the linuxolator userland to an up-to-date version (see
above).
_________________________________________________________________
VFS SMP
Contact: Jeff Roberson <jeff@freebsd.org>
FreeBSD's VFS layer has been fine grain locked along with the FFS
filesystem for the FreeBSD 6.0 release. The locking has been underway
for several years, with the project really picking up over the last 6
months thanks largely to sponsorship provided by Isilon Systems, Inc.
a leading vendor of clustered storage systems. The project has entered
a stabilization phase, with a few bugs being reported in extreme
circumstances while the majority of users have seen no problems. Tests
on a 8 and 16 way machines yield reasonable parallelization, however,
it will be beneficial to do lock contention analysis once things are
fully stable.
For those interested in technical details, there have been a few
relatively significant changes with vnode life-cycle management. Vnode
reference counting and recycling is now no longer an ad-hoc process
involving a variety of flags, a use count and the hold count. A single
hold count is used to track all vnode references and a destroyed vnode
is freed in the context of the caller when the last ref is lost. The
old system would never reclaim memory used by vnodes and also had
pathlogical behavior with unreferenced vnode caching under pressure.
The new system is much simpler than the old one, however, callers are
now required to vhold a vnode that they lock directly without going
through vget to prevent it from being recycled while they are waiting
on a lock. Relying on 'location stable storage', which is a more
strict version of 'type stable storage' is no longer a valid
approach.
Some other side effects include a much simpler and faster nullfs
implementation, an improved buf daemon flushing algorithm which
eliminated high latency that caused audio skipping, and a lots of
minor cleanups and debugging aids.
_________________________________________________________________
Wireless Networking Support
Contact: Sam Leffler <sam@freebsd.org>
A lot of bugs were fixed in preparation for the 6.0 release. 6.0 will
be the first release to include full WPA support (both supplicant and
authenticator).
A presentation on the forthcoming multi-bss support was given at
BSDCan 2005. The slides from the talk are available at
http://www.freebsd.org/~sam/BSDCan2005.pdf . The plan is to commit
this work to HEAD after 6.0 is released which means the first release
that will have it is 7.0.
Open tasks:
1. hostapd needs work to support the IAPP and 802.11i
preauthentication protocols (these are simple conversions of
existing Linux code).
_________________________________________________________________