Poul-Henning Kamp
2004-Dec-20 04:49 UTC
[FreeBSD-Announce] Final report on funded FreeBSD development project
Final report on funded FreeBSD development project [1]
------------------------------------------------------
As promised, here follows the status report on how my six months
of user-community funded FreeBSD development went.
The goal of this effort was to get as far in the "buf-junta" plans
(http://people.freebsd.org/~phk/plan.html) as possible, and as I
have outlined below, a significant dent was made in that list
over the last six months.
I would like to thank 187 persons and companies for, though their
donations small and large, making this work possible. The future
of FreeBSD is very much in the hands of the developers, and by
helping this particular developer get his hands free for half a
year certainly moved a lot of stuff from the future into the present.
If you, my donors, are not happy with what you got out of your
donations, the fault is entirely mine. (and I would like you to tell
me about it as fast as possible.) Please do not fault the FreeBSD
project as such or the FreeBSD Foundation for that matter, if this
was not a success, it is all my personal fault.
If on the other hand you think you got your moneys worth and you
would be willing to help support FreeBSD development financially
also in the future, then, and only then, will I declare my project
to have been a complete success.
Feedback either way is most welcome!
The future
----------
A lot of people have asked me "what about the future ?"
I promised them to make up my mind and let them know in this
report: I have decided to not do another fundraising drive at
this time.
There are a number of reasons for this.
The primary reason is that I do not want to be stuck in an ivory
tower doing only the FreeBSD work I can see from my lab in the
basement. I need to get out amongst people and get inspiration
and input from real-world usage of FreeBSD. To that end I hope
to land a couple of contracts currently under discussion and
occupy half of my time this way.
The other reason is that I hope the FreeBSD Foundation will be
the place where this kind of fundraising happens.
Channeling the funds through the FreeBSD Foundation means that
this will not grow into a personality cult around my person and
that money can be spent where it is most needed rather than on
where I happen to fancy spending it.
The minor reason is that the paperwork associated with fundraising
is not a favourite hobby of mine and I'm still not done resolving
various sticky details [4]
But since there are still a lot of work to be done in this area
of the kernel and since that is what I think I am best at, I
have sent a proposal to the FreeBSD Foundation for funding me
part time for a period to continue working on this stuff.
I was somewhat late in sending my proposal to the Foundation,
and have not yet received an answer from them, so the outcome
of this is not yet known. If it falls through, I will find a
couple of contracts for now, and then reconsider the situation
later in the 2005.
If you want to support FreeBSD development financially, the way
to do it right now is to donate money to the FreeBSD Foundation.
If you do so, some of the money may or may not end up funding
my work, either way, it will be money well spent.
High-lights from my personal score-card:
----------------------------------------
June ... October -- TTY code/driver cleanup
In order to not clash with the RELENG_5 release work, I
attacked the tty code/drivers in order to try to drive Giant
out of the pty driver.
I found that a massive amount of code which should have
lived in the generic tty code was copy&pasted into almost
every and all tty device drivers.
In the end I did not manage to do any of the actual locking
work, but I did get a lot of preparation for it done: I
eliminated the 3100 lines of copy&paste'd code so that there
will be many fewer places where locking will have to be
handled (and bringing a lot of consistency to our tty drivers
as a side effect).
I also managed to get struct tty properly reference counted,
but some sticky issues about unloading tty drivers with
dangling sessions still need to be handled.
June ... November -- Device and Fifo vnode bypass
In terms of architecture, this is one of the most significant
changes I have made in the last six months.
Despite being a relatively small change in terms of number
of lines of code changed, this project was spread out over
a long time in order to be sure to catch all the little
details in the corners. One of my test machines have been
more or less dedicated to to this subproject for 5 months.
The end result is that access to devices and fifos ('named
pipes"), from programs now does not go through the vnode
layer for the "traffic" calls, read, write & ioctl. (For
device drivers we may still need Giant if the driver is
marked as needing this.)
August -- Rewrite Floppy driver.
The internal state engine was rewritten and the driver made
INTR_FAST, Giant safe and GEOM aware.
September ... October -- DEVFS/SPECFS consolidation
Devices are now only supported in the devfs filesystem, and
supporting code was removed from ufs, ext2fs, cd9660 and a
few minor filesystems.
In practice this have had very little impact when seen from
outside the kernel, but the simplifications in the kernel
are extensive and very significant.
The end result is that specfs is retired and only devfs
handles devices now.
A significant part of this work has been to properly reference
count access into device drivers in order to make it safe(r)
to unload device drivers. There are still many hurdles here
before it is 100% safe to unload a driver, but at least now
the cdevsw interface should not be one of them.
October ... [ongoing] -- struct buf/vnode -> buf/bufobj/vnode
This is also one of the architecturally very significant
changes which had its fingers all over filesystems and buffer
cache code.
The work is not carried all the way through, but has progressed
sufficiently far to reap the first major benefits (see below).
It is hard to fairly summarize this stuff without it turning
into a long architectural monologue, but the gist of it is
that the aspect of vnods which serve as the attachment point
for stuff in the buffer cache has been given an indenpendent
data structure (bufobj) which in the future will not be tied
to vnodes only.
The untangling has reached and stopped right in front of the
syncer code, and will continue from there as time permits.
October -- Move copyonwrite and prewrite into FFS where they belong.
These two functions are private to snapshots and softupdates,
and with a private strategy implementation in FFS they can be
isolated there entirely.
This simplifies the buffer cache code a bit.
October -- Proper waiting for pending geom events
A process which caused events to be created in geom will now
wait for them to be completed before returning to userland.
This removed an insufficient lot of explicit waits.
October ... --- Put local filesystems directly on GEOM
Another piece of significant architecture [2]
Instead of taking a tour through the vnode layer, DEVFS and
geom_dev, go directly to the new Geom class geom_vfs to access
disk devices for local filesystems.
Amongst many other benefits, this gives us correct read/write
tracking on devices, and it is now possible to mount the same
filesystem r/o many places.
Together with the device vnode bypass, this entirely takes
I/O traffic from devices out of the vnode layer (and more
significantly: vnode locking).
November ... December -- FILEDESC locking
In order to make the locking of file descriptors work better
with devices and fifos bypassing vnodes, the locking was
reworked and struct filedesc got a reference count more for
protecting the memory allocation.
November ... December -- omount(2) -> nmount(2) kernel transition.
All filesystems converted to implement the nmount(2) API
now. Calls to the old omount(2) API are converted in a
special compatibility function in the filesystems.
Userland conversion to nmount(2) should happen in the spring
so that RELENG_6 will be entirely nmount(2) but still support
omount(2) calls. Support for omount(2) will be dropped in
RELENG_7.
December -- Make VOP_* implementations typesafe.
By using sparse struct initialization, it was possible to avoid
a lot of the magic pointer gymnastics in creation of the VOP
method vectors for filesystems. The result is typesafe and
faster VOP_ method calling.
December -- Root filesystem mount rework
Since all filesystems now accept nmount(2) calls, the magic
code mounting the root filesystem was cleaned up, removing
the need for magic per-filesystem code to handle the root
file system.
This eliminated the main source of "bogo-vnodes" in the
system, leaving only the syncer-vnodes now.
June ... December -- Bugs introduced, code broken.
Throughout a number of mistakes, minunderstandings, keyboard
misfires and just plain stupidity on my part have introduced
bugs and broken code.
This will be fixed as soon and as fast as I can.
Pointless statistics:
---------------------
In the mock sprit of sports pages everywhere, and general journalistic
"that was the year that was" reporting approaching the end of a
year:
Total donations: 33336 USD
Average donation (sans outliers): 72 USD
Total number of commits to CVS: 577
Cost per commit: 58 USD
Average time between commits to CVS: 7h36m
Number of submits to p4: 996 (minium ?)
Average time between submits to p4: 4h25m
Netto lines of code added,
not counting retired device drivers: -7291
Pointy hats awarded: 3
Cost per codeline: -.22 USD
Disks crashed: 3
Super strong fridge magnets retrieved: 6
Shoulder injuries: 1
Electricity used by computers 5275 kWh
Successful attempts to communicate
non-criminal status to Paypal: 0
Futile attempts to communicate
non-criminal status to Paypal: 10
Random acts of kindness received: 3
Random acts of kindness not delivered
due to botch by amazon.com: 3 [3]
Merry X-mas & happy new-year,
Poul-Henning
[1] For those of you who don't know what I'm talking about in this
email, a quick visit to http://people.freebsd.org/~phk/funding.html
would be a good idea.
[2] I rather fear I sound as boring as the tourist guides on
the ruins of Akropolis, my apologies.
[3] Amazon.com had a wrong shipping address on my "wish-list", so
at least three packages of gifts from kind people have travelled
all the way from Amazon to my old address and back again. I'm told
people will get a full refund from Amazon once the package makes
it back to them. My apologies for this mess. Much appreciated
anyway.
[4] Paypal is very well and nice, but trying to get through their
CMS system to explain to them that just because you are located in
the EU and received money doesn't make you a member of the mafia
is still not an art I have managed. If you happen to know an email
address of a senior executive let me know.
--
Poul-Henning Kamp | UNIX since Zilog Zeus 3.20
phk@FreeBSD.ORG | TCP/IP since RFC 956
FreeBSD committer | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.