freebsd announce - May 2001 - ANNOUNCE: Status update on ftp.freebsd.org AKA ftp.freesoftware.com

Hi folks,

This is just a short note to discuss the current state of affairs with
our master FTP site, explain what we're doing about it and combat some
of the FUD floating around which has everyone from space aliens to
Wind River Systems (who supply the space aliens with navigational
software, of course) intentionally killing the site off.

On April 20th, 2001, the day that FreeBSD 4.3 was due to be released,
we lost access to our main FTP site, ftp.freesoftware.com, without any
advance warning.  Further investigation revealed that the problem was
due to an unforeseen network outage which caused the hosting ISP to
have to block access to the site.  With one of their major links down,
it was overloading the ISPs backup links and denying bandwidth to
their other customers.

It was also not clear to us, then or now, just how long the outage
would last and whether this was to be a short-term or a long-term
problem.  With the release date for 4.3 already well publicized in
advance and many people asking me for it, I decided to use one of our
backup sites, the usw[1-6].freebsd.org cluster, and at least get the
current releases bits up somewhere on an interim basis.

Unfortunately, I really underestimated both the extent of the demand
and the degree to which our big, fat Gigabit pipe at ftp.freesoftware.com
has made distributing the bits to our mirrors look comparatively easy.
Even with the login limits set *significantly* greater in favor of the
registered mirror sites, and with multiple machines pulling the load,
it quickly overwhelmed the new hosting infrastructure and resulted in
significant bandwidth limiting on FTP traffic being instituted by the
ISP.  Even delaying the official announcement by 24 hours had no
measurable effect on the overload since word of mouth and anticipation
had already built demand beyond the saturation point.

In short, what many of us have suspected for years turned out to be
proven rather abruptly true: The FreeBSD.org services infrastructure
has become overly reliant on resources which constitute single points
of failure and lacks both sufficient tiering and redundancy in the
face of such failure.  We lost a key FTP resource and our entire
distribution service essentially collapsed.  The same may be true for
CVSup, mail and WWW services and that's something we definitely need
to look at.

For now, the most critical item is to finish implementing the
discussed-but-not-implemented ftp-master site scheme.  A machine to
serve as the project's "master FTP host" has been procured will
shortly be available for FTP access.  ONLY mirror sites will be
allowed to connect to it, and it will always be pre-loaded with
release bits, CERT advisories and anything else which requires the
mirrors to have a head-start in advance of any public announcement.
Announcements will only be made after a hand-inspection of the mirror
sites reveals that a significant degree of propagation has taken place
from the master site and not beforehand.

What people currently regard as "ftp.freebsd.org" will become another
tier of sites, probably served in round-robin DNS fashion, which have
all agreed to meet the minimum requirements for being "full and
complete mirrors" of the bits offered from ftp-master.freebsd.org.
Existing mirror sites which wish to maintain only a subset of the
master bits will become clients of these second-tier sites and subject
to whatever distribution policies they institute.  It would be my
expectation that certain 2nd-tier sites will also have mirror access
lists which favor the 3rd-tier sites over end-users, but that's a
detail to be worked out over time.

It's also not clear what, if anything, needs to be done to make sites
like www.freebsd.org and hub.freebsd.org more reliable in the face of
failure, but people can certainly expect that a good deal of thought
will be going into answering those questions over the next few weeks.

It certainly would have been nice to have had all of this been fixed
before the release of 4.3, of course, and it's somewhat ironic that
discussions about these very problems were occurring right around the
same time, but it sometimes requires a truly painful experience to
actually force an organization (or an individual for that matter) to
both recognize and act on its shortcomings.

The failure of the FTP site, the timing of FreeBSD 4.3 RELEASE and
Wind River's acquisition of BSDi's BSD assets were simply unfortunate
coincidences which had no causal link and I'd also be happy if people
would stop inferring as much.  Wind River has, in fact, indicated some
interest in keeping the ftp.freesoftware.com site alive and we simply
need to figure out the best way of doing so.  As one might expect, a
Gigabit connection and collocation facilities for a full 19" rack's
worth of equipment doesn't come cheap!

I will be releasing more information as the situation develops,
both here and to the hubs@freebsd.org mailing list.  Thanks,

- Jordan

This is the moderated mailing list freebsd-announce.
The list contains announcements of new FreeBSD capabilities,
important events and project milestones.
See also the FreeBSD Web pages at http://www.freebsd.org


To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-announce" in the body of the message