Jordan Hubbard
2001-May-02 20:58 UTC
ANNOUNCE: Status update on ftp.freebsd.org AKA ftp.freesoftware.com
Hi folks, This is just a short note to discuss the current state of affairs with our master FTP site, explain what we're doing about it and combat some of the FUD floating around which has everyone from space aliens to Wind River Systems (who supply the space aliens with navigational software, of course) intentionally killing the site off. On April 20th, 2001, the day that FreeBSD 4.3 was due to be released, we lost access to our main FTP site, ftp.freesoftware.com, without any advance warning. Further investigation revealed that the problem was due to an unforeseen network outage which caused the hosting ISP to have to block access to the site. With one of their major links down, it was overloading the ISPs backup links and denying bandwidth to their other customers. It was also not clear to us, then or now, just how long the outage would last and whether this was to be a short-term or a long-term problem. With the release date for 4.3 already well publicized in advance and many people asking me for it, I decided to use one of our backup sites, the usw[1-6].freebsd.org cluster, and at least get the current releases bits up somewhere on an interim basis. Unfortunately, I really underestimated both the extent of the demand and the degree to which our big, fat Gigabit pipe at ftp.freesoftware.com has made distributing the bits to our mirrors look comparatively easy. Even with the login limits set *significantly* greater in favor of the registered mirror sites, and with multiple machines pulling the load, it quickly overwhelmed the new hosting infrastructure and resulted in significant bandwidth limiting on FTP traffic being instituted by the ISP. Even delaying the official announcement by 24 hours had no measurable effect on the overload since word of mouth and anticipation had already built demand beyond the saturation point. In short, what many of us have suspected for years turned out to be proven rather abruptly true: The FreeBSD.org services infrastructure has become overly reliant on resources which constitute single points of failure and lacks both sufficient tiering and redundancy in the face of such failure. We lost a key FTP resource and our entire distribution service essentially collapsed. The same may be true for CVSup, mail and WWW services and that's something we definitely need to look at. For now, the most critical item is to finish implementing the discussed-but-not-implemented ftp-master site scheme. A machine to serve as the project's "master FTP host" has been procured will shortly be available for FTP access. ONLY mirror sites will be allowed to connect to it, and it will always be pre-loaded with release bits, CERT advisories and anything else which requires the mirrors to have a head-start in advance of any public announcement. Announcements will only be made after a hand-inspection of the mirror sites reveals that a significant degree of propagation has taken place from the master site and not beforehand. What people currently regard as "ftp.freebsd.org" will become another tier of sites, probably served in round-robin DNS fashion, which have all agreed to meet the minimum requirements for being "full and complete mirrors" of the bits offered from ftp-master.freebsd.org. Existing mirror sites which wish to maintain only a subset of the master bits will become clients of these second-tier sites and subject to whatever distribution policies they institute. It would be my expectation that certain 2nd-tier sites will also have mirror access lists which favor the 3rd-tier sites over end-users, but that's a detail to be worked out over time. It's also not clear what, if anything, needs to be done to make sites like www.freebsd.org and hub.freebsd.org more reliable in the face of failure, but people can certainly expect that a good deal of thought will be going into answering those questions over the next few weeks. It certainly would have been nice to have had all of this been fixed before the release of 4.3, of course, and it's somewhat ironic that discussions about these very problems were occurring right around the same time, but it sometimes requires a truly painful experience to actually force an organization (or an individual for that matter) to both recognize and act on its shortcomings. The failure of the FTP site, the timing of FreeBSD 4.3 RELEASE and Wind River's acquisition of BSDi's BSD assets were simply unfortunate coincidences which had no causal link and I'd also be happy if people would stop inferring as much. Wind River has, in fact, indicated some interest in keeping the ftp.freesoftware.com site alive and we simply need to figure out the best way of doing so. As one might expect, a Gigabit connection and collocation facilities for a full 19" rack's worth of equipment doesn't come cheap! I will be releasing more information as the situation develops, both here and to the hubs@freebsd.org mailing list. Thanks, - Jordan This is the moderated mailing list freebsd-announce. The list contains announcements of new FreeBSD capabilities, important events and project milestones. See also the FreeBSD Web pages at http://www.freebsd.org To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-announce" in the body of the message