CentOS - May 2005 - Re: Demonizing generic Linux issues as Fedora Core-only issues -- WAS: Hi, Bryan

From: Les Mikesell <lesmikesell at gmail.com>
> Yes, but... whose choice was it to ship 2.6 with lots of broken
> and omitted stuff when 2.4 works better for many things?
Again, 2-2-2, 6-6-6

At some point, Red Hat has to start the new series for "early
adopters."
That means being the first to adopt the new GLibC, GCC, kernel, etc...
Looking at just the GLibC 2+ generations (post-LibC4/5)...

EL0 series:  GLibC 2.0, GCC 2.8.1.1, kernel 2.0
EL1 series:  GLibC 2.1, GCC 2.91.66 (EGCS 1.1.2), kernel 2.2
EL2 series:  GLibC 2.2, GCC 2.96, kernel 2.4
EL3 series:  GLibC 2.3, GCC 3.2, kernel 2.4
EL4 series:  GLibC 2.4, GCC 3.4, kernel 2.6

In each 2-4 "Community Linux" revisions of those "Enterprise
Linux" series,
Red Hat minimizes changes.  Typically the .0 can go through a number of
refinements from .0 to .1, but then they taper off in preparation for the
.2 and 18 month EL series.

But even what I call CL4.0 (Fedora Core 2) wasn't nearly as much of a
shift as CL0.0 (Red Hat Linux 5.0), and still a bit better than CL2.0
(Red Hat Linux 7[.0]).  One of my only complaint starting with Red Hat
Linux 9 (_well_before_ Fedora Core) is the lack of revisions to warn us of
".0" like shifts.
> Just because a developer writes some code and tacks on a higher
> version number doesn't mean it's ready for prime time.
Correct.  I never said so and I have been _very_critical_ of Red Hat no
longer using revisions to tell us that its either a major shift (.0) or
an evolutionary update (.1, .2, etc...).  And this was _before_ evne
Fedora Core.

Red Hat should have called Red Hat Linux 9 as Red Hat Linux 8.1.
Yes, they added NTPL, but there have always been a few changes
from a .0 to a .1 as things are more finalized.  Every .0 revision has
made some poor considerations in a few regards, and after the
"early adopters" start using it, they are exposed.

The reality is that most things are _not_ exposed _until_ people
actually start using things.  Even Red Hat Linux 7[.0] was regression
tested to the anal power with GCC 2.96 and its ANSI C++ requirement,
and it wasn't until after Red Hat released it that the projects that
weren't writing ANSI C++ compliant software took note.

People like to throw up the fact that Red Hat included GCC 2.91.66
(EGCS 1.1.2) for kernel building and that's why they should have gone
with GCC 2.96, but I will return point out that many distros were shipping
GCC 2.95 as well, and GCC 2.95 had a lot of issues with various software
too.
> Well, an occasional rant is therapeutic - or at least cheaper than
> smashing the equipment that no longer works after you upgrade the
> software.
Constructive rants are good.  They identify real technical issues.  But
"brand name" rants are typically full of ignorance, blaming companies
for things outside their control, or outside their focus (e.g., features
on a SLA-centric distro).  Again, I think it's good to see projects like
CentOS, but that doesn't mean you have to complain about Red Hat
on things that are not Red Hat's doing or focus.

The principles of configuration management are universal, and one must
be wary of any new version, regardless of where it comes from.  In the
past, Red Hat made this very easy with revisions.  With RHL9+, I think
Red Hat really needs to bring them back.
> I'm questioning the sensibility of shipping a kernel in what should be
> an upgrade
If Red Hat doesn't do it, another distro will.  And then that distro can
be chastized by people like yourself instead of Red Hat, and Red Hat
will benefit from being on the trail that other distro's issues.

I purposely kept my CL3.2 (Fedora Core 1) systems well after
CL4.0 (Fedora Core 2) came out, and didn't upgrade until CL4.1
(Fedora Core 3) was released.  I did the same of CL2.3
(Red Hat Linux 7.3) until CL3.1 (Red Hat Linux 9) came out, CL1.2
(Red Hat Linux 6.2) until CL2.1 (Red Hat Linux 7.1), etc...

It's best to wait on a new distro series until 1 year after it first starts
development, typically 6 months after it's first revision's release. 
Fedora Core 2 was _very_stable_, but it was _very_incompatible_ with
many things because of its changes.  Those compatibility issues will
_not_ get resolved until someone finally releases an "early adopter"
release and people then start throwing things at it.

Heck, even SuSE does this too.  And SuSE even came out with their
Linux 2.6-based enterprise distro much sooner than Red Hat.  So you could
easily "double" your chastizing of Red Hat towards SuSE in the same
regard.  At some point, Red Hat has to "give up" further development
on Linux 2.4, or anything else, and "move on."  There's nothing
stopping
you from delaying your adoption for 6+ months, or even 36+ months
(in the case of RHEL/CentOS).

I've had this argument over and over with people -- not just Linux.
You can't always blame vendors for your own professional negligence.
Sometimes vendors just ship what consumers want, even when they
tell them "we don't recommend this."  A perfect example was
Windows
95/98/Me -- something I _never_ allowed on my networks (only
Windows NT 3.1, 3.50, 3.51, 4.0 and 2000).  And that means you don't
just go off and upgrade when the vendors says "we've changed all
the core stuff" which is a definite sign that things might break.

Your option is always that you can stick with the older release.
Again, 6+ months for Fedora Core today, 36+ months for RHEL/
CentOS.
> with many things that don't work at all that worked in
> the previous version.
Of course not!  That's what change does!  ;->

Red Hat release 2-4 revisions every 6 months before changing things up.
SuSE does much of the same as well, and more distros are moving to
this model too.  Things break, but the idea is to maintain 2-4 revisions
over 12-24 months that don't.  In the case of Red Hat and SuSE, they also
release a distro every 18 months based on the "most mature" revision
and
then support that for 60+ months under a subscription model.

Otherwise, the overwhelming distros don't do such a good job, and _any_
release of theirs can dork up anything!  I'd really like to find a vendor
who balances "early adoption" with "anal SLA-level reliably"
so well and
for so long.  It's the 2-2-2, 6-6-6 model that everyone, even Novell-SuSE,
have adopted.
> As you point out, these aren't surprises.
> While I hate to complain about free software, I think that user's
> experiences are relevant and should be reported even if they are
> painful.  
That's fine, if you report them.  But then people go beyond that and start
"analyzing."  And when I say "analyzing" I mean they
"blame the vendor."
And the illogic doesn't stop there -- they will blame Red Hat for issues
with
CentOS, or not realize that Red Hat really did take time, effort and extensive
consideration to get it to work, and couldn't.

And not only that, but when other distros ran into the same issues.
> I'm not sure I believe that in the case of CIPE, since the 1.6 version
> specifically addresses the changes in the 2.6 kernel and was available
> well before FC3 or RHEL4 releases which did not include it again.
Both FC3 and RHEL3 started with FC2.  And FC2 has already been through
6 months of development and Fedora Core 3 was already entering into
Test by the time CIPE 1.6 was finally available.  And even then there were
issues -- let alone issues with security and the lack of interest.

At some point Red Hat crosses a threshhold where they have to say it's
too close to release date.  There's just not enough time to not only
integrate the package, but give its "due dilligence" in package test
(Rawhide/Development), distro regression test (Beta/Test) and post-release
resolution.

And when a package doesn't get modified to support kernel 2.6 until
some 9 months after kernel 2.6 release, and still has issues, at some
point, Red Hat's gotta say "this is going to cause us headaches to
support."
> If I were to speculate about intentions as you suggest above, I'd
> guess that someone at RedHat read the one negative review published
> about CIPE, decided it was no longer a selling point, and walked away
> from the integration they had done before.
And I'd say you haven't looked through Bugzilla.  I don't know how
many
times I've seen people complain about Red Hat and then a few of them
actually take the time to submit Bugzilla reports and then realize what
happens.  The few that do then change their tune.

Any and every major decision on EL begins with their development of CL.
And that has not varied in Fedora Core since the days of Red Hat Linux 4.0
on-ward.  If you want to see something change, you have to get involved.
And that means taking the time to understand what you _can_ do to see
things change.

It's very easy in the Red Hat Linux world.  And its even easier now that
Fedora Core is no longer the "product" that Red Hat Linux is, even
though
Red Hat Enterprise Linux has a 1:1 package relationship just like Red Hat
Linux prior.
> But of course I wouldn't speculate about something like that...
You don't need to.  Hit Bugzilla.

There are people who like to complain from ignorance.
And then there are people who like to complain from familiarity.

I typically choose to be the latter, although I'm sure I've done the
former
more than once too.
> It's a complicated system, and it isn't immediately obvious that
other
> distros have exactly the same problems.
Do you know how many times I hear people say that?  And I just have to
shake my head because it can't be categorized as anything else but
ignorance?

Do you know how many times I hear someone call something a "Fedora
Core 2-only issue" yet it plagues SuSE Linux 9.1, even SuSE Linux
Enterprise Server (SLES) 9 and Mandrake 10.x as well?  And they will
actually sit there and say "it's only a Fedora Core 2 issue" no
matter
how many times I send them links?

Red Hat has its model and it is extremely effective.  But it also means
that you have to understand why the "end" of 2-2-2 and 6-6-6 is more
"reliable" than the "beginning."  Fedora Core 2 was the
beginning.
Red Hat Enterprise Linux 4 was the end.  At some point, if you keep
introducing latest'n greatest that still have compatibility issues, you
are going to affect the quality of the end.
> If you follow www.distrowatch.com for a few months you'll see that they
> each keep trying to improve things in different ways. 
Yes, and the overwhelming majority of distros on distrowatch.com don't
have a good release model, and they break all sorts of things everytime
they come out with a new version -- at least in comparison to Red Hat
or SuSE.
> Switching distros should always be an option - that's one of the
> big reasons for using open source.
But sometimes people switch distros out of "brand name blame" instead
of actual "technical focus."  I don't know how many times I've
seen people
argue over 2 distros that use the _exact_same_ base!  I've literally
had people argue to death that APT is Deb-centric, and the RPM world
doesn't have anything equivalent.

And sometimes I think people aren't "up-to-date" on different
distros
because they used one from years ago, and their preference more currently.
As someone who deploys Debian, Fedora, Gentoo, SuSE and RHEL and SLES
regularly, there's a lot of overlap.
> But, in spite of what I consider an occasional bad choice,
Like CIPE I assume?  At what point are you going to recognize that
CIPE was really "no choice" until Fedora Core 3 was in Test?
> Fedora/RH seem to be among the best and I do give the fact that
> they've been responsible for getting buggy software in front of a
> vast number of people credit for it eventually being fixed.
"Buggy"?  Or "incompatible?"  Big difference!

If GNU Tar is incompatible with POSIX, and someone introduces a
new version or replacement program, like Star, that is, do you know
how much the "ignorant majority" absolutely derail that person?

Or when LibC5 is not getting the attention it needs, has security
issues, yet GLibC2 is active, solves a lot of security and compatibility
issues, etc...?

Or when lack of ANSI C++ compliance hurts compatibility with future
code, and GCC 3 is being finally pushed?  Or when the fact that
GCC 2.95 was adopted by many distros even when the GCC team
said they should not and stick with GCC 2.91.66 (EGCS 1.1.2) which
is proven (and the kernel uses)?
> If you remember the state of free software before RH 4.0 was
> released you will know what I mean.  I do sometimes wonder
> how things would have turned out if someone had built an equally
> easy to install CD based on freebsd first, though.  
Who says it isn't and I don't also deploy FreeBSD and NetBSD?
[ If you own the book "Samba Unleashed," you might open up the
Appendix on BSD UNIX and see someone's name you recognize. ;-]
> Hmmm, just who would you blame for WindowsME?
Oh, that was Microsoft.

Although oeople professionally negligent enough to ever allow _any_
"Chicago" release on their network were the start of the issue.
Windows Me was Microsoft's attempt to get their own application
developers to stop using "Chicago" interfaces.
The result was that they created a release that was both incompatible
and unreliable.

As I always said about Windows, you had your choice between
incompatible (NT) and unreliable ("Chicago").  I choose the former,
and I was in the minority.  But I didn't have 98% of the problems
other people had.
> Who would you blame for the upgrade that made interfaces not
> start if the hardware address in the configs didn't match
> the NIC?  That was bad news for my remote machines where all
> the drives had been cloned and the IP addresses set before
> shipping.
I'd blame you because you cloned, not installed.  ;->
You can't blame Red Hat for an install approach they don't test for.

[ BTW, Ethernet hardware addresses are used in IPv6. ]
> I'm not positive about this, but I think it happened
> at about the same time across Fedora/RH/Centos updates instead
> of following the scenario you mentioned for testing things that
> are supposed to change behavior.
What version?  I can tell you exactly if it was a major or minor change.

.0 always changes stuff.
Sometimes .1 changes a few things that didn't make it or needed to
be addressed.
.2 rarely does, as well as the "enterprise" release.


--
Bryan J. Smith   mailto:b.j.smith at ieee.org

On Wed, 2005-05-25 at 13:51, Bryan J. Smith  wrote:
> The reality is that most things are _not_ exposed _until_ people
> actually start using things.  
Agreed.  But note that the standards are set long before that...
> Your option is always that you can stick with the older release.
> Again, 6+ months for Fedora Core today, 36+ months for RHEL/
> CentOS.
If you are doing anything complicated, I'm not sure you have any
other choice. But then you run into ugliness where you want to
run perl programs that need perl=>5.8.3 to get character set handling
right, and you can't, or you have to do a local install trying not
to break the system version.  Or you need mysql 4.x for transactions.
> > with many things that don't work at all that worked in
> > the previous version.
> 
> Of course not!  That's what change does!  ;->
I guess my real complaint is that the bugfixes and improvements to
the old releases that you are forced by the situation to run are
minimal as soon as it becomes obvious that the future belongs to
a different version - that may be a long time in stabilizing to
a usable point. 
> Otherwise, the overwhelming distros don't do such a good job, and _any_
> release of theirs can dork up anything!  I'd really like to find a
vendor
> who balances "early adoption" with "anal SLA-level
reliably" so well and
> for so long.  It's the 2-2-2, 6-6-6 model that everyone, even
Novell-SuSE,
> have adopted.
It will be interesting to see how this works out for Ubuntu.  I think it
would be possible to be more aggressive with the application versions
but hold back more than RH on the kernel changes.  Of course they'll
have it easy for the first few since they don't have to be backwards
compatible to a huge installed base.
> Both FC3 and RHEL3 started with FC2.  And FC2 has already been through
> 6 months of development and Fedora Core 3 was already entering into
> Test by the time CIPE 1.6 was finally available.
How was the CIPE author supposed to know that what would be released as
FC2 would have a changed kernel interface?  He did respond with 1.6
as quickly as could be expected after a released distribution didn't
work with it and a user reported problems on the mailing list.
> And when a package doesn't get modified to support kernel 2.6 until
> some 9 months after kernel 2.6 release, and still has issues, at some
> point, Red Hat's gotta say "this is going to cause us headaches to
support."
About the kernel, or all of the drivers' authors that assumed that
kernel interfaces shouldn't change?
> If you want to see something change, you have to get involved.
> And that means taking the time to understand what you _can_ do to see
> things change.
And if you want the things that work to not change???
> > It's a complicated system, and it isn't immediately obvious
that other
> > distros have exactly the same problems.
> 
> Do you know how many times I hear people say that?  And I just have to
> shake my head because it can't be categorized as anything else but
> ignorance?
Where do you find the real info?  Specific example: I'm trying to make a
software RAID1 partition with one internal IDE and a matching external
drive in a firewire case work.  FC1 worked flawlessly once the raid was
established but would not autodetect the drive, either when hotplugged
or at bootup.  Everything later that I've tried is worse. Some get the
autodetect right on a hotplug but crash after some amount of runtime.
Is any distro likely to autodetect at bootup in time to cleanly connect
the RAID and then keep working?  Maybe it matters that the filesystem
is reiserfs - I see some bug reports about that, but rarely have
problems when the internal IDE is running as a broken mirror.
> > But, in spite of what I consider an occasional bad choice,
> 
> Like CIPE I assume?  At what point are you going to recognize that
> CIPE was really "no choice" until Fedora Core 3 was in Test?
Like letting it be a surprise to the CIPE author that it didn't work
at release time.  I don't remember seeing a hint about this on the
CIPE mail list when someone must have known well ahead that it was
coming.
> > Fedora/RH seem to be among the best and I do give the fact that
> > they've been responsible for getting buggy software in front of a
> > vast number of people credit for it eventually being fixed.
> 
> "Buggy"?  Or "incompatible?"  Big difference!
Buggy. I don't mean RH bugs, I mean bugs from the upstream packages
that got their first huge real world exposure because RH bundled them
on a CD that you could drop into just about any PC and have a working
system.  I mean things like the bind, sendmail, and ssh versions that
went out with RH 4.0 and were installed on more machines than ever
before - all with remote exploits that weren't known yet.
> If GNU Tar is incompatible with POSIX, and someone introduces a
> new version or replacement program, like Star, that is, do you know
> how much the "ignorant majority" absolutely derail that person?
The real world is a complicated place.  If you want to substitute
a different program for an old but non-standard one you need to
make sure it handles all the same things.  Until just recently,
star didn't even come close to getting incrementals right and still.
unlike gnutar, requires the runs to be on filesystem boundaries for
incrementals to work.  And, it probably doesn't handle the options
that amanda needs.  Theory, meet practice.
> > If you remember the state of free software before RH 4.0 was
> > released you will know what I mean.  I do sometimes wonder
> > how things would have turned out if someone had built an equally
> > easy to install CD based on freebsd first, though.  
> 
> Who says it isn't and I don't also deploy FreeBSD and NetBSD?
By 'first', I meant before RH 4.x, which in my opinion really drove
the popularity of Linux simply because it had a decent installer
that came up working on most hardware at the time.  Freebsd was
around at the time but in a form that was much more difficult
for a new user to get working.
> As I always said about Windows, you had your choice between
> incompatible (NT) and unreliable ("Chicago").  I choose the
former,
> and I was in the minority.  But I didn't have 98% of the problems
> other people had.
That depends on when you started.  Before SP6, NT was also unreliable.
Remember that in the timeframe of RH4, you could kill an NT box
by sending it an oversized ping packet - and have a fair chance of
corrupting the filesystem beyond repair from the ungraceful shutdown.
> > Who would you blame for the upgrade that made interfaces not
> > start if the hardware address in the configs didn't match
> > the NIC?  That was bad news for my remote machines where all
> > the drives had been cloned and the IP addresses set before
> > shipping.
> 
> I'd blame you because you cloned, not installed.  ;->
It wasn't so much the cloning as it was setting the IP address
with their GUI tool (and I did that because when I just made
the change to the /etc/sysconfig/network-scripts/ifcfg-xxx file
the way that worked in earlier RH versions it sometimes mysteriously
didn't work).  Then I shipped the disks in their hot-swap carriers to
the remote sites to be installed.
> You can't blame Red Hat for an install approach they don't test
for.
That procedure can't be uncommon.  And the same thing would happen
if you swapped a NIC card - hmmm, I wonder about PCMCIA cards now.
> > I'm not positive about this, but I think it happened
> > at about the same time across Fedora/RH/Centos updates instead
> > of following the scenario you mentioned for testing things that
> > are supposed to change behavior.
> 
> What version?  I can tell you exactly if it was a major or minor change.
I'm not sure exactly when it happened because it was an update that
did not include a new kernel so the machines weren't rebooted
immediately.  I think the ones where I noticed it first were somewhere
in the Centos 3.4 range with some subsequent updates (maybe a month
or so ago).
> .0 always changes stuff.
> Sometimes .1 changes a few things that didn't make it or needed to
> be addressed.
> .2 rarely does, as well as the "enterprise" release.
I think this was just an update without a version number change.  As it
turned out, I saw the screen of one that was rebooted at a nearby site
and knew more or less what happened but before I got around to fixing
them all a machine crashed at a distant site and I had to talk someone
who didn't know vi through finding the file and removing the hardware
address entry.  If I hadn't already known about it by then or if a
lot of the remote servers had been rebooted and came up with no network
access it might have ruined my whole day.

-- 
  Les Mikesell
    lesmikesell at gmail.com