On Mon, Dec 29, 2014 at 3:03 PM, Warren Young <wyml at etr-usa.com> wrote:> > As a software developer, I think I can speak to both halves of that point. > > First, the world where you design, build, and deploy The System is disappearing fast.Sure, if you don't care if you lose data, you can skip those steps. Lots of free services that call everything they release 'beta' can get away with that, and when it breaks it's not the developer answering the phones if anyone answers at all.> The world is moving toward incrementalism, where the first version of The System is the smallest thing that can possibly do anyone any good. That is deployed ASAP, and is then built up incrementally over years. >That works if it was designed for rolling updates. Most stuff isn't, some stuff can't be.> Instead of trying to go from 0 to 100 over the course of ~7 years, you deliver new functionality to production every 1-4 weeks, achieving 100% of the desired feature set over the course of years.If you are, say, adding up dollars, how many times do you want that functionality to change?> This isn?t pie-in-the-sky theoretical BS. This is the way I?ve been developing software for decades, as have a great many others. Waterfall is dead, hallelujah! >How many people do you have answering the phone about the wild and crazy changes you are introducing weekly? How much does it cost to train them?> I don?t mean that glibly. I mean you have made a fundamental mistake if your system breaks badly enough due to an OS change that you can?t fix it within an iteration or two of your normal development process. The most likely mistake is staffing your team entirely with people who have never been through a platform shift before.Please quantify that. How much should a business expect to spend per person to re-train their operations staff to keep their systems working across a required OS update? Not to add functionality. To keep something that was working running the way it was? And separately, how much developer time would you expect to spend to follow the changes and perhaps eventually make something work better?> Again, this is not theoretical bloviation. The software system I?ve been working on for the past 2 decades has been through several of these platform changes. It started on x86 SVR4, migrated to Linux, bounced around several distros, and occasionally gets updated for whatever version of OS X or FreeBSD someone is toying with at the moment.How many customers for your service did you keep running non-stop across those transitions? Or are you actually talking about providing a reliable service?> Everyone?s moaning about systemd, and how it?s taking over the Linux world, as if it would be better if Red Hat kept on with systemd and all the other Linux distro providers shunned it. Complain about its weaknesses if it you like, but at least it?s looking to be a real de facto standard going forward.Again, it's only useful to talk about if you can quantify the cost. What you expect to pay to re-train operations staff -just- for this change, -just- to keep things working the same.. And separately, what will it cost in development time to take advantage of any new functionality?>> So, seven, even ten, years of stability is really nothing at all. And as >> Linux seeks to enter into more and more profoundly valuable employment the >> type of changes that we witnessed from v6 to v7 are simply not going to be >> tolerated. > > Every other OS provider does this.> (Those not in the process of dying, at any rate. A corpse is stable, but that?s no basis for recommending the widespread assumption of ambient temperature.) > > Windows? Check. (Vista, Windows 8, Windows CE/Pocket PC/Windows Mobile/Windows RT/Windows Phone)We've got lots of stuff that will drop into Windows server versions spanning well over a 10 year range. And operators that don't have a lot of special training on the differences between them.> And when all these breakages occurred, what was the cry heard throughout the land of punditry? ?This is Linux?s chance! Having forced everyone to rewrite their software [bogus claim], Bad OS will make everyone move to Linux!? Except it doesn?t happen. Interesting, no?No, Linux doesn't offer stability either.> Could it be that software for these other platforms *also* manages to ride through major breaking changes? >Were you paying attention when Microsoft wanted to make XP obsolete? There is a lot of it still running.>> What enterprise can afford to rewrite all of its software >> every ten years? > > Straw man.Not really. Ask the IRS what platform they use. And estimate what it is going to cost us when they change.>> What enterprise can afford to retrain all of its personnel to >> use different tools to accomplish the exact same tasks every seven years? > > Answer: Every enterprise that wants to remain an enterprise. > > This is exactly what happens with Windows and Apple, only on a bit swifter pace, typically. > > (The long dragging life of XP is an exception. Don?t expect it to occur ever again.)No, that is the way things work. And the reason Microsoft is in business.> Tell that to Google.With their eternally beta software? With the ability to just drop things they don't feel like supporting any more? Not everyone has that luxury.> What, you think they?re still building Linux boxes based on the same kernel 2.2 custom distro they were using when they started in the mid 1990s? > > We don?t have to guess, they?ve told us how they coped: > > http://events.linuxfoundation.org/sites/events/files/lcjp13_merlin.pdf > > Check out the slide titled "How did that strategy of patching Red Hat 7.1 work out?? > > Read through the rest of it, for that matter. > > If you come away from it with ?Yeah, that?s what I?m telling you, this is a hard problem!? you?re probably missing the point, which is that while your resources aren?t as extensive as Google?s, your problem isn?t nearly as big as Google?s, either.So again, quantify that. How much should it cost a business _just_ to keep working the same way? And why do you think it is a good thing for this to be a hard problem or for every individual user to be forced to solve it himself?> Bottom line: This is the job. This is what you get paid to do.But it could be better, if anyone cared. -- Les Mikesell lesmikesell at gmail.com
On Dec 29, 2014, at 4:03 PM, Les Mikesell <lesmikesell at gmail.com> wrote:> On Mon, Dec 29, 2014 at 3:03 PM, Warren Young <wyml at etr-usa.com> wrote: >> >> the world where you design, build, and deploy The System is disappearing fast. > > Sure, if you don't care if you lose data, you can skip those steps.How did you jump from incremental feature roll-outs to data loss? There is no necessary connection there. In fact, I?d say you have a bigger risk of data loss when moving between two systems released years apart than two systems released a month apart. That?s a huge software market in its own right: legacy data conversion. If your software is DBMS-backed and a new feature changes the schema, you can use one of the many available systems for managing schema versions. Or, roll your own; it isn?t hard. You test before rolling something to production, and you run backups so that if all else fails, you can roll back to the prior version. None of this is revolutionary. It?s just what you do, every day.> when it breaks it's not the developer answering > the phones if anyone answers at all.Tech support calls shouldn?t go straight to the developers under any development model, short of sole proprietorship, and not even then, if you can get away with it. There needs to be at least one layer of buffering in there: train up the secretary to some basic level of cluefulness, do everything via email, or even hire some dedicated support staff. It simply costs too much to break a developer out of flow to allow a customer to ring a bell on a developer?s desk at will.>> The world is moving toward incrementalism, where the first version of The System is the smallest thing that can possibly do anyone any good. That is deployed ASAP, and is then built up incrementally over years. > > That works if it was designed for rolling updates. Most stuff isn?t,Since we?re contrasting with waterfall development processes that may last many years, but not decades, I?d say the error has already been made if you?re still working with a waterfall-based methodology today. The first strong cases for agile development processes were first made about 15 years ago, so anything started 7 years ago (to use the OP?s example) was already disregarding a shift a full software generation old.> some stuff can't be.Very little software must be developed in waterfall fashion. Avionics systems and nuclear power plant control systems, for example. Such systems make up a tiny fraction of all software produced. A lot of commercial direct-to-consumer software also cannot be delivered incrementally, but only because the alternative messes with the upgrade treadmill business model. Last time I checked, this sort of software only accounted for about ~5% of all software produced, and that fraction is likely dropping, with the moves toward cloud services, open source software, subscription software, and subsidized software. The vast majority of software developed is in-house stuff, where the developers and the users *can* enter into an agile delivery cycle.>> Instead of trying to go from 0 to 100 over the course of ~7 years, you deliver new functionality to production every 1-4 weeks, achieving 100% of the desired feature set over the course of years. > > If you are, say, adding up dollars, how many times do you want that > functionality to change?I?m not sure what you?re asking. If you?re talking about a custom accounting system, the GAAP rules change several times a year in the US: http://www.fasb.org/jsp/FASB/Page/SectionPage&cid=1176156316498 The last formal standard put out by FASB was 2009, and they?re working on another version all the time. Chances are good that if you start a new 7-year project, a new standard will be out before you finish. If instead you?re talking about the cumulative cost of incremental change, it shouldn?t be much different than the cost of a single big-bang change covering the same period. In fact, I?d bet the incremental changes are easier to adopt, since each change can be learned piecemeal. A lot of what people are crying about with EL7 comes down to the fact that Red Hat is basically doing waterfall development: many years of cumulative change gets dumped on our HDDs in one big lump. Compare a rolling release model like that of Cygwin or Ubuntu (not LTS). Something might break every few months, which sounds bad until you consider that the alternative is for *everything* to break at the same time, every 3-7 years. I?m not arguing for CentOS/RHEL to turn into Ubuntu Desktop. I?m just saying that there is a cost for stability: every 3-7 years, you must hack your way through a big-bang change bolus. (6-7 years being for those organizations that skip every other major release by taking advantage of the way the EL versions overlap. EL5 was still sunsetting as EL7 was rising.)>> This isn?t pie-in-the-sky theoretical BS. This is the way I?ve been developing software for decades, as have a great many others. Waterfall is dead, hallelujah! > > How many people do you have answering the phone about the wild and > crazy changes you are introducing weekly?The burden of tech support has more to do with proper QA and roll-out strategies than with the frequency of updates. For the most part, we roll new code to a site in response to a support call, rather than field calls in response to an update. The new version solves their problem, and we don?t hear back from them for months or years. We don?t update all sites to every new release. We merely ship *a* new release every 1-4 weeks, which goes out to whoever needs the new features and fixes. It?s also what goes out on each new server we ship.> How much does it cost to train them?Most of our sites get only one training session, shortly after the new system is first set up. We rarely get asked to do any follow-up training. The users typically pick up on the incremental feature updates as they happen, without any additional help from us. We attribute that to solid UX design. That first session is mostly about giving the new users an idea of what the system can do. We teach them enough to teach themselves. How often do most people get trained to use a word processor? I?ll bet a lot of people got trained just once, in grade school. They just cope with changes as they come. The worst changes are when you skip many versions. Word 97 to Word 2007, for example. *shudder*>> I don?t mean that glibly. I mean you have made a fundamental mistake if your system breaks badly enough due to an OS change that you can?t fix it within an iteration or two of your normal development process. The most likely mistake is staffing your team entirely with people who have never been through a platform shift before. > > Please quantify that. How much should a business expect to spend per > person to re-train their operations staff to keep their systems > working across a required OS update? Not to add functionality. To > keep something that was working running the way it was?If you hire competent people, you pay zero extra to do this, because this is the job they have been hired to do. That's pretty much what IT/custom development is: coping with churn. Most everything you do on a daily basis is a reaction to some change external to the IT/development organization: - Capacity increases - Obsolete ?ware upgrades - New seat/site deployments - Failed equipment replacements - Compatibility breakage repair (superseded de facto standard, old de jure standard replaced, old proprietary item no longer available?) - Tracking business rule change (GAAP, regulations, mergers?) - Effecting business change (entering new markets, automation, solving new problems developing from new situations?) - Tracking business strategy change (new CEO, market shift?) Setting aside retail software development, IT and internal development organizations *should* be chasing this kind of thing, not being ?proactive.? We?re not trying to surprise our users with things they didn?t even ask for, we?re trying to solve their problems. Maybe we solve problems in a *manner* our users did not expect ? hopefully a better way ? but we?re rarely trying to innovate, as such.> how much developer time would you expect to spend to > follow the changes and perhaps eventually make something work better?Pretty much 100%, after subtracting overhead. (Meetings, email, breaks, reading?) Again: This is what we do. Some new thing happens in the world, and we go out and solve the resulting problems. The only question is one of velocity: the more staff you add, the faster you go. So, how fast do you want to go? (Yes, I?ve read ?The Mythical Man Month.? The truths within that fine book don?t change the fact that Microsoft can develop a new OS faster than I can all by my lonesome.)>> The software system I?ve been working on for the past 2 decades has been through several of these platform changes. > > How many customers for your service did you keep running non-stop > across those transitions?Most of our customers are K-12 schools, so we?re not talking about a 24/7 system to begin with. K-12 runs maybe 9 hours a day (7am - 4pm), 5 days a week, 9 months out of the year. That gives us many upgrade windows. We rarely change out hardware or the OS at a particular site. We generally run it until it falls over, dead. This means we?re still building binaries for EL3. This also means our software must *remain* broadly portable. When we talk about porting to EL7, we don?t mean that it stops working on EL6 and earlier. We might have some graceful feature degradation where the older OS simply can?t do something the newer one can, but we don?t just chop off an old OS because a new one came out. All that having been said, we do occasionally roll a change to a site, live. We can usually do it in such a way that the site users never even notice the change, except for the changed behavior. This is not remarkable. It?s one of the benefits you get from modern centralized software development and deployment stacks.>> Everyone?s moaning about systemd...at least it?s looking to be a real de facto standard going forward. > > What you expect to pay to re-train operations staff -just- for this > change, -just- to keep things working the same..You ask that as if you think you have a no-cost option in the question of how to address the churn. Your only choices are: 1. Don?t upgrade 2. Upgrade and cope 3. Switch to something else Each path carries a cost. You think path 1 is free? If you skip EL7, you?re just batching up the changes. You?ll pay eventually, when you finally adopt a new platform. One change set plus one change set equals about 1.9 change sets, plus compound penalties. Penalties? Yes. You know the old joke about how you eat an elephant? [*] By the time you eat 1.9 elephants, you?ve probably built up another ~0.3 change sets worth of new problems. Time you spend grinding through nearly two full change sets is time you don?t spend keeping your current backlog short. We call this technical debt in the software development world. It?s fine to take out a bit of technical debt occasionally, as long as you don?t let it build up too long. The longer you let it build, the more the interest & penalties accrue, so the harder it is to pay down.> We've got lots of stuff that will drop into Windows server versions > spanning well over a 10 year range.Yes, well, Linux has always had a problem with ABI stability. Apparently the industry doesn?t really care about this, evidenced by the fizzling of LSB, and the current attacks on the work at freedesktop.org. Apparently we?d all rather be fractious than learn to get along well enough that we can nail down some real standards. Once again, though, there?s a fine distinction between stable and moribund.> And operators that don't have a > lot of special training on the differences between them.I?ve never done much with Windows Server, but my sense is that they have plenty of churn over in their world, too. We?ve got SELinux and SystemD, they?ve got UAC, SxS DLLs, API deprecation, and tools that shuffle positions on every release. (Where did they move the IPv4 configuration dialog this time?!) We get worked up here about things like the loss of 32-bit support, but over in MS land, they get API-of-the-year. JET, ODBC, OLE DB, or ADO? Win32, .NET desktop, Silverlight, or Metro? GDI, WinG, DirectX, Windows Forms or XAML? On and on, and that?s just if you stay within the MSDN walls.>> Could it be that software for these other platforms *also* manages to ride through major breaking changes? > > Were you paying attention when Microsoft wanted to make XP obsolete? > There is a lot of it still running.Were you paying attention when Target?s XP-based POS terminals all got pwned? Stability and compatibility are not universal goods.>>> What enterprise can afford to rewrite all of its software >>> every ten years? >> >> Straw man. > > Not really. Ask the IRS what platform they use. And estimate what > it is going to cost us when they change.Monopolies are inherently inefficient and plodding. Government is special only because it is the biggest monopoly. (That?s why we have antitrust law: not because it?s good for the consumer, but because it fights the trend toward zaibatsu rule.) Few organizations are working under such stringent constraints, if only because it?s a danger to the health of the organization. Only monopolies can get away with it.>> (The long dragging life of XP is an exception. Don?t expect it to occur ever again.) > > No, that is the way things work. And the reason Microsoft is in business.Microsoft stopped retail sale of Windows 7 a few months ago, and Vista back in April. A few months ago, there was a big stink when MS killed off Windows 8.0 updates, requiring that everyone upgrade to 8.1. Yes, I know about downgrade rights for pro versions of Windows. Nevertheless, the writing is on the wall.>> while your resources aren?t as extensive as Google?s, your problem isn?t nearly as big as Google?s, either. > > So again, quantify that. How much should it cost a business _just_ to > keep working the same way?Google already did that cost/benefit calculation: they tried staying on RH 7.1 indefinitely, and thereby built up 10 years of technical debt. Then when they did jump, it was a major undertaking, though one they apparently felt was worth doing. There?s a cost to staying put, too.> And why do you think it is a good thing > for this to be a hard problem or for every individual user to be > forced to solve it himself?I never said it was a good thing. I?m just reporting some observations from the field. ????? [*] One bite at a time.
On Mon, Dec 29, 2014 at 8:04 PM, Warren Young <wyml at etr-usa.com> wrote:> >>> >>> the world where you design, build, and deploy The System is disappearing fast. >> >> Sure, if you don't care if you lose data, you can skip those steps. > > How did you jump from incremental feature roll-outs to data loss? There is no necessary connection there.No, it's not necessary for either code interfaces or data structures to change in backward-incompatible ways. But the people who push one kind of change aren't likely to care about the other either.> In fact, I?d say you have a bigger risk of data loss when moving between two systems released years apart than two systems released a month apart. That?s a huge software market in its own right: legacy data conversion.I'm not really arguing about the timing of changes, I'm concerned about the cost of unnecessary user interface changes, code interface breakage, and data incompatibility, regardless of when it happens. RHEL's reason for existence is that it mostly shields users from that within a major release. That doesn't make it better when it happens when you are forced to move to the next one.> If your software is DBMS-backed and a new feature changes the schema, you can use one of the many available systems for managing schema versions. Or, roll your own; it isn?t hard.Are you offering to do it for free?> You test before rolling something to production, and you run backups so that if all else fails, you can roll back to the prior version.That's fine if you have one machine and can afford to shut down while you make something work. Most businesses aren't like that.> None of this is revolutionary. It?s just what you do, every day.And it is time consuming and expensive.>> when it breaks it's not the developer answering >> the phones if anyone answers at all. > > Tech support calls shouldn?t go straight to the developers under any development model, short of sole proprietorship, and not even then, if you can get away with it. There needs to be at least one layer of buffering in there: train up the secretary to some basic level of cluefulness, do everything via email, or even hire some dedicated support staff. > > It simply costs too much to break a developer out of flow to allow a customer to ring a bell on a developer?s desk at will.Beg your pardon? How about not breaking the things that trigger the calls in the first place - or taking some responsibility for it. Do you think other people have nothing better to do?> Since we?re contrasting with waterfall development processes that may last many years, but not decades, I?d say the error has already been made if you?re still working with a waterfall-based methodology today. >We never change more than half of a load-balenced set of servers at once. So all changes have to be compatible when running concurrently, or worth rolling out a whole replacement farm.>> some stuff can't be. > > Very little software must be developed in waterfall fashion.If you run continuous services you either have to be able to run new/old concurrently or completely duplicate your server farm as you roll out incompatible clients.> Last time I checked, this sort of software only accounted for about ~5% of all software produced, and that fraction is likely dropping, with the moves toward cloud services, open source software, subscription software, and subsidized software. > > The vast majority of software developed is in-house stuff, where the developers and the users *can* enter into an agile delivery cycle.OK, but they have to not break existing interfaces when they do that. And that's not the case with OS upgrades.>> If you are, say, adding up dollars, how many times do you want that >> functionality to change? > > I?m not sure what you?re asking.I'm asking if computer science has advanced to the point where adding up a total needs new functionality, or if you would like the same total for the same numbers that you would have gotten last year. Or more to the point, if the same program ran correctly last year, wouldn't it be nice if it still ran the same way this year, in spite of the OS upgrade you need to do because of the security bugs that keep getting shipped while developers spend their time making arbitrary changes to user interfaces.> Compare a rolling release model like that of Cygwin or Ubuntu (not LTS). Something might break every few months, which sounds bad until you consider that the alternative is for *everything* to break at the same time, every 3-7 years.When your system requires extensive testing, the few times it breaks the better. Never would be nice...>>> I don?t mean that glibly. I mean you have made a fundamental mistake if your system breaks badly enough due to an OS change that you can?t fix it within an iteration or two of your normal development process. The most likely mistake is staffing your team entirely with people who have never been through a platform shift before. >> >> Please quantify that. How much should a business expect to spend per >> person to re-train their operations staff to keep their systems >> working across a required OS update? Not to add functionality. To >> keep something that was working running the way it was? > > If you hire competent people, you pay zero extra to do this, because this is the job they have been hired to do.That's nonsense for any complex system. There are always _many_ different OS versions in play and many different development groups that only understand a subset, and every new change they need to know about costs time and risks mistakes.> That's pretty much what IT/custom development is: coping with churn.And it is expensive. Unnecessarily so, in my opinion.>> How many customers for your service did you keep running non-stop >> across those transitions? > > Most of our customers are K-12 schools, so we?re not talking about a 24/7 system to begin with. K-12 runs maybe 9 hours a day (7am - 4pm), 5 days a week, 9 months out of the year. That gives us many upgrade windows.That's a very different scenario than a farm of data servers that have to be available 24/7.> We rarely change out hardware or the OS at a particular site. We generally run it until it falls over, dead. > > This means we?re still building binaries for EL3.I have a few of those, but I don't believe that is a sane thing to recommend.> This also means our software must *remain* broadly portable. When we talk about porting to EL7, we don?t mean that it stops working on EL6 and earlier. We might have some graceful feature degradation where the older OS simply can?t do something the newer one can, but we don?t just chop off an old OS because a new one came out. >You'd probably be better off in java if you aren't already.>>> Everyone?s moaning about systemd...at least it?s looking to be a real de facto standard going forward. >> >> What you expect to pay to re-train operations staff -just- for this >> change, -just- to keep things working the same.. > > You ask that as if you think you have a no-cost option in the question of how to address the churn.I ask it as if I think that software developers could make changes without breaking existing interfaces. And yes, I do think they could if they cared about anyone who built on those interfaces.>> We've got lots of stuff that will drop into Windows server versions >> spanning well over a 10 year range. > > Yes, well, Linux has always had a problem with ABI stability. Apparently the industry doesn?t really care about this, evidenced by the fizzling of LSB, and the current attacks on the work at freedesktop.org. Apparently we?d all rather be fractious than learn to get along well enough that we can nail down some real standards.Well, that has done a great job of keeping Microsoft in business.> I?ve never done much with Windows Server, but my sense is that they have plenty of churn over in their world, too. We?ve got SELinux and SystemD, they?ve got UAC, SxS DLLs, API deprecation, and tools that shuffle positions on every release. (Where did they move the IPv4 configuration dialog this time?!) > > We get worked up here about things like the loss of 32-bit support, but over in MS land, they get API-of-the-year. JET, ODBC, OLE DB, or ADO? Win32, .NET desktop, Silverlight, or Metro? GDI, WinG, DirectX, Windows Forms or XAML? On and on, and that?s just if you stay within the MSDN walls.Yes, there are changes - and sometimes mysterious breakage. But an outright abandonment of an existing interface that breaks previously working code s pretty rare (and I don't like it when they do it either...).>> Were you paying attention when Microsoft wanted to make XP obsolete? >> There is a lot of it still running. > > Were you paying attention when Target?s XP-based POS terminals all got pwned? > > Stability and compatibility are not universal goods.Well, some things you have to get right in the first place - and then stability is good.> Google already did that cost/benefit calculation: they tried staying on RH 7.1 indefinitely, and thereby built up 10 years of technical debt. Then when they did jump, it was a major undertaking, though one they apparently felt was worth doing.And conversely, they felt is was worth _not_ doing for a very very long time. So can the rest of us wait until we have google's resources?>> And why do you think it is a good thing >> for this to be a hard problem or for every individual user to be >> forced to solve it himself? > > I never said it was a good thing. I?m just reporting some observations from the field.Maybe I misunderstood - I thought you were defending the status quo - and the fedora developers that bring it to us. -- Les Mikesell lesmikesell at gmail.com
On 12/29/2014 09:04 PM, Warren Young wrote:> On Dec 29, 2014, at 4:03 PM, Les Mikesell <lesmikesell at gmail.com> wrote: > >> On Mon, Dec 29, 2014 at 3:03 PM, Warren Young <wyml at etr-usa.com> wrote: >>> the world where you design, build, and deploy The System is disappearing fast. >> Sure, if you don't care if you lose data, you can skip those steps. > How did you jump from incremental feature roll-outs to data loss? There is no necessary connection there. > > In fact, I?d say you have a bigger risk of data loss when moving between two systems released years apart than two systems released a month apart. That?s a huge software market in its own right: legacy data conversion. > > If your software is DBMS-backed and a new feature changes the schema, you can use one of the many available systems for managing schema versions. Or, roll your own; it isn?t hard. > > You test before rolling something to production, and you run backups so that if all else fails, you can roll back to the prior version. > > None of this is revolutionary. It?s just what you do, every day. > >> when it breaks it's not the developer answering >> the phones if anyone answers at all. > Tech support calls shouldn?t go straight to the developers under any development model, short of sole proprietorship, and not even then, if you can get away with it. There needs to be at least one layer of buffering in there: train up the secretary to some basic level of cluefulness, do everything via email, or even hire some dedicated support staff. > > It simply costs too much to break a developer out of flow to allow a customer to ring a bell on a developer?s desk at will. > >>> The world is moving toward incrementalism, where the first version of The System is the smallest thing that can possibly do anyone any good. That is deployed ASAP, and is then built up incrementally over years. >> That works if it was designed for rolling updates. Most stuff isn?t, > Since we?re contrasting with waterfall development processes that may last many years, but not decades, I?d say the error has already been made if you?re still working with a waterfall-based methodology today. > > The first strong cases for agile development processes were first made about 15 years ago, so anything started 7 years ago (to use the OP?s example) was already disregarding a shift a full software generation old. > >> some stuff can't be. > Very little software must be developed in waterfall fashion. > > Avionics systems and nuclear power plant control systems, for example. Such systems make up a tiny fraction of all software produced. > > A lot of commercial direct-to-consumer software also cannot be delivered incrementally, but only because the alternative messes with the upgrade treadmill business model. > > Last time I checked, this sort of software only accounted for about ~5% of all software produced, and that fraction is likely dropping, with the moves toward cloud services, open source software, subscription software, and subsidized software. > > The vast majority of software developed is in-house stuff, where the developers and the users *can* enter into an agile delivery cycle.Where did you get the 5% from according to google there are "over 200 billion lines of existing COBOL code, much of it running mission-critical 24/7 applications, it is simply too costly (in the short run) for many organizations to convert." And what about Fortran, RPG etc. Also how big is the outfit you work for? Sounds like you have no shortage of help, a lot of place don't have unlimited resources like you seem to have.>>> Instead of trying to go from 0 to 100 over the course of ~7 years, you deliver new functionality to production every 1-4 weeks, achieving 100% of the desired feature set over the course of years. >> If you are, say, adding up dollars, how many times do you want that >> functionality to change? > I?m not sure what you?re asking. > > If you?re talking about a custom accounting system, the GAAP rules change several times a year in the US: > > http://www.fasb.org/jsp/FASB/Page/SectionPage&cid=1176156316498 > > The last formal standard put out by FASB was 2009, and they?re working on another version all the time. Chances are good that if you start a new 7-year project, a new standard will be out before you finish. > > If instead you?re talking about the cumulative cost of incremental change, it shouldn?t be much different than the cost of a single big-bang change covering the same period. > > In fact, I?d bet the incremental changes are easier to adopt, since each change can be learned piecemeal. A lot of what people are crying about with EL7 comes down to the fact that Red Hat is basically doing waterfall development: many years of cumulative change gets dumped on our HDDs in one big lump. > > Compare a rolling release model like that of Cygwin or Ubuntu (not LTS). Something might break every few months, which sounds bad until you consider that the alternative is for *everything* to break at the same time, every 3-7 years. > > I?m not arguing for CentOS/RHEL to turn into Ubuntu Desktop. I?m just saying that there is a cost for stability: every 3-7 years, you must hack your way through a big-bang change bolus. > > (6-7 years being for those organizations that skip every other major release by taking advantage of the way the EL versions overlap. EL5 was still sunsetting as EL7 was rising.) > >>> This isn?t pie-in-the-sky theoretical BS. This is the way I?ve been developing software for decades, as have a great many others. Waterfall is dead, hallelujah! >> How many people do you have answering the phone about the wild and >> crazy changes you are introducing weekly? > The burden of tech support has more to do with proper QA and roll-out strategies than with the frequency of updates. > > For the most part, we roll new code to a site in response to a support call, rather than field calls in response to an update. The new version solves their problem, and we don?t hear back from them for months or years. > > We don?t update all sites to every new release. We merely ship *a* new release every 1-4 weeks, which goes out to whoever needs the new features and fixes. It?s also what goes out on each new server we ship. > >> How much does it cost to train them? > Most of our sites get only one training session, shortly after the new system is first set up. > > We rarely get asked to do any follow-up training. The users typically pick up on the incremental feature updates as they happen, without any additional help from us. We attribute that to solid UX design. > > That first session is mostly about giving the new users an idea of what the system can do. We teach them enough to teach themselves. > > How often do most people get trained to use a word processor? I?ll bet a lot of people got trained just once, in grade school. They just cope with changes as they come. > > The worst changes are when you skip many versions. Word 97 to Word 2007, for example. *shudder* > >>> I don?t mean that glibly. I mean you have made a fundamental mistake if your system breaks badly enough due to an OS change that you can?t fix it within an iteration or two of your normal development process. The most likely mistake is staffing your team entirely with people who have never been through a platform shift before. >> Please quantify that. How much should a business expect to spend per >> person to re-train their operations staff to keep their systems >> working across a required OS update? Not to add functionality. To >> keep something that was working running the way it was? > If you hire competent people, you pay zero extra to do this, because this is the job they have been hired to do. > > That's pretty much what IT/custom development is: coping with churn. > > Most everything you do on a daily basis is a reaction to some change external to the IT/development organization: > > - Capacity increases > > - Obsolete ?ware upgrades > > - New seat/site deployments > > - Failed equipment replacements > > - Compatibility breakage repair (superseded de facto standard, old de jure standard replaced, old proprietary item no longer available?) > > - Tracking business rule change (GAAP, regulations, mergers?) > > - Effecting business change (entering new markets, automation, solving new problems developing from new situations?) > > - Tracking business strategy change (new CEO, market shift?) > > Setting aside retail software development, IT and internal development organizations *should* be chasing this kind of thing, not being ?proactive.? We?re not trying to surprise our users with things they didn?t even ask for, we?re trying to solve their problems. > > Maybe we solve problems in a *manner* our users did not expect ? hopefully a better way ? but we?re rarely trying to innovate, as such. > >> how much developer time would you expect to spend to >> follow the changes and perhaps eventually make something work better? > Pretty much 100%, after subtracting overhead. (Meetings, email, breaks, reading?) > > Again: This is what we do. Some new thing happens in the world, and we go out and solve the resulting problems. > > The only question is one of velocity: the more staff you add, the faster you go. So, how fast do you want to go? > > (Yes, I?ve read ?The Mythical Man Month.? The truths within that fine book don?t change the fact that Microsoft can develop a new OS faster than I can all by my lonesome.) > >>> The software system I?ve been working on for the past 2 decades has been through several of these platform changes. >> How many customers for your service did you keep running non-stop >> across those transitions? > Most of our customers are K-12 schools, so we?re not talking about a 24/7 system to begin with. K-12 runs maybe 9 hours a day (7am - 4pm), 5 days a week, 9 months out of the year. That gives us many upgrade windows. > > We rarely change out hardware or the OS at a particular site. We generally run it until it falls over, dead. > > This means we?re still building binaries for EL3. > > This also means our software must *remain* broadly portable. When we talk about porting to EL7, we don?t mean that it stops working on EL6 and earlier. We might have some graceful feature degradation where the older OS simply can?t do something the newer one can, but we don?t just chop off an old OS because a new one came out. > > All that having been said, we do occasionally roll a change to a site, live. We can usually do it in such a way that the site users never even notice the change, except for the changed behavior. > > This is not remarkable. It?s one of the benefits you get from modern centralized software development and deployment stacks. > >>> Everyone?s moaning about systemd...at least it?s looking to be a real de facto standard going forward. >> What you expect to pay to re-train operations staff -just- for this >> change, -just- to keep things working the same.. > You ask that as if you think you have a no-cost option in the question of how to address the churn. > > Your only choices are: > > 1. Don?t upgrade > > 2. Upgrade and cope > > 3. Switch to something else > > Each path carries a cost. > > You think path 1 is free? If you skip EL7, you?re just batching up the changes. You?ll pay eventually, when you finally adopt a new platform. One change set plus one change set equals about 1.9 change sets, plus compound penalties. > > Penalties? Yes. > > You know the old joke about how you eat an elephant? [*] By the time you eat 1.9 elephants, you?ve probably built up another ~0.3 change sets worth of new problems. Time you spend grinding through nearly two full change sets is time you don?t spend keeping your current backlog short. > > We call this technical debt in the software development world. It?s fine to take out a bit of technical debt occasionally, as long as you don?t let it build up too long. The longer you let it build, the more the interest & penalties accrue, so the harder it is to pay down. > >> We've got lots of stuff that will drop into Windows server versions >> spanning well over a 10 year range. > Yes, well, Linux has always had a problem with ABI stability. Apparently the industry doesn?t really care about this, evidenced by the fizzling of LSB, and the current attacks on the work at freedesktop.org. Apparently we?d all rather be fractious than learn to get along well enough that we can nail down some real standards. > > Once again, though, there?s a fine distinction between stable and moribund. > >> And operators that don't have a >> lot of special training on the differences between them. > I?ve never done much with Windows Server, but my sense is that they have plenty of churn over in their world, too. We?ve got SELinux and SystemD, they?ve got UAC, SxS DLLs, API deprecation, and tools that shuffle positions on every release. (Where did they move the IPv4 configuration dialog this time?!) > > We get worked up here about things like the loss of 32-bit support, but over in MS land, they get API-of-the-year. JET, ODBC, OLE DB, or ADO? Win32, .NET desktop, Silverlight, or Metro? GDI, WinG, DirectX, Windows Forms or XAML? On and on, and that?s just if you stay within the MSDN walls. > >>> Could it be that software for these other platforms *also* manages to ride through major breaking changes? >> Were you paying attention when Microsoft wanted to make XP obsolete? >> There is a lot of it still running. > Were you paying attention when Target?s XP-based POS terminals all got pwned? > > Stability and compatibility are not universal goods. > >>>> What enterprise can afford to rewrite all of its software >>>> every ten years? >>> Straw man. >> Not really. Ask the IRS what platform they use. And estimate what >> it is going to cost us when they change. > Monopolies are inherently inefficient and plodding. Government is special only because it is the biggest monopoly. > > (That?s why we have antitrust law: not because it?s good for the consumer, but because it fights the trend toward zaibatsu rule.) > > Few organizations are working under such stringent constraints, if only because it?s a danger to the health of the organization. Only monopolies can get away with it. > >>> (The long dragging life of XP is an exception. Don?t expect it to occur ever again.) >> No, that is the way things work. And the reason Microsoft is in business. > Microsoft stopped retail sale of Windows 7 a few months ago, and Vista back in April. > > A few months ago, there was a big stink when MS killed off Windows 8.0 updates, requiring that everyone upgrade to 8.1. > > Yes, I know about downgrade rights for pro versions of Windows. > > Nevertheless, the writing is on the wall. > >>> while your resources aren?t as extensive as Google?s, your problem isn?t nearly as big as Google?s, either. >> So again, quantify that. How much should it cost a business _just_ to >> keep working the same way? > Google already did that cost/benefit calculation: they tried staying on RH 7.1 indefinitely, and thereby built up 10 years of technical debt. Then when they did jump, it was a major undertaking, though one they apparently felt was worth doing. > > There?s a cost to staying put, too. > >> And why do you think it is a good thing >> for this to be a hard problem or for every individual user to be >> forced to solve it himself? > I never said it was a good thing. I?m just reporting some observations from the field. > > > ????? > > [*] One bite at a time. > _______________________________________________ > CentOS mailing list > CentOS at centos.org > http://lists.centos.org/mailman/listinfo/centos >-- Stephen Clark *NetWolves Managed Services, LLC.* Director of Technology Phone: 813-579-3200 Fax: 813-882-0209 Email: steve.clark at netwolves.com http://www.netwolves.com