Get it Right From The Start. Visit www.critical-logic.com

So here’s the question: what do IT budgets and government entitlement programs have in common? And here’s the answer: both are budgeted and funded on the basis of entitlement. While government budgeting for entitlement programs is familiar, the IT corollary, and its implications for software development processes, is what we intend to focus on here.

How can an IT organization determine and justify its software development budget? In principle, budgets should tie business requirements to the effort necessary to deliver those requirements in new or existing systems. The results should be reliable, with the business having every expectation that high-quality software will be consistently delivered within the budget allocated.

Unfortunately, most IT organizations cannot do this type of budgeting. After all, how does the developmental budget document the fact that 20% of development projects are never delivered at all, or that 40% will substantially exceed project-level cost estimates and timeframes? How many CIOs could get approval for a budget that explicitly allocates 40–50% of development resources to rework (again, the industry average)? Obviously, another rationale for funding is needed.

Our observation is that IT budgets end up being derived from past spending patterns rather than clear cost-to-deliver calculations. The budget is loosely based on head-count and fixed costs, with adjustments for the corporate cost goals or special funding for large, high-visibility software initiatives. In effect, like government entitlement, funding is largely predetermined. If the budget last year was $2 million, then, unless there is a major business disruption, it will be about the same the following year.

Unfortunately, this budget approach hides the core drivers that determine how ineffective the organization’s software development processes really are. In a disruptive economic situation such as we are now experiencing, this leaves CIOs without a meaningful way to price and guarantee delivery of high-quality software at the lowest cost. Technologies and processes that could eliminate substantial rework costs or improve quality get ignored because those costs are not directly evaluated in the budget process.

We strongly encourage our clients to start measuring and understanding the cost drivers that truly impact their software development budgets. It’s also important that they measure rework rates and root causes of major quality problems. This gives CIOs the information to make the right technology and process investments for a more reliable and higher quality software delivery. And that is the best way to ensure adequate, rational IT budgets that everyone’s entitled to.

Every year we attend several software development and testing conferences, including the STAR conferences. In principle, these events provide QA managers and CIOs with information on new advances in software quality technology. We look for important new technologies and services that truly will make a difference­—that is, new stuff that substantially raises the ability to routinely deliver complex software on time, on budget, and with few, if any, functional defects. We look for advancements that will make any IT organization more productive, not just a few of them.

And every year we come away disappointed. The solutions for software quality remain very ad hoc and situation specific. One person commented that the technical sessions all seem to boil down to “test harder and work smarter.” Training seminars represent a huge potpourri of approaches to testing, with no clear way of discerning which approach delivers reliably and under what circumstances. As a CIO, will my ability to deliver quality software benefit more from training in Exploratory Testing or the course on Systematic Testing? How will I know? On the technology side, everyone wants to buy or sell a tool. The “tool” CD has become the aspirin for relieving software quality woes. Unfortunately, aspirin is not the solution when the patient has pneumonia.

IT budgets are going to shrink in these economic times. And if we cannot deliver defect-free software within these tougher resource constraints, our companies will be at risk. The big cost- and quality-drivers are known: poor specification of software requirements followed up by poor and inconsistent testing. These issues double the cost of many software projects and add significant unnecessary cost to the rest. In the average IT budget, these avoidable costs represent big numbers.

It’s time to stop merely scratching the surface of software quality. Quality cost-drivers are not going to go away by buying a few seats of a $4,000 tool or sending a few QA folks to a training class and hoping for some incremental improvement. If you want your software development costs to substantially decrease, then it will require commitment to changing processes and skills as well as technology. The essential change is to begin integrating requirements specification with testing, particularly test case design. Create testable functional specifications and the suite of test cases that will fully exercise that functionality in the code—both at the same time, both before the code is written (with all the bugs introduced). Call it “requirements-based quality” or getting it “Right From The Start.” The processes and technology are different from traditional testing. Instead of writing test cases, test engineers evaluate the logical completeness and consistency of the functional specifications and let advanced technology write the detailed test cases. Software design and software testing proceed in tandem, not sequentially. Testing skills emphasize critical analysis of specifications over trying to “break” the software during testing.

Moving to a requirements-based quality approach does not represent risk. It works. The processes and supporting technologies are well understood and have been consistently proven in a wide range of software development environments and organizations. Implementation can be accomplished on an incremental, save-as-you-go basis. There are several very experienced resources in the marketplace to make your organization successful.

Software cost and quality pressures are only going to rise in the foreseeable future. It’s time to stop merely scratching the surface of the problem with another tool or class and get to the root of software costs and defects.

A short time ago we were at a meeting of IT executives. A major new CIO survey had been released, and, once again, one of the top ten concerns of CIOs is the impact poor requirements have on delivering software on time and on budget. So in the Q&A session, the question was posed: Since the requirements problem has been a top concern of CIOs for at least the last ten years, why hasn’t anyone done anything about it?

The answer was summed up succinctly by one of the CIO panelists: “Requirements are the elephant in the room that no one wants to talk about.” Poor requirements specification accounts for about 80% of all software development rework, and rework itself is typically half of the cost of software development overall. So it is a big elephant indeed, especially as budgets get tight in tougher economic times.

Why Is This?

After all, proven, reliable, technology-backed solutions are out there that guarantee correct implementation of requirements into applications with few defects and very low rework costs (we should know; we’ve been doing it for years!). What keeps the requirements problem on the CIO top-ten issues list? We think there are two factors.

How to Improve

First, improving requirements cannot be done by simply buying another tool and some training classes. It also requires changes to roles, responsibilities, and skills, especially in the QA and testing organization. Many IT organizations are simply unwilling or unable to make these changes. Managers are willing to take risks with new technology but unwilling to do so when it comes to changes in how the organization does its work.

Second, the financial rewards for delivering higher-quality software at less cost are difficult to recognize. Suppose the annual software development budget for an IT organization is $2 million. They implement proven requirements validation and testing solutions and remove $500,000 in testing and rework costs (a typical result). How does that “profit” benefit the IT organization? Better salaries? Probably not. More likely, it translates into additional projects or goes to mitigate our chronic tendency to under-estimate software development costs in the first place.

There is, however, an overriding incentive to recognize the elephant: customer satisfaction. When applications and releases are routinely delivered on time and with few, if any, functional defects, the business customer is spared a tremendous amount of frustration and anxiety, not to mention cost. The trust between the business customer and IT goes up substantially. In these times of budget austerity and limited investment, this can be critical to the success of the CIO and the business as a whole. So pack up the elephant gun and let’s go on a safari!

Imagine you are an architect (the house kind, not the IT kind). A client comes to you and says, “I need a four-bedroom, three-bathroom house with living room, dining room and a big kitchen with all of the latest equipment. It should be large enough to meet all of my lifestyle needs. Now, tell me how much it will cost.” Could you give an answer? Would you give an answer? Probably not, because there is not enough information.

Yet how many times in IT software development does the business sponsor challenge us with the same type of question: “We want a system that does such-and-such, but tell me the total cost before you start.” We have brought a lot of this on ourselves by being notoriously poor at estimating the effort to build software and then delivering on that estimate. Some 60% of all projects are substantially over their original estimates, so you can’t blame business for wanting to somehow get a fence around its IT expenditures.

Two Costs, Not One

In reality, there are two costs to worry about: the cost to define the software and the cost to build and implement the resulting application. Allowing separate estimates for each activity can control cost risks and improve accuracy of estimates. But the trick is to be very clear about the exit criteria for each of these efforts, especially the define phase. Define means completely specify the expected behavior of the system: all business rules, every GUI, all user navigation, each electronic interface. The user should know exactly what the system will look like. Build and implement covers the technical software design, construction and testing of the software as well as the transition into production.

Traditionally, too much define work gets merged into the software build and implement activities, which not only makes it hard to segregate costs, but also drives up the cost by adding expensive definition rework. So being sure that the define phase meets its goals is critical. Reliable completion criteria for define phase specifications are

• Developers can build the system without requiring additional analysis and feedback from the business and functional analysts
• Detailed functional test cases can be designed from the specifications
• User manuals and training materials can be drafted from the defined specifications

Reduce Total Cost

Since the majority of defects come from poor or incomplete requirements, a strong define phase virtually eliminates the opportunity for requirements defects to be introduced into code. Total software defects and associated rework costs plummet. Properly executed, this project structure can reduce total costs by 20–30% and add predictability to schedules. Cost risks are better managed. Further, it still allows for agile or iterative software construction and implementation.

One of the things we observe too often in our engagements is quality being adversely impacted by seemingly reasonable project management decisions. We are not talking about the obvious quality-killers of tasks with impossible time frames or inadequate resources (dollars or people) for the amount of effort required. Rather, there are decisions that a PM makes during the project that sound good at the time but have serious impact on the testing and delivery phases. Let’s look at three examples:

Switch to iterative. Sometime during design and coding, the PM decides to deliver code to the test team in several iterations rather than as a single release. Makes sense to the developers, but it adds a major burden to the testing. Now for each iteration, not only must the new functionality be tested, but functionality delivered in previous iterations must be thoroughly regressed. Before switching the delivery strategy, the PM needs to understand and account for the impact on regression testing.

Assumptions. Every project estimate and plan comes with assumptions. Assumptions are to be managed, not assumed. We see too many cases where test schedules are based on important assumptions about specifications, test environments, or test data. The PM fails to ensure these assumptions hold true, then is surprised when the test effort is delayed because the need has not been fulfilled. In our proposals, we are relabeling assumptions as dependencies to ensure they get the attention they deserve.

The Project vs. MS Project. Just because MS Project says a task is done does not make it so. Is the work truly complete? Has the user of the task’s deliverables accepted them, without being under pressure to “meet the date?” The reality is that there is no free lunch; you always get caught by taking shortcuts, and it usually is when testing commences that the pain begins. Shortcuts in requirements definition become excessive defects. Shortcuts in test coverage translate into long nights during implementation, when the bugs surface.

Any time the project tasks are restructured in mid-project for the purpose of shaving time or effort, be very careful what the impact is on the testing effort. You may be adding to the testing burden or extending the rework costs by lowering the likely quality of the delivered application. Nobody wins in that case.

Critical Logic has been holding a series of Webinars entitled “Accelerating IT Performance.” Based on the feedback, two major issues seem to be on most senior IT managers’ minds:

• “We seem to have a lot of challenges getting IT projects delivered successfully; is everyone else having problems?” and

• “What can I do about it?”

This edition of Critical Knowledge examines the first of these questions. Future editions will address the second.

Is Everyone Having These Problems? Yes!

We’ve been working and talking with senior IT professionals about project challenges for over two decades and their litany of problems never seems to change: missed deadlines, blown budgets, failure to deliver business value, and lack of quality.

We wondered if this is just the nature of who we talked to, or is it a global complaint?

To get an objective insight into the question, we looked at the various literature on the subject and found that the best seemed to be the bi-annual Standish Group’s Chaos reports. They’ve been looking at these issues since 1994, and their latest study (based on over 15,000 projects) shows that not only is there a problem, but people know it’s there, and know it’s big. How big? Twenty percent of IT projects completely fail or are cancelled, half are challenged (late, over budget, missed objectives) and only 35% are completely successful!

Why?

Saying there’s a problem begs the other half of the question: Why? Here again, our anecdotal evidence seems to be supported by what the studies are saying: lack of user/executive involvement and unclear/poorly communicated business objectives … But the single biggest factor is poor requirements (Forrester Group - 2006).*

We think that most IT execs would agree that given the first two causes, it’s not suprising that it’s so hard to get requirements right - and the code written from them.

Upcoming editions will look at what people are doing about it. Experience the difference of getting it Right from the Start™. Visit critical-logic.com or email moreinfo@critical-logic.com for more information.

The 2008 Desert Thunder Cup was held in Richland, Washington, on the Columbia River May 17th and 18th. The weekend saw unseasonably hot weather, with temperatures over 100 degrees, which was wonderful for the fans, but difficult for the racers. Not only did the racers have to battle the heat and sun, but also a fast-moving Columbia River swollen by a rapid mountain snow melt and compounded by a record-setting winter snowfall. The rapid current made the “first turn” (the downstream turn) extremely hazardous because the waves caused by the boats in the corner were held in place by the swirling water, making each lap more hazardous than the last.

Putting together an all-out effort for the final heat, the Miss Critical Logic crew, headed by Trevor Hull, tuned the motor perfectly for the thin air caused by the high temperatures. Driver Paul Becker and radio operator Donnie Smith strategized carefully against the league-leading UL-1 Miss Boat Electric Team and their driver, Kayleigh Perkins. The plan was to take lane two, forcing Kayleigh to the inside and into the rougher water in turn one. But the key to that winning strategy was a perfectly tuned motor and a great flying start that enabled the Miss Critical Logic to beat the UL-1 to the turn and hold Kayleigh to the buoy line and the rougher water, all of the way through the turn.

While the strategy worked out very well against the UL-1, Greg Hopp got a perfect start on the outside in the UL-15 Graham Trucking, and his significantly higher boat speed at the start carried him to victory (average speed 111.951 mph). The UL-14 Miss Critical Logic placed second (average speed 109.056 mph). Video of the final heat can been seen on the www.ulhra.org website in additional to other hydroplane news and upcoming events. Miss Critical Logic driver Paul Becker was seen smiling after the race, but with a large ice bag held on his right arm, which was badly bruised racing through that first turn. “Two races, twice the bridesmaid,” he said. “Look out in Chamberlain. We’re going to win that one!” Chamberlain, South Dakota, is the next event on the circuit, July 12th and 13th. See you there!

Congratulations to the Miss Critical Logic team for placing second at Phoenix Lakefest! Here is a photo taken from David McEldowney, CIO of Bar-S Foods Inc.

- Paul

We are happy to launch our very own blog. There’s so much on our minds these days and now we have a place where we can share these ideas with you. And conversely, we’d like to hear from our friends out there. This is the first of many entries to come!

 - Critical Logic