By Tammi Nguyen

It is the first day on a new project and I am on the client’s site to sit in on the kick-off meeting that introduces Critical Logic to the key players on the project team.  My manager is getting ready to make a presentation on how the project will benefit from our process and the services provided by our people.  Sitting through the presentation, I quietly observe the reaction of the people in the room.  I notice that most people are making an effort to actively listen to the presentation while others are uncomfortably fidgeting in their chairs.  The presentation progresses and we are met with a lot of questions and requests for clarification.  There is an individual in the room that seems particularly tuned into the presentation only to refute the benefits of our services.  The conversation escalates and this person becomes agitated - a good indication that he is not quite ready for the change that is about to take place on the project.  The director of the project who has also been quietly observing people during the presentation has to intervene to bring things to order. Despite the hiccup, the presentation ends on a good note and everyone thanks us as they exit the conference room.  I take a deep breath and smile at my manager as we are collecting our things.  We have about an hour to regroup before our next meeting with the project team starts.

Most projects that I have been assigned to work on have started out this way, but they vastly differ beyond this point.  No two projects have ever been the same.  Even when projects are similar in nature, the amount of support from management and resources allocated to a project have always determined the direction and pace of the project.  What can and does remain the same is the level of commitment and professionalism we provide on the project.  Critical Logic’s mantra has always been to focus on providing our clients with expertise in requirements validation and test development which leads to defect-free software at the functional level.  Flexibility is among one of the great advantages of the client-contractor relationship with Critical Logic.  We will adapt to the project’s budget, schedule, processes, and standards as a part of our service and commitment to you.  In short, we are here to serve the needs of the project and ensure the overall quality on the final product.

Even with all the benefits of contracting, many people are still reserved about working with contractors.  The level of resistance can be used as a gauge of how well a project is progressing.  In my experience, the more resistance there is to share information with us, the more at-risk a project is in of not meeting key milestones and delivery dates.  In many cases, the struggle to obtain information is often due to incomplete requirements that are still in progress.  At Critical Logic, our philosophy is to embed ourselves in the project right from the start. This means that we can begin analyzing information as soon as it becomes available.
Being able to provide meaningful feedback early in the process is only the first step in a series of steps that leads to clear-cut requirements.  Our ambiguity review process provides a thorough dissection of the requirements documents to sort out questions that may lead to defects in development.  We have consistently found defects are generally introduced during the requirements phase of a given project.  The cost to fix these defects increases exponentially the later in the development cycle the defect is discovered.

Another large component of our process is to provide our clients with 100% test coverage with traceability.  Using our own technology, we leverage the clarified requirements developed since the start of the project to produce comprehensive test cases.  Individual requirements can be traced back to one or more test cases.  The format of the test cases can be customized to fit the needs of the client.  Upon delivery, the test cases come with a certification package that will provide an explanation of the test coverage for each set of test cases.  We guarantee complete test coverage.

Critical Logic has a long history of having successful relationships as contractors.  I have found that one key factor to a successful client-contractor relationship is that contractors are vested only in the project.  Our goal is to provide our client with the best service and our work remains unaffected by internal events or decisions made by the organization that can sometimes cloud objectivity.  Our process is transparent to give our clients full visibility to our productivity by providing our clients with weekly metrics to track our progress.  As experts in the field, we provide support and training for all our clients to promote a better understanding in forming a partnership between the organizations.  This is often seen as the desirable alternative to hiring and training new resources onto the project.  So despite the common reservations regarding the utilization of contracting services, the list of benefits provided by contractors goes on and on.  There are a number of variables that cannot be controlled in a given project… however the one thing that you can always count on is the consistency and efficiency in services that Critical Logic can bring to your organization as the experts in our field.

So here’s the question: what do IT budgets and government entitlement programs have in common? And here’s the answer: both are budgeted and funded on the basis of entitlement. While government budgeting for entitlement programs is familiar, the IT corollary, and its implications for software development processes, is what we intend to focus on here.

How can an IT organization determine and justify its software development budget? In principle, budgets should tie business requirements to the effort necessary to deliver those requirements in new or existing systems. The results should be reliable, with the business having every expectation that high-quality software will be consistently delivered within the budget allocated.

Unfortunately, most IT organizations cannot do this type of budgeting. After all, how does the developmental budget document the fact that 20% of development projects are never delivered at all, or that 40% will substantially exceed project-level cost estimates and timeframes? How many CIOs could get approval for a budget that explicitly allocates 40–50% of development resources to rework (again, the industry average)? Obviously, another rationale for funding is needed.

Our observation is that IT budgets end up being derived from past spending patterns rather than clear cost-to-deliver calculations. The budget is loosely based on head-count and fixed costs, with adjustments for the corporate cost goals or special funding for large, high-visibility software initiatives. In effect, like government entitlement, funding is largely predetermined. If the budget last year was $2 million, then, unless there is a major business disruption, it will be about the same the following year.

Unfortunately, this budget approach hides the core drivers that determine how ineffective the organization’s software development processes really are. In a disruptive economic situation such as we are now experiencing, this leaves CIOs without a meaningful way to price and guarantee delivery of high-quality software at the lowest cost. Technologies and processes that could eliminate substantial rework costs or improve quality get ignored because those costs are not directly evaluated in the budget process.

We strongly encourage our clients to start measuring and understanding the cost drivers that truly impact their software development budgets. It’s also important that they measure rework rates and root causes of major quality problems. This gives CIOs the information to make the right technology and process investments for a more reliable and higher quality software delivery. And that is the best way to ensure adequate, rational IT budgets that everyone’s entitled to.

Imagine you are an architect (the house kind, not the IT kind). A client comes to you and says, “I need a four-bedroom, three-bathroom house with living room, dining room and a big kitchen with all of the latest equipment. It should be large enough to meet all of my lifestyle needs. Now, tell me how much it will cost.” Could you give an answer? Would you give an answer? Probably not, because there is not enough information.

Yet how many times in IT software development does the business sponsor challenge us with the same type of question: “We want a system that does such-and-such, but tell me the total cost before you start.” We have brought a lot of this on ourselves by being notoriously poor at estimating the effort to build software and then delivering on that estimate. Some 60% of all projects are substantially over their original estimates, so you can’t blame business for wanting to somehow get a fence around its IT expenditures.

Two Costs, Not One

In reality, there are two costs to worry about: the cost to define the software and the cost to build and implement the resulting application. Allowing separate estimates for each activity can control cost risks and improve accuracy of estimates. But the trick is to be very clear about the exit criteria for each of these efforts, especially the define phase. Define means completely specify the expected behavior of the system: all business rules, every GUI, all user navigation, each electronic interface. The user should know exactly what the system will look like. Build and implement covers the technical software design, construction and testing of the software as well as the transition into production.

Traditionally, too much define work gets merged into the software build and implement activities, which not only makes it hard to segregate costs, but also drives up the cost by adding expensive definition rework. So being sure that the define phase meets its goals is critical. Reliable completion criteria for define phase specifications are

• Developers can build the system without requiring additional analysis and feedback from the business and functional analysts
• Detailed functional test cases can be designed from the specifications
• User manuals and training materials can be drafted from the defined specifications

Reduce Total Cost

Since the majority of defects come from poor or incomplete requirements, a strong define phase virtually eliminates the opportunity for requirements defects to be introduced into code. Total software defects and associated rework costs plummet. Properly executed, this project structure can reduce total costs by 20–30% and add predictability to schedules. Cost risks are better managed. Further, it still allows for agile or iterative software construction and implementation.

One of the things we observe too often in our engagements is quality being adversely impacted by seemingly reasonable project management decisions. We are not talking about the obvious quality-killers of tasks with impossible time frames or inadequate resources (dollars or people) for the amount of effort required. Rather, there are decisions that a PM makes during the project that sound good at the time but have serious impact on the testing and delivery phases. Let’s look at three examples:

Switch to iterative. Sometime during design and coding, the PM decides to deliver code to the test team in several iterations rather than as a single release. Makes sense to the developers, but it adds a major burden to the testing. Now for each iteration, not only must the new functionality be tested, but functionality delivered in previous iterations must be thoroughly regressed. Before switching the delivery strategy, the PM needs to understand and account for the impact on regression testing.

Assumptions. Every project estimate and plan comes with assumptions. Assumptions are to be managed, not assumed. We see too many cases where test schedules are based on important assumptions about specifications, test environments, or test data. The PM fails to ensure these assumptions hold true, then is surprised when the test effort is delayed because the need has not been fulfilled. In our proposals, we are relabeling assumptions as dependencies to ensure they get the attention they deserve.

The Project vs. MS Project. Just because MS Project says a task is done does not make it so. Is the work truly complete? Has the user of the task’s deliverables accepted them, without being under pressure to “meet the date?” The reality is that there is no free lunch; you always get caught by taking shortcuts, and it usually is when testing commences that the pain begins. Shortcuts in requirements definition become excessive defects. Shortcuts in test coverage translate into long nights during implementation, when the bugs surface.

Any time the project tasks are restructured in mid-project for the purpose of shaving time or effort, be very careful what the impact is on the testing effort. You may be adding to the testing burden or extending the rework costs by lowering the likely quality of the delivered application. Nobody wins in that case.