20 October 2009 - 15:54 To Spec or Not to Spec? The Project Manager’s Dilemma by Adam Richards, QA Manager

A project manager owns schedule and budget.  This holds true whether you’re an engineering project manager on a construction site or a software project manager on a new software application.

Despite the similarities, the software project manager has it worse.  Her deliverables are more abstract, the dependencies she manages tend to be less obvious, and the specifications her teams work with are, on average, woefully inadequate compared to the specifications used in construction.  All of this makes software projects less predictable, and thus more difficult to manage effectively.

Given the inherent difficulties of dealing in such an abstract environment, one might assume that software would go to even greater lengths than construction to define and document the interconnected layers of the system, from the highest level end-user need down to the lowest technical foundational level.

Shockingly, the reverse is true.

Study after study reveals a software industry hopelessly mired in the muck created by incomplete specs.  Nearly 2/3 of all software defects are traceable back to incorrect or incomplete requirements.  On average, half of the final budget of a large software project is spent after the first code is delivered for test.  This is not a good result for project managers who thought they were 80% done.

Granted, some systems require less documentation.  The contractor who re-landscaped my backyard was able to meet my needs based on a few sketches and periodic conversations about design decisions.

But complexity isn’t the only factor.  Risk must also be considered.  If it were my front yard, I would have worked with more detailed specifications.  Rework is expensive.

A recent NY Times piece on how Google develops new software provides a case in point for how lack of specification results in unpredictability.  The article, entitled ”Bug by Bug, Google Fixes a New Idea,” is a day in the life of a test engineer at Google, working on a new tool to push ads to mobile phones—not an overly complex system, but one that generates ad revenue, affecting the bottom line.

http://www.nytimes.com/2009/10/05/business/media/05adco.html

In one segment the test engineer was asked, “Can we stay on schedule?”  “It depends on how the testing goes,” she replied. “We’re not expecting any disasters, but you never expect any disasters.”  What is her definition of disaster?  It’s one thing to not plan for earthquakes, power outages, and tornadoes.  But something did come up that did delay their release date, and it was not an act of God.

A new kind of phone that had not been in the original list needed to be compatible with the new tool.  And some older phones had been misclassified and not included.  The specification for what phones should be included was incorrect, incomplete, and evidently not managed through formal change control.

The form of testing that was employed—an experiment employing select end-users with the new software compared with a control group—was itself unpredictable, and it betrays what was likely an incomplete functional specification.

A complete specification would specify the exact variables per phone type that mattered to the software, and document expected behavior for each variation per phone.  This would allow complete functional testing of the software, without the need for a dragnet approach based on probabilities.  The approach that was used gave no clear definition of what/how much functionality had been exercised, beyond a guess based on probabilities.

Google pulled it off, which is common on systems of this size.  They had some delays, and probably plenty of production defects, but it worked well enough.  But how much more money was spent developing it than planned (because of the late rework and schedule delays)?  What was the impact on other projects the test engineer and developers were scheduled to work on?  And what was the impact from lost ad revenue to Google’s bottom line?

One of the dirty secrets of software development is that you can get away with poor software specifications for the exact same reason that it is absolutely essential to develop good software specifications.  The stuff is so abstract and complex, it’s sometimes hard for project managers to know whether the documentation is sufficient.  Sadly, too often they don’t find out until their project goes 30% over budget.