After automation modeling on two projects since starting with Critical Logic, I can honestly say one thing: automation is a huge benefit for our clients.  In most of the projects I have done, our deliverable was only 100% coverage test cases.  Imagine the power of then taking that product and automating it.  With the integration between DTT and TMX, we can!  It’s like having twin turbo kits installed into a Ford Mustang. They’re not required for the car to be great, and it may take some time and money in order to have them installed, but once they’re installed, the car has greater value and much better performance. The same principle applies with automated test cases.

Since starting with Critical Logic, I’ve been involved with many major projects, and all of them have had at least a hundred test cases or more. Some have had up to a thousand test cases. Imagine having to run all of those manually every time you have to do a regression test. This is the benefit of automation. Enter our modeling technology –  automated tests that can be easily maintained. Making a change to the model changes all of the test cases simultaneously, a process that saves our clients time and money. Imagine what could be accomplished in all the time that was saved.

Modeling for automation is something that takes time, though, and it is something that has to be done perfectly. Once we have received a design spec from a client and have completed an ambiguity review, we can begin modeling. In order for the automation to work, the model descriptions must match what is going on inside the software. Even the most obvious typo will be recorded as a defect in automation. Every item or value that needs to be verified, every button that is clicked or value that is entered into a field, and every identification done by page names or field names all have to be exact. That is part of the work we do on the test development (i.e., “modeling”) side in order to make the automation fly. By going through this process we gain further understanding of the inputs and outputs of the “Black Box”.  This is a tremendous benefit for our clients, because it acts as another method to make sure everything is working as it should. Just imagine what the Ford Mustang might run like if those turbo kits were not installed correctly. We want to give our clients the best work we can possibly deliver.

It is a process, and it does take time, but I have heard reviews from clients about how much time has been saved for them because of automation. Not every test case can be automated; some will have to be done manually, such as test cases that also use programs outside of the one that is being tested. However, a majority of them probably can be scripted and automated, and that alone can help save enough time and money to be worth ten times over all the time it takes to get the automation right.

Bottom line, my experience with automation is this: our test cases by themselves are like the Ford Mustang without all the nice trimmings that can be added. The Mustang is still a great car, and our test cases still cover 100% of the functionality in the specification.  With some extra work, though, we can make those test cases blast.

A project manager owns schedule and budget.  This holds true whether you’re an engineering project manager on a construction site or a software project manager on a new software application.

Despite the similarities, the software project manager has it worse.  Her deliverables are more abstract, the dependencies she manages tend to be less obvious, and the specifications her teams work with are, on average, woefully inadequate compared to the specifications used in construction.  All of this makes software projects less predictable, and thus more difficult to manage effectively.

Given the inherent difficulties of dealing in such an abstract environment, one might assume that software would go to even greater lengths than construction to define and document the interconnected layers of the system, from the highest level end-user need down to the lowest technical foundational level.

Shockingly, the reverse is true.

Study after study reveals a software industry hopelessly mired in the muck created by incomplete specs.  Nearly 2/3 of all software defects are traceable back to incorrect or incomplete requirements.  On average, half of the final budget of a large software project is spent after the first code is delivered for test.  This is not a good result for project managers who thought they were 80% done.

Granted, some systems require less documentation.  The contractor who re-landscaped my backyard was able to meet my needs based on a few sketches and periodic conversations about design decisions.

But complexity isn’t the only factor.  Risk must also be considered.  If it were my front yard, I would have worked with more detailed specifications.  Rework is expensive.

A recent NY Times piece on how Google develops new software provides a case in point for how lack of specification results in unpredictability.  The article, entitled ”Bug by Bug, Google Fixes a New Idea,” is a day in the life of a test engineer at Google, working on a new tool to push ads to mobile phones—not an overly complex system, but one that generates ad revenue, affecting the bottom line.

http://www.nytimes.com/2009/10/05/business/media/05adco.html

In one segment the test engineer was asked, “Can we stay on schedule?”  “It depends on how the testing goes,” she replied. “We’re not expecting any disasters, but you never expect any disasters.”  What is her definition of disaster?  It’s one thing to not plan for earthquakes, power outages, and tornadoes.  But something did come up that did delay their release date, and it was not an act of God.

A new kind of phone that had not been in the original list needed to be compatible with the new tool.  And some older phones had been misclassified and not included.  The specification for what phones should be included was incorrect, incomplete, and evidently not managed through formal change control.

The form of testing that was employed—an experiment employing select end-users with the new software compared with a control group—was itself unpredictable, and it betrays what was likely an incomplete functional specification.

A complete specification would specify the exact variables per phone type that mattered to the software, and document expected behavior for each variation per phone.  This would allow complete functional testing of the software, without the need for a dragnet approach based on probabilities.  The approach that was used gave no clear definition of what/how much functionality had been exercised, beyond a guess based on probabilities.

Google pulled it off, which is common on systems of this size.  They had some delays, and probably plenty of production defects, but it worked well enough.  But how much more money was spent developing it than planned (because of the late rework and schedule delays)?  What was the impact on other projects the test engineer and developers were scheduled to work on?  And what was the impact from lost ad revenue to Google’s bottom line?

One of the dirty secrets of software development is that you can get away with poor software specifications for the exact same reason that it is absolutely essential to develop good software specifications.  The stuff is so abstract and complex, it’s sometimes hard for project managers to know whether the documentation is sufficient.  Sadly, too often they don’t find out until their project goes 30% over budget.