It is the first day on a new project and I am on the client’s site to sit in on the kick-off meeting that introduces Critical Logic to the key players on the project team.  My manager is getting ready to make a presentation on how the project will benefit from our process and the services provided by our people.  Sitting through the presentation, I quietly observe the reaction of the people in the room.  I notice that most people are making an effort to actively listen to the presentation while others are uncomfortably fidgeting in their chairs.  The presentation progresses and we are met with a lot of questions and requests for clarification.  There is an individual in the room that seems particularly tuned into the presentation only to refute the benefits of our services.  The conversation escalates and this person becomes agitated - a good indication that he is not quite ready for the change that is about to take place on the project.  The director of the project who has also been quietly observing people during the presentation has to intervene to bring things to order. Despite the hiccup, the presentation ends on a good note and everyone thanks us as they exit the conference room.  I take a deep breath and smile at my manager as we are collecting our things.  We have about an hour to regroup before our next meeting with the project team starts.

Most projects that I have been assigned to work on have started out this way, but they vastly differ beyond this point.  No two projects have ever been the same.  Even when projects are similar in nature, the amount of support from management and resources allocated to a project have always determined the direction and pace of the project.  What can and does remain the same is the level of commitment and professionalism we provide on the project.  Critical Logic’s mantra has always been to focus on providing our clients with expertise in requirements validation and test development which leads to defect-free software at the functional level.  Flexibility is among one of the great advantages of the client-contractor relationship with Critical Logic.  We will adapt to the project’s budget, schedule, processes, and standards as a part of our service and commitment to you.  In short, we are here to serve the needs of the project and ensure the overall quality on the final product.

Even with all the benefits of contracting, many people are still reserved about working with contractors.  The level of resistance can be used as a gauge of how well a project is progressing.  In my experience, the more resistance there is to share information with us, the more at-risk a project is in of not meeting key milestones and delivery dates.  In many cases, the struggle to obtain information is often due to incomplete requirements that are still in progress.  At Critical Logic, our philosophy is to embed ourselves in the project right from the start. This means that we can begin analyzing information as soon as it becomes available.
Being able to provide meaningful feedback early in the process is only the first step in a series of steps that leads to clear-cut requirements.  Our ambiguity review process provides a thorough dissection of the requirements documents to sort out questions that may lead to defects in development.  We have consistently found defects are generally introduced during the requirements phase of a given project.  The cost to fix these defects increases exponentially the later in the development cycle the defect is discovered.

Another large component of our process is to provide our clients with 100% test coverage with traceability.  Using our own technology, we leverage the clarified requirements developed since the start of the project to produce comprehensive test cases.  Individual requirements can be traced back to one or more test cases.  The format of the test cases can be customized to fit the needs of the client.  Upon delivery, the test cases come with a certification package that will provide an explanation of the test coverage for each set of test cases.  We guarantee complete test coverage.

Critical Logic has a long history of having successful relationships as contractors.  I have found that one key factor to a successful client-contractor relationship is that contractors are vested only in the project.  Our goal is to provide our client with the best service and our work remains unaffected by internal events or decisions made by the organization that can sometimes cloud objectivity.  Our process is transparent to give our clients full visibility to our productivity by providing our clients with weekly metrics to track our progress.  As experts in the field, we provide support and training for all our clients to promote a better understanding in forming a partnership between the organizations.  This is often seen as the desirable alternative to hiring and training new resources onto the project.  So despite the common reservations regarding the utilization of contracting services, the list of benefits provided by contractors goes on and on.  There are a number of variables that cannot be controlled in a given project… however the one thing that you can always count on is the consistency and efficiency in services that Critical Logic can bring to your organization as the experts in our field.

While the largest benefit to a project from functional modeling comes when it is introduced from the think-tank stage, functional modeling can be used long after the code is written for requirements thought to be chiseled in stone. It is often at this middle stage of the product development cycle that testing finds that incomplete or ambiguous requirements have caused code deficiencies. These deficiencies yield a mid-cycle quagmire of rework escalating bugs and burgeoning budgets.

Try the following quick exercise: draw three large circles, one on top of the other. Draw two lines coming from the middle circle. Draw three small circles in the middle circle and draw two small circles in the top circle. Draw a triangle on top of the top circle and draw another circle at the top of the triangle. Draw a curved line in the top circle underneath your two smaller circles. Tada! You can fill in the rest to be a season appropriate picture.

To explain why functional modeling can help mid-development, it helps to understand the process that leads to the point where management is asking for a snowman and development is handing over an alien hybrid. This happens because when ideas become abstracted into requirements and words alone try to capture intricate ideas and logic, some implied functionality or features become lost. While everyone knows that it should be, for example, a system for tracking inventory, and while each person has the same general concept of how the end product should work, each person comes away with a separate idea of exactly how this is accomplished. As with our “Use Case” of drawing the snowman, the written instructions will not always produce the envisioned outcome. Typically, when a project enters the aforementioned mid-cycle quagmire, the “instructions” (a.k.a. requirements) are then re-written and the code reworked to represent those changes. These changes tend to add new dimensions to the software and disconnect previously connected code chains. The fixes cause more fissures in the program than they solve. This, in turn, necessitates new instructions, new rework and even more fissures. The effect…. forgive the pun, snowballs. The budget and time allotted for the project also snowballs.

Functional modeling can help if you’ve just received an alien hybrid, or even if you are on the second generation of the alien hybrid. When introduced mid-project functional modeling circumvents the costly snowballing cycle of churn. Instead of distilling complex relationships into words alone, functional modeling can start providing pictures to accompany the text. When in a mid-cycle quandary, functional modeling is a sure-fire way to avoid complete derailment as the hot fixes and redevelopment avalanche through the project. To put a project back on track, and sometimes even ahead, functional modelers create an image of the current software (this step will often help clarify where the holes and broken links in the software already are) and then, after new requirements are received, model the proposed fixes and changes to determine interaction with the current functionality. Before developers even begin coding, faulty interactions with the current code and the proposed changes can be discovered in the process of functional modeling. This prevents the effort and expense of coding contradictory or ambiguous requirements that will necessitate fixes or delivery with bugs.

Even better than improved requirements, completed functional models provide an expansive picture to the developers because each step in the software logic, for each variation, is made known to them through the model. The software and development team can use the models as instructions to implement all needed changes and fixes without cross-examining several sets of instructions and external specifications. They can code these instructions confidently, as the “bugs” have been found in advance of their effort and corrected in advance in the functional models.

Over 50% of the cost in any software development project is rework caused by the mid-cycle quagmire described here. The pressure to deliver coupled with the rework often makes it necessary to deliver a hybrid alien snowman with missing features.  By using the flexible and adaptable tool of functional modeling, even after the project has started, it is possible to deliver a robust, full size snowman on time; not a Calvin and Hobbes special or melting snow pile because the season has changed.

Now I could go on and on with analogies, like the risk of death to you or your project if it has an underlying condition (e.g., a tight budget or timeline), but I will stop there. The important thing to remember is that there are process improvements available that will work like that vaccine—to prevent bugs from occurring and to mitigate the possibilities of costly rework. At Critical Logic we have shown our clients that the process of effective QA can start earlier in a project, short-circuiting the cycle of testing and rework. Rock-solid and unambiguous requirements are the result of this process, the ingredients of the vaccine that will prevent your project from getting sick. Isn’t it time for you to schedule a free needs assessment to see if your project would benefit from a shot in the arm?

For more information about or to schedule a free needs assessment, please visit us at critical-logic.com.

Since starting with Critical Logic, I’ve been involved with many major projects, and all of them have had at least a hundred test cases or more. Some have had up to a thousand test cases. Imagine having to run all of those manually every time you have to do a regression test. This is the benefit of automation. Enter our modeling technology –  automated tests that can be easily maintained. Making a change to the model changes all of the test cases simultaneously, a process that saves our clients time and money. Imagine what could be accomplished in all the time that was saved.

Modeling for automation is something that takes time, though, and it is something that has to be done perfectly. Once we have received a design spec from a client and have completed an ambiguity review, we can begin modeling. In order for the automation to work, the model descriptions must match what is going on inside the software. Even the most obvious typo will be recorded as a defect in automation. Every item or value that needs to be verified, every button that is clicked or value that is entered into a field, and every identification done by page names or field names all have to be exact. That is part of the work we do on the test development (i.e., “modeling”) side in order to make the automation fly. By going through this process we gain further understanding of the inputs and outputs of the “Black Box”.  This is a tremendous benefit for our clients, because it acts as another method to make sure everything is working as it should. Just imagine what the Ford Mustang might run like if those turbo kits were not installed correctly. We want to give our clients the best work we can possibly deliver.

It is a process, and it does take time, but I have heard reviews from clients about how much time has been saved for them because of automation. Not every test case can be automated; some will have to be done manually, such as test cases that also use programs outside of the one that is being tested. However, a majority of them probably can be scripted and automated, and that alone can help save enough time and money to be worth ten times over all the time it takes to get the automation right.

Bottom line, my experience with automation is this: our test cases by themselves are like the Ford Mustang without all the nice trimmings that can be added. The Mustang is still a great car, and our test cases still cover 100% of the functionality in the specification.  With some extra work, though, we can make those test cases blast.

Despite the similarities, the software project manager has it worse.  Her deliverables are more abstract, the dependencies she manages tend to be less obvious, and the specifications her teams work with are, on average, woefully inadequate compared to the specifications used in construction.  All of this makes software projects less predictable, and thus more difficult to manage effectively.

Given the inherent difficulties of dealing in such an abstract environment, one might assume that software would go to even greater lengths than construction to define and document the interconnected layers of the system, from the highest level end-user need down to the lowest technical foundational level.

Shockingly, the reverse is true.

Study after study reveals a software industry hopelessly mired in the muck created by incomplete specs.  Nearly 2/3 of all software defects are traceable back to incorrect or incomplete requirements.  On average, half of the final budget of a large software project is spent after the first code is delivered for test.  This is not a good result for project managers who thought they were 80% done.

Granted, some systems require less documentation.  The contractor who re-landscaped my backyard was able to meet my needs based on a few sketches and periodic conversations about design decisions.

But complexity isn’t the only factor.  Risk must also be considered.  If it were my front yard, I would have worked with more detailed specifications.  Rework is expensive.

A recent NY Times piece on how Google develops new software provides a case in point for how lack of specification results in unpredictability.  The article, entitled ”Bug by Bug, Google Fixes a New Idea,” is a day in the life of a test engineer at Google, working on a new tool to push ads to mobile phones—not an overly complex system, but one that generates ad revenue, affecting the bottom line.

http://www.nytimes.com/2009/10/05/business/media/05adco.html

In one segment the test engineer was asked, “Can we stay on schedule?”  “It depends on how the testing goes,” she replied. “We’re not expecting any disasters, but you never expect any disasters.”  What is her definition of disaster?  It’s one thing to not plan for earthquakes, power outages, and tornadoes.  But something did come up that did delay their release date, and it was not an act of God.

A new kind of phone that had not been in the original list needed to be compatible with the new tool.  And some older phones had been misclassified and not included.  The specification for what phones should be included was incorrect, incomplete, and evidently not managed through formal change control.

The form of testing that was employed—an experiment employing select end-users with the new software compared with a control group—was itself unpredictable, and it betrays what was likely an incomplete functional specification.

A complete specification would specify the exact variables per phone type that mattered to the software, and document expected behavior for each variation per phone.  This would allow complete functional testing of the software, without the need for a dragnet approach based on probabilities.  The approach that was used gave no clear definition of what/how much functionality had been exercised, beyond a guess based on probabilities.

Google pulled it off, which is common on systems of this size.  They had some delays, and probably plenty of production defects, but it worked well enough.  But how much more money was spent developing it than planned (because of the late rework and schedule delays)?  What was the impact on other projects the test engineer and developers were scheduled to work on?  And what was the impact from lost ad revenue to Google’s bottom line?

One of the dirty secrets of software development is that you can get away with poor software specifications for the exact same reason that it is absolutely essential to develop good software specifications.  The stuff is so abstract and complex, it’s sometimes hard for project managers to know whether the documentation is sufficient.  Sadly, too often they don’t find out until their project goes 30% over budget.

REQ A: Records entered after January 1st, 2000, will be converted to the new format.

Sounds straightforward, right? In this requirement the boundary for records being converted or not converted lies between January 1st and January 2nd. Those are the only dates that we need to test to verify the functionality of the software. January 1st’s record is not converted; January 2nd’s record is converted. There is no need to test December 31st or January 3rd to see what happens. When I see a requirement like this, I will often log an ambiguity to clarify the boundary. Does the business truly want the boundary between the 1st or 2nd, or is the business intending the boundary to be between the years 1999 and 2000? Just to see the difference, here is the same requirement with that boundary:

REQ A: Records entered January 1st, 2000, or later will be converted to the new format.

There is not much of a difference in wording here, and when the requirements are being written by business experts and not software engineers, these issues can easily slip into a specification.

Another boundary issue I have seen in specifications of logical comparisons concerns dollar values. Recently, I reviewed a document with a cost limit for processing an order without manager approval. The requirement read something like this:

REQ B: Orders over $100 must be routed to the manager for approval.

Does it sound straightforward again? Not really. This requirement should always result in an ambiguity question being asked. One problem with defining the intended boundaries of the logical comparison here is the currency format being used by the system. Is the logical comparison really between $100 and $101, or actually $100.00 and $100.01? Most likely, the ambiguity question would reveal that orders equal to $100 would need approval as well. In that case the requirement would have to be rewritten; given here using the decimal:

REQ B: Orders equal to or over $100.00 must be routed to the manager for approval.

This requirement implies a testing boundary of $99.99, no approval required; $100.00, approval required.

Although the examples given here concern minor differences, eventually comparison boundaries that are misunderstood by a programmer will have a serious enough impact to the business as to require the additional cost of rework. At Critical Logic our Right From The Start™ process aims to eliminate that possibility. We’ve had a great amount of success in the past reducing the amount of rework required in application development through our process, rework which is often 50%-75% of a project’s entire budget. This not only reduces the overall cost of the project, but also the time-to-market of the application being developed.

If you would like to know more about what we do and conduct a free needs assessment for your organization, contact us today.

Not many people outside academia tend to concern themselves with the specific details of logical processing. Your average Joe doesn’t think too heavily about deductive and inductive reasoning, or parallel versus sequential processing. The term logic equates for most of us with common sense, so why bother with that level of detail? But when it comes to software testing that level of detail can make a huge difference in terms of both time and money.

When I first began training with Critical Logic, I found that unraveling the logic behind functional requirements was much harder than it seemed. Something that seemed so straightforward involved an inordinate amount of questioning of basic assumptions and details. The level of detail required to create a model, to develop the necessary if-then statements, and to decide if these operations are happening in sequence or in parallel was mind-boggling even for a simple GUI screen. Initially, the advantage seemed to be that the graphing or modeling process broke things down visually so that the user could see the intricacies of the process. Instead of thinking through everything that could go wrong with the given piece of software, DTT© allowed the user to explore the system and question the logical possibilities of each function or field. Breaking down the logic behind software requirements is not an easy task, but once it’s done DTT© can provide an amazing number of benefits to both testers and designers. The irony of all this, of course, is that many months into it this has become second-nature to me, and I now find myself thinking of the process as common sense – given the benefits of modeling as a tool for functional software analysis, why would you do it any other way?

In addition to modeling, the Requirements-based Testing (RBT) approach has exposed me to the world of ambiguity review. Ambiguity review is at the core of Critical Logic’s approach, and stands as one of the most valuable tools offered by the company. Ideally, ambiguity identification proceeds in two steps. The first portion of the ambiguity review is done by someone who is not a subject matter expert (SME).The focus is on ambiguities in the logic and structure of the wording, unclear references and potential missing details. However, since the initial reviewer is not a SME he or she cannot read the requirements for facts that are not explicitly there. Thus, the second stage of the process is conducted by subject matter experts to ensure that the document is complete and correct in terms of the application. Often, in my experience, this is a fluid process where SMEs and Critical Logic staff exchange information, shifting between the first and second stage of this process numerous times. The key element of this process is eliminating problems in the beginning when they are easy and inexpensive to fix. Having one set of specifications and requirements that are clear, determinate, and specific is an asset for developers, testers, and business analysts.

Finding ambiguities is a skill that involves learning how to ask the right questions. Although it is not an automated technology process like DTT©, ambiguity review is easily the most important step in paving the way for DTT’s© effectiveness. In my limited time I have seen first-hand how poorly written requirements can ruin a project. Editing and revision early in the developmental phase is the key to eliminating inefficiencies later in the test cycle.

It is exciting to be part of a company that has the potential to improve existing quality and testing processes to such a startling degree.

If you can’t make it down to Lake Washington, you can watch the event LIVE at www.ulhra.org.

For more information on Critical Logic racing visit www.critical-logic.com

CIC_004: The system will be able to handle a constantly growing inventory.
CIC_005: The system will be able to add new categories of inventory.
CIC_006: The system will be able to record inventory changes.

These are sometimes referred to as business requirements, that is, the requirements and constraints the business puts on the software. They define the boundaries of the software and what it must be able to do. What they do not define is how it will do as required. Unfortunately, all too often, business requirements like the above are all a software team or an analyst receives. Consider these ambiguities that the software and analyst teams must work around:

Ambiguity #1: How large an inventory must the system realistically support? Would 3,000 items be overkill? (Perhaps in the case of a company specializing in custom-made Renaissance Faire costumes, this would be more than enough to fill their needs. In the case of Amazon.com, it would be woefully inadequate.)

Ambiguity #2: Will each item, even if it is the same as another item, be listed separately within the inventory? Or will the inventory be able to hold an item count?

Ambiguity #3: How will items be added to the inventory?

As you can see, these ambiguities could be resolved in any number of ways. Using functional requirements like the ones that follow can eliminate creative coding by defining clearly how each qualitative requirement will be implemented. Also it will enable testers to delineate between expected functionality and unexpected functionality without constant meetings, reviews and rework.

CIC_104: The system will have slots for 10,000 individual inventory items.
CIC_114: Each inventory slot will have fields for an inventory number, an item description, inventory category, item price and item quantity.
CIC_124: An inventory number will be 10 digits, starting at 0000000001.
CIC_134: An inventory description may contain 140 characters.
CIC_144: An inventory category may contain 10 characters.
CIC_154: Item price must follow the format of dollar sign ($), numerical characters, decimal point, two numerical characters. The acceptable range for item price is $0.01 - $9999.99.
CIC_164: Item quantity must be a number from 0-9999.

Each qualitative requirement will need its own functional requirements that describe how the needed quality is implemented in the software. Before any development process begins, the following process should be used to create qualitative and functional requirements that will prevent ambiguities:

1. Define what problems the software should solve for the company. (e.g., the inventory is poorly tracked).
2. Define what the general business needs are for a Central Inventory Control system. (i.e., write and edit all the qualitative business requirements).
3. Consider how the software should behave to meet all of those business requirements. (i.e., write and edit all functional software requirements).

Of course, the third step is where things often become confused. There is a need in most specifications to separate the business requirements from software requirements. While business requirements do impact the software requirements, it is our experience that there is a trend to use them in place of software requirements, which leaves functionality ambiguous to both the developers and the analysts. The design will, of course, influence the development and functionality of the software, but the software must have its own functional requirements to avoid missing a key business necessity.

Consider functional requirements to be a blueprint; no one would let an electrician wire their new custom home on the fly without even the roughest of sketches. Nor should the plumber and electrician find that they both intend to put infrastructure in the same place in the wall. By considering each step of the way, and clearly designing even small details in advance of the development of the software, your software project can be done right from the start with a minimum of rework due to misunderstandings, ambiguities or on-the-fly development decisions. The time and money savings from using defect-free software requirements is enormous. Like a carefully planned custom home, a carefully planned software project can be a value for years to come.

One of the ways we can accomplish this is through a modified exploratory testing technique, sometimes called “exploratory modeling.” Even exploratory modeling needs a plan. We’ve used many ways to break up applications into “chunks” for test planning, but in the case of a prototype we usually start with a screen inventory. We can put together this screen inventory ourselves (yes, by “exploring” the application) and learn about how the application works in the process. Based on past experience and the amount of features found, we can usually put together a good estimate of how long it will take to model each area of the application as well.

Now to the part where having good technology at hand really makes a difference when practicing an exploratory testing technique. Our modeling technology, DTT©, is very apt at allowing the tester to model the functionality of an already-existing prototype. The logical possibilities of each function or field can be captured in the graphical model, although what is captured is still at the discretion of the tester exploring the application. Sometimes the parameters of this discretion are a predefined scope decision by the QA manager or our client. Other times it is simply a limit on time or budget. Either way, the functionality is captured in the model, and DTT© can generate a repeatable script of what was explored and exercised. Even if never exercised again, this documentation is valuable if it is needed for regulatory or change-management purposes, or, more likely, for the basis of a future automation effort. Like any other technique we use, bugs have to be tracked and stomped if found.

At the end of the day, exploratory testing is really reverse engineering of the application (exploratory) and then deciding if you like what you see (testing). Exploratory testing is the test engineer’s last resort for poor specification of expected software behavior. What the model facilitates is the capturing of the lost functionality so that we can migrate from discovery to formal testing of the application, all in one step. Also, the model lets us generate a functional spec so that the next test engineer does not have to re-explore the same application, and knows what has already been covered.

Using the above technique allows Critical Logic to continue delivering value at all stages of the test cycle. To learn more about the DTT© modeling technology discussed here and our other capabilities, contact us at www.critical-logic.com/contact.php.