Test Procedure/Test Report Roadmap

Documents in the Section

· Test Procedure Template

· Test Procedure Example

· Test Report Template

· Test Report Example

What is the Test Procedure

The Test Procedure is the step-by-step process you go through to verify that your product meets all the requirements established in your SRD.

What is the Test Report

The Test Report is the result of formally running the Test Procedure and recording the results. The Test Report summarizes the results of the test, draws conclusions about the fitness of your product for its intended purpose, and recommends to your customer whether she should accept it or not.

Purpose of the Test Procedure and Test Report

The Test Procedure has to demonstrate that your program works for all the equivalence classes for input for all the requirements. The Test Report presents and honestly interprets the results of the test. With these two items, your customer can tell to what extent you met your requirements and have a feel for how long it would take to get the application into complete compliance.

How do you make a Test Procedure/Test Report?

· To make the Test Procedure, take the User Manual use case tables and change the topic headings to Test Step #, User Action and Expected Results.

· Add columns headed Pass/Fail and Comments.

· Expand the User Manual use cases to cover all the equivalence classes. Equivalence classes are representative elements of all the possible inputs to fulfill a requirement. The idea is to select a subset of the possible inputs so you don't spend the rest of your life testing, but the subset should be such that you can convince yourself and your customer that the product will handle all inputs. Doing this, you might end up running some of the use cases four or five times with different inputs - if you chose a board game such as Checkers, you could run the "Make Move" use cases 10 or 15 times.

The technique (and it can get really formal) is to select representatives for each of the "edges" and each different configuration. For example, if you were doing Tic Tac Toe, you would have at least three Win equivalence classes: a horizontal row, a vertical row and a diagonal row. If you have a betting function on your game, you need to test that (1) a bet is declined when the player has $0, (2) a bet is declined when the player has $1 too little money, (3) a bet is accepted if the player has exactly the right amount of money, (4) a bet is accepted if the player has more than enough money, (5) a bet smaller than what you allow is not allowed, and (6) a bet larger than you allow is not allowed.

· Add a Traceability Matrix, just like you did for the User Manual, but in the Comments section, identify what aspect of the requirement you are testing. For example, in the Tic Tac Toe example above, you'd have three sets of test steps testing the same thing, so in the Traceability Matrix, you'd say that each of them tested the same requirement, but you'd comment next to one that it tested horizontal, the next one would be commented with vertical, etc. You won't have to do this much if you itemized your requirements so that, in the example, the three win types were each identified and separately called out.

· Now that you have the Test Procedure, start running the tests as soon as you're done with the procedure and you have even fragmentary code. As you find things that don't pass the test, either fix the bugs or renegotiate the requirement.

· When you are no longer finding defects or time runs out, whichever comes first, you run the formal, usually witnessed, test. The results recorded during this run of the test are the ones used to judge your product. When you run the test, you record whether each step passed or failed by putting a "Pass" or "Fail" in the Pass/Fail column of the table. If a test step fails, you may put in a comment, describing the conditions under which it failed or a guess as to what caused it, whatever.

· Start a new document, titled Test Report.

· To write the Test Report, you start by summarizing the results. Make a table listing all the failed test steps. The table has the following headings: Test Step, Description of Failure, Requirement Not Met, Assessment of Effect. Fill out the table. Under the Description of Failure column, say what you observed that caused you to characterize the test step as a failure. Under Assessment of Effect, state if you think this is a minor or major fault. If you have a sense for it, you may say what you think the cause of the fault was.

If you don't find any failures, include the table and across the first row, write "Not applicable - no failures detected."

· Count the number of total test steps. Count the number of failed test steps. Write a paragraph such as the following:

This test was run by _______ (your name) and witnessed by ________ (whoever witnesses it). The test was run on _______(date) in JBE 105 using a ________(platform type). A total of ____(total number of test steps) tests were run, of which _____ failed and _____ passed. Of the failed tests, ____ were judged to be minor or cosmetic failures and ____ were judged to be major. Based on these results we recommend _________(acceptance or rejection) of this product. <Possibly add constraints on acceptance.>

If you don't find any failures, your paragraph will state "No failures were detected during the run of the test." The witness is either the customer or her authorized representative - usually someone from QA. For this class, we'll deputize someone in your group because we (the customers) will be re-running some part of the test on the last day of class during the final demos.

· Put the summary paragraph at the top of the Test Report page (tell 'em what you're going to tell 'em), followed by the Failure Summary Table (tell 'em), and staple that page to the filled out Test Procedure that you ran (tell 'em what you told 'em).

· Put a signature/date block below the table on the first page and, when you've completed the table and written the summary paragraph, each team member signs and dates it. The completed Test Procedure with the signed Test Report cover sheet is your Test Report package.