This is a continuation of the discussion on testing techniques that started with the Basis Path, White Box, testing technique (see Lecture #4). Following, is a detailed discussion of two well known Black Box techniques. That is, Equivalence Partitioning and Boundary Value Analysis. Another technique, Error Guessing is also briefly mentioned.
To motivate the discussion, consider the following functional description of the function, CheckoutCost, that is intended to compute the total cost of a basket of products purchased at an electronic retailer.
Functional Description:
CheckoutCost is a funtion of three variables: nextday, amount, and type. It determines shipping charge, applicable surcharges, applicable discounts, and tax, if any. It returns a single total amount which is the sum of amount, shipping charge, and tax, plus any surcharges and less any discounts.
The input variables are subject to the following constraints:
The following requirements apply:
NextDay must be Y or N
Amount must be greater than zero
NextDay must be Y or N Type must be B, E or H
Issue:
Given this functional description alone, is it possible to derive a thorough set of test cases that may be used to determine if the specified functionality is correctly implemented? The answer is Yes. Several techniques have been developed for accomplishing this. We will examine two of these techniques in some detail. They are the complementary techniques, Equivalence Partitioning and Boundary Value Analysis. We will also briefly mention the Error Guessing technique.
Equivalence Partitioning:
The Equivalence Partitioning technique is usually attributed to Glenford Myers. Myers provided a comprehensive outline of the technique in his classic text, The Art of Software Testing that was first published in 1979. It is based on the idea of Equivalence Classes from Set Theory (Discrete Mathematics). For the curious, the Set Theory aside below provides some details on this idea. However, to understand the technique it is not absolutely necessary to understand the mathematics. The point is that the use of equivalence classes as the basis for deriving test cases has two motivations.
Following is an outline of the actual technique that may be used to discover these equivalence classes and, as a result, may be used to derive test cases.
Identifying Equivalence Classes:
Equivalence classes may be identified from a functional description by carefully examining the constraints and functional requirements for each input. For each input, the constraints and functional requirements should be sufficient to partition the input domain into two or more groups. The groups are partitions of the input domain such that each partition consists of values that the functional requirements indicate will be processed in the same way (i.e. equivalently). As a result, these partitions are referred to as equivalence classes.
Remember that two types of equivalence classes should be identified. That is, valid equivalence classes for input values that are valid (may be determined from the constraints) and invalid equivalence classes for input values that are invalid (may be determined from the constraints).
Identifying equivalence classes is largely a heuristic process. You may apply the following guidelines based on guidelines first outlined by Myers in the 1979 text:
Identifying Test Cases:
Once equivalence classes have been identified then test cases may be derived from the equivalence classes. The procedure described below is more robust than that outlined by Myers in his 1979 text. That is, it results in a more complete set of test cases than Myers would have derived. It is similar to a procedure referred to as Strong Robust Equivalence Class Testing in the second edition of Software Testing by Paul Jorgensen (2002).
Valid labels: IT-V1, IT-V2 Invalid labels: IT-I1, IT-I2
IT = {IT-V1, IT-V2, IT-I1, IT-I2} A = {A-V1, A-I1, A-I2}Notice that, using our labeling convention, it is evident that there are two valid equivalence classes and two invalid equivalence classes for ItemType. Similarly, there is one valid equivalence class and two invalid equivalence classes for Amount. So, the resulting Cartesian product would be a set of 12 tuples:
IT X A = {(IT-V1, A-V1), (IT-V1, A-I1), (IT-V1, A-I2), (IT-V2, A-V1), (IT-V2, A-I1), (IT-V2, A-I2), (IT-I1, A-V1), (IT-I1, A-I1), (IT-I1, A-I2), (IT-I2, A-V1), (IT-I2, A-I1), (IT-I2, A-I2)}
Example:
We are now ready to derive test cases for the function, CheckoutCost described above. We will do this in two steps. First we will define equivalence classes by referring to the functional description for CheckoutCost and using the heuristics above. Second, we will derive test cases using the procedure above.
A-V1 = {0 < Amount =< 200} A-V2 = {200 < Amount =< 1000} A-V3 = {Amount > 1000}
ND = {ND-V1, ND-V2, ND-I1} A = {A-V1, A-V2, A-V3, A-I1} T = {T-V1, T-V2, T-V3, T-I1}
ND X A X T = {(ND-V1, A-V1, T-V1), (ND-V1, A-V1, T-V2), (ND-V1, A-V1, T-V3), (ND-V1, A-V1, T-I1), (ND-V1, A-V2, T-V1), (ND-V1, A-V2, T-V2), (ND-V1, A-V2, T-V3), (ND-V1, A-V2, T-I1), (ND-V1, A-V3, T-V1), (ND-V1, A-V3, T-V2), (ND-V1, A-V3, T-V3), (ND-V1, A-V3, T-I1), (ND-V1, A-I1, T-V1), (ND-V1, A-I1, T-V2), (ND-V1, A-I1, T-V3), (ND-V1, A-I1, T-I1), (ND-V2, A-V1, T-V1), (ND-V2, A-V1, T-V2), (ND-V2, A-V1, T-V3), (ND-V2, A-V1, T-I1), (ND-V2, A-V2, T-V1), (ND-V2, A-V2, T-V2), (ND-V2, A-V2, T-V3), (ND-V2, A-V2, T-I1), (ND-V2, A-V3, T-V1), (ND-V2, A-V3, T-V2), (ND-V2, A-V3, T-V3), (ND-V2, A-V3, T-I1), (ND-V2, A-I1, T-V1), (ND-V2, A-I1, T-V2), (ND-V2, A-I1, T-V3), (ND-V2, A-I1, T-I1), (ND-I1, A-V1, T-V1), (ND-I1, A-V1, T-V2), (ND-I1, A-V1, T-V3), (ND-I1, A-V1, T-I1), (ND-I1, A-V2, T-V1), (ND-I1, A-V2, T-V2), (ND-I1, A-V2, T-V3), (ND-I1, A-V2, T-I1), (ND-I1, A-V3, T-V1), (ND-I1, A-V3, T-V2), (ND-I1, A-V3, T-V3), (ND-I1, A-V3, T-I1), (ND-I1, A-I1, T-V1), (ND-I1, A-I1, T-V2), (ND-I1, A-I1, T-V3), (ND-I1, A-I1, T-I1)}
# Test Case Expected Result --------------------------------------------------------- 1. ND=Y, A=100, T=B $223.00 2. ND=Y, A=100, T=E $210.00 3. ND=Y, A=100, T=H $215.00 4. ND=Y, A=100, T=G Type must be B, E or H 5. ND=Y, A=500, T=B $700.00 6. ND=Y, A=500, T=E $635.00 7. ND=Y, A=500, T=H $660.00 8. ND=Y, A=500, T=G Type must be B, E or H 9. ND=Y, A=5000, T=B $6000.00 10. ND=Y, A=5000, T=E $5350.00 11. ND=Y, A=5000, T=H $5600.00 12. ND=Y, A=5000, T=G Type must be B, E or H 13. ND=Y, A=-1, T=B Amount must be greater than zero 14. ND=Y, A=-1, T=E Amount must be greater than zero 15. ND=Y, A=-1, T=H Amount must be greater than zero 16. ND=Y, A=-1, T=G Amount must be greater than zero Type must be B, E or H 17. ND=N, A=100, T=B $123.00 18. ND=N, A=100, T=E $110.00 19. ND=N, A=100, T=H $115.00 20. ND=N, A=100, T=G Type must be B, E or H 21. ND=N, A=500, T=B $600.00 22. ND=N, A=500, T=E $635.00 23. ND=N, A=500, T=H $660.00 24. ND=N, A=500, T=G Type must be B, E or H 25. ND=N, A=5000, T=B $5900.00 26. ND=N, A=5000, T=E $5250.00 27. ND=N, A=5000, T=H $5500.00 28. ND=N, A=5000, T=G Type must be B, E or H 29. ND=N, A=-1, T=B Amount must be greater than zero 30. ND=N, A=-1, T=E Amount must be greater than zero 31. ND=N, A=-1, T=H Amount must be greater than zero 32. ND=N, A=-1, T=G Amount must be greater than zero Type must be B, E or H 33. ND=Z, A=100, T=B NextDay must be Y or N 34. ND=Z, A=100, T=E NextDay must be Y or N 35. ND=Z, A=100, T=H NextDay must be Y or N 36. ND=Z, A=100, T=G NextDay must be Y or N Type must be B, E or H 37. ND=Z, A=500, T=B NextDay must be Y or N 38. ND=Z, A=500, T=E NextDay must be Y or N 39. ND=Z, A=500, T=H NextDay must be Y or N 40. ND=Z, A=500, T=G NextDay must be Y or N Type must be B, E or H 41. ND=Z, A=5000, T=B NextDay must be Y or N 42. ND=Z, A=5000, T=E NextDay must be Y or N 43. ND=Z, A=5000, T=H NextDay must be Y or N 44. ND=Z, A=5000, T=G NextDay must be Y or N Type must be B, E or H 45. ND=Z, A=-1, T=B NextDay must be Y or N Amount must be greater than zero 46. ND=Z, A=-1, T=E NextDay must be Y or N Amount must be greater than zero 47. ND=Z, A=-1, T=H NextDay must be Y or N Amount must be greater than zero 48. ND=Z, A=-1, T=G NextDay must be Y or N Amount must be greater than zero Type must be B, E or H
As mentioned earlier, the procedure used to derive these test cases is robust. Some versions of equivalence partitioning may yield fewer test cases. However, it should be noted that this robust procedure is particularly appropriate when the testing of error conditions is a high priority.
If the testing of error conditions is not a priority then a weaker version of this technique may be used. Rather than identifying both valid and invalid equivalence classes, only valid equivalence classes are defined. The Cartesian product of these valid equivalence classes is then constructed and test cases are derived. Paul Jorgensen identifies such a technique in his Software Testing text but, he uses the phrase Strong Normal Equivalence Class Testing to refer to this approach.
Independence Presumption:
It is also worth mentioning that, like all other equivalence partitioning procedures, the presumption here is that the input variables are independent. If any dependencies occur, the procedure is likely to generate erroneous test cases or may mask important test conditions. An erroneous test case is a test case which suggests an expected result that cannot occur.
Many types of dependencies between input variables are possible. Unfortunately, no single strategy to resolve this dilemma exists. Some practitioners address the issue by first eliminating any erroneous test cases and then supplementing the remaining test cases with new test cases derived using experience, domain knowledge, or the error guessing technique mentioned below. However, the Strong Robust technique discussed above may be readily extended to accommodate dependencies. One approach is to construct the Cartesian product from the equivalence classes in stages. We illustrate the idea with two examples:
One way of resolving this is to construct the Cartesian product in stages. That is, consider the independent equivalence classes first and construct the Cartesian product for these classes. Then, for each dependent variable (i.e. BusinessSubType for this example), construct a Cartesian product with the related subset of dependent tuples only. If you use this approach for our extended example you will notice that for each tuple that contains T=B, three tuples that contain the variable BusinessSubType will result. So, instead of forty eight test cases, we would have seventy two test cases.
Exercise: Derive the additional test cases for the extended example above using the dependence strategy mentioned.
One way of resolving this is, again, to construct the Cartesian product in stages. The trick here is to first derive equivalence classes, construct Cartesian products, and derive test cases for valid values of Type={B, E} only. Next, with Type=H, derive equivalence classes for Amount and then construct the Cartesian product and derive test cases. If you use this approach for our extended example, you will notice that the twelve test cases with T=H will be replaced by fifteen test cases, three of which will result in the required error message.
Exercise: Derive the additional test cases for the extended example above using the dependence strategy mentioned.
Boundary-Value Analysis:
Experience shows that test cases that explore boundary conditions have a higher payoff than test cases that do not. Boris Beizer captured the essence of this principle in the following often quoted statement:
Boundary conditions are those situations directly on, above, and below the edges of equivalence classes. In his text, Myers points out that Boundary-value analysis differs from equivalence partitioning in two respects:
The following guidelines (also based on those from Myers) may be used to derive test cases:
Exercise: Derive additional test cases for the forty eight test case example above using the boundary-value analysis heuristics.
Error Guessing:
Error Guessing is the process of using intuition and past experience to augment the set of test cases derived using any technique. There are no rules. The tester should review the set of test cases with an eye toward recognizing important missing test cases.
Error Guessing is considered by many practitioners to be a legitimate technique that is as important as more structured techniques because it is intended to compensate for the inherent limitations of any structured technique (see the section titled Independence Presumption above).
To illustrate the spirit of error guessing, Myers proposes a sorting example. Let us say you were testing a sorting function where the input is either a list or a file. Myers suggests that you may want to explore the following situations:
In other words, one enumerates those special cases that may have been overlooked when the program was coded.
Exercise: See if your intuition, or experience, suggests any additional test cases for the forty eight test case example above.
Aside - Set Theory:
This aside is a review of two topics that are usually covered in a first Discrete Mathematics course. The first topic is a brief high level review of Partitions and Equivalence Classes. The second topic is a review of the idea of a Cartesian Product.
Given a set B, and subsets A1, A2,..., An of B, the subsets are said to be partitions of B iff:
A1 U A2 U ... U An = Bwhere, i not equal to j implies Ai intersection Aj is the empty set.
The two parts of this definition are of particular importance to testers. The first part indicates that every element of B is in some subset, while the second part guarantees that no element of B is in two of the subsets. This leads to two important notions. That is, completeness (everything is somewhere) and nonredundancy.
Now, suppose we have some partition
A1,
A2,...,
An
of a set B. Let us say that two elements
b1 and
b2 of B are related if
b1 and
b2 are in the same partition
(say
Aj).
As it turns out, the relation in defined from the partition has
special properties (i.e. it is reflexive, symmetric, and transitive) and
is called the equivalence relation induced by the partition. The partitions
are then referred to as equivalence classes.
Note: For more details see any introductory Discrete Mathematics
text (e.g. Chapter 2 of Richard Johnsonbaugh's
Discrete Math text).
Recall that the two properties of a partition are the notions of completeness and nonredundancy. When translated into testing situations, these notions allow teters to make powerful statements about the extent to which a software product has been tested. Furthermore, considerable effort is saved by testing just one element of an equivalence class and assuming that the remaining elements will behave similarly.
The Cartesian product of two sets A1 and A2 is the set of tuples:
A1 X A2 = {(x, y): x is in A1 and y is in A2}For example, let us say that A1 = {1, 2, 3} and A2 = {w, x, y, z} then the Cartesian product would be:
A1 X A2 = {(1, w), (1, x), (1, y), (1, z), (2, w), (2, x), (2, y), (2, z), (3, w), (3, x), (3, y), (3, z)}Notice that the resulting set contains twelve tuples. Recall that the cardinality of a set A is the number of elements in A and is denoted |A|. Thus, we see that for sets A1 and A2, the cardinality of the Cartesian product is the multiplicative product of the cardinalities. That is:
|A1 X A2| = |A1| x |A2|We can easily see that in general, if we have k sets then the Cartesian product of the k sets is the set of k-tuples where the cardinality of the Cartesian product is the multiplicative product of the cardinalities of each of the individual k sets.