All Combinations

This is the second of a suite of fixtures that make up the AllPairsExample. Like with AllFiles, we expand wildcard characters in path specifications to get lists of tests. But this time we run all combinations of the tests. This is another warmup for AllPairs testing.

A project can expect to have dozens or hundreds of files testing specific features in a stand-alone way. We may wonder if there are any interactions between features that will cause any of these tests to fail. AllCombinations explores this. It is a fixture that enumerates cases by drawing one file at a time from each of a number of lists; running each in turn, and then repeating the drawing and running for all possible combinations.

For the purpose of illustration we've created a number of files that contain small tests of the CalculatorExample. We've categorized the files as magnitudes, signs and functions, which enter values, changes signs, and compute functions, respectivly.

This table runs all possible sequences of magnitude, sign and function. Try it with http:run.cgi.

eg.AllCombinations

input/AllPairs/magnitude/*.html

 

input/AllPairs/sign/*.html

 

input/AllPairs/function/*.html

 

The number of cases will as large as the product of the size of each list of tests. The searches in the table find four magnitues, three signs and two functions.

Any file of any case can fail. We save the results from failed tests only. A single file can fail different ways in different cases. Look for differences in the counts or examine each FileFootnote.

If the tests are intended to interact, as ours are here, then some care is required to be sure that checks in the tests are independent of this interaction. No checks at all my be suitable if we are only testing that there are no exceptions thrown. Or the simplest checks may be adaquate so long as features are tested well in isolation. It may be convenient to run a simulation, a so called oracale, along side the application to generate expected results.

Implementation

The fixture inherits path expansion from AllFiles, but captures these by overrinding doRows(Parse, List) and saving the list of files in a list of lists. Then, at the end of doTable, the fixture gets to work computing and running combinations.

See source.

We had a little trouble setting up the test data for this example so we wrote this little Trigonometry Refresher.

eg.ArithmeticColumnFixture

x

sin()

cos()

0

0.0000

1.0000

30

0.5000

0.8660

60

0.8660

0.5000

90

1.0000

0.0000

120

0.8660

-0.5000

150

0.5000

-.8660

180

0.0000

-1.0000

fit.Summary

Last edited November 4, 2002