A Simple Testing Framework
The primary goal of this framework[89] is to verify the output of the examples in the book. You have already seen lines such as
private static Test monitor = new Test();
at the beginning of most classes that contain a main( ) method. The task of the monitor object is to intercept and save a copy of standard output and standard error into a text file. This file is then used to verify the output of an example program by comparing the contents of the file to the expected output.
We start by defining the exceptions that will be thrown by this test system. The general-purpose exception for the library is the base class for the others. Note that it extends RuntimeException so that checked exceptions are not involved:
//: com:bruceeckel:simpletest:SimpleTestException.java
package com.bruceeckel.simpletest;
public class SimpleTestException extends RuntimeException {
public SimpleTestException(String msg) {
super(msg);
}
} ///:~
A basic test is to verify that the number of lines sent to the console by the program is the same as the expected number of lines:
//: com:bruceeckel:simpletest:NumOfLinesException.java
package com.bruceeckel.simpletest;
public class NumOfLinesException
extends SimpleTestException {
public NumOfLinesException(int exp, int out) {
super("Number of lines of output and "
+ "expected output did not match.\n" +
"expected: <" + exp + ">\n" +
"output: <" + out + "> lines)");
}
} ///:~
Or, the number of lines might be correct, but one or more lines might not match:
//: com:bruceeckel:simpletest:LineMismatchException.java
package com.bruceeckel.simpletest;
import java.io.PrintStream;
public class LineMismatchException
extends SimpleTestException {
public LineMismatchException(
int lineNum, String expected, String output) {
super("line " + lineNum +
" of output did not match expected output\n" +
"expected: <" + expected + ">\n" +
"output: <" + output + ">");
}
} ///:~
This test system works by intercepting the console output using the TestStream class to replace the standard console output and console error:
//: com:bruceeckel:simpletest:TestStream.java
// Simple utility for testing program output. Intercepts
// System.out to print both to the console and a buffer.
package com.bruceeckel.simpletest;
import java.io.*;
import java.util.*;
import java.util.regex.*;
public class TestStream extends PrintStream {
protected int numOfLines;
private PrintStream
console = System.out,
err = System.err,
fout;
// To store lines sent to System.out or err
private InputStream stdin;
private String className;
public TestStream(String className) {
super(System.out, true); // Autoflush
System.setOut(this);
System.setErr(this);
stdin = System.in; // Save to restore in dispose()
// Replace the default version with one that
// automatically produces input on demand:
System.setIn(new BufferedInputStream(new InputStream(){
char[] input = ("test\n").toCharArray();
int index = 0;
public int read() {
return
(int)input[index = (index + 1) % input.length];
}
}));
this.className = className;
openOutputFile();
}
// public PrintStream getConsole() { return console; }
public void dispose() {
System.setOut(console);
System.setErr(err);
System.setIn(stdin);
}
// This will write over an old Output.txt file:
public void openOutputFile() {
try {
fout = new PrintStream(new FileOutputStream(
new File(className + "Output.txt")));
} catch (FileNotFoundException e) {
throw new RuntimeException(e);
}
}
// Override all possible print/println methods to send
// intercepted console output to both the console and
// the Output.txt file:
public void print(boolean x) {
console.print(x);
fout.print(x);
}
public void println(boolean x) {
numOfLines++;
console.println(x);
fout.println(x);
}
public void print(char x) {
console.print(x);
fout.print(x);
}
public void println(char x) {
numOfLines++;
console.println(x);
fout.println(x);
}
public void print(int x) {
console.print(x);
fout.print(x);
}
public void println(int x) {
numOfLines++;
console.println(x);
fout.println(x);
}
public void print(long x) {
console.print(x);
fout.print(x);
}
public void println(long x) {
numOfLines++;
console.println(x);
fout.println(x);
}
public void print(float x) {
console.print(x);
fout.print(x);
}
public void println(float x) {
numOfLines++;
console.println(x);
fout.println(x);
}
public void print(double x) {
console.print(x);
fout.print(x);
}
public void println(double x) {
numOfLines++;
console.println(x);
fout.println(x);
}
public void print(char[] x) {
console.print(x);
fout.print(x);
}
public void println(char[] x) {
numOfLines++;
console.println(x);
fout.println(x);
}
public void print(String x) {
console.print(x);
fout.print(x);
}
public void println(String x) {
numOfLines++;
console.println(x);
fout.println(x);
}
public void print(Object x) {
console.print(x);
fout.print(x);
}
public void println(Object x) {
numOfLines++;
console.println(x);
fout.println(x);
}
public void println() {
if(false) console.print("println");
numOfLines++;
console.println();
fout.println();
}
public void
write(byte[] buffer, int offset, int length) {
console.write(buffer, offset, length);
fout.write(buffer, offset, length);
}
public void write(int b) {
console.write(b);
fout.write(b);
}
} ///:~
The constructor for TestStream, after calling the constructor for the base class, first saves references to standard output and standard error, and then redirects both streams to the TestStream object. The static methods setOut( ) and setErr( ) both take a PrintStream argument. System.out and System.err references are unplugged from their normal object and instead are plugged into the TestStream object, so TestStream must also be a PrintStream (or equivalently, something inherited from PrintStream). The original standard output PrintStream reference is captured in the console reference inside TestStream, and every time console output is intercepted, it is sent to the original console as well as to an output file. The dispose( ) method is used to set standard I/O references back to their original objects when TestStream is finished with them.
For automatic testing of examples that require user input from the console, the constructor redirects calls to standard input. The current standard input is stored in a reference so that dispose( ) can restore it to its original state. Using System.setIn( ), an anonymous inner class is set to handle any requests for input by the program under test. The read( ) method of this inner class produces the letters “test” followed by a newline.
TestStream overrides a variety of PrintStream print( ) and println( ) methods for each type. Each of these methods writes both to the “standard” output and to an output file. The expect( ) method can then be used to test whether output produced by a program matches the expected output provided as argument to expect( ).
These tools are used in the Test class:
//: com:bruceeckel:simpletest:Test.java
// Simple utility for testing program output. Intercepts
// System.out to print both to the console and a buffer.
package com.bruceeckel.simpletest;
import java.io.*;
import java.util.*;
import java.util.regex.*;
public class Test {
// Bit-shifted so they can be added together:
public static final int
EXACT = 1 << 0, // Lines must match exactly
AT_LEAST = 1 << 1, // Must be at least these lines
IGNORE_ORDER = 1 << 2, // Ignore line order
WAIT = 1 << 3; // Delay until all lines are output
private String className;
private TestStream testStream;
public Test() {
// Discover the name of the class this
// object was created within:
className =
new Throwable().getStackTrace()[1].getClassName();
testStream = new TestStream(className);
}
public static List fileToList(String fname) {
ArrayList list = new ArrayList();
try {
BufferedReader in =
new BufferedReader(new FileReader(fname));
try {
String line;
while((line = in.readLine()) != null) {
if(fname.endsWith(".txt"))
list.add(line);
else
list.add(new TestExpression(line));
}
} finally {
in.close();
}
} catch (IOException e) {
throw new RuntimeException(e);
}
return list;
}
public static List arrayToList(Object[] array) {
List l = new ArrayList();
for(int i = 0; i < array.length; i++) {
if(array[i] instanceof TestExpression) {
TestExpression re = (TestExpression)array[i];
for(int j = 0; j < re.getNumber(); j++)
l.add(re);
} else {
l.add(new TestExpression(array[i].toString()));
}
}
return l;
}
public void expect(Object[] exp, int flags) {
if((flags & WAIT) != 0)
while(testStream.numOfLines < exp.length) {
try {
Thread.sleep(1000);
} catch (InterruptedException e) {
throw new RuntimeException(e);
}
}
List output = fileToList(className + "Output.txt");
if((flags & IGNORE_ORDER) == IGNORE_ORDER)
OutputVerifier.verifyIgnoreOrder(output, exp);
else if((flags & AT_LEAST) == AT_LEAST)
OutputVerifier.verifyAtLeast(output,
arrayToList(exp));
else
OutputVerifier.verify(output, arrayToList(exp));
// Clean up the output file - see c06:Detergent.java
testStream.openOutputFile();
}
public void expect(Object[] expected) {
expect(expected, EXACT);
}
public void expect(Object[] expectFirst,
String fname, int flags) {
List expected = fileToList(fname);
for(int i = 0; i < expectFirst.length; i++)
expected.add(i, expectFirst[i]);
expect(expected.toArray(), flags);
}
public void expect(Object[] expectFirst, String fname) {
expect(expectFirst, fname, EXACT);
}
public void expect(String fname) {
expect(new Object[] {}, fname, EXACT);
}
} ///:~
There are several overloaded versions of expect( ) provided for convenience (so the client programmer can, for example, provide the name of the file containing the expected output instead of an array of expected output lines). These overloaded methods all call the main expect( ) method, which takes as arguments an array of Objects containing expected output lines and an int containing various flags. Flags are implemented using bit shifting, with each bit corresponding to a particular flag as defined at the beginning of Test.java.
The expect( ) method first inspects the flags argument to see if it should delay processing to allow a slow program to catch up. It then calls a static method fileToList( ), which converts the contents of the output file produced by a program into a List. The fileToList( ) method also wraps each String object in an OutputLine object; the reason for this will become clear. Finally, the expect( ) method calls the appropriate verify( ) method based on the flags argument.
There are three verifiers: verify( ), verifyIgnoreOrder( ), and verifyAtLeast( ), corresponding to EXACT, IGNORE_ORDER, and AT_LEAST modes, respectively:
//: com:bruceeckel:simpletest:OutputVerifier.java
package com.bruceeckel.simpletest;
import java.util.*;
import java.io.PrintStream;
public class OutputVerifier {
private static void verifyLength(
int output, int expected, int compare) {
if((compare == Test.EXACT && expected != output)
|| (compare == Test.AT_LEAST && output < expected))
throw new NumOfLinesException(expected, output);
}
public static void verify(List output, List expected) {
verifyLength(output.size(),expected.size(),Test.EXACT);
if(!expected.equals(output)) {
//find the line of mismatch
ListIterator it1 = expected.listIterator();
ListIterator it2 = output.listIterator();
while(it1.hasNext()
&& it2.hasNext()
&& it1.next().equals(it2.next()));
throw new LineMismatchException(
it1.nextIndex(), it1.previous().toString(),
it2.previous().toString());
}
}
public static void
verifyIgnoreOrder(List output, Object[] expected) {
verifyLength(expected.length,output.size(),Test.EXACT);
if(!(expected instanceof String[]))
throw new RuntimeException(
"IGNORE_ORDER only works with String objects");
String[] out = new String[output.size()];
Iterator it = output.iterator();
for(int i = 0; i < out.length; i++)
out[i] = it.next().toString();
Arrays.sort(out);
Arrays.sort(expected);
int i =0;
if(!Arrays.equals(expected, out)) {
while(expected[i].equals(out[i])) {i++;}
throw new SimpleTestException(
((String) out[i]).compareTo(expected[i]) < 0
? "output: <" + out[i] + ">"
: "expected: <" + expected[i] + ">");
}
}
public static void
verifyAtLeast(List output, List expected) {
verifyLength(output.size(), expected.size(),
Test.AT_LEAST);
if(!output.containsAll(expected)) {
ListIterator it = expected.listIterator();
while(output.contains(it.next())) {}
throw new SimpleTestException(
"expected: <" + it.previous().toString() + ">");
}
}
} ///:~
The “verify” methods test whether the output produced by a program matches the expected output as specified by the particular mode. If this is not the case, the “verify” methods raise an exception that aborts the build process.
Each of the “verify” methods uses verifyLength( ) to test the number of lines of output. EXACT mode requires that both output and expected output arrays be the same size, and that each output line is equal to the corresponding line in the expected output array. IGNORE_ORDER still requires that both arrays be the same size, but the actual order of appearance of the lines is disregarded (the two output arrays must be permutations of one another). IGNORE_ORDER mode is used to test threading examples where, due to non-deterministic scheduling of threads by the JVM, it is possible that the sequence of output lines produced by a program cannot be predicted. AT_LEAST mode does not require the two arrays to be the same size, but each line of expected output must be contained in the actual output produced by a program, regardless of order. This feature is particularly useful for testing program examples that contain output lines that may or may not be printed, as is the case with most of the examples dealing with garbage collection. Notice that the three modes are canonical; that is, if a test passes in IGNORE_ORDER mode, then it will also pass in AT_LEAST mode, and if it passes in EXACT mode, it will also pass in the other two modes.
Notice how simple the implementation of the “verify” methods is. verify( ), for example, simply calls the equals( ) method provided by the List class, and verifyAtLeast( ) calls List.containsAll( ). Remember that the two output Lists can contain both OutputLine or RegularExpression objects. The reason for wrapping the simple String object in OutputLines should now become clear; this approach allows us to override the equals( ) method, which is necessary in order to take advantage of the Java Collections API.
Objects in the expect( ) array can be either Strings or TestExpressions, which can encapsulate a regular expression (described in Chapter 12), which is useful for testing examples that produce random output. The TestExpression class encapsulates a String representing a particular regular expression.
//: com:bruceeckel:simpletest:TestExpression.java
// Regular expression for testing program output lines
package com.bruceeckel.simpletest;
import java.util.regex.*;
public class TestExpression implements Comparable {
private Pattern p;
private String expression;
private boolean isRegEx;
// Default to only one instance of this expression:
private int duplicates = 1;
public TestExpression(String s) {
this.expression = s;
if(expression.startsWith("%% ")) {
this.isRegEx = true;
expression = expression.substring(3);
this.p = Pattern.compile(expression);
}
}
// For duplicate instances:
public TestExpression(String s, int duplicates) {
this(s);
this.duplicates = duplicates;
}
public String toString() {
if(isRegEx) return p.pattern();
return expression;
}
public boolean equals(Object obj) {
if(this == obj) return true;
if(isRegEx) return (compareTo(obj) == 0);
return expression.equals(obj.toString());
}
public int compareTo(Object obj) {
if((isRegEx) && (p.matcher(obj.toString()).matches()))
return 0;
return
expression.compareTo(obj.toString());
}
public int getNumber() { return duplicates; }
public String getExpression() { return expression;}
public boolean isRegEx() { return isRegEx; }
} ///:~
TestExpression can distinguish regular expression patterns from String literals. The second constructor allows multiple identical expression lines to be wrapped in a single object for convenience.
This test system has been reasonably useful, and the exercise of creating it and putting it into use has been invaluable. However, in the end I’m not that pleased with it and have ideas that will probably be implemented in the next edition of the book (or possibly sooner).
