Section 17.4.  The Test Harness

17.4. The Test Harness

We usually invoke tests, either as the developer or the user, by using make test . The Makefile uses the Test::Harness module to run the tests, watch the output, and report the results.

Tests live in files in the t directory at the top level of the distribution, and each test file ends in .t. The test harness invokes each file separately, so an exit or die terminates only the testing in that file, not the whole testing process.^[*]

^[*] If we decide that we need to stop all testing, we can "bail out" of the process by printing "bail out" to standard output. This can be handy if we encounter an error that is going to cause a cascade of errors that we don't want to wait to encounter or sift through.

The test file communicates with the test harness through simple messages on standard output.^[] The three most important messages are the test count, a success message, and a failure message.

^[] The exact details are documented in the perldoc Test::Harness::TAP, if you're curious. Actually, they're there, even if you're not curious.

An individual test file comprises one or more tests. These tests are numbered as counting numbers, starting with one. The first thing a test file must announce to the test harness (on STDOUT) is the expected number of tests, as a string that looks like a Perl range, 1..N. For example, if there are 17 tests, the first line of output should be:

1..17

followed by a newline. The test harness uses the upper number to verify that the test file hasn't just terminated early. If the test file is testing optional things and has no testing to do for this particular invocation, the string 1..0 suffices.

We don't have to worry about printing that ourselves, though, because Test::More does it for us. When we load the module, we tell it the number of tests that we want to run, and the module outputs the test range for us. To get that 1..17, we tell Test::More we're running 17 tests.

use Test::More tests => 17;

After we print the test range, we send to standard output one line per test. That line starts with ok if the test passed, or not ok if the test failed. If we wrote that by hand, a test to check that 1 plus 2 is 3 might look like this:

print +(1 + 2 =  = 3 ? '', 'not '), "ok 1\n";

We could also print the not only if the test failed, with the ok as a separate step. But on some platforms, this may fail unnecessarily, due to some I/O-related oddity. For maximum portability, print the entire string of ok N or not ok N in one print step. Don't forget the space after not!

print 'not ' unless 2 * 4 =  = 8;
print "ok 2\n";

We don't have to do that much work, though, and we don't have to count the tests ourselves. What if we wanted to add a test between the two previous tests? We'd have to give that new test the test number 2 and change the last test to test number 3. Not only that, we haven't output anything useful if the test fails, so we won't know what went wrong. We could do all that, but it's too much work! Luckily, we can use the convenience functions from Test::More.