Section 17.3.  The Art of Testing

17.3. The Art of Testing

Good tests also give small examples of what we meant in our documentation. It's another way to express the same thing, and some people may like one way over the other.^[*] Good tests also give confidence to the user that your code (and all its dependencies) is portable enough to work on her system.

^[*] A few of the modules we've used from CPAN are easier to learn from test examples than by the actual POD. Of course, any really good example should be repeated in your module's POD documentation.

Testing is an art. People have written and read dozens of how-to-test books (and then ignore them, it seems). Mostly, it's important to remember everything we have ever done wrong while programming (or heard other people do), and then test that we didn't do it again for this project.

When you create tests, think like a person using a module, not like one writing a module. You know how you should use your module, because you invented it and had a specific need for it. Other people will probably have different uses for it and they'll try to use it in all sorts of different ways. You probably already know that given the chance, users will find every other way to use your code. You need to think like that when you test.

Test things that should break as well as things that should work. Test the edges and the middle. Test one more or one less than the edge. Test things one at a time. Test many things at once. If something should throw an exception, make sure it doesn't also have bad side effects. Pass extra parameters. Pass insufficient parameters. Mess up the capitalization on named parameters. Throw far too much or too little data at it. Test what happens for unexpected values such as undef.

Since we can't look at your module, let's suppose that we want to test Perl's sqrt function, which calculates square roots. It's obvious that we need to make sure it returns the right values when its parameter is 0, 1, 49, or 100. It's nearly as obvious to see that sqrt(0.25) should come out to be 0.5. We should also ensure that multiplying the value for sqrt(7) by itself gives something between 6.99999 and 7.00001.^[*]

^[*] Remember, floating-point numbers aren't always exact; there's usually a little roundoff. The Test::Number::Delta module can handle those situations.

Let's express that as code. We use the same stuff that we used for our first example. This script tests things that should work.

#!/usr/bin/perl

use Test::More tests => 6;

is( sqrt(  0),  0, 'The square root of 0   is  0' );
is( sqrt(  1),  1, 'The square root of 1   is  1' );
is( sqrt( 49),  7, 'The square root of 49  is  7' );
is( sqrt(100), 10, 'The square root of 100 is 10' );

is( sqrt(0.25), 0.5, 'The square root of 0.25 is 0.5' );

my $product = sqrt(7) * sqrt(7);

ok( $product > 6.999 && $product < 7.001,
        "The product [$product] is around 7" );

We should make sure that sqrt(-1) yields a fatal error and that sqrt(-100) does too. What happens when we request sqrt(&test_sub( )), and &test_sub returns a string of "10000"? What does sqrt(undef) do? How about sqrt( ) or sqrt(1,1)? Maybe we want to give our function a googol: sqrt( 10**100 ). Because this function is documented to work on $_ by default, we should ensure that it does so. Even a simple function such as sqrt should get a couple dozen tests; if our code does more complex tasks than sqrt does, expect it to need more tests, too. There are never too many tests.

In this simple script, we tested a lot of odd conditions.^[]

^[] And in writing this, we discovered that we can't write sqrt(-1) because eval doesn't trap that. Apparently, Perl catches it at compile time.

#!/usr/bin/perl

use Test::More tests => 9;

sub test_sub { '10000' }

is( $@, '', '$@ is not set at start' );

{
$n = -1;
eval { sqrt($n) };
ok( $@, '$@ is set after sqrt(-1)' );
}

eval { sqrt(1) };
is( $@, '', '$@ is not set after sqrt(1)' );

{
my $n = -100;
eval { sqrt($n) };
ok( $@, '$@ is set after sqrt(-100)' );
}

is( sqrt( test_sub(  ) ), 100, 'String value works in sqrt(  )' );

eval { sqrt(undef) };
is( $@, '', '$@ is not set after sqrt(undef)' );

is( sqrt, 0, 'sqrt(  ) works on $_ (undefined) by default' );

$_ = 100;
is( sqrt, 10, 'sqrt(  ) works on $_ by default' );

is( sqrt( 10**100 ), 10**50, 'sqrt(  ) can handle a googol' );

If you write the code and not just the tests, think about how to get every line of your code exercised at least once for full code coverage. (Are you testing the else clause? Are you testing every elsif case?) If you aren't writing the code or aren't sure, use the code coverage facilities.^[*]

^[*] Basic code coverage tools such as Devel::Cover are available in CPAN.

Check out other test suites, too. There are literally tens of thousands of test files on CPAN, and some of them contain hundreds of tests. The Perl distribution itself comes with thousands of tests, designed to verify that Perl compiles correctly on our machine in every possible way. That should be enough examples for anyone. Michael Schwern earned the title of "Perl Test Master" for getting the Perl core completely tested and still constantly beats the drum for "test! test! test!" in the community.