some ideas for practical QuickCheck

I think I've found some answers to my practical QuickCheck questions. This post may be fairly long as I'm trying to make it concrete and explicit enough to overcome the kind of inertia I had when I was still resisting testing.

How do I make my tests easy to run?

1. Use test-framework

The key thing to know about test-framework is that it is very easy to get started. Just visit the friendly web page and copy the example.

Note: An earlier post suggested the testrunner package developed for Darcs, but at the time we didn't realise that test-framework already had all the features needed.

2. Support cabal test

Here's a Setup.hs recipe I copied. It has the handy property of the code is that it runs your tests straight from your dist/build directory.

-- EXAMPLE Setup.hs FILE 1 -----------------------------------------------
import System.FilePath

main = defaultMainWithHooks hooks
  where hooks = simpleUserHooks { runTests = runTests' }

runTests' :: Args -> Bool -> PackageDescription -> LocalBuildInfo -> IO ()
runTests' _ _ _ lbi = system testprog >> return ()
  where testprog = (buildDir lbi) </> "test" </> "test"
-- -----------------------------------------------------------------------

The code snippet for your Setup.hs file comes from Greg Bacon's Setting up a Simple Test with Cabal (I tacked on an import). As you can see, the recipe assumes you're building an executable called "test" (see Greg's post on how to do this)

3. Bake your unit tests in

This may go down as the kind of bad advice that "seemed like a good idea at the the time". For now, I can justify this by saying that it may be reassuring to users to be able to just run the same tests that I'm running and see for themselves that their program thinks it's working.

I've been working on a program called GenI. To help people test this program, I've added a simple "--tests" switch. Now people can run geni --tests for a self check. If they want, they can also "cabal test", using this slight modification to Greg's setup file (to call geni itself and to pass the --tests flag in).

-- EXAMPLE Setup.hs FILE 2 -----------------------------------------------

import System.FilePath

main = defaultMainWithHooks hooks
  where hooks = simpleUserHooks { runTests = runTests' }

runTests' :: Args -> Bool -> PackageDescription -> LocalBuildInfo -> IO ()
runTests' _ _ _ lbi = system testprog >> return ()
  where testprog = (buildDir lbi) </> "geni" </> "geni --tests"

-- -----------------------------------------------------------------------

As for GenI, whenever I see --tests in my arguments (for example "--tests" `elem` args), I just pass control to another module, which in turn strips the switch out and passes the rest of the arguments to test-framework.

-- EXAMPLE TEST-FRAMEWORK WRAPPER ------------------------------------------
module NLP.GenI.Test where

import System.Environment ( getArgs )
import Test.Framework

import NLP.GenI.GeniVal ( testSuite )
import NLP.GenI.Tags ( testSuite )
import NLP.GenI.Simple.SimpleBuilder ( testSuite )

runTests :: IO ()
runTests =
 do args <- filter (/= "--tests") `fmap` getArgs
    flip defaultMainWithArgs args
     [ NLP.GenI.GeniVal.testSuite
     , NLP.GenI.Tags.testSuite
     , NLP.GenI.Simple.SimpleBuilder.testSuite
     ]
-- -----------------------------------------------------------------------

There's some other things going on in this file, notably the organisation of test suites. More on that later.

Where should I put my properties?

4. Put tests in the same module (where relevant)

If a test is specific to one module, I tend to put them in that same source file. I do this because

1. It lets me test functions that I don't want to export
2. The tests serve as documentation
3. It forces me to update my tests along with my code

This approach is in contrast to (a) having one big tests module and (b) having a separate test hierarchy. It may turn out to be useful to have a single big tests module as well, for example, for tests that cross the boundary from one module to the next. That need has not arisen for me yet. Likewise, I don't particularly believe in a separation between tests and code, although on the other hand some very experienced hackers seem to do so, so I'll just have to let experience teach me why.

How do I avoid repeating myself?

5. Provide a testSuite function for each module

Commenting on my last post, Josef kindly pointed out that the book-keeping I feared isn't so bad in practice. He's right. Nevertheless, I want to avoid it. To do this, I make each of my modules export a testSuite function. Here is what one of my modules looks like, just focusing on the test suite

-- EXAMPLE MODULE --------------------------------------------------------
module NLP.GenI.GeniVal where

-- SKIPPED MAIN IMPORTS ...

import Test.Framework
import Test.Framework.Providers.HUnit
import Test.Framework.Providers.QuickCheck
import Test.QuickCheck
import Test.HUnit

-- SKIPPED MAIN CODE

testSuite = testGroup "unification"
 [ testProperty "self" prop_unify_sym
 , testProperty "anonymous variables" prop_unify_anon
 , testProperty "symmetry" prop_unify_sym
 , testCase "evil unification" test_evil
 ]

-- SKIPPED THE TESTS THEMSELVES
-- -----------------------------------------------------------------------

If you'll scroll up to the example that's marked TEST-FRAMEWORK WRAPPER, you'll see how these test suites are used in practice. Note the small trick of using the qualified module name to identify the test suite.

Anyway, the general principle of having a per-module test suite comes from Aidan Delaney's Organising Unit Tests in Haskell. The main difference between his approach and my approach are that I mix tests and code rather liberally.

Conclusion

I hope that some of these hints will make testing easier for you, or perhaps even get you started. If you still find yourself putting testing off, let me know. I'll be curious to see what else makes us resist. One thing that would probably be helpful is an extra guide to writing Arbitrary instances for QuickCheck, and also writing good properties that control the space well. Maybe even getting started with SmallCheck.

Note that I am still somewhat new to testing and have only recently started these practices. So take these ideas with the usual salt. Thanks to Greg, Reinier, Aidan, and also folks who commented on my previous posts.

I think I've found some answers to my practical QuickCheck questions. This post may be fairly long as I'm trying to make it concrete and explicit enough to overcome the kind of inertia I had when I was still resisting testing.

How do I make my tests easy to run?

1. Use test-framework

The key thing to know about test-framework is that it is very easy to get started. Just visit the friendly web page and copy the example.

Note: An earlier post suggested the testrunner package developed for Darcs, but at the time we didn't realise that test-framework already had all the features needed.

2. Support cabal test

Here's a Setup.hs recipe I copied. It has the handy property of the code is that it runs your tests straight from your dist/build directory.

-- EXAMPLE Setup.hs FILE 1 -----------------------------------------------
import System.FilePath

main = defaultMainWithHooks hooks
  where hooks = simpleUserHooks { runTests = runTests' }

runTests' :: Args -> Bool -> PackageDescription -> LocalBuildInfo -> IO ()
runTests' _ _ _ lbi = system testprog >> return ()
  where testprog = (buildDir lbi) </> "test" </> "test"
-- -----------------------------------------------------------------------

The code snippet for your Setup.hs file comes from Greg Bacon's Setting up a Simple Test with Cabal (I tacked on an import). As you can see, the recipe assumes you're building an executable called "test" (see Greg's post on how to do this)

3. Bake your unit tests in

This may go down as the kind of bad advice that "seemed like a good idea at the the time". For now, I can justify this by saying that it may be reassuring to users to be able to just run the same tests that I'm running and see for themselves that their program thinks it's working.

I've been working on a program called GenI. To help people test this program, I've added a simple "--tests" switch. Now people can run geni --tests for a self check. If they want, they can also "cabal test", using this slight modification to Greg's setup file (to call geni itself and to pass the --tests flag in).

-- EXAMPLE Setup.hs FILE 2 -----------------------------------------------

import System.FilePath

main = defaultMainWithHooks hooks
  where hooks = simpleUserHooks { runTests = runTests' }

runTests' :: Args -> Bool -> PackageDescription -> LocalBuildInfo -> IO ()
runTests' _ _ _ lbi = system testprog >> return ()
  where testprog = (buildDir lbi) </> "geni" </> "geni --tests"

-- -----------------------------------------------------------------------

As for GenI, whenever I see --tests in my arguments (for example "--tests" `elem` args), I just pass control to another module, which in turn strips the switch out and passes the rest of the arguments to test-framework.

-- EXAMPLE TEST-FRAMEWORK WRAPPER ------------------------------------------
module NLP.GenI.Test where

import System.Environment ( getArgs )
import Test.Framework

import NLP.GenI.GeniVal ( testSuite )
import NLP.GenI.Tags ( testSuite )
import NLP.GenI.Simple.SimpleBuilder ( testSuite )

runTests :: IO ()
runTests =
 do args <- filter (/= "--tests") `fmap` getArgs
    flip defaultMainWithArgs args
     [ NLP.GenI.GeniVal.testSuite
     , NLP.GenI.Tags.testSuite
     , NLP.GenI.Simple.SimpleBuilder.testSuite
     ]
-- -----------------------------------------------------------------------

There's some other things going on in this file, notably the organisation of test suites. More on that later.

Where should I put my properties?

4. Put tests in the same module (where relevant)

If a test is specific to one module, I tend to put them in that same source file. I do this because

1. It lets me test functions that I don't want to export
2. The tests serve as documentation
3. It forces me to update my tests along with my code

This approach is in contrast to (a) having one big tests module and (b) having a separate test hierarchy. It may turn out to be useful to have a single big tests module as well, for example, for tests that cross the boundary from one module to the next. That need has not arisen for me yet. Likewise, I don't particularly believe in a separation between tests and code, although on the other hand some very experienced hackers seem to do so, so I'll just have to let experience teach me why.

How do I avoid repeating myself?

5. Provide a testSuite function for each module

Commenting on my last post, Josef kindly pointed out that the book-keeping I feared isn't so bad in practice. He's right. Nevertheless, I want to avoid it. To do this, I make each of my modules export a testSuite function. Here is what one of my modules looks like, just focusing on the test suite

-- EXAMPLE MODULE --------------------------------------------------------
module NLP.GenI.GeniVal where

-- SKIPPED MAIN IMPORTS ...

import Test.Framework
import Test.Framework.Providers.HUnit
import Test.Framework.Providers.QuickCheck
import Test.QuickCheck
import Test.HUnit

-- SKIPPED MAIN CODE

testSuite = testGroup "unification"
 [ testProperty "self" prop_unify_sym
 , testProperty "anonymous variables" prop_unify_anon
 , testProperty "symmetry" prop_unify_sym
 , testCase "evil unification" test_evil
 ]

-- SKIPPED THE TESTS THEMSELVES
-- -----------------------------------------------------------------------

If you'll scroll up to the example that's marked TEST-FRAMEWORK WRAPPER, you'll see how these test suites are used in practice. Note the small trick of using the qualified module name to identify the test suite.

Anyway, the general principle of having a per-module test suite comes from Aidan Delaney's Organising Unit Tests in Haskell. The main difference between his approach and my approach are that I mix tests and code rather liberally.

Conclusion

I hope that some of these hints will make testing easier for you, or perhaps even get you started. If you still find yourself putting testing off, let me know. I'll be curious to see what else makes us resist. One thing that would probably be helpful is an extra guide to writing Arbitrary instances for QuickCheck, and also writing good properties that control the space well. Maybe even getting started with SmallCheck.

Note that I am still somewhat new to testing and have only recently started these practices. So take these ideas with the usual salt. Thanks to Greg, Reinier, Aidan, and also folks who commented on my previous posts.

some ideas for practical QuickCheck