Hobby-hacking Eric

2009-07-28

some ideas for practical QuickCheck

I think I've found some answers to my practical QuickCheck questions. This post may be fairly long as I'm trying to make it concrete and explicit enough to overcome the kind of inertia I had when I was still resisting testing.

How do I make my tests easy to run?

1. Use test-framework
The key thing to know about test-framework is that it is very easy to get started. Just visit the friendly web page and copy the example.

Note: An earlier post suggested the testrunner package developed for Darcs, but at the time we didn't realise that test-framework already had all the features needed.
2. Support cabal test
Here's a Setup.hs recipe I copied. It has the handy property of the code is that it runs your tests straight from your dist/build directory.
-- EXAMPLE Setup.hs FILE 1 -----------------------------------------------
import System.FilePath

main = defaultMainWithHooks hooks
where hooks = simpleUserHooks { runTests = runTests' }

runTests' :: Args -> Bool -> PackageDescription -> LocalBuildInfo -> IO ()
runTests' _ _ _ lbi = system testprog >> return ()
where testprog = (buildDir lbi) </> "test" </> "test"
-- -----------------------------------------------------------------------
The code snippet for your Setup.hs file comes from Greg Bacon's Setting up a Simple Test with Cabal (I tacked on an import). As you can see, the recipe assumes you're building an executable called "test" (see Greg's post on how to do this)
3. Bake your unit tests in
This may go down as the kind of bad advice that "seemed like a good idea at the the time". For now, I can justify this by saying that it may be reassuring to users to be able to just run the same tests that I'm running and see for themselves that their program thinks it's working.

I've been working on a program called GenI. To help people test this program, I've added a simple "--tests" switch. Now people can run geni --tests for a self check. If they want, they can also "cabal test", using this slight modification to Greg's setup file (to call geni itself and to pass the --tests flag in).
-- EXAMPLE Setup.hs FILE 2 -----------------------------------------------

import System.FilePath

main = defaultMainWithHooks hooks
where hooks = simpleUserHooks { runTests = runTests' }

runTests' :: Args -> Bool -> PackageDescription -> LocalBuildInfo -> IO ()
runTests' _ _ _ lbi = system testprog >> return ()
where testprog = (buildDir lbi) </> "geni" </> "geni --tests"

-- -----------------------------------------------------------------------
As for GenI, whenever I see --tests in my arguments (for example "--tests" `elem` args), I just pass control to another module, which in turn strips the switch out and passes the rest of the arguments to test-framework.
-- EXAMPLE TEST-FRAMEWORK WRAPPER ------------------------------------------
module NLP.GenI.Test where

import System.Environment ( getArgs )
import Test.Framework

import NLP.GenI.GeniVal ( testSuite )
import NLP.GenI.Tags ( testSuite )
import NLP.GenI.Simple.SimpleBuilder ( testSuite )

runTests :: IO ()
runTests =
do args <- filter (/= "--tests") `fmap` getArgs
flip defaultMainWithArgs args
[ NLP.GenI.GeniVal.testSuite
, NLP.GenI.Tags.testSuite
, NLP.GenI.Simple.SimpleBuilder.testSuite
]
-- -----------------------------------------------------------------------
There's some other things going on in this file, notably the organisation of test suites. More on that later.

Where should I put my properties?

4. Put tests in the same module (where relevant)
If a test is specific to one module, I tend to put them in that same source file. I do this because
  1. It lets me test functions that I don't want to export
  2. The tests serve as documentation
  3. It forces me to update my tests along with my code
This approach is in contrast to (a) having one big tests module and (b) having a separate test hierarchy. It may turn out to be useful to have a single big tests module as well, for example, for tests that cross the boundary from one module to the next. That need has not arisen for me yet. Likewise, I don't particularly believe in a separation between tests and code, although on the other hand some very experienced hackers seem to do so, so I'll just have to let experience teach me why.

How do I avoid repeating myself?

5. Provide a testSuite function for each module
Commenting on my last post, Josef kindly pointed out that the book-keeping I feared isn't so bad in practice. He's right. Nevertheless, I want to avoid it. To do this, I make each of my modules export a testSuite function. Here is what one of my modules looks like, just focusing on the test suite
-- EXAMPLE MODULE --------------------------------------------------------
module NLP.GenI.GeniVal where

-- SKIPPED MAIN IMPORTS ...

import Test.Framework
import Test.Framework.Providers.HUnit
import Test.Framework.Providers.QuickCheck
import Test.QuickCheck
import Test.HUnit

-- SKIPPED MAIN CODE

testSuite = testGroup "unification"
[ testProperty "self" prop_unify_sym
, testProperty "anonymous variables" prop_unify_anon
, testProperty "symmetry" prop_unify_sym
, testCase "evil unification" test_evil
]

-- SKIPPED THE TESTS THEMSELVES
-- -----------------------------------------------------------------------
If you'll scroll up to the example that's marked TEST-FRAMEWORK WRAPPER, you'll see how these test suites are used in practice. Note the small trick of using the qualified module name to identify the test suite.

Anyway, the general principle of having a per-module test suite comes from Aidan Delaney's Organising Unit Tests in Haskell. The main difference between his approach and my approach are that I mix tests and code rather liberally.

Conclusion

I hope that some of these hints will make testing easier for you, or perhaps even get you started. If you still find yourself putting testing off, let me know. I'll be curious to see what else makes us resist. One thing that would probably be helpful is an extra guide to writing Arbitrary instances for QuickCheck, and also writing good properties that control the space well. Maybe even getting started with SmallCheck.

Note that I am still somewhat new to testing and have only recently started these practices. So take these ideas with the usual salt. Thanks to Greg, Reinier, Aidan, and also folks who commented on my previous posts.


12 comments:

Wilkes Joiner said...

I've been looking for a post like this, especially the cabal integration. Can't wait to try it out.

I waffle on where to keep the tests. I've grown up with test being separate, but then I have to export functions that I don't want exposed outside of the module.

kowey said...

Good luck with that! Don't forget to set your build type with Custom and do heed the disclaimer about me not necessarily being in a position to give advice about these sorts of things :-)

Björn said...

Very useful post; this is exactly the kind of stuff that helps smooth the bumps when starting Haskell. Thanks a lot!

With this motivation, I'll just have to (finally!) write some tests when I get off work. ;-)

LAca said...

I don't think it's a good idea to put test code in the production code. Check this out!

kowey said...

Thanks for the link! For the moment, my test logic looks like it's cleanly separated from the rest of my code, no crazy interleaving (thanks to the functional style, perhaps?). But it's interesting to consider.

I know some great programmers who are pretty clear that this is a bad idea, so maybe I will burn my fingers on this :-)

Echo Nolan said...

I've another reason to separate tests from functionality: simplifying dependencies. Since my tests are in their own files and have their own executable section in my .cabal, the main of the program doesn't need to depend on QuickCheck/HUnit/etc. This is a win if someone ever wants to link my library and use, e.g. QuickCheck-1 rather than QuickCheck-2.

Ivan said...

My problem with having a --test option to your program is that you then require that users (who usually don't care about tests) must install QC just to build it.

Peter said...

Mark Wotton and I have recently tried to address this problem. TBC has Cabal integration, a fact I should have pushed more in my haskell-cafe announcement.

Hackage: http://hackage.haskell.org/package/TBC

It makes tests easy to run by adopting conventions for identifiers, e.g. "prop_blah" is a QuickCheck property, etc. No more forgetting to add tests to lists!

The other key feature is that it will run all runnable tests, so you can actually do test-driven development.

"cabal test" is broken, forget about it for now. TBC provides a hook but it is a lot more primitive than the 'tbc' binary. A quick look at Cabal's trac will show that the 'test' hook has been in need of love for years.

TBC is agnostic about where the tests go - conventionally, separate tests go in $PROJECT/Tests, but you could bake them into your source code if you like.

Please try it out and give us some feedback.

Thanks,
Peter.

kowey said...

Peter: anything that will let me write less code is welcome! I'll add it to my list of things to try.

Echo and Ivan: points taken. Ivan, I don't really mind making folks install QC because it just automagically happens with cabal anyway, although I do sympathasise more with Echo's point that people might like more control, for example over QC version numbers.

I definitely seem to be in the minority here about mixing tests with code, and also about baking them in and probably for good reason. Hopefully I will figure it out before I have to learn the hard way!

polux said...

Thank you Eric, this has been helpful !

kowey said...

There are loads of imports missing in that Setup.hs example: Distribution.PackageDescription, Distribution.Simple, Distribution.Simple.{LocalBuildInfo,Setup}.

Sorry for that!

Gabriel said...

The System.FilePath import appears to be problematic on Hackage- it produces build failures as this one.

A slighly less clean (but hackage-friendly) variant picks the path separators using CPP, see e.g. this Setup.hs.