Hobby-hacking Eric

Showing posts with label quickcheck. Show all posts
Showing posts with label quickcheck. Show all posts

2011-02-18

practical QuickCheck revisited - separate testing hierarchy

I'll begin this post with a quote from 2009-Eric:
This may go down as the kind of bad advice that "seemed like a good idea at the the time".
The advice in question was to "bake unit tests in".  The basic idea was that whatever module you write should have its own testSuite function exposing unit tests for that particular module.  The advantages were simplicity (no parallel test hierarchy), the ability to ship a binary with self-tests, and the ability to non-exported functions, helper code with a granularity that lends itself more to testing (easier to think of tests for them).

I was unconvinced by the counterargument that it was not a good idea to mix testing and business logic.  To be clear, I did agree with the spirit of the advice -- I'm not about go around questioning the kind of wisdom a community gains by watching rockets blow up -- but I felt that I was not advocating any such mixing.  All I wanted was to put my testing code in the same file as the business code, cordoned off in a testing section at the end of the file if you want without any sort of if-testing-mode-do-X logic.  So I thought that the counterargument was right, but that it didn't apply to this particular context.  (I'd be interested to see when/if I change my mind on this, maybe it leads to temptation to mix logic, which is bad.)

In any case, I don't need to change my mind on that particular point. Being the kind of person that only learns the hard way, I've found myself forced to divorce my test code from the business code after all. It's mainly a practical problem of dependencies (this was pointed out by Echo Nolan and Ivan Miljenovic). Forcing users to install QuickCheck and test-framework, when they probably don't care about testing, when they just see your module as yet another dependency on the the road to some other more pressing goal, is really a bit anti-social.

The problem isn't installing the package per se (it all happens automatically with cabal install), but dealing with package version dependencies.  So GenI depends on test-framework 2.x and QuickCheck 1.2.  What if I go away for a few years, stop hacking on GenI and in the meantime the rest of the world moves on to using QuickCheck 2.x and test-framework 3?  What happens when they try to install GenI and cabal install needs to rebuild the random package, which then breaks QuickCheck-2.4 because it depends on random too.  Headaches all around.

I think I can live with a separate hierarchy. Arguing with past-Eric a bit:
  1. All the extra modules and what not are not that big a deal (and I could probably let myself go wrt imports, etc).
  2. Who cares if there's an extra geni-test binary, which only gets enable with -ftest anyway?
  3. Self tests, shmelf tests.  Seriously, who is going to run that geni --test function anyway?
  4. If I forget to cabal configure -ftest, I can always cabal configure again and build
  5. If I'm really desperate to test some internal function, I could always export an alias like testingFoo for every foo I want to test, applying a sort of Pythonesque we're-all-grownups-here principle. 
  6. Also maybe forcing yourself to test only the exported functions, enforces a kind of general black-box thinking which is healthy if you're writing a library.
So, with apologies to Ivan for not understanding his rants 2 years ago; and also anyone that may have listened to 2009-Eric for any messes I got you and your users in, I'm retracting that particular bit of advice and separating my test hierarchies like a good boy.   Let's see if 2013-Eric decides to post some kind of retraction retraction.


2009-07-28

some ideas for practical QuickCheck

I think I've found some answers to my practical QuickCheck questions. This post may be fairly long as I'm trying to make it concrete and explicit enough to overcome the kind of inertia I had when I was still resisting testing.

How do I make my tests easy to run?

1. Use test-framework
The key thing to know about test-framework is that it is very easy to get started. Just visit the friendly web page and copy the example.

Note: An earlier post suggested the testrunner package developed for Darcs, but at the time we didn't realise that test-framework already had all the features needed.
2. Support cabal test
Here's a Setup.hs recipe I copied. It has the handy property of the code is that it runs your tests straight from your dist/build directory.
-- EXAMPLE Setup.hs FILE 1 -----------------------------------------------
import System.FilePath

main = defaultMainWithHooks hooks
where hooks = simpleUserHooks { runTests = runTests' }

runTests' :: Args -> Bool -> PackageDescription -> LocalBuildInfo -> IO ()
runTests' _ _ _ lbi = system testprog >> return ()
where testprog = (buildDir lbi) </> "test" </> "test"
-- -----------------------------------------------------------------------
The code snippet for your Setup.hs file comes from Greg Bacon's Setting up a Simple Test with Cabal (I tacked on an import). As you can see, the recipe assumes you're building an executable called "test" (see Greg's post on how to do this)
3. Bake your unit tests in
This may go down as the kind of bad advice that "seemed like a good idea at the the time". For now, I can justify this by saying that it may be reassuring to users to be able to just run the same tests that I'm running and see for themselves that their program thinks it's working.

I've been working on a program called GenI. To help people test this program, I've added a simple "--tests" switch. Now people can run geni --tests for a self check. If they want, they can also "cabal test", using this slight modification to Greg's setup file (to call geni itself and to pass the --tests flag in).
-- EXAMPLE Setup.hs FILE 2 -----------------------------------------------

import System.FilePath

main = defaultMainWithHooks hooks
where hooks = simpleUserHooks { runTests = runTests' }

runTests' :: Args -> Bool -> PackageDescription -> LocalBuildInfo -> IO ()
runTests' _ _ _ lbi = system testprog >> return ()
where testprog = (buildDir lbi) </> "geni" </> "geni --tests"

-- -----------------------------------------------------------------------
As for GenI, whenever I see --tests in my arguments (for example "--tests" `elem` args), I just pass control to another module, which in turn strips the switch out and passes the rest of the arguments to test-framework.
-- EXAMPLE TEST-FRAMEWORK WRAPPER ------------------------------------------
module NLP.GenI.Test where

import System.Environment ( getArgs )
import Test.Framework

import NLP.GenI.GeniVal ( testSuite )
import NLP.GenI.Tags ( testSuite )
import NLP.GenI.Simple.SimpleBuilder ( testSuite )

runTests :: IO ()
runTests =
do args <- filter (/= "--tests") `fmap` getArgs
flip defaultMainWithArgs args
[ NLP.GenI.GeniVal.testSuite
, NLP.GenI.Tags.testSuite
, NLP.GenI.Simple.SimpleBuilder.testSuite
]
-- -----------------------------------------------------------------------
There's some other things going on in this file, notably the organisation of test suites. More on that later.

Where should I put my properties?

4. Put tests in the same module (where relevant)
If a test is specific to one module, I tend to put them in that same source file. I do this because
  1. It lets me test functions that I don't want to export
  2. The tests serve as documentation
  3. It forces me to update my tests along with my code
This approach is in contrast to (a) having one big tests module and (b) having a separate test hierarchy. It may turn out to be useful to have a single big tests module as well, for example, for tests that cross the boundary from one module to the next. That need has not arisen for me yet. Likewise, I don't particularly believe in a separation between tests and code, although on the other hand some very experienced hackers seem to do so, so I'll just have to let experience teach me why.

How do I avoid repeating myself?

5. Provide a testSuite function for each module
Commenting on my last post, Josef kindly pointed out that the book-keeping I feared isn't so bad in practice. He's right. Nevertheless, I want to avoid it. To do this, I make each of my modules export a testSuite function. Here is what one of my modules looks like, just focusing on the test suite
-- EXAMPLE MODULE --------------------------------------------------------
module NLP.GenI.GeniVal where

-- SKIPPED MAIN IMPORTS ...

import Test.Framework
import Test.Framework.Providers.HUnit
import Test.Framework.Providers.QuickCheck
import Test.QuickCheck
import Test.HUnit

-- SKIPPED MAIN CODE

testSuite = testGroup "unification"
[ testProperty "self" prop_unify_sym
, testProperty "anonymous variables" prop_unify_anon
, testProperty "symmetry" prop_unify_sym
, testCase "evil unification" test_evil
]

-- SKIPPED THE TESTS THEMSELVES
-- -----------------------------------------------------------------------
If you'll scroll up to the example that's marked TEST-FRAMEWORK WRAPPER, you'll see how these test suites are used in practice. Note the small trick of using the qualified module name to identify the test suite.

Anyway, the general principle of having a per-module test suite comes from Aidan Delaney's Organising Unit Tests in Haskell. The main difference between his approach and my approach are that I mix tests and code rather liberally.

Conclusion

I hope that some of these hints will make testing easier for you, or perhaps even get you started. If you still find yourself putting testing off, let me know. I'll be curious to see what else makes us resist. One thing that would probably be helpful is an extra guide to writing Arbitrary instances for QuickCheck, and also writing good properties that control the space well. Maybe even getting started with SmallCheck.

Note that I am still somewhat new to testing and have only recently started these practices. So take these ideas with the usual salt. Thanks to Greg, Reinier, Aidan, and also folks who commented on my previous posts.


2009-06-08

testrunner for practical quickcheck

I had mentioned in a previous post three practical problems I had getting started with QuickCheck. My third question in this post was:
How do I make my tests easy to run? Do I have to write my own RunTests module? Should I just use something like quickcheck-script?
And one of the replies I got:
I'm sure people are writing tests, but we all hack up harnesses in our own idiosyncratic ways.... -- blackdog
Maybe we can do better. Instead of everybody hacking up their own harness, how about having one test harness that everybody wants to use? We may even have a candidate for such a harness. Reinier Lamers has recently released a "testrunner" package which supports some rather nice features:
  • It can run unit tests in parallel.
  • It can run QuickCheck and HUnit tests as well as simple boolean expressions.
  • It comes with a ready-made main function for your unit test executable.
  • This main function recognizes command-line arguments to select tests by name and replay QuickCheck tests.
That's all really good stuff, but I think the number one best feature for me would be the little tutorial on its homepage.

Testrunner is work that Reinier started in the context of the darcs project. We were trying to make our own custom test suite faster and more useful. Seeing ahead, Reinier did it not just by tweaking and tuning the harness we have, but by writing a more general purpose harness that did the things we wanted it to do and hopefully which other projects would want to do as well. So do you have a Haskell project that needs testing? Or maybe you already are doing some tests, but you just wish you could squeeze a little more out of your tests? Give testrunner a try!

Edit 2009-06-08 17:15
It turns out there is a second candidate, or rather a first candidate since test-framework has been around for months. Embarrassingly enough, I had started to use test-framework for my own stuff, but I never realised how feature complete it was. Maybe it'll be time to merge projects? I'll see what Reinier thinks. Apologies to Max...


2009-02-04

practical quickcheck (wanted)

Despite all the glowing reports on how useful QuickCheck is, I find that I still have a lot of resistance to using it. A lot of resistance comes from uncertainty, so in this post, I'm going to write down some of my half-formulated questions about using QuickCheck.

Now, there may not be any right answer to these questions, but I'm writing them down anyway so that other people in my shoes know that they are not alone. Later on, as I find the answers that work for me, I'll hopefully put together some notes on 'Practical QuickCheck'.
  1. Where should I put my properties? Xmonad and darcs seem to put them in a single properties module, but it would seem more natural to me to stick them in the same module as the functions I'm quickchecking. That said, I imagine that some properties can be thought of as being cross-module, so maybe a properties module would make sense.
  2. How do I avoid redundancy, and generally repeating myself? Ideally, I would just write a property and be done with it. It would annoy me to have to keep updating some list of properties somewhere else (duplication). That said, maybe it's not really duplication if the list serves a secondary purpose of grouping the properties into some sensible hierarchy. Maybe the real question is "how do I make sure I don't forget to run all my properties?"
  3. How do I make my tests easy to run? Do I have to write my own RunTests module? Should I just use something like quickcheck-script?
I might update this list later as I think of more "best practices" questions. Hopefully I can follow this up with a short article teaching myself and others that really getting started with QuickCheck is easy easy easy (or maybe a link to a pre-existing article of the sort). The Real World Haskell chapter on it seems helpful.