Hobby-hacking Eric


testrunner for practical quickcheck

I had mentioned in a previous post three practical problems I had getting started with QuickCheck. My third question in this post was:
How do I make my tests easy to run? Do I have to write my own RunTests module? Should I just use something like quickcheck-script?
And one of the replies I got:
I'm sure people are writing tests, but we all hack up harnesses in our own idiosyncratic ways.... -- blackdog
Maybe we can do better. Instead of everybody hacking up their own harness, how about having one test harness that everybody wants to use? We may even have a candidate for such a harness. Reinier Lamers has recently released a "testrunner" package which supports some rather nice features:
  • It can run unit tests in parallel.
  • It can run QuickCheck and HUnit tests as well as simple boolean expressions.
  • It comes with a ready-made main function for your unit test executable.
  • This main function recognizes command-line arguments to select tests by name and replay QuickCheck tests.
That's all really good stuff, but I think the number one best feature for me would be the little tutorial on its homepage.

Testrunner is work that Reinier started in the context of the darcs project. We were trying to make our own custom test suite faster and more useful. Seeing ahead, Reinier did it not just by tweaking and tuning the harness we have, but by writing a more general purpose harness that did the things we wanted it to do and hopefully which other projects would want to do as well. So do you have a Haskell project that needs testing? Or maybe you already are doing some tests, but you just wish you could squeeze a little more out of your tests? Give testrunner a try!

Edit 2009-06-08 17:15
It turns out there is a second candidate, or rather a first candidate since test-framework has been around for months. Embarrassingly enough, I had started to use test-framework for my own stuff, but I never realised how feature complete it was. Maybe it'll be time to merge projects? I'll see what Reinier thinks. Apologies to Max...


Echo Nolan said...

Hi Eric.

Are you/Reiner aware of test-framework? It predates testrunner and offers all of: parallel tests, QuickCheck (1 and 2), ready made main function, selecting tests at the command line. If the answer is yes, what motivated starting a new project? I've been using test-framework for a personal project - nothing on the scale of darcs - quite successfully.

kowey said...

Hi Echo,

Actually, I started playing around test-framework in my previous post, and I'm actually rather embarrassed because I didn't notice it had the kinds of features that Reinier implemented (I just noticed what I used, being able to group properties, and having the driver work). Doh!

I guess I should update my post to mention test-framework as well...