Despite all the glowing reports on how useful QuickCheck is, I find that I still have a lot of resistance to using it. A lot of resistance comes from uncertainty, so in this post, I'm going to write down some of my half-formulated questions about using QuickCheck.
Now, there may not be any right answer to these questions, but I'm writing them down anyway so that other people in my shoes know that they are not alone. Later on, as I find the answers that work for me, I'll hopefully put together some notes on 'Practical QuickCheck'.
- Where should I put my properties? Xmonad and darcs seem to put them in a single properties module, but it would seem more natural to me to stick them in the same module as the functions I'm quickchecking. That said, I imagine that some properties can be thought of as being cross-module, so maybe a properties module would make sense.
- How do I avoid redundancy, and generally repeating myself? Ideally, I would just write a property and be done with it. It would annoy me to have to keep updating some list of properties somewhere else (duplication). That said, maybe it's not really duplication if the list serves a secondary purpose of grouping the properties into some sensible hierarchy. Maybe the real question is "how do I make sure I don't forget to run all my properties?"
- How do I make my tests easy to run? Do I have to write my own RunTests module? Should I just use something like quickcheck-script?
I might update this list later as I think of more "best practices" questions. Hopefully I can follow this up with a short article teaching myself and others that really getting started with QuickCheck is easy easy easy (or maybe a link to a pre-existing article of the sort). The Real World Haskell chapter on it seems helpful.
Despite all the glowing reports on how useful QuickCheck is, I find that I still have a lot of resistance to using it. A lot of resistance comes from uncertainty, so in this post, I'm going to write down some of my half-formulated questions about using QuickCheck.
Now, there may not be any right answer to these questions, but I'm writing them down anyway so that other people in my shoes know that they are not alone. Later on, as I find the answers that work for me, I'll hopefully put together some notes on 'Practical QuickCheck'.
- Where should I put my properties? Xmonad and darcs seem to put them in a single properties module, but it would seem more natural to me to stick them in the same module as the functions I'm quickchecking. That said, I imagine that some properties can be thought of as being cross-module, so maybe a properties module would make sense.
- How do I avoid redundancy, and generally repeating myself? Ideally, I would just write a property and be done with it. It would annoy me to have to keep updating some list of properties somewhere else (duplication). That said, maybe it's not really duplication if the list serves a secondary purpose of grouping the properties into some sensible hierarchy. Maybe the real question is "how do I make sure I don't forget to run all my properties?"
- How do I make my tests easy to run? Do I have to write my own RunTests module? Should I just use something like quickcheck-script?
I might update this list later as I think of more "best practices" questions. Hopefully I can follow this up with a short article teaching myself and others that really getting started with QuickCheck is easy easy easy (or maybe a link to a pre-existing article of the sort). The Real World Haskell chapter on it seems helpful.
practical quickcheck (wanted)
8 comments:
Some notes on the thought process: if I put properties in a big properties module, then I have to export the functions I want to test, but to hide (ugh).
If I put all my properties in the same module as the functions I'm testing, then I may have to also (a) export them (b) import them somewhere else, and if I write my own custom driver (c) stick them in a list of properties. I guess I can live with that sort of pain if things I set up so I don't forget to do these things. But then again, maybe the convenience of just putting them all in one module makes sense.
Harumph.
Playing with test-framework
Second thought: maybe what I really want is a sort of driver like quickcheck-script that scans my source code for properties and runs them. The twist is that it should accept some sort of text file that lets me organise my properties if I want. If it finds properties that aren't in the text file, it could emit a sort of warning about orphan properties.
I started writing a script to scan source code for tests, then run them via Test.Framework. (But then I hosed my cabal install so stopped before I got very far. I need to fix that.)
My approach was to write tests in the module they test, and for the script to rewrite a copy of the source code if necessary to ensure they are exported.
Ah-hah! So these dilemmas that I'm facing are not only in my head then, if somebody is willing to write some scripts for them. Thanks! Hope to see something on hackage... :-) (still playing with the test-framework package at the moment)
I totally recognize these thoughts. At one point I had had a lot of exposure to QuickCheck (the authors sit down the hall) but I had various not particularly clear doubts about how to make it work for my code. But then I had to teach some people Haskell, including using QuickCheck, which meant that I had to learn it myself. And after that I was hooked.
Ok, so address your questions I can tell you how have done it. But I should say that I mostly have experience with QuickChecking small code bases.
I tend to write tests in the same module as the thing they're testing. I find this especially appropriate when I want to test for stuff that isn't exported, like the internal state of a data structure. However, I use cpp so that my tests are only there if I want them to, that is, they don't ship with the final compiled code. Since I'm using cpp anyway I might just as well have exported the internals in that case but I find this approach fits my mind more easily.
I do maintain a list of properties that are going to be run. I don't find that cumbersome at all, but maybe that's because I've mostly worked on small projects. I use this list for various things, like giving the properties nice names which I print out while testing them. I sometimes also have special testing parameters that I give in this list.
As for making sure that the properties are being tested, I make sure darcs run the tests each time I record a change.
Generally I find the bookkeeping that you seem to be afraid of not to be a problem in practice. The important thing is to decide on a setup, once you've done that it's not so hard to maintain.
Happy QuickChecking!
For Bimap, I wrote a little haskell module that searches a file for functions starting with "prop_" and uses TH to splice them into a little test harness.
There might be something better up on Hackage these days, but if not check out my solution.
I've been wrestling with the same thing - my experience in the perl world is that there's a nice infrastructure for writing tests, and I haven't found it in Haskell. I'm sure people are writing tests, but we all hack up harnesses in our own idiosyncratic ways....
Thanks for the comments, everyone! This makes me even more motivated to work something out (Josef's remark that the bookkeeping is also not so bad in practice particularly interests me).
@blackdog: the test-framework package may help. Also, I checked out Stuart's TH solution, and commented in an email:
I wonder if it's worthwhile to spin something like this into a general
solution. Maybe it could print out some JSON with the property, its
module and the results, which one could then pass to a display script.
Or maybe I'm making things overly complicated :-)
Post a Comment