Hobby-hacking Eric

2011-02-18

practical QuickCheck revisited - separate testing hierarchy

I'll begin this post with a quote from 2009-Eric:
This may go down as the kind of bad advice that "seemed like a good idea at the the time".
The advice in question was to "bake unit tests in".  The basic idea was that whatever module you write should have its own testSuite function exposing unit tests for that particular module.  The advantages were simplicity (no parallel test hierarchy), the ability to ship a binary with self-tests, and the ability to non-exported functions, helper code with a granularity that lends itself more to testing (easier to think of tests for them).

I was unconvinced by the counterargument that it was not a good idea to mix testing and business logic.  To be clear, I did agree with the spirit of the advice -- I'm not about go around questioning the kind of wisdom a community gains by watching rockets blow up -- but I felt that I was not advocating any such mixing.  All I wanted was to put my testing code in the same file as the business code, cordoned off in a testing section at the end of the file if you want without any sort of if-testing-mode-do-X logic.  So I thought that the counterargument was right, but that it didn't apply to this particular context.  (I'd be interested to see when/if I change my mind on this, maybe it leads to temptation to mix logic, which is bad.)

In any case, I don't need to change my mind on that particular point. Being the kind of person that only learns the hard way, I've found myself forced to divorce my test code from the business code after all. It's mainly a practical problem of dependencies (this was pointed out by Echo Nolan and Ivan Miljenovic). Forcing users to install QuickCheck and test-framework, when they probably don't care about testing, when they just see your module as yet another dependency on the the road to some other more pressing goal, is really a bit anti-social.

The problem isn't installing the package per se (it all happens automatically with cabal install), but dealing with package version dependencies.  So GenI depends on test-framework 2.x and QuickCheck 1.2.  What if I go away for a few years, stop hacking on GenI and in the meantime the rest of the world moves on to using QuickCheck 2.x and test-framework 3?  What happens when they try to install GenI and cabal install needs to rebuild the random package, which then breaks QuickCheck-2.4 because it depends on random too.  Headaches all around.

I think I can live with a separate hierarchy. Arguing with past-Eric a bit:
  1. All the extra modules and what not are not that big a deal (and I could probably let myself go wrt imports, etc).
  2. Who cares if there's an extra geni-test binary, which only gets enable with -ftest anyway?
  3. Self tests, shmelf tests.  Seriously, who is going to run that geni --test function anyway?
  4. If I forget to cabal configure -ftest, I can always cabal configure again and build
  5. If I'm really desperate to test some internal function, I could always export an alias like testingFoo for every foo I want to test, applying a sort of Pythonesque we're-all-grownups-here principle. 
  6. Also maybe forcing yourself to test only the exported functions, enforces a kind of general black-box thinking which is healthy if you're writing a library.
So, with apologies to Ivan for not understanding his rants 2 years ago; and also anyone that may have listened to 2009-Eric for any messes I got you and your users in, I'm retracting that particular bit of advice and separating my test hierarchies like a good boy.   Let's see if 2013-Eric decides to post some kind of retraction retraction.


6 comments:

Johan Tibell said...

Instead of creating an executable stanza you can use Cabal's new testing support (the test-suite stanza). Available in Cabal 1.10.

kowey said...

Nice, Johan! I'll be looking forward to try it. I knew I forgot something. Was mentioning to add a note about hoping to put Thomas' GSoC work to use. We'll see if I have anything to say having tried it out at some point.

Erik said...

Another option for testing internal functions is exporting them in a .Internal module. This is a relatively common pattern (e.g. in bytestring and text), and it also allows people to use your internal building blocks in unexpected ways.

Yitz said...

I agree with the spirit of your original idea, though - that there should be a standard place to find the tests. While the new test-suite stanza in Cabal will make it easy for machines to find the tests, there is still value in having a standard place where humans can expect to find them. Perhaps .Tests.testSuite would be good.

Johan Tibell said...

Yitz, many libraries seems to put them directly into tests/. I guess once you have a large collection you'll need to start a hierarchy under that directory.

Christopher Lewis said...

Somewhat related to 6. I've also found that my ability to write clear tests for my exported functions can serves as a check on the completeness of the functionality offered by a module. Of course, this is not always the case, but it's a nice additional check to have.