practical QuickCheck revisited - separate testing hierarchy

I'll begin this post with a quote from 2009-Eric:

This may go down as the kind of bad advice that "seemed like a good idea at the the time".

The advice in question was to "bake unit tests in".  The basic idea was that whatever module you write should have its own testSuite function exposing unit tests for that particular module.  The advantages were simplicity (no parallel test hierarchy), the ability to ship a binary with self-tests, and the ability to non-exported functions, helper code with a granularity that lends itself more to testing (easier to think of tests for them).

I was unconvinced by the counterargument that it was not a good idea to mix testing and business logic.  To be clear, I did agree with the spirit of the advice -- I'm not about go around questioning the kind of wisdom a community gains by watching rockets blow up -- but I felt that I was not advocating any such mixing.  All I wanted was to put my testing code in the same file as the business code, cordoned off in a testing section at the end of the file if you want without any sort of if-testing-mode-do-X logic.  So I thought that the counterargument was right, but that it didn't apply to this particular context.  (I'd be interested to see when/if I change my mind on this, maybe it leads to temptation to mix logic, which is bad.)

In any case, I don't need to change my mind on that particular point. Being the kind of person that only learns the hard way, I've found myself forced to divorce my test code from the business code after all. It's mainly a practical problem of dependencies (this was pointed out by Echo Nolan and Ivan Miljenovic). Forcing users to install QuickCheck and test-framework, when they probably don't care about testing, when they just see your module as yet another dependency on the the road to some other more pressing goal, is really a bit anti-social.

The problem isn't installing the package per se (it all happens automatically with cabal install), but dealing with package version dependencies.  So GenI depends on test-framework 2.x and QuickCheck 1.2.  What if I go away for a few years, stop hacking on GenI and in the meantime the rest of the world moves on to using QuickCheck 2.x and test-framework 3?  What happens when they try to install GenI and cabal install needs to rebuild the random package, which then breaks QuickCheck-2.4 because it depends on random too.  Headaches all around.

I think I can live with a separate hierarchy. Arguing with past-Eric a bit:

1. All the extra modules and what not are not that big a deal (and I could probably let myself go wrt imports, etc).
2. Who cares if there's an extra geni-test binary, which only gets enable with -ftest anyway?
3. Self tests, shmelf tests.  Seriously, who is going to run that geni --test function anyway?
4. If I forget to cabal configure -ftest, I can always cabal configure again and build
5. If I'm really desperate to test some internal function, I could always export an alias like testingFoo for every foo I want to test, applying a sort of Pythonesque we're-all-grownups-here principle. 
6. Also maybe forcing yourself to test only the exported functions, enforces a kind of general black-box thinking which is healthy if you're writing a library.

So, with apologies to Ivan for not understanding his rants 2 years ago; and also anyone that may have listened to 2009-Eric for any messes I got you and your users in, I'm retracting that particular bit of advice and separating my test hierarchies like a good boy.   Let's see if 2013-Eric decides to post some kind of retraction retraction.

I'll begin this post with a quote from 2009-Eric:

This may go down as the kind of bad advice that "seemed like a good idea at the the time".

The advice in question was to "bake unit tests in".  The basic idea was that whatever module you write should have its own testSuite function exposing unit tests for that particular module.  The advantages were simplicity (no parallel test hierarchy), the ability to ship a binary with self-tests, and the ability to non-exported functions, helper code with a granularity that lends itself more to testing (easier to think of tests for them).

I was unconvinced by the counterargument that it was not a good idea to mix testing and business logic.  To be clear, I did agree with the spirit of the advice -- I'm not about go around questioning the kind of wisdom a community gains by watching rockets blow up -- but I felt that I was not advocating any such mixing.  All I wanted was to put my testing code in the same file as the business code, cordoned off in a testing section at the end of the file if you want without any sort of if-testing-mode-do-X logic.  So I thought that the counterargument was right, but that it didn't apply to this particular context.  (I'd be interested to see when/if I change my mind on this, maybe it leads to temptation to mix logic, which is bad.)

In any case, I don't need to change my mind on that particular point. Being the kind of person that only learns the hard way, I've found myself forced to divorce my test code from the business code after all. It's mainly a practical problem of dependencies (this was pointed out by Echo Nolan and Ivan Miljenovic). Forcing users to install QuickCheck and test-framework, when they probably don't care about testing, when they just see your module as yet another dependency on the the road to some other more pressing goal, is really a bit anti-social.

The problem isn't installing the package per se (it all happens automatically with cabal install), but dealing with package version dependencies.  So GenI depends on test-framework 2.x and QuickCheck 1.2.  What if I go away for a few years, stop hacking on GenI and in the meantime the rest of the world moves on to using QuickCheck 2.x and test-framework 3?  What happens when they try to install GenI and cabal install needs to rebuild the random package, which then breaks QuickCheck-2.4 because it depends on random too.  Headaches all around.

I think I can live with a separate hierarchy. Arguing with past-Eric a bit:

1. All the extra modules and what not are not that big a deal (and I could probably let myself go wrt imports, etc).
2. Who cares if there's an extra geni-test binary, which only gets enable with -ftest anyway?
3. Self tests, shmelf tests.  Seriously, who is going to run that geni --test function anyway?
4. If I forget to cabal configure -ftest, I can always cabal configure again and build
5. If I'm really desperate to test some internal function, I could always export an alias like testingFoo for every foo I want to test, applying a sort of Pythonesque we're-all-grownups-here principle. 
6. Also maybe forcing yourself to test only the exported functions, enforces a kind of general black-box thinking which is healthy if you're writing a library.

So, with apologies to Ivan for not understanding his rants 2 years ago; and also anyone that may have listened to 2009-Eric for any messes I got you and your users in, I'm retracting that particular bit of advice and separating my test hierarchies like a good boy.   Let's see if 2013-Eric decides to post some kind of retraction retraction.

practical QuickCheck revisited - separate testing hierarchy