Dynamic Testing Tools

Question

I am looking for some sort of dynamic testing tool. Let me explain what I mean. I have a modular application and part of my business strategy is to license each module independently. I would like to test things such as the fact that one module's tests all still pass even when the other module is not present on the file system.

Another thing I would like to be able to verify during my test suite, is that un-related settings should not change behavior they are not supposed to be coupled to.

To write an ad-hoc script for each test case and inject the defect would be trivial, but sounds like a violation of DRY. Is there any tools designed for this kind of defect injection, specifically in PHP?

So in an ideal world I would like to be able to write a set of "environment modifying" scripts, each of which will alter the environment in some way. For each "environment modification" phpunit should run all my tests. I should be able to view in some kind of report which tests failed under which "environment extremes"

So with N "environment modifications" and M tests, The sum of passes & failures should equal N times M, similar to how GUI test tools such as selenium run the suite on each different environment (browser)

Answer 1

This might work using phpUnit and some magic inside the fixtures. But I'd look into some build tool (like Ant or Phing) which creates a checkout from your source repository, set's up the required modules and then runs the phpUnit tests.

Phing or Ant can be called from an continous integration system like phpUnderControl or Hudson on a regular basis so you get regular automated feedback.