Been while since I’ve been blogging! I’m just off a nice months long project so I’m easing my way back onto the open source scene. Since I take these times of renewed effort to work on crazy stuff (gets the inspiration going again), I revisited an old piece of code I first mentioned way back a year or so ago that I had dubbed MutateMe.

You can find it (and the exhaustive README) over at http://github.com/padraic/mutateme. You can install it using PEAR (I’ll get it to a PEAR channel soon) and it sits at a notional 0.5 development version.

MutateMe is a PHP Mutation Testing framework. Mostly. There is still work to be done, but the core of the framework has been completed with that fancy new namespace magic PHP 5.3 introduced. It doesn’t even crash anymore…that I’ve seen in…oh…at least a couple of hours ;) . Special thanks to Arvid Bergelmir who played a real good guinea pig over the past week in testing this out :P .

Onto some explanations…

Mutation Testing is a little known (in PHP) practice of deliberately adding errors into source code. The idea is that, your sparkling perfect 100% code coverage jewel of a test suite should immediately pop out a failed test when the error is introduced. If it doesn’t, you get to ask why the error did not produce a test failure. Maybe it’s a missing test, or maybe the error was simply so minor or inconsequential it had no test-worthy effect. Either way, it’s food for thought.

The range of errors that can be introduced range from simple swaps (like sneakily swapping TRUE and FALSE somewhere) to more complex blunders (like replacing a regular expression with one matching only the Lead Developer’s cat’s name). At present MutateMe just implements a dozen or so simple swaps (the rest will follow now that the core framework is stable).

So, in a sense, Mutation Testing is quality control for unit tests. It helps indicate that you have good tests (or not) that will prevent a future apocalypse. MutateMe currently supports PHPUnit out of the box (since doing otherwise would be project suicide).

Back to MutateMe. It’s a relatively simple command line utility (much like PHPUnit). It operates by examining your source code, figuring out how to screw it all up, and then applies each screw up (mutations) to the source code in turn while re-running your test suite. All mutations are applied in isolation, of course – we run tests in a new PHP process each time to maintain some isolation. Also, the changes are not applied to the filesystem since we use the marvellous runkit extension to alter classes in memory.

Please be sure to read the README on how to install MutateMe and runkit, and then how to commence some amazing source code mangling ;) .

As far as I’m aware, MutateMe is currently the only (I always hope I’m wrong) Mutation Testing framework in PHP. If you have some small source code pile to experiment with it, I’d love to hear about your experiences, comments, and suggestions for the future. I say “small” with meaning – remember that Mutation Testing runs the assigned test suite (or any subset thereof) for every single mutation generated from the source code (likewise, or any subset thereof). Mutation Testing is something you do occassionally rather than every other minute ;) .

Final notes? Given the runkit requirements, I’m afraid Windows users may feel neglected. We’re using a patched version of the official runkit (far better feature set and fewer bugs) but there’s no precompiled DLL for it. If anyone wants to link me to a working PHP 5.3 (with static method support) Windows DLL, I’ll happily note it in the README. I’ll work on some Windows-dependent workarounds to avoid static method mutations otherwise.

Happy mutating! Don’t forget the feedback. Github has an issue tracker available for any bugs and/or feature requests.

Related posts:

  1. Mutation Testing: MutateMe 0.2.0alpha Released
  2. PHP Mutation Testing With MutateMe
  3. Mutation Testing Brain Dump
  4. PHPMutagen: Mutation Testing for PHP5
  5. Unit Testing: Multiple Assertions And Lazy/Shallow Testing Are Evil