PHP, Zend Framework and Other Crazy Stuff
Posts tagged maugrim
MutateMe: A PHP Mutation Testing Framework
May 12th
Been while since I’ve been blogging! I’m just off a nice months long project so I’m easing my way back onto the open source scene. Since I take these times of renewed effort to work on crazy stuff (gets the inspiration going again), I revisited an old piece of code I first mentioned way back a year or so ago that I had dubbed MutateMe.
You can find it (and the exhaustive README) over at http://github.com/padraic/mutateme. You can install it using PEAR (I’ll get it to a PEAR channel soon) and it sits at a notional 0.5 development version.
MutateMe is a PHP Mutation Testing framework. Mostly. There is still work to be done, but the core of the framework has been completed with that fancy new namespace magic PHP 5.3 introduced. It doesn’t even crash anymore…that I’ve seen in…oh…at least a couple of hours 
. Special thanks to Arvid Bergelmir who played a real good guinea pig over the past week in testing this out 
.
Onto some explanations…
Mutation Testing is a little known (in PHP) practice of deliberately adding errors into source code. The idea is that, your sparkling perfect 100% code coverage jewel of a test suite should immediately pop out a failed test when the error is introduced. If it doesn’t, you get to ask why the error did not produce a test failure. Maybe it’s a missing test, or maybe the error was simply so minor or inconsequential it had no test-worthy effect. Either way, it’s food for thought.
The range of errors that can be introduced range from simple swaps (like sneakily swapping TRUE and FALSE somewhere) to more complex blunders (like replacing a regular expression with one matching only the Lead Developer’s cat’s name). At present MutateMe just implements a dozen or so simple swaps (the rest will follow now that the core framework is stable).
So, in a sense, Mutation Testing is quality control for unit tests. It helps indicate that you have good tests (or not) that will prevent a future apocalypse. MutateMe currently supports PHPUnit out of the box (since doing otherwise would be project suicide).
Back to MutateMe. It’s a relatively simple command line utility (much like PHPUnit). It operates by examining your source code, figuring out how to screw it all up, and then applies each screw up (mutations) to the source code in turn while re-running your test suite. All mutations are applied in isolation, of course - we run tests in a new PHP process each time to maintain some isolation. Also, the changes are not applied to the filesystem since we use the marvellous runkit extension to alter classes in memory.
Please be sure to read the README on how to install MutateMe and runkit, and then how to commence some amazing source code mangling .
As far as I’m aware, MutateMe is currently the only (I always hope I’m wrong) Mutation Testing framework in PHP. If you have some small source code pile to experiment with it, I’d love to hear about your experiences, comments, and suggestions for the future. I say “small” with meaning - remember that Mutation Testing runs the assigned test suite (or any subset thereof) for every single mutation generated from the source code (likewise, or any subset thereof). Mutation Testing is something you do occassionally rather than every other minute .
Final notes? Given the runkit requirements, I’m afraid Windows users may feel neglected. We’re using a patched version of the official runkit (far better feature set and fewer bugs) but there’s no precompiled DLL for it. If anyone wants to link me to a working PHP 5.3 (with static method support) Windows DLL, I’ll happily note it in the README. I’ll work on some Windows-dependent workarounds to avoid static method mutations otherwise.
Happy mutating! Don’t forget the feedback. Github has an issue tracker available for any bugs and/or feature requests.
Volunteer Technical Reviewers, Proofreaders and Contributors Wanted
Jun 19th
As readers may have surmised, when I write chapters/articles I tend to leave behind in my glorious epic works of perfection (well, that’s what Mr. Ego likes to call them ) numerous grammatical, spelling and technical errors. It’s a flaw that persists because I proofread and QA my own writing and this is in strict opposition to the “many eyes” approach to QA. Self-reviewing only goes so far - once you are used to the text you wrote it becomes extremely difficult to see problems because you have the article/chapter and your own background concepts mingling. Errors get missed, technical foolishness escapes attention.
To rectify this problem I would like to ask for volunteer reviewers and proofreaders who can examine the final drafts of Chapters for the ZF Book “Surviving The Deep End” before they go live. This would be facilitated by offering unrestricted access to the private repository I version the book at (that way you can correct my failings and make me look godlike). When I say volunteer, I mean peanut paid monkeys - I don’t have the funds for decent fees but I will offer a prominant mention in the book’s credits and a once off donation of $50 to cover the cost of the electricity/generator cranking monkey used by your PC while reviewing. This should be a fairly light weight role - it’s not like I finish chapters every other day. Being a member of the PHP Blogosphere is obviously a serious advantage since I know the work and professional attitude of most people on my feed list.
In case you are unaware of where my money spinning tactic exists with the book, it lies in a dual donation/advertising revenue stream which is expected to be as predictable as the weather in Ireland. If the income exceeds my own costs I’ll put together some additional funds for reviewers/proofreaders. I expect to want no more than two technical reviewers/proofreaders which should be sufficient to keep me honest and free of major misteps.
In the third category, I am also inviting people who are interested in contributing specific topics chapters for the book. In my mind, these are highly specific chapters which address some nugget of utility that isn’t obviously provided by the Zend Framework itself but is useful to have in your toolbox. For example, my longstanding example had been using Zend_Feed to build a generic feed reader which is capable of interpreting feed formats without constant hand wrangling. Or perhaps there is a specific area of the framework I’m unlikely to use in a protracted project based book which you really want to write about. Or maybe it’s a practice related topic, like my Performance Optimisation appendix (no ZF strictly but necessary information for anyone building a generic application using the ZF). Yeah, like that’s NOT vague 
. If you have ideas let me know by email (padraic dot brady at yahoo dot com) or twitter (padraicb) message and we can discuss a chapter fee.
Now, back to those two chapters I intend having online by tomorrow…
