PHP, Zend Framework and Other Crazy Stuff
Archive for February, 2009
Mutation Testing: MutateMe 0.2.0alpha Released
Feb 25th
I love git! After the initial alpha, the git repository was forked on github by Sebastian Bergmann and Benjamin Eberlei. The new release incorporates all changes pulled from their forks, and huge thanks to both for their feedback and fixes. Also, see the revised README (copied in this blog post) with much improved detailed installation and usage instructions included.
You can download the PEAR package at http://dev.phpspec.org/MutateMe-0.2.0alpha.tgz - the source code is maintained in git at http://www.github.com/padraic/mutateme
The other major changes included funtionality to ensure mutated methods retain their class visibility, i.e. whether public, protected or private. In addition, I also implemented optional static method support which is missing from the current Runkit extension CVS. The static support can be implemented in one of two ways.
The preferred option is to compile a new runkit extension using a slightly patched version of the extension from CVS, which I have forked to http://github.com/padraic/runkit. The only change from PECL CVS is to add a patch written by David Sklar a while ago to add static method support to runkit. As suggested on Twitter, I could simply request to maintain runkit but my C skills are rarely exercised so I wouldn’t be comfortable doing that. The small fork will do until a new runkit release appears on PECL.
The alternative option, currently applicable mainly to Windows users, is to disable E_STRICT error reporting when using MutateMe. I haven’t had time to compile a Windows DLL for the patched version of runkit but I’ll get around to it soon. E_STRICT kicks in, since mutated static methods only retain the public/protected/private visibility and don’t retain their static nature. Using a method statically, if not flagged as static, results in a deprecation error. This error will not impact anyone using the patched version of runkit.
The final major change was adding support for SimpleTest! It wouldn’t be much of a Mutation Testing framework if it only supported PHPUnit, now would it?
Here’s the updated README
MutateMe is a Mutation Testing framework for PHP5, currently released as an alpha version for interested developers to assist in offering feedback. It is not yet fully functional, and stability is a relative term. Requirements: PHP: 5.2.4, probably less but I haven't checked PEAR: Text_Diff 1.1.0 Extensions: ext/runkit, and optionally ext/xdiff Note: Do not install runkit from PECL since the current release does not support PHP 5.2. Follow the installation instructions below to grab a patched version of runkit which supports both PHP 5.2 and static methods. Windows users should download the DLL from the given link - a patched version of the DLL will be made available soon. If you have installed from PECL delete the module and follow the instructions below to replace it with a slightly improved patched version. Installation (with PEAR): 1. Download the latest PEAR packaged release and run the following command: pear install MutateMe-0.2.0alpha.tgz 2. Follow the manual instructions below from Step 3 to install the PHP runkit extension. Installation (Manual Installation without PEAR): 1. Copy the contents of /library to a location on the PHP include_path, e.g. /usr/share/php. 2. Copy /bin/mutateme and /bin/mutateme.bat to a location on PATH so they are accessible from the command line. Edit the contents of each to replace @[email protected], @[email protected] with the path to the php binary (e.g. /usr/bin/php) and the path to the directory you're putting these scripts respectively. 3. Install the runkit extension but NOT from PECL. Windows users can grab the DLL from http://dev.phpspec.org/php_runkit.dll but should disable E_STRICT error reporting (I will compile a patched DLL for use with MutateMe soon). Linux users should compile runkit from source. I've put up a patched version supporting static methods (via a David Sklar patch) and PHP 5.2 (from CVS HEAD in PECL) at http://github.com/padraic/runkit Use the following commands to compile from source (install git first!): git clone git://github.com/padraic/runkit.git runkit cd runkit phpize ./configure make make install Finally, enable the extension in the php.ini file which is loaded for the cli environment using: extension=runkit.so Check the output of "php -i" or phpinfo() to ensure the extension is loaded. Usage: MutateMe has just a few necessary command line options: --adapter Adapter to use: phpunit or simpletest. PHPUnit is the default. --basedir Base directory containing source code and tests --srcdir Directory containing source code to mutate --testdir Directory containing the tests to run --testfile Name of the test file to execute, e.g. AllTests.php or all_tests.php --test (PHPUnit only) Name of the test class to run, e.g. MyLibrary_AllTests If options are omitted, MutateMe will attempt to autodiscover the correct values using the following conventions: 1. The base directory is assumed to be the current working directory 2. The source directory is assumed to be a subdirectory of the base directory called either "src", "lib" or "library" 3. The test directory is assumed to be a subdirectory of the base directory called either "tests" or "specs" 4. The test file is assumed to be either AllTests.php or all_tests.php depending on the adapter used 5. (PHPUnit only) The test class is assumed to be AllTests If you cannot use these conventions completely, please use the command line options appropriately. Example: $ mutateme --adapter phpunit --basedir ./ --srcdir ./src --testdir ./tests \ --testfile AllTests.php --test MyLib_AllTests In the example above, we could have omitted the --basedir, --srcdir, --testdir and --testfile options since they meet the requirements of the autodetectable convention as described above. Output: A successful Mutation Test would output something like: MutateMe Alpha: Mutation Testing for PHP All initial checks successful! The mutagenic slime has been activated. PHPUnit 3.3.14 by Sebastian Bergmann. . Time: 0 seconds OK (1 test, 1 assertion) . 1 Mutant born out of the mutagenic slime! 1 Mutant exterminated! No Mutants survived! Muahahahaha! A failed Mutation Test would indicate a test failed to detect an introduced error in the source code. This might be a completely spurious failed mutation, since not all introduced errors actually cause problems, but often it will highlight a new condition that you should a new test for so a future real error will be detected and not completely missed by the test suite: All initial checks successful! The mutagenic slime has been activated. PHPUnit 3.3.14 by Sebastian Bergmann. . Time: 0 seconds OK (1 test, 1 assertion) . 1 Mutant born out of the mutagenic slime! 0 Mutants exterminated! 1 Mutant escaped; the integrity of your suite may be compromised by the following Mutants: 1) Index: ./Math.php =================================================================== @@ -1 +1 @@ -return$op1+$op2; +return$op1-$op2; Happy Hunting! Remember that some Mutants may just be Ghosts (or if you want to be boring, false positives).
PHP Mutation Testing With MutateMe
Feb 23rd
Writing a while ago about the problem of shallow testing (when tests appear to verify code is working on the surface but actually fail when you examine them more closely) dragged up some enthusiasm to revive my attempt to write a more publicly accessible library for Mutation Testing (other than the mish mash I concocted for my own use over a year ago).
Mutation Testing for PHP is currently almost non-existent though there are a few pieces of work buried in subversion repositories here and there. None of them however are capable of supporting ALL testing tools leaving many tools out in the cold. I never did like the constant parade of tightly integrated solutions in PHP, so I’ve been working on a standalone version that can operate with any testing tool once a suitable adapter is available for it.
I’ve dubbed it “MutateMe” and you can find the git repository at http://github.com/padraic/mutateme.
Update #1: The git repo now contains a preliminary SimpleTest adapter supporting both the TextReporter and HtmlReporter - simply pass “-adapter simpletest” on the command line and ensure -testfile contains the relevant group file, e.g. all_tests.php. If you want to do a manual install from git, the main thing is editing the files in /bin - they use similar conventions to PHPUnit so you can use the phpunit and phpunit.bat files for your system as rough examples of what PHP paths are needed to be edited.
Update #2: Added installation instructions for the required runkit extension. Added a Windows binary download, and some simple Linux compilation instructions. Note - do not install from PECL since the current release does not support PHP 5.2.
Update #3: I’ve discovered the runkit CVS does not yet support static methods. There is a patch by David Sklar so I may independently release a git fork and windows DLL with the necessary changes until the runkit CVS is updated in the future. In the meantime, be aware static methods will likely result in errors - oh, the fun of alpha releases . I’ll update git later with some changes contributed by Benjamin Eberlei in his MutateMe fork at http://github.com/beberlei/mutateme/tree.
Update #4: Those on Linux can now compile a slightly patched version of runkit which supports static methods using http://www.github.com/padraic/runkit. This is strictly a bridging fork until a similar patch finds its way into a new runkit release to PECL (though last release was in 2007, so…depends on Sara ).
Update #5: Git repository now contains full static method support. I’ll roll out a new alpha release to reflect the changes during tomorrow. Thanks to everyone who gave feedback and even fixes since the initial 0.1.0 alpha!
What is Mutation Testing?
Mutation Testing is basically testing…for tests. It ensures that your tests are truly capable of detecting errors and problems with the source code. It does this by mutating the source code itself (using ext/runkit) in such a way that an error is created in the code. If your tests detect the error, all is well with the world. If your tests do not detect the error…well, you better add a new test that does .
These source code mutations are applied consistently against specific areas. For example, MutateMe can replace a plus sign with a minus sign, or substitute a variable value for the original (e.g. $state=0 becomes $state=123), or pretty much any other mutation or change you can imagine. These mutations are applied one by one, with the tests run after every change. At the end of the process you receive a report on which mutations did not cause a test to fail (in Mutation Testing, passing tests are a bad thing!) including a pretty diff of the mutation itself so you can replicate the errors that no test detected.
Obviously, this is a resource intensive operation since you run the test suite for every possible mutation. Unlike Unit Testing, Mutation Testing runs your tests for every mutation - and there can be a lot of mutations! It’s a form of testing best reserved for occasional use, and preferably on another machine so you can let it run in the background while you’re doing something else.
The other facet of Mutation Testing is that it is not definitive all the time. It may raise false positives because changing some code might not actually break anything. It may also raise very obvious errors which aren’t worth the time testing (otherwise we’d all have 100% code coverage ). Nevertheless, properly utilised it can point to areas where testing needs improvement.
Mutation Testing vs Code Coverage
In PHP, there is a persistent lack of focus on test quality. Code Coverage has become one of the primary and most celebrated measures of ensuring test quality even though it’s real purpose is to ensure tests execute all source code - i.e. it ensures a test is written which executes certain lines of code, but doesn’t inform you whether the test is poorly written and incapable of detecting the problems it’s suppossed to.
Mutation Testing is a complementary practice which does detect the existence of shallow or low quality tests since it deliberately introduces errors into the source code to see if tests will catch it. It’s a very simple notion .
Without Mutation Testing, the quality of a suite of tests is very difficult to assess without in-depth reading of the tests, a complete and godlike knowledge of the underlying source code, and the ability to see flaws in how the tests operate. In other words - you need a highly skilled human to do the task. Any human involvement is bound to be unreliable - and expensive. Just as with unit testing, automated tests are cheaper, faster and far more methodical when properly designed.
Introducing MutateMe
Rising from the ashes of my initial pass, then called PHPMutagen, MutateMe is the rewritten fully-tested version of my Mutation Testing tool for PHP. It is designed to hook into any testing framework (like PHPUnit or PHPT), mutate the underlying source code within the current PHP process (source code is mutated in memory using ext/runkit), and run the relevant tests against the mutated form to check whether they detect the deliberate error.
To install, download the current development release from http://dev.phpspec.org/MutateMe-0.1.0alpha.tgz.
You should install MutateMe using the PEAR installer, so from the location of the download run:
pear install MutateMe-0.1.0alpha.tgz
This will also download and install Text_Diff 1.1.0, and suggests you might consider installing ext/diff from PECL (optional). MutateMe does however require the ext/runkit extension which is the most problematic part - you must NOT install this from PECL! The current PECL release does not support PHP 5.2.
If you are on Windows, you can try a precompiled version that works with PHP 5.2, grab a copy from http://dev.phpspec.org/php_runkit.dll, then just copy it to your extensions directory for PHP5 and enable it in php.ini.
Under Linux, you need to compile ext/runkit (takes just a few minutes) from CVS:
1. Using your package manager, install autoconf, automake, libtool, bison, flex and re2c to compile PHP modules.
2. Log into the PHP CVS server anonymously using:
cvs -d :pserver:[email protected]:/repository login
The password is “phpfi”.
3. Grab the latest runkit HEAD using:
cvs -d:pserver:[email protected]:/repository co pecl/runkit
4. Compile the extension and copy it into your default PHP extensions directory using (assuming you are not already in the runkit source directory):
cd runkit phpize ./configure make make install
5. Edit whichever php.ini config file that governs the PHP CLI to include an enabling line like:
extension=runkit.so
6. Check the output of “php -i” or phpinfo() to ensure the extension is loaded.
MutateMe is now ready for action - well, the alpha version with it’s initial payload of 5 possible mutations and a dodgy PHPUnit adapter is . Unfortunately you’ll have to wait a bit longer for the SimpleTest and PHPT adapters.
The command line tool is accessed by calling “mutateme”. It accepts a few command line options:
-basedir: The base directory containing the source code and tests (see conventions later)
-srcdir: The directory where the source code to mutate is located
-testdir: The directory where unit tests are maintained
PHPUnit options (subject to improvement):
-testfile: The name of the test file to run (e.g. AllTests.php)
-test: The test class to run contained in the test file (e.g. My_AllTests - the actual class name in that file can differ from the file name)
-adapter: Select a specific adapter (currently phpunit or simpletest)
If you want to minimise options, MutateMe assumes the current directory is the base directory unless otherwise specified. On that assumption, it will attempt (unless options are passed) that source code is held in one of /src, /lib or /library subdirectories of the base directory. It also assumes tests are stored in either the /tests or /specs subdirectory of the base directory, where there exists an AllTests.php test file containing an AllTests class (assuming you’re using the PHPUnit adapter which is the default).
Unless you meet these conventions exactly - use the specific options! For example:
cd ~/projects/mathlib mutateme --srcdir ./libs --testdir ./tests --testfile AllTests.php --test Mathlib_AllTests
You should ensure, that before using MutateMe, your test suite is recording no Errors or Failures (Skips and Incompletes are fine though), since this a precondition before Mutation Testing can be performed. You’ll be told by MutateMe if it doesn’t anyway.
You can then look forward to the results…
MutateMe Alpha: Mutation Testing for PHP All initial checks successful! The mutagenic slime has been activated. PHPUnit 3.3.14 by Sebastian Bergmann. . Time: 0 seconds OK (1 test, 1 assertion) . 1 Mutant born out of the mutagenic slime! 1 Mutant exterminated! No Mutants survived! Muahahahaha!
If you see anything different, it’s time to see if the issue warrants an additional test to catch that defect if it appears in the future:
MutateMe Alpha: Mutation Testing for PHP All initial checks successful! The mutagenic slime has been activated. PHPUnit 3.3.14 by Sebastian Bergmann. . Time: 0 seconds OK (1 test, 1 assertion) . 1 Mutant born out of the mutagenic slime! 0 Mutants exterminated! 1 Mutant escaped; the integrity of your suite may be compromised by the following Mutants: 1) Index: ./Math.php =================================================================== @@ -1 +1 @@ -return$op1+$op2; +return$op1-$op2; PHPUnit 3.3.14 by Sebastian Bergmann. . Time: 0 seconds OK (1 test, 1 assertion) Happy Hunting! Remember that some Mutants may just be Ghosts (or if you want to be boring, false positives).
The mutations enabled are very limited for now. There are four primary mutations:
1. Replacing “+” with “-”
2. Replacing “-” with “+”
3. Replacing TRUE with FALSE
4. Replacing FALSE with TRUE
Before an actual beta release, I’ll be adding adapters for SimpleTest and PHPT, and obviously adding a lot more mutations to the mix. The current release is mainly there for those interested to take a look at.