Maugrim The Reaper's Blog

HTML Sanitisation: The Devil's In The Details (And The Vulnerabilities)

nospam@example.com (Pádraic Brady) — Mon, 09 Aug 2010 22:00:31 +0000

HTML Sanitisation (defined below) has been with us for a long time, ever since the first genius who came up with the idea of allowing potentially untrustworthy third party HTML to be dynamically patched into their own markup. The years have not taken this kindly, and third-party HTML inclusion has remained one of the most complex and underappreciated vectors for security vulnerabilities.

In this article, I take a look at some of the solutions PHP developers rely upon to perform HTML Sanitisation. Mostly because few others have done it or written about such solutions in any great detail (at least publicly). HTML Sanitisation has a very low profile in PHP. It's rarely mentioned, usually not understood all that well, and examining some of the solutions in this area with more deliberate attention is worth doing. Also, it's valuable research since I am writing my own HTML Sanitisation library (bias alert!) for a future Zend Framework 2.0 proposal. Knowing what the competition is up to does no harm! Finally, I was simply curious. Nobody seems too pushed to look closely at all these HTML Sanitisation solutions despite the fact that there are other developers (I think) who wouldn't touch most of them with a barge pole.

One somewhat remarkable example, just to illustrate why I figured this article was worth the time, is HTMLPurifier's Comparison analysis where HTMLPurifier is compared against a number of other HTML Sanitisers. The comparison is remarkable because it seems inclined to err on the side of giving HTMLPurifier's competitors the benefit of the doubt. Unfortunately, this means the analysis is often flawed and its conclusions suspect. Also, it assists in legitimising other solutions in the minds of readers by making assumptions of safety. Not that this reflects on HTMLPurifier's functionality, incidentally, which I have always maintained is the only HTML Sanitiser worth looking at.

Back on track…

What is HTML Sanitisation?

HTML is an amazingly dangerous thing. It can contain Javascript, CSS, or malformed markup, or even gigantic images that laugh at your dual 32" monitor setup. Each of these, in their own way, can damage the experience of an end user, whether it be by Cross-Site Scripting (XSS), Phishing or simply mangling the page until it's unusable and/or defaced with scriptkiddie jibes.

There are two ways of dealing with these threats to the HTML output of an application: escape output so that the only HTML rendered by the browser is the application's (anything else being neutered by HTML entities), or by sanitising output so that any additional HTML it contains, that is renderable by a browser, is stripped of any potentially damaging markup. This article concerns the second option.

HTML Sanitisation may therefore be defined as any means of filtering HTML to ensure that a) Cross-Site Scripting (XSS) vulnerabilities are removed, b) Phishing vulnerabilities are removed, c) the HTML is well formed and adheres to an acceptable HTML standard, and d) the HTML contains no obvious means of breaking expected web page rendering.

I won't claim this is a perfect definition but it covers most of the salient points you'll likely encounter.

So there are, broadly speaking, four primary objectives of HTML Sanitisation, any one of which is capable of preventing damage to end users or web application functionality (including javascript powered client side functionality). Each is, in its own way, quite a difficult proposition requiring suitable tools and specialised knowledge. However, with some objectives we can measure our success somewhat reliably. The question of this article being: how well do HTML Sanitisers in PHP measure up to these objectives?

The Candidates

Since this is intended as a brief examination (just a few million words long!), I decided to select four candidate HTML Sanitisers meeting certain conditions. These conditions included:

1. Having a release at some point in the past two years;
2. Describing itself as a HTML sanitiser/filter to prevent Bad Things;
3. Having a design clearly in line with an intent to filter XSS/Phishing; and
4. Having no publicly acknowledged long standing security vulnerabilities.

The great part about applying these conditions is that I pretty much eliminated stacks of HTML Sanitisers (as some might claim them as being). Outside of those, it also eliminates anything users might misconstrue as a HTML Sanitiser (for example PHP's strip_tags() function or Zend Framework's Zend_Filter_StripTags class). What we are left with is pretty thin on the ground, but fits what I'd expect a reasonably educated PHP developer to swing with. From what remained, I selected four candidates (or maybe these were the only four left - I'll never tell):

1. PEAR's HTML_Safe
2. htmLawed
3. Wordpress' Kses
4. HTMLPurifier

With four candidates in tow, I proceeded to examine each against the four objectives of HTML Sanitisation I noted earlier. Just to emphasise to readers, this examination was not so in-depth as to identify every possible flaw or issue with each candidate. My intent was to attempt to locate one security vulnerability and assess each candidate in general terms for the other non-security related HTML Sanitisation objectives.

Before I go any further, let me clarify that all security vulnerabilities discovered were notifed almost immediately to the parties responsible for each candidate solution. All such parties confirmed receipt of these reports within one week, and all were given a period (approx. six or more weeks to today) in which to apply fixes, make new releases, update documentation, perform additional security reviews, etc. I've sat on this article for a long time. Regardless of the effectiveness of any such actions (or lack of action as the case may be), this article discloses all perceived security vulnerabilities discovered during my examination whether or not all parties agree with my opinion. In some cases a disclosure may clearly indicate a fundamental flaw in the underlying design of the candidate. In these cases, it was emphasised to the responsible parties that reported vulnerabilities were limited to the scope of my examination and that I believed it was likely that additional and possibly related vulnerabilities remained unreported but easily discoverable as a result of public disclosure. This concludes my rendition of "Cover Your Ass" .

Note that all discussions below relate to the whichever version of each candidate solution was initially examined (before any fixed releases). I have noted resolutions as necessary for each.

PEAR's HTML_Safe

To get us started, PEAR's HTML_Safe is one of the older candidates despite its examined release being in April 2010 (the previous release being a beta in 2005). HTML_Safe's description states that "This parser strips down all potentially dangerous content within HTML". HTML_Safe operates on the basis of parsing HTML with regular expressions and applying filtering logic which is dependent on predefined blacklists of potentially harmful elements, attributes and CSS properties.

Unfortunately, HTML_Safe's blacklists prove its undoing. The problem with blacklists is that they require constant attention and updates for new problems. A cursory examination of the CSS property blacklist showed that it omitted many browser specific CSS properties such as -ms-behavior. The -ms-behavior CSS property (specific to Internet Explorer 8) may contain as a value a URI reference to a locally hosted HTC file (which contains executable Javascript). While such a file would need to exist on the local domain, this is obviously a security vulnerability in that it allows the execution of any arbitrary Javascript an attacker can store or reuse on the local domain thus opening up XSS possibilities.

HTML_Safe also has another vulnerability shared by practically all HTML Sanitisation solutions based on the use of regular expressions. Regular expression parsing typically assumes that all HTML special characters are encoded in ASCII (an encoding subset which is common across other encodings such as ISO-8859-1 and UTF-8). However, UTF-7 encodes the greater than (>) and less than (<) characters differently. This means that typical regular expression parsing does not detect these characters when encoded in UTF-7. If you can't detect them, you can't sanitise them! This sanitisation bypass requires a secondary vulnerability where an attacker either forces a webpage containing unsanitised UTF-7 encoded markup to be rendered with a charset of UTF-7 (IE has vulnerabilities here, as do some versions of Firefox) or where the target application actually allows a user to select a nonsensical custom character set (as happened with Google and Yahoo when struck with this same exploit, and several CMS applications as recently as last Spring).

Finally, HTML_Safe's blacklisting also misses out on CSS properties not directly tied to XSS (e.g. position), but which may be used to perform Phishing attacks by using CSS styling to alter or overlay HTML elements. This could, for example, allow an attacker to re-style their injected HTML to replace (as in positioning above) specific page elements (if not the entire page). This can lead to Clickjacking among other forms of Phishing.

In terms of HTML well formedness, HTML_Safe does not necessarily emit standards compliant or well formed HTML, nor does it check some other common page breaking tactics such as overlarge images as a precaution.

HTML_Safe may well be the least secure of our four candidates. Its use of blacklisting, its relative age, and a lack of peer review have left it woefully outdated for the task of HTML Sanitisation. In short, it should be avoided at all costs and my main request to PEAR at the time of reporting the above vulnerabilities was to seriously consider removing it from PEAR. Personally, I find it almost tragic that a library of such limited capability may benefit from PEAR's reputation and lead to users trusting it over far more secure alternatives.

Of the issues noted above, the UTF-7 vulnerability has been resolved in a new release. The CSS blacklist has not yet been revised though I remain confident that the security advisory will go out any day now. PEAR does do security advisories, right? My recommendation, as originally suggested, remains that HTML_Safe should be removed, or overhauled, given it is not up to the task of HTML Sanitisation in its current condition. Blacklists are simply the worst approach ever to HTML sanitisation.

htmLawed

Our second candidate is, at least based on Google's omnipotence, one of the more popular open source standalone HTML Sanitisers. htmLawed has garnered a reputation as something of a rebellious spirit built for speed, a stark contrast to the slow resource intensive operation of HTMLPurifier with which is often compared. htmLawed's description states that it is "PHP code to purify & filter HTML" and that it can "make HTML markup in text secure and standard-compliant".

The problem with htmLawed is that its operation is not much more than a very short stone's throw away from HTML_Safe. On the face of it, htmLawed is significantly more complex than HTML_Safe, being loosely based on Kses (an older HTML Sanitisation script). Its documentation is also huge, finely detailed, and packed full of options. It's source code is also complex and heavily obscured (presumably to discourage anyone from trying to examine it too closely). But...complexity and options a HTML Sanitiser does not make.

Similar to HTML_Safe, htmLawed carries no functionality to thwart character encoding based attacks. The UTF-7 exploit as described for PEAR's HTML_Safe works perfectly well.

In addition, htmLawed has a number of oddities which are not vulnerabilities but may potentially become such in the future. These are related to attempts by htmLawed to fix insecure attribute values and CSS values. I have no idea why htmLawed tries to fix them (just remove them!), but the results come very close to enabling reconstructive logic based attacks (i.e. where you can trick a fixing mechanism to reconstruct a more obvious attempt at a vulnerability into a less obvious but equally dangerous one). I can't say these are anything more than oddities however. One example, is that you can get the parser to create a CSS string containing something like "exp ress ion" (expression properties allow the execution of Javascript). The spaces in the string prevent it from being used by all current browsers, but what if a new browser release decides to ignore a little whitespace? Several other oddities also exist but are either harmless or fixed in the most recent htmLawed release.

Back firmly in current reality, htmLawed's CSS filtering is seriously flawed. For example, it does not filter out the CSS behavior property which leads to the same result as it would for HTML_Safe's allowance of -ms-behavior, the execution of local domain hosted HTC files containing Javascript.

And, again, htmLawed is open to the exact same Phishing and Clickjacking vulnerabilities as HTML_Safe, allowing all CSS position/height/width and other properties capable of re-styling web pages.

htmLawed is not quite as easy to fool as HTML_Safe, but both share a remarkably similar lack of attention to specific areas of HTML Sanitisation. The conclusion I've come to hold is that both of these libraries (and others in the wild) are reading from the same book. Which might sound ridiculous before you consider that these libraries all seem to revolve around the year 2005 or so - the same period when Kses was King of HTML Sanitisation. There's no originality when you write new libraries in various styles which all follow the exact same assumptions and knowledge level.

htmlLawed performs tag balancing and other cleanup tasks. Nevertheless, it will not necessarily output wellformed or standards compliant HTML. You should bring ext/tidy to the party as a post-processor.

My main gripe with htmLawed, besides the above, is that it does not appear these are going to be fixed any time soon. The script's author has taken the approach that htmLawed filters HTML, and only HTML. Its CSS filtering can't be trusted and is not secure, and this IS part of HTML Sanitisation (HTML contains CSS!). Nevertheless, htmLawed documentation has been updated since my reports to clarify what developers may need to do before using htmLawed (i.e. normalise input character encoding, ensure all CSS is stripped/disabled or sanitised by another process, etc.). In short, htmLawed is not even remotely in the race as a fully featured HTML Sanitiser. It is lacking too many features, or using features which are obviously incomplete, and it is pushing too much complex responsibilities on end users who we all know will never bother doing them.

The issues noted above have not, to date, been fully resolved to my satisfaction though the updated documentation means I can't call them vulnerabilities anymore. While documentation updates and minor fixes have resulted in some improvement and clarification, it appears that the fundamental flaws in CSS sanitisation and character encoding will not be implemented until some future unscheduled major revision of htmLawed (encoding has been documented as an end user concern instead). In effect, the security vulnerabilities reported resulted not in fixes, but in pushing the fixes back to the end users to deal with and denying them as a responsibility of htmLawed. In my book, that simply removes htmLawed entirely as a usable HTML sanitiser. Pushing responsibility back to users may let you off the security vulnerability hook but it does call the entire description and goals of htmLawed into question.

Users are advised to ensure that they follow the documentation to the letter (read it in detail!) in disabling all CSS styling and enabling safe mode for htmLawed (sanitisation is disabled by default for this one). Using ext/iconv to do some minimal character encoding normalisation is also highly recommended. As with PEAR, there has been no formal security advisory issued, also the release notes for the latest release make no reference to any security vulnerability existing.

Wordpress (Kses)

Candidate three is Wordpress 3.0, or more specifically the Kses script bundled with Wordpress that is used internally for HTML Sanitisation. It should be noted that while the original Kses is stuck in 2005 (and should be avoided like the plague), Wordpress have heavily updated their internal copy. This creates a viable HTML Sanitisation solution which is simple to extract for personal use (just needs a handful of extra functions borrowed from an includes file), and which may well be the defacto winner in the popularity stakes just because it's a core part of everyone's favourite blogging platform.

What could possibly go wrong? Well, surprisingly not as much as I was half expecting after my earlier candidates!

Wordpress' Kses proved far more challenging than the previous two candidates. It's obvious that the widespread use of Wordpress has enforced constant peer review and improvement. Nevertheless, where there's a will (and a quiet afternoon), there's a way.

Wordpress' Kses first of all offers no protection when used as a standalone sanitiser against character encoding attacks, thus the previously described UTF-7 XSS attack works quite well. I do NOT consider this a security vulnerability since the use of Kses within Wordpress prevents manipulation of the HTML's declared charset (though presumably this protection could be mangled by some Wordpress template author). Nevertheless, for anyone using Kses in standalone mode you should note that character encoding normalisation is necessary and should be implemented.

Fast forward three coffees, and it finally clicked that Kses did have one obvious flaw in that it uses a little blacklisting for CSS filtering. In short, it removes all CSS attribute values which contain any of the characters /, \, * and (. It's actually very clever (in that it's indiscriminately simple) since it just removes CSS using these characters altogether. Unfortunately, it never considered the use of the right-handed curly brace, }. Under all versions of Internet Explorer, in either quirks or standards mode, the right curly brace is treated as a CSS terminator (i.e. just like a semi-colon). This means that something like "position: absolute; top: 5px; left: 10px;" can be written for Internet Explorer as "position: absolute} top: 5px} left: 10px}". IE extends another unwanted helping hand... The result is that such CSS styling values are not intercepted by Kses (which relies on semi-colon terminators), and may allow for Phishing and Clickjacking attacks where Kses filtered user input is rendered to an end user from Wordpress.

After those three coffees and some head banging, I thanked my numb brain for finally giving me something and left well enough alone. Later on, I also reported to the Wordpress team that Internet Explorer in quirks mode (doesn't work in standards mode) also accepted the equals sign in place of a colon, i.e. "position: absolute;" could be written as "position=absolute;" or even "position=absolute}". This is of far less concern than the previous issue - and more of a funny aside on IE's incredible silliness.

As with previous candidates, Kses will not always output well formed HTML not will it check for other page breaking tactics. Again, this will require some custom checks and perhaps ext/tidy in some cases.

Wordpress' Kses is surprisingly (well, it should be!) adept at the HTML Sanitisation game. If it bundled encoding normalisation, and packed HTML tidying and a few other bits and pieces I'd almost use it myself. Almost. At the end of the day, however, I just don't trust regular expressions enough.

The issues noted above were fixed in Wordpress 3.0.1 released in late July 2010 (and weeks beforehand in SVN). The reported Phishing issue was mysteriously absent from the list of 54 fixed issues for 3.0.1, which isn't really all that surprising (would not have been a public issue anywhere) but it would have been nice to see a public disclosure from the vendor before I published this article.

HTMLPurifier

HTMLPurifier is the brain child (no doubt with Angelina Jolie) of Edward Z. Yang. In short it is nothing remotely like the other HTML Sanitisation candidates. It bundles a HTML parser/validator/supercomputer (rumour says it might be The Stig's brain). HTMLPurifier describes itself with "HTML Purifier is a standards-compliant HTML filter library written in PHP. HTML Purifier will not only remove all malicious code (better known as XSS) with a thoroughly audited, secure yet permissive whitelist, it will also make sure your documents are standards compliant, something only achievable with a comprehensive knowledge of W3C's specifications."

Short answer. I completely failed to find anything that got past it. I tried. Then I tried some more. Then I tried multiple caffeine injections but to no avail. It just sat there looking smug. HTMLPurifier is seemingly impervious to pretty much anything. Worse, it refused to produce any mangled attributes, or reconstructions, or anything else I normally expect of a typical HTML Sanitisation solution.

Out of four candidates, HTMLPurifier was the only one to successfully meet all four objectives of HTML Sanitisation. It's only problem? It eats RAM and sucks CPU cycles far more than any other candidate. A point which at least some HTML Sanitisers may note as a disadvantage since they can't hang it on anything else.

Since I cannot just leave it there, and since it is always worth noting, this does not mean that HTMLPurifier is invincible. The problem with the world of Cross-Site Scripting is that there are constantly evolving new ways to attack users. HTMLPurifier has fixed several vulnerabilities in the past, and these are publicly disclosed by the author (as security advisories even). Many of them are reported directly by discoverers of new vectors which adds to HTMLPurifier security in one respect since it gets such attention. The author is also quite proactive about locating and researching fixes to possible future vulnerabilities.

HTMLPurifier is, quite simply, the only fully functioning and fully featured HTML Sanitiser in PHP. It literally stands alone.

What Does It All Mean?

Besides the fact that PHP developers are sheep to the slaughter? It means that HTML Sanitisation is incredibly misunderstood even by the authors of HTML Sanitisation solutions. It means that such solutions have minimal peer review by individuals who are relatively knowledgeable of security issues. It probably means that the days of combating XSS and Phishing and other concerns linked to HTML Sanitisation are far from over, at least for PHP.

Consider the nature of the vulnerabilities. None of them are particularly obscure or poorly documented in the public domain. There are several excellent sources of XSS/Phishing vectors for browsers, but it truly appears that everyone relies entirely on just one - the XSS (Cross Site Scripting) Cheatsheet and it's blatantly obvious that it's the only source for XSS vectors utilised by most solutions in whatever passes for their testing (I doubt anyone checks the ha.ckers.org forum for more either). This is a limited exposure issue - if you focus on the same subset of possible exploits and ignore anything else, you are entirely at the mercy of a single source of knowledge that omits quite a lot (the XSS Cheatsheet doesn't contain all possible exploits - far from it!). For example, the CSS vulnerability approach for Wordpress Kses is clearly documented on the HTML5 Security Cheatsheet but not mentioned anywhere on the more commonly referenced XSS Cheatsheet. It's a very old CSS vector.

In addition, many HTML Sanitisation solutions share common approaches and vulnerabilities. I was only looking for one vulnerability, but cross checks netted those in common. This is one indication of an isolated evolutionary line - everybody feeds off everybody else's work in the area with no attempt to look outside the PHP house to see what the weather is like over in Rubyville or Javatown (it looks really sunny in both incidentally) or even just to keep up to date on new resources, ideas and research. More interestingly, all this in-feeding seems to start with Kses in 2005. It's like the past six years dropped into a timeless black hole insulated from the harsh realities of Terra Firma even as tidal forces slowly tore them apart. The singularity being a library from 2005 that is no longer maintained (outside of Wordpress).

What is surprising, is that this isn't surprising to me anymore. It's par for the course in PHP and that hasn't changed since the 90s. I love PHP to bits, but it's a language that is tragically short on security expertise, and where a security expert can be nothing more than someone who read a book long ago and who has no incentive to move beyond the obvious or do something extraordinary - like using Google. If this article achieves nothing more than a blip on some faraway radar forgotten the next day as the download count for HTML_Safe, htmLawed, and all the other solutions I haven't examined, goes unchanged then at least I got a blip. It's a start. Maybe it's a few less XSS reports on Bugtraq or a few less people choosing an insecure solution because it claims to be fast and routinely misleads people about its efficacy in between taking senseless potshots at the only reliable solution out there. Maybe a few more developers will question all these miraculous HTML sanitisers and stop believing their summary descriptions at face value.

Developers need to start using their heads when it comes to selecting any security related solution. It's truly amazing to see developers recommend something purely on the basis of speed, or to watch them argue against all logic in support of an insecure option and just plain ignore the security implications (or assume there are none, or point at an RFC and misquote/misread it). The whole point of a security related solution IS security. If you want to compromise security to gain performance then at least be brutally honest about what it entails for the applications you build.

Do yourself a favour, use HTMLPurifier. And Ambush Commander, update the fracking comparison page already!

HTML Sanitisation Benchmarking With Wibble (ZF Proposal)

nospam@example.com (Pádraic Brady) — Thu, 08 Jul 2010 20:50:31 +0000

In January of this year, I had the idea of writing a HTML Sanitiser for PHP. Why not? All PHP has is HTMLPurifier and a bunch of random solutions that are about as secure as the average wooden gate. If you think that's harsh, wait for my next blog post . HTMLPurifier is the only secure by default HTML Sanitiser in PHP. Fact. But the darn thing is gigantic and slow. That has never stopped me using it (for years), even if I had to do a little funky engineering so I could minimise the performance hit. Other developers, however, have often abandoned HTMLPurifier, falling into the trap of believing that alternative solutions will serve them just as well.

That's the state of HTML Sanitisation in PHP - pick a big slow library that crushes Cross-Site Scripting and Phishing attacks, or use yet another regular expression based sanitiser that a) barely manages a fraction of HTMLPurifier's features and b) can probably be exploited by any scriptkiddie working with a stack of data cards. It says an awful lot about security standards among PHP developers that such delusions are uncomprehendingly rampant.

In case you haven't noticed, I'm biased. Sue me.

I have opined since forever that regular expression sanitisers are nothing short of insane. Since the problem with HTMLPurifier is speed and size, I started thinking about ways to build something like HTMLPurifier that was fast, small and almost as feature packed as HTMLPurifier. At first, this sounds like an impossible task. The typical suggestion is to use regular expressions, but I'm not completely insane...yet. Instead I borrowed a concept called a DOM Filter and chucked in a helpful dose of HTML Tidy. The result was Wibble.

Wibble is basically a DOM Filter. It loads up HTML into PHP DOM, applies a set of filters against all nodes in the DOM, passes the output through HTML Tidy, and then hands it back to the user - sanitised and well-formed. It's almost stupid in its obviousness. Better, this allows Wibble to skip regular expression dependence. It operates far more like HTMLPurifier by relying on a DOM representation (no string parsing to funk around with) partnered with Tidy for cleanup.

Of course, there have to be regular expressions somewhere. And whitelists. And other stuff. Wibble is really an amalgamation of borrowed concepts. It's hard to be too original in HTML Sanitisation because originality is a good way to shoot yourself in the foot (hence regex is EVIL!), so I wasn't going to spend too long digging my own grave when there is a wealth of sanitisation resources in the programming world. Wibble's approach borrows elements from Ruby's loofah, Python's HTML5Lib, and Java's AntiSamy. Wibble mixes and matches from the useful design elements each of these offers, serving them up on top of PHP's DOM and Tidy extensions with its own distinctive twists.

I completed the first Wibble prototype recently, so I figured that with something that was at that 90% point where the remaining 10% would be in-depth sanity testing, cleanup and documentation, it was time to see how it compared to some other PHP solutions (HTMLPurifier and HtmLawed). I had some fairly conservative performance objectives so the results came as a pleasant surprise.

If you are a benchmark fiend, you can download and independently fiddle with my benchmark process from http://github.com/padraic/wibble-benchmarks. Note that the current benchmark uses a Wibble prototype - there are additional elements that need to be added over time. The benchmark currently uses three sample snippets of HTML: Small (blog comment size), Medium (markup heavy with limited textual content), and Big (markup light with lots of textual content). It operates by filtering each HTML sample 200 times with each benchmarked HTML sanitisation solution. Each iteration includes the instantiation and setup phases of each solution (where relevant) to reflect the most likely real world experience of using sanitisation as a once off (non-repeating in same request) process. I use PEAR's Benchmark package to record the aggregate run time per loop of sanitisation tasks. All operations occur within one single PHP process with HTMLPurifier caching enabled (Wibble and HtmLawed do not use caching). Each solution is configured as close as possible to target total stripping of all HTML from the content.

You can view a sample result at http://gist.github.com/468426.

The results show that both Wibble and HtmLawed outperform HTMLPurifier by a very wide margin. Wibble underperforms HtmLawed by a variable margin - from twice as slow on small to medium sized input, to four times slower on large inputs with minimal HTML tags. In Wibble's slowest benchmark, it outperformed HTMLPurifier by a factor of four.

Wibble intent is to try and replicate the completeness of HTMLPurifier, so it's speed deficit when compared to HtmLawed is expected (when stripping all tags). There is not a lot to be done to improve this specific benchmark result since Wibble does a lot of stuff behind the scenes like encoding normalisation, DOM manipulation and HTML tidying. It also does all three of these things far more consistently and completely than HtmLawed is capable of.

So how does Wibble match up against Big Daddy? Wibble is a prototype, so obviously it still has ground to gain in terms of features with HTMLPurifier. But on the most significant points it only has one specific problem - it's not HTML 5 ready. Neither DOM or Tidy support HTML 5, though you can "pretend" it's HTML 4.01 (or even XHTML 1.0) for HTML 5 fragments so long as you are aware Tidy will strip unsupported HTML 5 tags and attributes.

The other points are syncing up with HTMLPurifier quite nicely. Wibble will santitise all HTML by default using strict filters (i.e. by default it strips every tag and only outputs plain text). It handles multiple encodings including conversion if necessary. It outputs standards compliant (other than HTML 5) HTML or XHTML. It fixes all the usual page breaking stuff like unclosed tags and illegal tag nesting. It is entirely reliant on whitelists and strict validation rather than blacklists and loose reconstructive parsing. It includes minimal regular expression usage (only needed for attribute and CSS validation) based on regular expressions widely used and tested in other languages. While testing will (and must) continue, it has so far proven resistant to XSS and Phishing attacks. This can't be absolutely assured until sufficient testing has been performed.

Otherwise, it will be interesting to see the final version of Wibble. HTMLPurifier has a tough reputation to follow, but having something which can even up the odds and do it with a pronounced advantage in speed will be really nice. Well, until someone needs to install it on CentOS .

Zend Framework Community Review Team

nospam@example.com (Pádraic Brady) — Tue, 08 Jun 2010 19:07:00 +0000

For those of you not presently keeping watch on the Contributors mailing list or IRC, a Community Review Team (CR Team) has been established to assist with contributions to the Zend Framework. The role of the team will take a bit of time to settle into and explore, but Matthew Weier O'Phinney summarised it as follows:

- Assist contributors in getting patches and features into existing components.
- Act as liaison for contacting a maintainer on behalf of a contributor
- If the maintainer refuses to accept a patch, act as an arbiter between the contributor and the maintainer
- If the maintainer does not respond after a set period of time, would evaluate and/or apply the patch for the contributor
- Would issue pull requests to the Zend team in such instances as the above
- Identify orphaned components
- Would identify when a component is no longer under active maintenance
- Solicit volunteers to take over maintenance of orphaned components
- Decide when an orphaned component should be marked as such and scheduled for removal (Note: removal can only happen in major revisions)
- Shepherd new proposals.
- Solicit community feedback on proposals
- Would put competing proposal authors in touch with each other to work on a unified proposal
- Provide feedback on proposals (including initial decision as to whether or not there is enough community interest in including the proposed functionality in the framework)
- Would notify the Zend team when a proposal is ready
- Would do initial code review on the proposal implementation
- Would notify the Zend team when the proposed feature is feature complete and ready to pull into the master branch

So essentially, the CR Team will have an advisory/liason role as it pertains to the proposing and maintenance of components. You should note that it will have limited decision capability, and Zend will continue to issue final approval for new proposals. However, if you do have a proposal in the works (or in a queue already), the Team will doubtlessly soon be looking for you .

The purpose of the CR Team is to assist in streamlining the noted areas: proposals, patches, maintenance of orphaned/abandoned components and communications with Zend and component maintainers as needed. Streamlining is a broad term, and while the specifics will be discussed by the team, it will as noted include component reviews (hopefully on an ongoing basis), offering feedback/advice, and blackmailing the community to take over from absent maintainers .

The CR Team presently has seven members (IRC nicks in brackets where known):

Pádraic Brady (PadraicB)
Rob Allen (Akrabat)
Steven Brown
Shaun Farrell (farrelley)
Pieter Kokx (kokx)
Dolf Schimmel (Freeaqingme)
Ben Scholzen (DASPRiD)

Most of the names above probably sound horribly familiar (including me!).

The Team is comprised of a fairly broad segment of active Zend Framework contributors with a variety of backgrounds. Most of the CR Team is on IRC on a daily (or weekly) basis and well known from the mailing lists. It's expected that the CR Team will serve for a fixed period (to be determined) and then have its membership reopened for review/replacements if needed. That process isn't defined at the moment but we'll get to it.

In the meantime, if you have any questions regarding the Community Review Team you can find many of us on IRC (#zftalk.dev on Freenode) and/or Twitter. You may also get our attention via the mailing lists. While it will take us a bit of time to spin up our engines and dig into our roles, you should be aware we are out there and willing to help. If you have any particularly urgent questions about proposals, patches or maintainance, I'm sure the Team will be happy to look at those while we're gearing up for full operation.

Mockery 0.6.1 Released

nospam@example.com (Pádraic Brady) — Wed, 02 Jun 2010 20:10:00 +0000

You can read more about Mockery at http://blog.astrumfutura.com/archives/427-Mockery-0.6-Released-PHP-Mock-Object-Framework.html.

Mockery 0.6.1 includes a functional fix which ensures mocking classes containing variants of the __call() method with or without typehinting are correctly mocked/replaced. I have also downgraded the PHP dependency to 5.3.0 from 5.3.2 by request. Thanks to everyone who so far has offered feedback! Mockery has been downloaded a total of 274 times since it's original release. Counting those of you doing it twice or three times on differing machines, that probably means around 100 or more people have installed Mockery (at a guess). Remember we have a mailing list if you wish to ask any in-depth questions, you can report issues or feature requests on Github, and I'm usually somewhere on IRC (and Twitter) in the evening times (GMT).

Mockery 0.6 Released - PHP Mock Object Framework

nospam@example.com (Pádraic Brady) — Thu, 27 May 2010 17:05:00 +0000

Mockery is a Mock Object framework for PHP, compatible with most unit testing frameworks including PHPUnit. Its purpose is to implement a lightweight grammer for the creation and testing of Mock Objects, Test Stubs, and Test Spies as an alternative to the built-in support offered by PHPUnit, etc.

Mockery is hosted on Github (http://github.com/padraic/mockery) where you can find an extensive README covering its API and uses. The Mockery 0.6 release may be installed from the SurviveTheDeepEnd.com PEAR channel at http://pear.survivethedeepend.com.

Mockery 0.6 features:

- Full Mock Object and Test Stub support
- Lightweight fluent API
- Flexible mocking and stubbing
- Object Interaction Recording
- Natural language syntax and expectation constructs
- Supports generic (untyped) mock objects for rapid prototyping
- Simple partial mocking of real objects
- Both local and global Mock Object call ordering
- Built-in return value queue for repeated method calls
- Support for default expectations
- Support for expectation replacement and stacking
- Fluent API/Law of Demeter mocking

If that sounds complex, it's not! Mockery can be picked up and used with little study.

Why Mockery?

Mockery's objective is to simplify Mock Objects in PHP while maintaining significant flexibility and a default level of intuitive behaviour. In Mockery, Mock Objects behave exactly as you write them with liberal interpretations otherwise applied. Mockery was born out of my own need to innovate the use of Mock Objects in PHP and draw away from the original import of aging Mocking approaches from Java. While Java (and almost every other programming language) has been steadily progressing its mock object libraries, and complementing them with new solutions, PHP has a relatively static approach depending on similarly static library components. That result has seen solutions using clunky APIs, poorly described syntax and behaviour, a lack of focus on the practice of using Mock Objects, user confusion, and raised barriers to new programmers trying to learn about Mock Objects. Mockery is one potential solution to these problems. Also, as a dedicated Test-Driven Design user, I really want something that clicks immediately and doesn't have any gotchas.

Installation

Mockery may be installed from its PEAR channel using:

pear channel-discover pear.survivethedeepend.com
pear install deepend/Mockery

Mockery is written in PHP 5.3 (I know, but all you 5.2 users will get there eventually ). It is released under a New BSD license.

Example

The README offers a good look at some examples, and explains the API in a lot of detail. If you are trying to figure something out, the README undoubtedly has a section for it. Here's an API example (assuming Mockery namespace used as MK). We're capturing an interaction where we login into a bookmarking service, check for the existance of a "php" tagged bookmark, add three more bookmarks and then recheck if a "php" tag exists (twice for fun). We're mocking the service since we don't actually want to mess with a real account! Following the description closely...

$service = \MK::mock('MyService');
$service->shouldReceive('login')->with('user', 'pass')->once()->andReturn(true);
$service->shouldReceive('hasBookmarksTagged')->with('php')->once()->andReturn(false);
$service->shouldReceive('addBookmark')->with('/^http:/', \MK::type('string'))->times(3)->andReturn(true);
$service->shouldReceive('hasBookmarksTagged')->with('php')->twice()->andReturn(true);

The example uses some of the basic parts of Mockery to describe some interaction with a mocked web service class (obviously
also stubbing the web service's responses in terms of booleans). The setup is straightforward, easy to follow, and there's zero
misinterpretations possible. Our description was likewise simple and uncomplicated. The third line just shows two argument matchers at work, a default regex (intrepreted from any string argument set so long as any eventual string comparison fails and it's a valid regex) and a Type matcher set to match any valid string.

To put this into some perspective, here's an equivalent attempt using PHPUnit in a similar order of thought (excerpt from a test).

$service = $this->getMock('MyService');
$service->expects($this->once())->method('login')->with('user', 'pass')->will($this->returnValue(true));
$service->expects($this->once())->method('hasBookmarksTagged')->with('php')->will($this->returnValue(false));
$service->expects($this->exactly(3))->method('addBookmark')
->with($this->matchesRegularExpression('/^http:/'), $this->isType('string'))
->will($this->returnValue(true));
$service->expects($this->exactly(2))->method('hasBookmarksTagged')->with('php')->will($this->returnValue(true));

Besides the differences in API, there are others. If MyService is just intended as a fake unimplemented object (the class doesn't exist), Mockery carries on and just uses a generic Mock instance without error. PHPUnit will throw an exception, however, stating that login() is not a valid method. If we assume the class is real, but missing some methods, the same thing happens and PHPUnit complains about missing methods. Eventually, you'll get the idea to implement the dependent class... If we add all the relevant methods (say, we mock an interface with all methods declared), PHPUnit STILL fails. This time complaining that hasBookmarksTagged() was expected only once. This occurs because PHPUnit has no capacity for stacking later expectations, and so, it ignores the second (and any later) ones. We can fix that by merging both into a single expectation using:

$service = $this->getMock('MyService');
$service->expects($this->once())->method('login')->with('user', 'pass')->will($this->returnValue(true));
$service->expects($this->exactly(3))->method('hasBookmarksTagged')->with('php')
->will($this->onConsecutiveCalls(false, true, true));
$service->expects($this->exactly(3))->method('addBookmark')
->with($this->matchesRegularExpression('/^http:/'), $this->isType('string'))
->will($this->returnValue(true));

Using OnConsecutiveCalls() to create a return value queue, and merging the two stacked expectations, allows the PHPUnit variant to pass. Unlike Mockery, if there were ten hasBookmarksTagged() calls, you would need to add all ten return values (Mockery let's you set the last return value to act infinitely). The merging simply demonstrates that complex class interactions across classes will fall victim to the need to constantly merge expectations until they are unreadable and explain little.

While your mileage may vary, Mockery just doesn't need reworking, deep thought or extra work. Just state what you want your Mock Object to do in plain unconfused English according to your natural thought order! If nothing else, it helps make the expected interaction obvious which makes your tests more readable and explicit.

Feedback

Any issues can be reported via our Github hosted issue tracker. If you wish to discuss Mockery in more detail, you're welcome to join the mailing list at http://groups.google.com/group/phpmockery.

Mockery: From Mock Objects to Test Spies

nospam@example.com (Pádraic Brady) — Sun, 23 May 2010 16:01:38 +0000

With next week seeing the formal release of Mockery 0.6 which is currently fermenting on Github at http://github.com/padraic/mockery, I'm already looking forward to next piece of the puzzle arriving with 0.7. Mockery is an opportunity to being something new and fresh to the PHP mock objects environment beyond a far neater flexible API.

If you follow the test double debate in other languages there are two popular concepts making their presence felt. The first is that the over-specified definitions of test doubles (we have dummy objects, test stubs, mock objects, test spies and fake objects) do more harm than good since in reality we never really distinguish (nor care to) between most of them. In most cases we just sort of wing it, using our own concepts and intent to create test doubles which do what we want with little thought as to the kind of test double we're creating. In the end, the definitions all merge together into a spectrum of possible behaviours we add as and if needed.

The second is that we're seeing a lot more attention over whether Mock Objects are the best way to do things. Nobody doubts they have the right idea with Mock Objects, but the upfront setting of expectations may come across as unintuitive. A typical test begins with exercising code, and thereafter we typically make assertions on the behaviour (whether it be a resulting state change or an object interaction). Mock Objects reverse that - we expect first, and then we exercise the code.

With the future (before end of June) 0.7 release, Mockery will offer an alternative approach to the traditional Expect-Exercise-Verify cycle called the Test Spy. 0.6 already offers a measure of that capability through its ability to record interactions, though recording is completely automated and geared primarily towards comparing two sets of source code which should behave identically (e.g. during refactoring). The new Test Spy implementation will be API driven, using a similar form to Mockery's existing Mock Object approach.

Why Test Spies?

Using Test Spies returns to the test based approach of exercising code first, and then making assertions. There's a reason why this cyclical approach is used in unit testing. It's easy to understand and makes sense from the start. Applying it to what we do with Mock Objects can make using Test Doubles in general easier to learn and teach. It may also remove barriers to entry for those who simply never understood Mock Objects or loathed their expectation setup. Test Spies also operate on a selective basis - where Mock Objects require upfront expectations to define how they will behave, Test Spies just do what you need them to and then allow selective assertions afterwards (i.e. you only need to test what you want to test).

The difference, once you see the code, is very subtle. Not unusually for testing practices, subtle differences in approach rely on the YMMV yardstick. Some people prefer the upfront expectation setting, others find the after-use assertion approach easier to understand or as more applicable to their thinking.

By supporting both approaches as equal citizens, Mockery once again lets go of the proverbial rope around your neck tied there by Java traditions. You'll be free to go with whatever approach you find yourself preferring. Better, they can both co-exist in relative peace since they are drawn from the same framework and both share nearly all of the exact same API elements, with the exact same meaning. Whether you prefer Mock Objects or Test Spies (both equally are capable of being used as Test Stubs), they are both equally understandable by anyone using Mockery.

As a means of showing the differences, here are two test cases. The first is written using the Mock Object terminology, the second using the Test Spy terminology - each actually achieves the exact same goal.

use \Mockery as M;

class StarshipTest extends PHPUnit_Framework_TestCase
{

public function testEngagingWarpDrive()
{
$engineering = M::mock('Engineering');
$engineering->shouldReceive('prepForWarpFactor')->with(8)->once()->andReturn(true);
$engineering->shouldReceive('engage')->once();

$starship = new Starship($engineering);
$starship->accelerateToWarp(8);
}

}

Or, using a Test Spy approach.

use \Mockery as M;

class StarshipTest extends PHPUnit_Framework_TestCase
{

public function testEngagingWarpDrive()
{
$engineering = M::mock('Engineering');
$engineering->whenReceives('prepForWarpFactor')->thenReturn(true);

$starship = new Starship($engineering);
$starship->accelerateToWarp(8);

$engineering->assertReceived('prepForWarpFactor')->with(8)->once();
$engineering->assertReceived('engage')->once();
}

}

To demonstrate the selective assertion approach, what if we just didn't care about whether or not the engage() method was used? In that case, we could just drop it from the assertions altogether - never mentioning it in the test at all. In the Mock Object approach, we cannot do this - mocking means we must set all methods expected (otherwise the methods would not exist on the mock). We could even drop the assertion on prepForWarpFactor, although we still need to retain the stubbing of its return value (since it's needed).

Supporting Test Spies and Mock Objects also has the other obvious benefit that you can switch modes effortlessly. There will always be cases where expectation setting is preferable over assertions and vice versa. Both have their uses even if you heavily prefer any one over the other.

Watch for the Mockery 0.6 release next week! I'll follow through with Test Spies in 0.7 once I get the final API down (working on making it even shorter than our Mock Object API can achieve).

The Mockery: PHP Mock Objects Made Simple

nospam@example.com (Pádraic Brady) — Thu, 20 May 2010 19:03:16 +0000

As I explored in my previous post, PHP Mock Objects: Sucking Ass Since Forever, Mock Objects in PHP have had difficulty gaining traction. One of the targets of this examination (in between variations of "I love PHPUnit") was PHPUnit's Mock Objects implementation. My main point, I suppose, was that with PHPUnit being the de-facto standard for unit testing, its mock objects implementation left a lot to be desired. As the later comments suggested, suger coating the situation won't change the fact I'm criticising one of the most deeply embedded libraries in PHP practice (well, actually a teeny tiny bit of it).

So what is Mockery? Mockery is a mock object framework. You can use it to mock, stub and dazzle your classes into submission - even when they don't exist yet . Mock objects are almost a given when it comes to Test-Driven Development, but even non-TDD unit testing can use mocks and stubs. They encourage test isolation, stand in for classes you haven't implemented yet, and can replace resource intensive tasks with mocked versions (since we like our tests to run really fast). Using Mock Objects (or Stubs) is just incredibly useful.

So what does Mockery offer you, the unit tester?

1. It integrates with pretty much any conceivable testing framework with almost no effort.
2. It's simple to use, intuitive, and specifically works with the language we use to describe object interactions.
3. The API is uncluttered and minimal.
4. Flexibility is its cornerstone allowing it to accurately and succintly express your requirements.
5. It enables simple one-off setups that can be later modified so you don't need to re-specify entire mocks for every single test.
6. If you use Ruby and PHP, it will make your day (yes, I also write Ruby - sue me).

Why consider it instead of PHPUnit's or (insert PHP testing framework)'s mock object implementation?

1. It ports to ANY testing framework with zero effort
2. It isn't inspired by Java
3. It has no gotchas, requirements for custom callback code, or complicated and frustrating setups.
4. It's capable of mocking real objects and performing mock object recordings of existing processes.
5. Expectations are not pre-constrained - you hold the rope.

Time for the inevitable API exploration... Mockery is written for PHP 5.3, so get your namespace booties on. If you cannot currently adopt PHP 5.3, read along anyway. You'll be using PHP 5.3 eventually .

You can find Mockery on Github at http://github.com/padraic/mockery. Besides the code, there is one massive README that will go into the API and usage in far more detail than I can do in this article. It is currently not released to a PEAR channel, but I'll be doing so once I feel happy that the framework has enough hard-core crazy features to last its lifetime in PHP. You can, however, clone from the git repository and install it via PEAR using the included package.xml file. I love issue reports - if you have them, don't be stingy and keep them to yourself.

For those who find themselves curious about how, specifically, Mockery pits against PHPUnit Mock Objects, I'll run a later article contrasting the two in terms of features and examples.

The basis of Mockery's API is the English language. It's a Domain Specific Language (DSL) in the form of a fluent interface. Given that basis, it's not surprising that Mockery's intent is to capture every possible object expectation within a DSL statement that reads like English.

For example, let's assume we have the class Engineering which is utilised by the class Starship. Hey, I wrote a whole article based on a dodgy Klingon once so get with the programme. We're testing the Starship class, and find that there is no Engineering class...yet. So we shrug, and decide to use a mocked object to replace Engineering. One of the consequences of this approach is that, down the line, the mock usage will actually specify the API our real Engineering class should use (this is test-driven DESIGN, afterall).

To boot up the creative juices, we start with a story:

Given our Starship has located a planet, it will instruct Engineering to turn off the warp drive, and then to divert 40% power to sensors, and then to divert 20% power to auxilliary engines, and finally to run a Level-1 diagnostic on itself. All instructions must be given once in this precise order. There may optionally be many diagnostic runs of any Level at any stage in the process. The Starship must report whether the operation was successful. Success is measured by successfully passing a Level 1 diagnostic.

Okay, so what exactly are we expecting?

use \Mockery as M;

require_once 'Starship.php';

class StarshipTest extends PHPUnit_Framework_TestCase
{
public function testEngineeringResponseToEnteringOrbit()
{
$mock = M::mock('Engineering');
$mock->shouldReceive('disengageWarp')->once()->ordered();
$mock->shouldReceive('divertPower')->with(0.40, 'sensors')->once()->ordered();
$mock->shouldReceive('divertPower')->with(0.30, 'auxengines')->once()->ordered();
$mock->shouldReceive('runDiagnosticLevel')->with(1)->once()->andReturn(true)->ordered();
$mock->shouldReceive('runDiagnosticLevel')->with(M::type('int'))->zeroOrMoreTimes();

$starship = new Starship($mock);
$this->assertTrue($starship->enterOrbit());
}

public function teardown()
{
M::close();
}
}

There's the test. Our mock has been setup according to the expectations implicit in the feature story. As you might notice, the final diagnostic calls are unordered (can occur at any time between ordered calls). For PHPUnit, the teardown() method is optional if you configure PHPUnit to use the bundled TestListener. I added it to show that it's the sole integration needed with a testing framework (unless using the bundled TestListener for PHPUnit).

Now let's write just enough code to pass this test in Starship.php:

class Starship
{

protected $_engineering = null;

public function __construct($engineering)
{
$this->_engineering = $engineering;
}

public function enterOrbit()
{
$this->_engineering->disengageWarp();
$this->_engineering->runDiagnosticLevel(5); // unordered call!
$this->_engineering->divertPower(0.40, 'sensors');
$this->_engineering->divertPower(0.30, 'auxengines');
$criticalResult = $this->_engineering->runDiagnosticLevel(1);
return $criticalResult;
}

}

Hurrah! The test passes. We've just tested Starship without needing a real Engineering object (or even a class of that name!). You can find this example in the examples directory on Github for Mockery, along with the phpunit.xml configuration and Bootstrap.php file used.

If something did go wrong with our expectations, Mockery would throw an Exception to complain about it.

The Mockery API also handles return values (including a return queue for multiple method invocations with the same arguments), of course, along with a host of other features and expectation term methods. Refer to the README up on Github. Like I said before, it's very complete and informative. We've barely touched on the basics of Mockery in this article, so get reading.

PHP Mock Objects: Sucking Ass Since Forever

nospam@example.com (Pádraic Brady) — Tue, 18 May 2010 17:21:34 +0000

Admit it, you look forward to my colourful blog posts . This one has been two years in the making, but it's only recently, with the wait for Zend Framework 2.0, that I've had time to pull together all the strands so I could offer something concrete as a solution. I don't like complaining without a solution to hand...

The theme of this article is Mock Objects, and how they suck in PHP. You could throw Stubs in here, but distinguishing between the two has little advantages other than Stubs are a simpler subset of Mocks. Both do something similar - they basically fake a real object, simulating all the real object's behaviour as closely as possible (within their respective goals) so that, for the purposes of isolating a System-Under-Test (SUT), e.g. a specific class or specific library, you don't need to drag in a stack of real objects which breach that isolation. The same holds true for ever present dependencies like databases, filesystems, web services, etc. Hint, even they are usually pulled in via classes - which can be mocked.

The unique property of mock objects, compared to all other test doubles (fake objects), is that they are tools for verifying behaviour. For example, if we expect to call method foo() on class Foo(), and we don't, a mock object would complain long and hard about it. A Stub, on the other hand, would not. Stubs setup canned responses to method calls - they don't care if the call is ever made.

Isolation is a noteworthy goal since it keeps our tests laser focused on just the class/system being tested without being worried if some other dependent class is screwing up, but Mock Objects are useful in others way. In Test-Driven Design, the unit tests for a class are written without regard for its current state of implementation (test-first!). Despite lacking an implementation, the API of such a class may assume interactions with other classes. The problem? Without an implementation of these "other" classes, how can we even test the current class? How can we even tell if our class is interacting (making calls on) the "other" classes properly?

Note: Usually we tend to see Stubs taking precedence here in PHP, since a Stub would return a value, which in turn impacts some real result from the class being tested. This is a verification of state, or state-verification. It's NOT an invalid approach but neither does it verify behaviour (it doesn't specify how the classes all interact, simply that they have in some magical unknown way). Mocks are more of a behaviour-verification approach with design benefits (we're exploring the APIs of these other theoretical classes prior to implementing them). The differences are explored really well in Martin Fowler's Mocks Aren't Stubs article.

The answer to verifying class interactions (behaviour of APIs) is to stick in a mock object which simulates the expected behaviour of the non-existing class. Better, the mocking strategy means we are also testing the interaction between classes (i.e. we're feeling out or exploring the API between classes), something that will make the new classes easier to write tests for since we already explored their API! It also helps avoid an overreliance on state. In unit tests state is always being tested, but sometimes state is nothing more than a collection of faked up data (e.g. using Stubs) which does nothing to help with DESIGN. In other cases, the state is just plain meaningless (e.g. our class is an Abstract or Proxy). Remember, TDD is Test-Driven DESIGN!

The alternative answer is, of course, to implement or stub-out (minimally implement) all the classes right now to facilitate state-verification. The is probably "The PHP Way" that's most common, even if not practicing TDD (MOs and TDD are mutually exclusive concepts). This lack of mock object use is easily proven by using grep on any conceivable PHP application and looking for the term "mock", or you can use "getMock" for PHPUnit tests. Seriously, try it now, go to your Zend Framework (or other) tests directory and try "grep -r getMock ." (assuming PHPUnit). Filter out the SVN trash, and count them up. Now eliminate the blind expectations (solely added to return values predictably - these are Stubs, not Mocks). Now figure out how many fingers and toes they equate to.

Mock Objects in PHP have just never really taken off in a huge way which is decidedly at odds with other programming languages.

Back to Sucking Ass, why is it that a wider PHP adoption of Mock Objects never happened? Sure, we say we "mock" all the time, but mostly we're talking about Stubs or once-off hand-painted mock objects derived from one of the main unit testing libraries (I'm getting there). We never actually use Mock Objects outside of very simple restrictive scenarios. In fact, mostly we just avoid them like the plague.

The problem, partly, is that PHP does not have a full featured mock object library in common use. Without one of those, trying to use mock objects is a bit hard. Well, impossible.

Incidentally, this is not a PHPUnit bashing exercise . I can imagine you can all see where I'm going with this, but PHPUnit doesn't evolve itself and blaming it or anyone involved would be pure nonsense. Like all open source projects, flaws are the fault of the greater community which has failed to rectify them. So it's OUR fault! Go look in a mirror and give yourself a headslap or find a Leroy Jethro Gibbs lookalike to do it for you. While you have one, please ask how the hell he gets those boats out of his basement!

Back on track, PHPUnit does offer a Mock Object solution. This is the crux of some lengthy arguments I enjoyed two years ago with different people who persisted in confusing mock objects from Java, Ruby, Python, .NET, ad infinitum and mock objects from PHPUnit. They are not the same! PHPUnit's implementation is heavily influenced (judging from the API) by Java's jMock (one of the original mock object libs). However, it comes with numerous limitations, some of which are taken from a jMock designed way back when in the 1920s (it looked suspiciously like a sewing machine) and some are simply omitted features or, possibly, misinterpretations from jMock. Combined, these limitations are destined to frustrate anyone who tries using mock objects, especially if they come from anywhere other than PHP. Try getting a Ruby developer to switch from flexmock or mocha to PHPUnit and they will go insane (which may not be a bad thing depending ) or run screaming to DHH who is the father of Ruby (if you believe the hype).

The first immediate flaw with PHPUnit Mock Objects is the API. The nested API is reminiscent of jMock, but without Java's shorter forms it's clunky and unintuitive. The second obvious flaw, something most likely to have frustrated you in the past as it does everyone, is that managing multiple method expectations isn't possible. Rather than queue up, they seem to overwrite previously made similar expectations. The third isn't actually a terrible flaw, but more of an inconvenience, whereby mocking classes still needs some input to deal with constructors and object instantiation. This leads to some gymnastics when instantiating a new mock. There are many other subtle "flaws" you probably never even thought of unless coming from another programming language.

You have to admit, that the flaws don't sound all that terrible in summary form, and that is why they are so bad. Their sheer simplicity hides their impact. In practice, however, they collide with our sense of object manipulation (complex, chainable, hierarchical and conditional). And this is why, ultimately, you can't really mock with PHPUnit mock objects. Mock objects simulate every possible behaviour of real objects - if you can't program a mock object to deal with the common intricacies of object interactions then you cannot use mock objects. Period. You can't put up with expectation overwriting, a kludgy API, overly complex mock setup, and the lack of features you never even knew existed, and expect to find mock objects fascinating. Instead, you'll just end up frustrated with a looming sense that mock objects are not worth using. Then you stop trying. Or you resort to Stackoverflow, and then stop trying. Either way, in the end, you stop.

The proof is everywhere... "grep -r getMock ." Que X-Files theme music.

Fresh off the frying pan, PHPUnit isn't exactly overly damaged . It's an excellent unit testing framework with excellent support with one fixable blemish. So, how do we fix Mock Objects?

Of course, there has to some self-promoting agenda to this blog post! About two years ago I prototyped a mock object framework called Mockery. Since it was a prototype, and a learning experience, it turned out to be a rusty implementation with a few decidedly ugly flaws. Nevertheless, I maintained a more sane version which I've been using privately all this time. It's fair to say, though I'm biased and you don't need to believe me, that it made me a better developer. I found it easier to write beautiful code (like mathmaticians, programmers are driven by aesthetics). Having a good mock object solution does have its benefits, particularly when you're trying to apply Test-Driven Design. And it works, so long as you stay away from any open source project (see later).

With the recent push out of a rewritten version of MutateMe (a Mutation Testing framework - another practice long overdue for a PHP outing), I gave the same treatment to Mockery. Mockery is now rewritten for PHP 5.3, and I'm busy rounding out its mocking ability (specifically creating class-type obedient mocks - the rest is done) before announcing a beta release.

I'll talk specifically about Mockery in another blog post later, but you can find the code and the exhaustive README over at http://github.com/padraic/mockery. It has a few known kinks, but another day or two with some free hours will finish it.

Back to why I want to unleash Mockery against PHP's Ass Sucking Mocks, it's because it will make life easier. For me. I'm selfish that way but, perhaps, it'll also make life easier for you. Open source is all about selfishly fulfilling our own needs before unselfishly slapping it with a New BSD sticker. My selfishness serves the community . So does yours!

Removing all the mumbo-jumbo, I need to get Mockery "out there". Out there is a weird place, a world filled with thousands upon thousands of unit test suites whose developers think Mock Objects were born in PHPUnit and will die there. In the Zend Framework, for example, I can't use mock objects (as neither can anyone else). Sure, I throw in the odd Stub (as with Zend_Feed_Pubsubhubbub). By the way, that grep list? Remove an index finger for Zend_Feed_Pubsubhubbub's mocks - they're stubs. We'll count them as a single digit deduction. Point is that trying to use Mockery mocks in ZF would kick up a massive fuss as unit tests the world over died from a missing dependency.

Why can't I use mocks? I want to use mocks! But first, I need all those unit tests suites to accept a dependency on a mock object library like Mockery (or another, assuming there is another outside of Mockery and yaymock (yes, there's a yaymock effort)). Until that happens, mock objects in PHP are a lost cause. So this opening post (I'll introduce Mockery later) is a call out to fellow Mocketeers (catchy?) and future Mocketeers (everyone reading this has been drafted ). Get thinking about Mock Objects! If we think as one, our psychic oneness might result in gaining a (i.e. my) common use mock object library that we are more free to use in all those PHPUnit test suites sweeping across Github.

And I said "removing all mumbo-jumbo"...

On a serious note, we need mock objects. If Mockery actually turns out okay (fingers crossed - detailed look tomorrow), I could use the help in spreading the word about its awesomeness.

MutateMe: A PHP Mutation Testing Framework

nospam@example.com (Pádraic Brady) — Wed, 12 May 2010 16:03:47 +0000

Been while since I've been blogging! I'm just off a nice months long project so I'm easing my way back onto the open source scene. Since I take these times of renewed effort to work on crazy stuff (gets the inspiration going again), I revisited an old piece of code I first mentioned way back a year or so ago that I had dubbed MutateMe.

You can find it (and the exhaustive README) over at http://github.com/padraic/mutateme. You can install it using PEAR (I'll get it to a PEAR channel soon) and it sits at a notional 0.5 development version.

MutateMe is a PHP Mutation Testing framework. Mostly. There is still work to be done, but the core of the framework has been completed with that fancy new namespace magic PHP 5.3 introduced. It doesn't even crash anymore...that I've seen in...oh...at least a couple of hours . Special thanks to Arvid Bergelmir who played a real good guinea pig over the past week in testing this out .

Onto some explanations...

Mutation Testing is a little known (in PHP) practice of deliberately adding errors into source code. The idea is that, your sparkling perfect 100% code coverage jewel of a test suite should immediately pop out a failed test when the error is introduced. If it doesn't, you get to ask why the error did not produce a test failure. Maybe it's a missing test, or maybe the error was simply so minor or inconsequential it had no test-worthy effect. Either way, it's food for thought.

The range of errors that can be introduced range from simple swaps (like sneakily swapping TRUE and FALSE somewhere) to more complex blunders (like replacing a regular expression with one matching only the Lead Developer's cat's name). At present MutateMe just implements a dozen or so simple swaps (the rest will follow now that the core framework is stable).

So, in a sense, Mutation Testing is quality control for unit tests. It helps indicate that you have good tests (or not) that will prevent a future apocalypse. MutateMe currently supports PHPUnit out of the box (since doing otherwise would be project suicide).

Back to MutateMe. It's a relatively simple command line utility (much like PHPUnit). It operates by examining your source code, figuring out how to screw it all up, and then applies each screw up (mutations) to the source code in turn while re-running your test suite. All mutations are applied in isolation, of course - we run tests in a new PHP process each time to maintain some isolation. Also, the changes are not applied to the filesystem since we use the marvellous runkit extension to alter classes in memory.

Please be sure to read the README on how to install MutateMe and runkit, and then how to commence some amazing source code mangling .

As far as I'm aware, MutateMe is currently the only (I always hope I'm wrong) Mutation Testing framework in PHP. If you have some small source code pile to experiment with it, I'd love to hear about your experiences, comments, and suggestions for the future. I say "small" with meaning - remember that Mutation Testing runs the assigned test suite (or any subset thereof) for every single mutation generated from the source code (likewise, or any subset thereof). Mutation Testing is something you do occassionally rather than every other minute .

Final notes? Given the runkit requirements, I'm afraid Windows users may feel neglected. We're using a patched version of the official runkit (far better feature set and fewer bugs) but there's no precompiled DLL for it. If anyone wants to link me to a working PHP 5.3 (with static method support) Windows DLL, I'll happily note it in the README. I'll work on some Windows-dependent workarounds to avoid static method mutations otherwise.

Happy mutating! Don't forget the feedback. Github has an issue tracker available for any bugs and/or feature requests.