Maugrim The Reaper's Blog

As I explored in my previous post, PHP Mock Objects: Sucking Ass Since Forever, Mock Objects in PHP have had difficulty gaining traction. One of the targets of this examination (in between variations of "I love PHPUnit") was PHPUnit's Mock Objects implementation. My main point, I suppose, was that with PHPUnit being the de-facto standard for unit testing, its mock objects implementation left a lot to be desired. As the later comments suggested, suger coating the situation won't change the fact I'm criticising one of the most deeply embedded libraries in PHP practice (well, actually a teeny tiny bit of it).

So what is Mockery? Mockery is a mock object framework. You can use it to mock, stub and dazzle your classes into submission - even when they don't exist yet . Mock objects are almost a given when it comes to Test-Driven Development, but even non-TDD unit testing can use mocks and stubs. They encourage test isolation, stand in for classes you haven't implemented yet, and can replace resource intensive tasks with mocked versions (since we like our tests to run really fast). Using Mock Objects (or Stubs) is just incredibly useful.

So what does Mockery offer you, the unit tester?

1. It integrates with pretty much any conceivable testing framework with almost no effort.
2. It's simple to use, intuitive, and specifically works with the language we use to describe object interactions.
3. The API is uncluttered and minimal.
4. Flexibility is its cornerstone allowing it to accurately and succintly express your requirements.
5. It enables simple one-off setups that can be later modified so you don't need to re-specify entire mocks for every single test.
6. If you use Ruby and PHP, it will make your day (yes, I also write Ruby - sue me).

Why consider it instead of PHPUnit's or (insert PHP testing framework)'s mock object implementation?

1. It ports to ANY testing framework with zero effort
2. It isn't inspired by Java
3. It has no gotchas, requirements for custom callback code, or complicated and frustrating setups.
4. It's capable of mocking real objects and performing mock object recordings of existing processes.
5. Expectations are not pre-constrained - you hold the rope.

Time for the inevitable API exploration... Mockery is written for PHP 5.3, so get your namespace booties on. If you cannot currently adopt PHP 5.3, read along anyway. You'll be using PHP 5.3 eventually .

You can find Mockery on Github at http://github.com/padraic/mockery. Besides the code, there is one massive README that will go into the API and usage in far more detail than I can do in this article. It is currently not released to a PEAR channel, but I'll be doing so once I feel happy that the framework has enough hard-core crazy features to last its lifetime in PHP. You can, however, clone from the git repository and install it via PEAR using the included package.xml file. I love issue reports - if you have them, don't be stingy and keep them to yourself.

For those who find themselves curious about how, specifically, Mockery pits against PHPUnit Mock Objects, I'll run a later article contrasting the two in terms of features and examples.

The basis of Mockery's API is the English language. It's a Domain Specific Language (DSL) in the form of a fluent interface. Given that basis, it's not surprising that Mockery's intent is to capture every possible object expectation within a DSL statement that reads like English.

For example, let's assume we have the class Engineering which is utilised by the class Starship. Hey, I wrote a whole article based on a dodgy Klingon once so get with the programme. We're testing the Starship class, and find that there is no Engineering class...yet. So we shrug, and decide to use a mocked object to replace Engineering. One of the consequences of this approach is that, down the line, the mock usage will actually specify the API our real Engineering class should use (this is test-driven DESIGN, afterall).

To boot up the creative juices, we start with a story:

Given our Starship has located a planet, it will instruct Engineering to turn off the warp drive, and then to divert 40% power to sensors, and then to divert 20% power to auxilliary engines, and finally to run a Level-1 diagnostic on itself. All instructions must be given once in this precise order. There may optionally be many diagnostic runs of any Level at any stage in the process. The Starship must report whether the operation was successful. Success is measured by successfully passing a Level 1 diagnostic.

Okay, so what exactly are we expecting?



There's the test. Our mock has been setup according to the expectations implicit in the feature story. As you might notice, the final diagnostic calls are unordered (can occur at any time between ordered calls). For PHPUnit, the teardown() method is optional if you configure PHPUnit to use the bundled TestListener. I added it to show that it's the sole integration needed with a testing framework (unless using the bundled TestListener for PHPUnit).

Now let's write just enough code to pass this test in Starship.php:



Hurrah! The test passes. We've just tested Starship without needing a real Engineering object (or even a class of that name!). You can find this example in the examples directory on Github for Mockery, along with the phpunit.xml configuration and Bootstrap.php file used.

If something did go wrong with our expectations, Mockery would throw an Exception to complain about it.

The Mockery API also handles return values (including a return queue for multiple method invocations with the same arguments), of course, along with a host of other features and expectation term methods. Refer to the README up on Github. Like I said before, it's very complete and informative. We've barely touched on the basics of Mockery in this article, so get reading.

Admit it, you look forward to my colourful blog posts . This one has been two years in the making, but it's only recently, with the wait for Zend Framework 2.0, that I've had time to pull together all the strands so I could offer something concrete as a solution. I don't like complaining without a solution to hand...

The theme of this article is Mock Objects, and how they suck in PHP. You could throw Stubs in here, but distinguishing between the two has little advantages other than Stubs are a simpler subset of Mocks. Both do something similar - they basically fake a real object, simulating all the real object's behaviour as closely as possible (within their respective goals) so that, for the purposes of isolating a System-Under-Test (SUT), e.g. a specific class or specific library, you don't need to drag in a stack of real objects which breach that isolation. The same holds true for ever present dependencies like databases, filesystems, web services, etc. Hint, even they are usually pulled in via classes - which can be mocked.

The unique property of mock objects, compared to all other test doubles (fake objects), is that they are tools for verifying behaviour. For example, if we expect to call method foo() on class Foo(), and we don't, a mock object would complain long and hard about it. A Stub, on the other hand, would not. Stubs setup canned responses to method calls - they don't care if the call is ever made.

Isolation is a noteworthy goal since it keeps our tests laser focused on just the class/system being tested without being worried if some other dependent class is screwing up, but Mock Objects are useful in others way. In Test-Driven Design, the unit tests for a class are written without regard for its current state of implementation (test-first!). Despite lacking an implementation, the API of such a class may assume interactions with other classes. The problem? Without an implementation of these "other" classes, how can we even test the current class? How can we even tell if our class is interacting (making calls on) the "other" classes properly?

Note: Usually we tend to see Stubs taking precedence here in PHP, since a Stub would return a value, which in turn impacts some real result from the class being tested. This is a verification of state, or state-verification. It's NOT an invalid approach but neither does it verify behaviour (it doesn't specify how the classes all interact, simply that they have in some magical unknown way). Mocks are more of a behaviour-verification approach with design benefits (we're exploring the APIs of these other theoretical classes prior to implementing them). The differences are explored really well in Martin Fowler's Mocks Aren't Stubs article.

The answer to verifying class interactions (behaviour of APIs) is to stick in a mock object which simulates the expected behaviour of the non-existing class. Better, the mocking strategy means we are also testing the interaction between classes (i.e. we're feeling out or exploring the API between classes), something that will make the new classes easier to write tests for since we already explored their API! It also helps avoid an overreliance on state. In unit tests state is always being tested, but sometimes state is nothing more than a collection of faked up data (e.g. using Stubs) which does nothing to help with DESIGN. In other cases, the state is just plain meaningless (e.g. our class is an Abstract or Proxy). Remember, TDD is Test-Driven DESIGN!

The alternative answer is, of course, to implement or stub-out (minimally implement) all the classes right now to facilitate state-verification. The is probably "The PHP Way" that's most common, even if not practicing TDD (MOs and TDD are mutually exclusive concepts). This lack of mock object use is easily proven by using grep on any conceivable PHP application and looking for the term "mock", or you can use "getMock" for PHPUnit tests. Seriously, try it now, go to your Zend Framework (or other) tests directory and try "grep -r getMock ." (assuming PHPUnit). Filter out the SVN trash, and count them up. Now eliminate the blind expectations (solely added to return values predictably - these are Stubs, not Mocks). Now figure out how many fingers and toes they equate to.

Mock Objects in PHP have just never really taken off in a huge way which is decidedly at odds with other programming languages.

Back to Sucking Ass, why is it that a wider PHP adoption of Mock Objects never happened? Sure, we say we "mock" all the time, but mostly we're talking about Stubs or once-off hand-painted mock objects derived from one of the main unit testing libraries (I'm getting there). We never actually use Mock Objects outside of very simple restrictive scenarios. In fact, mostly we just avoid them like the plague.

The problem, partly, is that PHP does not have a full featured mock object library in common use. Without one of those, trying to use mock objects is a bit hard. Well, impossible.

Incidentally, this is not a PHPUnit bashing exercise . I can imagine you can all see where I'm going with this, but PHPUnit doesn't evolve itself and blaming it or anyone involved would be pure nonsense. Like all open source projects, flaws are the fault of the greater community which has failed to rectify them. So it's OUR fault! Go look in a mirror and give yourself a headslap or find a Leroy Jethro Gibbs lookalike to do it for you. While you have one, please ask how the hell he gets those boats out of his basement!

Back on track, PHPUnit does offer a Mock Object solution. This is the crux of some lengthy arguments I enjoyed two years ago with different people who persisted in confusing mock objects from Java, Ruby, Python, .NET, ad infinitum and mock objects from PHPUnit. They are not the same! PHPUnit's implementation is heavily influenced (judging from the API) by Java's jMock (one of the original mock object libs). However, it comes with numerous limitations, some of which are taken from a jMock designed way back when in the 1920s (it looked suspiciously like a sewing machine) and some are simply omitted features or, possibly, misinterpretations from jMock. Combined, these limitations are destined to frustrate anyone who tries using mock objects, especially if they come from anywhere other than PHP. Try getting a Ruby developer to switch from flexmock or mocha to PHPUnit and they will go insane (which may not be a bad thing depending ) or run screaming to DHH who is the father of Ruby (if you believe the hype).

The first immediate flaw with PHPUnit Mock Objects is the API. The nested API is reminiscent of jMock, but without Java's shorter forms it's clunky and unintuitive. The second obvious flaw, something most likely to have frustrated you in the past as it does everyone, is that managing multiple method expectations isn't possible. Rather than queue up, they seem to overwrite previously made similar expectations. The third isn't actually a terrible flaw, but more of an inconvenience, whereby mocking classes still needs some input to deal with constructors and object instantiation. This leads to some gymnastics when instantiating a new mock. There are many other subtle "flaws" you probably never even thought of unless coming from another programming language.

You have to admit, that the flaws don't sound all that terrible in summary form, and that is why they are so bad. Their sheer simplicity hides their impact. In practice, however, they collide with our sense of object manipulation (complex, chainable, hierarchical and conditional). And this is why, ultimately, you can't really mock with PHPUnit mock objects. Mock objects simulate every possible behaviour of real objects - if you can't program a mock object to deal with the common intricacies of object interactions then you cannot use mock objects. Period. You can't put up with expectation overwriting, a kludgy API, overly complex mock setup, and the lack of features you never even knew existed, and expect to find mock objects fascinating. Instead, you'll just end up frustrated with a looming sense that mock objects are not worth using. Then you stop trying. Or you resort to Stackoverflow, and then stop trying. Either way, in the end, you stop.

The proof is everywhere... "grep -r getMock ." Que X-Files theme music.

Fresh off the frying pan, PHPUnit isn't exactly overly damaged . It's an excellent unit testing framework with excellent support with one fixable blemish. So, how do we fix Mock Objects?

Of course, there has to some self-promoting agenda to this blog post! About two years ago I prototyped a mock object framework called Mockery. Since it was a prototype, and a learning experience, it turned out to be a rusty implementation with a few decidedly ugly flaws. Nevertheless, I maintained a more sane version which I've been using privately all this time. It's fair to say, though I'm biased and you don't need to believe me, that it made me a better developer. I found it easier to write beautiful code (like mathmaticians, programmers are driven by aesthetics). Having a good mock object solution does have its benefits, particularly when you're trying to apply Test-Driven Design. And it works, so long as you stay away from any open source project (see later).

With the recent push out of a rewritten version of MutateMe (a Mutation Testing framework - another practice long overdue for a PHP outing), I gave the same treatment to Mockery. Mockery is now rewritten for PHP 5.3, and I'm busy rounding out its mocking ability (specifically creating class-type obedient mocks - the rest is done) before announcing a beta release.

I'll talk specifically about Mockery in another blog post later, but you can find the code and the exhaustive README over at http://github.com/padraic/mockery. It has a few known kinks, but another day or two with some free hours will finish it.

Back to why I want to unleash Mockery against PHP's Ass Sucking Mocks, it's because it will make life easier. For me. I'm selfish that way but, perhaps, it'll also make life easier for you. Open source is all about selfishly fulfilling our own needs before unselfishly slapping it with a New BSD sticker. My selfishness serves the community . So does yours!

Removing all the mumbo-jumbo, I need to get Mockery "out there". Out there is a weird place, a world filled with thousands upon thousands of unit test suites whose developers think Mock Objects were born in PHPUnit and will die there. In the Zend Framework, for example, I can't use mock objects (as neither can anyone else). Sure, I throw in the odd Stub (as with Zend_Feed_Pubsubhubbub). By the way, that grep list? Remove an index finger for Zend_Feed_Pubsubhubbub's mocks - they're stubs. We'll count them as a single digit deduction. Point is that trying to use Mockery mocks in ZF would kick up a massive fuss as unit tests the world over died from a missing dependency.

Why can't I use mocks? I want to use mocks! But first, I need all those unit tests suites to accept a dependency on a mock object library like Mockery (or another, assuming there is another outside of Mockery and yaymock (yes, there's a yaymock effort)). Until that happens, mock objects in PHP are a lost cause. So this opening post (I'll introduce Mockery later) is a call out to fellow Mocketeers (catchy?) and future Mocketeers (everyone reading this has been drafted ). Get thinking about Mock Objects! If we think as one, our psychic oneness might result in gaining a (i.e. my) common use mock object library that we are more free to use in all those PHPUnit test suites sweeping across Github.

And I said "removing all mumbo-jumbo"...

On a serious note, we need mock objects. If Mockery actually turns out okay (fingers crossed - detailed look tomorrow), I could use the help in spreading the word about its awesomeness.

Some recent attention in the PHP framework community has been focused on the recent publication of Symfony 2 Preview benchmarks showing that Symfony 2 outperforms Zend Framework by a factor of 3.5. It also outperforms every other benchmarked framework. This reminded me of the earlier Yii Framework benchmarks which allegedly proved that Yii outperforms Zend Framework by a factor of 7. At this point, the spirit of Glenn Beck would have me demonstrate that 3.5 multiplied by 2 (the number of eyes in Barrack Obama's skull) equals 7, thus proving the existence of a Liberal conspiracy led by a closet Muslim. That's probably bullshit though.

My fellow Zend Frameworkers, we cannot allow this to stand. We must put paid to these wild claims and prove, once and for all, that Zend Framework is the fastest. To present this undeniable truth rooted in solid statistics and Paul M. Jones (aka Benchmark God) inspired techniques, I have created the benchmark of benchmarks. Well, to be honest, I only really edited another benchmark. But still, it will prove Zend Framework is faster than everything else out there.

Before everyone gets too excited, the Symfony 2 Preview performance is quite impressive and it's consistent in my own rerun of the original benchmark. A lot of effort has obviously gone into speeding things up and it shows. My benchmark is not about busting some untruth, but about demonstrating how it is possible to outrace Symfony 2 and Yii with the Zend Framework. I'll dissect the methods after the results are reported .

To ensure the unchallenged fairness of my benchmark, I will publish its entire source code to Github. Unless there are any more sarcastic Irishmen out there, that should be relatively safe. I will also follow in the path of the Almighty Paul M. Jones, in whose shadow all PHP framework benchmarkers will find peace, flowing waters, milk and honey and, if not too much trouble, the extermination of all console noobs who insist that PC gaming is dying. Is that too much to ask, Paul? A little biblical gnashing of teeth would do them good. Paul (may he live forever) promotes using a standardised benchmarking platform, so I will be using an AWS instance reflecting Fabien Potencier's except it will be an m1.large instance instead (the other seeing a few issues in testing) and include some baseline HTML to make sure siege or ab are not misbehaving.

With all these preparations, the stage is set to demolish Symfony 2 and Yii and run them into the ground. Prior to running the benchmark, it's suggested to play the Battlestar Galactica theme tune in the background to set the right mood. Just think of Fabien Potencier dressed up as a Cylon (if only my Photoshop skills were better...sigh). It fits in well with the siege feedback. You can try some HipHop, but those Facebook guys must have been smoking something illegal because it had no impact during the Zend Framework runs. I also experimented with Metallica (surely that would work!) but no...nothing. Quite disappointing.

For those with attention deficit disorder, here are the results:

framework                |      rel |      avg |        1 |        2 |        3 |        4 |        5

------------------------ | -------- | -------- | -------- | -------- | -------- | -------- | --------

baseline-html            |   1.1972 |  5702.79 |  5527.29 |  5775.74 |  5779.59 |  5671.24 |  5760.10

baseline-php             |   1.0000 |  4763.50 |  4718.24 |  4751.51 |  4760.75 |  4763.68 |  4823.32

zend-1.10                |   0.1596 |   760.45 |   766.90 |   764.70 |   758.62 |   752.04 |   759.98

symfony-2.0.0alpha1      |   0.1366 |   650.61 |   641.98 |   655.41 |   653.86 |   656.68 |   645.12

solar-1.0.0beta3         |   0.1131 |   538.86 |   539.76 |   536.95 |   540.63 |   540.10 |   536.87

yii-1.1.1                |   0.0821 |   390.87 |   401.90 |   392.59 |   386.18 |   395.98 |   377.72

symfony-1.4.2            |   0.0441 |   210.22 |   211.20 |   209.78 |   210.72 |   210.49 |   208.92

cakephp-1.2.6            |   0.0406 |   193.56 |   193.84 |   193.35 |   193.27 |   192.57 |   194.75

As you can clearly see, Zend Framework is faster than all other frameworks. More enlightening, we beat both Symfony 2 and Yii. Lithium and Flow were omitted from the original benchmark because besides them being non-functional when forked from Fabien's benchmark (for whatever reason), they are both fairly young frameworks (Flow3 is alpha). Note: Lithium works fine outside of the benchmark (I've been working with it) and it is fairly fast.

Zend Framework wins. You can go now.

Still here? I suppose you're trying to figure out how I manipulated...er...improved the benchmark to reflect my reality of Zend Framework being superior at all levels. Well, I'm not sure I should tell you. It's hard enough to create and run these benchmarks without having to explain how to fuck them over so they show whatever you need them to. Alright then, I'll 'fess up.

To create a positive benchmark, you need to understand that all frameworks were born as festering piles of unoptimised stinking crap. They were all born bad and get worse with age. This sounds quite sad, but actually it's an inevitable compromise between performance and features. It's also a compromise between performance and ease-of-use. So you see, performance is unfairly faced by two opponents: features and ease-of-use. All performance is sacrificed in the name of serving the needs of rapid development, flexibility, prototyping, and making your source code look prettier than the other guy's. As if.

What happens if you move away from the enemies of performance and do some propping up behind the scenes? You get...wait for it...oodles of extra performance!

This is partly how both Symfony 2 and Yii manage to outperform Zend Framework. But hey, we can do it too!

To ascertain the perfect method of "propping up" to get the Zend Framework into speed demon territory without it costing me dozens of hours of in-depth analysis, I created four variants to play with: All-In, Optimised, More-Optimised and What-The-Fuck-Optimised. Here's the four final alternatives benchmarked against themselves (same run as the previous results).

framework                |      rel |      avg |        1 |        2 |        3 |        4 |        5

------------------------ | -------- | -------- | -------- | -------- | -------- | -------- | --------

baseline-html            |   1.1972 |  5702.79 |  5527.29 |  5775.74 |  5779.59 |  5671.24 |  5760.10

baseline-php             |   1.0000 |  4763.50 |  4718.24 |  4751.51 |  4760.75 |  4763.68 |  4823.32

zend-1.10-wtfoptimised   |   0.1748 |   832.63 |   830.64 |   837.20 |   836.20 |   824.90 |   834.23

zend-1.10-moreoptimised  |   0.1596 |   760.45 |   766.90 |   764.70 |   758.62 |   752.04 |   759.98

zend-1.10-optimised      |   0.0586 |   279.25 |   280.71 |   279.30 |   279.62 |   276.51 |   280.10

zend-1.10                |   0.0292 |   138.89 |   138.43 |   137.56 |   139.96 |   139.93 |   138.56

Pretty illuminating . Yes, the What-The-Fuck-Optimised variant is faster than the All-In variant by a factor of 6. The More-Optimised variant (fairer for reasons explained below) is 5.5 times faster than the base variant. In fact, the improvements are fairly smooth with each level of optimisation, at least doubling each time.

So just how does this miraculous optimisation work to such a degree that we can beat Symfony 2 and Yii at their speed games? There are four solid steps to being a speed demon. I only implemented these in the More-Optimised variant which was enough to destroy Symfony 2 (as configured) and Yii in the benchmarks. The Optimised version simply discarded some optional ease-of-use features. What-The-Fuck-Optimised is identical to More-Optimised except it exits in the middle of a controller (we only had to echo Hello, right? ).

1. Don't use Zend_Application. While Zend_App is great for creating consistent complex bootstraps within a standardised structure, it doesn't come without a significant performance hit to baseline performance. A more direct bootstrap (typical of ZF until Zend_App arrived) is far faster and can also be done without configuration files.

2. Skip using the ViewRenderer plugin. Without the ViewRenderer, you need to manually configure Zend_View and add render() calls to Controllers. This is actually very simple to do and is fairly fast - fast was never really part of the ViewRenderer's genetics.

3. Use autoloading. Strip require_once calls from the framework library so unneeded files are ignored. Replace uses of Zend_Loader_Autoloader with a not-so-crazy autoloader function. In fact, pray Zend_Loader is never used - it does a lot of file ops that, to date, have never been explained to me as having any value.

4. Preload everything (Symfony 2 Preview does!). It buys you some performance cookies and equalises the speed baseline. Using a simple preload script is not that hard.

With these four techniques, Symfony 2 and Yii are left in the dust.

It's time to ask the question on everybody's minds: what does it all mean? Give up. There is no meaning in benchmarks. They are designed to compare the relative performance of frameworks with different goals, design philosophies, development practices, and features. Might as well flip a coin, roll some dice or perhaps examine the entrails of a sacrificed goat for answers. The goat is your best bet. If all else fails at least you can have a nice goat-stew dinner while you think things over some more.

Benchmarks. Useless. Final words?

Know your framework! All this benchmarking nonsense does little good unless it's plastered with disclaimers. Symfony's preloading is an obvious example mainly because it's the only framework in Fabien's benchmark (that I know of) using it. Does it increase base performance out of the box? Yes. Can every other framework use it with little effort? Yes. So what's the point? Should everything else preload by default? Maybe, who can tell. Most developers simply don't work in scenarios where such a performance boost would make a difference. And if they do, it's easy to make the necessary changes.

What's interesting is that I just fiddled with the Zend Framework. You could probably find ways of making every other framework in there run a hell of a lot faster than currently presented. It's just a bit sweeter for Zend Framework since we're the self-proclaimed use-at-will framework. And its really fast when developers need it to be.