PHP Framework Benchmarks: Entertaining But Ultimately Useless

Some recent attention in the PHP framework community has been focused on the recent publication of Symfony 2 Preview benchmarks showing that Symfony 2 outperforms Zend Framework by a factor of 3.5. It also outperforms every other benchmarked framework. This reminded me of the earlier Yii Framework benchmarks which allegedly proved that Yii outperforms Zend Framework by a factor of 7. At this point, the spirit of Glenn Beck would have me demonstrate that 3.5 multiplied by 2 (the number of eyes in Barrack Obama's skull) equals 7, thus proving the existence of a Liberal conspiracy led by a closet Muslim. That's probably bullshit though.

My fellow Zend Frameworkers, we cannot allow this to stand. We must put paid to these wild claims and prove, once and for all, that Zend Framework is the fastest. To present this undeniable truth rooted in solid statistics and Paul M. Jones (aka Benchmark God) inspired techniques, I have created the benchmark of benchmarks. Well, to be honest, I only really edited another benchmark. But still, it will prove Zend Framework is faster than everything else out there.

Before everyone gets too excited, the Symfony 2 Preview performance is quite impressive and it's consistent in my own rerun of the original benchmark. A lot of effort has obviously gone into speeding things up and it shows. My benchmark is not about busting some untruth, but about demonstrating how it is possible to outrace Symfony 2 and Yii with the Zend Framework. I'll dissect the methods after the results are reported .

To ensure the unchallenged fairness of my benchmark, I will publish its entire source code to Github. Unless there are any more sarcastic Irishmen out there, that should be relatively safe. I will also follow in the path of the Almighty Paul M. Jones, in whose shadow all PHP framework benchmarkers will find peace, flowing waters, milk and honey and, if not too much trouble, the extermination of all console noobs who insist that PC gaming is dying. Is that too much to ask, Paul? A little biblical gnashing of teeth would do them good. Paul (may he live forever) promotes using a standardised benchmarking platform, so I will be using an AWS instance reflecting Fabien Potencier's except it will be an m1.large instance instead (the other seeing a few issues in testing) and include some baseline HTML to make sure siege or ab are not misbehaving.

With all these preparations, the stage is set to demolish Symfony 2 and Yii and run them into the ground. Prior to running the benchmark, it's suggested to play the Battlestar Galactica theme tune in the background to set the right mood. Just think of Fabien Potencier dressed up as a Cylon (if only my Photoshop skills were better...sigh). It fits in well with the siege feedback. You can try some HipHop, but those Facebook guys must have been smoking something illegal because it had no impact during the Zend Framework runs. I also experimented with Metallica (surely that would work!) but no...nothing. Quite disappointing.

For those with attention deficit disorder, here are the results:

framework                |      rel |      avg |        1 |        2 |        3 |        4 |        5

------------------------ | -------- | -------- | -------- | -------- | -------- | -------- | --------

baseline-html            |   1.1972 |  5702.79 |  5527.29 |  5775.74 |  5779.59 |  5671.24 |  5760.10

baseline-php             |   1.0000 |  4763.50 |  4718.24 |  4751.51 |  4760.75 |  4763.68 |  4823.32

zend-1.10                |   0.1596 |   760.45 |   766.90 |   764.70 |   758.62 |   752.04 |   759.98

symfony-2.0.0alpha1      |   0.1366 |   650.61 |   641.98 |   655.41 |   653.86 |   656.68 |   645.12

solar-1.0.0beta3         |   0.1131 |   538.86 |   539.76 |   536.95 |   540.63 |   540.10 |   536.87

yii-1.1.1                |   0.0821 |   390.87 |   401.90 |   392.59 |   386.18 |   395.98 |   377.72

symfony-1.4.2            |   0.0441 |   210.22 |   211.20 |   209.78 |   210.72 |   210.49 |   208.92

cakephp-1.2.6            |   0.0406 |   193.56 |   193.84 |   193.35 |   193.27 |   192.57 |   194.75

As you can clearly see, Zend Framework is faster than all other frameworks. More enlightening, we beat both Symfony 2 and Yii. Lithium and Flow were omitted from the original benchmark because besides them being non-functional when forked from Fabien's benchmark (for whatever reason), they are both fairly young frameworks (Flow3 is alpha). Note: Lithium works fine outside of the benchmark (I've been working with it) and it is fairly fast.

Zend Framework wins. You can go now.

Still here? I suppose you're trying to figure out how I manipulated...er...improved the benchmark to reflect my reality of Zend Framework being superior at all levels. Well, I'm not sure I should tell you. It's hard enough to create and run these benchmarks without having to explain how to fuck them over so they show whatever you need them to. Alright then, I'll 'fess up.

To create a positive benchmark, you need to understand that all frameworks were born as festering piles of unoptimised stinking crap. They were all born bad and get worse with age. This sounds quite sad, but actually it's an inevitable compromise between performance and features. It's also a compromise between performance and ease-of-use. So you see, performance is unfairly faced by two opponents: features and ease-of-use. All performance is sacrificed in the name of serving the needs of rapid development, flexibility, prototyping, and making your source code look prettier than the other guy's. As if.

What happens if you move away from the enemies of performance and do some propping up behind the scenes? You get...wait for it...oodles of extra performance!

This is partly how both Symfony 2 and Yii manage to outperform Zend Framework. But hey, we can do it too!

To ascertain the perfect method of "propping up" to get the Zend Framework into speed demon territory without it costing me dozens of hours of in-depth analysis, I created four variants to play with: All-In, Optimised, More-Optimised and What-The-Fuck-Optimised. Here's the four final alternatives benchmarked against themselves (same run as the previous results).

framework                |      rel |      avg |        1 |        2 |        3 |        4 |        5

------------------------ | -------- | -------- | -------- | -------- | -------- | -------- | --------

baseline-html            |   1.1972 |  5702.79 |  5527.29 |  5775.74 |  5779.59 |  5671.24 |  5760.10

baseline-php             |   1.0000 |  4763.50 |  4718.24 |  4751.51 |  4760.75 |  4763.68 |  4823.32

zend-1.10-wtfoptimised   |   0.1748 |   832.63 |   830.64 |   837.20 |   836.20 |   824.90 |   834.23

zend-1.10-moreoptimised  |   0.1596 |   760.45 |   766.90 |   764.70 |   758.62 |   752.04 |   759.98

zend-1.10-optimised      |   0.0586 |   279.25 |   280.71 |   279.30 |   279.62 |   276.51 |   280.10

zend-1.10                |   0.0292 |   138.89 |   138.43 |   137.56 |   139.96 |   139.93 |   138.56

Pretty illuminating . Yes, the What-The-Fuck-Optimised variant is faster than the All-In variant by a factor of 6. The More-Optimised variant (fairer for reasons explained below) is 5.5 times faster than the base variant. In fact, the improvements are fairly smooth with each level of optimisation, at least doubling each time.

So just how does this miraculous optimisation work to such a degree that we can beat Symfony 2 and Yii at their speed games? There are four solid steps to being a speed demon. I only implemented these in the More-Optimised variant which was enough to destroy Symfony 2 (as configured) and Yii in the benchmarks. The Optimised version simply discarded some optional ease-of-use features. What-The-Fuck-Optimised is identical to More-Optimised except it exits in the middle of a controller (we only had to echo Hello, right? ).

1. Don't use Zend_Application. While Zend_App is great for creating consistent complex bootstraps within a standardised structure, it doesn't come without a significant performance hit to baseline performance. A more direct bootstrap (typical of ZF until Zend_App arrived) is far faster and can also be done without configuration files.

2. Skip using the ViewRenderer plugin. Without the ViewRenderer, you need to manually configure Zend_View and add render() calls to Controllers. This is actually very simple to do and is fairly fast - fast was never really part of the ViewRenderer's genetics.

3. Use autoloading. Strip require_once calls from the framework library so unneeded files are ignored. Replace uses of Zend_Loader_Autoloader with a not-so-crazy autoloader function. In fact, pray Zend_Loader is never used - it does a lot of file ops that, to date, have never been explained to me as having any value.

4. Preload everything (Symfony 2 Preview does!). It buys you some performance cookies and equalises the speed baseline. Using a simple preload script is not that hard.

With these four techniques, Symfony 2 and Yii are left in the dust.

It's time to ask the question on everybody's minds: what does it all mean? Give up. There is no meaning in benchmarks. They are designed to compare the relative performance of frameworks with different goals, design philosophies, development practices, and features. Might as well flip a coin, roll some dice or perhaps examine the entrails of a sacrificed goat for answers. The goat is your best bet. If all else fails at least you can have a nice goat-stew dinner while you think things over some more.

Benchmarks. Useless. Final words?

Know your framework! All this benchmarking nonsense does little good unless it's plastered with disclaimers. Symfony's preloading is an obvious example mainly because it's the only framework in Fabien's benchmark (that I know of) using it. Does it increase base performance out of the box? Yes. Can every other framework use it with little effort? Yes. So what's the point? Should everything else preload by default? Maybe, who can tell. Most developers simply don't work in scenarios where such a performance boost would make a difference. And if they do, it's easy to make the necessary changes.

What's interesting is that I just fiddled with the Zend Framework. You could probably find ways of making every other framework in there run a hell of a lot faster than currently presented. It's just a bit sweeter for Zend Framework since we're the self-proclaimed use-at-will framework. And its really fast when developers need it to be.