PHP, Zend Framework and Other Crazy Stuff
Archive for June, 2011
Zend Framework Contributors Mailing-List Summary; Edition #1 (June 2011)
Jun 27th
What’s this nonsense then? Well, a few weeks ago I shot myself in the foot (I was aiming for the cat who spilled coffee all over my desk) and before my sanity returned to normal, I found myself hoodwinked on IRC into writing up weekly summaries of what is discussed in Zend Framework land. The moral of the story is that the attempted murder of any ungrateful coffee-spilling animals sharing your home never ends well.
Let’s see how good a verbose meandering writer can be at summarising things. I decided to refer to myself by name throughout to avoid confusion.
Discussion Time: In ZF2, where do things go?
Ralph Schindler sprang this topic on us back in April and it has stubbornly continued on ever since. Ralph’s initial question boiled down to where should we put resource files, i.e. files utilised by PHP class files but not written in PHP themselves. The two options presented were to store them relative to the class files inside the library directory or store them in a completely separate parallel directory specifically for resources.
Opinions varied quite a bit and Mike Willbanks opined that we should follow PEAR standards rather doing our own thing and seek to limit include_path performance issues. Matthew Weier O’Phinney noted that include_path performance concerns should be minimal using ZF2′s autoloader solution which he has researched, and the intention was to use PEAR or Pyrus. Pádraic Brady (I know that name from somewhere!) chipped in that any decision ought to be made independent of the packaging used, referencing possible weaknesses in how PEAR handles installation, unit tests and documentation viewing. Ralph responded to clarify possible workings of a separate resource directly using simple constants and allowing users to selectively override this noting the existence of the Assetic project (used by Symfony 2). Kevin McArthur added a vote to avoiding PEAR citing the need for multi-version installation support in a final solution and suggested the PHAR format for consideration.
Short version: Someone will make a choice…eventually .
How to Package ZF2
Pádraic spawned a new thread from the above earlier topic outlining the options available for packaging source code including PEAR, Pyrus, Git and a Symfony related project (now known as Composer). He also reiterated concerns previously raised regarding PEAR/Pyrus. There followed a side discussion on how individuals were actually deploying applications and managing QA and patches. Matthew raised an objection to the concept of centralised multi-version installs of Zend Framework citing alternative solutions such as deploying applications already containing the Zend Framework version required as easing maintenance and uncertainty. He also asked Kevin McArthur to clarify the use of PHAR. Kevin responded to offer an answer as to why centralised multi-installs were useful citing benefits in minimising the APC cache memory (centralised libraries offering minimal chances of having identical copies being cached), and offering an example bootstrap script for such an architecture to manage version selection.
Matthew also posted responses to points brought up in respect of Pyrus noting, among other things, that it was closer to stable than suspected, that centralised multi-versioning was possibly not as popular as believed, that git support may be possible to add independently, and that XML package definitions had a number of advantages. The debate over centralised multi-version installations of Zend Frameworks continues for a large number of emails without resolution (too much to summarise other than to note each side is firmly divided by the benefits their particular approach and multi-versioning proponents seem more numerous than expected). No concensus was reached over the method of installation with the best summarisations of the respective opinions being emailed in by Matthew and Kevin McArthur. Pádraic chimed in briefly to prompt adoption of PEAR in preference to Pyrus on the basis PEAR is already widely adopted, understood and is easily manipulated at present. This was seconded but there remained a lack of concensus. The topic ends with a suggestive note that adoption of Pyrus may be accepted recognising the absence of another realistic solution at the current time.
Short version: ZF2 may be distributed using Pyrus. Additional needs beyond that may be proposed to PEAR for Pyrus or via another tool. It’s clear Pyrus will be crop up again in a future discussion.
ZF2′s View: Some thoughts for discussion
Pádraic Brady dropped an email offering his thoughts on the direction of ZF2′s View which hadn’t seen huge feedback on the Wiki. The short version was that Zend_View was a God Class, View Helpers were confusing, integration needed improvement and templates needed additional control over layouts/placeholders. He suggested a couple of steps including elimination of the ViewRenderer helper, the replacement of View Helpers with a Controller oriented entity referred to as a “Cell”, ensuring the base template of a View had greater control over the rendering process and reiterated previously agreed changes. Marc Bennewitz added several additional concerns and posted a discussion he had with Matthew on the Zend\View\Variables class. Matthew responded with a number of points including keeping the barrier to entry low, recognising all Views are not HTML, and other areas for consideration. Nice to see everything in one place for discussion.
Short version: Not much in the way of disagreement. Seems like a topic that just needs sufficient code for someone to run off and write some proposals.
Proposal: Don’t implement BC requirement until ZF 2.1
Rob Allen emailed a proposal suggesting that backwards compatibility be deferred as a requirement until ZF 2.1. His reasoning focuses on the experience with ZF 1.0 where the frozen compatibility hurt ZF 1.x more than it helped. The proposal was quickly seconded by Ryan Mauger, Anthony Shireman, Rob Zienert (on condition of communicating this clearly to users), and H. Hatfield. Opposing views were aired by Till Klampaeckel on the grounds of keeping migrations between versions simpler. Tomáš Fejfar commented on this being a psychological proposal to increase adoption and raise feedback before the API is finally frozen. Matthew Weier O’Phinney noted his agreement that bigger features were required to increase early adoption.
Bradley Holt took the opportunity to propose alternative version/release strategies setting the context for the rest of the debate to date. His two points were to a) utilise an odd/even version system where odd numbered minor releases were considered betas and even numbered considered stable, similar to how the Apache HTTP server does things, and b) increase the pace of major releases to shorten the period between allowable compatibility breaks and speed up rolling out such improvements. The debate suggested Rob Allen would agree to faster major releases.
Short version: Implementing BC may be necessary. Might be better to shorten the release cycle and roll out compatibility breaking changes more regularly.
Proposal: Shorter Release Cycle for Major Versions
On the back of the previous topic, Bradley Holt elaborated on a proposal for shortening the release cycle for major versions. Pádraic Brady responded in agreement noting that by the time ZF2 was released, there was a possibility that PHP 5.4 with potentially advantageous features would be well on the way to a 2012 release. Based on this he suggested that ZF3 development could be executed quickly with a release date no later than end of 2012 (i.e. 18 months away) with a maximum allowed period of 2 years. Kevin McArthur inquired into a reasonable minimum period before major releases but this seems to the number needing more discussion. There has been no input from the Zend guys to date so this remains up in the air.
Short version: We want ZF3 relatively quickly and not in 4-5 years time.
Encouraging Usage of ZF 2.0 Beta
Another discussion opener from Bradley Holt. Bradley suggested an extended beta period, a communication campaign, treating all betas as regular GA releases and highlight applications build on ZF2 to encourage uptake. Kevin McArthur reiterated the need to maintain current versioning and noted his agreement to shortening the major version release cycle and having an extended alpha/beta period. Alessandro Pellizzari emailed in his thoughts from the perspective of a user and the difficulties that currently exist with checking the status of any one ZF2 component. Derek Miranda voiced his agreement with Alessandro’s thoughts.
Short version: Maybe we need a beta first?
New dev snapshot released
On the back of the work going into Zend\DI, Matthew announced the release of a new development snapshot for testing and feedback. Ralph Schindler subsequently posted links to Zend\DI examples. Feedback is ongoing. Anyone is free to check it out and offer some opinion!
Short version: Isn’t that short enough?
For those of you wondering where to go and track the inner thoughts of the Zend Framework developers, you can join us on the zf-contributors mailing list (available on Nabble here) or on IRC channel #zftalk.dev on Freenode.net. Until next time, remember, coffee + cat = bad.
How Would You Engineer A PEAR2/Pyrus Distribution Architecture?
Jun 20th
I was recently accused on the Zend Framework Contributors mailing list of having “strong feelings” towards Pyrus (i.e. the PEAR Group’s Installer/Packager for PEAR2) and not in a positive way. It’s a fair description. PEAR is, putting it lightly, a very old architecture which makes it very resistant to change. With the idea of PEAR2 and Pyrus, I had hoped to see a renewal - the advancement of a PEAR architecture for the 21st Century. Instead, and this is just my opinion, PEAR2/Pyrus were a relatively simple iteration on a very old theme.
A Ranting We Shall Go
Now, I may be biased since I gave up on PEAR becoming PHP’s core distribution mechanism after I found myself using alternative strategies for hosting and deployment. This is not to say PEAR is not useful for everyone. It is - just not in my specific case when developing/testing/deploying applications. It still remains a good distribution means regardless by virtue of its ubiquitous installation with PHP.
I surprised even myself, however, with my vehement outcry over the idea of adopting Pyrus as Zend Framework 2′s package distribution method, lambasting both it and the PEAR concept of distribution in equal measures while piling up questions on Pyrus’ status (currently released in alpha) and suitability in the near term. That thread showed a fairly divided sentiment. Once I jokingly threatened to mow down my zombified colleagues with a minigun, I figured it was time to go forth and rant (miniguns are too expensive for these recessionary times).
If the PEAR ecosystem has a failing, it is one of staggered evolution. Over time it has picked up additional features tacked on top of a base model. The classic example is the use of Channels (to support multiple repositories) that has more recently prompted calls for the use of a Channel Aggregator to avoid the use cost in locally managing a channel registry or even hosting a Channel. This is the way of many PEAR features. They each do something incredibly useful but do it in a way that has many developers looking for a better approach - usually to discover the better approach requires breaking compatibility.
My vehemence in the afore mentioned mailing list was down to a simple case of disappointment. We all deal with PEAR because we have it, we know it, and have done so for years. Seeing PEAR2 and Pyrus take the incremental improvement route without apparently doing anything to change the core experience seemed…pointless. It improved a lot of what PEAR already did without actually doing very much different. All the same advantages, disadvantages, features and lack thereof were present and accounted for with a handful of nice headline changes (e.g. we now have package signing capability). What exactly was the purpose of rewriting the entire toolchain if not to seize the opportunity to answer the accusations of those who doubt PEAR is even relevant these days - by making it the single most relevant development in PHP today?
One Possible Path Forward
Since this is a brain dump post, as much to gather my own throughts in one place as anything else, feel free to call me bat shit crazy. There are days even I think that. Below I’ve raised what I perceive as problems in the PEAR/Pyrus system, obviously from a personal perspective, and possible solutions under the categories of Packaging, Distribution, Installation and Usage. I’ve tried to avoid getting into technical details - broad strokes will suffice for now. For your sanity, only the Packaging and Distribution areas are presented today. I will add a similar post for Installation and Usage later in the week. First one to mention “TL:DR” gets a minigun round to the head (will have to make do with throwing it at you until I can scrape more cash together for the hardware). To avoid any confusion, I use the terms PEAR and Pyrus to refer to the entire workflow from package generation to end usage for each respectively.
Packaging
The packaging of source code for PEAR is performed using the PEAR/Pyrus Installer coupled with a Package Definition (i.e. package.xml) to create a distributable archive file. Pyrus utilises a slightly more friendly Package Definition by also allowing for some elements of the definition to be defined in files other than package.xml (e.g. for setting up a changelog file or version numbers). The basic goal of this Package Definition is to have at least one XML file which tells PEAR/Pyrus which files to package, while role a file has (code/docs/tests), where each file goes in a relative filesystem, optionally the file’s MD5 hash, and a set of metadata like the package name, changelog, version, dependencies, etc. Using Pyrus offers the additional feature of being able to cryptographically sign packages, use a larger number of archive formats including PHAR, and bundle certain package dependencies internally.
Problems:
The main problem with the current Package Definition is that it often must be generated by a separate tool since it’s XML (it’s that thing everyone used before discovering YAML/JSON), and must explicitly list every file and piece of data within that format (with the exception of Pyrus which allows specifically formatted files to carry version and changelog information among other nuggets) optionally with each file’s digest hash. Even the Pyrus improvements still require specific files using specific formatted text and/or file names. Using XML just ends up imposing extra work to maintain package details unless you are lucky enough to have a small stable enough package.xml that it can be manually maintained rather then persistently needing generation. A minor aesthetic detail is that XML is harder to read.
Secondly, packages are therefore bound to their archiving restraints. Since package.xml generation is tied to a secondary process, installing from source code may not be feasible whether performed on a local git clone or similarly automated from a remote source where the remote package.xml may well be out of sync with the actual source code or where it may not even exist.
Possible Solutions:
The one solution that keeps occuring to me is to simply make a Package Definition programmable, i.e. a small consumable low-maintenance PHP script. Using native PHP, one can create ether a generic array, or a newfangled closure, which can be executed through PHP to populate all the necessary data for a Package Definition for consumption by a package installer.
Since I’ve dabbled a bit, here’s what such a Package Definition could look like:
<?php
$package = function ($s) {
$s->name = 'Overlord';
$s->authors = 'Padraic Brady, Sauron[[email protected]]';
$s->version = '0.0.1-dev';
$s->api_version = '0.0.1-dev';
$s->summary = 'Monitoring library for Hobbit Detector 1.0';
$s->description = file_get_contents(__DIR__ . '/description.txt');
$s->homepage = 'http://en.wikipedia.org/wiki/Sauron';
$s->changelog = file_get_contents(__DIR__ . '/changelog.txt');
$s->files['php'][] = 'library/**/*.php';
$s->files['tests'][] = 'tests/**/*.*';
$s->files['ignore'][] = '*.project';
$s->files['bin'][] = 'scripts/overlord.bat';
$s->include_path = 'Overlord/Monitor/';
$s->dependencies[] = 'PHP[>=5.3.1]';
$s->dependencies[] = 'Pear[>=1.6.5]';
$s->dependencies[] = 'MutateMe[0.5.0]';
$s->dependencies[] = 'ext/runkit';
$s->optional_dependencies[] = 'ext/eyeofsauron';
$s->license = 'New BSD';
};
I’ll assume PHP 5.4 will have some sort of short array notation to cut down the array size. Well, let’s hope so . Would be nice to reduce the line count more. Yes, I did indeed borrow the idea from elsewhere
.
This has a few advantages. No XML to maintain. No need to keep an XML Package Definition synced up for every file change in a VCS. No need for secondary XML generation tools or build tool plugins. Supports downloading files from remote herarchical sources and not just archives (including any VCS source). Developers are already used to versioning build scripts from tools like Phing (just not the end products which are usually ignored whereas package.xml is not). Being plain old PHP, it can be just as complex or as minimal as you want and anyone with basic PHP knowledge can write one.
One can still generate signable archive files using this approach - the point is to increase the kind of installation sources that can be used rather than replace existing ones. In place of signable packages, for those requiring the security, other package files could be limited to download over HTTPS. For example, Github offers git read-only access via HTTPS for all repositories as standard.
Distribution
In order to distribute source code using PEAR/Pyrus, you need to make use of either a PEAR Channel or a standalone archive download (i.e. a downloadable tarball). A Channel is basically a whole bunch of XML files served up for access as a REST API. Using a Channel, you can upload packages to the Channel host, update the XML files, and publicise your Channel URI so users can discover your Channel and install your packages.
Problems:
PEAR (PHP Extension and Application Repository) was originally founded to serve as a central package distribution channel. For various real and imagined reasons, the concept of a central repository did not succeed in PHP and instead developers insisted on using alternative means. This was aggravated even further by the arrival of frameworks like Zend Framework offering discrete components not originally served over a PEAR Channel at all. PEAR Channels were introduced to allow anyone host their own distinct PEAR Channel as one of those means.
The PEAR Installer has only ever shipped with the main PEAR Channels pre-registered. All other Channels needed to be manually located before use - usually by referring to the packaged library’s documentation. Since all Channels are independent entities, there is no global lookup point for querying package details, dependencies and availability. There is also no scope for true package name uniqueness (technically this is accomplished by requiring all packages (except core-PEAR ones) are prefixed with a Channel alias term, e.g. mychannel/MyPackage).
The generation of the REST API, which was the backbone of a Channel, was also complex (the release of Pirum by Fabien Potencier has gone a long way towards simplifying this). Obviously, Channels are also tied to the concept of archive packages and cannot operate directly with a VCS like git. There is a workaround possible for Github using Github Pages to host the REST API.
As alluded to, the REST API is itself a complex graph of XML files that requires a generation tool to manage initial setup and package updates.
Possible Solutions:
The best concept to gain early traction was that of a Channel Aggregator expressed by Stuart Herbert. Sadly, I haven’t seen much more action on that front. In commenting on that idea, I considered it a move towards a decentralised distributed Channel mechanism (mouthful of gibberish, I know!). Here’s a couple of thoughts on how this could work:
The players would include a Package Authority, a Channel Aggregator (any number of them), and Channels (optional).
The Package Authority would be a centralised location basically for reserving package names and ensuring there is a point of reference and authority to prevent package name duplication and to manage ownership of such. It’s possible this could also be developed with additional purposes but let’s keep it simple. This would help, primarily, in removing the need for Channel prefixes on package names and preventing package name confusion. For security reasons, the Package Authority would associate a package name to a specific URI representing a download source (e.g. a PEAR Channel or Git URI)
Channel Aggregators are the more complex beasts. They may be utilised by Channel operators to distribute Package metadata to end-users on demand. The Aggregator would track available packages at source, their basic details, their available versions, and information on the location of host Channels, version control systems, and Package URIs and so forth. In effect, the Aggregator might well replace Channels for many purposes - and potentially eliminate one more source of work in distributing source code using PEAR/Pyrus.
The ideal scenario here is that any PEAR/Pyrus Installer would pre-register a couple of well-maintained Aggregators saving the users and package distributors the annoyance of dealing with Channels altogether. Hence, we’re back to a core Channel of sorts but with control of package/source hosting decentralised to individual developers. Again, Aggregators could easily repurpose themselves as package hosters if they wish (such as Pearfarm are doing) though this would be entirely optional.
Channels, as suggested, could well be optional. Use an Aggregator instead and register either a package URI, git repository, or anything else so long as it lets you download the package files (and the PHP programmable Package Definition ). Painless hosting? Maybe.
I will point out this would require at least one authentication in the system. You’d need a Package Authority account to allow for reserving a package name and perhaps transferring it between maintainers. The Aggregator may operate without authentication since it acts much like any aggregator based on your source data (and one would hope a few simple crosschecks with the Package Authority to ensure it’s not unwittingly aggregating false data from the hackers . Package/source hosters could ping the Aggregator as a hint to update its date in a more timely manner.
I won’t touch the issue of who gets to reserve the package name “DB”. The Package Authority may need to enforce specific rules against overly generic names on a common sense basis.
I think that’s enough for a Monday read (you’ll all need enough brain capacity to finish out the week!). Feedback is, as usual, welcome. If anyone has a pre-existing solution or one in planning along these or similar lines, drop a comment!