PHP, Zend Framework and Other Crazy Stuff
Archive for July, 2009
Writing A Simple Twitter Client Using the PHP Zend Framework’s OAuth Library (Zend_Oauth)
Jul 29th
During yesterday, I finally got around to patching and finishing Zend_Oauth’s Consumer implementation for the OAuth Core 1.0 Revision A specification. Once I had it finished, I used it to write a quick and simple interface to post some Tweets on Twitter while I was testing it out. With some documentation and a few extra unit tests, the Consumer implementation should find its way into Zend Framework 1.10…along with the Server implementation I think. In this article I’ll explore how to writea quick Twitter client so you can post tweets (those short messages of less than 140 characters) once authorised across the OAuth protocol.
You can download all the necessary files (just be sure to edit them as described) or pull them from git from: http://github.com/padraic/Tweet-Lite/tree/master
What is OAuth?
If you’re not aware of what OAuth is, the OAuth specification puts it this way:
The OAuth protocol enables websites or applications (Consumers) to access Protected Resources from a web service (Service Provider) via an API, without requiring Users to disclose their Service Provider credentials to the Consumers. More generally, OAuth creates a freely-implementable and generic methodology for API authentication.
In other words, it’s a means of allowing websites to access your data on other services via a service API, like Twitter’s API or Google Gdata, without actually providing those websites with your username and password. Instead, OAuth allows you to authorise such websites to access your data so that they don’t need your username or password - they just use an Access Token supplied by your service provider - and you can easily deauthorise them if desired. The benefit is immediately obvious - your username and password are not shared or handed out to potentially untrustworthy sites. The glut of services using Twitter are a prime example - until recently they all needed your Twitter username and password and honestly, how would you know they wouldn’t misuse that? Because they said so? OAuth eliminates this problem.
The protocol works like this. The website (consumer) that wants to access your data from a service provider, contacts the provider using HTTP to retrieve a Unauthorised Request Token. The consumer will then redirect you, the user, back to your service provider so you can authorise the consumer’s access. The redirect URL will contain the Unauthorised Request Token as a parameter. If you approve the access, you are redirected back to the original website with a verification code attached to the URL. The website now knows you approved its access, so it contacts the service provider, including both the newly approved Request Token (once again) and the verification code in the URL. The response to this should be a fully authorised Access Token (associated with the User) which the consumer can use in all future requests when accessing your data (until either it times out or you deauthorise the access). The Request Access token can be discarded now - in OAuth parlance you exchanged an unauthorised Request Token for an authorised Access Token.
Preparations Are Always Inevitable!
With this understanding in hand, let’s get to writing this small Twitter client as an example! You will need to download the Zend Framework (whether the latest release or via subversion). You will also need to download/checkout the Zend Framework Incubator since Zend_Oauth is not yet part of the main trunk. Once you have stored both somewhere, make a note of the paths to their “library” directories so you can add them to the PHP include_path later.
On the Twitter side there are three steps.
First, get a Twitter account! Hopefully you already have one and are following @padraicb (i.e. me!).
Second, you need to configure your operating system for a new local domain. Under Linux, this is done by editing /etc/hosts. You’ll need root privileges here so use “sudo myeditor /etc/hosts”. Under Windows, the same file is located at C:\Windows\System32\drivers\etc\hosts and will require Administrator privileges to edit. Adding a local domain is dead simple, and it’s needed to use Twitter’s API on a local machine - Twitter refuse all requests from localhost or 127.0.0.1 but they won’t filter out a local domain name since it’s unfeasable and would make a lot of application developers extremely crazy. Edit the file to include a new entry like:
127.0.0.1 mytwitterclient.tld
Once saved, your browser should immediately respond to any address in the form of http://mytwitterclient.tld by attempting to call localhost/127.0.0.1 on your system. If you have a default web document root configured with a web server running, this is where the request will be mapped to and where you should store any files. If you are really troublesome and can’t take how easy that was, you can run off to play with Virtual Host configurations to use a different path.
Third! OAuth providers like Twitter don’t hand out access tokens to every Tom, Dick and Harry. Well, they do - but they like to know whether your name is Tom, Dick or Harry! You need to register all applications with them for this purpose and to get hold of a key to access their OAuth service. This is pretty much standard for all similar providers. Visiting http://twitter.com/oauth_clients and logging in using your Twitter account opens up a menu where you can register new applications/clients or delete them. Create a new client record here. Naming is not important - just make absolutely sure that a) you select “Browser” as the Application Type, b) set a callback URL of http://mytwitterclient.tld/callback.php (edit domain and path for your preferred location), and c) select “Read & Write” as the Default Access since we will be sending new Tweets and need that write access. We will not be using Twitter for logins. Once registered - it’s immediate - you’ll be faced with a page of details including URLs to various OAuth endpoints, and most importantly, the Consumer Key and Consumer Secret you should use for your application. Don’t worry! You can revisit this page at any time.
A Glimmer Of Actual Source Code
Since we’re sticking with the concept of “simple” here, this will be a scripted effort not an exercise in writing the next Zend Framework powered super app. Any permanent (these can be refreshed indefinitely so don’t worry about losing them) Access Token will be stored to the current session for reuse so we can skip a database.
Our mini application is comprised of six files (none of them big): config.php, common.php, index.php, tweet.php, callback.php and clear.php. Splitting this out across a few files makes it easier to follow, understand and edit. Let’s start with config.php which contains our OAuth configuration which you will need to edit for your own details.
[geshi lang=php]
define('URL_ROOT', 'http://mytwitterclient.tld');
$configuration = array(
'callbackUrl' => ‘http://mytwitterclient.tld/callback.php’,
‘siteUrl’ => ‘http://twitter.com/oauth’,
‘consumerKey’ => ‘xxxxxxxxxxxxxxxxxxxxxxx’,
‘consumerSecret’ => ‘xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx’
);[/geshi]
In a perfect world, this should be everything you need to add to your configuration. You should always be aware that a Service Provider is capable of setting their own requirements, service endpoints, and even preferred request method. A fuller configuration might look like this (still valid for Twitter since it mostly demonstrates internal defaults used by Zend_Oauth):
[geshi lang=php]
define('URL_ROOT', 'http://mytwitterclient.tld');
$configuration = array(
'version' => ’1.0′, // there is no other version…
‘requestScheme’ => Zend_Oauth::REQUEST_SCHEME_HEADER,
‘signatureMethod’ => ‘HMAC-SHA1′,
‘callbackUrl’ => ‘http://mytwitterclient.tld/callback.php’,
‘requestTokenUrl’ => ‘http://twitter.com/oauth/request_token’,
‘authorizeUrl’ => ‘http://twitter.com/oauth/authorize’,
‘accessTokenUrl’ => ‘http://twitter.com/oauth/access_token’,
‘consumerKey’ => ‘xxxxxxxxxxxxxxxxxxxxx’,
‘consumerSecret’ => ‘xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx’
);[/geshi]
We won’t worry about the details for now. The only two to bear in mind are “requestScheme” and “signatureMethod”. The request scheme is a means of sending the OAuth Protocol Paramaters to a service provider. By default (and it’s the preferred option per the spec) we send these parameters by way of an Authorization header. There is a fallback to two other methods - as raw data in a POST body, or as part of the query string (GET or POST). The default method should be supported as recommended by the OAuth Specification. The signature method has four supported options in Zend_Oauth: HMAC-SHA1, HMAC-SHA256, RSA-SHA1 and PLAINTEXT. HMAC-SHA1 is a reasonable default, but the others may be needed on some service providers. Finally, you will note there are three URL options and no “siteUrl” as in the short version. Setting a siteUrl is perfectly fine if the actual endpoints use the convention of /request_token, /authorize and /access_token as the paths in their endpoints.
Not let’s look at common.php (predictably the file all other files except config.php will be including):
[geshi lang=php]
// If you haven't edited php.ini to add the Zend Framework and the
// Zend Framework Incubator to the PHP include_path, then do so here.
// Don't use mine!
set_include_path(
'/home/padraic/projects/zf/trunk/library'
. PATH_SEPARATOR . '/home/padraic/projects/zf/incubator/library'
. PATH_SEPARATOR . get_include_path()
);
// Make sure Zend_Oauth's Consumer is loaded
require_once 'Zend/Oauth/Consumer.php';
// Start up the ol' session engine
session_start();
// Include the configuration data for our OAuth Client (array $configuration)
include_once './config.php';
// Instantiate an instance of the Consumer for use
$consumer = new Zend_Oauth_Consumer($configuration);[/geshi]
The common file is pretty simple stuff. You'll note we set the include path here (you can do it in php.ini either), start our session, and instantiate a new OAuth Consumer. Zend_Oauth currently offers a full Consumer, the Server/Provider implementation will follow in the very near future.
Time to look at the starting point to our little app, index.php:
[geshi lang=php]
// include some common code
include_once './common.php';
// Do we already have a valid Access Token or need to go get one?
if (!isset($_SESSION['TWITTER_ACCESS_TOKEN'])) {
// Guess we need to go get one!
$token = $consumer->getRequestToken();
$_SESSION['TWITTER_REQUEST_TOKEN'] = serialize($token);
// Now redirect user to Twitter site so they can log in and
// approve our access
$consumer->redirect();
}
// Got past that if block! Must have an Access Token. Let’s kick out a simple
// form for the user to submit their tweet with.
// echo the xml declaration in case shorty tags enabled - grrr. They’re
// a bloody nuisance at times.
echo ‘‘;
?>
"http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
You successfully sent your tweet!
Oops! Tweet wasn’t accepted by Twitter. Probable failure:
All that work on Zend_Oauth, and all you do is send Tweets with it?
Click below to delete the Access Token and force start another authorisation leg:
Clear Access Token
[/geshi]
In the top half of this script, we’re checking to see if we previously stored an Access Token (needed to use the Twitter API on behalf of the current user) as a serialized session variable. If it’s there, we’ll print the form, accept a status textfield, and send it to tweet.php. If we don’t have an Access Token, we need to get one. Calling “$consumer->getRequestToken();”, asks the Zend_Oauth_Consumer to fire a request to Twitter asking for a new Unauthorised Request Token, and we should get one back. With this in hand, we may then redirect the user to Twitter (the redirect URI will include the token) so they can approve our access. Before we redirect the user, we’ll serialise the token and store it as a session variable. Note that all tokens in Zend_Oauth are objects of type Zend_Oauth_Token.
The next step happens outside the application since we redirected the user back to the provider.
Here, a user is given the option to approve our access or deny it. As Twitter notes, users may also retract their authorisation at any time. Once the user clicks the big “Allow” button, Twitter will redirect the user to us. Within the redirect URI should be a verification code (think of it like a PIN number) with which we can now request an authorised Access Token. Now, as we configured, and as it was added during registration, a user should be redirected to callback.php:
[geshi lang=php]
// include some common code
include_once './common.php';
// Someone's knocking at the door using the Callback URL - if they have
// some GET data, it might mean that someone's just approved OAuth access
// to their account, so we better exchange our current Request Token
// for a newly authorised Access Token. There is an outstanding Request Token
// to exchange, right?
if (!empty($_GET) && isset($_SESSION['TWITTER_REQUEST_TOKEN'])) {
$token = $consumer->getAccessToken($_GET, unserialize($_SESSION['TWITTER_REQUEST_TOKEN']));
$_SESSION['TWITTER_ACCESS_TOKEN'] = serialize($token);
// Now that we have an Access Token, we can discard the Request Token
// Keep on eye on gathering RTs in real life which are never used.
$_SESSION['TWITTER_REQUEST_TOKEN'] = null;
// With Access Token in hand, let’s try accessing the client again
header(‘Location: ‘ . URL_ROOT . ‘/index.php’);
} else {
// Mistaken request? Some malfeasant trying something?
exit(‘Invalid callback request. Oops. Sorry.’);
}[/geshi]
In the callback file, we check the incoming request for a query string and if we also have an outstanding Request Token, we pass the $_GET array and the Request Token (unserialised from the session) to the Zend_Oauth_Consumer::getAccessToken() method. This sets up another request from the application to Twitter which includes both the Request Token and the verification (PIN) code from the user redirect. Twitter should now respond with an authorised Access Token, which we’ll serialise to a session variable for future use (in a serious app, this would be associated with the user account more permanently), and with an Access Token in hand we can discard the now useless Request Token. With this done, we can redirect the user back to index.php where they will be greeted by our application in all it’s glory (or not).
Now that we are here, our OAuth authorisation has been completed. But wait, there’s more!
Users may now input a status message of no more than 140 characters and submit it to Twitter. The form on the page is sent to tweet.php, so let’s see what it is doing:
[geshi lang=php]
// include some common code
include_once './common.php';
// Check for a POSTed status message to send to Twitter
if (!empty($_POST) && isset($_POST['status'])
&& isset($_SESSION['TWITTER_ACCESS_TOKEN'])) {
// Easiest way to use OAuth now that we have an Access Token is to use
// a preconfigured instance of Zend_Http_Client which automatically
// signs and encodes all our requests without additional work
$token = unserialize($_SESSION['TWITTER_ACCESS_TOKEN']);
$client = $token->getHttpClient($configuration);
$client->setUri(‘http://twitter.com/statuses/update.json’);
$client->setMethod(Zend_Http_Client::POST);
$client->setParameterPost(‘status’, $_POST['status']);
$response = $client->request();
// Check if the json response refers to our tweet details (assume it
// means it was successfully posted). API gurus can correct me after.
$data = json_decode($response->getBody());
$result = $response->getBody(); // report in error body (if any)
if (isset($data->text)) {
$result = ‘true’; // response includes the status text if a success
}
// Tweet sent (hopefully), redirect back home…
header(‘Location: ‘ . URL_ROOT . ‘?result=’ . $result);
} else {
// Mistaken request? Some malfeasant trying something?
exit(‘Invalid tweet request. Oops. Sorry.’);
}[/geshi]
When I said there’s more, I meant it. From now on, every single API request concerning our user must be authorised using our OAuth Access Token. This is done, handily enough, by signing all requests using HMAC-SHA1. Now, doing this manually can be a pain, so using the Access Token object you can simple retrieve an instance of Zend_Http_Client (subclasses by Zend_Oauth_Client) which will handle all the OAuth protocol parameters and request signing transparently. Just use as normal as you would Zend_Http_Client.
So using this new client, we merely implement part of the Twitter API. Sending status messages is done by sending a POST request to http://twitter.com/statuses/update.format (where “format” must be changed to one of “json”, “xml”, “atom” or “rss”) which includes a “status” parameter (in the POST body) containing our tweet. Once the response is received, we do a little checking to ascertain whether it was a success or failure, and redirect the user back to index.php once more.
The final file we haven’t mentioned is linked to from the bottom of index.php’s output. clear.php is a simple script to delete the current Access Token and trigger a new OAuth authorisation process back through Twitter (just in case you missed it the first time around):
[geshi lang=php]
// include some common code
include_once './common.php';
// Clear the Access Token to force the OAuth protocol to rerun
$_SESSION['TWITTER_ACCESS_TOKEN'] = null;
// Redirect back to index and the protocol legs should run once again
header('Location: ' . URL_ROOT . '/index.php');[/geshi]
Now go and do some Twittering! In any other Twitter client, your new tweets should also display nearby the name of the application you registered with Twitter.
Other Considerations on OAuth?
There are a few things I’d like to mention before closing. The first is that Zend_Oauth implements Revision A of the OAuth specification (1.0a) which accounts for a session fixation vulnerability in the original specification. There is no need to worry about that here, and 1.0a is being rolled out to other service providers as we speak, if not already. Twitter implemented the improvements back in June. The component is still compatible with the original specification for service providers who haven’t yet updated their implementation - it’s completely transparent. Zend_Oauth also makes use of Zend_Crypt (in the Incubator also) so we offer full support for HMAC and RSA hashing via its subcomponents.
Conclusion
Well, there you have it. A simple little app for making tweets using the Twitter API over OAuth. Obviously this is a very simple example to show off OAuth but I think I kept it neat enough to explore its operation without confusing everyone thoroughly . In reality, tokens will need to be managed with much more care since they are valid for extended periods (in fact many times the provider won’t expire them for the forseeable future). You also need to ensure they are associated with the correct user.
Have fun with OAuth!
Unapologetic To The End? I think Zend_OpenId hates me… Hates it back!
Jul 27th
Disclaimer: The following is obviously a spur of the moment tirade of sarcastic commentary and lots of (probably bad) humour. A lot of it refers to events from two years ago (that’s around ZF 0.8!). Ancient history . Mostly. In any case, I have to rant about something once a year related to the Zend Framework. It’s traditional. Park any attempt to take this too seriously at the door. I get a bit dramatic at times…
In certain places, I do sometimes get a reputation of being a crazy person. Usually it happens shortly after I find myself going down a road that leads somewhere truly undesireable (or when I throw in a weird blog post like this one). I had one of those moments today that served as a flashback to such a point in time from 2007. It’s not the first time, but it will be the last since I’ll ignore it completely from now on. Believe it or not, I do get a bit upset at myself when I let these times get to me. They are completely unimportant and silly - but when I have to relive them I will react in a pretty irritated fashion (ed: that should be kneejerk incendiary fashion).
This one started in 2007, right about the time I was first starting to write proposals for the Zend Framework. I felt pretty great at the time. I had lots of ideas, plenty of motivation, and then it all seemed to go downhill from there. The turning point was in June 2007, when after three months of work I felt I had enough done to propose OpenID support for the framework. Obviously a worthy cause, and also one which shouldn’t cause much grief since the ZF keeps advertising its wicked web service support. Before long, that boat was scuttled with news that Zend itself had been working on just such a proposal. Okay, that was disappointing. Zend had somehow managed to completely miss a three month effort on OpenID. Just another day on the farm. Next step? Collaborate? So I waited a few days and Zend’s code for what would become Zend_OpenId appeared in my inbox. Let the review commence! Okay, that ended a bit abruptly…it’s a bit short. Almost like it’s missing stuff and was written in a handful of days as a prototype. Talking went on and off for a while, and eventually my interest in collaborating petered out. Like any open source contributor, I participate to scratch an itch. My OpenID itch was well and truly scratched by now with my own source code, and continuing to throw time at another OpenID solution based on a different paradigm with a different ideology, little of which I agreed with, when I already had one functional was pretty pointless compared to something that still needed scratching like the Zend_View Enhanced proposal (and I really needed that shepherded through). So Zend_OpenId continued without my assistance, with something for Yadis persisting in the Incubator as an orphaned proposal. It still persists waiting that one final patch and call to the trunk… One day…
This all sounds reasonable and a bit dry, but in reality I found it all intolerable. A self-professed OpenID expert who is a member of an OpenID Foundation decides to add OpenID support to the Zend Framework after working on it for three months and who has actual code to prove it, is then told about the other proposal which just happens to be coming at the same time, and has code…of course. Well, a bit of code. If I were a real nut…I’d have a field day. Collaborate? I’d already finished this twice. I’m supposed to go for round three? Yeah, I admit, I was getting pissed off very quickly back then. I was close to constructing an elaborate conspiracy between Zend, George W. Bush and the Illuminati (how could you NOT have the Illuminati ). What was nagging me from the start was the obvious - I should have told someone to stuff that proposal somewhere and forged ahead with my own. Of course I didn’t. I was new, and had green stuff growing out of my ears, and it was Zend. The Zend. Silly me…
The summer of 2007 left me wishing the Zend Framework Proposal Process would shrivel, die and get gobbled down by Orcses (I was hoping for Gollum but he’s a hotshot movie star now). It remains the single worst open source experience I have had since I joined the open source gig in 1995. Not only was it discouraging, it was demoralising. There were points when I was driven to contemplate whether my input was even wanted. Which is probably why I remain so overly sensitive about it - I can understand the new guy is immediately suspect. But not like this. In September 2007 I finally wrote the blog post “How a proposal process could work - if retaining contributors is a goal” out of frustration. As the title suggested, and others guessed, I was borderline on taking a hike from contributing. Some months later I gave some ground, and deleted Zend_Yaml as a proposal. Waiting for a review, or challenging non-existent reviews, or noticing the distinct lack of communication between people gets tiresome eventually. I have a spectacular tendency to vanish for months on projects, but not a whole year. Also, using it as a condition when reviewing another proposal was just fucking stupid. After a few bouts of initial enthusiasm, Zend_Service_Yadis joined it for any worthwhile purpose. That is my fault of course - my motivation to finish a library already released on PEAR had simply evaporated. That scratch your itch thing again - it was scratched on PEAR already. By now Yadis had been in use for over a year in production in other forms, and Zend_OpenId wasn’t going anywhere.
Thankfully, since mid-2008, that horrible system seems to have mostly vanished for good. The change in topics may have helped too. Not many people have considered reimplementing Zend_Feed, and for a change, nobody had been busy on any secret OAuth components. It’s taken roughly a year, but these are finding themselves in the trunk and Incubator respectively and should see ZF 1.9 and ZF 1.10 in turn. Hurrah, I committed actual finished code for once…;). I’ve also dropped any pretense at applying Agile methods to proposals. When I propose something, it will have backing code, class skeletons and use cases (basically the entire integration test suite). It won’t be complete, but it will be sufficiently functional. Overall, I haven’t been irritated in months (well, before today).
But why, why does the whole Zend_OpenId episode still carry weight? It hit me today, when I was scanning my email and I spotted a note about some Zend Framework libraries showing their age and lack of maintenance. Perfectly normal for something the size of Zend Framework - code rot is persistent anywhere. It can be difficult to diagnose though unless you are a regular user or the maintainer, so I shot off a reply adding that the same applies to Zend_OpenId. It was only intended as a supportive example, something I’m personally familiar with since I’m the guy with “Zend Framework” and “OpenID Europe Foundation Representative” written all over their email signatures, so apparently this makes me an expert on the topic. I field a lot of questions about Zend_OpenId at times. As anyone using it knows, Zend_OpenId just doesn’t work at the moment depending on the Provider. It’s woefully outdated.
However, it appears that…I am the Zend_OpenId maintainer! Really? Huh? Wait a minute… WTF?
Que 2007 flashback… Que obligatory irritated kneejerk response disavowing all alleged responsibility for afore mentioned proposal I have never contributed any code to…
I know where this started. At some point in 2008 I did agree to look into OpenID 2.0 support. It was one of those goodwill moments, where you know you’ll get around to it…eventually…after A, B, C and probably D. And you really mean it too, until you’re dragged into someone’s office and told about E. Then the mysterious Z pops by for a few months with a couple of mates. Anyway…it never went further. No proposals, no code, no discussions, no nothing. I only vaguely remember the email even. I imagine I forgot about it within a week until I was reminded about it today, and am apparently being held to it by superglue and ISP records of the email (just joking about the ISP records). God help me. Why are you laughing? This is fucking serious!
Now, being the apparent Zend_OpenId maintainer and therefore responsible for all issues filed since Spring/Summer 2008…I resign, effective immediately. It’s bad for my heart, and my irritation levels. More so now that I know about it. Now that I have resigned, I can see something of the funny side. If I complain about Zend_OpenId - it really is all my fault. Even if I was unaware of it. Allegedly.
Seeing a funny side, and having the benefit of hindsight, I can recognise where it all went off the rails back in 2007. Back in 2007 I should have been more of an ass. Yessir, I should done a Hugh Laurie on someone! Instead I retreated into politeness, followed by disinterest. I should have done exactly what I did with Zend_View Enhanced - write the bloody thing to death in a multi-part monologue that would have directed my legion of rag wearing subjects (er…readers) to assault the edifice of Zend Technologies Inc. until they caved and decided proper OpenID support was popular enough to warrant some real attention, i.e. raise bloody murder. If nobody listens in private, then shout it from the rooftops. But I didn’t. And Zend_OpenId is broken. And the OpenID Community is all the poorer for it. So it really truly is all my fault. I wasn’t a big enough ass. I will try to endeavour to be a bigger asshole in the future. May god have mercy on all your souls.
Now that we’ve established blame, and didn’t even need svn to do it (since I’m not listed there…aha), can we update a few records? In the maintainer field, right on the table called Zend_OpenId, can we change that to NULL or “Zend” since they actually wrote it so presumably are responsible also for its maintenance? While we’re at it, can we make it a new rule that the ex-proposer of features later proposed by a completely different party, not be held responsible for the completely different party’s screwups? Thanks.
This years “Paddy Rants About Something In The Zend Framework You Couldn’t Care Less About” was brought to you by…well…me.