Archive for February, 2006

It’s not everyday you see both these terms side by side. PHP has poor UTF-8 support. Well, actually its non-existent once you ignore the optional libraries that many shared hosts probably won’t compile with PHP. Now many will know all about setting up HTML pages for UTF-8, for many its the latest programming mantra. Unfortunately just setting a content-type is not enough. This will work completely for English webpages – and only English webpages using unaccented roman characters [a-z0-9].

So what is this mysterious character encoding and why is it so very important to get it right?

The problem is the 16 ISO-8859′s (ISO-8859-16 is a necessity for the EU Euro symbol I think), ASCII, and dozens of other language and custom character encodings to support native character sets. In short its a case of too many cooks. To try and unify all these variants and offer a single standard supporting all character sets (and therefore all languages) we have Unicode, and by extension UTF-8 encoding of the Unicode character set. With UTF-8, a UTF-8 encoded file can represent anything from standard ASCII (read English) to Hindi and more. For a multi-lingual web application this is the best thing since fire was invented – or would be if PHP supported UTF-8 and PHP developers figured out that string functions in PHP at present are doomed to failure with UTF-8.

Consider the example of a Sourceforge page – https://sourceforge.net/users/maugrim_t_r/ . Yes, that’s my profile . However there’s a small problem. The page insists it is encoded in UTF-8 (view the source). That’s not true. You see, a simple European accented character in my first name is not displayed – its replaced with a ?. Why? Because its not UTF-8 encoded. Ha! So much for internationalisation on Sourceforge. It claims to be UTF-8 – but its obviously not UTF-8. The truth is that only parts (if any) are actually UTF-8. The rest is most likely ISO-8859-1 – i.e. ASCII English.

One could blame me – I’m a bad person who uses a name containing a Gaelic accented character (equivalent to the french a-acute character). Why not blame the French why we’re at it, or the rest of those pesky non-English speaking people?

Its because of this that many self-proclaimed web apps purporting to support internationalisation or localisation (we use the abbreviation I18N for the latter) are being unrealistic. Many will support I18N for English, and a subset of European languages where accented characters can be given an English character variant, or maybe if they use a template system with UTF-8 encoded templates that require no manipulation – i.e. no database storage or use of PHP string functions. Some will require the mbstring library be available for PHP.

There are still problems. Take a simple example. I create a form accepting a username of no more than 32 characters. So someone from India drops by and enters a Hindi username – 32 Hindi characters. Now the form is submitted. Everything is great, right? We just check for 32 characters. But wait – there is no character length function in PHP! And no, strlen() is not that – not at all.

strlen() does NOT count characters – it counts BYTES. Never forget that distinction lest it burn you. Now UTF-8 is a multi-byte encoding. Keeping in mind UTF-8 incorporates single-byte ASCII we can say the string “Reaper” has 6 bytes, equalling 6 characters. This is what most of us already do – its the evil assumption. But what about Hindi? Hindi uses multi-byte characters.

A 32 character Hindi string will actually contain a lot more than 32 bytes – some languages may take up to 32 * 6bytes, i.e. 192 bytes to represent a 32 character string! Result – our filtering logic fails miserably, a victim of our false assumption that the whole world speaks English and writes with Roman characters.

There are other deeply set practices that fail to account for UTF-8. While Perl supports UTF-8 in RegExp, PHP’s PCRE does not. How many of us use regular expressions in form validation? Probably the same number who use strlen() to count characters… There are other examples. Basically you could list all the string functions.

There is a solution however – we can compile (or edit php.ini for dynamic inclusion) the mbstring library for PHP. However this has one flaw – it’s not a default library which means many shared hosts will not actually support it. So much for PHP’s UTF-8 support. The other solution is to wait for PHP6 which will have native UTF-8 support. But then what about all those hosts who’ll persist in using PHP4 because PHP5 is not backwards compatible?

In short, unless you’re a native English speaker – people using PHP and outputting UTF-8 html while claiming to support I18N are (in the majority) completely ignorant. Afterall for an English speaker there is zero difference between using ISO-8859-1, ASCII and UTF-8. Its the rest of the world that must jump through hoops. Even worse – many of the UTF-8 HTML out there is not even encoded in UTF-8. Often its just a typical ISO-8859 document (written on WindowsXP in almost all cases) with a UTF-8 content-type charset – this largely explains why name is commonly corrupted with question marks and boxes on many English sites claiming (falsely) to be UTF-8 encoded.

My name is not P?draic or Pdraic – it’s Pádraic!

Last September I decided to make a switch from the ADOdb Database Abstraction library to a smaller more memory efficient alternative. Of course, I’m talking about ADOdb-Lite created by Panama Jack (the guy in charge of Alien Assault Trader’s development). Making the switch was not done lightly. ADOdb is an extremely popular abstraction package with a lot of solid support and features. On the switch side it hogs memory. ADOdb-Lite‘s claims to reduce the memory usage substantially while still offering a similar (though reduced) feature set was very attractive.

The switchover overall has been worthwhile. Its worth noting that at the time ADOdb-Lite was a very new library so it took some time to settle down into a stable form. Since that time I’ve reported two bugs and requested one function change, and seen other issues like the unexpected return of full arrays not associated arrays (no numeral keys) a few versions back resolved.

Some ADOdb features are yet missing. There are no Caching functions, which removes the possibility of caching results to file. Now PanamaJack justifies this by noting in-memory caching features of MySQL – incidentally Postgres has something similar. Lets talk DBMS’s in general before justifying everything with MySQL. I rarely even use MySQL. But at the moment its not a concern for me – to be honest the library is open source, so if I do need it I’ll hack it in there with the nifty module system.

Also missing is the Performance Monitor offered by ADOdb. Again this is one of those extra rarely used features. It is quite useful, but I’m not certain how in-depth this was originally integrated into ADOdb. This doesn’t particularly bother me since I use ADOdb-Lite as the abstraction layer in a Data Access object – I can fit in a perfmon clone without touching the ADOdb-Lite source code. Of course it’s pointless at the moment given a DAO does not necessitate optimised SQL. Unfortunately its a complex item to add to a clearly defined DAO-TransferObject or ActiveRecord setup without turning the DAO into a collection of transaction scripts…

Another small niggle in the system are the potential typo-style errors. ADOdb-Lite was either developed on a plain text editor, or a non error reporting IDE. Eclipse still complains a lot about ADOdb-Lite having notice level errors, and in earlier versions potential parse errors. Small niggle, I have a preference for IDE quality assurance as a first line error pickup mechanism. The library does ship with tests, which although not Unit Tests do a fair job of exercising the Library. They’re imperfect however for a full test suite – they can’t cover all DBMS’s supported at one session. Hardly the developer’s fault – it’s just the nature of an abstraction library.

Overall ADOdb-Lite is definitely worth a look at. It does a fantastic job of reducing memory usage, enough that it makes up a substantial percentage of QSE’s memory use reduction over QS2. It’s matured over the last half year to a stable dependable library. It lacks some features of ADOdb, but these tend to be small-use features most developers wouldn’t even touch. Of course that may be a bitter pill for some developers to swallow. A good trend is a steady pickup of ADOdb-Lite – I’ve seen a few projects make the conversion recently.

In summary, if you’re expecting to develop an application for public consumption in enviroments that may include non-accelerator enabled shared hosts then ADOdb-Lite is a solid alternative to ADOdb. It’s still maturing, and has some small issues but for most common use-cases it’s worth looking at.