Juozas devBlog

Every now and then I see questions about ACL and how to use it. A lot of web developers are using it without actually knowing what it is and how it works, even though it’s powering one of the most important part of applications – user access management. So this series will explain that from raw basics to complete implementation with Zend Framework component Zend_Acl.

What is ACL?

ACL is the last step you make with implementing users’ access control – you first start with basic PHP website, then add login for users to post comments, then you create admin interface for administrators, some user groups to access private data etc. Probably you’ve seen code like this:

if ($user->type == ADMIN)
{
    $html .= '<a href="/index.php?delete=' . $id . '"></a>';
}

It works fine, but the problem is that it becomes very difficult to extend as all access rules are stored in places all over the code, hence changing it isn’t that easy. It’s also not “correct”, as with all things in programming, things (logic) should be encapsulated. This problem is solved by using ACL, which makes code to look like this:

if (user_can($user, 'page', 'delete'))
{
    $html .= '<a href="/index.php?delete=' . $id . '"></a>';
}

This code does exactly the same, but actual permissions logic is hidden (encapsulated) inside. It’s very clear what it is doing – can user $user execute delete on page? It doesn’t show that ACL is used at all, but this is a first step towards right direction.

Outside code shouldn’t care how these permissions are being calculated, as long as that one entry point is used (in this case a function user_can) pretty much any logic can lie inside, even the same as shown in first example, but when needed a proper ACL layer can be added, so lets look at that.

Basics

ACL stands for “Access control list” and it is a system designed to deal with any complexity permissions specifications. ACL systems consist of three basic components: roles, resources and permissions. They form a graph and are as follows:

• Roles – “admin”, “editors” etc. User has one or more roles, depending on architecture.
• Resources – “product#123″, “categories” etc. It’s just an object which is controlled by ACL.
• Privileges – “list”, “delete”, “create” etc. Actions role can perform on resource.

Once you have these components defined you can start forming relations. For example, string “admin can delete products” can be seen as: role “admin” has privilege “delete” on resource “products”.

One thing to understand from this is that all 3 components are very dynamic – I don’t know about any situation where ACL cannot be applied, it’s only a matter of understanding how it works. For example Zend_Acl relies on you for constructing ACL tree, so DB backend, lazy-loading etc. should be created yourself, that’s why it’s too hard to start with and why I’m writing this in a first place.

Web

MVC is very easy to use with ACL, as you have a very clean separation of resources – every controller (or controller and module) is a resource and privileges are usually controllers’ actions. For example products controller can have “list”, “add”, “edit” and “delete” actions and those clearly are permissions on products resource.

$acl->isAllowed($user, $request->getController(), $request->getAction());

Looks very clean and logical. Of course direct mapping from controller/action name to resource and privilege is not perfect; ideally you should construct some Zend_Acl_Resource object from request data and check on it. As you will see in later posts, you can use same logic to auto-generate menus only including links user can actually see.

Quite commonly resources are much smaller and represent actual objects in system – news item, product, comment etc. Some users can only delete their own comments, where user belonging to sport news admin group can delete any in that category. Even though these rules are very different for different apps, but actual ACL implementation is pretty much the same – just create resources for each object (or lazy load) and add some permissions logic depending on requirements.

Problems

ACL management is problematic when you do not have GUI for that. This is because ACL is a complicated graph and just by looking at the code it’s very hard to understand how all elements are connected to each other.

One of the biggest causes of problematic ACL is the fact that it supports inheritance and privileges have weird inheritance by themselves (I don’t know how much of this applies to ACL in general; later observations are mainly based on Zend_Acl behaviour). For example if role has access to resource with permission “null“, user can access resource with any permission (even if there is no information about it). Whereas with resources, ACL system checks parent resource until it finds allow/deny.

Because of those issues, one of the first things I recommend doing is… unit test. It’s not that hard, but can help you to understand how everything works and if in fact ACL tree represents your required logic. Once you know that it does work in backend, create user interface (like seen in many CMS systems) to make management easy and clean. Ideally you shouldn’t touch it once it’s created.

Conclusion

ACL is very powerful system (if your library supports all features of course), although it’s definitely not the easiest one to work with. In next posts of this series I’m going to look at actual implementations and how to make use of them.

All parts:

1. ACL made easy. Part 1

Zend Framework is always considered as being the slow/bloated one. I don’t think this is right, so I decided to prove that it’s not correct and in fact ZF is as good as other frameworks are. This post doesn’t cover any benchmarks though; this is more like a architecture review and some misconceptions disproof.

Before we go any further, let’s check with Wikipedia:

Software bloat is a term used to describe the tendency of newer computer programs to have a larger installation footprint, or have many unnecessary features that are not used by end users, or just generally use more system resources than necessary, while offering little or no benefit to its users.

So what I need to provide is contra arguments that Zend Framework doesn’t do those.

“Large installation footprint”

25.8 MB and 2338 files. This is how much ZF weights and this is usually quoted as the main factor of bloatedness. First of all, 25.8 MB is a joke – full application takes maximum 50 MB. So unless you want to squeeze in 100’s of projects into a shared hosting, size of ZF shouldn’t matter at all. Price of hard-drive and RAM (for APC caches) is so low nowadays, that I don’t even remember last time I checked how much space my project takes up.

Same applies with files count – why would anyone even care how much files it has? As you probably know, PHP doesn’t compile your project into one binary, thus making it a big file. What it does, is it loads external code using include/require statements. Hence, from those 2338 you usually going to use around 300 (based on my tests). One can argue that this is still a lot of files, but this big number of files comes from ZF extensibility. A lot of functionality is separated into plugins, adapters etc.

“Unnecessary features”

ZF has massive amount of functionality: starting from web framework components and finishing up with services classes (I have written more about this here). There is a big chance that you are not going to use all of them, but because all components are decoupled (as much as possible) and as mentioned above, you don’t even notice them if you don’t include those files.

I’m one of those people who think that ZF has too much components, but this is rather my own personal preference rather than a problem. Nevertheless, I haven’t used only a few of available components as once you need something, it’s always very quick to just use it. No need to download anything more or look for available components. For serious components like Doctrine, you probably want to look around, but just for some regular code Zend provides all required features.

“Use more system resources”

This one is hot topic in ZF world. But the point here is that “features = slow”, especially in PHP. This happens because PHP as a language is not that fast and as you start adding more features and code, you get slower code. This where framework’s benchmarks are born, but as Pádraic mentions in his post about benchmarks:

To create a positive benchmark, you need to understand that all frameworks were born as festering piles of unoptimised stinking crap. They were all born bad and get worse with age. This sounds quite sad, but actually it’s an inevitable compromise between performance and features. It’s also a compromise between performance and ease-of-use

It’s inevitable to provide those features that ZF provides and still work faster than other frameworks. So considering the speed of development you need to know how much performance you actually need. Maybe rapid development is much more important? At least it’s for me, and by sacrificing some tenths of a second I can roll out projects in a few days. And as “bloated” definition states, something is not bloated if it provides benefits while using more system resources.

How to make it not bloated?

If you still consider ZF as bloated framework, let me give you some ideas how to make it work for you. First of all – profile, profile, profile! I keep mentioning this every time I talk about performance, but don’t judge performance of something without actually knowing why it’s not performing well. Xdebug helps here as it can provide profiling data, which you then can use to detect slow parts of code. From my experience this is usually I/O – DB calls, file reads etc. Get rid of those and you have some decent performance increase.

A lot of other performance factors come from knowing how framework works internally or really reading manual. For example Zend_Application is kind of slow, but it can be easily optimized by just getting rid of it. But this is only needed if you are running very big application, where every millisecond of execution matters, for normal application you’d hardly get any benefit. This is very ZF’s decoupled components start to shine, as you can just use those which work well and add some special sauce for other parts.

Conclusion

I use Zend Framework extensively, so I’m biased a lot here, but having this much of experience, I can at least judge it objectively (up to some degree). And what I can say is that ZF is not bloated, it can be slower than some other available frameworks, but widely popular method to judge its performance by the number of files it has is just wrong. Of course it’s not the easiest framework to get working with (and get it to perform efficiently), but don’t get put off just by archive size it comes in.

While attending PHPUK conference in London, I noticed how much talk there is about PHP 5.3 and “when to upgrade?”, there was even a presentation about that. Because I have been using PHP 5.3 for more than half a year, I decided to share my views on this topic. This topic is very important as the earlier PHP 5.3 is adopted, the sooner second iteration of frameworks can be released (story for Symfony).

New features

PHP 5.3 includes a lot of new features. Some of them are new language features like namespaces, closures, exceptions chaining, jump labels etc., and even though these are very recommended to learn, they are not required. You can continue to write code as you are used to and chances are that it will still work (more about that in Legacy code).

What is important from new features is fixes are enhancements. For example new garbage collector and other fixes reduce memory usage and increases performance. There are also some new extensions like phar, sqlite3 and fileinfo and some libraries (like mysql or openssl) have some major improvements. Finally, Windows support is improved even further and now is expected to be even more production-ready.

Legacy code

Apparently the biggest problem is with legacy applications – somehow everyone just decided that because of all these new features, everything should break. If you’d look at backward incompatibility list here, you notice that not that much has changed. Or in other words – if your code was “correct” before upgrade, it will work fine with PHP 5.3 too. For example static and/or private setters, ereg and some more other features shouldn’t have been used in a first place as manual clearly stated that.

Personally I have upgraded around 10 projects to run on PHP 5.3, each took less than an hour. On my development machine I also have strict error reporting level so I even had to remove marked-as-deprecated code, but for larger part of code everything worked fine. Furthermore, some of that code has been written using PHP 4, a couple years ago, and it still worked just fine.

Debian…

As far as I’m aware of, main linux distros like Debian or Red Hat do not have PHP 5.3 in their stable repositories, which makes upgrading potentially very hard or impossible. I have come across this before – sysadmins do not want to install anything outside from stable, so the only option is to wait (that happened to me when PHP 5 was still not well adopted). This is probably the only reasonable argument to not switch to PHP 5.3, but only if you don’t have option to switch servers.

I’m lucky with this problem, because I do not administer servers, but still can request for any PHP version I want. If you do not have this option, maybe it’s time to switch hosting server provider? There can be different situations, but software version shouldn’t be a limiting factor, especially when new version is available and known to be stable enough. You can even use multiple PHP versions if switching server would require testing a lot of applications.

Why you should be using PHP 5.3 right now

Firstly because it has those new features, which mean that you can start to use them as an early adopter and benefit later. All second iteration frameworks (Zend Framework 2.0, Symfony 2.0, Lithium etc.) coming out late this year will use PHP 5.3 extensively, so knowing how namespaces work (I’d say they are the ones you need to look first) and how to utilize closures will mean that you can get going very quickly.

Doctrine 2.0 is probably the best example of why you should learn PHP 5.3 now – most of its features could be written in older versions too, but not as clean and easy to understand. And it’s already in alpha stage, so stable version might be coming out very soon and yet again – if you want to stay on top of your game, you need to know how it works and how to leverage all new features.

As mentioned above, because of new garbage collector implementation PHP 5.3 performs much better (benchmarks). From my personal tests I have noticed that Zend Framework uses much less memory, just because there is less memory leaks, Doctrine of course gets a big speed up too.

Conclusion

I’m yet unsure what is all that buzz about – PHP 5.3 is stable, way faster than previous versions and works with legacy code just fine. Considering that it has new core features, which will be expected from developers to be known in just a few months, I don’t see a reason why you shouldn’t use it. And if you have any questions you can just ask me directly on twitter.

It’s always good to use great tools, but you need to make sure that you use them correctly, not just trying to code “just for it to work”. For this reason I decided to write down my usual list of things I mention when taking over some legacy project or just consulting someone how to start.

Most of the outlined problems and solutions are focused on testability, maintainability and other good code practices. If you are not familiar with them, I recommend read about them ASAP as there is big chance that you are doing those things described in this post and don’t even realize how wrong they are. Believe me, you will soon find yourself a way better developer.

Separate logic

This one is the most obvious one, but trust me, I have found cases of it in every single project I have worked before (more than 10 with Zend Framework in a past half year and counting). If it’s controller, don’t do business logic, if it’s model don’t base behavior on POST parameters etc. Same applies to forms, bootstrap, views, various helpers and many more other components – logic should be separate and have its place.

Move all logic from controller to a model or service. Use forms only to handle validation and filtering, not to actually process data and persist it in any fashion. Hide session and authentication handling in one place and provide API for other algorithms. I can go on, but I hope it’s starting to get pretty clear, or at least it will eventually when you start testing your code: when you need to setup frontController, request, cookie and mail server just to test a form you realize that something is really wrong.

Globals

If you haven’t seen this video, please do it immediately – you won’t regret it. Global state makes testing problematic and is completely opposite to what OOP proposes. This applies for use of $_SERVER, $_SESSION etc. values and all of them are accessible via request object (look at Zend_Controller_Request_Http) methods or separate classes, like Zend_Session.

Yet again I’m going to mention testability, because that’s something you always need to keep in mind. Test should not modify global variables, but rather inject mocked request/other objects which just return expected values (like client IP), because Zend Framework does pretty good job abstracting access to all global variables.

Use form values, not request

This one is both security and ease of coding issue, let’s look at this code sample:

$form = new Form();
 
if ($this->_request->isPost()) {
 
    if ($form->isValid($this->_request->getPost())
    {
        $model = new Model($this->_request->getPost());
        $model->save();
    }
    else
    {
        $form->populate($this->_request->getPost());
    }
}

After form is validated (and hence values are filtered), raw values from request are still used from $this->_request->getPost(). Not only do you lose Zend_Form functions like ignored elements (for submit buttons for example) but also none of the filters are applied (which you use, yes?). Also I can pass pretty much anything I want and model needs to do way more validation. Hence $form->getValues() should always be used as it returns form data with respect to all defined rules and filters.

Form populate() method is also very overused. This method is designed to set default values for a form, from for example GB entry (when editing something) apart from that, you simply don’t need to use it as method isValid() sets values for all elements, so there is no need to apply default values too.

Do not rely or use exit()/die()

One of the first things I do is remove all these exit() calls and handle such cases with exceptions or return statements. There is only very very limited amount of situations where you actually need to use one of those functions. Just for example, imagine this controller action code:

if (!$this->userHasPermissions())
{
    $this->_redirect('/');
}
 
$form = new Form_Add();
 
if (//submit form)
{
    // save with $form->getValues();
    $this->_redirect('/index');
    exit();
}
 
// do something else

First problem – thinking that $this->_redirect() will call exit() and hence nothing else will be executed. Even though this is true by default, this should be avoided in all cases. Not only it makes all post* events to not be fired, but also it makes testing impossible or incorrect. Zend_Test disables use of exit() in controller helpers, so in this case while testing you cannot test permissions checking as in all cases it will still execute later code. To fix this just add return in front of redirect (return $this->_redirect(‘/’)) and you are safe.

Furthermore, second exit() is completely useless and makes code even more untestable. Again you can just use return to cancel later code (view will not be rendered as viewRenderer helper checks for redirection header and does nothing if detects such). From my experience, after saving entry, there is only some assignments to view in left code so you won’t suffer a lot if you just leave redirect, of course without exit().

Use a framework, not PHP

This can sound wrong at first, but if you use a framework (Zend Framework in this case) don’t start throwing in hacks from 5 year old PHP apps. Like this (controller action):

$object = new Some_Object();
 
$image = $object->generateImage();
 
header ('Content-type: image/jpeg');
echo $image;

I don’t even know where to start… All this logic is incorporated in response object, so you can do things like:

$this->getResponse()->setHeader("Content-type", 'image/jpeg');
$this->getResponse()->setBody($image);

It might seem as same thing, but it’s not. Yet again you can actually test it, you are not working with global state (header() is global state function) and request dispatch process is left working as it should. Controllers do not output any data (hence no echo should be used) they just get request object and return response object, that’s it. In a similar fashion as exit() breaks dispatch process, outputting from controller does that too, so don’t forget to make sure that you preserve this flow.

Some small ones

Application.ini has a property includePaths, which is used to add additional paths to include path. Even though it works great, I still recommend not to use it because it will add those paths every time you create new application instance (true for 1.9, can change in future). If you try to do some controllers testing, you are probably going to instantiate it before every single test and after some hundreds of tests you will notice that somehow it’s getting slower and slower. That took me a few hours to find, though fix was easy, but still keep that in mind.

If you are using jQuery or any other javascript view helpers, take full power of them. By that I mean use functions like addJavascriptFile(), addJavascript(), addStylesheet(), addOnload() etc. to add some additional resources and code from views. If you add javascript straight to view everything will still work, but by using view helper container you will have all your code nicely placed in one place and not scattered all other the place.

Conclusion

This is just a small list of problems and issue I have seen in my work – there are way more to look at (you can share some tips too). I hope those will give you some idea how to work with Zend Framework in a clean fashion and you will soon find that your code is starting to look nicer and coding time is decreasing as everything is nicely separated and transparent to other code.

As you might have noticed I mostly write about Zend Framework and related content. This has a lot of underlying reasons, so I decided to wrap up my thoughts about it and why to (not)use Zend Framework for your own projects. This is not a comparison of frameworks though, because I don’t feel like having enough experiences with other frameworks to make a fare comparison, that’s why this is going to be only a Zend Framework analysis.

History

First time I have actually built an application with Zend Framework was about a half year ago, if I remember correctly version 1.8 was already around. Although my first applications were more than two years ago, but hey were more than tests and playing with at that time new features. So as I said above, not that long ago I switched most of my company’s code to use Zend Framework, contributed some code and have also started working as Zend Framework consultant (drop me an email if interested), so I think I have a pretty decent experience to judge it.

Before diving in to the analysis, I would like to clarify, that the reasons why I’m not using other frameworks are mainly caused by the fact that I didn’t had a chance to do that. Since I’m usually working with projects lasting more than 3 months, I need to be confident enough to know that after 2 months I don’t need to throw everything away and start from scratch. Also to test a framework for me is not to create a Hello World application, I need to build something serious to know what features are missing and what problems can be a big issue. Nevertheless, I’m planning to do some research sometime.

Advantages

For me the most important piece of software: how customizable it is. As you might have noticed from the posts like this, I usually work with complex web application and would like to have ability to built it from ground up. Zend Framework was perfect for this, because everything is just a collection of building blocks, literally (or loosely coupled in proper terms). I really liked that there is no default or even worse required structure and everything can be configured very easily.

Because I’m also in a position responsible for a whole project, quality for me is very important. I don’t know how other frameworks handle this, but Zend is pretty serious with what’s in framework and how the code is maintained. Starting from requirement to sign Contributor License agreement (known as CLA) and requirement of 80% unit-tests coverage, ending with bug hunt days. Somehow it helps me to have a confidence to rely on the framework, and backwards compatibility has never been an issue (it’s surprisingly good).

Variety of components and libraries makes development faster because I do not need to search for another library (there is also coding standards problem and a lot of smaller libraries are just a bunch of files, without any clear structure). As of today, I probably have used more than 90% of available components and because they tend to share the same configuration style or just work style, it didn’t took long to start using them. For example, just last night I converted whole application to use custom routes (one custom route per action) in about 4 hours, without having done it before at all.

Community is number one best thing of Zend Framework. Apart from contributors, there are thousands of people willing to help and share their knowledge. Usually I hangout in twitter (here), but there are also IRC channels, mailing lists, forums and many more. I’m subscribed to hundreds of PHP blogs and from what I can see, Zend Framework articles are most common and usually in a very good quality. For me web applications development is not just writing code, so having a wonderful community really helps.

Disadvantages

Hard to learn. This is not a problem (any more) for me, but can be tricky when working with others, especially with things like form decorators. Even though Zend Framework has a very good documentation, I would say, that used techniques are sometimes not straightforward for starters and can lead to a very bad code. As with example with form decorators, once you get them, work with forms will be a joy, however at first they seem as pointless complication. I guess the best thing is just read the manual properly (RTFM in more rude words) – you will benefit later.

For some an issue can be that Zend Framework has no scaffolding. In short, scaffolding is a user interface and logic generation based on database description and business logic definitions. If you have used frameworks like Django (my favorite for Python, has perfect scaffolding), you will miss these things and it can be a first “brick wall” for your learning. Zend Framework has Zend_Tool component which provides basic project generation (you can customize it easily), but this is pretty much everything.

Connected to the both issues outlined above, Zend Framework in general is sometimes too much loosely coupled. Because all the components are just separate classes and out of the box you don’t get any sort of web application, one might have problems trying to understand what to do with all this stuff at all. It’s a great feature for me, but for someone not having any experience with Zend Framework it is probably impossible to build application without a help of blogs, books or (my recommendation) manual.

As you might have guessed, the fact that Zend Framework has lots of components also means that there should be enough developers to support them. However, this is not always the case and some components are really lacking new features or updates. For example Zend_Pdf is quite out-dated and not moving as fast as a lot of us would like to. But hey, it’s an open-source framework, so if you have ideas of how to improve things – you are free to do, believe me, it’s a great fun to contribute to such a influential software application. Nevertheless, always evaluate available components and choose what’s best for your project.

There is also the funny part – misconceptions and myths. I’ve just started working on a new article about these, so expect some interesting things very soon.

Conclusion

I hope it’s clear, that most of the advantages and disadvantages are coming from the same root – depending on what experience you have and what you are looking for Zend Framework can be a good or absolutely awful choice. For me it’s a perfect tool and currently I have no plans (even though Symfony is in my todo list) to use something else, and if you look at things which are coming in version 2.0 of both Doctrine and Zend Framework (yes, they will have integration)… Simply a good solution for serious work.

During last two months I spent massive amount of time tweaking Doctrine ORM framework and making it to perform as fast as possible (as you might have noticed from my never ending tweets). This post is devoted to performance and efficiency, with practical tips & tricks how to reduce memory usage, make it work faster and save resources.

Doctrine is a very powerful framework, however you should study its behavior a little bit to get it working properly. As it turns out – it’s not that hard.

Speed

One of the first things I recommend looking at is query cache. Because there is quite a lot happening in turning DQL statement into the SQL query, caching that process can increase performance by a big margin. Best choice – APC, although robo47.net has an example code for use of Zend_Cache adapters, just don’t forget that file back-end is probably not the best pick.

Hydrators are probably the easiest thing to misuse. Hydration in Doctrine language is turning SQL query results into data graph. It can be an array, object or single value. It’s very nice to get results as PHP objects (in this case – models), however it’s slow. Slower than hydrating like array, and much slower than getting raw result from PDO. That’s why I always recommend answering one simple question: “will you be updating/deleting records?”. If the answer is no – hydrate as array, because you are not going to need an actual model (usually).

For this reason I always try to use array notation to access properties rather than like-object-vars one. Basically instead of $product->name I tend to write $product['name'], which returns the same result, but makes switching to array hydration very easy. You can find more tips and tricks at Doctrine manual here, but the key moment is to remember that the faster data structure in PHP is array (I believe so), so if application is not performing well – start playing with hydrations first.

Doctrine has a very nice support for relations, where they work like proxies and data can be retrieved when it’s needed (lazy-loading). However, this is wrong:

$user = Doctrine_Core::getTable('User')->find(1);
 
foreach ($user->Comments as $comment)
{
   print $comment->NewsItem->title . '<br />';
}

This code is bad, because for each comment you will load news item using a separate query, hence you are wasting db server resources and making code slower. Optimization is pretty straightforward:

$query = Doctrine_Query::create();
 
$query->from('Comments C')
        ->innerJoin('C.NewsItem N')
        ->where('C.user_id = ?');
 
foreach ($query->execute(array(1)) as $comment)
{
   print $comment->NewsItem->title . '<br />';
}

Here all the data is loaded in one query and everything happens much faster (and in this case making it to hydrate as array would also improve the performance).

However, make sure you know what you are joining. For example imagine that in previous example news item is also one-to-many (comment has many news items) relation and user 1 has written 1000 comments where each comment is attached to average news items. Query above will return 50′000 records (50 * 1000) which Doctrine will need to hydrate then. This will be very slow and probably going to kill your web server after some time. One day I had a query which was returning 2GB of data, server admins where probably not very happy about it…

Memory

One of my favorite parts of software development is playing with memory usage and making it efficient. Even though it sounds really simple, debugging it and finding it where the memory is leaking is not an easy task. Recently I was using Doctrine to work with quite big datasets (on average 50′000 of records) and probably have tried all the possible tricks to make Doctrine memory efficient. My code looked really simple:

$data = Doctrine_Query::create()->execute();
 
foreach ($data as $item)
{
    // do some work here
}

You might expect that memory usage would be steady, however it is not. Doctrine uses identity map and objects also have a lot of references which makes freeing up memory a tricky job. As my experience showed, even though records have a method free() which is supposed to de-reference it, sometimes it doesn’t help.

So to make memory management work, make sure to try these:

• $record->free(true) – deep free-up, calls free() on all relations too
• $collection->free() – free all collection references
• Doctrine_Manager::connection()->clean() – cleanup connection (and remove identity map entries)

With some debugging and profiling (and these methods) you hopefully can make memory usage to be low. I’m also using some custom iterators (available here) to divide query into chunks, because loading 50′000 of objects in one go is not going to work, so you might want to look at it too.

Another recommended tip: make sure to free queries too. As collections hold references to all records, query object also has some references to parsed sub-parts of query. For this I use auto-free setting enabled by (available in my first part post also here):

// enable automatic queries resource freeing
$manager->setAttribute(
	Doctrine_Core::ATTR_AUTO_FREE_QUERY_OBJECTS,
	true
);

It’s very easy to forget to free a query and it’s again creating these references which makes garbage collector’s work hard. Nevertheless, after enabling this one I don’t know any other place where the memory can start to leak.

Last step – PHP 5.3. I’ve been working with this version for a few months and it works great, so if it’s possible – I recommend using it (you will also help with testing frameworks, but both Zend Framework and Doctrine already should work fine). One important function here is improved garbage collector, so the actual script takes less memory to execute. I haven’t recorded benchmarks on this one, but what I’ve noticed during development is that 5.3 does in fact has a lower peak memory usage.

Conclusion

I think I haven’t missed anything here, or at least these are the tips which made applications work fast (if I have – let me know). To finish with – good optimizations are done by comprehensive benchmarking and profiling, so the fact that Doctrine is a big framework doesn’t necessary mean it’s slow too. At the end of the day, it’s usually only a matter of reading a manual.

All parts:

1. Zend Framework and Doctrine. Part 1
2. Zend Framework and Doctrine. Part 2
3. Zend Framework and Doctrine. Part 3

In my professional live I mostly work with enterprise web applications which are quite demanding for big layer of business logic (that’s another article I guess) and decoupling of application layers. During this year I invested quite a lot for a search of a good ways to architecture a big application and make it simply good. Quite a while ago Matthew Weier O’Phinney introduced service layer in one of his great talks about models, since then service layer become one of the key architectural component one my applications. Here I’m going to show a few examples and use cases where it’s very useful.

“Old-style” interaction with data

I’ve used it for different projects, but one of the best examples of how great this concept is SaaS or any other users-based application. Some years ago I used to have code which worked like this (let’s say this is controller action):

public function userInfo()
{
     $userDao = new Users();
     $user = $userDao->getUser($_SESSION['id']);
}

And then in my database access class Users I just execute sql query with a given user id. However, after some thinking I looked at it again and though – why does controller need to know the userId? I mean, of course it’s a job of controller to process requests and control application flow, but logically – if an action is named userInfo and we got to the point where we need the user info (hence the user is authenticated and validated) why do we need to pass user id? It’s clear that some part of code already knows it.

One more case: if a site is a e-commerce it’s clear that user has only access to his orders, addresses information etc. but in a basic MVC you either fetch by Id and then check if it’s in fact user’s order or create a method like in a first example. Again, not very clear and not easy to maintain. There are more problems too: by passing user id and id of a record you assume that controller knows that this is a key to get the information. But it’s wrong – business layer knows that user has orders; controller only knows that there is such a thing like orders and it can be retrieved by id. That’s it.

Service layer

For such things I use service layer: it has user info injected from bootstrap (or directly to a constructor) and operates with data only accessible to the user. So previous method becomes to:

public function userInfo()
{
     $service = new UsersService();
     $user = $service->getUser();
}

of course you would have a separate method to fetch other users data, but for sake of simplicity let’s just say that this method returns some private info (like address for example). Here my action is completely unaware of what user id actually is – it expects a user object, it gets it. Very simple, very clean and very easy to maintain.

Getting back to the e-commerce example, action for a view order would look like this:

public function viewOrder()
{
     $service = new OrdersService();
     $order = $service->getOrder($_GET['id']);
}

Again – this controller action doesn’t care what user id is in current session, it just gets an order by its id. If this action returns false (or throws exception with error type) then it means order cannot be retrieved, or in other words – service cannot return id by given id. It can be permissions problem, it can be something else – but controller is completely freed from checking all this unnecessary things.

Practical usage

As you might have noticed, service layer is intermediate layer between models and controllers – in the same way as you would use Flickr or Youtube API to work with remote data, you use service layer API to work with application resources. All the business logic resides in service layer, where also using other service layer, models are retrieved, changed, saved, returned etc. Controller has zero lines which contain a word Doctrine (or any other database layer class). None.

Another advantage of using a service layer – it’s a good place to merge information sources. I think it’s more like a style of mine, but I tend to have models very clean – only with logic on that model. This is mainly because I usually use Doctrine models and I don’t want to put anything in them. For example just yesterday Adam from jazzslider.org posted a very good article about using Acl with models by creating custom listeners. With all respect, even though it works really great, I don’t think it’s a clean approach – I see models and permissions control as separate layers.

Furthermore, having this layer makes replacing database layer (or even models layer) a little bit easier – because all the other code communicates with data using given API (from service layer) so as long as results returned are the same, they don’t care how they are actually retrieved (for example they can come from cache, text file or created on-the-fly). But if you would have Zend_Db_Select calls all other the place, migrating to Doctrine’s DQL can be a pain. From my personal experience, I successfully migrated my own ORM code with about 50 models to Doctrine in about 4 days without changing a line in controllers (also because of tests I had).

To be honest, I’ve only tried various service layer implementations with Zend Framework. Even a default autoloader has a resource namespace Service_ with services folder inside application, so I didn’t need to do any hacking to get it working. Nevertheless, this pattern doesn’t require any specific framework futures, but if a framework has good dependency injector (like this one from Symfony) it can make things even cleaner.

Conclusion

I don’t know if I have convinced you to look at this pattern, but I definitely recommend looking at it. Especially when your application gets quite big and you need some sort of functionality to work with all these models (on average, I used service layer with applications having roughly 80 models). Nevertheless, there are tons of different ways to do this, so I definitely recommend reading a book by M. Fowler called “Patterns of Enterprise Application Architecture (P of EAA)”. One of the best sources for enterprise applications I’ve read so far.

^* Image copyright: http://martinfowler.com/eaaCatalog/serviceLayer.html and http://www.tutorialspoint.com/images/soa-additional-service-layer.jpg

Today we start actual development with Doctrine and Zend Framework. Base of this post is my code which I have been using for quite a few projects and it worked really well.

These are the steps required to setup Doctrine:

1. Create MySQL (or any other adapter supported by Doctrine) database
2. Download Doctrine 1.2 (as of today – 1.2.0beta3). Believe, it’s stable enough (no problems at all for me) and supports functions all functions we are going to use later
3. Setup application.ini and application resource
4. Generate models from database
5. Profit!

To start with, download this archive (or get updated version from my public repository). It has everything you will need. Some of the files are quite long so I’m not going to post them here, it’s better that you download them and have them ready to be used as the article progresses.

Create MySQL database

For MySQL databases I use a product called “MySQL Workbench“. If you haven’t tried it I definitely recommend to give it a go – it basically allows you create a database as a diagram and then export it to sql file or update actual database (can be risky). Since I expect you to have enough knowledge how to create a database I won’t write anything more – you can find tutorials all over the web.

Download Doctrine

After downloading Doctrine extract it to library folder. You should have ./library/Doctrine.php in your library folder. Although, you can use svn:externals (if you are using SVN at all) to remove Doctrine code from repository, but in this case you probably need to setup paths in application resource file explain below to use Doctrine_Core if you just get Doctrine folder contents (from here).

Most important point here – don’t forget to download 1.2, not version 1.1. Even though it doesn’t have a stable release, 1.1 doesn’t support models generation as we want them to be (auto-loadable). Also this week all bugs have been fixed which I reported and now models generation works perfectly. From my personal experience, I haven’t even used 1.1 at all (started with 1.2 when it was in alpha) and didn’t had any problems, so my recommendation – use 1.2.

Setup application.ini and application resource

From the file listed above open a file called application.ini. Append you current configuration (more on getting Zend Framework project running can be found here) with settings in that file (leave compiled and cache options to false for now). Doctrine uses DNS strings to connect to a database, there are quite a few examples in the documentation. If you have used PDO before you be very familiar.

In archive there is a file called library/resource.php. Depending on what namespace you use for outside-Zend code, paste it to chosen folder. If you choose to have “App” as a namespace, just copy that file to library/App/Application/Resource/Doctrine.php. And don’t forget to have:

autoloaderNamespaces[] = "App_"
pluginpaths.App_Application_Resource = "App/Application/Resource"

in application.ini also, otherwise library code won’t be auto-loaded. I don’t recommend putting any code in Zend folder, because it will create tons of problems in the future. Having App or ProjectName library folders allows to have your own code in separate packages which you can use in later projects (I usually have App, ProjectName, CompanyName).

Provided application.ini file isn’t that much configurable, mainly because it wasn’t supposed to be released at all. If you are looking for something more dynamic, look no further than this proposal in Zend incubator. Nevertheless, my code should work fine – it has all options required to get you started and have been tweaked with settings which I found to be really useful.

Generate models from database

This part is the most awesome. To start, just copy and paste everything from scripts folder in the zip archive to your scripts folder in application (I have scripts folder in the same level as application and library, in the root of project). There actually only two files – doctrine-cli.php and custom task in Doctrine/Task/GenerateModels.php. You can run doctrine-cli.php like this:

php doctrine-cli.php

and you should get a list of all the possible tasks (if you setup everything properly). To run my custom task, append “generate-models” to the end of previous command to get:

php doctrine-cli.php generate-models

This task will load your database schema and create all classes (table, base model, model) required for models. Doctrine also supports generating models from yaml configuration files, though I’ve never used it before, but configuration should be almost the same.

It works

To test the models you can look at the generate code or start coding you application (or even run dql task from doctrine-cli.php). For example you can test simply like this (adjust by models you have):

$product = new Model_Product();
$product->title = 'Product name';
$product->save();
 
print_r(Doctrine_Core::getTable('Model_Product')->findAll()->toArray());

If everything has been configured properly you shouldn’t get any errors and this code (put it in controller) should output array of your saved data. Because of the settings in application resource, save() method also runs validation so if you missed a field and it breaks a not null constrain, this code will throw a validator exception.

What’s next?

This article has enough information to get you started and almost all the code provided it left as simple as possible and open for further modifications. I hope that in coming months code for Doctrine and Zend Framework integration will be completed and you can use Doctrine without any outside code.

Since we have application running now we can dive into actual usage, testing and more advanced stuff. I have code for these also and will be posting more articles very soon, even though the best resource for Doctrine is documentation. If you have any problems with this code just leave a comment here or find my on twitter (here) and I will try to explain more.

All parts:

1. Zend Framework and Doctrine. Part 1
2. Zend Framework and Doctrine. Part 2
3. Zend Framework and Doctrine. Part 3

If you are following twitter (you can find me there also) or any other social network, you might have noticed that there is a huge interest in Doctrine and Zend Framework integration. Since I’ve been using these libraries for quite a while now, so I’m going to explain some best practices and ways you can do that.

History

To start with, in my opinion, Zend Framework never had a proper M from MVC. It was quite common to use Zend_Db_Table as base models class, but it was simply not practical. When you start dealing with relations and hierarchical data types it starts to get really tricky, because simply Zend_Db_Table doesn’t provide an extensive enough functionality.

So half a year ago Zend Framework developers started to look for better solutions. Quite obvious one was to start implementing their own Domain layer. You can find quite a few different implementations of that in blogosphere, and I chose this path also. However, after quite some time of developing my own code I realized that it’s simply not a right thing to do – I’m expected to deliver a product and I was “wasting” way too much time tweaking, testing, extending etc. my domain model code.

So about 3-4 months ago I completely switched to Doctrine. After evaluating possible solutions I decided to stay with Doctrine for a long time. I don’t know any other solution coming, I definitely don’t want (mainly because I don’t have time) to invest on creating my own library and Doctrine is simply awesome when you get used to it. After all this time I can say that it was a right call – Doctrine is on a way to being officially supported in Zend Framework (Symfony has it right now) and with Doctrine 2.0 (you can see a short presentation of its new features right here) it will be just a perfect tools combination. I would very much agree with Giorgio that:

Thus, Doctrine 2 is going to become the first production-ready Orm for php and to be favored with seamless integration in both Zend Framework and Symfony.

Benefits

Some time ago I was actually involved in making a decision of choosing Doctrine over other libraries (main Zend Framework components) as part of my work, so here I’m going to outline some points which made that decision easier.

Relations. I keep repeating this term every time I speak about Doctrine (or ORM’s in general), but for me it’s a key factor in writing code fast. Quick example: let’s image that our application consists of Order which has a list of Items where Item is Product and price, quantity information. Code for this situation:

$order = new Order();
 
$item = new OrderItem();
$item->Product = $product; // product object from somewhere
$item->quantity = 10;
$item->price = 9.99;
 
$order->Items->add($item);
 
$order->save();

This is how much code I need to write myself. I’m going to show more examples in the future posts, but because Doctrine is ORM (object relation mapper) all the hierarchical structure of your application data becomes very easy to work with.

What is more, Doctrine can generate all the model classes by itself. Now that’s a treasure! Just specify database connection in the config file, open up a terminal and run:

php doctrine-cli.php generate-models-db

In a few seconds all schema information is retrieved from the actual database and all the classes are created. My biggest project so far with Doctrine was worth more than 10′000 lines of models code. I don’t know how to count the time it saved and despite a few minor bugs in Doctrine library it worked exactly as we wanted.

I can list things I like about it for hours probably, so I’m going to finish this list with the future I consider to be a very big time saver. This feature is dynamics of models: inheritance and behaviors. Inheritance makes your objects have different classes even though they could be a same table (with for example type_id). All the work needed to handle the actual schema is done in background.

Behaviors work in the same way – one line of code and a model becomes physically undeletable (called soft-delete) or has a separate table to save revisions. Behaviors are very similar to decorators in Zend_Form for example – they are small classes which can be added to the main class to extend its functionality. So in the end your mode could have all sorts of different functionality depending on how you “decorate” it.

What’s next?

I only wrote a few examples how great Doctrine actually is today, but you can expect a lot more. Especially in Zend Framework code side: application.ini settings, application resource to setup auto-loading, script to generate Zend Framework compatible models, testing Doctrine models and much more. I’m also involved in a work group making Doctrine and Zend Framework integration possible so you can expect great things to come (first proposal here).

If you have been following me on twitter you might have noticed that I tweet a lot of interesting material on both Zend Framework, Doctrine and general PHP so I would recommend following me there or subscribing to the RSS feed. I haven’t been writing for a while now, but I will try as hard as I can to change it and write much more frequently – during past months I tried a lot of cool stuff which I would love to talk about.

All parts:

1. Zend Framework and Doctrine. Part 1
2. Zend Framework and Doctrine. Part 2
3. Zend Framework and Doctrine. Part 3

It’s been more than a month since this competition started, but the time ran very quickly. Today I’m going to summarize what I’ve used to create my entry and what I’ve learned.

Final application available at http://winphp.juokaz.com:82/.

Goal was:

My project will allow people to upload huge collections of photos (probably archived in one zip file) and get nice online gallery.

I decided not to use archived files and simply allowed to upload multiple files using Silverlight control, but pretty much everything left the same. My approach to this competition was to focus on technologies and not on functionality. That’s why I spent huge amount of time creating abstractions and making parts of application to be very customizable and not adding a lot of functions.

Functions in my app are really easy to add and in a matter of some lines one can add various parameters to images (tags, name etc.) and then sort/filter them – a lot of hard work is done under the hood. Silverlight based gallery is also absolutely independent from whole application and will work as long as front-end supplies correct gallery xml file, hence it can be easily customized.

Rules had 4 criterias:

1. Application originality – I think I passed this one
2. Completeness – I have done everything what I’ve wanted
3. Use of Windows specific features/services – uses COM objects, Silverlight
4. Documentation of the process – articles in this blog and a lot of tweets

I think I have done everything that was required and now will try to push this application even further. Maybe Microsoft itself will show interest in it, because what I’ve done and used has been key topics in MIX 09.

Silverlight part and images processing is based on Jellyfish library. At first I used Microsoft libraries, but soon I got stuck because of lack of documentation and functionality. However Jellyfish is far from perfect – a lot of things are hardcoded, made private and hard to change. Also, it has some functions which are useful only in rare cases and need to be removed. For example, each mouse move, click and scroll used to look through all images in scene (using loop) and detect which one is under the mouse cursor. Very inefficient (especially if you don’t need it).

Zend Framework was used to power whole website. I didn’t find any difference running it on Windows from running on Linux, so I can’t say much about anything specific. However, I decided to use the new Sql driver implementation for PHP, but it wasn’t included in Zend Framework supported adapters list. That’s why me and Rob Allen started a project at codeplex to create an adapter for sql driver. It’s almost complete and at least both of us use it for our projects – everyone is free to test it and suggest changes though.

Windows was good enough. I haven’t used Windows for any development for more than two years, but it worked surprisingly good. NetBeans works the same in all platforms, but Visual Studio was almost a new thing. Nevertheless, both have done their job. IIS was a new thing too, but I didn’t find a big difference for a developer, only that it allows to change everything without touching config files – I found it faster when settings are in multiple places.

Community is outstanding. Stuart Herbert,  Rob Allen, Alton Crosslen and organizer Bram Veenhof were all chatting on Twitter and it felt more like a community project and not a competition. Also the level of entries, from my point of view, is really high – this competition doesn’t had a lot of entrants (only slightly more than 10), but all of them are really competitive.

It was really a great time.