Final application available at http://winphp.juokaz.com:82/.

Goal was:

My project will allow people to upload huge collections of photos (probably archived in one zip file) and get nice online gallery.

I decided not to use archived files and simply allowed to upload multiple files using Silverlight control, but pretty much everything left the same. My approach to this competition was to focus on technologies and not on functionality. That’s why I spent huge amount of time creating abstractions and making parts of application to be very customizable and not adding a lot of functions.

Functions in my app are really easy to add and in a matter of some lines one can add various parameters to images (tags, name etc.) and then sort/filter them – a lot of hard work is done under the hood. Silverlight based gallery is also absolutely independent from whole application and will work as long as front-end supplies correct gallery xml file, hence it can be easily customized.

Rules had 4 criterias:

1. Application originality – I think I passed this one
2. Completeness – I have done everything what I’ve wanted
3. Use of Windows specific features/services – uses COM objects, Silverlight
4. Documentation of the process – articles in this blog and a lot of tweets

I think I have done everything that was required and now will try to push this application even further. Maybe Microsoft itself will show interest in it, because what I’ve done and used has been key topics in MIX 09.

Silverlight part and images processing is based on Jellyfish library. At first I used Microsoft libraries, but soon I got stuck because of lack of documentation and functionality. However Jellyfish is far from perfect – a lot of things are hardcoded, made private and hard to change. Also, it has some functions which are useful only in rare cases and need to be removed. For example, each mouse move, click and scroll used to look through all images in scene (using loop) and detect which one is under the mouse cursor. Very inefficient (especially if you don’t need it).

Zend Framework was used to power whole website. I didn’t find any difference running it on Windows from running on Linux, so I can’t say much about anything specific. However, I decided to use the new Sql driver implementation for PHP, but it wasn’t included in Zend Framework supported adapters list. That’s why me and Rob Allen started a project at codeplex to create an adapter for sql driver. It’s almost complete and at least both of us use it for our projects – everyone is free to test it and suggest changes though.

Windows was good enough. I haven’t used Windows for any development for more than two years, but it worked surprisingly good. NetBeans works the same in all platforms, but Visual Studio was almost a new thing. Nevertheless, both have done their job. IIS was a new thing too, but I didn’t find a big difference for a developer, only that it allows to change everything without touching config files – I found it faster when settings are in multiple places.

Community is outstanding. Stuart Herbert,  Rob Allen, Alton Crosslen and organizer Bram Veenhof were all chatting on Twitter and it felt more like a community project and not a competition. Also the level of entries, from my point of view, is really high – this competition doesn’t had a lot of entrants (only slightly more than 10), but all of them are really competitive.

It was really a great time.

1. PHP extensions
2. COM objects, programs executed with exec()

Today I’m going to look at all of them and explain my choice to use COM objects and not others from above.

Three choices

PHP extensions (how to create them read here) are fast and (at least should be) stable. However they are quite hard to create at start and uses C. For example, I’m now using library written in C# and rewriting it in C just to use it as extension is way to complicated and time consuming (and probably not worth it). I can only image PHP extensions as libraries to optimize some parts of code or to be used in multiple projects. In my case, I only need to some limited tasks.

Next logical choice would be executing programs with exec(). In my opinion it’s the easiest way because programs can be created with any language you like, can be supplied with source code and none of system settings need be changed. If exec() is not blocked in php.ini it’s very practical solution.

Windows users has an option to use COM objects. COM objects stand somewhere between PHP extensions and exec(). They can be created in many languages, but libraries are not executed outside PHP script – they behave as normal PHP objects (almost). They are more convenient to work with than compiled programs, but need to registered with Windows.

The best

I was testing last two solutions for more than two weeks and my final choice is… COM objects.  To explain this choice here is a short story:

I have a webpage which allows uploading multiple pictures. Two files are uploaded in parallel and once uploaded they are processed (resized, thumbnail and special structure for DeepZoom is created). All this processing is written in C# and uses various windows and 3rd-party libraries.

At first I tried compiling a program as “console application” (in VisualStudio) and then run it with exec(). It worked as expected, but was a little bit too slow. Script was taking around 3 s. to execute, even though actual program ran in 1 s. I didn’t spent much time analyzing where overhead was coming from but it was clear that it can be optimized.

So I compiled program as “class library”, registered it with the Windows assembly cache and then used with a DOTNET class. First problem was IIS server – it kept throwing 500 errors no matter what I did. I didn’t wanted to waste my time trying to find where was a problem – after changing to COM class it magically started to work (even though it uses the same library). Using my library inside PHP was much more faster and execution time reduced to less than a second (depends on a size of image).

Conclusion

I have tested most of possible ways to execute external code. Even though COM objects perform better they can’t be suggested as default choice, because libraries need to registered with a system what can be problematic and maybe not even possible. So if you have limited permissions to server – use exec(), but in all other cases – I definitely recommend using COM. PHP extensions are even better, but harder to code and limited to C.

What’s Wrong With The Existing MSSQL Extension For PHP?

… or, why do we need an improved SQL Server extension for PHP? :)

The existing MSSQL extension works well, but has a few practical limitations that have to be worked around.

• Limited to varchar(255) support
• No support for unicode columns like nvarchar
• No PDO drivers
• Poor error reporting

Now some of these issues are fixed, however Microsoft native Sql driver is still not used. As I’ve mentioned before, there are to ways to use native driver in PHP now:

1. As PHP extension
2. Through ODBC

However, none of these is available in Zend Framwork. So my task now (because of Winphp competition) is to come up with something what can be used to connect to Microsft Sql database with the new driver. To start with, lets look how standard DB config looks like:

;application/config/application.ini, ZF 1.8
resources.db.adapter = PDO_MYSQL
resources.db.params.host = localhost
resources.db.params.username = usr
resources.db.params.password = psw
resources.db.params.dbname = zf-tutorial

My goal is to change it as minimal as possible to allow easy db’s migration. To test ODBC connector I downloaded this class and renamed it to Add_Db_Adapter_Pdo_Odbc (because it’s not a good idea to change something in actual Zend Framework library). To use it in your code you need to change config file to something like this:

resources.db.adapter = PDO_ODBC
resources.db.params.adapterNamespace = App_Db_Adapter
resources.db.params.username = usr
resources.db.params.password = psw
resources.db.params.dbname = "Driver={SQL Native Client};
Server=localhost\SQLEXPRESS;Database=zf-tutorial;"

Note dbname string – now it has not only database name, but also driver information and host. I guess such format is chosen because dns’ for ODBC are very different for different providers, so to make things easier you just supply whole string, not it’s parts (driver, server/file, etc.). Don’t forget to use correct server host – for some strange reasons 127.0.0.1 refused to work for me.

Problem with ODBC is that it’s too general: ODBC supports a lot databases, hence they each can have something different (correct me if I’m wrong). That’s why I’m thinking about refactoring this class to App_Db_Adapter_Pdo_Odbc_Abstract abstract class and separate classes for different drivers. Although this class works fine for MSSQL, so can be used as is.

To test PHP extension there are no classes at all (at least I haven’t found, only in Adodb). So I decided to implement one myself – with a help of SqlSrv API Reference and Zend_Db_Adapter_Abstract abstract class I quite quickly came up with a working solution App_Db_Adapter_Mssql. Connection configuration looks the same:

resources.db.adapter = MSSQL
resources.db.params.adapterNamespace = App_Db_Adapter
resources.db.params.host = localhost\SQLEXPRESS
resources.db.params.username = usr
resources.db.params.password = psw
resources.db.params.dbname = zf-tutorial

I’ve tested both classes with the same application from previous post and benchmark results surprised me. They were equal! With a small variations page generation times were about 140 ms (all default settings, IIS7, MSSQL Express). Why? SQL server driver for PHP is open source, so if you look at it’s source you would find (C++):

SQLRETURN build_connection_string_and_set_conn_attr(
       sqlsrv_conn const* conn, const char* server, zval const* options,
       __inout std::string& connection_string TSRMLS_DC )
{
   ...
 
   connection_string = "Driver={SQL Native Client};Server=";
   connection_string += server;
   connection_string += ";";
 
   ...
 
   for( zend_hash_internal_pointer_reset( oht );
         zend_hash_has_more_elements( oht ) == SUCCESS;
         zend_hash_move_forward( oht )) {
 
      ...
 
       if( NO_ATTRIBUTE == ret.attr ) {
           if( ret.add ) {
               connection_string += key;
               connection_string += "={";
               connection_string += ret.str_value;
               connection_string += "};";
           }
       }
       ...
   }
 
   ...
 
   return SQL_SUCCESS;
}

If you are familiar with C++ you can look at full source at codeplex.com. What this function does is it creates ODBC connection string, so basically it’s the same thing as using PDO_ODBC (no?). Both are implemented as extensions (not as PHP code) so there are no performance differences, test proves that.

After today’s analysis I can’t say which driver is better to use: ODBC or native driver. However, since Pdo driver is already being used in Zend Framework it’s easier and faster to use ODBC. Also, ODBC doesn’t require Sql PHP extension to be installed, so as long as you have Sql server and native client it will work without php.ini modifications.

In conclusion, if you want to use SQL Native driver with Zend Framework solutions are almost here. I can promise, that something stable and proven to be working by actual application will be released before middle of June. I would love to hear comments from Microsoft people or ODBC users – I’m confused a little bit, because I didn’t have a lot of experience with Microsoft’s Sql and ODBC (you can share find me in twitter also @juokaz).

To start with, there are many ways how Deep Zoom application can be created. As I have written before here, easiest way is to use Deep Zoom Composer. However, I want to use it as a part of my application, so any outside application is not a choice.

Microsoft has released independent Deep Zoom library, but documentation is very limited, so I mostly used various MSDN blogs. For example, only in this article I have found list of available classes (with some useful comments). My first try was to use this Bryan Like’s example (at the end of post you can find a link to a working application).

Given code works just fine, but it can be improved. It allows doing a lot of post-processing modifications inside Silverligt client application, but in most cases such approach is very slow. Problem is, for every zoom level it loads not “grouped” image like this but required size copies of all visible images.

If, for example, there are 1000 images in a window, it will load all of them even if a single image size is just some pixels. What correct application should do is to load “grouped” images – downloaded amount of data is the same, but required HTTP calls is minimal. I only recommend using that code if you need to have dynamic layout or something like that. As I said before, in most cases, different approach should be chosen.

For my tests I downloaded big collection (>5000) of books covers from my project ManoKnyga.lt and when tried to show them in one screen. Working demo (requires Silverlight installed), JavaScript example:

After doing some in-depth analysis of Composer generated files I almost ended up witting my own Deep Zoom implementation, because I couldn’t find any flexible solution. However, later I found out that SparseImageCreator class is the only thing I need. It works almost the same as previously discussed CollectionCreator class, but generated output is exactly what I need. Code:

string source = @"C:\tests\Books";
string dest = @"C:\tests\GeneratedImages";
 
List images = GetImagesInDirectory(source);
List<Image> data = new List<Image>();
 
int start = 200;
int limit = start;
 
double x = 0.0;
double y = 0.0;
 
double rows = Math.Sqrt(start);
 
foreach (var image in images)
{
    if (limit < 0)
        break;
 
    limit--;
 
    Image img = new Image(image);
 
    img.ViewportWidth = rows+1;
    img.ViewportOrigin = new System.Windows.Point(-x, -y);
 
    x += 1;
 
    if (x >= rows)
    {
        y += 1.333;
        x = 0.0;
    }
 
    data.Add(img);
}
 
SparseImageCreator cc = new SparseImageCreator();
 
cc.TileSize = 256;
cc.TileFormat = ImageFormat.Jpg;
cc.ImageQuality = 0.95;
cc.TileOverlap = 1;
 
cc.Create(data, dest + "\\output");

This code loops through all images (with limit) in folder and adds them to a list of Images. Also it sets the ViewportOrigin property, which is used to position Image in space. Last step is to create a collection, what is done by SparseImageCreator class. Depending on images count this takes a while and consumes quite a lot of memory, so for using in a web application I need to create a scheduler too.

Another part of Deep Zoom is creating a Silverlight application. It can be easily done with Composer (just create a project with one image, and then customize generated source code for your needs). However, I’m not going to explain how to do it, because there are lots of examples online and I’m going to write another article about Silverlight only.

Currently I have some problems with library throwing Exceptions with big collections, nevertheless another milestone is reached (first was using .NET assemblies in PHP). Now I need to work on front-end part.

Making your library COM visible requires that you register the DLL in the GAC and registering something in the GAC should only be done when multiple applications need to access the same version of the same library. Even then, most software houses will deploy each app with it’s own copy of the DLL instead of using the GAC.

Nevertheless, C# (and most of other languages) support Reflection. Basically using reflection, programs can “observe and modify its own structure and behavior” (wiki). .NET has System.Reflection namespace with all needed methods. Purpose of this post is to find a way how to load DLL’s dynamically – without worrying about registering, re-registering, unregistering them in running system/production servers.

DLL’s are library assemblies, which “contain code in CIL, which is usually generated from .NET languages, and then compiled into machine language at runtime by the CLR just-in-time compiler.” (from). For all not familiar with assemblies, assembly “is a reusable, versionable, and self-describing building block of a common language runtime application.” (from). Consider this code:

using System;
using System.Reflection;
 
namespace Php
{
    public class Reflection
    {
        public Object get(string assembly, string className)
        {
            Assembly asm = Assembly.LoadFrom(assembly);
 
            return asm.CreateInstance(className, true);
        }
    }
}

This is very basic class, which can be used to do such things as:

Php.Reflection obj = new Php.Reflection ();
Object my = obj.get("my.dll", "namespace.class");

You may have thought that this class is absolutely useless, because you can simply use System.Reflection.Assembly without wrapping it into separate class. However, I was writing this code with an intention to use it as COM object in PHP and because PHP can’t access static COM class methods (which I find very disappointing) I needed to create this class. Now in PHP you can write:

$obj = new Com ("Php.Reflection");
$math = $obj->get("math_class.dll", "phpclass3.first");

This method, in my opinion, is much much more better than registering every single class with global assembly cache. It’s more practical and maintainable to have all required libraries in the same project directory, and not somewhere in Windows core. However, benchmarks are still coming so stay tuned.

* For everyone interested in Winphp contest, and people building PHP apps in Windows, I recommend reading www.eurowinphp.com blog, which aggregates all participants blogs. For example, Timmy Kokke, another WinPHP participant, recently wrote another great tutorial how to make your .NET classes accessible in PHP. Also, you can follow my on Twitter at twitter.com/juokaz.

My project will allow people to upload huge collections of photos (probably archived in one zip file) and get nice online gallery. Users will have selection of Ajax or Silverlight version and password protection for galleries. If I have time, I will also add ability to select different photos shuffle style, background theme and/or music, and maybe even automatic tour with captions.

PHP will work as front-end language (with Zend Framework), and C# will be responsible for all Deep Zoom work (which I haven’t tried/tested yet, but I believe it will work). Not only it’s quite useful app, but also showcases a lot of cool technologies. Leaseweb, sponsor of this competition, provided us with dedicated servers so I will hopefully publish alpha version in some weeks.

Why I chose to participate? Firstly, it’s good experience, because I do a lot of PHP, but also more and more C#, so it’s great opportunity to learn more. Secondly, in my belief, such competitions are really useful for portfolios and CV’s – I’m currently looking for a job, so it can help. Lastly, competitions are fun, so, I’m in.

Part of competition requirements is “blog about your experiences with Windows Server 2008 during the contest” so I’ll write around 2-3 posts every week about project status, problems with switching from Linux back to Windows and more. Also, I’m really bad in designing websites, so if anyone would like to help me – please feel free to contact.