To start with, max-execution time is first problem. By default, Apache process will kill itself if script has been running for more than 30 sec. Actual time depends on various parameters, but it’s nowhere near some hours of running. So first thing is to remove time limit:

set_time_limit(0);

Zero means no time limit at all. However, problem is not solved yet. If you are calling your script through Apache it’s most likely that script without any output in about 5 will be killed too. I believe that this depends on web server settings, but it can be easily tested – just create infinity loop and try to load it in Firefox.

After some time Firefox will display “Download” window with your script name – this means that your process has just been killed. I haven’t spent much time analyzing this behaviour, but easiest thing to do is just printing some text, for example:

for ($i = 0; $i < $pageCount; $i++)
{
   print $i . ' out of  ' . $pageCount . ', working with: '.$pages[$i];
   flush();
   hardWork ($page[$i]);
}

This not only prevents script from being killed, but also displays completion (x of N) information. It’s very useful when code may have bugs, because it shows actual unit where your code has stuck. Also, you need to make sure that there is enough memory. I have this code:

ini_set('memory_limit', '128M');

Not all scripts require that much of memory, but since all of this is used only for CRON tasks, it’s not unsafe.

Furthermore, when scripts are called by wget or just browser, they will be killed as soon as user aborts them. So if you click “Stop loading this page” in Firefox – execution stops. It’s good, but my experience showed, that sometimes wget (or other similar tool) decides not to wait longer and simply stops loading. Process gets killed again.

I don’t know why, but I spend whole day trying to make script complete its execution. Memory wasn’t an issue, there were no bugs, but still it kept being killed. Nevertheless, there is solution for this problem also:

ignore_user_abort(true);

Ignore user abort – it does what it says.

Last thing to make sure – disable output caching. When running CRON jobs, gzip‘ing content is absolutely useless and also uses memory and creates more problems with buffer flushing. I have it disabled by this:

apache_setenv('no-gzip', 1);
ini_set('zlib.output_compression', 0);
ini_set('implicit_flush', 1);
header("Content-Encoding: none");

My server uses gzip by default, so these settings makes sure that it’s not compressed.

That’s all. I use all these lines in start of my CRON jobs front controller and everything works fine. Please, better don’t try them on user-side scripts, because they can create problems – if you have no access to running processes, stuck processes with 0 time limit will probably kill your web server.

For example, if you are developing CMS you need to make sure that user input don’t break whole template. But that’s not so easy, because you need very clever HTML validations as even one missing closing tag for <div> or <p> can brake website’s layout completely. Editors like TinyMCE can check and try to fix errors, but in my experience, they sometimes create more of them.

However, problem can be solved, and quite easily. Almost a year ago I was reading some random blog when I find out about HTML Purifier. Basically, it’s library which can filter and fix any HTML. Compared to other libraries, it looks very promising, but since then I haven’t had a chance to test it – other libraries have been working fine.

Today I was working with web scrapper again and ended up stuck because of very badly formatted HTML. When regular expressions are used, code validity isn’t (shouldn’t) a case at all, but XPath fails immediately. I tried simplifying queries, hard-coded source fixing, but all that required so many effort that I introduced Purifier filter between source fetching and DOM constructing. It worked!

require_once 'HTMLPurifier.includes.php';
 
$config = HTMLPurifier_Config::createDefault();
 
$config->set('HTML', 'Doctype', 'XHTML 1.0 Transitional');
$config->set('HTML', 'TidyLevel', 'heavy');
// Don't remove IDs (<div id="first" />)
$config->set('Attr', 'EnableID', true);
 
$obj = new HTMLPurifier($config);
 
$clean_html = $obj->purify($html);

In this example I chose worst way – include all files. Library uses PEAR-like directory structure so simple auto-loader  can include all required files in background, but for simplicity it’s not used here. This sample code filters $html variable using XHTML 1.0 and does heavy level tidying (quite clear from source code itself).

XSS? Purifier protects from them also – full list of tests. Library is also highly customizable (configuration manual), but documentation is not very clear – I have spent more than a hour trying to make it to return HTML with <head> part. I haven’t found any nice solution (maybe because the library is not made for such things).

HTML Purifier contains about 350 files so it’s relatively big library, however it performs good and shouldn’t kill you web server. Today I Purified and using XPath extracted information from more than 1000 pages and it worked really stable – none of the results where filtered unexpectedly. I definitely recommend it for HTML inputs filtering because it just does wonderful job – you can try it online here.

Parallel map functions does all work for you – you don’t even need to know when it’s right time to fork another thread. Implementing parallel computation in Haskell was very easy and it almost gave theoretical decrease in processing time by N times (where N is processor cores count). Today I will talk a little bit about possible parallelism with PHP’s internal POSIX functions.

PHP is neither a functional language, nor is made with threading support (correct me if I’m wrong). Still, it’s possible to create parallel processes by using pcntl_* functions family. Some days ago I tried just to get something working, what uses parallel processes and can possible be extended to deal with time-consuming mathematical algorithms.

Download my sample code here (rename extension to php) and run it in terminal (it probably wont run with Apache). This code will try to compute x^3 for 0-1M integers two times, parallel runs both in one call, normal behaviour runs one after another. Output should look similar to this:

juokaz@thinkpad:~/Desktop/php/parallel$ php run.php
We start
I am the child, pid = 0
I am the parent, pid = 2004
yep, finished, I have 2004 ID
yep, finished, I have 0 ID
Ran parallel 5.81009602547 seconds
yep, finished, I have 1 ID
yep, finished, I have 2 ID
Ran normal 9.76671719551 seconds

As you can see, parallel computation on dual core processor ran almost 2x faster. And yes, it does use both cores simultaneously.

If you want to look more deeply, there is great article called Thorough look at PHP’s pcntl_fork(). You will quite quickly find that there are problems when using pcntl_fork, because it basically clones running process and then continues. Everything you define before forking child process will be accessible inside child – it’s not always what you really want.

Parallel processes can be easily simulated by using asynchronous calls (over HTTP, for example) from one script to others, but for something more calculations-based, using pcntl_fork() can be much more practical. But I will probably still chose asyncronous calls over parallelizing because they are much more flexible.