Juozas devBlog » Websites

HTML filtering and XSS protection

Juozas — Sat, 21 Mar 2009 20:40:24 +0000

If you have been programming websites long enough you would know that user input is first think to worry about when thinking about security. It’s really hard to decide what data is acceptable, especially when user has permission to insert HTML content through form.

For example, if you are developing CMS you need to make sure that user input don’t break whole template. But that’s not so easy, because you need very clever HTML validations as even one missing closing tag for

can brake website’s layout completely. Editors like TinyMCE can check and try to fix errors, but in my experience, they sometimes create more of them.

However, problem can be solved, and quite easily. Almost a year ago I was reading some random blog when I find out about HTML Purifier. Basically, it’s library which can filter and fix any HTML. Compared to other libraries, it looks very promising, but since then I haven’t had a chance to test it – other libraries have been working fine.

Today I was working with web scrapper again and ended up stuck because of very badly formatted HTML. When regular expressions are used, code validity isn’t (shouldn’t) a case at all, but XPath fails immediately. I tried simplifying queries, hard-coded source fixing, but all that required so many effort that I introduced Purifier filter between source fetching and DOM constructing. It worked!

require_once 'HTMLPurifier.includes.php';
 
$config = HTMLPurifier_Config::createDefault();
 
$config->set('HTML', 'Doctype', 'XHTML 1.0 Transitional');
$config->set('HTML', 'TidyLevel', 'heavy');
// Don't remove IDs ()
$config->set('Attr', 'EnableID', true);
 
$obj = new HTMLPurifier($config);
 
$clean_html = $obj->purify($html);

In this example I chose worst way – include all files. Library uses PEAR-like directory structure so simple auto-loader can include all required files in background, but for simplicity it’s not used here. This sample code filters $html variable using XHTML 1.0 and does heavy level tidying (quite clear from source code itself).

XSS? Purifier protects from them also – full list of tests. Library is also highly customizable (configuration manual), but documentation is not very clear – I have spent more than a hour trying to make it to return HTML with part. I haven’t found any nice solution (maybe because the library is not made for such things).

HTML Purifier contains about 350 files so it’s relatively big library, however it performs good and shouldn’t kill you web server. Today I Purified and using XPath extracted information from more than 1000 pages and it worked really stable – none of the results where filtered unexpectedly. I definitely recommend it for HTML inputs filtering because it just does wonderful job – you can try it online here.

Debugging with FirePHP and Firebug

Juozas — Thu, 12 Mar 2009 15:09:43 +0000

One may debug his application with print(”) statements all other the place or alert(”) for JavaScript. Luckily some years ago Firebug extension for Firefox was introduced, which introduced (?) console. Console works absolutely the same as in Linux and can be used not only to execute commands, but receive information from various sources.

To start with, I just love FirePHP. This extension extends Firebug itself and allows PHP to show messages in the console.

FirePHP use special headers to send required information, so browses without FirePHP doesn’t feel any difference, but others can easily extract information. It doesn’t mean that DB profiling information should be visible in production server, but it definitely helps in development stages. If you look at normal website’s headers you would see something like this:

HTTP/1.x 200 OK
Date: Thu, 12 Mar 2009 14:29:29 GMT
Server: Apache/2.2.9 (Ubuntu) PHP/5.2.6-2ubuntu4.1 ...
X-powered-by: PHP/5.2.6-2ubuntu4.1
Content-Length: 0
Keep-Alive: timeout=15, max=100
Connection: Keep-Alive
Content-Type: text/html

where FirePHP modifies it to:

HTTP/1.x 200 OK
Date: Thu, 12 Mar 2009 14:28:39 GMT
Server: Apache/2.2.9 (Ubuntu) PHP/5.2.6-2ubuntu4.1 ...
X-powered-by: PHP/5.2.6-2ubuntu4.1
X-Wf-Protocol-1: http://meta.wildfirehq.org/Protocol/JsonStream/0.2
X-Wf-1-Structure-1: http://meta.firephp.org/Wildfire/Structure/FirePHP/FirebugConsole/0.1
X-Wf-1-Plugin-1: http://meta.firephp.org/Wildfire/Plugin/ZendFramework/FirePHP/1.6.2
X-Wf-1-1-1-1: 56|[{"Type":"INFO","File":"","Line":""},"This is a debug!"]|
X-Wf-1-1-1-2: 58|[{"Type":"ERROR","File":"","Line":""},"This is an error!"]|
Content-Length: 0
Keep-Alive: timeout=15, max=100
Connection: Keep-Alive
Content-Type: text/html

FirePHP website has libraries for sending these headers, but since I was trying it when using Zend Framework, I chose Zend_Log_Writer_Firebug module. It works as writer interface for Zend_Log and since Zend_Log is (or should be) used for all logging in ZF, you can get very nice access to all information (execution time, queries, warnings, etc.) from browser. For example, to send something with Zend Framework, code would look like this:

// Place this in your bootstrap
$writer = new Zend_Log_Writer_Firebug();
$logger = new Zend_Log($writer);
 
// Use this in your model, view and controller files
$logger->log('This is a log message!', Zend_Log::INFO);

Simple and clean (it won’t work if Zend_Controller_Front is not used, look at manual).

Logging to files still should be used for information which will be required for further analysis, eg. payment gateway errors, but such information as execution time works perfectly in FirePHP. Users doesn’t care how fast page was rendered, but developers sometimes needs this information and in my opinion, using FirePHP sounds most reasonable. Have you tried it?

Scraping login requiring websites with cURL

Juozas — Mon, 23 Feb 2009 20:50:19 +0000

Scraping websites with XPath is very easy (read here), but how to scrape user’s friends list from social website if it can be viewed only when user is logged in?

What we need to do is to implement algorithm, which posts login and password fields to website login form and uses the same PHPSESSID id for further calls. For example, if login form is POSTed with 123 session id, then all requests with 123 session id could access users-only pages. This works because PHP (or other language) sets session data to loggedin=true for given session id.

But how you are going to do all this work with cookies and session id? Luckily, PHP has cURL extension which simplifies connecting to remote addresses, using cookies, staying in one session, POSTing data, etc. It’s really powerful library, which basically allows you to use all HTTP headers functionality.

For secure pages crawling, I’ve created very simple Secure_Crawler class, which works like this:

include ("crawler.php");
 
$crawler = new Secure_Crawler();
 
// Login to website
$crawler->login('my_username', 'secure_password');
 
// Get Content
$content = $crawler->get('http://www.example.com/secure/profile.php');
 
// modifications...

If you look at class source code, you would see that class has these specifications:

When Secure_Crawler instance is created, default cURL options are set
Login() method POSTs given credentials to login page (hard-coded)
Get(url) method loads page by given URL (previous session data is used)

Class itself is very easily extendible – as long as you pass Cookies file to cURL object, login information will (should) be used and all users-only content would be available.

Using similar class, I’ve pseudo-reverse engineered API. I needed to enter information to other website multiple times per day by hand because they didn’t offered any remote services (like XML-RPC or REST), so I created class which mimics API functionality. From outside it looks like normal API object, but inside code actually POSTs everything to actual website.

All websites works differently so you need to spend some time analysing how login form submission is handled on that specifix website. For example, maybe you need to use SSL protocol or even you own certificate. It depends and differs from site to site, but basics are the same – it will work as long as you are calm enough to tweak it’s work-flow.

PayPal payment with encryption

Juozas — Sun, 22 Feb 2009 14:34:40 +0000

Recently phpfour.com posted very interesting library for payment gateways. In my situation, PayPal is only used to pay for orders – cart and order setup is done in our shop, so I do not want to have additional problems with users changing orders numbers, amount to be paid, etc. Today I’m going to show how to encrypt PayPal transactions.

I chose to use Encrypted Website Payment, which allows you to encrypt all form fields and send them as one encrypted parameter. Only PayPal knows how to decrypt it, because it uses public key encryption technology (you need to upload your certificate in PayPal account).

My recommended PHP library for creating such buttons is written by Ivor Durham and is available on-line here. It’s not as flexible as phpfour.com one and is probably old, but it does what it needs to do. I have been using it for over a year now and haven’t had any problems (some hundreds payments).

To create encrypted button you need to write something like this:

$paypal = new PayPalEWP();
 
$paypal->setTempFileDirectory('/tmp');
// Certificate and private key
$paypal->setCertificate('mycompany_cert.pem', 'mycompany_key.pem');
// Uploaded certificate id
$paypal->setCertificateID('ABCDEFGHIJKL');
// PayPal certificate
$paypal->setPayPalCertificate('paypal_cert_sandbox.pem');
 
$parameters = array("cmd" => "_xclick",
      "business" => "sales@mycompany.com",
      "item_name" => "Order #ID",
      "amount" => "12.95",
      "no_shipping" => "1",
      "return" => "http://mycompany.com/paypal_ok.php",
      "cancel_return" => "http://mycompany.com/paypal_cancel.php",
      "no_note" => "1",
      "currency_code" => "USD",
      "bn" => "PP-BuyNowBF"
);
 
$encryptedButton = $paypal->encryptButton($parameters);
 
echo <<
 




END_HTML;

I have customized it a little bit to be modular (I use over 5 different payment gateways), but main concepts left the same. If you just starting PayPal integration, I recommend rewriting it to be more object-oriented and maybe integrating payments validation (which works the same as normal payments).

PayPal has sandbox mode and big manuals library – testing PayPal gateway is very easy and shouldn’t be a problem. I definitely recommend creating sandbox users (merchant and buyer) and playing with virtual money – it not only allows you to test gateway’s functionality, but feels very good to have unlimited amount of money.

To finish with, I recommend using encrypted PayPal buttons – additional security is not bad. How do you handle payments?

Web scraping with PHP and XPath

Juozas — Tue, 17 Feb 2009 20:23:04 +0000

When I was writing about how I use web scraping, I was still hadn’t tried using Xpath (shame on me). sssscripting blog responded to my article with very good and rich post about all sorts of different techniques for scraping (with Ruby examples) and after reading this post in Kore Nordmann blog I finally decided to try making something with Xpath.

It turned out, that using Xpath is extremely easy, really. When you master it, you can do everything in seconds. Yes, you need to know how XML works and how to write correct Xpath queries (brief explanation of Xpath syntax is available at W3Schools), but hey – these topics are in 1st year of university.

Also, there are good tools like XPath checker for Firefox which allows you to debug and test your queries without writing any code. Stupid to say, but XPath queries looks a lot like CSS selectors, but with much more power and flexibility. Without further talking, lets look at example (idea from Kore’s article):

 
 
$oldSetting = libxml_use_internal_errors( true ); 
libxml_clear_errors(); 
 
$html = new DOMDocument(); 
$html->loadHtmlFile(
    'http://www.bhphotovideo.com/c/shop/6222/SLR_Digital_Cameras.html'); 
 
$xpath = new DOMXPath( $html ); 
$links = $xpath->query( "//div[@class='productBlock clearfix']" ); 
 
$return = array();
 
foreach ( $links as $item ) {
	$newDom = new DOMDocument;
	$newDom->appendChild($newDom->importNode($item,true));
 
	$xpath = new DOMXPath( $newDom ); 
	$title = trim($xpath->query("//div[@id='productTitle']")
                   ->item(0)->nodeValue);
	$price = trim($xpath
                   ->query("//li[@class='price']/span[@class='value']")
                   ->item(0)->nodeValue);
 
	$return[] = array(
		'title' => $title,
		'price' => $price,
		);
} 
 
// Products array with title and price
print_r($return);
 
libxml_clear_errors(); 
libxml_use_internal_errors( $oldSetting ); 
 
?>

This code gets all products titles and prices from Bhphotovideo.com (read below) category page. You must have noticed that XPath queries are really simple: one selects all products, others selects only specific elements of each product. How fast you can get same results with plain regexp? I made this in 3 minutes (downloading XPath extension for Firefox, reading php manual, etc. included).

I used to write queries as regular expressions, but now I see that I’ve been just wasting time – using XPath is much more easier. I don’t know why I haven’t tried them sooner (maybe because of my believing, that XPath only works with correctly structured documents), but now I see that this technology is just awesome. I don’t know what to say more – web scraping with XPath is supper easy.

Bhphotovideo.com is really good shop and I chose it as example of scraping. I don’t encourage you to steal their information and it’s your responsibility to scrape only these sites, which allows it. My code it’s just an example and shouldn’t be used to affect Bhphotovideo.com sales.

Correct headers for dynamically generated content

Juozas — Sun, 15 Feb 2009 19:00:39 +0000

Static images have correct headers – Apache sends them by default. Different story is with all dynamic generating content – if you don’t send correct headers user’s browser will load it every time. It’s not always good, because generated thumbnails doesn’t change every time and should be cached in browser’s cache. If you want to reduce server load and increase loading speed you definitely need to send correct headers.

Our task is to give browser enough information to identify content, which later can be used to decided if content has changed, and if not – say it to browser without sending content again. Bigger picture is explained here, I really recommend analysing it a little bit – it gives better understanding how headers work and what they mean.

As you saw in headers map, there are many possible headers to send, but we can focus on three of them:

Etag. Unique identified for output data (maybe hash)
Last-Modified. Date of last data modification
Expires. When to expire content

Constructing algorithm is pretty simple, it can be explained in pseudo-code:

get request headers
   is content expired
      send content, new headers
   else is content different (Etag)
      send content, new headers
   else
      304 header, no content

I’ve created outputCacheHeaders(…) function which (available here) outputs all required headers and returns true if you need to output content or false if not. All you need to do is pass modification time, live time and data (two last ones are not required). Live time says how long should cache be kept and data is used to generate Etag. Very simple!

After enabling correct headers sending, my websites started to feel much faster, because HTTP requests takes much less time. Also, Yahoo performance tips are really worth reading, really. There are many great tips and I have been using many of them for a long time. These tricks are much much more significant than 0.1 s. faster PHP rendering time.

Web scraping – easy way to monitor market

Juozas — Sat, 14 Feb 2009 15:54:52 +0000

Sssssssscripting blog yesterday wrote about writing your own web crawler (in Ruby) and I immediately remember that I have done similar projects in past. Only difference is, I had developed web scrappers which were used to monitor e-commerce websites market. Believe me, when you run online shop, having ability to look at rivals prices in simple way is priceless.

Web scrapers which I wrote were not very dynamic – everyone of them was made for specific website. It may seem not very good practice, but having different spiders for different sites helps a lot – source code is dramatically simpler and tweaking one doesn’t affect others.

Currently, I have two spiders running every week (they have been running for half year). They have one task: go to website X and get all information about products they sell. Products price is most valuable information, so I cache it for later use (for information “price on other stores”, etc.).

Generally, writing web scraper is not very hard if you know how to write regexp’s fast (currently running web scrappers are only 100 lines each). Getting information from website is extremely easy if you know how to use regular expressions – website structure doesn’t change frequently, so correctly written expressions can last years. Or until server administrator blocks you, but if you really need this information, you can use N anonymous proxies.

Creating such spider involves analysing website structure and HTML source code – web scraper is basically “automatic copy paste” so if you know where to search for information in website, then you can simulate it code. For example, pseudo-code for price getting spider could look like this:

foreach category in categories
   goto category page
       pages = get {category} pages list
       foreach page in pages
           information = extract products from {pages}

Some people may think that it’s not legal. Probably it is. If you look at Legal issues in this article, you will find that:

Web scraping may be against the terms of use of some websites. The enforceability of these terms is unclear.

In my opinion, web scraping should be treated as legal thing if and only if you don’t directly use scraped information in your website. Scraping news, blog entries, etc. and showing them in your site is bad thing, but scraping information and using it somewhere in your back-end script is perfectly legal by me. What do you think?

Easy javascript packing with PHP

Juozas — Thu, 05 Feb 2009 00:14:00 +0000

“JavaScript packing” is method for reducing JavaScript files size by removing all unnecessary data (obfuscating) and compressing it’s contents. Most popular is Dean Edward’s packer, which transforms JavaScripts into something like this:

eval(function(p,a,c,k,e,r){...

To start with, it’s very easy to have dynamically packed files (no one wants to pack them by hand, packed versions are only useful for production, debugging them is harder) with PHP. One of many PHP packers is available on-line at http://joliclic.free.fr/php/javascript-packer/en/. It’s simple one-file script which works really well.

JavaScript packing is as simple as:

require 'class.JavaScriptPacker.php';
 
$src = 'my_script.js'; // Script file
$script = file_get_contents($src);
 
$packer = new JavaScriptPacker($script, 'Normal', true, false);
$packed = $packer->pack();
 
header ("Content-type: text/javascript; charset=utf-8");
header ("Content-Length: " . strlen($packed));
echo $packed;

* Parameters for JavaScriptPacker object can be easily tested in project website. Sometimes packed javascripts don’t work because of missing “;” symbols. I used FireBug extension for Firefox to find actual problematic lines in unpacked version and correct them.

Using this class I created controller for all my JavaScripts, which gets JavaScript name from GET call, checks for cached version and if packed version is not cached – creates 1 day cache and outputs it. Also, I pass JavaScript name in special form: “filename1-filename2-… .js” what let’s me merge multiple files into one packed version. (it can be optimized even more by storing cached versions as static files and accessing them directly)

Packer alone cannot make a big difference, but by combining it with:

multiple files merging
caching (correct headers)
then packing
putting JavaScripts at the bottom of document

gives very good results, not only decreasing HTTP calls count, but also time required to render whole website. Still, it’s very easy to implement whole system since all required libraries (packer, caching, etc.) are available on-line for free.