Juozas devBlog » security

HTML filtering and XSS protection

Juozas — Sat, 21 Mar 2009 20:40:24 +0000

If you have been programming websites long enough you would know that user input is first think to worry about when thinking about security. It’s really hard to decide what data is acceptable, especially when user has permission to insert HTML content through form.

For example, if you are developing CMS you need to make sure that user input don’t break whole template. But that’s not so easy, because you need very clever HTML validations as even one missing closing tag for

can brake website’s layout completely. Editors like TinyMCE can check and try to fix errors, but in my experience, they sometimes create more of them.

However, problem can be solved, and quite easily. Almost a year ago I was reading some random blog when I find out about HTML Purifier. Basically, it’s library which can filter and fix any HTML. Compared to other libraries, it looks very promising, but since then I haven’t had a chance to test it – other libraries have been working fine.

Today I was working with web scrapper again and ended up stuck because of very badly formatted HTML. When regular expressions are used, code validity isn’t (shouldn’t) a case at all, but XPath fails immediately. I tried simplifying queries, hard-coded source fixing, but all that required so many effort that I introduced Purifier filter between source fetching and DOM constructing. It worked!

require_once 'HTMLPurifier.includes.php';
 
$config = HTMLPurifier_Config::createDefault();
 
$config->set('HTML', 'Doctype', 'XHTML 1.0 Transitional');
$config->set('HTML', 'TidyLevel', 'heavy');
// Don't remove IDs ()
$config->set('Attr', 'EnableID', true);
 
$obj = new HTMLPurifier($config);
 
$clean_html = $obj->purify($html);

In this example I chose worst way – include all files. Library uses PEAR-like directory structure so simple auto-loader can include all required files in background, but for simplicity it’s not used here. This sample code filters $html variable using XHTML 1.0 and does heavy level tidying (quite clear from source code itself).

XSS? Purifier protects from them also – full list of tests. Library is also highly customizable (configuration manual), but documentation is not very clear – I have spent more than a hour trying to make it to return HTML with part. I haven’t found any nice solution (maybe because the library is not made for such things).

HTML Purifier contains about 350 files so it’s relatively big library, however it performs good and shouldn’t kill you web server. Today I Purified and using XPath extracted information from more than 1000 pages and it worked really stable – none of the results where filtered unexpectedly. I definitely recommend it for HTML inputs filtering because it just does wonderful job – you can try it online here.

Scraping login requiring websites with cURL

Juozas — Mon, 23 Feb 2009 20:50:19 +0000

Scraping websites with XPath is very easy (read here), but how to scrape user’s friends list from social website if it can be viewed only when user is logged in?

What we need to do is to implement algorithm, which posts login and password fields to website login form and uses the same PHPSESSID id for further calls. For example, if login form is POSTed with 123 session id, then all requests with 123 session id could access users-only pages. This works because PHP (or other language) sets session data to loggedin=true for given session id.

But how you are going to do all this work with cookies and session id? Luckily, PHP has cURL extension which simplifies connecting to remote addresses, using cookies, staying in one session, POSTing data, etc. It’s really powerful library, which basically allows you to use all HTTP headers functionality.

For secure pages crawling, I’ve created very simple Secure_Crawler class, which works like this:

include ("crawler.php");
 
$crawler = new Secure_Crawler();
 
// Login to website
$crawler->login('my_username', 'secure_password');
 
// Get Content
$content = $crawler->get('http://www.example.com/secure/profile.php');
 
// modifications...

If you look at class source code, you would see that class has these specifications:

When Secure_Crawler instance is created, default cURL options are set
Login() method POSTs given credentials to login page (hard-coded)
Get(url) method loads page by given URL (previous session data is used)

Class itself is very easily extendible – as long as you pass Cookies file to cURL object, login information will (should) be used and all users-only content would be available.

Using similar class, I’ve pseudo-reverse engineered API. I needed to enter information to other website multiple times per day by hand because they didn’t offered any remote services (like XML-RPC or REST), so I created class which mimics API functionality. From outside it looks like normal API object, but inside code actually POSTs everything to actual website.

All websites works differently so you need to spend some time analysing how login form submission is handled on that specifix website. For example, maybe you need to use SSL protocol or even you own certificate. It depends and differs from site to site, but basics are the same – it will work as long as you are calm enough to tweak it’s work-flow.

PayPal payment with encryption

Juozas — Sun, 22 Feb 2009 14:34:40 +0000

Recently phpfour.com posted very interesting library for payment gateways. In my situation, PayPal is only used to pay for orders – cart and order setup is done in our shop, so I do not want to have additional problems with users changing orders numbers, amount to be paid, etc. Today I’m going to show how to encrypt PayPal transactions.

I chose to use Encrypted Website Payment, which allows you to encrypt all form fields and send them as one encrypted parameter. Only PayPal knows how to decrypt it, because it uses public key encryption technology (you need to upload your certificate in PayPal account).

My recommended PHP library for creating such buttons is written by Ivor Durham and is available on-line here. It’s not as flexible as phpfour.com one and is probably old, but it does what it needs to do. I have been using it for over a year now and haven’t had any problems (some hundreds payments).

To create encrypted button you need to write something like this:

$paypal = new PayPalEWP();
 
$paypal->setTempFileDirectory('/tmp');
// Certificate and private key
$paypal->setCertificate('mycompany_cert.pem', 'mycompany_key.pem');
// Uploaded certificate id
$paypal->setCertificateID('ABCDEFGHIJKL');
// PayPal certificate
$paypal->setPayPalCertificate('paypal_cert_sandbox.pem');
 
$parameters = array("cmd" => "_xclick",
      "business" => "sales@mycompany.com",
      "item_name" => "Order #ID",
      "amount" => "12.95",
      "no_shipping" => "1",
      "return" => "http://mycompany.com/paypal_ok.php",
      "cancel_return" => "http://mycompany.com/paypal_cancel.php",
      "no_note" => "1",
      "currency_code" => "USD",
      "bn" => "PP-BuyNowBF"
);
 
$encryptedButton = $paypal->encryptButton($parameters);
 
echo <<
 




END_HTML;

I have customized it a little bit to be modular (I use over 5 different payment gateways), but main concepts left the same. If you just starting PayPal integration, I recommend rewriting it to be more object-oriented and maybe integrating payments validation (which works the same as normal payments).

PayPal has sandbox mode and big manuals library – testing PayPal gateway is very easy and shouldn’t be a problem. I definitely recommend creating sandbox users (merchant and buyer) and playing with virtual money – it not only allows you to test gateway’s functionality, but feels very good to have unlimited amount of money.

To finish with, I recommend using encrypted PayPal buttons – additional security is not bad. How do you handle payments?