It turned out, that using Xpath is extremely easy, really. When you master it, you can do everything in seconds. Yes, you need to know how XML works and how to write correct Xpath queries (brief explanation of Xpath syntax is available at W3Schools), but hey – these topics are in 1st year of university.

Also, there are good tools like XPath checker for Firefox which allows you to debug and test your queries without writing any code. Stupid to say, but XPath queries looks a lot like CSS selectors, but with much more power and flexibility. Without further talking, lets look at example (idea from Kore’s article):

<?php 
 
$oldSetting = libxml_use_internal_errors( true ); 
libxml_clear_errors(); 
 
$html = new DOMDocument(); 
$html->loadHtmlFile(
    'http://www.bhphotovideo.com/c/shop/6222/SLR_Digital_Cameras.html'); 
 
$xpath = new DOMXPath( $html ); 
$links = $xpath->query( "//div[@class='productBlock clearfix']" ); 
 
$return = array();
 
foreach ( $links as $item ) {
	$newDom = new DOMDocument;
	$newDom->appendChild($newDom->importNode($item,true));
 
	$xpath = new DOMXPath( $newDom ); 
	$title = trim($xpath->query("//div[@id='productTitle']")
                   ->item(0)->nodeValue);
	$price = trim($xpath
                   ->query("//li[@class='price']/span[@class='value']")
                   ->item(0)->nodeValue);
 
	$return[] = array(
		'title' => $title,
		'price' => $price,
		);
} 
 
// Products array with title and price
print_r($return);
 
libxml_clear_errors(); 
libxml_use_internal_errors( $oldSetting ); 
 
?>

This code gets all products titles and prices from Bhphotovideo.com (read below) category page. You must have noticed that XPath queries are really simple: one selects all products, others selects only specific elements of each product. How fast you can get same results with plain regexp? I made this in 3 minutes (downloading XPath extension for Firefox, reading php manual, etc. included).

I used to write queries as regular expressions, but now I see that I’ve been just wasting time – using XPath is much more easier. I don’t know why I haven’t tried them sooner (maybe because of my believing, that XPath only works with correctly structured documents), but now I see that this technology is just awesome. I don’t know what to say more – web scraping with XPath is supper easy.

Bhphotovideo.com is really good shop and I chose it as example of scraping. I don’t encourage you to steal their information and it’s your responsibility to scrape only these sites, which allows it. My code it’s just an example and shouldn’t be used to affect Bhphotovideo.com sales.

Web scrapers which I wrote were not very dynamic – everyone of them was made for specific website. It may seem not very good practice, but having different spiders for different sites helps a lot – source code is dramatically simpler and tweaking one doesn’t affect others.

Currently, I have two spiders running every week (they have been running for half year). They have one task: go to website X and get all information about products they sell. Products price is most valuable information, so I cache it for later use (for information “price on other stores”, etc.).

Generally, writing web scraper is not very hard if you know how to write regexp’s fast (currently running web scrappers are only 100 lines each). Getting information from website is extremely easy if you know how to use regular expressions – website structure doesn’t change frequently, so correctly written expressions can last years. Or until server administrator blocks you, but if you really need this information, you can use N anonymous proxies.

Creating such spider involves analysing website structure and HTML source code – web scraper is basically “automatic copy paste” so if you know where to search for information in website, then you can simulate it code. For example, pseudo-code for price getting spider could look like this:

foreach category in categories
   goto category page
       pages = get {category} pages list
       foreach page in pages
           information = extract products from {pages}

Some people may think that it’s not legal. Probably it is. If you look at Legal issues in this article, you will find that:

Web scraping may be against the terms of use of some websites. The enforceability of these terms is unclear.

In my opinion, web scraping should be treated as legal thing if and only if you don’t directly use scraped information in your website. Scraping news, blog entries, etc. and showing them in your site is bad thing, but scraping information and using it somewhere in your back-end script is perfectly legal by me. What do you think?