When I was writing about how I use web scraping, I was still hadn’t tried using Xpath (shame on me). sssscripting blog responded to my article with very good and rich post about all sorts of different techniques for scraping (with Ruby examples) and after reading this post in Kore Nordmann blog I finally decided to try making something with Xpath.
It turned out, that using Xpath is extremely easy, really. When you master it, you can do everything in seconds. Yes, you need to know how XML works and how to write correct Xpath queries (brief explanation of Xpath syntax is available at W3Schools), but hey – these topics are in 1st year of university.
Also, there are good tools like XPath checker for Firefox which allows you to debug and test your queries without writing any code. Stupid to say, but XPath queries looks a lot like CSS selectors, but with much more power and flexibility. Without further talking, lets look at example (idea from Kore’s article):
<?php $oldSetting = libxml_use_internal_errors( true ); libxml_clear_errors(); $html = new DOMDocument(); $html->loadHtmlFile( 'http://www.bhphotovideo.com/c/shop/6222/SLR_Digital_Cameras.html'); $xpath = new DOMXPath( $html ); $links = $xpath->query( "//div[@class='productBlock clearfix']" ); $return = array(); foreach ( $links as $item ) { $newDom = new DOMDocument; $newDom->appendChild($newDom->importNode($item,true)); $xpath = new DOMXPath( $newDom ); $title = trim($xpath->query("//div[@id='productTitle']") ->item(0)->nodeValue); $price = trim($xpath ->query("//li[@class='price']/span[@class='value']") ->item(0)->nodeValue); $return[] = array( 'title' => $title, 'price' => $price, ); } // Products array with title and price print_r($return); libxml_clear_errors(); libxml_use_internal_errors( $oldSetting ); ?>
This code gets all products titles and prices from Bhphotovideo.com (read below) category page. You must have noticed that XPath queries are really simple: one selects all products, others selects only specific elements of each product. How fast you can get same results with plain regexp? I made this in 3 minutes (downloading XPath extension for Firefox, reading php manual, etc. included).
I used to write queries as regular expressions, but now I see that I’ve been just wasting time – using XPath is much more easier. I don’t know why I haven’t tried them sooner (maybe because of my believing, that XPath only works with correctly structured documents), but now I see that this technology is just awesome. I don’t know what to say more – web scraping with XPath is supper easy.
Bhphotovideo.com is really good shop and I chose it as example of scraping. I don’t encourage you to steal their information and it’s your responsibility to scrape only these sites, which allows it. My code it’s just an example and shouldn’t be used to affect Bhphotovideo.com sales.
