Comments on: Web scraping – easy way to monitor market

By: Xpath

Xpath — Fri, 17 Apr 2009 22:09:53 +0000

xpath is overrated shit.

actually any language with a solid parsing Library will be easier to use, and faster.

By: Daily Digest for 2009-02-18 | Pedro Trindade

Daily Digest for 2009-02-18 | Pedro Trindade — Thu, 19 Feb 2009 08:12:51 +0000

[...] Web scraping – easy way to monitor market | Juozas devBlog [...]

By: Web scraping with PHP and XPath | Juozas devBlog

Web scraping with PHP and XPath | Juozas devBlog — Tue, 17 Feb 2009 20:23:37 +0000

[...] I was writing about how I use web scraping, I was still hadn’t tried using Xpath (shame on me). sssscripting blog responded to my [...]

By: Geo

Geo — Sun, 15 Feb 2009 15:18:09 +0000

I also wrote an article related to web scraping . You can find it here : http://ssscripting.wordpress.com/2009/02/15/web-scraping-techniques/

By: Juozas

Juozas — Sat, 14 Feb 2009 17:36:39 +0000

Today I found another article here (http://kore-nordmann.de/blog/0081_parse_html_extract_data_from_html.html).

It seems that Dom PHP libraries can work with malformed documents. Still, both sites which I crawl are very bad made (they made 5-10 years ago) so Xpath can have problems.

However, when I will be doing another spider, I will definitely use Xpath. Probably I’ll try next week.

By: Geo

Geo — Sat, 14 Feb 2009 17:18:17 +0000

I forgot to mention about that . You should pass the source’s content through tidy , or BeautifulSoup , or a parser that can work with malformed documents . I know that XPath can become useless if the site’s structure changes , but it’s very easy to replace .

By: Juozas

Juozas — Sat, 14 Feb 2009 16:49:49 +0000

Yes, XPath is wonderful thing, but always worry about document validity? What if document is badly formed?

Also, do you use XPath by “html/body/table/tr/td/span…”? If so, it can be breakable easier.

I haven’t used it much (only for well structured xml) so regular expressions for me does the job quite good :)

By: Geo

Geo — Sat, 14 Feb 2009 16:10:46 +0000

I find it easier to scrape using XPath lately . Firefox has some great addons for retrieving a path for a element in a page . By using that with a xpath library , the development speed is blazing fast . There is a ruby module , Scrubyt , built on top of mechanize , which simplifies that even more . As for the legal issues , I think that if users respect robots.txt , everything should be okay .