I read your articles on Web Scraping with Python and I’m wondering if over the years you’ve come across more advanced ways to solve your problems.

More specifically, do you know of any practical way to extract the same type of data (ie. camera products) from multiple websites of varying structure, without having to custom code for each website (or perhaps very minimal custom code).

I know there are some websites capable of doing this all automatically. Take for instance http://www.vast.com – they have loads of data.

Do you have any insights into this kind of technology?

Thanks Sig – I’d love to hear what you have to say…

Here is my answer :

First of all, the ideal solution remains to have the sites publish this data in a structured way (say RDF/OWL or JSON for instance). Most often, if they don’t publish in such a structured way, it may mean that they don’t allow you to scrape their data and you may get into legal troubles because of copyright laws.

That being said, there have been attempts at easing the process of custom-coding the scraping of specific sites. The 2 most interesting solutions I played with (a couple of years ago) are Openkapow and Dapper.

The advantage of Openkapow on custom script-based scraping is that it offers a rich scrape-robot development environment (GUI) which eases the process of analyzing the HTML structure. But running and exploiting these robots has revealed to be not as flexible and easy as running your own homemade scrapers.

Dapper has a significant strength : it allows structure to be learnt by the machine based on examples. You provide dappers with several samples of pages to extract data from and it “automagically” identifies recurrent HTML patterns which allow it to extract data. There must be some machine learning algorithm behind it AFAICS. But the drawbacks of dapper are : these algorithms are OK for 80% cases but the other 20% won’t be parseable by Dapper, and Dapper requires the page to be a list of many items (think paginated list of a search results). Dapper does not seem to be suitable for the technical sheet of a camera for instance. And Dapper scrapers can’t easily be combined : you can’t easily script the navigation in a complex site unless you combine dapper with things like Yahoo pipes.

As a conclusion, I would say that simple and easily accessible paginated lists of results deserve some dappering some hesitation. Openkapow is the tool to use if you can’t script by yourself. But the definite answer to complex and robust scraping remains homemade scripts.

There may be other valuable alternatives I don’t know. I have not been spending much time on scraping since I wrote this article.

Please come and share the results of your own experiments as further comments !

I’d hope I will have some opportunity to refresh this code a bit and to extend its functionalities. Unfortunately, at the moment, I am struggling with these clauses in my new job contract that let my employer kind-of own anything I create (in case there would be some software patents to produce out of it…)… :(

If your installation doesn’t have cTypes installed, then uTidylib has its own version that it tries to use (in pvt_ctypes). But that version didn’t work on my computer either – i think it is designed for python 2.3.

Solution, just install the latest cTypes library from sourceforge on your computer, and then utidy will use that version, and not its own private version. I hope that helps.

Sig – thx much for the code. I’ve been playing with it extensively for some time now.

I apologize for having forgotten to answer you sooner (job switch + holidays in-between)…

The traceback you provides says that your tidy lib complains about some DLL that can’t get loaded. I suggest that you check your uTidylib installation. Maybe try to uninstall/re-install and see if it fixes your problem.

If it does not work, I suggest that you ask for support to the uTidylib project team.

Once again I sincerely apologize and I hope that you could fix this problem.

The error messages output to shell console is following:
Traceback (most recent call last):
File “D:\download\python\web_parser\web_parser.py”, line 8, in ?
import tidy
File “D:\Python24\Lib\site-packages\tidy\__init__.py”, line 38, in ?
from tidy.lib import parse, parseString
File “D:\Python24\Lib\site-packages\tidy\lib.py”, line 16, in ?
import ctypes
File “D:\Python24\lib\site-packages\tidy\pvt_ctypes\ctypes.zip\ctypes\__init__.py”, line 13, in ?
ImportError: DLL load failed: 找不到指定的模块。

Can you help me,thanks.

So anyway, yes, you can certainly parse HTML generated by ASP.Net pages.

try:
option = finish['pseudotal:option']
path.append(option) # my change

near the end of template.shortest_path(), and it works, becuase extract() looks for that option value. what u think?

Thanks all for your positive feedback : keep on commenting, I enjoy that a lot ! :-)

It works right out of box, and made web scrapping so much easier. Great work. Thanks!!