Here is a set of resources for scraping the web with the help of Python. The best solution seems to be Mechanize plus Beautiful Soup.
See also :
- ClientTable (see also this comment)
- DOMForm
- Pull parser
- wwwsearch ’s FAQ
- Simon Willison’s weblog on this topic
- Dive Into Python’s tutorial on HTML processing
Off-topic : proxomitron looks like a nice (python-friendly ?) filtering proxy.
- SMIL-animated SVG for accessible textbooks...
[...] wling design is based on the John L. Lee’s mechanize framework for python. There are other tools available in Python (see also . And several other approaches are available when you want to [...]
Check also Web scraping with Python (part II) where I present a python app that helps you scraping the web.
Beautifulsoup is ok, but it does have lexical issues on certain constructs. Nor it is consistent.
Something I find much better is lxml and PyQuery. lxml is extremely fast. PyQuery’s advantage is that if you know jQuery already you bypass the learning curve in using the tool.
JohnMc, thanks for pointing to PyQuery and lxml. I was a bit concerned that lxml would not handle malformed HTML files. But it now can interface with BeautifulSoup and use its parsing abilities for malformed HTML files. Here is the lxml module for this :
http://codespeak.net/lxml/elementsoup.html