Web scraping with Python

Here is a set of resources for scraping the web with the help of Python. The best solution seems to be Mechanize plus Beautiful Soup.

See also :

Off-topic : proxomitron looks like a nice (python-friendly ?) filtering proxy.

5 réflexions au sujet de « Web scraping with Python »

  1. Ping : AkaSig » Web scraping with python (part 1 : crawling)

  2. JohnMc

    Beautifulsoup is ok, but it does have lexical issues on certain constructs. Nor it is consistent.

    Something I find much better is lxml and PyQuery. lxml is extremely fast. PyQuery’s advantage is that if you know jQuery already you bypass the learning curve in using the tool.

  3. Sig Auteur de l’article

    JohnMc, thanks for pointing to PyQuery and lxml. I was a bit concerned that lxml would not handle malformed HTML files. But it now can interface with BeautifulSoup and use its parsing abilities for malformed HTML files. Here is the lxml module for this :


Les commentaires sont fermés.