Comments on: Web scraping with Python (part II) http://www.akasig.org/2005/03/11/web-scraping-with-python-part-ii/ Innover, servir, entreprendre. Thu, 04 Dec 2008 01:17:50 +0000 http://wordpress.org/?v=2.6.2 By: Sig http://www.akasig.org/2005/03/11/web-scraping-with-python-part-ii/#comment-44853 Sig Mon, 24 Apr 2006 15:02:33 +0000 http://sig.levillage.org/?p=599#comment-44853 Juancho: thx for your thx. :) I'd hope I will have some opportunity to refresh this code a bit and to extend its functionalities. Unfortunately, at the moment, I am struggling with these clauses in my new job contract that let my employer kind-of own anything I create (in case there would be some software patents to produce out of it...)... :( Juancho: thx for your thx. :)

I’d hope I will have some opportunity to refresh this code a bit and to extend its functionalities. Unfortunately, at the moment, I am struggling with these clauses in my new job contract that let my employer kind-of own anything I create (in case there would be some software patents to produce out of it…)… :(

]]> By: Juancho http://www.akasig.org/2005/03/11/web-scraping-with-python-part-ii/#comment-44599 Juancho Wed, 29 Mar 2006 01:08:25 +0000 http://sig.levillage.org/?p=599#comment-44599 I know this is way after the fact, but Adam, I had that problem as well. uTidy lib wraps a C lib (i think) and therefore uses the Ctypes package to execute C stuff from Python. If your installation doesn't have cTypes installed, then uTidylib has its own version that it tries to use (in pvt_ctypes). But that version didn't work on my computer either - i think it is designed for python 2.3. Solution, just install the latest cTypes library from sourceforge on your computer, and then utidy will use that version, and not its own private version. I hope that helps. Sig - thx much for the code. I've been playing with it extensively for some time now. I know this is way after the fact, but Adam, I had that problem as well. uTidy lib wraps a C lib (i think) and therefore uses the Ctypes package to execute C stuff from Python.

If your installation doesn’t have cTypes installed, then uTidylib has its own version that it tries to use (in pvt_ctypes). But that version didn’t work on my computer either - i think it is designed for python 2.3.

Solution, just install the latest cTypes library from sourceforge on your computer, and then utidy will use that version, and not its own private version. I hope that helps.

Sig - thx much for the code. I’ve been playing with it extensively for some time now.

]]> By: Sig http://www.akasig.org/2005/03/11/web-scraping-with-python-part-ii/#comment-40987 Sig Tue, 13 Sep 2005 13:08:54 +0000 http://sig.levillage.org/?p=599#comment-40987 Adam, I apologize for having forgotten to answer you sooner (job switch + holidays in-between)... The traceback you provides says that your tidy lib complains about some DLL that can't get loaded. I suggest that you check your uTidylib installation. Maybe try to uninstall/re-install and see if it fixes your problem. If it does not work, I suggest that you ask for support to <a href="http://developer.berlios.de/projects/utidylib" rel="nofollow">the uTidylib project team</a>. Once again I sincerely apologize and I hope that you could fix this problem. Adam,

I apologize for having forgotten to answer you sooner (job switch + holidays in-between)…

The traceback you provides says that your tidy lib complains about some DLL that can’t get loaded. I suggest that you check your uTidylib installation. Maybe try to uninstall/re-install and see if it fixes your problem.

If it does not work, I suggest that you ask for support to the uTidylib project team.

Once again I sincerely apologize and I hope that you could fix this problem.

]]> By: adam http://www.akasig.org/2005/03/11/web-scraping-with-python-part-ii/#comment-36009 adam Fri, 03 Jun 2005 08:18:58 +0000 http://sig.levillage.org/?p=599#comment-36009 Dear Sig, Thanks for your hard work! I have a trouble when I use your the project 's ZIP distribution . I installed these packages obey your guide step by step,but when I run the "web_parser",I had been told "ImportError: DLL load failed". The error messages output to shell console is following: Traceback (most recent call last): File "D:\download\python\web_parser\web_parser.py", line 8, in ? import tidy File "D:\Python24\Lib\site-packages\tidy\__init__.py", line 38, in ? from tidy.lib import parse, parseString File "D:\Python24\Lib\site-packages\tidy\lib.py", line 16, in ? import ctypes File "D:\Python24\lib\site-packages\tidy\pvt_ctypes\ctypes.zip\ctypes\__init__.py", line 13, in ? ImportError: DLL load failed: 找不到指定的模块。 Can you help me,thanks. Dear Sig,
Thanks for your hard work!
I have a trouble when I use your the project ’s ZIP distribution .
I installed these packages obey your guide step by step,but when I run the “web_parser”,I had been told “ImportError: DLL load failed”.

The error messages output to shell console is following:
Traceback (most recent call last):
File “D:\download\python\web_parser\web_parser.py”, line 8, in ?
import tidy
File “D:\Python24\Lib\site-packages\tidy\__init__.py”, line 38, in ?
from tidy.lib import parse, parseString
File “D:\Python24\Lib\site-packages\tidy\lib.py”, line 16, in ?
import ctypes
File “D:\Python24\lib\site-packages\tidy\pvt_ctypes\ctypes.zip\ctypes\__init__.py”, line 13, in ?
ImportError: DLL load failed: 找不到指定的模块。

Can you help me,thanks.

]]> By: Sig http://www.akasig.org/2005/03/11/web-scraping-with-python-part-ii/#comment-35834 Sig Mon, 30 May 2005 13:58:46 +0000 http://sig.levillage.org/?p=599#comment-35834 Adam : yes, you may. This piece of code does not care about the technology that generates your web pages. Indeed, once your ASP.Net code has been accessed and run by your web server, the output sent to the web browser is pure HTML. The only technology that it may have problems with is Javascript when there is too much of it in a web page (when most of the HTML is generated at run-time on client-side). Other limit : when your HTML is really really far from valid, then you may have to tidy it with an included pre-processor (see the source code for more explanation on this point). So anyway, yes, you can certainly parse HTML generated by ASP.Net pages. Adam : yes, you may. This piece of code does not care about the technology that generates your web pages. Indeed, once your ASP.Net code has been accessed and run by your web server, the output sent to the web browser is pure HTML. The only technology that it may have problems with is Javascript when there is too much of it in a web page (when most of the HTML is generated at run-time on client-side). Other limit : when your HTML is really really far from valid, then you may have to tidy it with an included pre-processor (see the source code for more explanation on this point).

So anyway, yes, you can certainly parse HTML generated by ASP.Net pages.

]]> By: adam http://www.akasig.org/2005/03/11/web-scraping-with-python-part-ii/#comment-35833 adam Mon, 30 May 2005 08:51:16 +0000 http://sig.levillage.org/?p=599#comment-35833 May I use it to parse asp.net pages? May I use it to parse asp.net pages?

]]> By: Sig http://www.akasig.org/2005/03/11/web-scraping-with-python-part-ii/#comment-35644 Sig Tue, 24 May 2005 14:54:23 +0000 http://sig.levillage.org/?p=599#comment-35644 Alex : it sounds nice. Do you have some piece of code so that we can try it ? Alex : it sounds nice. Do you have some piece of code so that we can try it ?

]]> By: alex http://www.akasig.org/2005/03/11/web-scraping-with-python-part-ii/#comment-35562 alex Mon, 23 May 2005 13:38:05 +0000 http://sig.levillage.org/?p=599#comment-35562 Thanks. Currently working further on a similar idea. Compare two pages from the same site to get the modifier fields. With very litle programmed inteligence you can then get article description, date, title, price ... a la Froogle. Thanks. Currently working further on a similar idea. Compare two pages from the same site to get the modifier fields. With very litle programmed inteligence you can then get article description, date, title, price … a la Froogle.

]]> By: taprackbang http://www.akasig.org/2005/03/11/web-scraping-with-python-part-ii/#comment-28153 taprackbang Wed, 13 Apr 2005 02:57:54 +0000 http://sig.levillage.org/?p=599#comment-28153 missing template part: <div class="buying" pseudotal:content="title_authors" pseudotal:option="FULL_CONTENT"> missing template part:
<div class=”buying” pseudotal:content=”title_authors” pseudotal:option=”FULL_CONTENT”>

]]> By: taprackbang http://www.akasig.org/2005/03/11/web-scraping-with-python-part-ii/#comment-28152 taprackbang Wed, 13 Apr 2005 02:56:36 +0000 http://sig.levillage.org/?p=599#comment-28152 since title and authors are mixed together in the div node, so I want to get the whole div tag and process it myself. In my template, i have . but my script still treats title_authors regularly and were not returning whole html string. after I put in my change, then it works return the whole div tag. since title and authors are mixed together in the div node, so I want to get the whole div tag and process it myself. In my template, i have . but my script still treats title_authors regularly and were not returning whole html string. after I put in my change, then it works return the whole div tag.

]]> By: Sig http://www.akasig.org/2005/03/11/web-scraping-with-python-part-ii/#comment-27967 Sig Tue, 12 Apr 2005 08:55:27 +0000 http://sig.levillage.org/?p=599#comment-27967 taprackbang : thank you for your suggestion ; it looks better than the current version. But why do you say that the option wasn't working for you ? Please describe the problem you were encountering. taprackbang : thank you for your suggestion ; it looks better than the current version. But why do you say that the option wasn’t working for you ? Please describe the problem you were encountering.

]]> By: taprackbang http://www.akasig.org/2005/03/11/web-scraping-with-python-part-ii/#comment-25775 taprackbang Thu, 07 Apr 2005 00:35:44 +0000 http://sig.levillage.org/?p=599#comment-25775 the option wasn't working for me. then I put one line: try: option = finish['pseudotal:option'] path.append(option) # my change near the end of template.shortest_path(), and it works, becuase extract() looks for that option value. what u think? the option wasn’t working for me. then I put one line:

try:
option = finish['pseudotal:option']
path.append(option) # my change

near the end of template.shortest_path(), and it works, becuase extract() looks for that option value. what u think?

]]> By: Sig http://www.akasig.org/2005/03/11/web-scraping-with-python-part-ii/#comment-19566 Sig Wed, 16 Mar 2005 11:01:36 +0000 http://sig.levillage.org/?p=599#comment-19566 Quick notice : Leonard Richardson (BeautifulSoup's author) gave me a positive feedback about my suggested modifications. He is to include these features into the version 2.0 of BeautifulSoup he is working on. I don't plan on making any additional contribution on BeautifulSoup in the near future but I will certainly update this web parser as soon as v.2.0 of BS is available. Thank you Leonard ! Quick notice : Leonard Richardson (BeautifulSoup’s author) gave me a positive feedback about my suggested modifications. He is to include these features into the version 2.0 of BeautifulSoup he is working on. I don’t plan on making any additional contribution on BeautifulSoup in the near future but I will certainly update this web parser as soon as v.2.0 of BS is available. Thank you Leonard !

]]> By: Sig http://www.akasig.org/2005/03/11/web-scraping-with-python-part-ii/#comment-19466 Sig Mon, 14 Mar 2005 08:45:49 +0000 http://sig.levillage.org/?p=599#comment-19466 taprackbang : I should package this stuff so that it gets installed more easily but I don't have any precise idea about how to do this. Regarding malformed HTML (Amazon's mismatched form tag), I suggest that you write an appropriate regex in a "MyProcessor" class-like preprocessor <a href="http://sig.levillage.org/?p=588" rel="nofollow">as suggested in the first part of this article</a>. Your pre-processor will help crawling Amazon and saving its pages in a more adequate format. Thanks all for your positive feedback : keep on commenting, I enjoy that a lot ! :-) taprackbang : I should package this stuff so that it gets installed more easily but I don’t have any precise idea about how to do this. Regarding malformed HTML (Amazon’s mismatched form tag), I suggest that you write an appropriate regex in a “MyProcessor” class-like preprocessor as suggested in the first part of this article. Your pre-processor will help crawling Amazon and saving its pages in a more adequate format.

Thanks all for your positive feedback : keep on commenting, I enjoy that a lot ! :-)

]]> By: taprackbang http://www.akasig.org/2005/03/11/web-scraping-with-python-part-ii/#comment-19343 taprackbang Sat, 12 Mar 2005 17:44:05 +0000 http://sig.levillage.org/?p=599#comment-19343 This is great. After spending great deal of time to setup all the prerequisite packages, i was able to scrape the amazon's webpage to get book title, price easily. However, amazon's webpage has mismatched form tag which caused tidy to choke, so I had to manually remove all the form tags before tidy and web_parser function calls. It works right out of box, and made web scrapping so much easier. Great work. Thanks!! This is great. After spending great deal of time to setup all the prerequisite packages, i was able to scrape the amazon’s webpage to get book title, price easily. However, amazon’s webpage has mismatched form tag which caused tidy to choke, so I had to manually remove all the form tags before tidy and web_parser function calls.

It works right out of box, and made web scrapping so much easier. Great work. Thanks!!

]]> By: kedai http://www.akasig.org/2005/03/11/web-scraping-with-python-part-ii/#comment-19334 kedai Sat, 12 Mar 2005 15:04:09 +0000 http://sig.levillage.org/?p=599#comment-19334 nice. thanks for the link to beautiful soup. probably will try and use that as a parser for KebasData (a zope product to scrape the web, what else). currently, KebasData uses regex, and as noted by Leornard Richardson, regex is a double edge sword; it can help and it can also cause more trouble ;) nice. thanks for the link to beautiful soup. probably will try and use that as a parser for KebasData (a zope product to scrape the web, what else).

currently, KebasData uses regex, and as noted by Leornard Richardson, regex is a double edge sword; it can help and it can also cause more trouble ;)

]]> By: Reinout van Rees http://www.akasig.org/2005/03/11/web-scraping-with-python-part-ii/#comment-18943 Reinout van Rees Fri, 11 Mar 2005 12:54:41 +0000 http://sig.levillage.org/?p=599#comment-18943 Now that's a nice solution! Halfway your article I had a real AH!!!-moment : this is simply a good idea. http://vanrees.org/weblog/1110545603 Now that’s a nice solution! Halfway your article I had a real AH!!!-moment : this is simply a good idea. http://vanrees.org/weblog/1110545603

]]>