Differences between revisions 2 and 3
⇤ ← Revision 2 as of 2007-03-20 09:39:36 →
Size: 1326
Editor: nickcheng
Comment:
← Revision 3 as of 2007-03-20 09:41:07 → ⇥
Size: 1346
Editor: nickcheng
Comment:
Deletions are marked like this. Additions are marked like this.
Line 6: Line 6:
#!python
Line 12: Line 13:

[[BR]][[BR]]

python的robotparser模块

读一个Python的爬虫代码的时候看到一个名字叫做”robotparser”的模块, 查了一下, 是用来解析网站的”robots.txt”文件的. 用法很简单, 举个例子 

   1 rp = robotparser.RobotFileParser() 
   2 rp.set_url("http://www.musi-cal.com/robots.txt") 
   3 rp.read() 
   4 rp.can_fetch("*", "http://www.musi-cal.com/cgi-bin/search?city=San+Francisco")  # return False 
   5 rp.can_fetch("*", "http://www.musi-cal.com/")  # return True

BRBR 程序很简单. 首先将类实例化, 然后用”set_url”方法设置”robots.txt”文件的路径, 这个文件一般都在网站的根目录. 接着调用”read”方法来解析这个文件. 最后就是用”can_fetch”方法来判断这个网站是否让你的程序(爬虫)访问某个网站内的链接了! BRBR 如果这个网站没有”robots.txt”文件的话, 在”read”方法后查看”rp.errcode”属性, 会看到404的返回值, 如果有这个文件就是200的返回值了! BRBR 相关链接

nickcheng/2007-03-20 (last edited 2009-12-25 07:11:13 by localhost)