Differences between revisions 1 and 2
⇤ ← Revision 1 as of 2007-03-20 09:37:41 →
Size: 1318
Editor: nickcheng
Comment:
← Revision 2 as of 2007-03-20 09:39:36 → ⇥
Size: 1326
Editor: nickcheng
Comment:
Deletions are marked like this. Additions are marked like this.
Line 15: Line 15:
[BR][BR] [[BR]][[BR]]
Line 17: Line 17:
[BR][BR] [[BR]][[BR]]

python的robotparser模块

读一个Python的爬虫代码的时候看到一个名字叫做”robotparser”的模块, 查了一下, 是用来解析网站的”robots.txt”文件的. 用法很简单, 举个例子 

rp = robotparser.RobotFileParser() 
rp.set_url("http://www.musi-cal.com/robots.txt") 
rp.read() 
rp.can_fetch("*", "http://www.musi-cal.com/cgi-bin/search?city=San+Francisco")  # return False 
rp.can_fetch("*", "http://www.musi-cal.com/")  # return True

程序很简单. 首先将类实例化, 然后用”set_url”方法设置”robots.txt”文件的路径, 这个文件一般都在网站的根目录. 接着调用”read”方法来解析这个文件. 最后就是用”can_fetch”方法来判断这个网站是否让你的程序(爬虫)访问某个网站内的链接了! BRBR 如果这个网站没有”robots.txt”文件的话, 在”read”方法后查看”rp.errcode”属性, 会看到404的返回值, 如果有这个文件就是200的返回值了! BRBR 相关链接

nickcheng/2007-03-20 (last edited 2009-12-25 07:11:13 by localhost)