A Standard for Robot Exclusion
使用 robots.txt 檔案可控制網頁自動搜尋程式將網站列入索引中。
The following example "/robots.txt " file specifies that no robots should visit any URL starting with "/cyberworld/map/ " or "/tmp/ ", or /foo.html :
# robots.txt for http://www.example.com/
User-agent: *
Disallow: /cyberworld/map/ # This is an infinite virtual URL space
Disallow: /tmp/ # these will soon disappear
Disallow: /foo.html
This example "/robots.txt " file specifies that no robots should visit any URL starting with "/cyberworld/map/ ", except the robot called "cybermapper ":
# robots.txt for http://www.example.com/
User-agent: *
Disallow: /cyberworld/map/ # This is an infinite virtual URL space
# Cybermapper knows where to go.
User-agent: cybermapper
Disallow:
This example indicates that no robots should visit this site further:
# go away
User-agent: *
Disallow: /
|