# WWW Robots (also called wanderers or spiders) are programs that # traverse many pages in the World Wide Web by recursively retrieving # linked pages. # # In 1993 and 1994 there have been occasions where robots have visited WWW # servers where they weren't welcome for various reasons. Sometimes these # reasons were robot specific, e.g. certain robots swamped servers with # rapid-fire requests, or retrieved the same files repeatedly. In other # situations robots traversed parts of WWW servers that weren't suitable, # e.g. very deep virtual trees, duplicated information, temporary # information, or cgi-scripts with side-effects (such as voting). # # These incidents indicated the need for established mechanisms for WWW # servers to indicate to robots which parts of their server should not be # accessed. This standard addresses this need with an operational # solution. # For more info, see: A Standard for Robot Exclusion # http://info.webcrawler.com/mak/projects/robots/exclusion.html # # mail webmaster@rickohio.com for constructive criticism # User-agent: * Disallow: /cgi-bin Disallow: /log Disallow: /images Disallow: /sounds Disallow: /images Disallow: /rjp Disallow: /membership Disallow: /family Disallow: /family_unsecure Disallow: /graphics Disallow: /stats Disallow: /football/membership Disallow: /natasha Disallow: /platt