According to search engine author Alan K'necht, website owners should create and maintain a robots.txt file. This file tells search engine spiders what directories they should and should not index or spider. Learn more about the robot.txt file at robotstxt.org.
|