A text file present in the root directory of a website which is used to direct the activity of search engine crawlers. This file is typically used to tell a crawler which portions of the site should be crawled and which should not be .