to stop unwanted articles while in the search indexes, webmasters can instruct spiders to not crawl certain documents or directories throughout the common robots.txt file in the foundation Listing of the area. In addition, a website page might be explicitly excluded from a search engine's databases through the use of a meta tag specific to robots (