To stop undesirable material inside the search indexes, webmasters can instruct spiders not to crawl sure documents or directories from the normal robots.txt file in the basis directory of your area. Additionally, a web page might be explicitly excluded from a online search engine's database through the use of a meta tag unique to robots (commonly