Robots.txt
robots.txt gives rules to cooperating crawlers; it provides neither access protection nor reliable exclusion from the search index.
Manage fetches, not confidential access
The robots.txt file tells crawlers which areas they should or should not fetch. Compliance depends on the crawler. In a fictional example, certain catalogue filter combinations should not be crawled because they generate many similar views.
Google may still discover a blocked URL through other links and index it without reading the page content. Keeping a page out of the index requires different handling of a suitable rule such as noindex: Google must be able to read it on the page or in its response. Confidential documents need actual access controls. Also check whether blocks affect necessary scripts or styles and thus the understanding of public pages.
Sources
Related terms
- CrawlingCrawling is the automated fetching of web pages and resources by a bot, for example for processing by a search engine.
- IndexingDuring indexing, a search engine analyses content and adds information about it to its search index.
- Technical SEOTechnical SEO addresses the technical conditions that let search engines fetch, understand and index suitable pages.
- Core Web VitalsCore Web Vitals describe loading experience, responsiveness and visual stability. The measurement environment and underlying data should be reported with the result.
- Structured dataStructured data describes content in a machine-readable format and explicitly identifies its meaning.
- XML sitemapAn XML sitemap gives search engines selected URLs from a website and may include additional information about them.
Articles on this topic
Your project with ASES
Describe your project to us. We will get back to you with an initial assessment.