Robots directive
Robots directives give indexing or presentation rules to search engines that support them. The crawler and the delivery route also matter.
A robots directive is an instruction for search engine crawlers, for example about indexing a page or displaying search snippets. Which rules apply depends on the crawler addressed and its support for them.
Check HTML and response headers together
Google documents rules in robots meta tags and HTTP response headers. The header is called X-Robots-Tag and can also be used for non-HTML files. The crawler must be able to reach the page to read the instruction served with it.
Identify settings inherited from templates
In a fictional project, noindex is accidentally left in the template when a preview becomes a public service page. The team needs to check the rule actually served and its scope. A setting in an editorial form alone does not prove what the server sends. By contrast, robots.txt rules control fetching; the two mechanisms need distinguishing. An instruction about search presentation also provides no access control for confidential content.
Sources
Related terms
- NoindexNoindex controls inclusion in search engines that support the instruction. The crawler must be able to read it; it does not replace access protection.
- Indexing statusIndexing status describes a URL’s inclusion in a search index. It is distinct from reachability and visibility for a particular search.
- Crawl budgetCrawl budget describes which URLs Google can and wants to fetch. Server capacity and crawl demand work together.
- HreflangHreflang identifies related language or regional versions of a page for search engines.
- Canonical URLA canonical URL is the preferred address for identical or very similar content; search engines assess several signals when choosing it.
- Technical SEOTechnical SEO addresses the technical conditions that let search engines fetch, understand and index suitable pages.
Articles on this topic
Online Marketing with ASES
SEO, GEO, ads and analytics