Crawling
Crawling is the automated fetching of web pages and resources by a bot, for example for processing by a search engine.
A bot fetches an address
During crawling, an automated program visits known or newly discovered URLs and fetches their contents. Search engines use such crawlers to examine pages. In a fictional example, a bot discovers a guide through an internal link and then attempts to fetch it.
Discovery and successful fetching are different steps. Server errors, access restrictions or crawling rules can affect retrieval; necessary resources may also matter for rendering. A logged bot visit proves neither complete understanding nor indexing or good placement. Check which address was reached, with what response and accessible content, rather than simply counting visits.
Sources
Related terms
- IndexingDuring indexing, a search engine analyses content and adds information about it to its search index.
- Robots.txtrobots.txt gives rules to cooperating crawlers; it provides neither access protection nor reliable exclusion from the search index.
- XML sitemapAn XML sitemap gives search engines selected URLs from a website and may include additional information about them.
- Core Web VitalsCore Web Vitals describe loading experience, responsiveness and visual stability. The measurement environment and underlying data should be reported with the result.
- Structured dataStructured data describes content in a machine-readable format and explicitly identifies its meaning.
Articles on this topic
Your project with ASES
Describe your project to us. We will get back to you with an initial assessment.