Crawl budget

Crawl budget refers to the URLs Google can and wants to fetch for a website. The term describes available crawling resources, rather than money allocated to search engine optimisation. Google distinguishes the crawl capacity limit from the demand to fetch content.

Consider technical limits and URL volume

In a fictional project, sorting parameters generate many URLs with largely identical content. The server also responds slowly at times. Duplicate URLs can create crawling work; server responses can affect how much can be fetched. This does not automatically mean new pages are absent from the search index solely for that reason.

Establish the problem first

Google’s guide primarily addresses large or frequently updated websites. Crawl budget should therefore not automatically be treated as the first cause of trouble on a small company website. Check evidence of fetching problems and the indexing status before planning a capacity change. More crawling guarantees neither indexing nor a particular search position.

Sources

• Google Crawling Infrastructure: Optimize your crawl budget