Crawl Budget
What is Crawl Budget?
The crawl budget is the amount of time and resources a search engine assigns for crawling a particular website. In other words, it is the maximum number of pages a search engine can crawl on your site within a specific timeframe. The crawl budget can vary for different search engines (or crawlers).
Google states that you shouldn’t worry about the crawl budget unless:
- Your site has 1 million+ unique pages with content that changes around once a week.
- You have a medium-sized website (around 10k pages) and change content frequently (daily).
- You run a news website.
- Search Console classifies the majority of your site’s URLs as "Discovered - currently not indexed."
Each website gets a different crawl budget based on two main factors:
- Crawl Demand: Defined by the number of pages, posting/updating frequency, and popularity of the pages.
- Crawl Rate Limit: Impacted by server capabilities, crawl limit set by the site owner in Search Console, the search engine’s crawling limit, and auto-adjustments by Google based on server response times.
Why is the Crawl Budget Important?
The crawl budget is important because it affects how many pages Googlebot can crawl on your site and how often Googlebot can recrawl your web pages to update its index. Google has enormous resources, yet it cannot crawl (and regularly recrawl) all pages of the Internet. As a result, Google allocates a crawl budget to websites.
To ensure that your crawl budget is not being wasted on unimportant pages, focus on optimizing your website’s crawl efficiency.
How to Increase the Crawl Budget?
1. Speed Up Your Server and Decrease Page Loading Times
Server response time and page loading speed directly affect crawling. When Googlebot crawls your site, it downloads resources first and then processes them. If your server responds quickly, Googlebot can crawl more pages on your website.
- Use a fast and reliable web hosting service and Content Delivery Network (CDN) to improve server initial response time.
- Decrease page loading times by preventing crawling of large but non-critical resources using robots.txt, avoiding long redirect chains, and removing heavy and poorly-coded themes and plugins.
2. Add More Links
The number of links to a page indicates its importance to Google. Googlebot prioritizes crawling pages with more backlinks and internal links. Increase your crawl budget by adding more external and internal links to your pages.
- For internal linking suggestions, audit your website with tools like Ranktracker's Site Audit.
3. Fix Broken Links and Decrease Redirects
Too many broken internal links (404 or 410 response codes) and redirected URLs (3xx) can waste your site’s crawl budget. Fix broken links and unnecessary redirects to optimize your crawl budget.
- Use tools like Ranktracker's Site Audit to find broken and redirecting URLs on your site.
4. Use the Indexing API if Possible
The Indexing API lets you notify Google directly whenever you add, remove, or update pages on your site. This helps get your pages crawled faster.
- Note: The Indexing API is currently available for specific use cases like live videos and job postings.
FAQs
Does Googlebot Respect the Crawl-Delay in Robots.txt?
No, Googlebot doesn’t respect the crawl-delay settings applied in a robots.txt file.
When Should You Care About the Crawl Budget?
You should care about the crawl budget if you’re operating a very large site (more than 1 million pages) or a medium-sized website with very frequent (daily) changes in content. Most sites don’t need to worry about the crawl budget.
How Can I Check the Crawl Budget for My Website?
You won’t find the exact number for the crawl budget anywhere. However, you can check the overview of Google crawl activity in the Crawl Stats report in Google Search Console.
For more insights on improving your website's SEO and crawl efficiency, visit the Ranktracker Blog and explore our comprehensive SEO Guide. Additionally, familiarize yourself with key SEO terms and concepts in our SEO Glossary.