Skip to content

Chapter 3 · Technical SEO

Crawlability and Crawl Budget

Make important URLs discoverable and reliably fetchable, then use verified crawl data to fix demonstrated bottlenecks rather than trying to maximize crawl volume.

Updated November 1, 2026 · By · Reviewed by Robin Rad

· 8 min read

Quick answer: Crawlability is a search engine’s ability to discover and fetch a website’s URLs. Crawl budget is the set of URLs Google can and wants to crawl, shaped by crawl capacity and demand. Improving both means keeping important URLs accessible, maintaining reliable server responses, and limiting unnecessary URL discovery—not guaranteeing indexing.

Key facts

TL;DR: Make important URLs discoverable and reliably fetchable, then use verified crawl data to fix demonstrated bottlenecks rather than trying to maximize crawl volume.

Crawlability
The ability of search engine crawlers to discover and fetch a website’s URLs.
Crawl budget
The URLs Google can and wants to crawl, determined jointly by crawl capacity and crawl demand.
Crawl capacity limit
The limit on crawling that reflects server responsiveness and the crawler’s available resources.
Crawl demand
Google’s interest in crawling URLs, influenced by factors such as perceived inventory, popularity, freshness, and relevance.
A small crawler enters a page network, illuminating reachable nodes while isolated and duplicate branches remain dim.

What are crawlability and crawl budget?

Crawlability is a search engine’s ability to find URLs and fetch their content. Crawl budget is the set of URLs Google can and wants to crawl. It depends on crawl capacity limits and crawl demand, as defined by Google.

Finding a URL and fetching it are separate steps. Googlebot may find a link but fail to fetch the page because the server is down. A page may be open to crawlers but have no links that help them find it. Google may also know about a URL but not yet choose to crawl it.

Site owners do not assign Googlebot a fixed allowance. Google weighs how much crawling a site can support against its demand for the site’s URLs. Better access alone does not set that demand, according to Google’s crawl-budget guidance.

Crawling fetches content. Indexing is a separate step that evaluates the content. A successful fetch does not ensure a place in search results, as Google explains.

To find the cause of a problem, ask: can Google find the URL, can Googlebot fetch it, and is Google choosing to request it? These questions help you avoid treating every uncrawled page as a budget issue.

When does crawl budget actually matter?

Crawl budget matters when crawl delays keep Googlebot from reaching important new pages or updates when needed. Most small, stable sites should focus on basic access instead. Look at site size, how often pages change, and actual crawl delays before starting advanced crawl-budget work.

Google’s examples include sites with roughly 1 million or more pages changing weekly, or 10,000 or more changing daily. These are estimates, not exact thresholds, according to Google Search Central. Many URLs marked ‘Discovered – currently not indexed’ are another reason to investigate. That status alone does not prove a lack of crawl capacity.

Focus on timely access, not crawl totals

Timely crawling lets search engines fetch new content and updates. Content that cannot be fetched may miss search opportunities. But a crawl alone does not secure indexing or rankings.

  1. Ask: Are important URLs being fetched when needed?
  2. Compare crawl delays with how often those pages change.
  3. Focus on delays that affect useful content, rather than trying to save every crawler request.

A technical SEO audit checks for barriers to search access. It can help you separate basic access problems from issues that need a deeper crawl review.

How Google balances crawl capacity and demand

Google balances crawl capacity and demand to decide which URLs to fetch and how much crawling a site can support.

Capacity and demand are separate constraints

The crawl capacity limit sets limits on what Googlebot can fetch. Crawl demand reflects what Google wants to fetch. Together, they determine crawl budget, as explained by Google Search Central.

Spare server capacity does not create demand for unimportant URLs. Crawl budget can change over time. It is not a fixed daily allowance that site owners can assign to chosen pages. Cutting needless URLs can reduce waste, but saved requests do not always shift to preferred pages.

Server health supports crawl capacity

Server health concerns the speed and reliability of responses sent to Googlebot. It is not a browser-side speed score. Slow or failed responses can limit crawling. HTTP 5xx errors and HTTP 429 responses cause Google to reduce crawling for a time, according to Google’s crawling documentation.

Focus on reliable hosting, less downtime, and prompt server responses. Check server logs and CDN logs to find delays and failures. Do not rely on browser scores alone.

For caching and other ways to improve content delivery, see page speed optimization.

Make important URLs discoverable and fetchable

Start with your important URLs. Then check whether search engines can find and fetch them.

  1. Build a URL list. Group important URLs by purpose, such as products, categories, and guides. Pick samples of new and recently updated pages from each group.
  2. Separate discovery from fetching. Check each sample for missing links, blocked access, or failed requests. A URL that Google knows but has not crawled needs a different review from a page with no internal links. Use server logs or CDN logs to check for fetch attempts.

Link to every important page from at least one other page on your site. Use an HTML anchor with an href attribute for reliable link crawling, as Google Search Central recommends. See internal linking for practical guidance.

Click depth counts navigation steps, not the length of a URL path. Focus on clear paths to important pages within your site architecture, rather than a fixed click-depth rule.

XML sitemaps offer crawl suggestions. Googlebot does not have to fetch every listed URL or fetch it right away, according to Google Search Central.

ExampleWhat to check first
Product URL with no internal linksAdd useful paths to the page, then check fetch access.
Well-linked URL with failed crawler requestsFind the cause of failed fetches before adding more links.

Reduce unnecessary URL discovery and repeated fetching

To cut needless crawling, review how your site creates URLs and exposes them to crawlers. Focus on duplicate pages and URLs with little useful content.

  1. Find crawl traps. These are URL patterns that keep creating more URLs without useful new content. Examples include endless calendars, session IDs that create copies of a page, and combinations of filters and sort options.
  2. Compare generated URLs with useful pages. Review the URLs each template creates. Which pages serve a real user or search need? Keep useful filtered views. A URL with parameters is not always wasteful.
  3. Use evidence to set priorities. Check server logs or CDN logs for URL groups that draw excessive Googlebot requests. Focus on repeated fetches and what causes them. Do not make clearing every not-found response your main goal.
  4. Fix the source. Stop uncontrolled URL creation and remove needless links. Avoid broad blocks that could hide useful content.

Canonicalization helps search engines choose a main URL for duplicate content. It can reduce needless duplicate crawling over time, but it does not block a crawl. Google treats redirects and canonical annotations as strong signals, while sitemap inclusion is a weak signal, according to Google Search Central.

For removed or moved content, return HTTP 404 or 410 when no similar replacement exists. Use a 301 redirect when content has moved or has a clear replacement, as Google Search Central recommends.

Common mistakes that make crawl optimization ineffective

These crawlability and crawl budget mistakes can waste requests, hide important content, or make useful changes look like a problem.

  1. Confusing crawl controls. Do not use noindex to save crawl budget. Google must still fetch the page to read that rule, as Google’s crawl budget guidance explains.
  2. Blocking too broadly. Robots.txt can prevent crawling, but it does not ensure removal from search. Google may index a blocked URL found through links without fetching its content, according to Google’s robots.txt guidance. Do not treat it as a removal tool. Avoid broad rules that block important pages or resources needed to render them.
  3. Misreading response codes. HTTP 429 responses cause a temporary drop in Google’s crawling. Other HTTP 4xx responses do not reduce the site’s crawl rate, according to Google’s HTTP status documentation. Do not redirect every failed URL just to clear an error report.
  4. Trying to manufacture crawl demand. Avoid minor edits and extra URLs made only to attract Googlebot. Publish useful changes rather than extra crawl targets.
  5. Chasing higher crawl counts. More requests may mean wasted fetches of duplicate URLs. Fewer requests may reflect useful cleanup, not worse access. Judge changes by whether important content can still be found and fetched.

How can you check and measure crawl improvements?

Better crawling means timely, reliable fetches of important content with less wasted work. It does not always mean more requests. Record a baseline, group the URLs, check samples, and make a targeted change. Then compare similar periods. Track indexing and traffic apart from crawl measures.

A crawl-specific measurement workflow

A crawl baseline records request activity before you make changes.

  1. Review regularly. Use Google Search Console Crawl Stats to review total requests, response times, response patterns, and host availability. See Google Search Console for interface guidance.
  2. Monitor logs. Use server logs or CDN logs for evidence about each URL. Verify Googlebot through Google’s documented IP or DNS methods. Do not trust user-agent strings alone.
  3. Group requests. Separate important content from duplicate or generated URL groups. Group responses by outcome, and keep page fetches separate from resource requests.
  4. Inspect samples. Use URL Inspection and optional site crawlers such as Screaming Frog to check problem pages. Crawler-assisted technical SEO audits can flag access and discovery issues. They do not reveal Googlebot’s actual crawl schedule.
  5. Change and compare. Apply a targeted fix. Then compare similar time windows before and after the change.

Measures and limits

Fetch measures show what happened, not Google’s total crawl demand. Track:

  • The first observed Googlebot fetch after a page is published.
  • Time between crawls of important updated URLs.
  • Important URLs not fetched during the period.
  • Repeated HTTP 5xx errors and 429 responses, which temporarily reduce crawling, according to Google.

A missing request in origin server logs does not always mean no crawl took place. Logs may be incomplete, or a CDN may have served the request. A successful fetch also does not ensure indexing. Google evaluates the fetched content separately, as explained in its HTTP status documentation.

Frequently asked questions

Can I set a daily crawl budget for Googlebot?

No. Googlebot’s crawl budget depends on what Google can and wants to crawl, based on capacity and demand, as explained by Google Search Central. You can control access to parts of your site, but you cannot assign Google a fixed daily URL allowance.

Does adding a URL to a sitemap make Google crawl it immediately?

No. A sitemap provides crawl suggestions. Googlebot does not have to crawl every listed URL or crawl it right away, according to Google Search Central. Use sitemaps to help Google find URLs, not to set a crawl deadline.

Is crawl budget shared between Googlebot and other search engines?

Googlebot’s crawl budget describes Google’s own crawl capacity and demand, as defined by Google Search Central. It is not a shared allowance for all search engines. Other search engines make their own crawl decisions. Their requests do not subtract URLs from a common quota, though all crawler traffic uses server resources.

Sources

  1. Optimize your crawl budget
  2. How HTTP status codes affect Google’s crawlers
  3. Link best practices for Google
  4. Troubleshoot Google Search crawling errors
  5. How to specify a canonical URL with rel="canonical" and other methods
  6. Introduction to robots.txt

About the author: writes the Deep SEO SEO Encyclopedia, a 2026 reference on technical, on-page, local and AI search. Every entry cites the sources linked in its text and is reviewed by Robin Rad before publication.

Back to top
WhatsApp +968 9182 2962