# Crawl budget > Crawl budget defined, how Google actually allocates it, when a SaaS site is big enough to care, and the log file check that settles the argument in an hour. Source: https://saas-marketing.net/glossary/crawl-budget/ Topic: SaaS SEO Type: glossary Published: 2026-09-11 Last updated: 2026-09-11 Publisher: SaaS Marketing (saas-marketing.net) License: CC BY 4.0. Quote or republish with attribution and a link to https://saas-marketing.net/glossary/crawl-budget/ ## Short answer Crawl budget is the number of URLs a search engine will fetch from a site in a given period. It is the product of two things: crawl rate limit, which is how fast a server can be crawled without degrading, and crawl demand, which is how much the engine wants the site's pages. Google has said sites under roughly a few thousand URLs almost never need to think about it. Below 10,000 URLs, crawl budget is rarely the real problem. ## Key takeaways - Crawl budget equals crawl rate limit multiplied by crawl demand, and either one can be the binding constraint on a site. - Under roughly 10,000 URLs, crawl budget is almost never the cause of indexing problems on a SaaS site. - Faceted parameters, session URLs, uncanonicalised pagination and orphaned programmatic tranches cause most genuine crawl waste. - Log files settle the argument: compare crawls per day by directory against indexed pages in that directory. - Most published crawl budget advice is enterprise ecommerce guidance applied to sites a hundredth the size. - Index bloat and crawl budget are different problems with different fixes, and SaaS sites usually have the first one. --- Crawl budget is the number of URLs a search engine will fetch from your site in a given window. Google describes it as the product of two separate things: the crawl rate limit, which is the ceiling your server can take before Googlebot backs off, and crawl demand, which is how badly Google wants your pages. Most people arguing about crawl budget are arguing about the wrong one. And most SaaS sites should not be having the argument at all. ## What crawl budget actually is, in one formula Crawl budget = crawl rate limit x crawl demand. The rate limit is defensive. Googlebot watches response times, and when your server slows under load, it crawls less. Crawl demand is appetite: popular URLs, freshly updated URLs and URLs that historically returned useful content get fetched more often. You cannot set either number. You influence the first with server speed and the second with links, freshness and traffic. The crawl rate setting in the old Search Console was retired in January 2024, and it only ever let you throttle down. **Few thousand URLs** Site size below which Google's own docs say crawl budget generally is not a concern ## When is a SaaS site actually big enough to care? Roughly 10,000 URLs. Below that, if pages are not getting indexed, the cause is almost always quality, internal linking or rendering, not crawl starvation. Google's documentation puts the threshold lower still, at a few thousand. The sites that genuinely hit crawl limits are the ones running [programmatic SEO](/guides/programmatic-seo-for-saas/) at scale: an integrations directory with 4,000 combinations, a template gallery, a location or use case matrix. Ahrefs, Zapier and Canva all operate at a size where this is a real operational concern. A 280 page Series A marketing site does not. | Site size | Is crawl budget likely the issue? | What to check first | | --- | --- | --- | | Under 1,000 URLs | Almost never | Internal links, content quality, rendering | | 1,000 to 10,000 | Rarely | Orphan pages, canonical consistency | | 10,000 to 100,000 | Sometimes | Parameter handling, log file distribution | | Over 100,000 | Often | All of the above plus sitemap segmentation | ## The log file check that settles the argument Pull 30 days of server logs. Filter to verified Googlebot by reverse DNS, not user agent string, because spoofing is common. Group by directory, count fetches per day, then set that against the indexed URL count for the same directory in Search Console. **The 90 minute crawl waste audit** ## The four genuine causes of crawl waste on SaaS sites **Faceted parameters.** A filterable directory that generates `?sort=`, `?filter=`, `?page=` combinations produces a combinatorial explosion. One integrations page with four filters can spawn thousands of crawlable URLs. **Session and tracking URLs.** UTM parameters, session IDs and click identifiers leaking into internal links. Every internal link carrying a `?ref=` creates a distinct URL to fetch. **Uncanonicalised pagination.** Deep paginated archives where page 47 carries the same canonical treatment as page 1, or worse, no canonical at all. **Orphaned programmatic tranches.** You shipped 2,000 [comparison pages](/guides/saas-comparison-pages/) and [alternatives pages](/guides/saas-alternatives-pages/), linked them only from the sitemap, and now Googlebot crawls each once a quarter. That is not a crawl budget shortage. It is a linking failure that presents as one. Teams find crawl waste and immediately block everything in robots.txt. A robots-blocked URL cannot be read, so Google never sees the noindex or canonical tag you also added. If you want a URL out of the index, let it be crawled and serve noindex. Block only genuine crawl traps that should never be fetched. ## Commonly confused with: index bloat These get used interchangeably and they are not the same thing. Index bloat hurts more and hurts sooner. It dilutes topical signals, and it puts low-quality URLs in front of both classical rankings and the retrieval layers behind [answer engine optimization](/glossary/answer-engine-optimization/) and [generative engine optimization](/glossary/generative-engine-optimization/). Crawl waste is a throughput issue. Bloat is a quality issue. ## My position: most crawl budget advice is misapplied Nearly everything written about crawl budget originated in enterprise ecommerce, where a retailer genuinely has 2 million SKU URLs and a real fetch ceiling. That guidance gets copied onto SaaS blogs and applied to sites with 400 pages, where it does nothing except consume a quarter of somebody's engineering time. If your pages are not ranking, the honest order of suspicion is: content quality, then intent match, then internal linking, then rendering, then crawl. Crawl comes fifth. Work out what a fixed page is worth first with the [SEO traffic value calculator](/calculators/seo-traffic-value/), and check whether the gap is competitive rather than technical using a [share of voice](/calculators/competitor-share-of-voice/) view. The honest tradeoff: a proper log file audit costs a day of analyst time plus whatever your logging pipeline costs, and on a site under 10,000 URLs it will usually conclude that nothing is wrong. That is still a useful result, because it lets you stop arguing and go fix the content. Just do not sell it internally as a growth project. It is a diagnostic, and its output is often a clean bill of health on a patient with a different illness. ## What to do next Count your indexable URLs. Under 10,000, close this page and go audit internal links and content quality instead. Over 10,000, run the log file check above before changing a single line of robots.txt. If you are building programmatic tranches as part of a broader [SaaS SEO](/saas-seo/) programme, put internal linking into the template on day one, because retrofitting links to 2,000 orphaned pages is far more expensive than shipping them connected. And if the crawl data does turn up waste, fix the four causes in the order listed: parameters, sessions, pagination, orphans. Tie the recovered pages back to pipeline the way you would any other [qualified lead](/glossary/marketing-qualified-lead/) source, so the work has a number attached to it. ## Frequently asked questions ### What is crawl budget in SEO? Crawl budget is how many URLs a search engine crawler will fetch from your site over a period, typically measured per day. It comes from two inputs: the crawl rate your server can handle without slowing down, and how much demand the engine has for your content based on popularity and freshness. Neither input is something you set directly. ### Does crawl budget matter for a SaaS website? For most SaaS sites, no. Google's own documentation says sites with up to a few thousand URLs are usually crawled efficiently without intervention. It starts to matter above roughly 10,000 URLs, or when a site generates parameter combinations faster than it publishes real pages. A 300 page marketing site with an indexing problem has a different problem. ### How do I check if crawl budget is actually a problem? Pull 30 days of server log files, filter to verified Googlebot requests, and group crawls per day by directory. Compare that against the number of indexed URLs in each directory from Search Console. If a directory with 4,000 real pages gets 60 crawls a day while a parameter directory gets 900, you have measurable waste worth fixing. ### What is the difference between crawl budget and index bloat? Crawl budget is about fetching. Index bloat is about what got kept. A site can be crawled perfectly well and still have 12,000 thin, near-duplicate URLs sitting in the index diluting its topical signals. Bloat is fixed with noindex, canonicals and deletion. Crawl waste is fixed with robots.txt, parameter handling and internal linking. ### Can I increase my crawl budget? Not directly. You can raise the crawl rate ceiling by making the server faster, since Googlebot backs off when response times climb. You can raise crawl demand by earning links and publishing pages that get traffic. You cannot request more crawling. The old Search Console crawl rate setting was removed in January 2024 and only ever lowered the rate anyway. ### Does blocking URLs in robots.txt save crawl budget? Yes, and it is the most direct lever available. Blocked URLs are not fetched, so the crawl goes elsewhere. The trap is that a robots-blocked URL can still be indexed without content if other pages link to it, and blocking a URL prevents Google from seeing a canonical or noindex tag on it. Block crawl traps, not pages you want removed.