Technical SEO crawlers

A SaaS marketing site breaks crawlers in ways a brochure site never does. Client rendered pricing tables, a docs subdomain running on a different stack, 600 programmatic integration pages and an app subdomain behind auth that keeps leaking into the sitemap. Choosing a crawler is really choosing which of those failure modes you find before a customer or an engineer does.

We ran the four tools most SaaS teams consider against the same site and the same configuration budget. The differences that showed up were not the ones the feature matrices advertise.

What a SaaS site does to a crawler that a marketing site does not

Four structural things, and every one of them is a crawler decision. The marketing site is usually React or Next.js, so raw HTML and rendered HTML diverge. Docs live on a subdomain or a /docs reverse proxy with its own framework and its own sitemap. Programmatic page sets run to thousands of near identical URLs. And the app itself sits behind a login, which means half the links a crawler follows end in a 302 to a sign in page.

That last one wastes more crawl time than anything else. On the sites we audit, unconfigured crawls typically spend 20 to 40 percent of their requests bouncing off auth redirects and asset URLs nobody cares about.

So the first question is not which crawler is most accurate. It is which one lets you express exclusion rules, subdomain inclusion and render settings without fighting the interface. The technical SEO for SaaS guide covers the rules themselves. This page covers the tool that enforces them.

What the four crawlers cost at the scale a SaaS site actually reaches

Prices move, so treat these as a shape rather than a quote. The pattern holds: dedicated crawlers are cheap and platform audits are free only because you already pay for the platform.

Tool	Rough list price	Practical URL ceiling	JS rendering	Log files
Screaming Frog	About £199 a year, one seat	200k in memory mode, millions in database mode	Headless Chromium, fully configurable	Separate Log File Analyser product
Sitebulb Desktop	Roughly $16 to $40 a month	Comfortable to about 500k URLs	Chrome based, with rendered versus raw diff	No native log analysis
Semrush Site Audit	Bundled, Pro near $140 a month	100k pages a month on Pro, more on higher tiers	Available on higher tiers	Log file tool included, limited
Ahrefs Site Audit	Bundled, Lite from about $129 a month	Credit based, scales with plan	Renders by default	No log file analysis

The honest read: if you already pay for Ahrefs or Semrush, their site audits cost nothing extra and cover the monitoring job adequately. Neither replaces a dedicated crawler for diagnosis. If you are deciding between the two platforms for other reasons, the Ahrefs vs Semrush comparison goes into the wider trade.

Enterprise platforms (Lumar, Botify, JetOctopus) start in the low thousands per year and go up fast. They earn their price at genuine scale, meaning 500,000 URLs upward with log integration and a dedicated SEO headcount. Below that you are buying dashboards.

What each crawler missed on the same Next.js test site

We crawled the same 1,240 URL Next.js App Router marketing site with each tool, using default settings first and then a tuned configuration. The site had a client rendered pricing comparison table, a client side docs search, a paginated changelog and 620 programmatic integration pages.

The headline finding is boring and important. Three of the four crawlers could see the client rendered pricing content. Only Sitebulb told us, without being asked, that the raw HTML and the rendered HTML disagreed. Screaming Frog will show you the same thing, but you have to know to compare the two views, and the default 5 second AJAX timeout means a slow page silently reports as empty.

Ahrefs won on orphan detection because it joins the crawl against its own backlink index and your Search Console data. On this site it found 47 integration pages with inbound external links and no internal links pointing at them. Screaming Frog can do that too, with a GSC and backlink import, and nobody ever does.

Log files answer the question crawl data only raises

A crawler tells you a page is linked, renderable and canonical. It cannot tell you whether Googlebot has requested it in the last 90 days, and for a programmatic page set that is the only question worth asking.

This is the failure mode that kills programmatic SaaS SEO: 500 pages published, 40 indexed, and the crawler reports all 500 as healthy. Server logs show the truth, which is usually that Googlebot requested 60 of them once, found them near identical, and lost interest.

Screaming Frog’s Log File Analyser is a separate purchase at about £99 a year, and it imports raw access logs and joins them to a crawl. Semrush includes a basic log tool. Sitebulb and Ahrefs do not do log analysis at all. If you run on Vercel, Cloudflare or a CDN that does not expose raw logs by default, sort that out before you buy anything, because the log export is the hard part and the analyser is the easy part.

Scheduling, CLI and pre deploy checks

Quarterly audits are theatre. The useful pattern is a rendered crawl of the preview deployment on every release that touches the marketing site, plus a scheduled full crawl weekly.

Screaming Frog’s paid licence includes a command line interface, so you can point it at a Vercel preview URL in CI, export the issues CSV and fail the build on new 404s, unexpected noindex tags or missing canonicals. It takes about half a day to wire into GitHub Actions and it has caught more real problems for our clients than any monthly report. Sitebulb Cloud exposes an API for the same workflow at a higher price point.

The report is the product

This is the part every comparison skips and the part that decides the outcome. You are not buying a crawl. You are buying a document that a front end engineer, who has four higher priority tickets, will read and act on.

Sitebulb wins here and it is not close. Its hints are prioritised, each one explains why it matters in plain language, and the rendered versus raw comparison shows the engineer the actual difference rather than asserting one. Screaming Frog exports a spreadsheet with 40 tabs, which is perfect for an SEO and useless as a handoff.

So my position: if you are a solo SEO or founder doing the fixes yourself, Screaming Frog, every time. If you are handing work to a team you do not manage, pay the Sitebulb premium and consider it a translation cost. The SaaS technical SEO audit checklist covers what belongs in the handoff regardless of which tool produced it, and the economics of who does the fixing are covered in SEO agency vs in house for SaaS.

We switched crawlers not because the old one was wrong but because engineering kept closing the tickets as unclear. Same findings, different report, four times the fix rate.

What I would buy at each stage

Seed to about $2M ARR: Screaming Frog alone, £199 a year, run monthly and before releases. You have a few thousand URLs and one person doing SEO. Anything else is spend that could be a writer.

Series A to Series B: Screaming Frog plus the Log File Analyser, and use whichever platform audit comes bundled with the rank tracker you already pay for. That combination costs under £400 a year and covers diagnosis, monitoring and crawl budget. The wider stack question is covered in SaaS SEO tools.

Past 100,000 URLs with a real programmatic page set: add Sitebulb Cloud or move to Lumar. At that point the scheduling, change detection and multi user reporting are worth four figures a month, and you have someone whose job is to read the output.

What none of these crawlers tell you

They all measure the same web that existed in 2019. None of them measure whether ChatGPT, Perplexity or Google AI Mode cite you, and none flag that your robots.txt blocks GPTBot or ClaudeBot while your CMO asks why the brand never appears in AI answers. That needs separate tooling, covered in AI search visibility tools.

Crawlers also cannot tell you whether a fix worked. That requires a controlled change and a holdout, which is the discipline in SaaS SEO experiments. A green audit is a hygiene signal, not a growth signal, and plenty of technically immaculate SaaS sites rank for nothing because the pages have no reason to exist.

Do this in the next week

Buy the Screaming Frog licence. Run one rendered crawl at a 20 second AJAX timeout, compare rendered against raw HTML on your pricing page and your three highest value templates, and write down every difference. Then wire the CLI into your deploy pipeline on a 300 URL list before you audit anything else, because catching the next regression is worth more than cataloguing the last twelve.

Start at the SaaS SEO hub if you need the wider programme this sits inside.