Fix what blocks indexing
Find and fix the six technical issues that keep SaaS pages out of Google's index: rendering, robots rules, canonical conflicts, sitemaps and internal links.
On this page 9 sections
- Check one: robots.txt and the staging rule that shipped
- Check two: meta robots and X-Robots-Tag conflicts
- Check three: canonical tags pointing somewhere else
- Check four: the rendering test, step by step
- Check five: sitemap coverage and why you should segment it
- Check six: internal links, the one you control
- When to escalate to engineering, and what the ticket must contain
- Why Core Web Vitals can wait
- Do this before the next lesson
- Frequently asked questions
The short answer
Six checks account for nearly every indexing failure on a SaaS marketing site: a robots.txt disallow left over from staging, a stray noindex meta tag or X-Robots-Tag header, a canonical pointing at the wrong URL, content rendered only in client side JavaScript, a sitemap that omits the page, and zero internal links to it. Run them in that order. Confirm with URL Inspection's live test and compare the rendered HTML against the raw source.
Key points before you start
Five pages went live last week. Some of them are not in the index, and the instinct is to rewrite them. Resist it. Indexing failures are almost never a writing problem, and rewriting a page that Googlebot has never successfully fetched is a week you do not get back.
Technical SEO for a SaaS marketing site collapses into six checks. Run them in order, because each one makes the next one meaningful. And skip Core Web Vitals entirely this week. A page that loads in 400 milliseconds and is not in the index is worth exactly as much as a page that loads in nine seconds and is not in the index.
Check one: robots.txt and the staging rule that shipped
Open yoursite.com/robots.txt in a browser right now. Read every line. You are looking for a Disallow: rule that catches your new URLs, and specifically for a blanket Disallow: / that somebody added when staging went up and nobody removed.
This is the most common cause and the most embarrassing one. It happens because staging and production share a deploy pipeline and the robots file is environment-dependent. Webflow sites hit it when the staging subdomain setting is copied. Next.js projects on Vercel hit it when a middleware rule keys off an environment variable that is not set in production.
Test the exact URL, not the pattern you assume applies. Search Console has a robots.txt report under Settings that shows the fetched file and the last fetch time. If your file changed two days ago and Google last fetched it nine days ago, you are looking at a cached version and need to wait or force a refetch.
Blocking your own CSS and JS
A rule like Disallow: /assets/ or Disallow: /_next/ blocks the resources the page needs to render. Google then renders a page with no layout and, on a JavaScript heavy site, no content at all. The page may still get indexed, but as something unrecognisable. Allow your static asset directories.
Check two: meta robots and X-Robots-Tag conflicts
Two places can carry a noindex: the <meta name="robots"> tag in the HTML head, and the X-Robots-Tag HTTP response header. Teams check the first and forget the second constantly.
Run this from a terminal for each of your five URLs:
curl -sI https://yoursite.com/vs/competitor/ | grep -i "x-robots-tag"An empty response is what you want. If it returns noindex, the tag is being set at the CDN or server layer and no amount of editing your CMS will remove it. That is an engineering ticket, not a marketing fix.
For the meta tag, view source rather than using developer tools, because a client side script can inject or remove the tag after load and the inspector shows you the post-script state. curl -s URL | grep -i "robots" gives you what the crawler receives first.
Check three: canonical tags pointing somewhere else
A canonical tag tells Google which URL is the preferred version of a page. Get it wrong and you have politely asked Google to index a different page instead of yours.
Three failure patterns show up repeatedly on SaaS sites. A CMS template that hardcodes the homepage canonical onto every page. A trailing slash mismatch where the canonical says /vs/competitor and the live URL is /vs/competitor/, which some setups treat as distinct. And an environment mismatch where the canonical points at a staging domain because the base URL is built from an env var.
| Symptom in URL Inspection | Likely cause | Fix owner |
|---|---|---|
| “Alternate page with proper canonical tag” | Your canonical points at another URL | Marketing, if CMS editable |
| “Duplicate, Google chose different canonical” | Two near-identical pages competing | Marketing, content differentiation |
| “Page with redirect” | The URL 301s somewhere else | Engineering |
| “Blocked by robots.txt” | Disallow rule matches the path | Engineering |
| “Excluded by noindex tag” | Meta robots or X-Robots-Tag | Either, depends on layer |
| “Crawled, currently not indexed” | Quality, duplication or low priority | Marketing |
That last row is the one people misdiagnose. “Crawled, currently not indexed” means Google looked and declined. There is no technical fix. Near-duplicate pages compete with each other here, which is the mechanism described in keyword cannibalization, and it hits comparison page sets hardest because five alternatives pages built from one template can look substantially identical to a crawler.
Editable working copy
Get this checklist as a working file
Save the checks on this page as a working copy and assign an owner, status and evidence for each action.
Check four: the rendering test, step by step
This is the check that finds the problems the other five miss, and almost nobody runs it properly.
Comparing raw source against rendered DOM
- Fetch the raw HTML
Run curl -s https://yoursite.com/your-page/ > raw.html in a terminal. This is exactly what the crawler receives before any JavaScript executes.
- Search the raw file for your H1
grep -i '<h1' raw.html. If your headline is not there, the page is client rendered and you have found your problem already.
- Search for your body copy
Pick a distinctive eight word phrase from the middle of the page and grep for it. Headlines sometimes render server side while body content does not.
- Search for internal links
grep -o 'href="/[^"]*"' raw.html | sort -u. Compare against the links you placed. Links injected by JavaScript pass no signal until render happens, and render is queued.
- Run URL Inspection live test
In Search Console, paste the URL, click Test Live URL, wait, then click View Tested Page and open the HTML tab. This is the DOM Google produced after rendering.
- Diff the two
Compare the Search Console HTML against raw.html. Anything present in the rendered version and absent from the raw one depends on JavaScript execution.
- Check the screenshot tab
The same panel shows a rendered screenshot. A blank area where your comparison table sits means Google did not get the table, regardless of what the HTML tab suggests.
- Read the page resources tab
It lists resources Google could not load. A blocked script or a failed API call here explains a blank render better than any theory you will form on your own.
The diff is the whole exercise. If your comparison table, your answer capsule or your internal links exist only after JavaScript runs, you are betting your indexation on a render queue with unpredictable latency. Google’s own documentation describes rendering as a separate stage that happens when resources allow. For AI crawlers the situation is worse: several fetch raw HTML and do not execute JavaScript at all, which means a client rendered page is invisible to them permanently, not just slowly.
The fix is server rendering or static generation for marketing pages. On Next.js that means avoiding client-only data fetching for above-fold content. On a headless CMS setup it usually means a build hook. This is the one item on the list worth an engineering conversation even if everything else is clean, and it is covered in more depth in technical SEO for SaaS.
6
Checks that account for the overwhelming majority of SaaS indexing failures
saas-marketing.net editorial
Check five: sitemap coverage and why you should segment it
Your sitemap will not make Google index anything. What it does is give you a measurement surface, and that is worth more than most people realise.
Submit one sitemap per page type rather than one file for the whole site. A BOFU sitemap with your five new URLs. A blog sitemap. A docs sitemap. When you launch a programmatic set later, that gets its own file too. Search Console reports discovered and indexed counts per submitted sitemap, so a template-wide indexation problem shows up as one segment sitting at 15 percent while the others sit at 90 percent.
Without segmentation you get a single site-wide number that hides everything. This becomes acute at scale, which is why the programmatic page indexation study reports indexation rates by template type rather than by domain.
Three sitemap hygiene rules. Only include canonical, indexable, 200-status URLs; a sitemap full of redirects and noindexed pages trains Google to trust it less. Keep lastmod accurate, because a file where every URL claims to have been modified today is a signal that gets discounted. And reference the sitemaps in robots.txt with a Sitemap: line, which costs nothing.
Check six: internal links, the one you control
A URL with no internal links pointing at it has no path for a crawler to follow and no evidence that you consider it important. Both matter.
Crawl your own site with Screaming Frog on the free tier (500 URLs is plenty at this stage) or use any crawler that reports inlinks. Filter for your five new URLs and read the inlink count. Anything showing zero or one needs attention today.
The target from lesson two was three inbound internal links per page, from pages that themselves have some authority. A link from your homepage or a hub in the main navigation is worth several from an old blog post. If you cannot find three natural placements, that is information: the page may not connect to anything else you have published, which usually means the topic sits outside your actual cluster.
Force discovery on a small site
On a site under a few hundred URLs, submitting each new page individually through URL Inspection’s Request Indexing button genuinely speeds up discovery, typically to a day or two. There is a daily quota, so five pages is fine and 500 is not. This does not work as a substitute for internal links, only as an accelerant alongside them.
Review request
Free SaaS marketing audit
Share your site, stage and priorities to request a review of your positioning, funnel and acquisition plan.
When to escalate to engineering, and what the ticket must contain
Escalate when the fix requires a code change or a server configuration change. That means: X-Robots-Tag headers, redirect chains, canonical tags generated by templates you cannot edit, client side rendering of main content, and robots.txt in a repository.
Do not escalate for “Crawled, currently not indexed”, thin content, missing internal links or sitemap contents. Those are yours.
A ticket that gets actioned in the same sprint contains five things:
- The exact URL, and whether the issue affects one page or a template
- The observed behaviour, with the command that reproduces it (
curl -sI URL) - The expected behaviour in one sentence
- The business impact with a number, for example “this template will cover 40 pages targeting roughly 2,000 monthly searches”
- A deadline that is real
A ticket saying “SEO issue on the comparison pages, please fix” sits in the backlog for a quarter. A ticket saying “curl -sI on /vs/competitor returns X-Robots-Tag: noindex; expected no such header; blocks 5 pages already published; needs fixing before the 30 September review” gets picked up. The common failure patterns here, including the organisational ones, are catalogued in why SaaS SEO fails.
Why Core Web Vitals can wait
Page experience metrics influence ranking between otherwise comparable results. They have no role in whether a URL enters the index. So a team spending week three chasing a Largest Contentful Paint score from 3.1 seconds to 2.4 seconds, while two of their five new pages are excluded by a noindex header, has inverted the priority order completely.
Revisit performance once all five pages are indexed and holding positions in the top 20. At that point, a speed improvement might move you from position 12 to position 9. Before that point it moves you from nowhere to nowhere. Treat it as an experiment with a measurable before and after, the way SaaS SEO experiments frames any technical change.
The honest cost of this lesson: the six checks take about three hours the first time and 40 minutes when repeated. The engineering fixes they surface can take a sprint or a quarter depending on your team’s backlog, and that delay is outside your control. Budget for it rather than being surprised by it. Broader context on how this fits the overall programme sits in the SaaS SEO overview.
Do this before the next lesson
Run all six checks on all five URLs and record the result in a table. Confirm indexation in Search Console for each. For anything still excluded, write the ticket today rather than at the end of the week.
Then produce a severity ranked list of what remains: blocking issues first (anything preventing indexation), then signal issues (canonical conflicts, missing internal links), then optimisation (performance, schema). Take the blocking list to engineering. Keep the rest for the maintenance queue.
If your five pages are indexed and you are already thinking about a larger page set, read the programmatic page brief template before you publish batch one, and study how a single integration page or alternatives page behaves before you multiply it by 200. Lesson four moves off the site entirely and onto links and citations.
Self-paced learning
Build your SaaS marketing study plan
Choose a free course and work through its published lessons at your own pace. Save the course index for later.
Frequently asked questions
Why is my SaaS page not getting indexed?
In order of frequency: a robots.txt disallow or noindex tag carried over from staging, a canonical tag pointing at a different URL, main content that only appears after JavaScript runs, or no internal links pointing at the page. Use URL Inspection in Search Console to see which of these Google actually encountered rather than guessing from the page source.
What does 'Crawled, currently not indexed' mean?
Google fetched the page, evaluated it, and chose not to index it. That is usually a quality or duplication judgement, not a technical error. The practical fixes are adding unique substance the page currently lacks, adding internal links from pages that already have authority, and checking whether a near-identical page on your site is competing with it.
How do I test whether Google can render my page?
Run URL Inspection on the live URL in Search Console, click Test Live URL, then View Tested Page and open the HTML tab. That is the rendered DOM Google produced. Compare it against the raw response from curl. If a heading, your body copy or the internal links appear in one and not the other, rendering is your problem.
Should SaaS marketing pages be server rendered?
Yes, for anything that needs to rank. Google does render JavaScript, but rendering is queued separately from crawling and can lag by days or longer, and other crawlers including several AI engines render far less reliably. Server side rendering or static generation removes an entire category of failure for the cost of a build step.
How long should I wait before assuming a page will not be indexed?
Seven to fourteen days after submitting via URL Inspection on an established domain, and up to thirty days on a new one. Before that window closes, resubmitting repeatedly changes nothing. Spend the waiting time adding internal links, which is the variable you actually control.
Do I need an XML sitemap if my site is small?
A sitemap will not force indexing, but it gives you a measurement surface. Segmenting sitemaps by page type (one for BOFU pages, one for the blog, one for programmatic templates) lets Search Console report indexation per segment, which is how you spot a template-wide problem in a week rather than a quarter.
Does Core Web Vitals affect indexing?
No. Page experience signals can influence ranking among comparable results, but they play no part in whether a URL enters the index. A page scoring poorly on Largest Contentful Paint still gets indexed if it is crawlable, renderable and worth including. Fix indexation first and performance later.
The saas-marketing.net editorial team Research and editorial
We research, write and maintain every page on this site. The library explains marketing decisions through practical frameworks, explicit assumptions and references. Corrections can be requested through the contact page.
Published September 11, 2026. Last updated .