Crawlability Audit Checklist: 25 Checks Before You Touch Content

A crawlability audit checks whether Googlebot can reach, fetch and follow every important page on your site. Test robots.txt and firewall rules, status codes on key templates, real a href links and click depth, XML sitemaps, rendered HTML, canonical and hreflang setup. Then fix any blockers before you start content work.

A new content plan is wasted if robots.txt still says Disallow: / after launch. This guide is the first step of a technical SEO audit, written as 25 pass or fail checks. Each gives the test, the pass condition, the usual cause and the fix.

What is a crawlability audit?

A crawlability audit is a set of tests that confirm search engine crawlers can find and fetch your pages. It looks at access rules, server responses, links, sitemaps, rendering, duplicates and language versions. It comes before indexing and ranking work, because a page Google cannot crawl cannot rank.

Crawlability and indexability are different. Crawlability means Googlebot can reach and download a page. Indexability means Google can, and chooses to, store that page in its index. A page can be crawlable but not indexed, which is a content or quality question we cover in our guide to crawled, currently not indexed pages.

If you want the full picture first, read how search engines crawl, index and rank content, or start with what crawling in SEO means.

Before you start: tools, access and the 5 checks to run first

You need Search Console, a desktop crawler and access to your server or CDN logs. Run the five checks that can block a whole site before anything else. If any of them fails, stop and fix it before you crawl the rest.

The tools

• Search Console: the Page indexing report, the URL Inspection tool and the Crawl Stats report.
• A crawler: Screaming Frog SEO Spider or Sitebulb, set to the Googlebot Smartphone user agent.
• Logs: raw server access logs, or the request logs from Cloudflare or your CDN.

Google’s Crawl Stats report is “available only for root-level properties”. If you only have a subfolder property in Search Console, add a domain property or a root URL-prefix property first.

The 5 checks to run first

• Check 1: robots.txt is not blocking key pages.
• Check 2: no staging block leaked to production.
• Check 4: the firewall or CDN lets Googlebot through.
• Check 6: key templates return 200.
• Check 15: the XML sitemap loads and lists live URLs.

These five can hide every page at once. On a JavaScript framework site, run the rendering check (check 18) in the same first pass.

Stage 1: Access (checks 1 to 5)

Access checks confirm that nothing tells Googlebot to stay out, or stops it at the door. One wrong rule here can hide the whole site. Run all five before you open a crawler.

Check 1: robots.txt allows your important pages

• Test: open yourdomain.com/robots.txt and test key URLs with a robots.txt tester or your crawler.
• Pass: no Disallow rule matches a page you want in search, and the file is under 500 KiB.
• Usual cause: a broad rule such as Disallow: /*? that also blocks filtered category pages you want found.
• Fix: narrow the rule, then recheck. Remove any crawl-delay line, since Google’s robots.txt documentation says crawl-delay is not supported.

Google also says robots.txt is generally cached for up to 24 hours. A fix may take a day to show.

While you are in the file, review your rules for AI crawlers by each vendor’s documented user agent name. Blocking Google-Extended, for example, “does not impact a site’s inclusion in Google Search”.

Check 2: no staging block leaked to production

• Test: compare the live robots.txt, the meta robots tag and the response headers with what you expect after launch.
• Pass: production has no Disallow: /, no site-wide noindex and no password prompt.
• Usual cause: a staging robots.txt or a CMS visibility setting copied across during a launch or redesign.
• Fix: restore the production file and settings, then add this check to every release list.

Check 3: meta robots and X-Robots-Tag are correct

• Test: crawl the site and filter for noindex in the meta robots tag and the X-Robots-Tag header.
• Pass: only pages you want out of search carry noindex.
• Usual cause: a template or plugin setting that applies noindex to a whole post type or language folder.
• Fix: remove noindex from the template. Never block a noindex page in robots.txt as well.

Google says that if robots.txt blocks a page, “the crawler will never see the noindex rule, and the page can still appear in search results.” Pick one tool per page.

Check 4: the firewall or CDN lets Googlebot through

• Test: search your Cloudflare or WAF security events for Googlebot user agents that got a 403, a challenge or a rate limit.
• Pass: verified Googlebot requests get the same 200 response a visitor gets.
• Usual cause: a country block, a bot rule or a rate limit added during an attack or a busy sale.
• Fix: allow verified Googlebot only. Do not allow every request that claims to be Googlebot.

To verify Googlebot, Google says to run a reverse DNS lookup on the IP and confirm it resolves to googlebot.com, google.com or googleusercontent.com. Then run a forward DNS lookup to confirm it matches the IP. You can also match against Google’s published IP lists. Cloudflare’s docs say verified bots have historically been excluded from its default bot configurations. In our view, if Googlebot is challenged, custom rules and rate limits are the first places to look.

Check 5: no login or geo wall on public pages

• Test: fetch key pages with URL Inspection and with your crawler, from outside the UAE.
• Pass: the full page loads without a login, a cookie wall or a redirect based on the visitor’s country.
• Usual cause: a geo redirect that sends all non-UAE visitors, including Googlebot, to a different store or a holding page.
• Fix: show a country banner or selector instead of a forced redirect, so every version stays reachable.

For the causes behind each problem, see our guide on why Google is not crawling your site.

Stage 2: Server and status codes (checks 6 to 9)

Status code checks confirm your server answers Googlebot with the right code. Errors on robots.txt itself can pause crawling for the whole site.

Check 6: key templates return 200

• Test: pick one URL per template (home, category, product, service, blog post) in each language and check the status code.
• Pass: every one returns 200 directly, with no redirect.
• Usual cause: a plugin, theme or app update that breaks one template.
• Fix: repair the template, then recrawl that whole template group.

Check 7: no redirect chains or loops

• Test: in your crawler, export redirect chains and loops.
• Pass: each redirect goes to its final URL in one hop, and internal links point to the final URL.
• Usual cause: old migrations stacked on top of each other, such as http to https to www to a new slug.
• Fix: point every old URL straight at the final one, and update internal links.

Google says its crawlers follow up to 10 redirect hops by default. One hop is still the target.

Check 8: 404, 410 and soft 404 pages are intended

• Test: list 4xx URLs from your crawler and soft 404s from the Page indexing report.
• Pass: only pages that are truly gone return 404 or 410, and no live page looks empty.
• Usual cause: seasonal pages. A Dubai Shopping Festival or White Friday page that shows “no offers” with a 200 code can be read as a soft 404.
• Fix: keep evergreen seasonal URLs live with useful content and next year’s dates, or redirect them to the closest relevant page.

Google lists an empty page or an error message as a soft 404 case.

Check 9: no 5xx or 429 errors, and robots.txt loads

• Test: check host status and response codes in Crawl Stats, and fetch robots.txt yourself.
• Pass: robots.txt returns 200, and the Crawl Stats host status shows no recent problems with robots.txt fetch, DNS or server connectivity.
• Usual cause: an overloaded host during a sale, or a CDN rule that rate limits Googlebot.
• Fix: add server capacity, cache heavy pages and remove rate limits on verified Googlebot.

Google’s HTTP status code documentation says “5xx and 429 server errors prompt Google’s crawlers to temporarily slow down with crawling.” URLs that keep returning server errors are eventually dropped. For robots.txt, a 4xx (except 429) is treated as no restrictions. A 5xx stops crawling for 12 hours, then Google uses the last cached copy for up to 30 days.

Stage 3: Links and architecture (checks 10 to 14)

Link checks confirm Google can follow your navigation and reach every page in a few clicks. This stage also catches crawl traps that waste requests.

Check 10: every link is a real a href link

• Test: view the rendered HTML of your menus, filters, pagination and “load more” buttons.
• Pass: each link is an a element with an href that holds a real URL.
• Usual cause: buttons or spans that use onclick, or href values like javascript:void(0).
• Fix: rebuild these as standard a href links. Keep the JavaScript for behaviour, not for the URL.

So which link is not crawlable? A link without a real a href. Google’s crawlable links guide says it “can only crawl your link if it’s an a HTML element” with an href attribute. For a elements without href, Google “can’t reliably extract URLs”. It also does not recommend routerLink or javascript: URLs.

Link markup	Can Google follow it?
a element with href=”/ar/services/”	Yes
a element with only an onclick handler	Not reliably
a element with href=”javascript:goTo()”	Not recommended
span or button with a routerLink	Not recommended

Check 11: important pages sit within 3 to 4 clicks

• Test: run a crawl from the home page and sort by crawl depth.
• Pass: your money pages sit within three to four clicks of the home page, a common rule of thumb.
• Usual cause: deep pagination, or products linked only from filtered views.
• Fix: add links from category hubs, breadcrumbs and related content blocks.

Check 12: no orphan pages

• Test: connect your crawler to Search Console and the XML sitemap, then list URLs found there but not in the crawl.
• Pass: every page you want indexed has at least one internal link.
• Usual cause: landing pages built for ads, or posts removed from menus during a redesign.
• Fix: link each page from a relevant hub, or retire it.

Our guide to how internal links help Google discover pages covers the linking fixes in detail.

Check 13: pagination works without rel=next/prev

• Test: open a long category or blog archive and check how page 2 and beyond are linked.
• Pass: each page links to the next with a href, and each page has its own canonical.
• Usual cause: old advice. Google’s pagination guide says Google “no longer uses” rel=next/prev, yet some themes still canonical every page to page 1.
• Fix: give each paginated page a self-referencing canonical. Google says “Don’t use the first page of a paginated sequence as the canonical page.”

Check 14: faceted navigation and parameters do not create traps

• Test: crawl a large category and count how many URLs come from filter, sort and tracking parameters.
• Pass: filter combinations do not multiply into thousands of near-copies.
• Usual cause: WooCommerce or Shopify filters that create a new crawlable URL for every combination.
• Fix: keep only useful filter pages crawlable, and stop linking to the rest.

On shops with many thousands of URLs, these traps are when crawl budget starts to matter.

Stage 4: Discovery (checks 15 to 17)

Discovery checks confirm your sitemap gives Google an accurate list of the pages you want found. A messy sitemap sends Google to redirects and errors.

Check 15: the XML sitemap loads and lists live URLs

• Test: open the sitemap, check the Sitemaps report in Search Console, and crawl the sitemap URLs.
• Pass: it returns 200, uses full absolute URLs, and lists only 200, indexable, canonical pages.
• Usual cause: plugins that add redirected, noindex or tag pages.
• Fix: exclude those page types. Split files that pass 50,000 URLs or 50MB uncompressed, which is Google’s limit per sitemap.

Check 16: sitemap, crawl and index line up

• Test: compare three lists: sitemap URLs, crawled URLs and indexed URLs from the Page indexing report.
• Pass: the three lists mostly match.
• Usual cause: pages in the sitemap with no internal links, or linked pages left out of the sitemap.
• Fix: link the orphans and add missing pages to the sitemap.

Check 17: lastmod dates are accurate

• Test: compare lastmod values with the real last edit date of a sample of pages.
• Pass: lastmod changes only when the main content changes.
• Usual cause: a plugin that sets every lastmod to today’s date.
• Fix: tie lastmod to real edits. Google uses lastmod only when it is consistently and verifiably accurate, and ignores priority and changefreq.

Stage 5: Rendering (checks 18 to 20)

Rendering checks confirm Google sees the same content and links that a visitor sees. Sites built on React, Next.js or Vue often load key content with JavaScript.

Check 18: rendered HTML holds the content and links

• Test: use URL Inspection and compare the crawled HTML with the page source.
• Pass: the main text, links and canonical tag appear in the rendered HTML.
• Usual cause: content that loads only after a click, a scroll or an API call that fails for bots.
• Fix: move key content and links into server-rendered or static HTML.

See rendered HTML vs source HTML for how to read the difference.

Check 19: CSS and JavaScript files are not blocked

• Test: check robots.txt and URL Inspection for blocked resources.
• Pass: Googlebot can fetch every file needed to build the page.
• Usual cause: an old Disallow rule on folders such as /wp-includes/ or /assets/.
• Fix: allow those folders, so Google can render the page the way visitors see it.

Check 20: navigation does not depend on JavaScript alone

• Test: turn off JavaScript in your browser, or crawl with rendering off, and try to reach key pages.
• Pass: main menus and category links still exist as a href links in the HTML.
• Usual cause: a menu built entirely by a client-side script.
• Fix: output the menu links in the HTML. Our guides on how Google renders JavaScript and JavaScript SEO for React and Next.js cover the fixes.

Stage 6: Duplicates and canonicals (checks 21 to 23)

Duplicate checks confirm each page lives at one URL that Google can pick as the main version. Duplicate URLs split crawling across copies.

Check 21: one version of each URL

• Test: request http and https, www and non-www, with and without a trailing slash, in upper and lower case.
• Pass: every variant redirects in one hop to a single version.
• Usual cause: WPML or Polylang language settings that add or drop the trailing slash on one language only.
• Fix: set one rule for every language and redirect the rest.

Google’s URL handling is case sensitive, so /Services and /services are two different URLs.

Check 22: canonical tags do not conflict

• Test: crawl and compare each page’s canonical with its sitemap entry, hreflang tags and internal links.
• Pass: all signals point to the same URL, and canonicals point to 200 pages.
• Usual cause: a template that canonicals to the wrong language or to a redirected URL.
• Fix: set self-referencing canonicals on main pages, and update the other signals to match.

Check 23: parameters do not create duplicate pages

• Test: filter your crawl for URLs with UTM tags, session IDs, sort orders and currency switches.
• Pass: parameter URLs canonical to the clean URL, and internal links never use them.
• Usual cause: campaign links pasted into menus, or a currency switcher for AED that adds a parameter.
• Fix: strip parameters from internal links and keep canonicals on the clean URL.

Stage 7: International and mobile (checks 24 and 25)

International checks confirm each language version is reachable and correctly paired. Mobile checks confirm the smartphone crawler sees the full page. For Arabic and English sites in the UAE, these two often fail together.

Check 24: /ar/ and /en/ hreflang pairs and Arabic URLs

• Test: crawl both language folders and check hreflang tags for return links. Test a few Arabic slugs.
• Pass: every page lists itself and its other language version, both pages link back, and Arabic slugs are percent encoded.
• Usual cause: an English page pointing to an Arabic page that does not point back, or Shopify Markets folders such as /ar-ae/ missing from the pairs.
• Fix: generate hreflang from one source, consider an x-default fallback, and keep Arabic URLs encoded in links and sitemaps.

Google’s hreflang guide says: “If two pages don’t both point to each other, the tags will be ignored.” Google also recommends percent encoding for non-ASCII characters in URLs, and gives an Arabic path as its example.

Check 25: mobile and desktop show the same content

• Test: crawl with the Googlebot Smartphone user agent and compare with a desktop crawl.
• Pass: the mobile version has the same text, links, structured data and meta tags.
• Usual cause: a mobile theme that hides tabs, footer links or reviews.
• Fix: keep the same content on mobile, and use layout to save space.

Google uses the mobile version of a site’s content, crawled with the smartphone agent, for indexing and ranking. It says to “make sure that your mobile site contains the same content as your desktop site.”

How to prioritise technical SEO audit findings before content work

Sort every failed check into blockers, leaks and hygiene. Blockers stop Google reaching whole sections, so fix them first. Leaks waste crawls or split signals, and hygiene items can wait for a normal sprint.

Finding	Impact	Effort	When
Disallow: / or site-wide noindex on production	Very high	Low	Today
Firewall challenging verified Googlebot	Very high	Low	Today
Key template returning 5xx or 404	High	Medium	This week
Navigation built without a href links	High	High	Before content work
Broken /ar/ and /en/ hreflang pairs	Medium	Medium	This sprint
Redirect chains after a migration	Medium	Low	This sprint
Inaccurate lastmod dates	Low	Low	Next sprint

Start the content plan once the blockers are fixed. If your team needs help fixing the developer side, our technical SEO services cover this work.

Verify with Crawl Stats and logs, then set a re-audit schedule

A crawler tool shows what could happen. Crawl Stats and server logs show what Googlebot actually did. Check both after each fix, then re-run the audit on a fixed schedule.

What to look for

The Crawl Stats report shows request counts, download size, response time and host status. You can break these down by response, file type, purpose and Googlebot type. After a fix, look for fewer errors and more 200 responses on the fixed sections.

Your logs fill the gaps. Filter for verified Googlebot requests, then check which sections it visits and which status codes it gets. Our guide on how Googlebot decides what to crawl explains what normal patterns look like.

When to re-audit

• After every launch, redesign or theme change.
• After a migration or a change of domain, CMS or CDN.
• Before and after big sales such as the Dubai Shopping Festival or White Friday.
• Every quarter for the five run-first checks, even when nothing changed.

If you want us to run this audit on your site, see our SEO audits and diagnostics service.

FAQ

Sources

• Google Search Central: Make your links crawlable
• Google Search Central: How Google interprets the robots.txt specification
• Google Search Central: Pagination, incremental page loading, and their impact on Google Search
• Google Search Central: How HTTP status codes, and network and DNS errors affect Google’s crawlers
• Google Search Console Help: Crawl Stats report
• Google Search Central: Verifying Googlebot and other Google crawlers
• Google Search Central: Block Search indexing with noindex
• Google Search Central: URL structure best practices
• Google Search Central: Tell Google about localized versions of your page
• Google Search Central: Build and submit a sitemap
• Google Search Central: Mobile-first indexing best practices
• Google: Google’s common crawlers (Google-Extended)
• Cloudflare: Verified bots