How to Handle URL Parameters Before They Waste Your Crawl Budget

URL parameters are the key=value pairs after the ? in a URL, used for tracking, sorting, filtering, pagination and sessions. Each combination is a separate URL Google can crawl. Handle them by type: link only to clean URLs, canonicalise tracking parameters, block crawl-trap filters in robots.txt, and turn high-demand filters into static pages.

What are URL parameters in SEO?

A URL parameter is a piece of text added after a question mark that changes or labels the page. Each parameter is a key and a value joined by an equals sign, and several parameters are joined with an ampersand. For SEO, the point is simple: every new combination is a new URL that search engines may crawl.

Take this category page from a Dubai fashion store:

/collections/abayas?color=black&size=54&sort_by=price-ascending&utm_source=instagram

It breaks down like this:

• Base URL: /collections/abayas is the page itself.
• The ? sign: marks where the query string starts.
• Keys and values: color=black, size=54, sort_by=price-ascending and utm_source=instagram.
• The & sign: separates one parameter from the next.

Google’s URL structure guidelines ask for exactly this format: = for key-value pairs and & between parameters. The same page warns that “overly complex URLs, especially those containing multiple parameters, can cause problems for crawlers”.

Parameters are not bad in themselves. Your shoppers need filters, and your marketing team needs UTM tags. The problem starts when crawlers find thousands of parameter URLs that show the same products in a different order.

The 8 types of URL parameters and what each does to crawling

Most parameters fall into eight types. Some change what the page shows, some only change the order, and some change nothing at all. That difference decides whether a URL should ever be indexed.

Parameter type	Example	Changes content?	Should it be indexed?
Tracking	?utm_source=instagram, ?gclid=, ?srsltid=	No	No
Sort	?sort_by=price-ascending, ?orderby=price	Order only	No
Filter	?color=black&size=54	Yes, narrows the list	Sometimes
Pagination	?page=2	Yes, next set of items	Yes, as its own page
Internal search	?q=gold+watch	Yes, but endless	No
Action	?add_to_cart=123	No, triggers a task	No
Session ID	?sessionid=8f3a	No	No
Language or currency	?lang=ar, ?currency=AED	Yes, or prices only	Language yes, as a subfolder

Tracking, session and action parameters are mostly crawl waste. Sort parameters reshuffle the same products. Filters are the hard case, because a few of them match real searches. Pagination is different again: Google treats each page in a sequence as a separate page, so it needs to stay crawlable.

How parameters waste crawl budget: what Google has actually said

Parameters waste crawl budget because Google has to fetch a URL before it can tell whether that URL is useful. Crawl budget is the number of URLs Google can and wants to crawl on your site in a given period. We cover the basics in our guide to what crawl budget is and when to worry about it.

Google has been open about how big this problem is. On its Search Off the Record podcast, Gary Illyes and Martin Splitt shared Google’s top crawling challenges, as reported by Search Engine Roundtable on 3 February 2026:

• Faceted navigation: 50%
• Action parameters such as ?add_to_cart=true: 25%
• Irrelevant parameters such as UTM tags and session IDs: 10%
• WordPress plugins or widgets: 5%
• Other: 2%

Faceted navigation, action parameters and irrelevant parameters alone make up 85 of those points. In an earlier episode, as Search Engine Journal reported, Illyes said a URL can take “de facto infinite” parameters, and that ecommerce sites are hit hardest.

Google’s faceted navigation guide explains why. Filters can create near-endless URL spaces, and crawlers “can’t determine whether the URLs are going to be useful without crawling first”. Four filters with five values each, plus three sort orders, already produce thousands of crawlable URLs for one category, once you count every mix of filters left on or off.

Who really needs to worry

Google’s crawl budget guide is written for sites with 1 million or more unique pages that change weekly, or 10,000 or more pages that change daily. Google calls these “not exact thresholds”. Smaller stores rarely hit these limits, but parameters can create far more crawlable URLs than you have products. Your Crawl Stats report shows whether that is happening. If Googlebot spends its time on sort orders, new pages can wait longer to be crawled. If you want the difference between demand and server limits, read crawl budget vs crawl rate limit.

Canonical, noindex, robots.txt, 404 or fragment: what each one does

Each control solves a different problem: robots.txt stops crawling, while canonical and noindex shape indexing but still need a crawl. A 404 or a fragment stops the URL from existing as a separate page at all.

Control	Stops crawling?	Stops indexing?	Best for
Robots.txt disallow	Yes	Not fully, blocked URLs can still be indexed from links	Sort, action, internal search, low-value filters
rel=canonical	No, may reduce it over time	Not directly, it is a hint for which URL to show	Tracking parameters, near-duplicate pages you want crawled
Noindex tag	No	Yes	Pages users need but Google should not show
404 status	Reduces it over time, Google still checks now and then	Yes	Empty or nonsensical filter combinations
URL fragment (#)	Yes, Google ignores fragments	No separate URL exists	Filters you never want crawled

Why stacking controls backfires

A common answer on forums, and in the AI Overview we saw for this topic in November 2026, is to add a canonical, a noindex tag and a robots.txt disallow at the same time. This does not work. If robots.txt blocks a URL, Google never fetches it, so it never sees the noindex or the canonical on that page. You can end up with “Indexed, though blocked by robots.txt” in Search Console.

Pick one main control per parameter. If you want Google to read a tag, the URL must stay crawlable. If you want to stop crawling, use robots.txt and accept that tags on those URLs will not be read.

Noindex and canonical do not save crawl budget

Google’s crawl budget guide says plainly not to use noindex to save crawl budget, because Google still has to request the page to see the tag. The faceted navigation guide says rel=canonical “may” reduce crawling of non-canonical versions over time. So canonical is a consolidation tool, not a crawl control. If crawl waste is the problem, robots.txt or removing the links is the fix.

The same guide adds that rel=nofollow has to be on every link to the URL, and is generally less effective long term than robots.txt. Google also warns against using robots.txt to temporarily shift crawl budget to other pages. Set rules you intend to keep.

The decision table: the right control for each parameter

Here is the main control for each parameter type, with examples from typical UAE stores. Start with internal links: every menu, filter and breadcrumb should point to clean URLs. Then apply the single control below.

Parameter	UAE example	Control	Why
Tracking	?utm_source=instagram&utm_campaign=dsf	Self-referencing canonical on the clean page, never link internally	Ad and social links must still land, and the canonical points to the clean URL
Merchant Center result ID	?srsltid=123xyz	Self-referencing canonical, leave crawlable	Added to clicks from free product listings
Sort	?sort_by=price-ascending	Robots.txt disallow	Same products in a new order
Low-demand filter	?color=black&size=54	Robots.txt disallow, or a fragment	Combinations explode with no search demand
High-demand filter	?color=black on abayas	Static page such as /abayas/black/	Real searches, needs its own title and copy
Pagination	?page=2	Crawlable, self-canonical	Google treats each page as separate
Internal search	?q=oud	Robots.txt disallow	Endless, thin result pages
Action	?add_to_cart=123	Robots.txt disallow, use buttons not links	25% of Google’s crawl challenges
Session ID	?sessionid=8f3a	Remove from URLs, use cookies	Google advises against session IDs in URLs
Language	?lang=ar	Move to /ar/ subfolder with hreflang	Each language needs a stable URL
Currency	?currency=AED	Do not create links to it, canonical to the clean URL	Prices change, content does not

Two rows need more detail. The srsltid parameter is a result ID that Merchant Center adds to the URLs people click from your free product listings. It is a tracking parameter, so treat it like a UTM tag. On pagination, Google’s ecommerce pagination guide says to give each page a unique URL and not to use the first page as the canonical for the whole sequence.

Clean internal linking does a lot of the work here. Our guide on how internal links help Google discover pages explains why every link you control should point to the version you want indexed.

Robots.txt patterns for parameters that actually match

Robots.txt rules match from the start of the URL path, so a parameter rule needs a wildcard in front. The * sign matches any characters, and $ marks the end of the URL. A rule written without these often blocks far less than you think.

Google’s robots.txt specification confirms how these work. Paths are case-sensitive. When an allow and a disallow rule both match, Google uses the most specific rule, and on a tie it uses the least restrictive one.

Rule	What it does
Disallow: /*?*sort_by=	Blocks any URL with sort_by= in its query string
Disallow: /*?*add_to_cart=	Blocks add to cart links that use an underscore
Disallow: /*?*add-to-cart=	Blocks the WooCommerce spelling with hyphens
Disallow: /*?*filter_	Blocks WooCommerce layered filter keys
Disallow: /*?*q=	Blocks internal search results
Disallow: /*?*products=	Google’s own example for a filter parameter
Allow: /*?products=all$	Google’s example of keeping one exact URL open
Disallow: /?sort=	Only matches the homepage, a common mistake

That last line appears in at least one popular guide on this topic. Because matching starts at the beginning of the path, it only covers URLs that start with /?sort=, which is the homepage. Your category pages with ?sort= stay fully crawlable.

Be careful with short keys. A rule for q= also matches any key that ends in q, such as faq=. Check your real URLs before you deploy.

Test before you deploy

Avoid a blanket Disallow: /*? unless you are sure no parameter URL should ever rank. It also blocks paginated pages and any filter you later want indexed. Before you go live, run a crawl of your site with the new rules applied, using a crawler that lets you test a custom robots.txt file. Make sure your money pages still return as allowed.

When a filter deserves its own indexable page

A filter deserves its own page when people search for it. If “black abaya dubai” has real demand, a page for black abayas can rank for it. A filtered parameter URL is a weak way to do that, so build a proper static page instead.

Use this demand test before you index any filter:

1. Type the category plus the filter value into Google and note the autocomplete suggestions.
2. Check Search Console for queries that already contain the filter value.
3. Look at the top results. If they are category pages for that exact filter, there is room for yours.
4. Confirm the filter returns enough products to make a useful page, not two or three items.

If a filter passes, give it a clean URL such as /abayas/black/, a unique title and heading, a short intro, and links from the main category. Seasonal pages for DSF or White Friday follow the same rule: build them as static pages, not as a sale=true filter. Everything that fails the test stays blocked or behind a fragment.

Shopify, WooCommerce and Magento: where parameters come from

Shopify uses filter.v. and sort_by, WooCommerce uses orderby, filter_ and add-to-cart, and Magento uses ___store and layered navigation. Knowing the names lets you write rules that match.

Shopify

Shopify collection filters create parameters that start with filter.v., and sorting adds sort_by. Point your rules at those keys. Check what your store’s robots.txt already blocks before adding rules, so you do not duplicate or contradict them. If you run Shopify Markets with subfolders such as /en-ae/ and /ar-ae/, check that each subfolder keeps its own canonical and hreflang. We list more store issues in Shopify SEO in Dubai: 9 common issues.

WooCommerce

WooCommerce adds ?orderby= for sorting, ?filter_ keys for layered filters and ?add-to-cart= for action links. Block orderby and add-to-cart in robots.txt. For filters, apply the demand test, then block the rest with Disallow: /*?*filter_ in robots.txt. Note the hyphens in add-to-cart, which a rule written for add_to_cart will not match.

Magento

Magento’s layered navigation creates filter parameters on every category, and store switching can add ___store to URLs. Keep store views on clean URLs and make sure the switcher does not leak ___store into internal links. Our Magento SEO technical checklist covers layered navigation in more depth.

For stores with tens of thousands of SKUs, these rules are part of a wider plan. Our technical SEO for large inventories work starts with exactly this kind of parameter map.

Bilingual and GCC sites: ?lang= and ?currency= vs subfolders

Language should live in the path, not in a parameter. Use /ar/ and /en/ subfolders (or /ar-ae/ and /en-ae/) with hreflang, and keep currency out of indexable URLs. This gives each language version a stable URL that Google can crawl, index and pair.

If a ?lang=ar parameter relies on a cookie or a redirect, Googlebot may only see the default language. It also mixes with other parameters, so you get ?lang=ar&sort_by=price and many more copies. Subfolders avoid all of that.

Keep these rules for bilingual stores:

• Each language page has a self-referencing canonical. Never canonicalise the Arabic page to the English one.
• Hreflang pairs point to clean subfolder URLs, never to parameter URLs.
• Currency switching (AED, SAR) changes prices, not content. It should not create new indexable URLs. If a ?currency= URL exists, canonical it to the clean page and keep it out of internal links.
• If you sell to the UAE and Saudi Arabia, use market subfolders such as /en-ae/ and /en-sa/ instead of a currency parameter.

How to find parameter waste in 15 minutes

You can size the problem with Search Console and a crawler in about 15 minutes. Look at what Googlebot requests, then at what Google indexed, then at where the parameters come from. Log files confirm the picture on bigger sites.

1. Crawl Stats (5 minutes). In Search Console, open Settings, then Crawl stats. Check total requests over time and the breakdown by response and file type. Then open the example URLs and count how many contain a ?.
2. Page indexing (5 minutes). Open the Page indexing report and look for parameter URLs under “Alternate page with proper canonical tag”, “Duplicate without user-selected canonical” and “Indexed, though blocked by robots.txt”. The first is usually fine. The other two point to a missing or conflicting control.
3. Crawler (5 minutes). Run Screaming Frog or Sitebulb on a key category and filter crawled URLs by ?. Any parameter URL found through internal links is a link you should clean.
4. Logs (for large sites). Filter Googlebot hits by query string and group them by parameter key. This shows which keys eat the most requests.

If the numbers look large, our SEO audits and diagnostics team can review your parameter URLs, robots.txt rules and crawl stats as part of a technical SEO audit.

What replaced the Search Console URL Parameters tool

Nothing replaced it. Google shut down the URL Parameters tool on 26 April 2022, saying it had become much better at guessing which parameters are useful. Your controls today are robots.txt, canonicals, clean links and better URL design.

Google said that only about 1% of the parameter settings in the tool were useful for crawling, as Search Engine Roundtable reported. Guides that still tell you to configure it are out of date.

What not to do now:

• Do not stack robots.txt with noindex or canonical on the same URLs.
• Do not rely on noindex to cut crawling.
• Do not use # fragments to change real content you want indexed. Google’s URL structure guide warns against it.
• Do not list parameter URLs in your XML sitemap. List only the clean URLs you want indexed.
• Do not block every parameter by default without checking pagination and filters that rank.

For a wider view of how crawling feeds indexing and ranking, see our guide to how search engines crawl, index and rank content.

FAQ

Sources

• Google Search Central: Managing crawling of faceted navigation URLs
• Google Search Central: Crawl budget management for large sites
• Google Search Central: URL structure best practices
• Google Search Central: How Google interprets the robots.txt specification
• Google Search Central: Pagination and incremental page loading
• Google Search Central Blog: Spring cleaning, the URL Parameters tool
• Google Merchant Center Help: About key event tracking and the result id
• Search Engine Roundtable: Google’s top crawling challenges
• Search Engine Roundtable: Google to kill the URL Parameters tool on April 26th
• Search Engine Journal: Google warns URL parameters create crawl issues