Centralized crawl/index control for faceted navigation, one classifier, one registry, five tiers.
Faceted navigation (filterable product/listing pages) is where users combine filters: brand, color, size, price, sort, page. Every combination is a distinct URL. With n facets of k values each you have on the order of (k+1)ⁿ crawlable URLs, four facets of six values is ~2,400 combinations per category, three or four orders of magnitude more URLs than you have products. Left unmanaged, crawlers burn their budget on near-duplicate and empty filter URLs and never reach the pages that matter, while the index fills with thin duplicates.
This repo is the disciplined way to decide, from one source of truth, which filter URLs are worth indexing, which are crawlable-but-not-indexed, which get canonicalized, and which are invisible to crawlers, and to emit the matching signals coherently. The core is stdlib-only Python that ports cleanly to any rendering layer.
Sibling projects fix SEO at the CDN edge or ship a small faceted-nav example inside a starter. This repo goes deep on faceted navigation specifically and stands alone.
Every faceted URL belongs to exactly one tier:
| Tier | Name | Signals emitted | Examples |
|---|---|---|---|
| 1 | Indexable | self-canonical, index, follow, in sitemap, internally linked |
base category page; curated single-facet ?color=black; occasionally two facets |
| 2 | Canonicalized | crawlable, rel="canonical" → underlying page |
presentation-only variants: ?sort=price, ?view=list, ?per_page=48 |
| 3 | Noindex | crawlable, noindex, follow, no cross-canonical (canonical = None) |
uncurated multi-facet combos; pagination beyond page 1 |
| 4 | Blocked | blocked in robots.txt |
tracking/session params, arbitrary ranges ?price_min=…, internal search ?q= |
| 5 | Not a URL | never appears in the address bar | pure client UI state (hover, compare-mode toggle), documented, never emitted |
The classifier emits the first four. Tier 5 never reaches a URL, so there is no enum member for it.
noindex+ a cross-canonicalto a different URL is contradictory. Google may drop both signals or de-index the canonical target. Tier-3 pages are therefore self-canonical or have no canonical, never canonical-to-base while alsonoindex. Here, Tier 3 returnscanonical=None.- Adding a new facet defaults to Tier 3. Promotion to Tier 1 is an explicit registry opt-in, so a new facet can never silently promote thousands of URLs.
- Empty-result facet pages should return a real 404 (or redirect), a
200with an empty product list is a soft 404 that wastes crawl budget. That is a rendering-layer responsibility (see wiring). urllib.robotparserdoes not support Google's*/$wildcards, it treats/*?*x=as a literal prefix and silently passes URLs that should be blocked. This repo ships its own faithful matcher (seerobots.py) and a test that demonstrates the stdlib bug.
No dependencies for the core; Python 3.11+.
git clone <this-repo> && cd faceted-nav-seo
python examples/classify_population.pyThe demo loads examples/catalog.json, registers its
curated Tier-1 set, synthesizes a realistic population of faceted URLs,
classifies every one, and prints the tier distribution, the "what % of crawl
is wasted" view:
Tier distribution
--------------------------------------------------------
Tier 1 INDEXABLE (index, follow) 13 3.6% #
Tier 2 CANONICALIZED (canonical -> base) 22 6.1% ###
Tier 3 NOINDEX (noindex, follow) 293 81.4% ########################################
Tier 4 BLOCKED (robots.txt) 32 8.9% ####
--------------------------------------------------------
Total URLs discovered : 360
Worth indexing (T1) : 13 (3.6%)
Wasted crawl (T3+T4) : 325 (90.3%)
It then prints the generated positive-list sitemap and robots.txt, and
validates that every Tier-1 URL stays crawlable while every Tier-4 URL is
blocked.
from faceted_nav import FacetRegistry, classify_request, IndexingTier
registry = FacetRegistry()
registry.register_indexable_combination("shoes") # base page
registry.register_indexable_combination("shoes", {"color": "black"}) # curated facet
tier, canonical, meta_robots = classify_request(
"shoes", "color=black", base_url="https://shop.example.com", registry=registry
)
# tier == IndexingTier.INDEXABLE
# canonical == "https://shop.example.com/category/shoes?color=black"
# meta_robots == "index, follow"
# An uncurated combination defaults to Tier 3, note the null canonical:
classify_request("shoes", "color=black&size=10", registry=registry)
# Classification(tier=NOINDEX, canonical=None, meta_robots="noindex, follow")The whole package is parameter-driven, not URL-pattern-driven: a single
classify_request(category, query_params) backed by
one FacetRegistry.
- The classifier reads the registry to decide whether a facet combination is Tier 1.
- The sitemap generator reads the same registry, so
the sitemap can never drift from the classifier: a URL is in the sitemap iff
the classifier calls it
INDEXABLE. - Default is noindex. A combination is Tier 1 only if it was explicitly registered. Promotion is deliberate.
- Normalization is consistent at write and read time (lowercase + trim), so
registering
{"color": "Black"}matches an incoming?color=black.
from faceted_nav import generate_sitemap
print(generate_sitemap("https://shop.example.com", registry, lastmod="2026-05-22"))?color=black&size=10 and ?size=10&color=black are distinct URLs with
identical content. normalize.py produces one
canonical ordering (sort by key, lowercase/trim values, drop empties, preserve
multi-valued params) and tells you when to 301:
from faceted_nav import canonical_redirect_target
canonical_redirect_target("https://x.com/category/shoes?size=10&color=black")
# -> "https://x.com/category/shoes?color=black&size=10" (301 to this)
canonical_redirect_target("https://x.com/category/shoes?color=black&size=10")
# -> None (already canonical; no redirect)classify_request returns a plain (tier, canonical, meta_robots) tuple, the
business logic. Your view just emits the tags. The library is framework-agnostic
on purpose; here is the shape in any templating layer:
result = classify_request(category, request.query_params,
base_url=SITE_BASE, registry=registry)
# 1. Empty result set is a real 404, NOT a 200 with an empty list.
if not products:
return Response(status_code=404)
# 2. Tier 4 is blocked at robots.txt; you generally won't render it, but if you
# do, honor the signal defensively.
if result.tier is IndexingTier.BLOCKED:
return Response(status_code=404)
# 3. Emit the head tags from the classification.
head = f'<meta name="robots" content="{result.meta_robots}">'
if result.canonical:
head += f'\n<link rel="canonical" href="{result.canonical}">'
# 4. Optionally 301 non-canonical parameter orderings before rendering.
target = canonical_redirect_target(str(request.url))
if target:
return RedirectResponse(target, status_code=301)Three responsibilities stay in the rendering layer because the classifier can't see them: serving a real 404 on empty results, performing the 301, and not exposing Tier-5 state as URLs.
Which filter combinations are index-worthy is a business/SEO decision specific to your search demand. This repo is the mechanism; the curated registry is yours to fill in based on keyword research and your own data. Register the combinations real users search for ("black shoes") and leave the long tail at the Tier-3 default.
Don't cloak. Serve crawlers the same content and signals you serve users. The tiers here are about which URLs to expose and how, never about showing different content to bots.
faceted_nav/
classifier.py # IndexingTier, classify_request, FacetRegistry (source of truth)
normalize.py # canonical query ordering + the 301 decision
sitemap.py # positive-list sitemap, read from the same registry
robots.py # Tier-4 robots.txt generator + wildcard-aware validator
examples/
catalog.json # demo catalog + facets + a curated Tier-1 set
classify_population.py # the out-of-the-box demo
robots.txt # shipped Tier-4 example
tests/ # all five tiers + normalization + sitemap + robots
pip install -e ".[dev]"
ruff check .
pytest
python examples/classify_population.pyCI (GitHub Actions) runs lint, the test suite, and the demo on every push/PR across Python 3.11–3.13.
Issues and PRs welcome. Please keep the core stdlib-only (the dependency
lightness is the point, it lifts cleanly into any stack), add tests for new
behavior, and run ruff + pytest before opening a PR. This project keeps to
one sharp theme: faceted-navigation crawl/index control.
MIT.