User-agent: *
Allow: /

# -----------------------
# Private / non-indexable areas
# -----------------------
Disallow: /admin/
Disallow: /cms/
Disallow: /dashboard/
Disallow: /preview/
Disallow: /staging/
Disallow: /internal/
Disallow: /api/

# Site search results: thin and infinitely generatable.
Disallow: /search/
Disallow: /*?s=

# -----------------------
# Sitemap
# -----------------------
# ONE entry on purpose. sitemap.xml is the canonical index; /sitemap_index.xml
# now 301s to it rather than serving a second, drift-prone copy.
Sitemap: https://ployo.ai/sitemap.xml

# -----------------------
# LLM discovery
# -----------------------
# Curated overview for AI assistants (ChatGPT, Claude, Perplexity, AI Overviews):
# https://ployo.ai/llms.txt  (llmstxt.org standard)

# -----------------------
# Removed 2026-08-01, do not reinstate without reading this
# -----------------------
# 1. WordPress rules (/wp-admin/, /wp-login.php, /xmlrpc.php, /wp-json/,
#    */trackback/, ?replytocom=). This has been a Next.js site since the
#    migration off vettio.com/blog. None of those paths exist.
#
# 2. Feed blocking (/feed/, /*rss*, /*atom*). `Disallow: /*rss*` was blocking
#    OUR OWN feed at /diary/rss.xml, which the diary layout advertises via
#    `alternates.types['application/rss+xml']`. We were telling crawlers to
#    ignore a feed we were simultaneously asking them to subscribe to.
#
# 3. Tracking-parameter and blanket query-string blocking (/*?utm_, /*?gclid=,
#    /*?fbclid=, /*?msclkid=, and the catch-all /*?* and /*&*). This is the
#    wrong tool for the job and actively harmful. Blocking a URL in robots.txt
#    stops Google FETCHING it, which means Google never sees the rel=canonical
#    on the page and cannot fold the tracked URL into the clean one. The
#    documented outcome is "Indexed, though blocked by robots.txt": a
#    contentless duplicate in the index. It mattered here because the diary
#    cross-poster shares UTM-tagged links to LinkedIn, so those are exactly the
#    URLs external traffic and external crawlers arrive on. Every page now
#    declares its own canonical, which is the correct mechanism for parameter
#    consolidation. Let Google crawl them and canonicalise them.