Guide for this site

Missing · Parseable robots.txt

Without robots.txt the agent has no entry policy.

Serve /robots.txt with User-agent and Allow/Disallow.

Missing · Directives for AI bots

OAI-SearchBot/PerplexityBot (AEO) are not GPTBot (training). Name them.

Allow search bots; Disallow training if that is your policy.

Missing · sitemap.xml

The sitemap is the inventory the agent can walk.

Publish /sitemap.xml and link it from robots.txt.

Missing · Content Signals

ai-train / search / ai-input is the policy new crawlers read.

Content-Signal: ai-train=no, search=yes, ai-input=yes (adjust to your case).

Missing · Rules by bot type

Search, agent, and training should not share a single rule.

Separate User-agent groups for search vs training.

Missing · Clear access policy

An ambiguous policy is read as 'touch nothing' or as 'everything is fair game'.

One sentence in llms.txt + consistent robots.

Partial · Markdown negotiation

Accept: text/markdown saves tokens versus navigation-heavy HTML.

If Accept includes text/markdown, respond with markdown for the page.

Missing · Quality llms.txt

An llms.txt with H1, summary, and curated links is the site map for agents.

Publish /llms.txt with H1, a blockquote, and 15–25 important URLs.

Partial · Structured data

Organization/Article/Product/Offer is what answer engines cite.

Minimum JSON-LD Organization; add Article or Product by mode.

Partial · AEO / GEO signals

Article, FAQ, VideoObject, or og:title is what answer engines cite. That is not charging.

JSON-LD Article/FAQPage/HowTo/VideoObject and canonical. Disallow training if you do not sell it.

Partial · Token efficiency

HTML packed with nav and scripts burns the agent's budget.

A markdown version or a main with content first.

Missing · Link headers (RFC 8288)

Link: rel=describedby / canonical helps discover llms.txt and feeds.

Add Link toward /llms.txt and the canonical.