agent-search-optimisation

Agent Search Optimisation

Quick start

When given a website URL, produce an evidence-based optimisation plan for AI agents and AI search.

1. Crawl the public site surfaces:
   • homepage
   • robots.txt
   • sitemap.xml and sitemap index
   • llms.txt / llms-full.txt if present
   • key navigation pages
   • representative article, product, project, docs, pricing, and about pages
2. Audit agent-readiness:
   • crawlability and renderability
   • canonical URLs
   • sitemap coverage
   • structured data
   • language alternates
   • entity clarity
   • answer-oriented page summaries
   • machine-readable indexes or APIs
   • internal linking and topic hubs
3. Research current AI search best practices before making claims about current platforms.
4. Create a prioritized roadmap with:
   • quick wins
   • technical fixes
   • content changes
   • agent-facing data surfaces
   • measurement plan

Workflow

1. Clarify the goal

Infer the likely goal from the user request. Ask only when necessary.

Common goals:

• increase AI answer citations
• make agents understand a product or portfolio
• expose documentation to coding agents
• improve local/business discovery in AI search
• prepare content for retrieval-augmented systems
• control AI crawler access

2. Collect evidence

Check these URLs where applicable:

{site}/
{site}/robots.txt
{site}/sitemap.xml
{site}/llms.txt
{site}/llms-full.txt
{site}/.well-known/

Also inspect at least 5 representative pages when the site has enough content:

• homepage
• about/company/profile page
• main collection/archive page
• one detail page
• one recent article/docs page

3. Score the site

Use a 0-3 score for each area:

Area	0	1	2	3
Crawlability	blocked/broken	partially crawlable	mostly crawlable	clean HTML + clear policy
Discovery	no sitemap	partial sitemap	complete sitemap	sitemap index + freshness
Structured data	none	basic metadata	JSON-LD on some templates	complete schema graph
Entity clarity	vague	some entities	clear entities	entity graph + IDs
Content extractability	thin/visual	prose only	summaries present	answer blocks + JSON
Language/canonicals	absent	inconsistent	mostly correct	canonical + hreflang complete
Agent surface	none	llms.txt only	index/feed	API/search/content endpoints
Measurement	none	traffic only	search console	AI/retrieval benchmark

4. Recommend changes

Prioritize in this order unless the site context suggests otherwise:

1. Fix public crawl/discovery basics.
2. Add canonical URLs, metadata, and language alternates.
3. Add JSON-LD and entity IDs.
4. Add answer-oriented summaries to important pages.
5. Add topic hubs and internal links.
6. Add llms.txt as an orientation layer.
7. Add machine-readable content index.
8. Add semantic search/API only when the corpus is large enough.
9. Add measurement and recurring evaluation.

5. Deliver the plan

Structure the final answer as:

1. Executive summary
2. What I checked
3. Current strengths
4. Gaps and risks
5. Prioritized roadmap
6. Implementation details
7. Measurement plan
8. Open questions / assumptions

Output rules

• Do not claim a file or feature exists unless verified.
• Mark unverified items clearly.
• Prefer durable web standards over hype.
• Treat llms.txt as additive, not a replacement for HTML, sitemaps, metadata, or structured data.
• Separate discovery from access control; robots.txt is not security.
• Include concrete examples when possible.
• Keep recommendations implementation-ready.
• Use current web research for AI search platform behavior, crawler policies, and new conventions.
• If the user provides a private repo or codebase, inspect implementation before suggesting exact code changes.

Advanced features

See REFERENCE.md for audit criteria, schema recommendations, llms.txt guidance, roadmap templates, and API examples.

See EXAMPLES.md for output examples and reusable prompts.

Use scripts/audit-agent-readiness.mjs for a lightweight first-pass technical audit.