Search and AI Optimizer
Enter a domain or a page URL. We read its robots.txt, sitemaps, feeds and llms.txt, check a sample of the pages they list, and show how well the site supports search engines, answer engines and AI assistants, with what to fix first.
What this checks
- robots.txtCrawler rulesRead the way RFC 9309 says crawlers should read it, for each major search and AI agent.
- SitemapsSitemaps and indexesValidity, limits, lastmod dates and URLs robots.txt blocks.
- FeedsRSS and AtomRequired fields, dates and how recently the site published.
- PagesListed URLsErrors, redirects, noindex and canonical tags on a sample.
Allowing a crawler or submitting a sitemap helps search and AI tools find a site. It does not guarantee indexing, ranking or citations in AI answers. llms.txt is reported as an optional, experimental file. Read the technical details for the limits, safe fetching and how each check works.