Ai Search Readiness (ai-search-readiness)

Summary

llms.txt, AI crawler rules in robots.txt, structured data depth, and snippet-oriented signals.

Classification

• Category: SEO
• Plugin id: ai-search-readiness
• Version: 1.0.0
• Target types: WEBSITE, WEBSHOP
• Weight class: LIGHT
• Risk level: not set on plugin object
• Browser required: No (HTTP / Node / other — see source)
• Tier / group: Tier 1 — HTTP-only

Schedule

• scheduleHints: { defaultInterval: '0 4 * * 1', priority: 3 }
• Prerequisites: None declared on this plugin.

Configurable inputs

Public keys are derived from inputSchema: z.object({ ... }) (underscore-prefixed keys are runtime-only and omitted here).

Field	Notes
timeout	Zod field in inputSchema — see source for defaults, min/max, and .describe().
samplePages	Injected by the worker for this plugin on every run (crawl snapshot, else sitemap): the product and category page used by the crawler access probe.

Runtime / injected config

• None documented beyond standard worker context (target, config, logger, reportProgress). Underscore-prefixed keys are internal.

What it does

• Runs as part of the worker test execution pipeline; results become Run rows and may create Finding records.
• llms.txt, AI crawler rules in robots.txt, structured data depth, and snippet-oriented signals.
• Metrics: aiCrawlersBlocked, aiCrawlersAllowed, aiCrawlerCount (8 known bots). -1 on the crawler counts means robots.txt was not fetched. The AI Visibility card treats 0 allowed as critical (red), mixed as warning, and all-allowed as healthy — it does not treat a missing allowed count as "0 allowed".
• Crawler access probe (packages/test-plugins/src/ai-crawler-access.ts): requests the homepage, one product page, one category page and /llms.txt with the User-Agent of OAI-SearchBot, ChatGPT-User, GPTBot, Claude-User, ClaudeBot, PerplexityBot, Perplexity-User, Googlebot and Bingbot, each compared against a TesteBot control request. Per crawler and page the outcome is ok, robots_disallow (not requested), firewall_block (HTTP 403/406/451 or a WAF action header), challenge (CAPTCHA / JS challenge), login_redirect, auth_required (401), rate_limited, http_error, unreachable or inconclusive (the TesteBot control was refused too). Findings: ai-ready:ai-crawlers-http-blocked WARNING for search/answer crawlers (OAI-SearchBot, ChatGPT-User, Claude-User, PerplexityBot, Perplexity-User) that robots.txt allows but infrastructure refuses; ai-ready:ai-training-crawlers-http-blocked INFO for GPTBot/ClaudeBot; ai-ready:search-engines-http-blocked INFO for Googlebot/Bingbot (IP-verifying WAFs legitimately refuse a borrowed UA). Metrics: aiCrawlerAccess (per-crawler, per-page results, shown on the AI visibility page), aiCrawlersHttpBlocked, aiCrawlersHttpBlockedNames.
• UA-only limitation: the probe goes out from teste.no's own IP. It sends the crawler's User-Agent unchanged plus X-Teste-Bot and From headers so TesteBot stays identifiable; it never spoofs the crawler's IP. Results therefore show UA-based blocking only; every finding says so.
• For finding titles, metrics, and artifacts specific to this plugin, refer to the implementation linked below.

Source

packages/test-plugins/src/plugins/ai-search-readiness.ts