Robots.txt (robots-txt)

Summary

robots.txt presence, parsing, sitemap reference, and risky bot rules.

Classification

• Category: SEO
• Plugin id: robots-txt
• Version: 1.1.0
• Target types: WEBSITE, WEBSHOP
• Weight class: LIGHT
• Risk level: not set on plugin object
• Browser required: No (HTTP / Node / other — see source)
• Tier / group: Tier 1 — HTTP-only

Schedule

• scheduleHints: { defaultInterval: '0 */6 * * *', priority: 3 }
• Prerequisites: None declared on this plugin.

Configurable inputs

Public keys are derived from inputSchema: z.object({ ... }) (underscore-prefixed keys are runtime-only and omitted here).

Field	Notes
timeout	Zod field in inputSchema — see source for defaults, min/max, and .describe().

Runtime / injected config

• None documented beyond standard worker context (target, config, logger, reportProgress). Underscore-prefixed keys are internal.

Checks that produce findings

These checks run inside this plugin during a single test run. They are not separate tests in the database — each becomes a Finding linked to the Run for robots-txt.

Check	Typical severity	Why
robots.txt unreachable (fetch failed)	WARNING	Crawlers cannot read crawl policy — behavior falls back to guessing, and Search Console may report fetch errors.
robots.txt not found (404)	WARNING	Missing robots.txt implies full crawl allowed; you also lose a standard place to advertise sitemap URLs.
robots.txt returns non-200 status	WARNING	Many crawlers treat non-200 as unreachable — policy directives may be ignored entirely.
robots.txt is empty	INFO	An empty file allows all crawling but provides no sitemap pointer or bot-specific rules.
No Sitemap: directive	INFO	Search engines may discover sitemaps elsewhere, but omitting Sitemap: slows discovery and weakens sitemap hygiene.
Wildcard blocks all crawlers (Disallow: /)	WARNING	User-agent: * with Disallow: / (and no Allow: /) blocks the entire site from indexing — often accidental on production.
Googlebot blocked	WARNING	When wildcard does not block all, a Googlebot-specific Disallow: / removes the site from Google Search while other bots may still crawl.
Googlebot-Image blocked (when Googlebot is allowed)	INFO	Blocks image indexing in Google Images while text search may work — product galleries and visual SEO suffer.

How each check works

1. Fetch — GET {origin}/robots.txt via fetchTarget() (default timeout 10s). Network failure → single WARNING robots.txt is unreachable and run ends.
2. HTTP status — 404 → WARNING missing file. Non-200 (except handled 404) → WARNING bad status. 200 continues analysis.
3. Content checks (200 only) — Empty body → INFO. Missing /^sitemap:/im → INFO. Non-empty body parsed with parseRobotsTxt() (groups consecutive User-agent lines per RFC-style blocks).
4. Crawl blocking — isAgentBlocked(rules, agent) returns true when the agent group has Disallow: / without Allow: /. Priority: explicit agent group, else fallback to *.
5. Finding precedence — If wildcard blocks all → WARNING blocks-all only (Googlebot finding suppressed). Else if Googlebot blocked → WARNING. Else if Googlebot-Image blocked → INFO. Metrics set wildcardBlocksAll, googlebotBlocked, googlebotImageBlocked flags (0/1).

Caveats and concerns

• Not a full RFC 9309 engine — No crawl-delay, allow precedence on partial paths, or sitemap URL validation; only root block detection.
• Staging intent — Deliberate Disallow: / on staging will WARN if the wrong URL is registered as production.
• Fetch vs Google — Google may cache robots.txt; fixes may not reflect immediately in Search Console.
• Unreachable vs 403 — Any non-200 (including 401/403/500) shares the same “bad status” finding without distinguishing auth from server error.

Source

packages/test-plugins/src/plugins/robots-txt.ts