Deep 404 Scanner (deep-404-scanner)

Summary

Heavy multi-phase crawl: inventory, SEO, assets, external links, security signals.

Classification

• Category: SEO
• Plugin id: deep-404-scanner
• Version: 2.0.0
• Target types: WEBSITE, WEBSHOP
• Weight class: HEAVY
• Risk level: not set on plugin object
• Browser required: Yes (Playwright)
• Tier / group: Tier 5 — Heavy / deep scan

Schedule

• scheduleHints: { defaultInterval: '0 */12 * * *', priority: 8 }
• Prerequisites: [{ type: 'uptime', mode: 'not_down' }]

Configurable inputs

Public keys are derived from inputSchema: z.object({ ... }) (underscore-prefixed keys are runtime-only and omitted here).

Field	Notes
maxDepth	Zod field in inputSchema — see source for defaults, min/max, and .describe().
maxPages	Zod field in inputSchema — see source for defaults, min/max, and .describe().
checkAssets	Zod field in inputSchema — see source for defaults, min/max, and .describe().
maxAssets	Zod field in inputSchema — see source for defaults, min/max, and .describe().
checkExternalLinks	Zod field in inputSchema — see source for defaults, min/max, and .describe().
maxExternalLinks	Zod field in inputSchema — see source for defaults, min/max, and .describe().
timeout	Zod field in inputSchema — see source for defaults, min/max, and .describe().
concurrency	Zod field in inputSchema — see source for defaults, min/max, and .describe().
crawlDelay	Zod field in inputSchema — see source for defaults, min/max, and .describe().

Runtime / injected config

• Prior throttle metrics may be injected for adaptive politeness.

What it does

• Runs as part of the worker test execution pipeline; results become Run rows and may create Finding records.
• Heavy multi-phase crawl: inventory, SEO, assets, external links, security signals.
• Crawl audit (deep-404-crawl-audit.ts, TESTE-NO-387): findings that need the link graph. All are INFO or WARNING and carry evidence.pages, so the per-page report lists them next to the page. Fingerprints are stable (no counts):
  • deep-crawl:orphan-pages: sitemap pages (200, indexable) the homepage cannot reach through followed links. Needs a complete crawl; INFO only when pages behind the depth limit were left unexplored.
  • deep-crawl:deep-pages: more than CRAWL_STRUCTURE_THRESHOLDS.maxClickDepth (3) clicks from the homepage, measured on the link graph (sitemap-seeded crawl depth is ignored).
  • deep-crawl:few-inlinks: one internal inlink, depth 2 or deeper.
  • deep-crawl:links-to-redirects / links-to-errors: evidence.pages are the linking pages (pagesRole: linking-pages), evidence.links the from/to pairs. Errors drop to INFO when the crawl was throttled.
  • deep-crawl:noindex-linked: noindex pages with followed internal links (evidence.details names the linking pages). Utility pages and nofollow links are exempt.
  • deep-crawl:faceted-urls / pagination-urls: crawled sort/display/filter URLs without noindex, robots block or canonical to the base page; paginated URLs without a self-referencing canonical.
  • deep-crawl:empty-anchors / generic-anchors: empty link text (image alt, aria-label and title count as text) and "read more" style anchors.
  • WEBSHOP: variant URLs (?color=, ?variant=, attribute_* ...) of one product are collapsed out of the duplicate title/description/H1/body groups. deep-crawl:variant-families is raised only for families whose variants lack a canonical to the product page.
  • Page inventory seoIssues gains the tags orphan, deep-click-depth, few-inlinks, links-to-redirect, links-to-error, noindex-linked, faceted-url, paginated-url, empty-anchor, generic-anchor, variant-family. Run metrics carry a crawlAudit JSON summary.
• Inbound links per page (TESTE-NO-400): the same link graph gives every 2xx page in the inventory inlinks (distinct crawled pages with a followed link to it; duplicates count once, self-links, nofollow links and links from session/app-auth pages do not count, links from noindex pages do; a link through a redirect counts for the page it lands on, and a requested URL that redirected gets no figures of its own), clickDepth (clicks from the homepage over followed links, null when no link path reaches it) and inlinkSources (the strongest linking pages, i.e. most inlinks themselves, as { url: path, anchor }, at most CRAWL_LINK_GRAPH_LIMITS.inlinkSourcesPerPage = 10, anchors cut to 60 characters, the best anchor per source: visible text over alt/aria-label over generic over empty). The run metric linkGraphComplete is 1 when the crawl visited everything it found, hit no depth limit and read the homepage; otherwise counts are lower bounds. The worker stores them on DiscoveredPage.inlinks / clickDepth / inlinkSources and CrawlSnapshot.linkGraphComplete; quick wins and the per-page site API read them.
• For finding titles, metrics, and artifacts specific to this plugin, refer to the implementation linked below.

Source

packages/test-plugins/src/plugins/deep-404-scanner.ts