Deep 404 Scanner (deep-404-scanner)

Summary

Heavy multi-phase HTTP crawl: page inventory, SEO analysis, asset validation, external links, and security signals.

Classification

• Category: SEO
• Plugin id: deep-404-scanner
• Version: 2.0.0
• Target types: WEBSITE, WEBSHOP
• Weight class: HEAVY
• Risk level: not set on plugin object
• Browser required: No (HTTP + cheerio — not Playwright)
• Tier / group: Tier 5 — Heavy / deep scan

Schedule

• scheduleHints: { defaultInterval: '0 */12 * * *', priority: 8 }
• Prerequisites: [{ type: 'uptime', mode: 'not_down' }]

Configurable inputs

Public keys are derived from inputSchema: z.object({ ... }) (underscore-prefixed keys are runtime-only and omitted here).

Field	Notes
maxDepth	BFS link depth from homepage (default 3, max 10).
maxPages	Max pages to crawl (default 200, max 10 000).
checkAssets	HEAD-check images, CSS, JS, fonts, and media (default true).
maxAssets	Cap on unique assets validated (default 1000, max 5000).
checkExternalLinks	HEAD-check outbound links (default true).
maxExternalLinks	Cap on external URLs checked (default 500, max 2000).
timeout	Per-request timeout in ms (default 15000, range 5–120 s).
concurrency	Parallel requests per batch (default 3, max 10).
crawlDelay	Ms wait between batches plus throttle base delay (default 250).

Runtime / injected config

• _priorThrottleHint — worker may inject prior run throttle metrics so the next crawl starts with elevated delay when the site previously rate-limited scans.

Checks that produce findings

These checks run inside this plugin during a single test run. They are not separate tests in the database — each becomes a Finding linked to the Run for deep-404-scanner.

Missing canonical is tracked in crawl metrics (seoIssuesSummary.missingCanonical) and Site Map data but not emitted as a standalone finding from this plugin — use SEO Meta Tags or the Site Map tab for that.

Broken resources

Check	Typical severity	Why
Broken internal pages (404/5xx/unreachable)	INFO if ≤10 pages; WARNING if >10	Dead internal URLs waste crawl budget, leak link equity, and strand users on error pages.
Broken images	INFO if ≤5; WARNING if >5	Missing images break product and content pages and signal CMS or CDN deployment gaps.
Broken CSS files	WARNING	Stylesheet 404s often break layout sitewide on every page that references the bundle.
Broken JavaScript files	WARNING	Script 404s can break cart, navigation, analytics, and consent — often worse than a missing image.
Broken fonts/media	INFO	Broken fonts and media degrade polish and accessibility but rarely block core flows.
Broken external links	INFO if ≤10; WARNING if >10	Stale outbound links hurt credibility and SEO; many broken externals suggest neglected content.
Mixed content on HTTPS pages	WARNING	HTTP subresources on HTTPS pages trigger browser blocking and weaken transport security.
Scan throttled (rate-limit/WAF detected)	INFO	Documents that results may be incomplete because the site slowed or blocked the crawler — not a site defect.

SEO checks (site-wide crawl)

Check	Typical severity	Why
Missing <title>	INFO if ≤5 pages; WARNING if >5	Pages without titles are harder to index and produce poor snippets in search results.
Very short title (< 15 chars) / too long (> 60 chars)	INFO	Titles under 15 chars are often placeholders; titles over ~60 display chars truncate in SERPs. Clear short titles (15–50+) are fine — there is no mandatory 30-char floor.
Duplicate titles (>1 URL per title string)	INFO if ≤5 duplicate groups; WARNING if >5 groups	Duplicate titles confuse search engines about which URL to rank for a topic.
Missing meta description	INFO	Missing descriptions force Google to auto-generate snippets, reducing messaging control.
Description too short (< 50 chars) / too long (> 160 chars)	INFO	Suboptimal snippet length reduces SERP real estate and clarity.
Duplicate descriptions	INFO if ≤5 groups; WARNING if >5 groups	Same description on many URLs looks like thin/duplicate content to crawlers.
Missing H1	INFO	Missing primary headings weaken on-page structure and accessibility signals.
Duplicate H1	INFO	Multiple URLs sharing one H1 text compete for the same topical relevance.
Title identical to H1 (>3 pages only)	INFO	Identical title and H1 waste a second keyword opportunity in headings vs SERP title.
Missing Open Graph tags (>3 pages only; both og:title and og:image absent)	INFO	Social shares without OG metadata show generic previews instead of rich cards.

How each check works

1. Phase 1 — BFS crawl — Breadth-first from target.url up to maxDepth / maxPages. Each URL is fetched with fetchWithTiming (GET, TesteBot, follows redirects). Only responses with Content-Type containing text/html become pages; other content types are skipped silently. Cheerio extracts links, assets, SEO tags, mixed-content URLs, cookies, page type, and feature signals (login, checkout, B2B, etc.). Status ≥400 or unreachable → broken page list (except session-dependent shop paths on WEBSHOP targets — see caveats). AdaptiveThrottleManager backs off on 429/403/WAF signals between batches.
2. Phase 2 — SEO analysis — In-memory pass over collected pages (no extra HTTP). Thresholds: title stubs <15** chars / long **>60 chars (no 30-char minimum), description 50–160 chars. Duplicate title/description/H1 detected via maps. titleSameAsH1 and missingOgTags findings emit only when affected page count >3.
3. Phase 3 — Asset validation (if checkAssets) — Deduplicated asset registry HEAD-checked in batches (images, CSS, JS, fonts/media separately), capped at maxAssets. Status ≥400 or HEAD error → broken asset finding by type.
4. Phase 4 — External links (if checkExternalLinks) — Up to maxExternalLinks unique outbound URLs HEAD-checked (10 s timeout cap per link).
5. Phase 5 — Security — Mixed content counted from HTML parse during crawl (HTTP resources on HTTPS pages).
6. Artifacts & persistence — Writes page-inventory.json, asset-inventory.json, optional external-links.json and discovered-features.json. Worker persists CrawlSnapshot + DiscoveredPage rows and feeds the adaptive orchestrator and Site Map UI.
7. Throttle finding — After scoring, if totalThrottleEvents > 0, adds an INFO finding with backoff stats.
8. Score — (totalChecked − totalBroken) / totalChecked × 100 where broken includes pages, assets, and external links checked.

Caveats and concerns

• Doc vs implementation — Platform docs historically label this “Playwright”; the plugin is HTTP + cheerio only (no JS execution). Links and content rendered client-side are invisible.
• Non-HTML pages skipped — JSON, PDF, or application/* URLs return null from fetch and are not counted as broken pages even if linked.
• HEAD false positives — Asset and external-link checks use HEAD; servers that disallow HEAD may report broken CSS/JS/images incorrectly.
• WEBSHOP session paths — URLs matching cart/checkout/account paths (isSessionDependentUrl) skip broken-page reporting when fetch fails or returns 4xx, avoiding false positives on cookie-gated shops.
• Missing canonical — Collected and stored in snapshot metrics but no finding is emitted; use seo-meta-tags or Site Map filters.
• Coverage caps — Defaults crawl 200 pages, 1000 assets, and 500 external links; large sites get a sample, not exhaustive coverage.
• Fingerprint includes counts — Finding fingerprints embed broken counts (e.g. deep-404:pages:12); resolving one broken page can change the count and create a “new” finding on the next run rather than updating the same fingerprint.
• OG “missing” definition — A page counts as missing OG only when both og:title and og:image are absent (og:description alone does not satisfy the check).

Source

packages/test-plugins/src/plugins/deep-404-scanner.ts