AI Source Review (ai-source-review)
Summary
LLM review of fetched HTML source for security and quality issue categories.
Classification
- Category:
SECURITY - Plugin id:
ai-source-review - Version:
1.0.0 - Target types:
WEBSITE,WEBSHOP - Weight class:
HEAVY - Risk level:
LOW - Browser required: No (HTTP / Node / other — see source)
- Tier / group: Tier 6 — AI-powered
Schedule
- scheduleHints:
{ defaultInterval: '0 3 1 * *', priority: 10 } - Prerequisites:
[{ type: 'uptime', mode: 'not_down' }]
Configurable inputs
Public keys are derived from inputSchema: z.object({ ... }) (underscore-prefixed keys are runtime-only and omitted here).
| Field | Notes |
|---|---|
maxPages |
Pages to analyze (1–10, default 5): homepage plus discovered internal links. |
timeout |
Per-page fetch timeout in ms (default 20000). |
model |
OpenAI model name (default gpt-4o); overridden at runtime by team aiSourceReviewModel when set. |
Runtime / injected config
_aiCapBlocked— when the team AI monthly cap is reached, the run skips withpassedstatus and askippedmetric (no findings).- Worker merges
teamSettings.aiSourceReviewModelintoconfig.modelbefore execution (not a separate_aiSourceReviewModelkey). - OpenAI usage returned as
_aiUsageon results for billing (recordAiUsagein worker).
Checks that produce findings
These checks run inside this plugin during a single test run. They are not separate tests in the database — each becomes a Finding linked to the Run for ai-source-review.
Finding titles are dynamic (AI-generated, deduplicated by category plus normalized title). Only the Exposed Secrets & Keys category can remain CRITICAL after post-processing.
| Category | Typical severity | Why |
|---|---|---|
| Exposed Secrets & Keys | CRITICAL (private/secret material); downgraded to INFO for known public keys | Hardcoded secrets in HTML/JS can be scraped by anyone viewing page source — immediate credential rotation risk. |
| Debug & Dev Artifacts | WARNING / INFO | Debug logging, source maps, and verbose errors in production leak internals and aid attackers. |
| Code Quality Issues | WARNING / INFO | Patterns like eval, unsafe innerHTML, and blocking scripts increase XSS and maintenance risk. |
| Security Vulnerabilities | WARNING / INFO | Open redirects, missing postMessage origin checks, and tokens in URLs are exploitable client-side issues. |
| Structured Data & SEO | WARNING / INFO | Malformed JSON-LD or wrong canonicals hurt rich results and can mislead crawlers. |
| Hardcoded URLs & Env Leaks | WARNING / INFO | Staging/localhost URLs and env hints in production source expose infrastructure and mis-deployments. |
How each check works
- Prerequisites — Requires
AI_ENRICHMENT_ENABLEDand a valid OpenAI API key (getAIClient()). Otherwise returnspassedwith askippedmetric. Skips similarly when_aiCapBlockedis set. - Page selection — Fetches homepage via
fetchTarget. IfmaxPages > 1, discovers same-origin<a href>links (query/hash stripped, deduped) and analyzes up tomaxPages − 1additional URLs. Failed secondary fetches are skipped with a log warning. - Source extraction — Strips
<style>,<svg>, and<noscript>. Builds a string from<head>, inline<script>bodies, and<body>HTML. Truncates at 60 000 chars with a<!-- TRUNCATED -->marker. Saves extracted source as per-page.txtartifacts. - AI call — One chat completion per page (
temperature 0.2,max_tokens 4000, JSON response format) with a fixed system prompt covering six categories. User message wraps source betweenBEGIN/END TARGET PAGE SOURCEmarkers and lists up to 20 external script URLs. - Severity normalization — Model CRITICAL on non-
secretscategories is demoted to WARNING.secretsplus evidence matching known public key patterns (Stripepk_*, GA4, GTM, Algolia search keys, reCAPTCHA site keys, etc.) → INFO. - Aggregation — Issues deduped by
category:normalized-titlewith fuzzy token overlap for secrets (same key reported under different AI titles merges). Worst severity wins per aggregate. One finding per aggregate withevidence.pages/evidence.detailsper URL. - Run outcome — Any CRITICAL →
failed; else WARNING →warning; elsepassed. Score is the minimum AI page score across analyzed pages (0–100). Writesai-assessment.jsonartifact with per-page assessments.
Caveats and concerns
- Static HTML only — No browser execution; secrets or trackers injected only after JavaScript runs may be missed entirely.
- LLM variability — The same site can get different titles and severities between runs; prompts ask for precision but false positives and negatives are possible.
- Truncation — Large inline bundles beyond 60 k chars are cut off; issues in the truncated tail are invisible.
- Homepage-only link discovery — Internal links for extra pages come from the homepage HTML only, not from subsequent pages — deep routes may never be analyzed unless linked from
/. - Monthly schedule — Default cron is 1st of month 03:00 UTC; issues can persist weeks between runs.
- AI cost — One OpenAI call per analyzed page (up to 10); counts against team AI usage cap and billing.
- Prompt injection surface — System prompt instructs the model to ignore directives embedded in page source, but malicious content could still bias output.
- No CRITICAL except secrets — Even severe XSS patterns from the model are capped at WARNING unless categorized under
secretswith non-public material.
Source
packages/test-plugins/src/plugins/ai-source-review.ts