Engine internals

Architecture for developers. The authoritative documents are product-docs/01_PRD.md and product-docs/04_ARCHITECTURE.md; this page is the working map.

Repo layout

app/
  (marketing)/            landing, pricing, security, methodology, sample-report, docs (this site), sign-in + onboarding
  (app)/                  operator + agency portal (overview, agencies, brands, runs, reports, requests, settings + sub-pages)
  (print)/                /reports/[id]/print — the report's own paper-frame print view (D-32)
  r/[token]/               public white-label shared report (noindex, print-first) + /print
  api/                    health, run progress polling, CSV export, /api/v1/{reports,brands} (D-35)
src/
  engine/                 the measurement engine (below)
  db/                     drizzle schema, driver switch, DAL, seed, demo fixtures
  auth/                   argon2id + jose sessions
  lib/                    env validation, templates, run execution, WL accent guard
  components/             ui/ viz/ app/ marketing/
content/                  sample-report.json (committed fixture feeding /sample-report)
scripts/                  worker.ts, user-set-password.ts, regen-demo-analyses.ts, export-sample-report.ts
reference/python-engine/  untouched Python reference + parity export harness
tests/                    unit/ parity/ e2e/

Engine module map (src/engine/)

The engine is deliberately DB-free: executeBatch() takes providers and job keys and returns samples; persistence is the caller's job. The same code path is wrapped by the local worker and (Supabase era) Vercel Workflows — FR-3.4.

Provider adapters

Engine = { name, model, sample(), unitCostMicros, estSecondsPerCall, configured, maxConcurrency? }.

• DataForSEO LLM Responses (primary leg) — drives the canonical chatgpt and gemini engines against the real consumer surfaces via /v3/ai_optimization/{vendor}/llm_responses/live. Cost is per-model, not a single constant — measured live 25 Aug 2026 at $0.0142–$0.1085/call depending on model (gpt-5.6-luna cheapest, gpt-5.6-sol priciest; see MODEL_COST_MICROS in src/engine/providers/dataforseo.ts). An earlier flat $0.004/call estimate here was 6.8x low and the wrong shape — price varies by model, not by provider. Env: DATAFORSEO_LOGIN + DATAFORSEO_PASSWORD. Gemini grounding returns vertexaisearch.cloud.google.com redirect URLs, not publisher URLs — they are resolved concurrently at ingest inside the per-call timeout budget, pass-through on failure.

• Google AI Overviews (ai_overviews) — the AI answer Google puts above its own results, read from DataForSEO's /v3/serp/google/organic/live/advanced with load_async_ai_overview: true. $0.004/call, measured live 2 Sep 2026. There is no dedicated AI Overview endpoint; the block arrives as an ai_overview item beside the organic results, which are ignored. Depth is pinned at the endpoint minimum of 10 because the AI Overview is unaffected by it and every increment above is paid-for SERP we discard (depth 20 costs $0.006, depth 100 costs $0.022, for the identical answer). Env: DATAFORSEO_LOGIN + DATAFORSEO_PASSWORD.

• Google AI Mode (ai_mode) — Google's conversational search surface, /v3/serp/google/ai_mode/live/advanced. $0.004/call, measured live 2 Sep 2026. Same item shape as AI Overviews, and the same credentials.

  Both Google engines differ from every other engine in two ways. Neither names a model: Google does not disclose what writes these answers, so they record google-ai-overview:undisclosed / google-ai-mode:undisclosed rather than a fabricated pin (D-16, D-28). And AI Overviews are not always shown — when Google shows none, that is a sample with zero citations and answerPresent: false, never an error and never a retry, and the report prints it as a rate with its interval. An overview that Google did show but whose content failed to load is the opposite: an error sample, because publishing it as "shown, cited nobody" would understate the presence rate. Locale is recorded on the run (configSnapshot.locale, default US English 2840/en); these are the only engines that take one.

• Perplexity Sonar (cross-check leg) — the canonical sonar engine. ~$0.0063/call. Env: PERPLEXITY_API_KEY. Citations from search_results[].url, falling back to the legacy citations array. If the two legs diverge, we find out before a client does.

• Mock — deterministic offline engine (seeded mulberry32 PRNG, ported fixtures including error/no-vendor/activation behaviors). Selectable only in dev and demo, explicitly.

Missing credentials ⇒ configured: false. Preflight shows ✗, run creation rejects unconfigured engines server-side, and calling sample() on an unconfigured engine throws. The Python reference's silent mock fallback is deliberately not ported (decision D-7): you can't accidentally ship fixture data as measurement.

Run execution model

Statuses: queued → running → aggregating → complete | failed | canceled. Run creation snapshots the full config (prompts, engines, runsPerPrompt, brand aliases/domains, competitors) onto the run; execution, retry, aggregation, and reports read only the snapshot.

• Job derivation is stateless and race-free: the full sample keyset (promptId, engine, 0..runsPerPrompt−1) derives deterministically from the snapshot; remaining work is the keyset anti-joined against existing sample keys. No counters assign indexes.
• Claim protocol: a worker claims a run with a single conditional UPDATE … RETURNING (status queued/running, claim absent or stale > 90s), heartbeats claimedAt after every batch. The stale-claim sweep is the crash-recovery mechanism — kill a worker mid-run and any worker resumes it 90 seconds later.
• Batches of 16, concurrency 8, per-call timeout 30s → worst case ~60s per batch, far under serverless limits. Every per-call failure becomes an error sample, never a thrown batch.
• Derived counters: after each batch, completed/error/cost counters are recomputed from the samples table in one statement — never incremented (FR-3.5). Sample storage is idempotent on UNIQUE(runId, promptId, engine, sampleIndex).
• Retry failed: re-executes exactly the keys where error IS NOT NULL; the upsert overwrites only errored rows (ON CONFLICT DO UPDATE … WHERE existing.error IS NOT NULL); the run returns to running and re-aggregates.
• Cancellation: the executor re-reads status between batches; on canceled it stops, recomputes counters, and stamps finishedAt. The in-flight batch is allowed to finish.
• Aggregation: analyse() produces the Analysis JSON stored on the run. Cost never enters the Analysis — it is returned beside it, so agency/client render paths cannot leak what the snapshot doesn't contain.

Demo mode

DEMO_MODE=1: in-memory PGLite, migrated and seeded once per instance at cold start (2 agencies, 4 brands, 3 completed runs with reports and deterministic share tokens). New runs are mock-only, clamped to ≤ 60 samples (runsPerPrompt capped at 3, prompts truncated), and execute synchronously in-request — no worker, no cross-instance polling dependency. The progress endpoint synthesizes complete for unknown demo run ids. Writes are non-persistent by design.

The worker

pnpm worker (scripts/worker.ts) is the local/VM execution path: polls every 3s for queued or stale-claimed runs, claims via the protocol above, executes through the shared src/lib/run-exec.ts helper (the same helper the demo synchronous path uses), and is Ctrl-C-safe by idempotency. Point it at any database — PGLite locally or Supabase in production — via the same env as the app. In the Supabase era, Vercel Workflows wraps the identical executor with one workflow step per batch.

Health

GET /api/health → { ok, driver, demoMode } — the smoke test used throughout the runbook.