Skip to content

AI Transparency

Statement

ClaimGuard uses generative AI in exactly one production code path: the master_tool component sends claim narratives and evidence summaries to Google Gemini (gemini-2.5-pro) for analysis assistance. This page documents what is sent, what is not sent, what comes back, who sees the result, and how a customer can opt out.

There is no other LLM integration in production today. The application does not use OpenAI, Anthropic, or any non-Google generative-AI service, and the user-facing UI does not call any LLM directly.

The control sits at partial because the integration is implemented and the data flow is documented, but customer-facing transparency artifacts (an in-product disclosure, a per-org opt-out, a published data-handling commitment from the AI vendor) are still being formalized.

Implementation

Where AI is used

  • tools/master_tool/master_api.py — a Python FastAPI service on the production VM. It calls the Google Gemini API (gemini-2.5-pro) to assist with claim analysis. Configuration:
  • Model: gemini-2.5-pro (overridable via GEMINI_MODEL).
  • Temperature: 0.2 (overridable via GEMINI_TEMPERATURE).
  • Max tokens: 16384 (overridable).
  • Request timeout: 180 seconds.
  • Up to 3 retries with exponential backoff on transient errors.

The other AI / ML tooling in the repo (tools/c2pa, tools/ai_detect_baseline, tools/secondary_ai_detector, tools/dtect_image-0.1.0) performs detection / forensic analysis on uploaded images. They run locally on the VM and do not call out to external generative-AI services. They are pattern detectors and classifiers, not LLM-based generators.

What data is sent to Gemini

When the master_tool is invoked for a claim:

  • Sent: the claim's narrative text, structured evidence summaries, and the master prompt that frames the analysis task.
  • Not sent: raw uploaded files (images, video, documents), user PII fields beyond what's already in the claim text, authentication credentials, or the contents of unrelated claims.

What comes back

  • A structured JSON response from Gemini (per the schema in the master prompt) summarizing the model's analysis of the claim.
  • The response is stored in the application database alongside the claim and is shown to authorized users on the claim-detail UI.
  • The response is assistive, not authoritative. A human reviewer (REVIEWER or ADMIN role) makes the final decision on the claim. The application surface labels the AI-generated text as such.

Who sees the AI output

Per the Authorization model:

  • Users in the same org as the claim can see the AI output on the claim-detail page (subject to the same role-and-org filter as any claim data).
  • Super-admins can see cross-org output.
  • The AI output is included in claim exports if the role allows.

Authentication and data flow

  • The Gemini API key (GOOGLE_API_KEY / GEMINI_API_KEY) lives in GCP Secret Manager as claim-guard-google-api-key. The Node backend fetches it from Secret Manager at boot via the VM's claim-guard-vm SA (which holds per-secret roles/secretmanager.secretAccessor on that secret). The Python master_tool does NOT call Secret Manager itself; it reads the key from its environment (GOOGLE_API_KEY), which pm2 injects from tools/master_tool/.env per the ecosystem.config.cjs config. The two ingestion paths read the same value of the same secret, but the Python side relies on the operator keeping tools/master_tool/.env in sync with Secret Manager. This is documented in Secrets management.
  • Outbound requests use TLS to the Gemini API endpoint. Outbound HTTP is wrapped by safeFetch in the Node backend; the Python master_tool uses Google's official google-genai SDK directly and inherits its TLS posture.

Vendor data-handling commitments

The Gemini API is operated by Google and falls under the same Google Cloud DPA enumerated on Subprocessors. The model-provider side data-handling commitments (e.g., training-on-prompts opt-outs, content-filter behavior) are governed by Google's Generative AI Terms of Service for the API tier.

How a customer can opt out

Honest current state (2026-05-02): the operator-side opt-out schema and backend gate exist in code on the chore/weekend-hardening-2026-05-01 branch but are not yet deployed. Once the branch lands and migration 022 runs in prod, an operator can opt an org out via SQL:

UPDATE organizations
SET ai_analysis_enabled = false
WHERE id = '<org-uuid>';

The gate sits in server/src/services/analysisQueue.js next to the existing shouldRunLayover(org) check, so a single early-return short-circuits both master_tool POST sites (/analyze-claim-summary and /analyze-evidence-comparison) for the org's claims. Layover image-classification analysis is also skipped for opted-out orgs, matching "opt out of AI analysis" semantics. Default is TRUE — existing orgs continue to receive analysis after the migration runs.

There is no in-product self-service opt-out yet — flipping the flag still requires operator SQL access. A per-org admin-UI toggle is the next item on the roadmap below; per-claim granularity follows.

Status

partial — verified 2026-05-02.

What's in place:

  • A documented and code-grounded AI integration: one model, one vendor, one code path. Configuration values (gemini-2.5-pro, temperature 0.2, max tokens 16384, 180s timeout, 3 retries) all match tools/master_tool/master_api.py.
  • A stable contract for what gets sent (narrative + evidence summaries) and what does not (raw files, credentials, unrelated data).
  • A human-in-the-loop reviewer workflow for every claim.
  • AI output is stored, role-and-org-scoped, and labeled.
  • Operator-side opt-out gate scaffolded in code (default-on) on chore/weekend-hardening-2026-05-01. Awaits Monday review + deploy + migration 022 before it can be exercised in prod. See "How a customer can opt out" above.

Known gaps

  • Opt-out gate exists in code but is not yet deployed. Until chore/weekend-hardening-2026-05-01 merges and migration 022 runs in prod, the runtime behaviour is unchanged from the pre-2026-05-01 state — every claim with the right inputs invokes the prompt. Once deployed, operator SQL is the only opt-out surface (no admin UI yet).
  • No in-product transparency disclosure. The claim-detail UI labels AI-generated content but does not link to a customer-facing AI-use page.
  • No per-claim or per-user opt-out granularity — listed even though the org-level path also doesn't exist, because future opt-out work will need to decide on granularity up front.
  • No stronger published data-handling commitment than the default Gemini API ToS coverage (e.g., a "no training on our prompts" written commitment from the vendor on a stronger plan).
  • No LLM-output evaluation cadence. The model's behavior is not periodically re-tested against a held-out evaluation set; if Google changes the model behind the same model name, we will not see it.
  • No formal AI risk register distinct from Risk management.

Roadmap

  • Operator-side opt-out flag — in-flight on chore/weekend-hardening-2026-05-01. Migration 022 adds the column; the gate lives in analysisQueue.js at the dispatch level so both master_tool POST sites are short-circuited together. Default-on. Closes once the branch is merged + the migration runs in prod.
  • Per-org AI opt-out toggle in the admin UI, persisted on the same column.
  • Per-claim opt-out for sensitive claims.
  • In-product AI-use disclosure — a one-screen affordance that explains where AI is used, what data is sent, and how to opt out.
  • Pin model behavior by either pinning to a versioned model identifier (e.g., gemini-2.5-pro-001 vs the floating alias) or by running a periodic evaluation set against the live model.
  • Stronger vendor-side commitments evaluated when a customer requires them.